[SOLVED] Can I have some help writing a recursive move command involving files with the same name

[SilverCookieDust]

(Hi, newbie here [former Windows user]. Sincere apologies if this is the wrong forum.)

I have a folder on my desktop named "Archive", which contains some txt files and a couple of dozen subfolders, each of which have one or two subfolders and millions of txt files between them. I would like to move all the txt files into "Archive". Some of the txt files have the same name, but may or may not have different contents, and I want to keep all of them so when conflicts occur during the move they'll need to be renamed, preferably without it asking me every single time. (Or, failing that, a command that skips moving any files that have the same name as a file already in "Archive", and I'll run a second command after to move them into a different folder.)

I know I can use mv *.txt for this, but I'm not sure what else to put. I know -r is for recursive stuff, but I heard it doesn't work with mv? Also, if I'm running the command line from the "Archive" folder, do I put /Archive as the destination or do I need to specify /Desktop/Archive? (Or have I got it completely wrong and it's something else entirely?)

Solved using rene's code:

Code: Select all

find -mindepth 2 -name "*.txt" -exec mv --backup=numbered -v \{} . \;

[Hoser Rob]

Code: Select all

man mv

[WharfRat]

You can use the -n, --no-clobber option with mv which will not overwrite an existing file.

Just out of curiosity how many files are there find ~/Desktop/Archive -type f |wc -l

[rene]

You can use the --backup option of mv for this, applied through a find -exec. I.e., from ~/Desktop/Archive:

Code: Select all

find -mindepth 2 -name "*.txt" -exec mv --backup=numbered -v \{} . \;

This finds all *.txt files in subdirectories of the current directory (note, not all *.TXT or *.Txt or ... files: we're case-sensitive) and for each of them invokes mv --backup=numbered -v <the file> .. Given that you can't really "--dry-run" something like this first you'd ideally run this on a test directory first to see if it does what you want.

[SilverCookieDust]

You can use the --backup option of mv for this, applied through a find -exec. I.e., from ~/Desktop/Archive:

Code: Select all

find -mindepth 2 -name "*.txt" -exec mv --backup=numbered -v \{} . \;

This finds all *.txt files in subdirectories of the current directory (note, not all *.TXT or *.Txt or ... files: we're case-sensitive) and for each of them invokes mv --backup=numbered -v <the file> .. Given that you can't really "--dry-run" something like this first you'd ideally run this on a test directory first to see if it does what you want.

I was going to try it on a test directory, but I didn't realise pasting it right into the terminal would immediately set it off. Fortunately I do have backup of this folder if this deletes stuff or whatever, but when I look in the folder it does seem to be doing what I want. I'll see how things are when it's done.

Just out of curiosity how many files are there find ~/Desktop/Archive -type f |wc -l

I tried that code and it returned an error about wc invalid? That said, I had to alter the file path because Archive is actually a subfolder whose parent had spaces in it. (Didn't seem relevant to mention before.) Not sure if I put it right to account for the spaces. (I tried "/Desktop/Files - Docs/Archive" and /Desktop/"Files - Docs"/Archive and they both returned the same error.)

I opened the properties window for the folder, but so many files take a while to add up in those and I accidentally set off the above code while it was doing it and I don't know if that might have interrupted to count. It's stopped at 1,027,053 items, 49.4 GB, which is a lot less than I expect. I know the "Files - Docs" comes in at over 300GB and nearly 7 million files because that's what it said when I was copying it from my external HD. (And I'm gonna swear something awful if "Archive" isn't the biggest folder because the other ones are even worse of a mess.)

[phd21]

Hi SilverCookieDust,

Welcome to the wonderful world of Linux Mint and its excellent forum!

I just read your post and the good replies to it. Here are my thoughts on this as well.

It would help to know more about your system setup. If you run "inxi -Fxzd" from the console terminal prompt, highlight the results, copy and paste them back here, that should provide enough information.

I have a folder on my desktop named "Archive", which contains some txt files and a couple of dozen subfolders, each of which have one or two subfolders and millions of txt files between them. I would like to move all the txt files into "Archive". Some of the txt files have the same name, but may or may not have different contents, and I want to keep all of them so when conflicts occur during the move they'll need to be renamed, preferably without it asking me every single time. (Or, failing that, a command that skips moving any files that have the same name as a file already in "Archive", and I'll run a second command after to move them into a different folder.)

"Millions" of text files?

Unlike MS Windows it's probably better to have that "Archive" folder somewhere else other than your Desktop folder like underneath your Documents or Home folder. You should be able to just "copy to" or "move to" the parent and or subfolders using your file manager to the "Archive" folder.

Is this for the purpose of actually creating an archive file (.zip, .7z, etc...)? Or, trying to strip off many subfolders?

command line - Recursively copy files from one directory to another - Ask Ubuntu
https://askubuntu.com/questions/802238/ ... to-another

Code: Select all

find /home/yourusername/Desktop/Archive/ -iname '*.txt' -exec mv {} /home/yourusername/Desktop/Archive \;

linux - How do I move files out of nested subdirectories into another folder in ubuntu? (Trying to strip off many subfolders)
https://superuser.com/questions/658075/ ... -in-ubuntu

Code: Select all

find ~/Desktop/Archive/ -type f -print0 | xargs -0 mv -t ~/Desktop/Archive

or

Code: Select all

find ~/Desktop/Archive/ -type f -iname "*.txt" -exec mv --backup=numbered -t ~/Desktop/Archive/ {} +

Hope this helps ...

[rene]

I was going to try it on a test directory, but I didn't realise pasting it right into the terminal would immediately set it off.

Appears you pasted it including a trailing newline (enter) then But yes, it'll do what you requested; I wouldn't myself necessarily be enthusiastic about the additional .~N~ suffix (N a number) that duplicates get, but renaming those to something more to your liking in a second step if so desired is a possibility.

As to the wc thing I expect you misread/typed "-l" (small l) as for example "-1" (one).

[SilverCookieDust]

As to the wc thing I expect you misread/typed "-l" (small l) as for example "-1" (one).

Ah, yeah, that's what I did. Done it correctly now; output: 6,724,925.

The mv was working... until the whole system froze up. I don't know if maybe it got overwhelmed by the amount of files it's working with, but it's a start and I don't mind having to do multiple runs, so thank you for all your help.

It would help to know more about your system setup. If you run "inxi -Fxzd" from the console terminal prompt, highlight the results, copy and paste them back here, that should provide enough information.

Code: Select all

System:    Host: ZJB-Linux Kernel: 4.15.0-39-generic x86_64
           bits: 64 gcc: 7.3.0
           Desktop: Cinnamon 3.8.9 (Gtk 3.22.30-1ubuntu1)
           Distro: Linux Mint 19 Tara
Machine:   Device: desktop System: ASUS product: All Series serial: N/A
           Mobo: ASUSTeK model: H81I-PLUS v: Rev X.0x serial: N/A
           BIOS: American Megatrends v: 0805 date: 03/05/2014
CPU:       Quad core Intel Core i5-4690 (-MCP-) 
           arch: Haswell rev.3 cache: 6144 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 27998
           clock speeds: max: 3900 MHz 1: 3808 MHz 2: 3750 MHz 3: 3866 MHz
           4: 3739 MHz
Graphics:  Card-1: Intel Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller
           bus-ID: 00:02.0
           Card-2: NVIDIA GK106 [GeForce GTX 660] bus-ID: 01:00.0
           Display Server: x11 (X.Org 1.19.6 )
           drivers: modesetting,nouveau (unloaded: fbdev,vesa)
           Resolution: 1920x1080@60.00hz
           OpenGL: renderer: NVE6 version: 4.3 Mesa 18.0.5 Direct Render: Yes
Audio:     Card-1 Intel 8 Series/C220 Series High Def. Audio Controller
           driver: snd_hda_intel bus-ID: 00:1b.0
           Card-2 NVIDIA GK106 HDMI Audio Controller
           driver: snd_hda_intel bus-ID: 01:00.1
           Card-3 Intel Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
           driver: snd_hda_intel bus-ID: 00:03.0
           Sound: Advanced Linux Sound Architecture v: k4.15.0-39-generic
Network:   Card: Realtek RTL8111/8168/8411 PCIE Gigabit Ethernet Controller
           driver: r8169 v: 2.3LK-NAPI port: d000 bus-ID: 04:00.0
           IF: enp4s0 state: up speed: 100 Mbps duplex: full mac: <filter>
Drives:    HDD Total Size: 1500.3GB (24.3% used)
           ID-1: /dev/sda model: ST500DM002 size: 500.1GB
           ID-2: /dev/sdb model: ST1000DM003 size: 1000.2GB
           Optical-1: /dev/sr0 model: PIOR BD-ROM  BDC-207D
           rev: 1.00 dev-links: cdrom,cdrw,dvd,dvdrw
           Features: speed: 125x multisession: yes
           audio: yes dvd: yes rw: cd-r,cd-rw,dvd-r state: running
Partition: ID-1: / size: 458G used: 340G (79%) fs: ext4 dev: /dev/sda1
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors:   System Temperatures: cpu: 57.0C mobo: 27.8C gpu: 33.0
           Fan Speeds (in rpm): cpu: 0
Info:      Processes: 208 Uptime: 4 min Memory: 1762.1/7848.0MB
           Init: systemd runlevel: 5 Gcc sys: 7.3.0
           Client: Shell (bash 4.4.191) inxi: 2.3.56 

Note that Linux is running from the 500GB HDD as primary boot; the 1000GB still has Windows 7 on it.

"Millions" of text files?
[...]
Is this for the purpose of actually creating an archive file (.zip, .7z, etc...)? Or, trying to strip off many subfolders?

At minimum, 1 million (the count the properties window reached last time I opened it). According the wc command another user suggested, 6,724,925. (Though I haven't quite figured out how that works; searching the wc command says it counts words/lines/bytes, but I'll look into more as I learn.)

I want to strip out the many subfolders for now. Later I'll go through the files to delete duplicates that are exact copies as well as any other unnecessary files, and then I'll redistribute to subfolders that are better named and organised than the current ones. I named the folder Archive because it's a collection of documents that I keep stored but rarely use (I'm an obsessive compulsive hoarder).

Unlike MS Windows it's probably better to have that "Archive" folder somewhere else other than your Desktop folder like underneath your Documents or Home folder. You should be able to just "copy to" or "move to" the parent and or subfolders using your file manager to the "Archive" folder.

Thanks for the tip, but can I ask what's the advantage to having it under the Documents or Home folder?

Also, you linked to some questions about Ubuntu; I was a bit uncertain when searching myself, but does that mean any (or most) commands that work on Ubuntu also work on Mint? I could never quite figure out the difference in the types of Linux and I mostly settled on Mint because I love anything mint-flavoured. (Uh... not that I'm trying to eat my PC or anything. Promise )

[WharfRat]

If you use find "~//Desktop/Files - Docs/Archive" -type f you'll see it returns every file (not folders) in that hierarchy.

The wc -l returns the count of the lines so that's how many files you would have.

What does du -sh "~//Desktop/Files - Docs/Archive" show for size maybe you're running low when moving

The desktop folder is used to display/launch applications from the desktop. There's an 'Add to desktop' option when right-clicked from the application menu.

And to answer your final question Mint is Ubuntu based so commands that work on Ubuntu will also work on Mint and most other Linux distributions.

[rene]

Later I'll go through the files to delete duplicates that are exact copies [ ... ]

As to that part; from the directory with now all the *.txt files and their former namewise-duplicates now named with a .~N~ suffix:

Code: Select all

shopt -s nullglob; for TXT in *.txt; do for BAK in "$TXT".*; do cmp "$TXT" "$BAK" && echo rm "$BAK"; done; done

Remove the "echo" once you're satisfied this does what you want it to. I.e., safe to try; doesn't do anything with the echo other than print what it would do.

As to your system freezing; definitely shouldn't have, but perhaps Nemo (the Cinnamon file manager) was being it's usual chronically-broken-yet-never-repaired self. If it was trying to refresh a directory with 6 million files I can't even hold it against it all that much...

[EDIT] and come to think of it, with that many files you probably want "cmp" itself to be silent: make "cmp" in the above be "cmp -s".

[SilverCookieDust]

What does du -sh "~//Desktop/Files - Docs/Archive" show for size

317G

And to answer your final question Mint is Ubuntu based so commands that work on Ubuntu will also work on Mint and most other Linux distributions.

Good to know, thanks.

If it was trying to refresh a directory with 6 million files I can't even hold it against it all that much...

Yeah, me neither. Thanks again for more code.

[phd21]

Hi SilverCookieDust,

I want to strip out the many subfolders for now. Later I'll go through the files to delete duplicates that are exact copies as well as any other unnecessary files, and then I'll redistribute to subfolders that are better named and organised than the current ones. I named the folder Archive because it's a collection of documents that I keep stored but rarely use (I'm an obsessive compulsive hoarder).

I would make a backup of these original folders and files before running any re-organizing (copying or moving) commands so that if something did not work as you expected, you do not lose anything.

Thanks for the tip, but can I ask what's the advantage to having it under the Documents or Home folder?

The main advantage would be organizational, documents like text files under documents. You could still have a Desktop shortcut launcher "link" to the "archive" folder if you wanted that.

Also, you linked to some questions about Ubuntu; I was a bit uncertain when searching myself, but does that mean any (or most) commands that work on Ubuntu also work on Mint? I could never quite figure out the difference in the types of Linux and I mostly settled on Mint because I love anything mint-flavoured. (Uh... not that I'm trying to eat my PC or anything. Promise )

Most references to Ubuntu will work with Linux Mint which their main versions are based on as long as the versions match. Linux Mint 19.x is based on Ubuntu 18.04, Linux Mint 18.x is based on Ubuntu 16.04, etc... There can be some differences with commands between the versions, especially with security-related commands. I do not think the copy or moving commands will matter though with non-root folders.

I just ran this command below to copy any music mp3 file in my home Music folder and its subfolders to an "archive" folder in my home folder and it worked perfectly. The tilda "~" represents the "/home/YourUserName/". Replace "cp" for copy with "mv" to move them. Also, uppercase and lowercase characters matter in Linux.

FYI: click "select all" above a code box to highlight the command, then right-click it to copy it (or Ctl+Insert, or Ctl+C), then you can paste that command using right-click paste or Shift+Insert or Ctl+Shift+V.

Code: Select all

find ~/Music -type f -iname "*.mp3" -exec cp --backup=numbered -t ~/archive {} +

So, using your original statements, this command below should work to move all ".txt" files from all subfolders in the "/Desktop/Archive" folder to its top parent folder "/Desktop/Archive", and it may take some time on that many files. Obviously, you can change the source and destination folder path locations. You will still have a bunch of empty folders (directories) to delete afterward, yet if you created a new empty Archive folder elsewhere like "/Documents/Archive" and moved all the txt files to that folder, you could easily delete the original (source) folder and all of its now empty subfolders.

Code: Select all

find ~/Desktop/Archive -type f -iname "*.txt" -exec mv --backup=numbered -t ~/Desktop/Archive {} +

or use this command to move all the txt files to a new Archive folder underneath the Documents folder, create the new folder first.

Code: Select all

find ~/Desktop/Archive -type f -iname "*.txt" -exec mv --backup=numbered -t ~/Documents/Archive {} +

Hope this helps ...

[SilverCookieDust]

I would make a backup of these original folders and files before running any re-organizing (copying or moving) commands so that if something did not work as you expected, you do not lose anything.

Oh yeah, I definitely did that. I know better than to risk losing so many files.

I'm running rene's code and it's working so I haven't used yours, but thank you for all your help and advice anyway.

[phd21]

Hi SilverCookieDust,

You are welcome from all of us that replied...

That's good you backed up first.

Use whatever commands (code) that work for you and your task.

FYI: Those are not my console terminal commands (code) but from those articles. Although I modified some of it for your situation.