SOLVED Copying onto a USB drive

[Martin1001]

I have a large folder named ARCHIVE on the laptop with many subfolders containing videos, jpegs, and text files. It's a family archive and I wanted copies for four family members. So in Dec 2021 using the program

Code: Select all

rsync -avh --delete --exclude=".*" ~/ARCHIVE/ /media/martin/FRED

I made a mirror copy for Fred onto a new 32GB Samsung USB drive, and mirror copies for three other family members, all onto identical and new 32GB Samsung USB drives.

Since then there have been additions to the folder ARCHIVE, so today in Nov 2022 I started to use the same program to update the four USB mirror images of the laptop folder (now containing 2,079 items, totalling 29.3 GB).

The update onto the first USB drive took some time, but not too excessive a wait. I checked the result and all was fine.

I then ran the update onto the second USB drive, but that took considerably longer than the first update. I checked the result, and it was again fine. But, what puzzles me is that the update ought to my thinking to have taken exactly the same amount of time. I can see that I soon need to invest in four 64GB drives to replace those 32GB drives, but that doesn't answer why the second update took so very much longer than the first. Perhaps I ought to have rebooted between the first and second operation? If so, why? Or might there be some other reason for the vast difference in update time?

A further question is, with so many files and subfolders within the folder ARCHIVE, would it be faster to reformat the USB drive and do a simple copy rather than use rsync?

[Doruletz]

A further question is, with so many files and subfolders within the folder ARCHIVE, would it be faster to reformat the USB drive and do a simple copy rather than use rsync?

Of course it will be the easiest way. Do that 4 times, and you're done.
Try to always keep things simple.
Just like update from say Mint 20 to Mint 21 vs wipe the drive and fresh install Mint 21...
Good luck.

[kato181]

Not ALL usb sticks are the same, just because they are the same brand and size doesn't mean the speed will be exactly the same, the same applies to all storage devices, also to recording media like cd/dvd I have found.You could compress all the data into one file and keep the master data files open without compressing it and when you add more files just create another compressed file with the same name and copy it over to the usb sticks and it will ask if you want to overwrite the existing file, just select yes.

[mikeflan]

with so many files and subfolders within the folder ARCHIVE, would it be faster to reformat the USB drive and do a simple copy rather than use rsync?

I would not expect that to be faster, but who knows.

[Martin1001]

Going through other threads I find comments like: "on my system 100 GB takes about 90 mins to copy to my external hard drive" from which I presume that it doesn't much matter what I do, use rsync, or reformat then block copy and paste, or some other method, my 32GB to a USB drive is going to take a long time anyway.

[mikeflan]

That is only 19 MB / sec. I get about 7 times that (around 130 MB / sec) going from internal EXT4 to external EXT4 over USB3.

[Petermint]

If a USB stick is mounted and 60% full, the controller on the USB stick has a chance to erase unused space in the 40% unused. The next rsync has lots of space to write on before requiring more erases. A 95% full drive does not have that option. Writing to several 95% full drives can produce huge differences in speed because thet have accumulated different erased blocks and other factors.

When using real USB SSDs, you can issue TRIM to force the erase of unused blocks. There can be differences in temperature as big writes cause big erases which overheat the drives. You can also have differences in use of the Linux file cache.

Whew. So many complications. An exact comparison would require a reboot between each rsync.

[Martin1001]

The relatives who have received copies of the archive have either no desire or time to get involved in the technicalities of the various systems they use, of which none are Linux. For which reason the copies are on FAT32 USB drives. So unfortunately several of the appreciated suggestions can't be implemented, but are noted for the future in other contexts.

If a USB stick is mounted and 60% full, the controller on the USB stick has a chance to erase unused space in the 40% unused. The next rsync has lots of space to write on before requiring more erases. A 95% full drive does not have that option.

Happily that does suggest a way forward with the present problem. The next time I need to update, I shall in any case have to copy onto USBs of larger capacity, and the larger the capacity, the greater the % of unused space. So, provided I use USBs of sufficiently large capacity, that should speed up any subsequent updates.

Thank you to all who contributed.