Varying File Transfer Times & Speeds

[dave1953]

Always wondered - why can't any file manager in Windows or Linux accurately predict the time taken to transfer file folders without keep changing its' mind and why does the speed of file transfer keep changing all the while during the process? For example - I wanted to transfer some folders from my Data Storage drive to my Archive drive (slow drive, I appreciate this is very slow due to its' age and specification, fair enough) - but the process started at 74 mb/sec predicted timescale of approximately 1hr 20mins, then after it had already moved 45 gb of files (14% of total) the time increased to 2hrs 6mins and the speed dropped to 35.9 mb/sec. Being diagnosed on the autistic scale I cannot follow why the speed slows and the time increases when the system has already transferred 14% of the files. Surely the time should always go down and the speed stay the same or thereabouts? Currently on Nemo, (Linux Mint Cinnamon) by the way.

[LanceM]

It's normal and has to do with the buffering and write process. Faster drives help a lot and choosing a different file system than NTFS or FAT32 helps. exFAT is much better. Ext4 is much faster. F2FS for flash drives is fast.

[dave1953]

Thanks. All of the five drives are formatted ext4 by the way; Internal Drives - 1) boot (root & system) drive; 2) spare 240gb ssd for parking folders whilst working on them; 3) 6tb SATA III Data drive, External Drives - 4) 6tb Data Backup Drive; 5) Archive Drive for very old media likely to be deleted and old Windows Drivers, software etcetera. Linux Mint only system now for a few years.

[LanceM]

One of my friends is using Dolphin file manager and gets greatly improved file transfer speeds with it. It's in the Software Manager.

[rene]

While Dolphin may be a good choice over Nemo (especially, I believe, with USB targets) as to the general question and expanding on LanceM's initial reply...

Yes, buffering or more fully (write-) caching is the culprit as to times/speed uncertainty at the level of e.g. Nemo. The kernel allows some given amount of data to be in-flight at any given time unless very specifically told to not to. This is to say that a copy starts off copying to RAM and as such very fast, whereas only after that RAM buffer has filled the speed with which it can be emptied onto the actual drive to make room for more incoming data determines how fast Nemo can dump yet more data into that buffer.

However, and even though I don't believe it's much better in Windows, certainly on Linux where something like Nemo is fully independently developed and miles removed from the kernel which does said buffering, it can be said that Nemo does not even know about any kernel-sides buffering/caching. As soon as the syscall that it ends up calling to transfer the data returns from the kernel indicating success the data has been successfully transferred to the destination as far as Nemo is concerned. At the start very fast then and only after enough data has been put into the kernel-sides buffer(s) at a realistic speed for the destination drive.

Better, it does of course "know" as such, but doing something about it means copying anything synchronously from the very start: telling the kernel to not return from the mentioned syscall until it synced the data to the actual destination --- but that's not why you have a fast, write-caching kernel at the ready. When done generally this slows down your system massively.

Random other remark: that slow drop to 36 MB/s likely says we're talking about USB2. If not Nemo may itself be contributing to the above general issue and would as such be worth investigating indeed.

[LanceM]

Some drives have large caches built into their firmware. I'm thinking that must be a speed assist. More expensive SSDs tested for reviews show big gains in file transfers.
Like https://www.pcworld.com/article/3440740 ... rd-of.html

[rene]

Yes, true enough, also device-side buffers add to that same mechanism of allowing "true" write speed to only be known once said buffer is due to constantly incoming data in basically write-through mode. As compared to the kernel-side caching more theoretically though; a device-side write-buffer is never só large that it'll hold gigabytes of in-flight data: even though it may have a large buffer, that'll mostly be a device-sides read-cache, with the write-cache severely limited also for e.g. power-loss scenarios.

[all41]

Good replies here.

The cache is like a regulator--it allows input by the litre, but dispenses it at ml rates
as needed by the destination.

That's why the transfer speed reports are initially exaggerated--cache filling--files get cached quickly
and get dispensed as needed.

Also know, files are often still being written to external media after the progress window closes---causing many
to remove or eject mediums before the write processes have completed.

After the progress window closes on big files open a terminal and run sync
When that sync command returns to a command prompt $ then the write is finished--not before.

[dave1953]

Acknowledged and thanks to all for insightful comments. Much appreciated. And yes the lowly archive drive is usb 2. The speed wasn't the issue. It's the changing from one time length to another and the fluctuations in data transfer speed. I know I have to live with it but being insatiably curious I wondered why it occurred. Thanks again for all your input.

[rene]

Seems then you managed to decode it anyway, but when rereading took the opportunity to fix a too badly broken sentence...

[dave1953]

Re: Dolphin. Must confess never used it much - simply because it does not allow root access on a 'quick and easy' basis. If I am using a kde installation for any length of time I tend to install Nautilus, Nautilus-Admin and Nautilus-Share rather than faff around with Dolphin, but intrigued by your observation about theoretical faster file transfer might give it another go. To be fair, only ever use root for editing fstab, cups for my Ricoh printer driver etc which I could use nano and terminal commands for. Just Lazy I suppose using Nemo/Nautilus.

[LanceM]

You can make Dolphin the default and put a launcher on the panel, then Nemo in your favorites. When needing root, just launch Nemo.

[dave1953]

Yes, great idea thanks very helpful I shall give this a spin (sorry for pun).

[LanceM]

I just tried Dolphin vs Nemo and timed a 19GB file transfer from Desktop to USB 30. HDD with Ext4 and it was exactly the same.