File transfer hangs for a long time before finishing

Please post suggestions for improvement of Cinnamon on:
https://github.com/linuxmint/Cinnamon
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
User avatar
derion1
Level 1
Level 1
Posts: 19
Joined: Sun Nov 03, 2019 8:08 am
Location: Denmark
Contact:

File transfer hangs for a long time before finishing

Post by derion1 »

Hi.
When copying or moving files from my SSD to USB flash drive (USB 3.0) it always starts super fast and then it pauses for a really, really long time (at, say, 64%) before the transfer is finished. I read previous treads, and people are writing that it's got to do with buffer times and whatnot, but is there a solution to this in Nemo/Cinnamon?
I tried to to the same process in XFE file manager and it is fast as lightning. No problems in Windows 10 neither.

Here's a screenshot (not that it shows a lot) of what I'm writing about.

Image
------------------
Desktop Monologues
User avatar
JoeFootball
Level 13
Level 13
Posts: 4674
Joined: Tue Nov 24, 2009 1:52 pm
Location: /home/usa/mn/minneapolis/joe

Re: File transfer hangs for a long time before finishing

Post by JoeFootball »

derion1 wrote: I tried to to the same process in XFE file manager and it is fast as lightning.
I too experience the same issue when transferring large amounts of data, such as the 6.8 GBs in your example, with Nemo and some other GUI file managers.

Xfe is my solution as well, or Midnight Commander, or just the cp command in the terminal.
User avatar
AZgl1800
Level 20
Level 20
Posts: 11146
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes Sweeping down the Plains
Contact:

Re: File transfer hangs for a long time before finishing

Post by AZgl1800 »

that is not restricted to Linux,
I just downloaded 3 Win7 ISO files, and after they were CRC verified, copied them over to a flash drive....

Exact same thing you mentioned, Win7 started real fast, slowed down, then at 98% or so, it almost seems like nothing happens.

It has to do with clearing the copy buffers and making sure the file is complete. ( and more I'm sure )
LM21.3 Cinnamon ASUS FX705GM | Donate to Mint https://www.patreon.com/linux_mint
Image
HilltopsGM
Level 4
Level 4
Posts: 231
Joined: Thu Mar 15, 2012 8:11 pm

Re: File transfer hangs for a long time before finishing

Post by HilltopsGM »

I'd like to +1 this . . . .

I have been running into this issue for some time with mint (all through 19.x and if I remember correctly 18.3 as well).

File transfer starts of fast, slows down, then stops at about 99% and just sits there and does nothing.
If I wait a REALLY long time it MIGHT 'finish', but not always.

Has anyone got something new we might try to solve this issue?

Thanks
HilltopsGM
Level 4
Level 4
Posts: 231
Joined: Thu Mar 15, 2012 8:11 pm

Re: File transfer hangs for a long time before finishing

Post by HilltopsGM »

Has anyone got a solution for this problem?

Same computer, same USB, same USB port, same file being copied . . . the ONLY difference is the OS.

Login to Mint 19.2 and copy the file to the USB drive and it haaaaaaaaaaangs forever before completing.
Login to Windows 10 and copy the file to the USB drive and it finishes quickly - in fact it would hang so long in Mint that it was quicker to kill everything, shutdown, restart windows and THEN copy the file that it was to wait for it to finish.

Anyone?
User avatar
JoeFootball
Level 13
Level 13
Posts: 4674
Joined: Tue Nov 24, 2009 1:52 pm
Location: /home/usa/mn/minneapolis/joe

Re: File transfer hangs for a long time before finishing

Post by JoeFootball »

HilltopsGM wrote: . . . the ONLY difference is the OS.
And what are you using in Linux Mint?
Synthetase
Level 1
Level 1
Posts: 17
Joined: Wed May 20, 2020 8:22 am
Contact:

Re: File transfer hangs for a long time before finishing

Post by Synthetase »

I have noticed the same thing on Mint since v17, but I always thought it was a bug in the progress bar, not a problem with the transfer speed itself. If you have a USB drive with an LED, you can see it's still blinking away while the progress bar appears to have stopped, when the LED stops blinking, the dialog disappears because the copy has finished. At least that seems to be the behaviour for USB 2 devices (~5MB/sec copy speed maintained over the time period).
User avatar
AZgl1800
Level 20
Level 20
Posts: 11146
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes Sweeping down the Plains
Contact:

Re: File transfer hangs for a long time before finishing

Post by AZgl1800 »

I'm of the opinion, that the Progress Bar is not an Exact Science as to how long it is going to take to perform the job.

Initially, the File Managers have to run thru the list of jobs, collect names, locations, and then start the actual copy process.

Once the actual copy process itself has started, there is no way for that progress bar to know how long it is going to take to finish the job.

I suppose someone who cares, could redesign the action of the progress bar to indicate the Percentage of actual data transferred, relative to the total of all of the files combined.

In the Windows environment, an after market utility called TeraCopy comes the closest to have a Progress Bar that is "real time" to the job.

https://www.codesector.com/teracopy

But, even with that utility, the same action is obvious.
LM21.3 Cinnamon ASUS FX705GM | Donate to Mint https://www.patreon.com/linux_mint
Image
User avatar
n0body_special
Level 3
Level 3
Posts: 130
Joined: Sat May 16, 2009 3:00 pm

Re: File transfer hangs for a long time before finishing

Post by n0body_special »

this bug has been around since forever. I'm surprised that nobody has a real solution yet.
No, it's not a matter of cache memory being used for writing, or inaccurate progress bar. Copying freezes (and not necessarily at 99%, but also at 10%). Freezes means I waited for it to finish a long time -I left the house and returned after 3 freaking hours. 5 GB of data should have been copied by then.

The vm.dirty_bytes=15000000 addition in /etc/sysctl.conf unfreezes but slows down copying by A LOT (400 kb/s) so it's no real solution. 5 GB takes 1,5 hours according to the progress bar - no I didn't wait that long.

I have noticed this problem since at least 6 years with different Mint versions but didn't always happen, haven't managed to find a pattern yet
Attachments
1.png
1.png (27.95 KiB) Viewed 14372 times
Windows is the world's biggest virus.
UEFI is a Microsoft sponsored boot sector hijacker.
Android is the world's biggest spyware.
But most importantly: Google. Is. Skynet
User avatar
JoeFootball
Level 13
Level 13
Posts: 4674
Joined: Tue Nov 24, 2009 1:52 pm
Location: /home/usa/mn/minneapolis/joe

Re: File transfer hangs for a long time before finishing

Post by JoeFootball »

n0body_special wrote: I'm surprised that nobody has a real solution yet.
What application are you using when this occurs? Have you tried Xfe, Midnight Commander, or the cp command? Those have been successful for me on multiple installations.
User avatar
n0body_special
Level 3
Level 3
Posts: 130
Joined: Sat May 16, 2009 3:00 pm

Re: File transfer hangs for a long time before finishing

Post by n0body_special »

Double commander does exactly the same, and it has its own interface. I haven't tried XFE, MC or cp command yet. I may try them some time - but I'm interested in a solution, and using another file manager than the system's default is not really a solution, is it.

Now I have disabled the vm.dirty_bytes=15000000 trick, and it has returned to freezing (or really slow copying?) at 29%. Double commander too
Windows is the world's biggest virus.
UEFI is a Microsoft sponsored boot sector hijacker.
Android is the world's biggest spyware.
But most importantly: Google. Is. Skynet
User avatar
JoeFootball
Level 13
Level 13
Posts: 4674
Joined: Tue Nov 24, 2009 1:52 pm
Location: /home/usa/mn/minneapolis/joe

Re: File transfer hangs for a long time before finishing

Post by JoeFootball »

n0body_special wrote: ... using another file manager than the system's default is not really a solution, is it.
Speaking for myself, if it works, it certainly is. :)
n0body_special wrote: Now I have disabled the vm.dirty_bytes=15000000 trick, and it has returned to freezing (or really slow copying?) at 29%. Double commander too
Try Xfe, or just cp. It's what I use for large file copy sessions, and for me, it works without incident.

I've never tried Double Commander, but I do experience the same issue with Nemo, and some other GUI file managers.
User avatar
n0body_special
Level 3
Level 3
Posts: 130
Joined: Sat May 16, 2009 3:00 pm

Re: File transfer hangs for a long time before finishing

Post by n0body_special »

Midnight commander has trouble too. It starts really fast, then slows down to 1 MB/s and takes hours to copy just 5 GB.
The same thing happens with both my usb drives. It's clearly a problem of how linux handles the flash drives, I remember I had this problem again many years ago. With Mint 19.3 the problem has returned (along with others).
I have stopped using flash drives for large file transfer for this reason.

(in the screenshot, ETA time is "fake", it changes by one min every 5 mins) :?

edit: copying actually slows down to less than 1MB/s but not zero. It was the Midnight Commander that first showed me this. It's definitely something that has to do with the way the filesystems are handled, which means it could be ages before it can be fixed) :(
Attachments
1.png
Windows is the world's biggest virus.
UEFI is a Microsoft sponsored boot sector hijacker.
Android is the world's biggest spyware.
But most importantly: Google. Is. Skynet
User avatar
AndyMH
Level 21
Level 21
Posts: 13582
Joined: Fri Mar 04, 2016 5:23 pm
Location: Wiltshire

Re: File transfer hangs for a long time before finishing

Post by AndyMH »

It starts fast because it is filling up the memory buffers and once full slows down to the the write speeds for the USB drive. No way round this.
Thinkcentre M720Q - LM21.3 cinnamon, 4 x T430 - LM21.3 cinnamon, Homebrew desktop i5-8400+GTX1080 Cinnamon 19.0
deepakdeshp
Level 20
Level 20
Posts: 12334
Joined: Sun Aug 09, 2015 10:00 am

Re: File transfer hangs for a long time before finishing

Post by deepakdeshp »

Can this be a kernel problem? Some kernel may well copy fast.
If I have helped you solve a problem, please add [SOLVED] to your first post title, it helps other users looking for help.
Regards,
Deepak

Mint 21.1 Cinnamon 64 bit with AMD A6 / 8GB
Mint 21.1 Cinnamon AMD Ryzen3500U/8gb
User avatar
n0body_special
Level 3
Level 3
Posts: 130
Joined: Sat May 16, 2009 3:00 pm

Re: File transfer hangs for a long time before finishing

Post by n0body_special »

The way I see it, the problem of the original poster (and for me) is actually three problems:

1) Stupid implementation of write-cache for removable usb drives. Writing buffers are useful for hard disks, but it's a completely stupid thing to have for usb flash drives. It offers nothing in terms of actual writing speed, deceives the user about the true writing speed, and if the user removes a drive without unmounting, you get a corrupt filesystem. All file managers I have used are affected by this (nemo, double commander, midnight commander) and also the cp shell command. This thing is moronic: it absolutely should not exist and future linux distros should find a way to disable it. Until then there MUST be a workaround or a way to disable it.

2) Actual writing speed is slow: adding vm.dirty_bytes=15000000 in /etc/sysctl.conf makes the writing buffers small 15MB, effectively takes the writing cache out of the way and brings you face to face with the real problem: while copying to usb drives, I get speeds of 300kb/s and that's for usb 3 devices! I was wrong in my previous post. Copying 5 GB of data does finish if you leave the flash drive alone, it just happens after 10 hours or so. Users complain about "freezing" but it's not freezing, it's actually very very very slow copying.

3) During this slow copying procedure, Nemo becomes slow to respond or unresponsive. And because Nemo is an integral part of the desktop, the whole system becomes unresponsive while the usb copies (i.e. you can't open another folder and view it's contents). I have had to hard-reset the system after some time. Also, if you change your mind and decide to cancel copying, it takes some time (for me about 5 minutes) before copying actually stops. I think the system is waiting for that last file to finish copying before stopping and that may take some time
Windows is the world's biggest virus.
UEFI is a Microsoft sponsored boot sector hijacker.
Android is the world's biggest spyware.
But most importantly: Google. Is. Skynet
HilltopsGM
Level 4
Level 4
Posts: 231
Joined: Thu Mar 15, 2012 8:11 pm

Re: File transfer hangs for a long time before finishing

Post by HilltopsGM »

n0body_special wrote: Sat Jun 27, 2020 10:02 am The way I see it, the problem of the original poster (and for me) is actually three problems:

1) Stupid implementation of write-cache for removable usb drives. Writing buffers are useful for hard disks, but it's a completely stupid thing to have for usb flash drives. It offers nothing in terms of actual writing speed, deceives the user about the true writing speed, and if the user removes a drive without unmounting, you get a corrupt filesystem. All file managers I have used are affected by this (nemo, double commander, midnight commander) and also the cp shell command. This thing is moronic: it absolutely should not exist and future linux distros should find a way to disable it. Until then there MUST be a workaround or a way to disable it.

2) Actual writing speed is slow: adding vm.dirty_bytes=15000000 in /etc/sysctl.conf makes the writing buffers small 15MB, effectively takes the writing cache out of the way and brings you face to face with the real problem: while copying to usb drives, I get speeds of 300kb/s and that's for usb 3 devices! I was wrong in my previous post. Copying 5 GB of data does finish if you leave the flash drive alone, it just happens after 10 hours or so. Users complain about "freezing" but it's not freezing, it's actually very very very slow copying.

3) During this slow copying procedure, Nemo becomes slow to respond or unresponsive. And because Nemo is an integral part of the desktop, the whole system becomes unresponsive while the usb copies (i.e. you can't open another folder and view it's contents). I have had to hard-reset the system after some time. Also, if you change your mind and decide to cancel copying, it takes some time (for me about 5 minutes) before copying actually stops. I think the system is waiting for that last file to finish copying before stopping and that may take some time
Best description of the problem (100% agree).
It's a real pain in my backside and a very basic function (copying to a USB drive) that shouldn't bring the system to its knees.
Is anyone on the Mint Dev Team aware of this?
User avatar
AZgl1800
Level 20
Level 20
Posts: 11146
Joined: Thu Dec 31, 2015 3:20 am
Location: Oklahoma where the wind comes Sweeping down the Plains
Contact:

Re: File transfer hangs for a long time before finishing

Post by AZgl1800 »

I am using 19.3 Cinnamon, and when I investigated Actual writing speed is slow: adding vm.dirty_bytes=15000000 in /etc/sysctl.conf

I found that value is already in effect. exactly.
LM21.3 Cinnamon ASUS FX705GM | Donate to Mint https://www.patreon.com/linux_mint
Image
HilltopsGM
Level 4
Level 4
Posts: 231
Joined: Thu Mar 15, 2012 8:11 pm

Re: File transfer hangs for a long time before finishing

Post by HilltopsGM »

Same Question.
ANYONE ???
HilltopsGM wrote: Fri Jul 17, 2020 2:31 pm
n0body_special wrote: Sat Jun 27, 2020 10:02 am The way I see it, the problem of the original poster (and for me) is actually three problems:

1) Stupid implementation of write-cache for removable usb drives. Writing buffers are useful for hard disks, but it's a completely stupid thing to have for usb flash drives. It offers nothing in terms of actual writing speed, deceives the user about the true writing speed, and if the user removes a drive without unmounting, you get a corrupt filesystem. All file managers I have used are affected by this (nemo, double commander, midnight commander) and also the cp shell command. This thing is moronic: it absolutely should not exist and future linux distros should find a way to disable it. Until then there MUST be a workaround or a way to disable it.

2) Actual writing speed is slow: adding vm.dirty_bytes=15000000 in /etc/sysctl.conf makes the writing buffers small 15MB, effectively takes the writing cache out of the way and brings you face to face with the real problem: while copying to usb drives, I get speeds of 300kb/s and that's for usb 3 devices! I was wrong in my previous post. Copying 5 GB of data does finish if you leave the flash drive alone, it just happens after 10 hours or so. Users complain about "freezing" but it's not freezing, it's actually very very very slow copying.

3) During this slow copying procedure, Nemo becomes slow to respond or unresponsive. And because Nemo is an integral part of the desktop, the whole system becomes unresponsive while the usb copies (i.e. you can't open another folder and view it's contents). I have had to hard-reset the system after some time. Also, if you change your mind and decide to cancel copying, it takes some time (for me about 5 minutes) before copying actually stops. I think the system is waiting for that last file to finish copying before stopping and that may take some time
Best description of the problem (100% agree).
It's a real pain in my backside and a very basic function (copying to a USB drive) that shouldn't bring the system to its knees.
Is anyone on the Mint Dev Team aware of this?
electric-wildflower
Level 1
Level 1
Posts: 49
Joined: Thu Apr 28, 2011 4:23 pm

Re: File transfer hangs for a long time before finishing

Post by electric-wildflower »

I'm having this same issue copying over Samba shares to my FreeNas, it's like sometimes a 5 gig file takes a minute and other times 28 minutes even though i have capable speeds, sometimes the desktop will lock up if i cancel and then it comes crashing down. It's gotten to the point when i copy large files from one location to another ill walk away and leave it for a few hours because if i open up any more locations and do anything i can break the copying and the system locks up.

This is a major bug that's happening to everyone?
Locked

Return to “Cinnamon”