Too fast uploading content on a flash drive

[rado84]

I noticed this problem in 18.3 and it seems it's still present in 19: when I format a USB flash drive and then I try to upload anything on it (even a simple docx file), the upload speed is 155 MB/second (the normal speed of this FDD is 10 MB tops). If the file is, for instance, 700 MB, uploading reaches 80% instantly and then hangs on this percentage forever.
At first I thought it was the flash drive problem, so I diagnosed it with different software, I even bought a new one and it was still the same - instant upload to a specific percentage and then hanging on that forever. Which is why I can't upload a Linux live ISO on a flash drive doing that in Linux env. I tried that with Mint, Antergos, Debian and even Fedora - in all distroes the upload speed is extremenly high.
I tried the same in Windows and all uploading processes happened normally, including uploading the ISO on a flash drive.

So, before I report this as a bug in Mint, I'd like to know has someone else ever had this problem and how did you fix it?

[ajgringo619]

How did you format the drive? What do you mean by "uploading an .ISO"?

[rado84]

With the integrated formatting tool which allows only FAT32 and no other options.

[gm10]

This sounds like write caching. What does sudo hdparm -W /dev/sdb say? Replace /dev/sdb with wherever your actual USB stick is.

[schi2m]

I have the same problem.

Code: Select all

> sudo hdparm -W /dev/sdb1

/dev/sdb1:
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 write-caching = not supported

[rado84]

Code: Select all

rado@Mint19:~$ sudo hdparm -W /dev/sdd1
[sudo] password for rado: 

/dev/sdd1:
SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 write-caching = not supported
rado@Mint19:~$ 

[gm10]

So not write-caching, or at least none configured that way. Was the only idea I had, sorry. Can't reproduce it, either. Files copy with a steady speed to my USB sticks.

[ajgringo619]

I just ran a test, using unison to sync my critical /home files to a USB3 stick that was formatted in FAT32. While the data seems to be copying OK, I'm getting tons of timeouts (although no errors that I can see). I've never noticed timeouts when using my USB sticks - slowdowns yes, but never complete stoppages. I've created USB images from ISOs that have all worked as advertised.

Hopefully someone knows of a fix, or at least a workaround. Maybe this is a FAT32 issue; I'll try the same test with an EXT4 USB stick and see what happens.

[gm10]

Maybe this is a FAT32 issue;

Doubt it. I had tried to repro with a FAT32 formatted stick. No issues whatsoever.

[ajgringo619]

Maybe this is a FAT32 issue;

Doubt it. I had tried to repro with a FAT32 formatted stick. No issues whatsoever.

Thanks for the quick reply. Just grasping at straws now...

[Flemur]

If the file is, for instance, 700 MB, uploading reaches 80% instantly and then hangs on this percentage forever.

I had that too with thunar/Xfce.
(Edit: except "not forever", it "hung" for as long as it took to copy the data).
It's a bug, but it's a bug you can ignore if you feel like it. I "fixed it" by using "cp" in a terminal.

Remember how - for years - when you copied something with windows it would say "3232328445 seconds remaining" then "25 seconds" then "98452394723 seconds" ?

[ajgringo619]

Tried another test, this time using rsync. After about 500 MB, the command timed out for almost 2 minutes before kicking back into gear and finishing the copy; same thing happens when I use Thunar.

This never happens with my WD Passport USB drive, only my stick drives (Sandisk 64GB).

[Mute Ant]

I think it is a write-cache phenomenon, but by Linux saving up data-to-send rather than the drive saving up data-received. Normally a Debian kernel is set to use 10% of your RAM to cache data being sent to a drive. So, if you write more than that, it goes RAM speed, then Drive speed.

You can smooth out transfers by limiting the kernel's write buffer to something similar to that in a drive...

sudo sysctl vm.dirty_bytes=8192000 ### Refuse to cache more than 8MB of new data, until the next boot.

I always eject the drive and insert it again to re-register the hardware and a new cache. Maybe that's not needed, I don't know.

"...when you copied something with windows..." https://xkcd.com/612/

[rado84]

sudo sysctl vm.dirty_bytes=8192000 ### Refuse to cache more than 8MB of new data, until the next boot.

That didn't work either. 128 MB/sec upload speed. The integrated tool in Mint 19 offers other file systems (finally) but the result is the same with NTFS:
https://www19.zippyshare.com/v/YqiEUQbd/file.html
There's also EXT4 FS but the set top box for the TV won't read the flash drive if I format it in EXT4. It supports many codecs and formats but only if they're on a Windows' file system.