Awfully slow USB Drive transfer

All Gurus once were Newbies
Forum rules
There are no such things as "stupid" questions. However if you think your question is a bit stupid, then this is the right place for you to post it. Please stick to easy to-the-point questions that you feel people can answer fast. For long and complicated questions prefer the other forums within the support section.
Before you post please read how to get help
Post Reply
TheDrot
Level 1
Level 1
Posts: 5
Joined: Sun Mar 23, 2014 2:12 pm

Awfully slow USB Drive transfer

Post by TheDrot » Mon Apr 24, 2017 6:02 am

Hi. As I was told by friends who are familiar with Linux, it's an old problem, but I couldn't find much information about it. When I copy files to a USB drive, at start it works fine (90 mbps), but then it gets slower and slower. After a couple minutes of copying it's 20 mbps and keeps slowing down. Then the file transfer simply freezes, then unfreezes for several seconds, freezes again and so on. Because of that, copying takes hours, while at the beginning estimated time was 4 minutes. When I move a 1.5 Gb file to a USB drive, it copies in seconds and then stays at 100% for 10 minutes or even longer before the transfer is done. And that behaviour is here for years. I've tried using the most recent kernel, and it helped a bit, at least freezes are gone now, but the speed reduction wasn't fixed. The thing is, I need to make a large backup (almost 1 Tb), and if I try to make it right now, it will take weeks to finish. As it's a one-time operation (in future I will just update the backup without transfering large amounts of data), I don't mind using some LiveCD or beta kernel or whatever just for that transfer. My questions are: is there any solution or a workaround for that USB problem and is there any distrubution (maybe even not Linux, but FreeBSD or something) without that bug? So I would copy my files and then continue using Mint.

My system:
System: Host: thedrot-X541UJ Kernel: 4.10.0-19-generic x86_64 (64 bit gcc: 5.4.0)
Desktop: MATE 1.16.1 (Gtk 3.18.9-1ubuntu3.2)
Distro: Linux Mint 18.1 Serena
Machine: System: ASUSTeK (portable) product: X541UJ v: 1.0
Mobo: ASUSTeK model: X541UJ v: 1.0
Bios: American Megatrends v: X541UJ.202 date: 12/02/2016
CPU: Dual core Intel Core i7-7500U (-HT-MCP-) cache: 4096 KB
flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 11616
clock speeds: max: 3500 MHz 1: 1737 MHz 2: 1736 MHz 3: 1725 MHz
4: 1741 MHz
Graphics: Card-1: Intel Device 5916 bus-ID: 00:02.0
Card-2: NVIDIA GK208M [GeForce 920M] bus-ID: 01:00.0
Display Server: X.Org 1.18.4 drivers: intel (unloaded: fbdev,vesa) FAILED: nouveau
Resolution: 1366x768@60.05hz
GLX Renderer: Mesa DRI Intel Kabylake GT2
GLX Version: 3.0 Mesa 11.2.0 Direct Rendering: Yes
Audio: Card Intel Device 9d71 driver: snd_hda_intel bus-ID: 00:1f.3
Sound: Advanced Linux Sound Architecture v: k4.10.0-19-generic
Network: Card-1: Realtek RTL8101/2/6E PCI Express Fast/Gigabit Ethernet controller
driver: r8169 v: 2.3LK-NAPI port: d000 bus-ID: 02:00.2
IF: enp2s0f2 state: down mac: 60:45:cb:20:cb:8e
Card-2: Qualcomm Atheros QCA9565 / AR9565 Wireless Network Adapter
driver: ath9k bus-ID: 03:00.0
IF: wlp3s0 state: up mac: 3c:a0:67:1f:fd:66
Drives: HDD Total Size: 2000.4GB (25.1% used)
ID-1: /dev/sda model: ST1000LM035 size: 1000.2GB
ID-2: USB /dev/sdb model: Expansion size: 1000.2GB
Partition: ID-1: / size: 909G used: 452G (53%) fs: ext4 dev: /dev/sda2
ID-2: swap-1 size: 8.47GB used: 0.00GB (0%) fs: swap dev: /dev/sda3
RAID: No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors: System Temperatures: cpu: 45.0C mobo: N/A
Fan Speeds (in rpm): cpu: N/A
Info: Processes: 208 Uptime: 33 min Memory: 2200.1/7863.4MB
Init: systemd runlevel: 5 Gcc sys: 5.4.0
Client: Shell (bash 4.3.461) inxi: 2.2.35

User avatar
austin.texas
Level 20
Level 20
Posts: 12054
Joined: Tue Nov 17, 2009 3:57 pm
Location: at /home

Re: Awfully slow USB Drive transfer

Post by austin.texas » Mon Apr 24, 2017 6:20 am

What method are you using to copy? The file manager?
rsync (or Grsync) does incremental backups, so only the new and changed files are copied. That cuts backup time dramatically.
Mint 18.2 Cinnamon, Quad core AMD A8-3870 with Radeon HD Graphics 6550D, 8GB DDR3, Ralink RT2561/RT61 802.11g PCI
Linux Linx 2018

TheDrot
Level 1
Level 1
Posts: 5
Joined: Sun Mar 23, 2014 2:12 pm

Re: Awfully slow USB Drive transfer

Post by TheDrot » Mon Apr 24, 2017 6:34 am

I use Caja and simply drag'n'drop the folder. But the same problem was on my old PC with both Ubuntu and Mint (so the file manager was Nautilus). And I don't mind making backups manually, as there's a lot of media data, which doesn't change and just needs to be stored, so I can simply update a couple of folders. The main problem is the first backup, when I need to transfer all the data. It would take some hours on Windows, but I wonder how long it will take on Mint because of transfer getting slower over time. Just for test I tried to copy a 13 Gb folder. At start, transfer speed was almost 100 mbps, but in the end it was 4 mbps. So if I try to copy 1 Tb, it may slow down to 500 kbps after an hour (I guess), and transfer my files for years.

User avatar
austin.texas
Level 20
Level 20
Posts: 12054
Joined: Tue Nov 17, 2009 3:57 pm
Location: at /home

Re: Awfully slow USB Drive transfer

Post by austin.texas » Mon Apr 24, 2017 6:45 am

I have never had any problem with Grsync slowing down with large backups.
See Part 2 of this tutorial.
TUTORIAL: Grsync backups of root and /home
Mint 18.2 Cinnamon, Quad core AMD A8-3870 with Radeon HD Graphics 6550D, 8GB DDR3, Ralink RT2561/RT61 802.11g PCI
Linux Linx 2018

ColdBoot
Level 3
Level 3
Posts: 196
Joined: Thu Feb 16, 2017 10:40 pm

Re: Awfully slow USB Drive transfer

Post by ColdBoot » Mon Apr 24, 2017 6:46 am

What you see is caching speed which is very fast particularly if you are copying from a SSD. In reality, even with a USB 3 device, you can't expect file transfers faster than 14MB/s and it totally depends on the quality of the USB stick.
Linux Mint Cinnamon 18.1
Intel G1820, DDR3 8GB, Nvidia GT720(2GB)

Mute Ant
Level 14
Level 14
Posts: 5135
Joined: Tue Sep 03, 2013 7:45 pm
Location: Norfolk UK

Re: Awfully slow USB Drive transfer

Post by Mute Ant » Mon Apr 24, 2017 6:58 am

The computer end looks fine ... https://www.asus.com/Notebooks/X541UJ/ ... USB3 ... "transferring a movie with a file size of 2GB takes less than two seconds".

https://en.wikipedia.org/wiki/USB
USB 1.0 ~ 1996-01 ~ Low Speed ~ 1.5 Mbit/s ~ 187 kByte/s
USB 1.1 ~ 1998-08 ~ Full Speed ~ 12 Mbit/s ~ 1.5 MByte/s
USB 2.0 ~ 2000-04 ~ High Speed ~ 480 Mbit/s ~ 60 MByte/s
USB 3.0 ~ 2008-11 ~ Super Speed ~ 5 Gbit/s ~ 625 MByte/s
USB 3.1 ~ 2013-07 ~ Super Speed Plus ~ 10 Gbit/s ~ 1.25 GByte/s

You should ignore the speed reported during the first 500 MByte of a transfer, it is badly distorted by the data going into RAM cache first. After the cache fills up, it sounds as though your transfer is stuck at USB2 speed. Use a USB3 device plugged in to a USB3 port. If you must have a cable, that has to be USB3 too, ideally the one that came with the drive...four big pins +five small pins +blue insulator would be typical. Any part in the chain that is USB2, software or hardware, will seriously limit the maximum possible rate.

Of Historic Interest Only
https://en.wikipedia.org/wiki/RS-232 RS232 ~ 1969 ~ 20kbit/s ~ 2 kByte/s ~ 1TB transfer down a RS232 cable...15844 years :shock:
Last edited by Mute Ant on Mon Apr 24, 2017 7:09 am, edited 1 time in total.
While you're waiting, read the free novel we sent you. It's a Spanish story about a guy named "manual".

ColdBoot
Level 3
Level 3
Posts: 196
Joined: Thu Feb 16, 2017 10:40 pm

Re: Awfully slow USB Drive transfer

Post by ColdBoot » Mon Apr 24, 2017 7:07 am

An interesting article if you wish to tinker with Linux disk caching & performance cause it may affect file transfer speed to usb(but I think not much):

https://lonesysadmin.net/2013/12/22/bet ... rty_ratio/
Linux Mint Cinnamon 18.1
Intel G1820, DDR3 8GB, Nvidia GT720(2GB)

TheDrot
Level 1
Level 1
Posts: 5
Joined: Sun Mar 23, 2014 2:12 pm

Re: Awfully slow USB Drive transfer

Post by TheDrot » Mon Apr 24, 2017 8:23 am

Thanks for the replies!
Mute Ant wrote:The computer end looks fine ... https://www.asus.com/Notebooks/X541UJ/ ... USB3 ... "transferring a movie with a file size of 2GB takes less than two seconds".
It's true when it comes to small files. a 2GB movie would actually transfer in a couple of seconds. But when I transfer a massive array of data, it slows down after ~3 gigabytes. So the first 3GB or something like that transfer fine, but then speed starts to decrease. While I was copying a 13GB folder, speed was going down, then it was ~6MBps for a minute or so, then started going down again. And when I say it's going down I mean that it's like a countdown every second: 20.9, 20.8, 20.7 and so on. Is it because of the caching system?
I've double checked that I'm plugging a USB3-compatible external HDD in a USB3-port using a USB3-cable, and everything is right with that.

Update: I've made a video of file transfer https://www.youtube.com/watch?v=7jtb-yFiuVo
As it was a 4.4GB file, the transfer didn't get too slow, but with a bigger file it would continue slowing down just like on the video.

Volunteer4F
Level 1
Level 1
Posts: 11
Joined: Wed Dec 07, 2011 3:48 am

Re: Awfully slow USB Drive transfer

Post by Volunteer4F » Mon Oct 23, 2017 6:52 pm

The only trick I found that really works:
https://unix.stackexchange.com/question ... 00-or-near

If you want to try some power-user tricks, you can reduce the size of the buffer that Linux uses by setting /proc/sys/vm/dirty_bytes to something like 15728640 (15 MB). This means the application can't get more than 15MB ahead of its actual progress.

A side effect is that your computer might have lower data-writing throughput with this setting, but on the whole, I find it helpful to see that a program is running a long time while it writes lots of data vs. the confusion of having a program appear to be done with its job but the system lagging badly as the kernel does the actual work. Setting dirty_bytes to a reasonably small value can also help prevent your system from becoming unresponsive when you're low on free memory and run a program that suddenly writes lots of data.

But, don't set it too small! I use 15MB as a rough estimate that the kernel can flush the buffer to a normal hard drive in 1/4 of a second or less. It keeps my system from feeling "laggy".

User avatar
catweazel
Level 19
Level 19
Posts: 9580
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Awfully slow USB Drive transfer

Post by catweazel » Tue Oct 24, 2017 1:38 am

Volunteer4F wrote:The only trick I found that really works
Please don't resurrect dead threads over 6 months old.
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

Post Reply

Return to “Newbie Questions”