USB3 transfer speeds

Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

USB3 transfer speeds

Post by Kefren »

For background, I had some issues doing backups (viewtopic.php?f=47&t=364853). As a result I changed my process and have seen the reliability go up, fixing my original issues. Originally I was backing up to NTFS so that in an emergency I could still access my files from any PC (e.g. my girlfriend's Windows laptop) - but now I just use ext4, and in an emergency would USB live-boot into Mint from her laptop, copy any files I need from the backup to her data drive, and then access them in Windows. I also now make sure the drive is unmounted before logout (which can take a few minutes sometimes) - the original errors were probably bad drive etiquette on my part, thinking the drive was safe to unplug when it was actually still working (Disks showed it as being in the process of unmounting).

Current setup

So I am now backing up my personal data from my internal ext4 HDD to an external ext4 HDD via USB3, using FreeFileSync. It is a brand new PC, all components pretty fast and fancy. Linux Mint Cinnamon 20.2.

Problem

When I first start a large backup it has speeds of over 150 MB/s. Great! Even with a huge backup of 200GB, FFS shows it will only take 20 mins or so. On my old PC (USB2 only) I'd have been waiting hours.

However, after a short period of time (from 30 seconds to ten minutes) the speed starts dropping massively. Doesn't matter if it is on a big file like an iso or loads of tiny ones. It ends up dropping below 1MBs, sometimes to 0KBs, but then it gets faster again, then slower, then faster. Kind of like a pulse, but the higher speeds are a bit lower each time, and it never goes back up to the fast speeds - max is c, 30MBs, but often at a crawl. It's as if the data transfer speed is being throttled.

During this time I have a system monitor running. My PC has 32GB RAM, but even during the backup only 2GB is being used by the OS. The CPU goes from c.1% usage up to c.20-30%.

If I reboot and try it again, I get the 150MB/s speed again, but by the time it's done 10GB or so it starts dropping again.

So my question is, what causes the slowdown? A cacheing issue with USB3? But in that case, why does it never catch up again, especially when I have so much memory? Instead of the backup taking half an hour, I had to stagger it over three days and multiple reboots to get it finished. Since it's obvious my PC can do 150MB/s, that's pretty frustrating. Since FreeFileSync estimates the end time based on current transfer speed, I watch with dismay as the "20 minutes" estimate grows and grows until it is saying "22 days"!

The issue of USB3 data backups to a HDD being fast, then suddenly dropping, seems to be quite common. One post (which I now can't find) said it might be related to a certain part of USB3 (it had an acronym) which can be turned off, and then people get faster transfer speeds because it no longer throttles them. I tried the procedure in the post at viewtopic.php?p=1636401#p1636401 but it made no difference (so I undid it again).

Obviously this is not a critical problem, just inconvenient and a bit irritating!
1.jpg
Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 1 time in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
cliffcoggin
Level 8
Level 8
Posts: 2297
Joined: Sat Sep 17, 2016 6:40 pm
Location: England

Re: USB3 transfer speeds

Post by cliffcoggin »

It is quite normal for data transfer to start fast and then slow down. Providing the overall time is not too long I wouldn't worry about it.
Cliff Coggin
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

Re: USB3 transfer speeds

Post by Kefren »

cliffcoggin wrote: Tue Jan 11, 2022 2:19 pm It is quite normal for data transfer to start fast and then slow down. Providing the overall time is not too long I wouldn't worry about it.
But 22 days seems extreme for a backup that would only take 15 minutes at the speed the drives are capable of (and which it begins at). So the question of why it suddenly becomes so slow (sometimes dropping to zero) and how to avoid it still remains.

I guess the frustrating thing is that tech companies and journalism go on about how fast new things are, and that everything can do 5Gb/s, and then when I try to use it I find that sometimes it can't even do 1MB/s!
User avatar
AndyMH
Level 21
Level 21
Posts: 13753
Joined: Fri Mar 04, 2016 5:23 pm
Location: Wiltshire

Re: USB3 transfer speeds

Post by AndyMH »

Some stick manufacturer's implementation of usb3 is not quite compliant, more like usb2.5 and it can cause problems. There have been posts on this in the past, but the forum search facilities are not brilliant... That may be the reason for the low transfer rates.
Thinkcentre M720Q - LM21.3 cinnamon, 4 x T430 - LM21.3 cinnamon, Homebrew desktop i5-8400+GTX1080 Cinnamon 19.0
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

Re: USB3 transfer speeds

Post by Kefren »

AndyMH wrote: Wed Jan 12, 2022 8:55 am Some stick manufacturer's implementation of usb3 is not quite compliant, more like usb2.5 and it can cause problems. There have been posts on this in the past, but the forum search facilities are not brilliant... That may be the reason for the low transfer rates.
Do you know if there any kind of test I can do, as in changing Mint settings to see if it helps? Or a better forum I should post this in? Cheers!
motoryzen
Level 10
Level 10
Posts: 3489
Joined: Sun Dec 08, 2019 12:25 am

Re: USB3 transfer speeds

Post by motoryzen »

Kind of like a pulse, but the higher speeds are a bit lower each time, and it never goes back up to the fast speeds - max is c, 30MBs, but often at a crawl. It's as if the data transfer speed is being throttled.
If it wasn't for the " but 22 days seems very excessive for what should only take around 15 minutes "....I'd ask are you ONLY backing up big files like video files?

THOSE...( if both ends of the transfer deal ARE fast + usb 3 or faster usb type A ports AND the transfer medium ( in this case...FFS) supports sustained usb 3.0 or faster speeds and the hardware itself in the middle acting like the bridge...ALSO supports usb 3.0 speeds or faster and is in good shape..

Why? Because video files will ALWAYS ( if you're comparing one 5 GB video file to 5GBs of text/document files) transfer much faster...regardless of it being hdd to hdd or flash storage to flash storage

Given your statement about 22 days and that dropping to 30MBs makes me wonder if a piece of hardware is dying...but beyond what everyone else has already mentioned...I don't know yet. Sorry I cannot be of much help.

*** Also Have you tried another usb 3.0 port on both the "reading " machine ( the one it's coming from) to the " writing" machine ( the one that is receiving the file/files) *** ??
Mint 21.2 Cinnamon 5.8.4
asrock x570 taichi ...bios p5.00
ryzen 5900x
128GB Kingston Fury @ 3600mhz
Corsair mp600 pro xt NVME ssd 4TB
three 4TB ssds
dual 1TB ssds
Two 16TB Toshiba hdd's
24GB amd 7900xtx vid card
Viewsonic Elite UHD 32" 144hz monitor
Rooo
Level 2
Level 2
Posts: 52
Joined: Sun Nov 21, 2021 9:00 am

Re: USB3 transfer speeds

Post by Rooo »

have you tested the physical read/write speed of the dongle?

could be your filling the cache using that 32gig of ram - progress bar zooms across,then the actual write takes ages cos its ancient and really slow :D (like mine :lol: )

i still use a portable,old program on windows called roadkills speed test,no idea what to use on linux sorry :(



edit:- Windows - copying over LMDE to my slow ventoy - speeds hit 171mb/s for maybe 15?seconds,then crawl miserably,leaving me with 3gb of my 16gb ram ...
Forum Pest ...

"Easy tips for Linux Mint and Ubuntu, both for beginners and for advanced users" https://easylinuxtipsproject.blogspot.com/p/1.html
DisturbedDragon
Level 5
Level 5
Posts: 574
Joined: Mon Oct 29, 2012 6:29 pm
Location: Texas

Re: USB3 transfer speeds

Post by DisturbedDragon »

Answer to this is most likely the hardware. I have USB3 and USB3.1 ports that write to my 1TB USB C nvme drive at full port capabilities , start to finish, 1Gb or 100GB. Other Adata flash drives write anywhere from 150 to 40 MB/s for breadth of the write for large files.
AMD Ryzen 9 5950X 16C/32T | MSI MPG x570 Gaming Plus | 2TB Mushkin Pilot-E NVMe | 1TB Crucial P1 NVMe | 2x 2TB Inland Gen4 NVMe | 32GB Trident Z DDR4 3600 | Nvidia RTX4090 | Fedora 39 Cinnamon | Linux Mint 21.3 Cinnamon | Kernel 5.15.x lowlatency
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

Re: USB3 transfer speeds

Post by Kefren »

motoryzen wrote: Thu Jan 13, 2022 11:56 am
If it wasn't for the " but 22 days seems very excessive for what should only take around 15 minutes "....I'd ask are you ONLY backing up big files like video files?

THOSE...( if both ends of the transfer deal ARE fast + usb 3 or faster usb type A ports AND the transfer medium ( in this case...FFS) supports sustained usb 3.0 or faster speeds and the hardware itself in the middle acting like the bridge...ALSO supports usb 3.0 speeds or faster and is in good shape..

Why? Because video files will ALWAYS ( if you're comparing one 5 GB video file to 5GBs of text/document files) transfer much faster...regardless of it being hdd to hdd or flash storage to flash storage
It's a mix of files. There are some video files, some isos etc; but also down to tiny txt files. It's just my personal and work backups. The same pattern of speed dropping seems to happen whatever it is backing up.
motoryzen wrote: Thu Jan 13, 2022 11:56 am Given your statement about 22 days and that dropping to 30MBs makes me wonder if a piece of hardware is dying...but beyond what everyone else has already mentioned...I don't know yet. Sorry I cannot be of much help.
Well, it's a brand new PC, everything seems rock solid apart from the speed drop. I have two identical external Seagate 5Gb/s USB3 hard drives (since I keep two backups in different places). Each has its own cable. Exactly the same results whichever I choose.
motoryzen wrote: Thu Jan 13, 2022 11:56 am *** Also Have you tried another usb 3.0 port on both the "reading " machine ( the one it's coming from) to the " writing" machine ( the one that is receiving the file/files) *** ??
Yep. The external HDDs just have a single cable, so no options there. But I have tried them in
- both front USB3 ports on my desktop case 2 x USB 3.2 Gen 1 5Gbps (Type-A) [blue]
- two different ones at the back, direct in the MB 1 x USB 3.2 Gen 1 5Gbps (Type-A) [blue] / 1 x USB 3.2 Gen 2 10Gbps (Type-A) [red]
- one on my monitor's SUB hub (5Gbps).

Same thing whichever I choose, even the 1-Gbps one - fast speeds (c.185MB/s), then drop, then phase up and down, anywhere between 0MBs and 50MBs.
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

Re: USB3 transfer speeds

Post by Kefren »

Rooo wrote: Thu Jan 13, 2022 4:38 pm have you tested the physical read/write speed of the dongle?

could be your filling the cache using that 32gig of ram - progress bar zooms across,then the actual write takes ages cos its ancient and really slow :D (like mine :lol: )

i still use a portable,old program on windows called roadkills speed test,no idea what to use on linux sorry :(

edit:- Windows - copying over LMDE to my slow ventoy - speeds hit 171mb/s for maybe 15?seconds,then crawl miserably,leaving me with 3gb of my 16gb ram ...
That sounds like my pattern, but the backup never touches my RAM - it always shows 30GB free during backups.

It's a brand new PC with a decent spec, everything seems rock solid apart from the speed drop. I have two identical external Seagate 5Gb/s USB3 hard drives (since I keep two backups in different places). Each has its own cable. Exactly the same results whichever I choose.

I have tried them in
- both front USB3 ports on my desktop case 2 x USB 3.2 Gen 1 5Gbps (Type-A) [blue]
- two different ones at the back, direct in the MB 1 x USB 3.2 Gen 1 5Gbps (Type-A) [blue] / 1 x USB 3.2 Gen 2 10Gbps (Type-A) [red]
- one on my monitor's SUB hub (5Gbps).

Same thing whichever I choose, even the 1-Gbps one - fast speeds (c.185MB/s), then drop, then phase up and down, anywhere between 0MBs and 50MBs.
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

Re: USB3 transfer speeds

Post by Kefren »

DisturbedDragon wrote: Thu Jan 13, 2022 9:00 pm Answer to this is most likely the hardware. I have USB3 and USB3.1 ports that write to my 1TB USB C nvme drive at full port capabilities , start to finish, 1Gb or 100GB. Other Adata flash drives write anywhere from 150 to 40 MB/s for breadth of the write for large files.
Well, it's a brand new PC with a decent spec, everything seems rock solid apart from the speed drop. I have two identical, new external Seagate 5Gb/s USB3 hard drives (since I keep two backups in different places). Each has its own cable. Exactly the same results whichever I choose.

I have tried them in
- both front USB3 ports on my desktop case 2 x USB 3.2 Gen 1 5Gbps (Type-A) [blue]
- two different ones at the back, direct in the MB 1 x USB 3.2 Gen 1 5Gbps (Type-A) [blue] / 1 x USB 3.2 Gen 2 10Gbps (Type-A) [red]
- one on my monitor's SUB hub (5Gbps).

Same thing whichever I choose, even the 1-Gbps one - fast speeds (c.185MB/s), then drop, then phase up and down, anywhere between 0MBs and 50MBs.
Kefren
Level 4
Level 4
Posts: 264
Joined: Fri Dec 10, 2021 3:45 pm
Location: Scotland
Contact:

Re: USB3 transfer speeds

Post by Kefren »

No idea if this helps.

System: Kernel: 5.13.0-25-generic x86_64 bits: 64 compiler: N/A Desktop: Cinnamon 5.0.7
wm: muffin dm: LightDM Distro: Linux Mint 20.2 Uma base: Ubuntu 20.04 focal
Machine: Type: Desktop Mobo: Micro-Star model: MAG B550M MORTAR WIFI (MS-7C94) v: 1.0
serial: <filter> UEFI: American Megatrends LLC. v: 1.80 date: 07/01/2021
CPU: Topology: 6-Core model: AMD Ryzen 5 5600X bits: 64 type: MT MCP arch: Zen 3
L2 cache: 3072 KiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm bogomips: 88798
Speed: 4048 MHz min/max: 2200/3700 MHz Core speeds (MHz): 1: 3170 2: 2880 3: 2880
4: 2922 5: 2879 6: 3434 7: 2200 8: 2200 9: 2213 10: 2515 11: 2410 12: 2662
Graphics: Device-1: NVIDIA driver: nvidia v: 470.86 bus ID: 2b:00.0 chip ID: 10de:2489
Display: x11 server: X.Org 1.20.13 driver: nvidia
unloaded: fbdev,modesetting,nouveau,vesa resolution: 2560x1440~60Hz
OpenGL: renderer: NVIDIA GeForce RTX 3060 Ti/PCIe/SSE2 v: 4.6.0 NVIDIA 470.86
direct render: Yes
Audio: Device-1: NVIDIA driver: snd_hda_intel v: kernel bus ID: 2b:00.1 chip ID: 10de:228b
Device-2: AMD Starship/Matisse HD Audio vendor: Micro-Star MSI driver: snd_hda_intel
v: kernel bus ID: 2d:00.4 chip ID: 1022:1487
Sound Server: ALSA v: k5.13.0-25-generic
Network: Device-1: Intel Wi-Fi 6 AX200 driver: iwlwifi v: kernel bus ID: 29:00.0
chip ID: 8086:2723
IF: wlo1 state: up mac: <filter>
Device-2: Realtek RTL8125 2.5GbE vendor: Micro-Star MSI driver: r8169 v: kernel
port: f000 bus ID: 2a:00.0 chip ID: 10ec:8125
IF: enp42s0 state: down mac: <filter>
Drives: Local Storage: total: 4.10 TiB used: 890.41 GiB (21.2%)
ID-1: /dev/nvme0n1 vendor: Samsung model: MZVL2512HCJQ-00B00 size: 476.94 GiB
speed: 63.2 Gb/s lanes: 4 serial: <filter>
ID-2: /dev/nvme1n1 vendor: Samsung model: SSD 970 EVO Plus 250GB size: 232.89 GiB
speed: 31.6 Gb/s lanes: 4 serial: <filter>
ID-3: /dev/sda vendor: Seagate model: ST4000DM004-2CV104 size: 3.64 TiB speed: 6.0 Gb/s
serial: <filter>
Partition: ID-1: / size: 467.02 GiB used: 23.07 GiB (4.9%) fs: ext4 dev: /dev/dm-0
ID-2: swap-1 size: 980.0 MiB used: 0 KiB (0.0%) fs: swap dev: /dev/dm-1
Sensors: System Temperatures: cpu: 39.8 C mobo: N/A gpu: nvidia temp: 39 C
Fan Speeds (RPM): N/A gpu: nvidia fan: 0%
Repos: No active apt repos in: /etc/apt/sources.list
Active apt repos in: /etc/apt/sources.list.d/additional-repositories.list
1: deb https: //dl.winehq.org/wine-builds/ubuntu/ focal main
Active apt repos in: /etc/apt/sources.list.d/official-package-repositories.list
1: deb https: //mirror.cov.ukservers.com/linuxmint uma main upstream import backport
2: deb http: //archive.ubuntu.com/ubuntu focal main restricted universe multiverse
3: deb http: //archive.ubuntu.com/ubuntu focal-updates main restricted universe multiverse
4: deb http: //archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
5: deb http: //security.ubuntu.com/ubuntu/ focal-security main restricted universe multiverse
6: deb http: //archive.canonical.com/ubuntu/ focal partner
Info: Processes: 332 Uptime: 1h 27m Memory: 31.28 GiB used: 2.36 GiB (7.5%) Init: systemd
v: 245 runlevel: 5 Compilers: gcc: 9.3.0 alt: 9 Client: Unknown python3.8 client
inxi: 3.0.38
Locked

Return to “Storage”