[SOLVED] Linux-->Linux File Transfer Speed

[ralplpcr]

Apologies if this has been asked before, but I couldn't find anything in my search that quite answered my concern?

I've got a large Linux Mint server that I use for storage of my old DVD collection, among other things. (Plex server)
I have another desktop system set up as dual-boot with Windows 10 and Linux Mint. I use this system to extract the movies with Handbrake, and then transfer them over to my Plex server. Both instances of Linux are Cinnamon.

What I've noticed is that for some reason, transferring a file using Linux is always much slower than via Windows.... this is what I'd like to correct!
Transferring a single 1GB .mkv file from Linux, the maximum speed I get is roughly 37-40MB/s. If I copy the same .mkv file into Windows, and transfer it, I usually get 96-110MB/s. So I know it's not a faulty network cable, and the network card works. I just don't quite understand why it's not reaching similar speeds under Linux?

I'm using SMB to connect by entering the smb://url/path into Nemo's address bar, and then mounting the resource with my server's username & password. In Windows, I use explorer to connect similarly.

FYI - This isn't limited to just 1 system. I've been able to replicate this issue on *multiple* Mint systems & Windows systems. Each are using different network cards, different network cables, etc, The same issue appears on all - - Linux transfers are always slower than sending files via Windows. This happens on Mint 17 through 20.

I *don't* see this problem with Warpinator. I suspect it's an issue with how SMB is configured. Any suggestions?

[Moonstone Man]

What I've noticed is that for some reason, transferring a file using Linux is always much slower than via Windows....
I *don't* see this problem with Warpinator. I suspect it's an issue with how SMB is configured. Any suggestions?

There are a whole host of factors involved here, far too many to chase down and troubleshoot on a web forum, and far too much explanation will be required if you want to know why it is so, so I'll give you some points to consider that you can do research on however at the end of the day, you may just have to put up with it.

First up, Windows uses a completely different caching algorithm to Linux. In some instances, Linux is faster but in others it's slower. You could be seeing the effects of that caching. On the server, you may want to adjust the caching:

Code: Select all

sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50

With the difference in caching, Linux will use what is called writeback for some transfers, which means the data goes directly to the remote disk. Windows will use memory caching in some instances, which means the file is loaded into memory then transferred to the remote system. The difference in speeds here should be obvious. The problem is discerning the conditions that trigger this behaviour.

Second, and this is possibly the killer, for some absolutely unknown reason that I've been able to fathom, the file content plays a large role in how fast Linux transfers data over a network. I have a situation where some very large files will transfer at break-neck speeds, in the order of 400MB/s and higher (this high speed is caused by caching, which I know is true because my internal network runs at 2.5Gb/s or 250MB/s), and small files will transfer at a snail's pace, in the order of KB/s. This slowness is caused by the overhead in allocating space in the filesystem inodes and writing a 4k block for a 1k file for example. However I also have some very large files that transfer at 80MB/s as opposed to others that transfer at 400MB/s, so I suspect that the remote file system is alternating where it places files, i.e. at the start of the disk, files transfer faster, and they transfer slower when placed at the end of the drive.

I've thought about adding a high speed NVMe drive as a write cache but Linux Mint's (Ubuntu's) dm-writecache has been broken for the last 12 months. Ubuntu built it with the wrong flags so it doesn't support a local write cache. Grrr. The mongrels still haven't decided a priority for the bug report after all this time.

Also, SMB isn't the fastest of protocols. Certainly the version of SMB being used can play a significant role here. Linux's nfs is much faster, but you need at least Windows 10 Pro, and a lot of research and Windows command-line-fu to install it and get it up and running, plus you'll probably hit the previous caching differences I mentioned.

So, as I said, you may just have to put up with it. I've spent an inordinate amount of time figuring out only what I've told you above, and an inordinate amount of time trying to fix it. It just so happens that last Friday I decided to just put up with it. Hopefully you can see why I'm more than reluctant to give you a definitive solution. My research has taught me that there isn't one.

[ralplpcr]

Thanks for the response, Kadaitcha Man!

Like you, I've spent a good deal of time trying to figure this out as well. I've looked into nfs, ftp, and a few other possible transfer methods. While I can make them work, and they do sometimes end up being faster than the "mount SMB share in Nemo" method, they're not always convenient.
The best I've found so far is to mount a share to my server in fstab using the cifs protocol. I had understood that cifs was essentially the same thing as SMB.... but it seems that there is some difference under the hood which I haven't yet grasped? If I mount a share into a temp folder using cifs, it generally results in transfer speeds that are just about on-par with what I see in Windows. While that's great, and could theoretically solve the issue, it's not nearly as convenient in many cases. I have multiple paths & directories on my server, and mounting to them all permanently in fstab is annoying. I'm not *always* needing to be connected to that server, so mounting the appropriate share only when needed seems more appropriate in my mind.

I'll try the swappiness suggestion, and see if that helps. I'm also considering changing the minimum SMB level to 3. I know it's not officially supported out of the box on Windows 7, but if it works I can easily avoid using Windows 7 for file transfers.

I can live with it if necessary....I just feel that if I can make it work closer to the maximum potential speed of 125MB/s, it would make me feel better about spending all the time & effort to upgrade my home network to all gigabit ethernet.

[Henry2000]

My experience is that smb on linux is just a lot slower. nfs, ftp, sftp, etc work a lot faster on linux. NFS can be pretty convenient if you just set it up properly client side. Either with automatic mounting, or a little script. Because of that I always used nfs, even on win10 game pc.

There are plenty of guides that explain how to setup nfs for ubuntu/mint. You can experiment a little with it, and check out the results for yourself.

[ralplpcr]

My experience is that smb on linux is just a lot slower. nfs, ftp, sftp, etc work a lot faster on linux. NFS can be pretty convenient if you just set it up properly client side. Either with automatic mounting, or a little script. Because of that I always used nfs, even on win10 game pc.

There are plenty of guides that explain how to setup nfs for ubuntu/mint. You can experiment a little with it, and check out the results for yourself.

I can verify that using a small bash script to mount a share on my server using cifs and then using rsync will also give me speeds in the 96-110MB/s range. So that is another alternative to putting the cifs mount into fstab. It has to be run with sudo, however... so not perfect, but do-able?

Code: Select all

sudo mount -t cifs //192.168.1.111/mnt/ /home/ralplpcr/share/ -o username=ralplpcr,file_mode=0777,dir_mode=0777

Does anyone know which transfer protocol is used by rsync &/or Warpinator? Is that a form of NFS?

[altair4]

If I mount a share into a temp folder using cifs, it generally results in transfer speeds that are just about on-par with what I see in Windows. While that's great, and could theoretically solve the issue, it's not nearly as convenient in many cases. I have multiple paths & directories on my server, and mounting to them all permanently in fstab is annoying. I'm not *always* needing to be connected to that server, so mounting the appropriate share only when needed seems more appropriate in my mind.

Not sure if this helps but since you are mounting these shares to your home directory you can have it in fstab in such a way that it will mount only when you need it.

Something like this:

Code: Select all

sudo mount -t cifs //192.168.1.111/mnt/ /home/ralplpcr/share/ -o username=ralplpcr,file_mode=0777,dir_mode=0777

In fstab becomes:

Code: Select all

//192.168.1.111/mnt/ /home/ralplpcr/share/ cifs username=ralplpcr,password=XXXXX,file_mode=0777,dir_mode=0777,noauto,user 0 0

noauto = makes it so it will not mount at boot.
user = allows an ordinary user ( not sudo )the ability to mount the share

In practice since the mount point is in your home folder an icon wil appear on the side panel of your file manager. When selected it will mount the share per the settings in fstab. You can also just do a mount /home/ralplpcr/share.

You can also go full systmed mode - but you will have to change your mount points: Automount Samba Shares with CIFS:

I had understood that cifs was essentially the same thing as SMB.... but it seems that there is some difference under the hood which I haven't yet grasped?

smb through the file manager is controlled by a samba client library invoked by gvfs. CIFS is under the control of the Linux kernel directly.

[ralplpcr]

In fstab becomes:

Code: Select all

//192.168.1.111/mnt/ /home/ralplpcr/share/ cifs username=ralplpcr,password=XXXXX,file_mode=0777,dir_mode=0777,noauto,user 0 0

noauto = makes it so it will not mount at boot.
user = allows an ordinary user ( not sudo )the ability to mount the share

In practice since the mount point is in your home folder an icon wil appear on the side panel of your file manager. When selected it will mount the share per the settings in fstab. You can also just do a mount /home/ralplpcr/share.

Thanks Altair4! I was not aware of the noauto and user flags. That is certainly another possible method I could try.
I will have to spend some time over the weekend to poke around & see if I can make that work.

smb through the file manager is controlled by a samba client library invoked by gvfs. CIFS is under the control of the Linux kernel directly.

Again, thank you - - that clarifies perfectly (for me, at least) the reason why the difference in speed between the two.

It looks as though I've got several tips & tricks to try out. Hopefully, one of them will do the trick & I can then mark this as [SOLVED].

[ralplpcr]

In fstab becomes:

Code: Select all

//192.168.1.111/mnt/ /home/ralplpcr/share/ cifs username=ralplpcr,password=XXXXX,file_mode=0777,dir_mode=0777,noauto,user 0 0

noauto = makes it so it will not mount at boot.
user = allows an ordinary user ( not sudo )the ability to mount the share

In practice since the mount point is in your home folder an icon wil appear on the side panel of your file manager. When selected it will mount the share per the settings in fstab. You can also just do a mount /home/ralplpcr/share.

Unfortunately, the noauto and user flags failed in my initial testing. This was on a Cinnamon 17.3 system - - so maybe I need to try on a newer system/kernel where these flags are permitted? When I ran sudo mount -a after modifying the /etc/fstab, I received an error that it had no permission and could not find the mount location? Even after verifying that read/write permissions were set to 777 on the mount location (/home/ralplpcr/share/), it still failed.
As soon as I removed the noauto and user flags and re-ran sudo mount -a, it was successful... so I know that it wasn't a typo or permissions issue.

HOWEVER, I was able to verify that the transfer speed is dramatically improved over SMB. Although it was only a small test transferring roughly 770MB (2 files: 1 of 543MB, 1 of 227MB), both files transferred over in just under 10 seconds - - so about 77MB/s. I'll have to try again with larger & more files, but even 77MB/s is a vast improvement over the 37-40MB/s I was getting previously.

I've marked this post as [Somewhat Solved?]. After I can run a few more tests to confirm, I'll mark the whole thread as [SOLVED] if it keeps that level of performance.

[altair4]

Can you post what your line in fstab looked like?

If you did not add the password within the fstab entry it would fail since there is no way for fstab to ask you for one.

[ralplpcr]

I was able to do a transfer test of a single 6.2GB .ISO file, and have verified that the transfer speed is most definitely improved using cifs instead of SMB.

I tested both copying to/from a RAID-6 HDD array on my server, as well as to/from an SSD. In all 4 cases, I reached quite good speeds:

SERVER SSD --> Desktop SSD: 108-135MB/s
Desktop SSD --> SERVER SSD: 119-173MB/s
SERVER HDD --> Desktop SSD: 96-127MB/s
Desktop SSD --> SERVER HDD: 112-168MB/s

I'm not quite sure how I reached speeds *above* the theoretical max of 125MB/s that my gigabit connection should have.... but I'm quite happy with these speeds either way. I *have not* yet adjusted the "swappiness" as suggested earlier. I may do that later to see if I can improve it even more, but for now, I'm satisfied with these results.

I'll still have to see if i can figure out why the noauto and user flags suggested don't work... but that's not as important. For the time being, I just know that using a cifs mount instead of SMB will be the way to go if I want to have transfers go quickly over my home network.

Thanks all for your help!

[ralplpcr]

Can you post what your line in fstab looked like?

If you did not add the password within the fstab entry it would fail since there is no way for fstab to ask you for one.

The line in fstab was essentially a copy/paste of the quote you provided:

Code: Select all

//192.168.1.111/mnt/ /home/ralplpcr/share/ cifs username=ralplpcr,password=ralplpcrPW,file_mode=0777,dir_mode=0777,noauto,user 0 0

I don't have a screenshot or the exact error message given when I tried to mount the share, but simply removing the ,noauto,user from that line allowed it to work:

Code: Select all

//192.168.1.111/mnt/ /home/ralplpcr/share/ cifs username=ralplpcr,password=ralplpcrPW,file_mode=0777,dir_mode=0777 0 0

I'll have to try the noauto,user bit on a different system when time allows. It's strange that it didn't work, but as I said, it was Cinnamon 17.3 - - which is quite long in the tooth at this point. Wouldn't surprise me if it works on a newer kernel?

[Moonstone Man]

I was able to do a transfer test of a single 6.2GB .ISO file, and have verified that the transfer speed is most definitely improved using cifs instead of SMB.

It's funny how the universe works. On Friday and yesterday (it's Sunday here) I was reconfiguring my internal network, which is a 2.5Gbe LAN being run by LMDE4 on a tiny 2-core machine, and bringing all of the connected machines and spare LMDE4 servers to an as common a configuration as possible. As part of that work I had to inspect and remove, add or modify all the custom systemd services on each machine. I found a systemd service on one of the spare tiny LMDE4 servers. It was the only machine with this particular service, and looking through the bash history I surmise I was experimenting around fixing the darned network bottlenecks that shouldn't really exist. Just for shits and giggles, I implemented the service on all my live machines. Lo and behold, many of the large files I move around the network are transferring much faster, and very small files (2B to about 4kB) are now transferring at up to 10MB/s instead of 5kb/s though some large files are still suffering a slowdown due to their content type. It seems that highly compressed content is the culprit here.

If you are interested, I'm happy to post an entire set of instructions for you to try. The steps will need to be implemented on all of your Linux systems to be effective. Whenever the system boots, the service will modify swappiness, the Linux kernel cache, and the amount of memory allocated to the entire TCP/IP stack. If it doesn't work out for you, it can be permanently undone with a single command.

Let me know if you're interested in trying it. There is no guarantee, of course.

Scratch that request to let me know. I created a tutorial. See here: viewtopic.php?p=1967523#p1967523

[ralplpcr]

It's funny how the universe works. ,,,,,<snip>,,,,,, I created a tutorial. See here: viewtopic.php?p=1967523#p1967523

That's pretty complex, and will take some time to figure out. It's worth a shot, however?
If I understand correctly, that would not do anything to improve the SMB-mounted transfers - correct? It would merely fine-tune the kernel to allow cifs and other kernel-handled transfer services to work more smoothly.

I'm thinking I ought to be able to put many of your instructions into a bash script that I can copy/execute from USB to avoid manual editing on multiple machines... but first I'll have to see if it works on 2 machines & would be worth it.

Thanks for posting back, and for your tutorial! I'll have to give it a try, and see if it works for me. I've been debating on setting up a 10GB SFP+ switch between some of my more heavily-used systems, and this could be a big benefit if I can make it work.

[senjoz]

In LM 20.1 Cinnamon, folder /usr/share/icons/ contains a lot of files. Nemo shows 102937 items and 249 MiB. I copied that folder using Nemo to another ext4 partition on the same disk, Samsung PM981a nvme. Copying finished in 82 seconds.

I restarted the machine, connected the M.2 sata SSD in the external USB enclosure and copied the same folder to the ext4 partition on the external disk. Copying was finished in 84 seconds.

I restarted the machine, connected to the shared folder on another Mint Machine with sata SSD using smb://192.168.1.150 in Nemo. Gigabit ethernet connection was perfect. I copied the same folder to the shared folder on ext4 partition. Copying was finished after more than 42 minutes. During copying I got errors "symlink not supported by backend" and "replace file xxx?". I confirmed the option "skip all". After copying Nemo showed 48454 items and 232 MiB.

Code: Select all

nvme0n1p2 -> nvme0n1p3			82 s
nvme0n1p2 -> usb sda3			84 s
nvme0n1p2 -> smb://192.168.1.150	>2500 s

I believe I will avoid all actions in Samba for a long time.

Regards, Jože

[ralplpcr]

.... During copying I got errors "symlink not supported by backend" and "replace file xxx?". ....

This doesn't surprise me in the least. Symlinks are specific to the machine on which they're created, and rely on the target to be in the same place to which they point during creation. And since /usr/share/icons/ is an existing directory on the target machine, it makes perfect sense that you'd be prompted whether or not to replace existing files/folders with the same name.

In other words, I don't think that's *entirely* the fault of Samba. Yes, Samba is slower.... but I suspect you'd see better results if you copied a folder that didn't have symlinks or duplicate files?