Nemo failing when moving big files

Please post suggestions for improvement of Cinnamon on:
https://github.com/linuxmint/Cinnamon
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Locked
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Nemo failing when moving big files

Post by macbeto »

Hi,
The most of the times when I move .mkv files with size greater of 200Mb with Nemo from my laptop disk to an external disk the most of the times it is inconsistent. I play the video file and it goes well, but after unplugging the external drive and plug it again the video file is with less size than the original and doesn't play.
My external drives are formatted as ext4 and hfsplus, and all of them suffer the same problem.
Anytime I move a file I fear it won't work, apparently it move well but at the end I loose the video file.
Even sometimes I move the files with mv command and the result is the same.
It happened the same from LM18 upto LM20.1
Is it a common error with Linux Mint / Nemo?
Thanks for your advice.
Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 1 time in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
User avatar
absque fenestris
Level 12
Level 12
Posts: 4124
Joined: Sat Nov 12, 2016 8:42 pm
Location: Confoederatio Helvetica

Re: Nemo failing when moving big files

Post by absque fenestris »

If you don't want to do it by terminal commands, try Double Commander.
The problem with Nemo comes up again and again here in the forums.
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Re: Nemo failing when moving big files

Post by macbeto »

Thanks for your reply.
I didn't know Double Commander. Does it move the files reliable?
I don't mind to use commands if it works ok, but sometimes I moved video files with mv coomand and after unplugging the external drive and plugin again the video file was inconsistent, with less size than the original file. So I assumed command was a good solution.
mikeflan
Level 17
Level 17
Posts: 7016
Joined: Sun Apr 26, 2020 9:28 am
Location: Houston, TX

Re: Nemo failing when moving big files

Post by mikeflan »

It happened the same from LM18 upto LM20.1
Is it a common error with Linux Mint / Nemo?
I say this is not a common error on LM, but I am only a single test case with about 3 LM machines.
I would like to know how prevalent this problem is. I suspect less than 2% of users, but maybe that is wishful thinking.

My source drive is a spinning HD formatted to EXT4. My target drive is a USB thumb drive formatted to NTFS, though df-Th says Vfat.

My largest MKV files is only 30 MB, so I went to my MP4 files

I copied a 200 MB MP4 via drag and drop in Nemo, then did a ctrl-C ctrl-V back to HHD (in a different directory). It played fine. Copy to thumb drive took about 10 seconds. Copy back was instantaneous (<1 sec).

Then I copied a 338 MB MP4 via drag and drop in Nemo, then did drag and drop back to HHD (in a different directory). It played fine. Copy to thumb drive took about 18 seconds. Copy back was instantaneous (<1 sec).

So thankfully my computer does not have the problem you report.
I would like to know what happens for you when you use FreeFileSync to transfer the file. I suspect that will work fine.
motoryzen
Level 10
Level 10
Posts: 3450
Joined: Sun Dec 08, 2019 12:25 am

Re: Nemo failing when moving big files

Post by motoryzen »

macbeto wrote: Wed May 05, 2021 5:07 pm Thanks for your reply.
I didn't know Double Commander. Does it move the files reliable?
I don't mind to use commands if it works ok, but sometimes I moved video files with mv coomand and after unplugging the external drive and plugin again the video file was inconsistent, with less size than the original file. So I assumed command was a good solution.
-- I've never encountered this problem on my target external ssd nor external 10TB hdd doing the same thing and I deal with full sized ripped bluray movies occasionally. My target drives are all EXT4 formatted except one that was formatted from within Windows 7 as exFat.

Have you tried formatting that thumb drive to EXT4 and see if the problem still happens?...sorry if that sounds pointless but at least it's one thing to rule out if nothing changes (shrugs) --
Mint 21.2 Cinnamon 5.8.4
asrock x570 taichi ...bios p5.00
ryzen 5900x
128GB Kingston Fury @ 3600mhz
Corsair mp600 pro xt NVME ssd 4TB
three 4TB ssds
dual 1TB ssds
Two 16TB Toshiba hdd's
24GB amd 7900xtx vid card
Viewsonic Elite UHD 32" 144hz monitor
Petermint
Level 9
Level 9
Posts: 2983
Joined: Tue Feb 16, 2016 3:12 am

Re: Nemo failing when moving big files

Post by Petermint »

Did the light on the external device stop blinking before you unplugged? Linux can be copying in the background long after it tells an application the file copy is finished. This is one annoying difference from Windows. You can end up with the problem you descibe.

I usually run the sync command before unplugging a storage device. Sync tells Linux to complete all disk operations stored in the memory cache.

Some types of USB 3 adapter chips in external disks can cause weird disk errors.
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Re: Nemo failing when moving big files

Post by macbeto »

Yes, I moved some of the external disks from hfs.plus (for compability with Mac OS) to ext4, and I didn't notice the change, sometimes I moved files to ext4 format disk and I got inconsistency later on. Less usual than the hfs.plus format but also happens.
But it is weird because I make sure the video file can be played from the external disk after moved, I assume is not a problem related with the light of the external device, even I expend so much time after copying the file before unplug it.
After de copy I make a sudo fsck.hfsplus to check the consistency of the disk and I got errors with some files recently copied, so I move again the file with the same result.
I get the most of the errors with a Wester Digital disk, but also with a Toshiba and a Seagate.
linolenux
Level 1
Level 1
Posts: 12
Joined: Mon Dec 21, 2020 5:35 pm

Re: Nemo failing when moving big files

Post by linolenux »

Not Mint, but I had a similar problem with Ubuntu with large file transfers over USB (no problem with SATA or network transfers). Whether using the GUI or command line (cp or rsync), copy operations of large files would fail in various ways using several different devices/cables and destination filesystems.

The first few GB would be fine but then the transfer rate would become slower and slower. If the transfer completed, sometimes the copy would be corrupt, despite appearing to pass a checksum check. Sometimes the transfer would not complete and the drive would unmount with an I/O error.

With the GUI, sometimes the progress indicator would show transfer to complete in an impossibly short amount of time, but writing was obviously continuing because the drive lights were still flashing. Despite the file not being fully copied, the checksums would already match making it necessary to verify hashes from a different machine.

It wasn't a hardware issue because the problem went away with an upgrade (it had appeared suddenly one day and persisted over several updates). It seems like some kind of buffer or caching glitch but I never did figure out what the problem was.
Petermint
Level 9
Level 9
Posts: 2983
Joined: Tue Feb 16, 2016 3:12 am

Re: Nemo failing when moving big files

Post by Petermint »

For USB 3 problems, run the lsusb command and find the USB chip id for the disk. You will get something like:

Code: Select all

Bus 001 Device 007: ID 0bc2:3322 Seagate RSS LLC
Search for the usb chip id + linux.

The problem devices usually work on USB 2. They have a USB 2 chip that was half converted to USB 3. I call them USB 2.5 chips. Windows has lots of workarounds to make the chips work without failing, even if they run randomly slow as a result. Linux does not have all the hacks and sludge required to hide the errors by continuously retrying failed operations.
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Re: Nemo failing when moving big files

Post by macbeto »

Hi,
My lsusb output is:

Code: Select all

Bus 002 Device 004: ID 1058:25a2 Western Digital Technologies, Inc. Elements 25A2
But my impression is that is nothing related with this external device. I've been working with it doing the same actions for 3 years, and it is just the last 6 months, when the disk is at 85% of capacity when I get the most of the errors.
In fact the hfs.plus command gives me errors:

Code: Select all

** /dev/sda2
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
** Checking multi-linked files.
** Checking Catalog hierarchy.
** Checking Extended Attributes file.
** Checking volume bitmap.
** Checking volume information.
   Invalid volume free block count
   (It should be 98009873 instead of 97517972)
** Repairing volume.
** Rechecking volume.
** Checking HFS Plus volume.
** Checking Extents Overflow file.
** Checking Catalog file.
** Checking multi-linked files.
** Checking Catalog hierarchy.
** Checking Extended Attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume Disk3 was repaired successfully.
In that situation anything I move to it with long size is not copied at all, I just did it with a couple of files of about 250 MB each, and after copied I played it well, but after unplug/plug it both files are at 0 bytes. It is pretty disturbing and annoying, I can't delete the file from the laptop because I am not sure it is well copied to the external drive.
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Re: Nemo failing when moving big files

Post by macbeto »

In fact I just used Double Command application to move files with the same result.
I don't think it is just a Nemo issue, I also used commands to love the files and I got the same error.
I will say it is related to the external device, but it happens the same in 3 external dives, one in hfsplus fortmat and two in ext4 format.
User avatar
absque fenestris
Level 12
Level 12
Posts: 4124
Joined: Sat Nov 12, 2016 8:42 pm
Location: Confoederatio Helvetica

Re: Nemo failing when moving big files

Post by absque fenestris »

That 3 external harddisks are faulty at the same time is rather unlikely - what about the connections?
You mentioned HFS+ - how do you have or had the harddisks connected to the Mac under macOS? USB-2, USB-3 or FireWire?
I ask because my Macs with USB-2 were notoriously unreliable.
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Re: Nemo failing when moving big files

Post by macbeto »

Actually, from a year ago I am not using the Mac computer, just a couple of PCs with LM20.1.
I still has one of the external devices in hfs+ format because currently I don't have 2 free Tb to move all the data from the external drive and then format it to ext4. But the Mac system is out of the ecuation before I notice these problems.
User avatar
all41
Level 19
Level 19
Posts: 9505
Joined: Tue Dec 31, 2013 9:12 am
Location: Computer, Car, Cage

Re: Nemo failing when moving big files

Post by all41 »

Is it possible you are unplugging the drive before the file is completely written?
With large files large chunks are cached waiting to be written. Especially considering an
external storage.

Try the same transfer but this time open a terminal and enter

Code: Select all

sync
Wait for the command prompt to return and you will know the write is finished.
Compare the files for size and operation.
Everything in life was difficult before it became easy.
linolenux
Level 1
Level 1
Posts: 12
Joined: Mon Dec 21, 2020 5:35 pm

Re: Nemo failing when moving big files

Post by linolenux »

It's probably a coincidence, but I also had installed something, possibly hfsplus or hfsutils to enable write access to hfs+ volumes. The upgrade which made the problem go away also seems to have removed those packages. The copying issue I experienced was not restricted to hfs+ volumes so I can't see why they would be involved but I mention it as something under the umbrella of disk access and writing.
macbeto
Level 3
Level 3
Posts: 134
Joined: Tue Jan 29, 2019 5:04 am

Re: Nemo failing when moving big files

Post by macbeto »

Thanks for this 2 las advices.
I tryed the sync command while a was moving files, but it finishes fast and without error, but no success, the file remained at 0 bytes.
The same installing hfsutils, no success.
I assume it is a faulty in the disks format, so as soon as I can I will format them.
Thanks all.
User avatar
Lady Fitzgerald
Level 15
Level 15
Posts: 5759
Joined: Tue Jan 07, 2020 3:12 pm
Location: AZ, SSA (Squabbling States of America)

Re: Nemo failing when moving big files

Post by Lady Fitzgerald »

all41 wrote: Fri May 07, 2021 7:39 pm Is it possible you are unplugging the drive before the file is completely written?
With large files large chunks are cached waiting to be written. Especially considering an
external storage...
That also was my first thought. Are you safely removing or ejecting the drive before removing it? I've found if I safely remove or eject the drive from the left pane of the drive's open window, I will get a pop-up in the right upper corner of my screen saying it's not safe to remove the drive yet which, once the drive has finished receiving data, will change to saying it is safe to remove the drive. If I try to reject or safely remove the drive by right clicking the drive's desktop icon, I won't get the message.
Jeannie

To ensure the safety of your data, you have to be proactive, not reactive, so, back it up!
Locked

Return to “Cinnamon”