<SOLVED> When copying large files, system freezes

Questions about other topics - please check if your question fits better in another category before posting here
Forum rules
Before you post please read how to get help
Locked
User avatar
thedoctar
Level 1
Level 1
Posts: 22
Joined: Wed Oct 26, 2011 5:23 am
Location: Sydney, Australia

<SOLVED> When copying large files, system freezes

Post by thedoctar » Sun Mar 04, 2012 1:24 am

When I copy large files, several gigabytes, my system freezes. I notice that when I do copy large files that all of my RAM is used up mostly by cache. I know this because I added the system monitor applet on the panel and made it display memory.

I use Linux Mint 11, linux kernel 2.6.38-8-generic 64bit and have 3gb of RAM, no swap.
Last edited by thedoctar on Tue Mar 06, 2012 8:46 am, edited 1 time in total.
Image

User avatar
xenopeek
Level 24
Level 24
Posts: 22660
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: When copying large files, system freezes

Post by xenopeek » Sun Mar 04, 2012 6:51 am

From where to where are you copying gigabytes of data? If it is on the same hard disk, yes this can appear to freeze your system. If you don't need to do this daily, perhaps get something to drink or a snack as you wait for copy action to complete :wink:

You can try to start Nautilus so as to be nicer to other programs, this may prevent freezing but the copy action might run longer if you continue to use your computer during the copy action. From a terminal or the Alt+F2 dialog, run the following and then do your copy action fully in that Nautilus window.

Code: Select all

nice -n 19 nautilus
RAM being used by cache is not a problem. The Linux kernel uses RAM not currently in use by programs as a file cache. But the moment some program needs memory that is currently in use as cache, that memory becomes immediately available to that program. So cache is using memory only as long as nobody else needs it, to optimize file system operations.
Image

User avatar
thedoctar
Level 1
Level 1
Posts: 22
Joined: Wed Oct 26, 2011 5:23 am
Location: Sydney, Australia

Re: When copying large files, system freezes

Post by thedoctar » Mon Mar 05, 2012 2:15 am

Thanks for the reply.
Vincent Vermeulen wrote:From where to where are you copying gigabytes of data? If it is on the same hard disk, yes this can appear to freeze your system. If you don't need to do this daily, perhaps get something to drink or a snack as you wait for copy action to complete :wink:
I was copying files from my hard disk to a portable hard disk. I was copying a few films, around 3 gigabytes all up
Vincent Vermeulen wrote:You can try to start Nautilus so as to be nicer to other programs, this may prevent freezing but the copy action might run longer if you continue to use your computer during the copy action. From a terminal or the Alt+F2 dialog, run the following and then do your copy action fully in that Nautilus window.

Code: Select all

nice -n 19 nautilus
RAM being used by cache is not a problem. The Linux kernel uses RAM not currently in use by programs as a file cache. But the moment some program needs memory that is currently in use as cache, that memory becomes immediately available to that program. So cache is using memory only as long as nobody else needs it, to optimize file system operations.
Initially, when I was copying the films, the films originated from the ntfs file system of my hard drive, so I tried what you suggested, I opened a nautilus window with niceness of 19, and copied something from my home folder to my desktop, 3 gigabytes, but after transferring about 1 gigabyte, my system froze. Although I know cache isn't really technically 'used', I still believe it has something to do with RAM or a memory leak or something.

Also, on a related point, I had a tar.gz file which was around 4 gigabytes, however when trying to open the archive, my system also froze. I thought there might be a possibility that the archive was being loaded into RAM, which there is less than 4 gigabytes, and when I had no RAM left, my system would freeze. Is it possible that something similar is happening with copying large files?
Image

User avatar
xenopeek
Level 24
Level 24
Posts: 22660
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: When copying large files, system freezes

Post by xenopeek » Mon Mar 05, 2012 4:50 am

Of course, as long as you have memory available to cache files, file copy operations go fast. As it can read data and independently write that data, where the memory acts as a buffer. As memory fills up (as read operations are faster than write operations), you come to a point that will cause the system to freeze. Either copy in smaller chucks, try a different file browser than Nautilus, or accept it and do you copy operations in a big bulk and leave your computer to its work while you do something else :wink:

As for the tar file, I expect if you would untar it from the command prompt it would not hang.
Image

User avatar
thedoctar
Level 1
Level 1
Posts: 22
Joined: Wed Oct 26, 2011 5:23 am
Location: Sydney, Australia

Re: When copying large files, system freezes

Post by thedoctar » Tue Mar 06, 2012 2:21 am

Do you know any other file managers which don't freeze when copying large files?

And also, I also attempted copying a large folder from my home folder to my desktop using the command

Code: Select all

nice -n 19 cp -rt ~/Desktop ~/folder
and my system also froze. Should this happen? As I am using the terminal, so it shouldn't freeze, right?
Image

User avatar
xenopeek
Level 24
Level 24
Posts: 22660
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: When copying large files, system freezes

Post by xenopeek » Tue Mar 06, 2012 3:02 am

Well, for copy actions that wouldn't differ. I mentioned for unarchive command it may differ, but on second thought if you have 4 GiB archives this will probably not work either.

I'm not sure how else to improve this. Perhaps somebody else can pipe in here with thoughts.

I'd just take a break from the computer when copying or unarchive files this large... A high-end SSD may improve disk-to-disk copying, but this won't help if you are copying data to external devices over slow connections (e.g., USB).
Image

User avatar
thedoctar
Level 1
Level 1
Posts: 22
Joined: Wed Oct 26, 2011 5:23 am
Location: Sydney, Australia

Re: When copying large files, system freezes

Post by thedoctar » Tue Mar 06, 2012 4:57 am

Oh, well. Thanks for your help. One last question, do you know why, in Windows, the system doesn't freeze when copying large files?
Image

User avatar
xenopeek
Level 24
Level 24
Posts: 22660
Joined: Wed Jul 06, 2011 3:58 am
Location: The Netherlands

Re: When copying large files, system freezes

Post by xenopeek » Tue Mar 06, 2012 6:08 am

Well, I'd be happy to find out, but since Windows is closed source we will never know :wink:

Edit: as griffon shares in next comment, on my system with SSD disk-to-disk copying is fast. One thing I did observe though is that it can be slowed significantly if you have an encrypted home folder. Is that perhaps your root cause? The overhead that each byte read has to be decrypted first, and each byte written has to be encrypted first?
Image

griffon
Level 1
Level 1
Posts: 19
Joined: Thu Feb 16, 2012 4:29 pm

Re: When copying large files, system freezes

Post by griffon » Tue Mar 06, 2012 6:54 am

I just don't see this problem. I have just tried copying a 7GB file first to another hard drive and then to the same drive. About 2mins for the first and about 5mins for the second. The system remained perfectly responsive throughout and RAM usage hardly changed from 450MB out of my paltry 2GB installed. I do have an SSD as the system 'drive'. (And I've reverted to plain Ubuntu 11.04 after problems with serial tty using Mint Lisa and equivalent Ubuntu).

User avatar
thedoctar
Level 1
Level 1
Posts: 22
Joined: Wed Oct 26, 2011 5:23 am
Location: Sydney, Australia

Re: When copying large files, system freezes

Post by thedoctar » Tue Mar 06, 2012 8:44 am

I may have found a solution. The problem existed on kernel 2.6.38-8 and 2.6.39-020639rc4, however on 3.0.18-030018, a lot of RAM was used, but my system didn't, and running free -ms 1, There was always 100MB of RAM left free. After however, the cache files in RAM weren't cleared and were stilling taking up space. Is this normal? I don't like using the 3.0 kernel because there's a problem with it, as it keeps waking up the CPU cores and the CPU temperature is always higher (my CPU is 2nd gen sandybridge), so I will try to find a 2.6 kernel without this freezing problem.

I'll mark the thread solved, as I've found the source of the problem.
Image

balsoft
Level 1
Level 1
Posts: 1
Joined: Mon Jul 09, 2018 10:04 am

Re: <SOLVED> When copying large files, system freezes

Post by balsoft » Mon Jul 09, 2018 10:07 am

I understand that this is an old thread, but now there is a solution to the problem:

Code: Select all

ionice -c 3 cp -r $FROM $TO

Locked

Return to “Other topics”