nemo insanely slow when many files in directory

Please post suggestions for improvement of Cinnamon here
https://github.com/linuxmint/Cinnamon
maxreason
Level 1
Level 1
Posts: 48
Joined: Tue May 24, 2016 4:28 pm
Location: phobos
Contact:

nemo insanely slow when many files in directory

Post by maxreason » Sat Apr 01, 2017 7:12 am

I have a directory that naturally and unavoidably contains a great many files... currently about 62,000 files.

When I try to perform certain operations in that directory with the nemo file browser, nemo "locks up" for a long time. Eventually it seems to break free, probably because it finally finished some process that was taking forever.

This behavior doesn't happen with the same directory with the same content in windows (whole directory just copied across network).

I don't know whether the problem is with nemo (perhaps the "tree widget" or something) or in the filesystem. But I find it difficult to believe that linux filesystems are less efficient than Windows, especially by a factor of many thousand times.

Anyone know what the problem is?

Background...
#1: the files are in a subdirectory of my home directory.
#2: the /home directory is a separate partition on the 500GB SSD that contains "/boot", "swap", "/", "/home".
#3: the /home partition has oodles of free space (387GB free). also, this is a fresh install of linux mint v18.1 as of roughly 1 week ago.

-----

The following probably-related problem is probably even worse. When I want to search for a file in this directory (in nemo), and click the magnifying glass (search) icon, nemo displays a one-line text entry field. When I start typing the characters of the filename, it apparently starts searching before I finish the filename, and sometimes locks-up for 10 or 20 minutes before it finally displays the matching files. The obvious two questions are:

#1: why on earth does the search work this way (not wait for a newline character to be entered to start the search).
#2: how can this possibly take so long?

Note that performing an "ls" or "ls -la" with equivalent wildcards in a terminal window doesn't lock up like this, or take very long. Of course it doesn't start until the newline is pressed, so that's part of the difference.

PS: I couldn't decide what forum this belongs in? Did I guess right?

maxreason
Level 1
Level 1
Posts: 48
Joined: Tue May 24, 2016 4:28 pm
Location: phobos
Contact:

nemo slow when many files in directory

Post by maxreason » Mon Apr 17, 2017 3:55 pm

When I have lots of files in a directory, nemo can get insanely slow... as in lock up for 30 seconds to several minutes. The problem seems to exist when a directory has on the order of 10,000 files or so, and gets horrendous when several times this. I am fairly certain based on running commands in a terminal that the underlying linux or filesystems isn't the cause. So it must be nemo or the tree-widget or something to do with GUI one way or other, I assume. But it really is intolerable!

The easiest way to make this happen is to rename a file. Then try to rename another. You'll find nemo doesn't respond to the mouse click or anything else for a long time... many seconds to many minutes. And I have a fresh new install with fastest Ryzen CPU and Linux Mint 18.1 running off a fast SSD drive. Sooooo... not much else to point the finger at that I can see.

Any ideas? Known issue?

User avatar
jimallyn
Level 18
Level 18
Posts: 8941
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: nemo slow when many files in directory

Post by jimallyn » Mon Apr 17, 2017 5:17 pm

I remember that this was posted once before - and solved - but I looked for the solution a few weeks ago for my own computer, and couldn't find it. Didn't spend much time looking for it, though. You may have better luck than I did.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan

ColdBoot
Level 3
Level 3
Posts: 196
Joined: Thu Feb 16, 2017 10:40 pm

Re: nemo slow when many files in directory

Post by ColdBoot » Mon Apr 17, 2017 5:19 pm

I find this difficult to believe...

Now, I could check what you're talking about only in /var/lib/dpkg/info which contains roughly 12.000+ files on my machine and it took Nemo approximately 1.5sec to display that folder on a slow Kingston SSD and with a much slower CPU than yours. There's nothing "natural"(as you said in another post - for this is a double post, actually) about having this amount of files in a single folder and you should foremost consider making a better file organization for those files. Folder is not database and I'm sure you'll come up with something more appropriate.

Lastly but not least, Windows's file manager is by no means any faster with 10,000+ files folders.
Linux Mint Cinnamon 18.1
Intel G1820, DDR3 8GB, Nvidia GT720(2GB)

User avatar
jimallyn
Level 18
Level 18
Posts: 8941
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: nemo slow when many files in directory

Post by jimallyn » Mon Apr 17, 2017 5:27 pm

Aha, here it is. I must have just used the wrong search terms when I looked for it before. I haven't tried it yet but will later today or tomorrow.

viewtopic.php?t=192984
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan

User avatar
jimallyn
Level 18
Level 18
Posts: 8941
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: nemo insanely slow when many files in directory

Post by jimallyn » Mon Apr 17, 2017 5:29 pm

Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan

User avatar
Moem
Level 19
Level 19
Posts: 9002
Joined: Tue Nov 17, 2015 9:14 am
Location: The Netherlands
Contact:

Re: nemo insanely slow when many files in directory

Post by Moem » Mon Apr 17, 2017 5:40 pm

(Two very similar topics merged. Please don't double-post.)
Image

If your issue is solved, kindly indicate that by editing the first post in the topic, and adding [SOLVED] to the title. Thanks!

User avatar
jimallyn
Level 18
Level 18
Posts: 8941
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: nemo insanely slow when many files in directory

Post by jimallyn » Mon Apr 17, 2017 5:48 pm

Thanks, Moem. Didn't notice they were both from the same poster. My "noticer" isn't working well these days.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan

Cosmo.
Level 23
Level 23
Posts: 17827
Joined: Sat Dec 06, 2014 7:34 am

Re: nemo insanely slow when many files in directory

Post by Cosmo. » Mon Apr 17, 2017 5:49 pm

There are (at least) 2 bug reports, which might be related with the issue: Regarding Nemo 3.2.2 and this about Nemo 3.0.6.

maxreason
Level 1
Level 1
Posts: 48
Joined: Tue May 24, 2016 4:28 pm
Location: phobos
Contact:

Re: nemo insanely slow when many files in directory

Post by maxreason » Tue Apr 18, 2017 3:11 am

Thanks for the link to the "resolve slow loading of large directories in nemo" thread. BTW, that wasn't my thread. I did create a similar thread, but before I just posted this new question I checked, and it showed ZERO replies. Which means, I did not consider my problem to be "solved"... at all.

One question though. My directory has no video files, only.jpgimage files. I notice the solution involves "ffmpegthumbnailer" which sounds like it is designed for video files (hopefully not just.mpgtype video files). So my [obvious] question is the following: does that solution also handle all/most image file types and/or all/most video file types?

Thanks for helping me figure this out. In retrospect I should probably have wondered whether the problem had anything to do with the types of files in the directory.

Oh, a related question. When I move files from directory to directory, this seems to cause problems in the thumbnail cache [synchronization], because sometimes nemo posts a message asking whether to fix the thumbnail cache or "dismiss". Why would nemo not fix the thumbnail cache automatically, since the moves and copies were done within nemo, and thus nemo is aware of everything it needs to know to keep its thumbnail cache in sync.

Another related question. Can the thumbnail cache be disabled so nemo has to re-create the thumbnails every time a directory is viewed? On a fast system, this might be less annoying that all the weirdness and slowness that happens now. Note that I rarely look at the directory contents as thumbnails anyway... my default and usual way to view and manipulate directories and files is individual lines of text (one line per directory/file), not thumbnails.

And one side question... is there an alternative to nemo that is essentially similar, but perhaps better in terms of performance and/or reliability? Maybe other linux fans work differently than me (more in terminals perhaps), but for people like me, a file-browser application is one of the most important tools, and one that I leave on the desktop and perform actions with dozens if not hundreds of times on a typical day. So I'm a bit surprised extra attention isn't made to assure this application is one of the most solid, reliable and efficient. Maybe one of the alternatives is? Can I assume there is no reason an alternative can't be substituted (if one exists)?

User avatar
jimallyn
Level 18
Level 18
Posts: 8941
Joined: Thu Jun 05, 2014 7:34 pm
Location: Wenatchee, WA USA

Re: nemo insanely slow when many files in directory

Post by jimallyn » Tue Apr 18, 2017 4:24 am

I made the change to ffmpegthumbnailer. It is indeed a lot faster. I opened a directory that contains a lot of images and a few videos. I would guess it works with any file type.

It might be difficult to substitute another file manager for Nemo, but you should certainly be able to install additional file managers. I have Gnome Commander on my computer. In some areas, it has advantages over Nemo, in other areas Nemo is better. Gnome Commander is not fancy, but it works well.
Image

“If the government were coming for your TVs and cars, then you'd be upset. But, as it is, they're only coming for your sons.” - Daniel Berrigan

maxreason
Level 1
Level 1
Posts: 48
Joined: Tue May 24, 2016 4:28 pm
Location: phobos
Contact:

Re: nemo insanely slow when many files in directory

Post by maxreason » Tue Apr 18, 2017 4:50 am

Thanks, glad to hear that worked for you.

So I started to perform the steps, but:

- [b]sudo aptitude install ffmpegthumbnailer[/b] ... worked
- [b]cd /user/share/thumbnailers[/b] ... worked
- [b]sudo cp totem.thumbnailer totem.thumbnailer.original[/b] ... did not work because no file with "totem" in the name exists in that directory.

That directory contains these files:

ffmpegthumbnailer.thumbnailer
gnome-font-viewer.thumbnailer
xplayer.thumbnailer
xreader.thumbnailer

None of those appear to be likely prospects... though maybe one is.

Therefore, I'm more-or-less stuck. Any ideas?

Cosmo.
Level 23
Level 23
Posts: 17827
Joined: Sat Dec 06, 2014 7:34 am

Re: nemo insanely slow when many files in directory

Post by Cosmo. » Tue Apr 18, 2017 6:30 am

If you want to disable the thumbnails you can try this (not tested by myself):
At first install the package dconf-editor. This tool gives you a GUI for the hidden settings for the system. It is organized similar to a file manager.
Now navigate inside of this editor to org -> cinnamon -> desktop -> thumbnailers. Activate the option to disable-all.
You can also try to go to org -> cinnamon -> desktop -> thumbnail-cache and reduce both offered options to 1 (or another value, if you prefer).

In my system (Mint 17.3 with Cinnamon) I do not encounter the problem, so I did never change those settings myself.

ColdBoot
Level 3
Level 3
Posts: 196
Joined: Thu Feb 16, 2017 10:40 pm

Re: nemo insanely slow when many files in directory

Post by ColdBoot » Tue Apr 18, 2017 10:06 am

IMO, if it's impossible to break down the file structure and organize them better within folders of this magnitude, it's best to keep them for reference only. Not browsed or searched by Nemo but left to some desktop-searching utility since they use indexed searches. Beagle, Recoll, Tracker, are all much better and faster tools for this purpose than Nemo.
Linux Mint Cinnamon 18.1
Intel G1820, DDR3 8GB, Nvidia GT720(2GB)

ColdBoot
Level 3
Level 3
Posts: 196
Joined: Thu Feb 16, 2017 10:40 pm

Re: nemo insanely slow when many files in directory

Post by ColdBoot » Tue Apr 18, 2017 9:21 pm

Besides, for strictly image browsing, image browsers might do better than Nemo. Just tried Pix (installed by default) and it looks very nice though as I have nowhere near that amount of pictures OP has, cannot tell how it performs in such a case.

Edit: nope... just compared it to XnView (a deb package available on website) and Pix is obviously not using an indexed image search because the speed difference between them is startling. Bet it can handle directories containing idiotic number of images with equal speed. :mrgreen:
Linux Mint Cinnamon 18.1
Intel G1820, DDR3 8GB, Nvidia GT720(2GB)

User avatar
greerd
Level 6
Level 6
Posts: 1055
Joined: Sat Jul 31, 2010 10:58 am
Location: Nova Scotia, Canada

Re: nemo insanely slow when many files in directory

Post by greerd » Wed Apr 19, 2017 9:20 pm

maxreason wrote:Thanks, glad to hear that worked for you.

So I started to perform the steps, but:

- [b]sudo aptitude install ffmpegthumbnailer[/b] ... worked
- [b]cd /user/share/thumbnailers[/b] ... worked
- [b]sudo cp totem.thumbnailer totem.thumbnailer.original[/b] ... did not work because no file with "totem" in the name exists in that directory.

That directory contains these files:

ffmpegthumbnailer.thumbnailer
gnome-font-viewer.thumbnailer
xplayer.thumbnailer
xreader.thumbnailer

None of those appear to be likely prospects... though maybe one is.

Therefore, I'm more-or-less stuck. Any ideas?
If you want to continue on, try replacing totem with xplayer, and use xed instead of gedit (in the next step).

crash21
Level 1
Level 1
Posts: 1
Joined: Sat Jun 10, 2017 11:13 am

Re: nemo insanely slow when many files in directory

Post by crash21 » Sat Jun 10, 2017 11:49 am

I am on 18.1 and nemo on 3.2. Nemo is slow form the start every time, on HOME, DESKTOP etc, All folders take 2-5 seconds to swap, every-time I click them. Even to close button takes some time. Also I see that my CPU is about 30-40% constantly when I have files browser open.

I have disabled all plugins/scripts but still same.

I have also installed thunar. The difference is that thunar, it's slow only the first time I open or reopen it. So when I open or reopen it it takes 3-6 seconds to browse to a folder, something like initialization lag and also see 30-40% CPU. But after that it's fast and browsing is fine....

Any ideas?

EDIT: One other difference I spot in thulnar, is that on all folders the size is 4.1KB which of course is wrong, in files it's right but ion folders it's wrong.... But this calculation error make it faster?

User avatar
Lumikki
Level 4
Level 4
Posts: 232
Joined: Wed Feb 01, 2012 5:21 pm

Re: nemo insanely slow when many files in directory

Post by Lumikki » Wed Jun 28, 2017 1:44 am

I will borrow this subject, even if my situation is little different. I have Mint 18.1 Cinnamon.

I have notice few small issue with Nemo / file browsing system.

These issues are hard to notice if you filesystem is very fast, but they are still there.

1. When You want to access files in some directory and that directory has subdirectories where is a lot of files. Nemo will not give access to current directory files before it has scanned how many files are in that subdirectory. I know, with Nemo setting this can be fixed as not to scan subdirectories. How ever issue is that Nemo isn't multitasking situations. Example allow access to directory files while still scanning subdirectories. Next example will shows it even better.

2. When you have directory where is NFS subdirectory, but Network access has been cut off. Nemo can't handle situation. Because it's single minded and tries to go stuff one by one. When it can't do someting it just get stuck in it. Problem is same as in number 1. Nemo can't priorities what to do or multitask situation.

These situation cause Nemo been extreme slow. Not because Nemo would be slow, but because it's can't handle sertain situation well. Nemo should run subdirectory file count calculation as seperate process after the currect directory is handeled. Giving access to directory and it's files before subdirectory calculation. Point been, in my opinion Nemo need better ways priorities how to handle directory access/content.
Asus P7P55D, i5 750 2.6Ghz, 4GB DDR3, GeForce 750Ti, 80GB Intel SSD, Dell 1600x1200, Dual boot, Linux Mint 18 Cinnamon, Windows 7

stans78
Level 1
Level 1
Posts: 4
Joined: Tue Jul 25, 2017 2:00 am

Re: nemo insanely slow when many files in directory

Post by stans78 » Wed Jul 26, 2017 2:00 am

I've been having a similar problem. The switch to ffmpegthumbnailer helped on the internal HDD, but not on an external 500GB USB drive.

Some thoughts/observations:

I first noticed this issue earlier this year when I was using Xubuntu and upgraded from 14.04 to 16.04. This is why I switched to Mint, but the problem has followed me.

I've tried a couple of other file managers (Gnome-Commander, EmelFM2) to no avail.

I also tried the Debian 8.8/Cinnamon live CD and everything was fine.

So I'm thinking that this may not be a Nemo problem, but one of the underlying OS that is shared by Ubuntu and Mint, but not Debian.
Linux Mint 18.1 64bit / Cinnamon

II-Trax-II
Level 2
Level 2
Posts: 67
Joined: Sun Jul 03, 2016 3:18 am

Re: nemo insanely slow when many files in directory

Post by II-Trax-II » Mon Jul 31, 2017 9:01 pm

maxreason wrote: Oh, a related question. When I move files from directory to directory, this seems to cause problems in the thumbnail cache [synchronization], because sometimes nemo posts a message asking whether to fix the thumbnail cache or "dismiss". Why would nemo not fix the thumbnail cache automatically, since the moves and copies were done within nemo, and thus nemo is aware of everything it needs to know to keep its thumbnail cache in sync.
I Dual boot as most yet the only thing I use Windows for is moving huge directories (~160 Gig) Linux takes twice as long.

Edit: Sata3 to Sata3

Post Reply

Return to “Cinnamon”