nemo insanely slow when many files in directory

[maxreason]

I have a directory that naturally and unavoidably contains a great many files... currently about 62,000 files.

When I try to perform certain operations in that directory with the nemo file browser, nemo "locks up" for a long time. Eventually it seems to break free, probably because it finally finished some process that was taking forever.

This behavior doesn't happen with the same directory with the same content in windows (whole directory just copied across network).

I don't know whether the problem is with nemo (perhaps the "tree widget" or something) or in the filesystem. But I find it difficult to believe that linux filesystems are less efficient than Windows, especially by a factor of many thousand times.

Anyone know what the problem is?

Background...
#1: the files are in a subdirectory of my home directory.
#2: the /home directory is a separate partition on the 500GB SSD that contains "/boot", "swap", "/", "/home".
#3: the /home partition has oodles of free space (387GB free). also, this is a fresh install of linux mint v18.1 as of roughly 1 week ago.

-----

The following probably-related problem is probably even worse. When I want to search for a file in this directory (in nemo), and click the magnifying glass (search) icon, nemo displays a one-line text entry field. When I start typing the characters of the filename, it apparently starts searching before I finish the filename, and sometimes locks-up for 10 or 20 minutes before it finally displays the matching files. The obvious two questions are:

#1: why on earth does the search work this way (not wait for a newline character to be entered to start the search).
#2: how can this possibly take so long?

Note that performing an "ls" or "ls -la" with equivalent wildcards in a terminal window doesn't lock up like this, or take very long. Of course it doesn't start until the newline is pressed, so that's part of the difference.

PS: I couldn't decide what forum this belongs in? Did I guess right?

[maxreason]

When I have lots of files in a directory, nemo can get insanely slow... as in lock up for 30 seconds to several minutes. The problem seems to exist when a directory has on the order of 10,000 files or so, and gets horrendous when several times this. I am fairly certain based on running commands in a terminal that the underlying linux or filesystems isn't the cause. So it must be nemo or the tree-widget or something to do with GUI one way or other, I assume. But it really is intolerable!

The easiest way to make this happen is to rename a file. Then try to rename another. You'll find nemo doesn't respond to the mouse click or anything else for a long time... many seconds to many minutes. And I have a fresh new install with fastest Ryzen CPU and Linux Mint 18.1 running off a fast SSD drive. Sooooo... not much else to point the finger at that I can see.

Any ideas? Known issue?

[jimallyn]

I remember that this was posted once before - and solved - but I looked for the solution a few weeks ago for my own computer, and couldn't find it. Didn't spend much time looking for it, though. You may have better luck than I did.

[ColdBoot]

I find this difficult to believe...

Now, I could check what you're talking about only in /var/lib/dpkg/info which contains roughly 12.000+ files on my machine and it took Nemo approximately 1.5sec to display that folder on a slow Kingston SSD and with a much slower CPU than yours. There's nothing "natural"(as you said in another post - for this is a double post, actually) about having this amount of files in a single folder and you should foremost consider making a better file organization for those files. Folder is not database and I'm sure you'll come up with something more appropriate.

Lastly but not least, Windows's file manager is by no means any faster with 10,000+ files folders.

[jimallyn]

Aha, here it is. I must have just used the wrong search terms when I looked for it before. I haven't tried it yet but will later today or tomorrow.

viewtopic.php?t=192984

[jimallyn]

viewtopic.php?t=192984

[Moem]

(Two very similar topics merged. Please don't double-post.)

[jimallyn]

Thanks, Moem. Didn't notice they were both from the same poster. My "noticer" isn't working well these days.

[Cosmo.]

There are (at least) 2 bug reports, which might be related with the issue: Regarding Nemo 3.2.2 and this about Nemo 3.0.6.

[maxreason]

Thanks for the link to the "resolve slow loading of large directories in nemo" thread. BTW, that wasn't my thread. I did create a similar thread, but before I just posted this new question I checked, and it showed ZERO replies. Which means, I did not consider my problem to be "solved"... at all.

One question though. My directory has no video files, only.jpgimage files. I notice the solution involves "ffmpegthumbnailer" which sounds like it is designed for video files (hopefully not just.mpgtype video files). So my [obvious] question is the following: does that solution also handle all/most image file types and/or all/most video file types?

Thanks for helping me figure this out. In retrospect I should probably have wondered whether the problem had anything to do with the types of files in the directory.

Oh, a related question. When I move files from directory to directory, this seems to cause problems in the thumbnail cache [synchronization], because sometimes nemo posts a message asking whether to fix the thumbnail cache or "dismiss". Why would nemo not fix the thumbnail cache automatically, since the moves and copies were done within nemo, and thus nemo is aware of everything it needs to know to keep its thumbnail cache in sync.

Another related question. Can the thumbnail cache be disabled so nemo has to re-create the thumbnails every time a directory is viewed? On a fast system, this might be less annoying that all the weirdness and slowness that happens now. Note that I rarely look at the directory contents as thumbnails anyway... my default and usual way to view and manipulate directories and files is individual lines of text (one line per directory/file), not thumbnails.

And one side question... is there an alternative to nemo that is essentially similar, but perhaps better in terms of performance and/or reliability? Maybe other linux fans work differently than me (more in terminals perhaps), but for people like me, a file-browser application is one of the most important tools, and one that I leave on the desktop and perform actions with dozens if not hundreds of times on a typical day. So I'm a bit surprised extra attention isn't made to assure this application is one of the most solid, reliable and efficient. Maybe one of the alternatives is? Can I assume there is no reason an alternative can't be substituted (if one exists)?

[jimallyn]

I made the change to ffmpegthumbnailer. It is indeed a lot faster. I opened a directory that contains a lot of images and a few videos. I would guess it works with any file type.

It might be difficult to substitute another file manager for Nemo, but you should certainly be able to install additional file managers. I have Gnome Commander on my computer. In some areas, it has advantages over Nemo, in other areas Nemo is better. Gnome Commander is not fancy, but it works well.

[maxreason]

Thanks, glad to hear that worked for you.

So I started to perform the steps, but:

- [b]sudo aptitude install ffmpegthumbnailer[/b] ... worked
- [b]cd /user/share/thumbnailers[/b] ... worked
- [b]sudo cp totem.thumbnailer totem.thumbnailer.original[/b] ... did not work because no file with "totem" in the name exists in that directory.

That directory contains these files:

ffmpegthumbnailer.thumbnailer
gnome-font-viewer.thumbnailer
xplayer.thumbnailer
xreader.thumbnailer

None of those appear to be likely prospects... though maybe one is.

Therefore, I'm more-or-less stuck. Any ideas?

[Cosmo.]

If you want to disable the thumbnails you can try this (not tested by myself):
At first install the package dconf-editor. This tool gives you a GUI for the hidden settings for the system. It is organized similar to a file manager.
Now navigate inside of this editor to org -> cinnamon -> desktop -> thumbnailers. Activate the option to disable-all.
You can also try to go to org -> cinnamon -> desktop -> thumbnail-cache and reduce both offered options to 1 (or another value, if you prefer).

In my system (Mint 17.3 with Cinnamon) I do not encounter the problem, so I did never change those settings myself.

[ColdBoot]

IMO, if it's impossible to break down the file structure and organize them better within folders of this magnitude, it's best to keep them for reference only. Not browsed or searched by Nemo but left to some desktop-searching utility since they use indexed searches. Beagle, Recoll, Tracker, are all much better and faster tools for this purpose than Nemo.

[ColdBoot]

Besides, for strictly image browsing, image browsers might do better than Nemo. Just tried Pix (installed by default) and it looks very nice though as I have nowhere near that amount of pictures OP has, cannot tell how it performs in such a case.

Edit: nope... just compared it to XnView (a deb package available on website) and Pix is obviously not using an indexed image search because the speed difference between them is startling. Bet it can handle directories containing idiotic number of images with equal speed.

[greerd]

Thanks, glad to hear that worked for you.

So I started to perform the steps, but:

- [b]sudo aptitude install ffmpegthumbnailer[/b] ... worked
- [b]cd /user/share/thumbnailers[/b] ... worked
- [b]sudo cp totem.thumbnailer totem.thumbnailer.original[/b] ... did not work because no file with "totem" in the name exists in that directory.

That directory contains these files:

ffmpegthumbnailer.thumbnailer
gnome-font-viewer.thumbnailer
xplayer.thumbnailer
xreader.thumbnailer

None of those appear to be likely prospects... though maybe one is.

Therefore, I'm more-or-less stuck. Any ideas?

If you want to continue on, try replacing totem with xplayer, and use xed instead of gedit (in the next step).

[crash21]

I am on 18.1 and nemo on 3.2. Nemo is slow form the start every time, on HOME, DESKTOP etc, All folders take 2-5 seconds to swap, every-time I click them. Even to close button takes some time. Also I see that my CPU is about 30-40% constantly when I have files browser open.

I have disabled all plugins/scripts but still same.

I have also installed thunar. The difference is that thunar, it's slow only the first time I open or reopen it. So when I open or reopen it it takes 3-6 seconds to browse to a folder, something like initialization lag and also see 30-40% CPU. But after that it's fast and browsing is fine....

Any ideas?

EDIT: One other difference I spot in thulnar, is that on all folders the size is 4.1KB which of course is wrong, in files it's right but ion folders it's wrong.... But this calculation error make it faster?

[Lumikki]

I will borrow this subject, even if my situation is little different. I have Mint 18.1 Cinnamon.

I have notice few small issue with Nemo / file browsing system.

These issues are hard to notice if you filesystem is very fast, but they are still there.

1. When You want to access files in some directory and that directory has subdirectories where is a lot of files. Nemo will not give access to current directory files before it has scanned how many files are in that subdirectory. I know, with Nemo setting this can be fixed as not to scan subdirectories. How ever issue is that Nemo isn't multitasking situations. Example allow access to directory files while still scanning subdirectories. Next example will shows it even better.

2. When you have directory where is NFS subdirectory, but Network access has been cut off. Nemo can't handle situation. Because it's single minded and tries to go stuff one by one. When it can't do someting it just get stuck in it. Problem is same as in number 1. Nemo can't priorities what to do or multitask situation.

These situation cause Nemo been extreme slow. Not because Nemo would be slow, but because it's can't handle sertain situation well. Nemo should run subdirectory file count calculation as seperate process after the currect directory is handeled. Giving access to directory and it's files before subdirectory calculation. Point been, in my opinion Nemo need better ways priorities how to handle directory access/content.

[stans78]

I've been having a similar problem. The switch to ffmpegthumbnailer helped on the internal HDD, but not on an external 500GB USB drive.

Some thoughts/observations:

I first noticed this issue earlier this year when I was using Xubuntu and upgraded from 14.04 to 16.04. This is why I switched to Mint, but the problem has followed me.

I've tried a couple of other file managers (Gnome-Commander, EmelFM2) to no avail.

I also tried the Debian 8.8/Cinnamon live CD and everything was fine.

So I'm thinking that this may not be a Nemo problem, but one of the underlying OS that is shared by Ubuntu and Mint, but not Debian.

[II-Trax-II]

Oh, a related question. When I move files from directory to directory, this seems to cause problems in the thumbnail cache [synchronization], because sometimes nemo posts a message asking whether to fix the thumbnail cache or "dismiss". Why would nemo not fix the thumbnail cache automatically, since the moves and copies were done within nemo, and thus nemo is aware of everything it needs to know to keep its thumbnail cache in sync.

I Dual boot as most yet the only thing I use Windows for is moving huge directories (~160 Gig) Linux takes twice as long.

Edit: Sata3 to Sata3