[SOLVED] Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Questions about applications and software
Forum rules
Before you post read how to get help. Topics in this forum are automatically closed 6 months after creation.
Post Reply
User avatar
Drugwash
Level 5
Level 5
Posts: 734
Joined: Fri Jun 07, 2019 6:40 am
Location: Ploieşti, RO
Contact:

[SOLVED] Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by Drugwash »

Chances are this won't get any (useful) answer but posting here nonetheless just in case.

This morning after a reboot there were no icons on the desktop anymore. Looking around I found nemo-desktop was crashing with segmentation fault and there were core dumps in /var/lib/systemd/coredump/ around 10-12MB in lz4 archive format.

Also in /var/log/syslog there are repeated mentions of Python3.6 (libpython3.6m.so.1.0) and others related to Python. Unfortunately I'm no expert in decoding the logs and crash dumps.

Before anyone suggests it: no, manually starting nemo-desktop in terminal won't help, it yields the same segfault over and over again on each start.

These days I have been working on a Python 3 script, performed lots of test until at some point the script started crashing on exit yesterday evening. I had not installed or removed anything before that for quite a while. After the crashes I downgraded a couple Python modules (urllib3 and chardet-normalizer) which the requests module was complaining about with a warning about those two dependencies being out of range or something (too high versions).
That was the only recent change I performed before the reboot and the subsequent segfaults. For some reason I couldn't reinstall the high versions of those modules anymore.

Anyway, I know installing additional modules possibly overriding system-managed packages is risky and there is already a mix of official and unoficial modules - some in global dist-packages/site-packages and others in user site-packages - so I'm reluctant on starting to mess with them now, more so when it all used to work.

I'd appreciate some educated pointers as to what exactly could be the problem, whether it's some module misbehaving or something else that could trigger the issue. Can't attach the core dump here as it's too large but if anyone thinks they could help I could upload to my Dropbox account. Only attached part of the syslog related to one of the crashes:
nemo-desktop crash.txt
(5.78 KiB) Downloaded 11 times
Last edited by Drugwash on Wed Mar 27, 2024 9:14 am, edited 1 time in total.
cptnoblivious
Level 1
Level 1
Posts: 45
Joined: Wed Jan 06, 2021 8:29 pm

Re: Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by cptnoblivious »

I mean, LM19 has been EOL for close to a year ...
User avatar
Drugwash
Level 5
Level 5
Posts: 734
Joined: Fri Jun 07, 2019 6:40 am
Location: Ploieşti, RO
Contact:

Re: Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by Drugwash »

I use what I want. It's the principle that matters: a crash, some details in a log.
Last edited by xenopeek on Mon Mar 25, 2024 4:24 pm, edited 1 time in total.
Reason: removed hostile reply
User avatar
zcot
Level 9
Level 9
Posts: 2838
Joined: Wed Oct 19, 2016 6:08 pm

Re: Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by zcot »

https://packages.ubuntu.com/focal/gir1.2-webkit2-4.0

note on that info, that specific package version is available and functional on focal and jammy, that's ubuntu 20.04 and 22.04 LTS bases, which translates to mint 20.x and mint 21.x, but it is not available anymore on bionic 18.04, which is mint 19.x.

You can not take the file from jammy version and put it on focal version and expect a good result. vice-versa. And bionic too.

There are other dependencies from there. Here's a simple example based from a focal(mint 20.x) system:

Code: Select all

Depends: gir1.2-javascriptcoregtk-4.0 (= 2.38.6-0ubuntu0.20.04.1), gir1.2-gtk-3.0 (>= 3.10.0), gir1.2-soup-2.4 (>= 2.42.0), libwebkit2gtk-4.0-37 (>= 2.37.1)
You could try to install a wrong one on another system and the dependency would only be checked for version, and that might actually pass, luckily! -only because the same version numbers, or close enough, are used.

But these packages are pre-compiled for us, which is all very specific, and using linkages to other dependency packages for that system version. That's an important note.

You can't mix them up between the different base versions.

that package, on an incompatible system, will try to use linkages to other modules that are going to reside in a different memory footprint. All the logs full of memory segfaults are showing a problem that is not correctable while using incompatible packaging like that.

A solution you might try would be to get the sources for these packages and compile them on this previous version system. The best scenario is that there would barely be any changes required of the source code but at least the calls, memory footprint, it would produce a functional operation, but the worse case of it, is that there could be 1000's of required changes to rewrite. -like for example HexChat, which is closer to the 1,000,000 lines mark.

The even more functional option, in the development of a Jargonaut fork, is to modify that existing code instead to where it doesn't need to use the calls of the more modern version, but you literally yourself backport the intended functionality to be able to accomplish the goal but using the older supported libs. The thing about that, is that then you only have to change a much more minimal amount of calls to the libs, instead of having to change not only the wrong version libs and the 10's of 1000's of lines of the other parts of the functionality of the newer libs.

To fix your system, you need to get the original correct versions for that system back into place, the ones made for bionic and bionic only.
JosephM
Level 6
Level 6
Posts: 1463
Joined: Sun May 26, 2013 6:25 pm

Re: Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by JosephM »

I'd appreciate some educated pointers as to what exactly could be the problem
Don't mess with your system python install if you don't know how to get back to what it should be. And maybe don't use unsupported versions of a distro if you want to use newer things.
When I give opinions, they are my own. Not necessarily those of any other Linux Mint developer or the Linux Mint project as a whole.
User avatar
Drugwash
Level 5
Level 5
Posts: 734
Joined: Fri Jun 07, 2019 6:40 am
Location: Ploieşti, RO
Contact:

Re: Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by Drugwash »

zcot wrote: Mon Mar 25, 2024 7:53 pm To fix your system, you need to get the original correct versions for that system back into place, the ones made for bionic and bionic only.
Thank you for the extensive reply.
The version of that package is precisely the original one for Mint 19: 2.32.4-0ubuntu0.18.04.1. It was already installed and I did not mess with its dependencies. But it is possible that other packages added at some later time could have introduced an incompatibility.
Is the webkit library mentioned in that crash report? Can't find any mention of it.

Thing is, I did install over time quite a few various system packages from an unofficial PPA that provides backports specifically for Bionic. Haven't had any problems with them. Last time I rebooted the system before yesterday was about six-seven days ago, after a batch of updates, and there was no nemo-desktop crash or anything - at least nothing visible or palpable.

Here's the list of updates performed on the 19th through Update Manager. Some of them are updates to unofficial versions already installed previously:

Code: Select all

19.03.2024 07:27:49 libwebpmux3 0.6.1-2ubuntu0.18.04.2 1.3.2-0.3~18.04.sav0
19.03.2024 07:27:48 libwebpdemux2 0.6.1-2ubuntu0.18.04.2 1.3.2-0.3~18.04.sav0
19.03.2024 07:27:47 libwebp-dev 0.6.1-2ubuntu0.18.04.2 1.3.2-0.3~18.04.sav0
19.03.2024 07:26:26 palemoon 32.5.2-1.gtk3.bionic 33.0.1-1.gtk2.bionic
19.03.2024 07:16:52 mainline 1.4.9-0~202401190113~ubuntu18.04.1 1.4.10-0~202403090046~ubuntu18.04.1
19.03.2024 07:16:51 openssl 1.1.1w-0ubuntu1~18.04.sav1 1.1.1w-0ubuntu1~18.04.sav2
19.03.2024 07:16:50 libuv1 1.43.0-1~18.04.sav0 1.43.0-1ubuntu0.1~18.04.sav0
19.03.2024 07:16:49 libunbound8 1.16.2-1ubuntu0.1~18.04.sav0 1.17.1-2ubuntu0.1~18.04.sav0
19.03.2024 07:16:48 libsynctex2 2019.20190605.51237-3ubuntu0.1~18.04.sav0 2019.20190605.51237-3ubuntu0.2+18.04.sav0
19.03.2024 07:16:47 libneon27-gnutls 0.32.5-2~18.04.sav0 0.33.0-1~18.04.sav0
19.03.2024 07:16:46 libminizip1 1.1-8build1 1:1.3.1.dfsg-0ubuntu1~18.04.sav1
19.03.2024 07:16:45 liblua5.3-0 5.3.3-1ubuntu0.18.04.1 5.3.6-1~18.04.sav0
19.03.2024 07:16:44 libedit2 3.1-20210910-1~18.04.sav0 3.1-20230828-1~18.04.sav0
19.03.2024 07:16:42 libedit2:i386 3.1-20210910-1~18.04.sav0 3.1-20230828-1~18.04.sav0
19.03.2024 07:16:41 update-manager-core 1:18.04.11.13 1:18.04.11.23
19.03.2024 07:16:40 python3-update-manager 1:18.04.11.13 1:18.04.11.23
19.03.2024 07:16:33 ubuntu-advantage-tools 30~18.04 31.2~18.04
19.03.2024 07:16:31 libssl1.1:i386 1.1.1w-0ubuntu1~18.04.sav1 1.1.1w-0ubuntu1~18.04.sav2
19.03.2024 07:16:30 libssl1.1 1.1.1w-0ubuntu1~18.04.sav1 1.1.1w-0ubuntu1~18.04.sav2
19.03.2024 07:16:29 libssl-dev 1.1.1w-0ubuntu1~18.04.sav1 1.1.1w-0ubuntu1~18.04.sav2
19.03.2024 07:16:28 libsqlite3-0:i386 3.37.2-2ubuntu0.1~18.04.sav0 3.37.2-2ubuntu0.3~18.04.sav0
19.03.2024 07:16:26 libsqlite3-0 3.37.2-2ubuntu0.1~18.04.sav0 3.37.2-2ubuntu0.3~18.04.sav0
19.03.2024 07:16:25 sqlite3 3.37.2-2ubuntu0.1~18.04.sav0 3.37.2-2ubuntu0.3~18.04.sav0
19.03.2024 07:16:24 zlib1g:i386 1:1.2.13.dfsg-1ubuntu4~18.04.sav0 1:1.3.1.dfsg-0ubuntu1~18.04.sav1
19.03.2024 07:16:24 libsqlite3-dev 3.37.2-2ubuntu0.1~18.04.sav0 3.37.2-2ubuntu0.3~18.04.sav0
19.03.2024 07:16:22 zlib1g 1:1.2.13.dfsg-1ubuntu4~18.04.sav0 1:1.3.1.dfsg-0ubuntu1~18.04.sav1
19.03.2024 07:16:21 zlib1g-dev 1:1.2.13.dfsg-1ubuntu4~18.04.sav0 1:1.3.1.dfsg-0ubuntu1~18.04.sav1
19.03.2024 07:16:20 zlib1g-dev:i386 1:1.2.13.dfsg-1ubuntu4~18.04.sav0 1:1.3.1.dfsg-0ubuntu1~18.04.sav1
Between the 19th when those updates were installed followed by a reboot to a working system, and the 25th when the nemo-desktop segfault started to happen there were two minor installs mentioned in Synaptic's log:

Code: Select all

Commit Log for Sat Mar 23 12:20:52 2024

Installed the following packages:
lz4 (1.9.4-1~18.04.sav0)
===================================================================================================
Commit Log for Sat Mar 23 12:21:57 2024

Installed the following packages:
python3-lz4 (0.10.1+dfsg1-0.2)
Python module is an official package not backport or anything, and it isn't actually required. However the lz4 package is needed for unpacking the core dumps recently produced by Jargonaut as the default archiver couldn't unpack them (which is strange considering the system creates them in that format). Just uninstalled the latter (python3-lz4) and checked: nemo-desktop still crashes.

I have to mention that the Jargonaut crashes happen only when trying to import any module that have to do with web connections: urllib.request, requests, urllib3. And it doesn't always happen on script start - it may crash once or twice and then starts normally; but it always crashes on exit, most likely due to threading daemons as I've read around recently.

However, I have other scripts that import and use one or the other of those modules, and they all work fine without problems. So there has to be an incompatibility between those modules and some other module(s) imported in Jargonaut. That however doesn't explain the crash in nemo-desktop.

While typing here it just dawned on me to try debugging through gdb. Unfortunately it doesn't yield anything useful, or at least I can't see it:

Code: Select all

Reading symbols from nemo-desktop...Reading symbols from /usr/lib/debug/.build-id/3b/29115d42fc7a478f7a42d66e781159843e0a03.debug...done.
done.
[New LWP 12141]
[New LWP 12142]
[New LWP 12145]
[New LWP 12143]
[New LWP 12147]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nemo-desktop'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f891b447be4 in ?? () from /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
[Current thread is 1 (Thread 0x7f894282da80 (LWP 12141))]
What else could I do? Is there any other way or tool that could uncover the real cause of the crash? Because reverting all recent updates would not be possible without uninstalling half the system. I've been building this system for the last five years and will not give it up just like that.
User avatar
Drugwash
Level 5
Level 5
Posts: 734
Joined: Fri Jun 07, 2019 6:40 am
Location: Ploieşti, RO
Contact:

Re: Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by Drugwash »

Turns out it wasn't any fancy system library at fault. It was simply a Python3 module that had been installed as a dependency to sjcl which has no official package available (at least not for Mint 19).
That faulty module is pycryptodome.

But that's not all. After uninstalling that module and the associated dependency Crypto, and confirming nemo-desktop was loading correctly once again, I tried a bunch of older versions (latest one was 3.20) and I went back even up to the official version 3.4.7 as is being shipped with this Mint. None of the versions would allow nemo-desktop to run - all of them crashed with segmentation fault.

Installing the official python3-pycryptodome 3.4.7 had no negative impact upon nemo-desktophowever the sjcl module bailed out with the error AttributeError: module 'Crypto.Cipher.AES' has no attribute 'MODE_CCM'. This means I can't use sjcl which unfortunately is an important piece in the Jargonaut script, dealing with uploading/downloading to a LUFI storage server.

To me this is bad programming. At the very least a lack of compatibility checks and failsafes. And I dare say both on module's maintainer and on Mint's developers'. Am entire desktop to become unusable due to a random Python module is... huge. But who am I to judge...

Well, the LUFI idea probably went down the drain unless I can find something else to replace sjcl's functionality with.
Fortunately now Jargonaut stopped crashing upon exit.
And the Cinnamon desktop is back up and functional.

This is how one fixes a system. Not by "upgrading".
Thank you for reading.
User avatar
spamegg
Level 14
Level 14
Posts: 5118
Joined: Mon Oct 28, 2019 2:34 am
Contact:

Re: [SOLVED] Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by spamegg »

This is how one fixes a system. Not by "upgrading".
Thank you for reading.
To be fair though, there isn't much we can do without being in front of your PC and knowing its package history, what Python modules you installed / removed, etc.
That Nemo crash log could not indicate in any way that Pycryptodome was the problem. (Not blaming you, blaming the log itself)
And again to be fair, suggesting an upgrade actually is helpful and good advice. You really should upgrade. Mint 19 is not safe if you are connected to the Internet.

You are correct that the system is horribly made, so ridiculously fragile due to Python dependency. I've been trying to move people away by suggesting to never, ever, ever use pip or install / remove anything whatsoever Python-related (whether apt package, PPA or anything else), instead using PyEnv to isolate modules from the system. viewtopic.php?f=42&p=2103213
Next Debian / Ubuntu 24.04 will apparently take some measures https://www.linuxuprising.com/2023/03/n ... ikely.html. How effective will that be I don't know.
Best way to go for projects is probably Poetry https://python-poetry.org/ which also isolates all the dependencies for your project.
User avatar
Drugwash
Level 5
Level 5
Posts: 734
Joined: Fri Jun 07, 2019 6:40 am
Location: Ploieşti, RO
Contact:

Re: [SOLVED] Mint 19 Cinnamon. nemo-desktop segfault. Python problem.

Post by Drugwash »

spamegg wrote: Wed Mar 27, 2024 9:39 am To be fair though, there isn't much we can do without being in front of your PC [...]
The way I see it it's not as much knowing all details as knowing the principle on which debugging is based. One could start with generic questions like "have you installed/removed anything lately?", "can you perform this or that?", in order to gather a few more information. Some of the questions may even trigger the right idea.

It's true those particular logs didn't offer any solid clue as to where the problem lied. Subsequently I had even installed the debug version of Python 3.6 and still didn't offer much in addition. Had to change strategy: search for Python modules installed within a limited time frame, both locally and globally, and start uninstalling one by one while checking if the app was still crashing. That was much easier than anylyzing the mumbo-jumbo in the logs. Thankfully I had Double Commander with a nice assortment of plug-ins, that helped with searching and other operations, as Nemo the file browser was as dead as the desktop - them being closely linked together.

Suggesting an upgrade is borderline humiliating. If one wanted to upgrade they wouldn't have asked for help on a particular and very technical matter. People who have certain degree of technical knowledge and are willing to delve into an operating system's inner working would never resort to upgrading unless everything else fails. And even in that case they wouldn't ask others whether they should upgrade or not - they would just do it.

Just as an aside, all the changes I noticed in versions of Mint later than 19.2 were the removal of options and features; since I do need and want some of those features I willingly decided to stay with this version. It is my choice and my right as much as it's anyone else's to do what they want with their own hardware and software.

Safety is not a concern for someone who spent almost 20 years on the Internet and testing alpha-grade software on a Windows 98SE machine with no firewall and no resident anti-virus. And by the way, that system was (and still is) not a vanilla installation but heavily upgraded with libraries taken from various newer OS versions (2000/XP/Vista/7) as well as KernelEx and other enhancements. It has never been reinstalled all this time; the main folder dates back to 2006 when it was first installed on that drive. Compared to that this Mint (installed in 2019) which only sports a few minor enhancements such as Nemo 4.6.0, a few newer Python modules, newer self-compiled applications, a custom theme, and a few system file improvements (check out my repositories below in the signature) is much closer to a vanilla installation. I would've been ashamed not to be able to fix it after all the work that's gone into building it for these last years.

Now, isolating each and every piece of software from the global environment would be hilarious at best. Flatpak does a lousy job at that by delivering multi-gigabyte packages for applications that wouldn't take more than 10-20 megabytes at most. On that rate everybody should soon buy exabyte-grade drives in order to hold all their applications and the OS. It's not at all hilarious when programming is getting more and more sloppy such that no two system libraries or modules can work together in a global environment.

BTW, I have been testing a couple of my scripts in Debian 12.1 and indeed it growls at trying to install modules through regular pip commands. This can at times be helpful as the user could then look through Synaptic or in Terminal through apt for official .deb packages of the needed modules, and sometimes find some. But that's not always the case, and having multiple - possibly identical - versions of same modules scattered all around virtual environments wastes space to say the least. Unfortunately backward compatibility is almost an utopia in the Linux world. And that's precisely what lead to current situation.

Anyway, my machine is a 2009 model, a Samsung NP-R580 notebook upgraded to an i5 CPU and 8GB of DDR3 RAM. Video is a 512MB NVIDIA which has no proprietary driver available in newer Linux. I have a lot of external drives and none has more than 1% free space, most of the times less than 200MB. Internal drive usually has only 500-800MB free. With the amount of extra applications and that scarce space any newer OS version would fall to its knees begging for mercy. So I'd very much like to keep this machine as it is now for as long as it will still function because I'll never afford anything else.

Thank you for your input.
Post Reply

Return to “Software & Applications”