Troubleshooting screen freezes.

[glesialo]

My system ('inxi -F'):

Code: Select all

System:    Host: Linux-2 Kernel: 5.13.0-41-generic x86_64 bits: 64 Desktop: Cinnamon 5.2.7 Distro: Linux Mint 20.3 Una 
Machine:   Type: Desktop System: Micro-Star product: MS-7D08 v: 2.0 serial: N/A 
           Mobo: Micro-Star model: MAG Z590 TORPEDO (MS-7D08) v: 2.0 serial: <filter> UEFI: American Megatrends LLC. 
           v: A.50 date: 04/14/2022 
CPU:       Topology: 6-Core model: 11th Gen Intel Core i5-11600K bits: 64 type: MT MCP L2 cache: 12.0 MiB 
           Speed: 4600 MHz min/max: 800/4900 MHz Core speeds (MHz): 1: 4600 2: 4600 3: 4600 4: 4600 5: 4600 6: 4600 7: 4600 
           8: 4600 9: 4616 10: 4600 11: 4613 12: 4593 
Graphics:  Device-1: Intel driver: i915 v: kernel 
           Display: server: X.Org 1.20.13 driver: modesetting unloaded: fbdev,vesa resolution: 3840x2160~60Hz 
           OpenGL: renderer: Mesa Intel Graphics (RKL GT1) v: 4.6 Mesa 21.2.6 
Audio:     Device-1: Intel driver: snd_hda_intel 
           Sound Server: ALSA v: k5.13.0-41-generic 
Network:   Device-1: Intel Ethernet I219-V driver: N/A 
           Device-2: Intel driver: igc 
           IF: enp2s0 state: up speed: 100 Mbps duplex: full mac: <filter>
Drives:    Local Storage: total: 14.55 TiB used: 3.35 TiB (23.0%) 
           ID-1: /dev/sda vendor: Seagate model: ST8000VN004-2M2101 size: 7.28 TiB 
           ID-2: /dev/sdb vendor: Seagate model: ST8000VN004-2M2101 size: 7.28 TiB 
Partition: ID-1: / size: 36.42 GiB used: 16.82 GiB (46.2%) fs: ext4 dev: /dev/sda3 
           ID-2: /home size: 36.42 GiB used: 11.54 GiB (31.7%) fs: ext4 dev: /dev/sda4 
           ID-3: swap-1 size: 5.59 GiB used: 0 KiB (0.0%) fs: swap dev: /dev/sda2 
Sensors:   System Temperatures: cpu: 45.0 C mobo: N/A 
           Fan Speeds (RPM): N/A 
Info:      Processes: 354 Uptime: 47m Memory: 31.17 GiB used: 4.01 GiB (12.9%) Shell: bash inxi: 3.0.38

Issue:

• Screen's image freezes, music still plays on and I can move mouse's cursor but I can't do anything with it.

• Keyboard shorcuts do not work, can't switch to a Tty (Alt-Ctrl-F1), ...

• Alt-Ctrl-Backspace, to kill Cinnamon, doesn't work either (works when there is no freeze).

• The only way to exit from a freeze is to use 'Alt-PrtScr s' (sync) + 'Alt-PrtScr b' (reboot).

I have already reported the issue here: https://github.com/linuxmint/cinnamon/issues/10688.

I originally thought it was an Intel Graphics driver problem and closed the issue after removing the obsolete driver, 'xserver-xorg-video-intel' but I have had to reopen the issue because the bug is still there.

The last freeze has made me think that, perhaps, it isn't a graphics driver problem but a Cinnamon session one:
I was downloading a file, with 'firefox', when the freeze happened, and I could check that the file downloading froze at the same time.

To troubleshoot the bug I could start a general purpose daemon with each Cinnamon session and have it monitor the session (by running some 'cinnamon' command, checking processes, files...) continuously in the hope of detecting a 'freeze' and, once detected, perform measurements/force a switch to a TTY/restart the Cinnamon session or 'lightdm'...

Is there a way to interact with a running Cinnamon session?
What does 'cinnamon-session --whale' do?

Thanks in advance for your help!

[Cosmo.]

Two observations by reading your description (here and at Github):

1: Does reverting to kernel 5.13.0-28 help (momentarily)? To my understanding it should. Confirmed? To get there reboot the machine and hold the right shift-key immediately after the restart begins; alternatively you can rapidly press ESC as often as you can. This should open the grub menu. Here enter the submenu for additional options and select the old kernel.

Is kernel 5.13 really needed?

2. Regarding your usage of Alt-PrtScr, aka SysReq:
You seem to miss the U(nmount) between S(ync) and B(oot). But this should be done to prevent greater dramas.
To prevent that you get knots in your fingers do it this way:
Press and hold Alt-PrtScr and during that you press s, u and b one after the other. Make a little break between u and b to give the system the needed time for proper unmounting.

[glesialo]

@Cosmo

Thank you.

Does reverting to kernel 5.13.0-28 help (momentarily)?

Kernel 5.13.0-28 is no longer there. I wrote 'I am going back to kernel 5.13.0-28' three months ago and reverting to that kernel didn't solve the issue.

Is kernel 5.13 really needed?

Yes. I installed Cinnamon Edge because of my system hardware: Pretty recent MoBo and 11th generation Intel Core uP.

Regarding your usage of Alt-PrtScr, aka SysReq...

I know about REISUB. 'R' (take control of KB), 'E' (SIGTERM to all processes) & 'I' (SIGKILL to all processes) do not have any effect: No KB control and processes apparently chugging along. That's why I skipped all but those that seemed to work. I'll include 'U' from now on.

Are you familiar with Cinnamon's inner details? I would like to known how to monitor a running Cinnamon session so that my daemon can detect the next Freeze.

[glesialo]

While waiting for a Cinnamon expert that can tell me how to monitor a running session, I have setup the mentioned daemon to check (with 'pgrep'), every 5 seconds, if the following processes are running:

• cinnamon-session --session cinnamon

• cinnamon-launcher

• cinnamon --replace

• cinnamon-killer-daemon

If any of those processes is missing, the daemon will just send an EMail to the user.

EDIT: added another check: 'timeout 0.2 cinnamon-dbus-command GetMonitorWorkRect 0'

Conditions (checked every 5 seconds) that trigger 'CinnamonReport' action:

• CinnamonDBusIsBlocked

• CinnamonKillerDaemonProcessMissing

• CinnamonLauncherProcessMissing

• CinnamonReplaceProcessMissing

• CinnamonSessionProcessMissing

[lofi]

I know about REISUB. 'R' (take control of KB), 'E' (SIGTERM to all processes) & 'I' (SIGKILL to all processes) do not have any effect: No KB control and processes apparently chugging along. That's why I skipped all but those that seemed to work. I'll include 'U' from now on.

Even during a screen freeze, there's not reason why some direct "magical" syscalls would work and not others. What's more likely is that some keys are allowed in your preferences (sync, reboot), but not unmount. viewtopic.php?t=206552

[glesialo]

@lofi
Thank you very much!

That's very useful to know. I thought all the REISUB key-codes were enabled by default.

In '/etc/sysctl.d/10-magic-sysrq.conf', 'kernel.sysrq' is set to 176, which I guess means:

• enable sync command

• enable remount read-only

• allow reboot/poweroff

I'll set it to 180 to 'enable control of keyboard' too.

[lofi]

You're welcome!

In '/etc/sysctl.d/10-magic-sysrq.conf', 'kernel.sysrq' is set to 176, which I guess means:

• enable sync command

• enable remount read-only

• allow reboot/poweroff

Yes.

[glesialo]

Yesterday, at 20:53, there was another freeze and the action script ran. Please check the following files:

• The action script report: 'CinnamonReport__Action_triggered.txt'. https://pastebin.com/Ht1cH4gg

• Frozen GUI session '.xsession-errors.old'. https://pastebin.com/NihdGT6c

• Relevant lines (20:50 .. next boot) of 'syslog'. https://pastebin.com/w7yk1eBs

• GUI session user's log '2022_06_01.log' (in my system every user has his own log). https://pastebin.com/eSH5r6A7

Notes:

• The action script was triggered when 'timeout 4 cinnamon-dbus-command GetRunState' failed.

• 'ExitCode: 124' in 'CinnamonReport__Action_triggered.txt' means timeout for 'timeout 1 cinnamon-dbus-command GetRunState' and that there was no output.

• I waited until 20:57 before rebooting because, if the action script was triggered, there would be a new local mail and that would generate a notification GUI dialog at 20:55. I could not see the dialog on the screen but, after rebooting, I could check that the dialog was indeed running 1 second after it was launched: See '2022/Jun/01 20:55:01' lines in '2022_06_01.log'. The GUI dialog is a program I wrote in Java and it seems it doesn't notice that there is anything wrong with the X server.

• I could not get control of the keyboard: Check 'sysrq: Keyboard mode set to system default' & 'sysrq: This sysrq operation is disabled' lines in 'syslog'.

• I would really like if someone knowledgeable in 'cinnamon' could tell me what to 'ask' (with the action script) 'cinnamon' during a freeze.

[lofi]

Did you find the cause for your freezes ?
I am not the cinnamon expert you were waiting for, not at all. All I know is that cinnamon originally is a fork from gnome-shell. Many gnome-shell commands and flags still work (-whale, that you mentioned, probably comes from it, that's where you'll find documentation).

I only looked at your first pastebin. I saw that kswapd is not a suspect (0% CPU). But still, I want to mention it, as for me, freezes on Mint cinnamon are history since I discovered the legendary linux kernel bug 12309 (cf. bugzilla, hidden). I'm on a kernel 4.xx, you're on 5.xx, but the legend says this bug is forever. Symptom is 99% CPU for kswapd0, during heavy I/O use (you mentioned downloading a file with Firefox), like when running out of memory. Causes very hard freezes where either you wait at least one hour (even with top-notch hardware (it's actually worse when you have an SSD)), or you hard reset. Facebook wrote their own fix for their servers. Sorry if unrelated. But I wish there was a pinned thread somewhere about this.