Linux Mint 19.2 Session gets OOM-killed overnight

[sebbo]

Update: My problem was solved, Go straight to my solution: viewtopic.php?f=147&t=301532&p=1688008#p1688008


Good morning,

since the upgrade to Mint 19.2 XFCE this is really bothering me.

When i leave my Mint running in a virtualbox over night (Host is also a Linux Mint), the session gets killed somehow.

That means when i log in, i see an empty desktop and all my running apps are gone.

When i run dmesg i see the following:

Code: Select all

[53069.217854] [25530]     0 25530    25228      311   233472        0             0 cupsd
[53069.217855] [25531]     0 25531    75881      380   344064        0             0 cups-browsed
[53069.217857] Out of memory: Kill process 1290 (Xorg) score 595 or sacrifice child
[53069.217872] Killed process 1290 (Xorg) total-vm:4818228kB, anon-rss:4167676kB, file-rss:0kB, shmem-rss:43308kB
[53069.589804] oom_reaper: reaped process 1290 (Xorg), now anon-rss:0kB, file-rss:0kB, shmem-rss:20824kB
[53070.046109] vgdrvLinuxIOCtlSlow: copy_to_user(0x7ffc772f6e90,,0x48); uCmd=0xc0485607!
[53071.381084] vgdrvLinuxIOCtlSlow: copy_to_user(0x7f755c179b20,,0x58); uCmd=0xc0585607!

I have already deactivated the OOM Killer, following this post https://ubuntuforums.org/showthread.php?t=2197016 that basically says to run

Code: Select all

sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf 

However... I undid this change pretty fast, as i had the feeling that this will simply make applications freeze and crash.

I can't be the only one experiencing something like this, am I?

I haven't found anything useful.. anyone has an idea how to tackle this?

THanks

Code: Select all

System:    Host: sebastian-mint192 Kernel: 4.15.0-60-generic x86_64 bits: 64 compiler: gcc v: 7.4.0 Desktop: Xfce 4.12.3 
           Distro: Linux Mint 19.2 Tina base: Ubuntu 18.04 bionic 
Machine:   Type: Virtualbox System: innotek product: VirtualBox v: 1.2 serial: <filter> 
           Mobo: Oracle model: VirtualBox v: 1.2 serial: <filter> BIOS: innotek v: VirtualBox date: 12/01/2006 
Battery:   ID-1: BAT0 charge: 50.0 Wh condition: 50.0/50.0 Wh (100%) model: innotek 1 status: Full 
CPU:       Topology: Dual Core model: Intel Core i5-7300U bits: 64 type: MCP arch: Kaby Lake rev: 9 L2 cache: 3072 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 bogomips: 10847 
           Speed: 2712 MHz min/max: N/A Core speeds (MHz): 1: 2712 2: 2712 
Graphics:  Device-1: VMware SVGA II Adapter driver: vmwgfx v: 2.14.0.0 bus ID: 00:02.0 
           Display: x11 server: X.Org 1.19.6 driver: vmware unloaded: fbdev,modesetting,vesa resolution: 1736x886~60Hz 
           OpenGL: renderer: llvmpipe (LLVM 8.0 256 bits) v: 3.3 Mesa 19.0.8 direct render: Yes 
Audio:     Device-1: Intel 82801AA AC97 Audio vendor: Dell driver: snd_intel8x0 v: kernel bus ID: 00:05.0 
           Sound Server: ALSA v: k4.15.0-60-generic 
Network:   Device-1: Intel 82540EM Gigabit Ethernet driver: e1000 v: 7.3.21-k8-NAPI port: d020 bus ID: 00:03.0 
           IF: enp0s3 state: up speed: 1000 Mbps duplex: full mac: <filter> 
           Device-2: Intel 82371AB/EB/MB PIIX4 ACPI type: network bridge driver: piix4_smbus v: N/A port: d200 bus ID: 00:07.0 
           Device-3: Intel 82540EM Gigabit Ethernet driver: e1000 v: 7.3.21-k8-NAPI port: d240 bus ID: 00:08.0 
           IF: enp0s8 state: up speed: 1000 Mbps duplex: full mac: <filter> 
Drives:    Local Storage: total: 40.43 GiB used: 24.52 GiB (60.7%) 
           ID-1: /dev/sda vendor: VirtualBox model: VBOX HARDDISK size: 40.43 GiB 
Partition: ID-1: / size: 37.86 GiB used: 24.16 GiB (63.8%) fs: ext4 dev: /dev/dm-1 
           ID-2: /boot size: 703.5 MiB used: 291.8 MiB (41.5%) fs: ext4 dev: /dev/sda1 
           ID-3: swap-1 size: 976.0 MiB used: 77.7 MiB (8.0%) fs: swap dev: /dev/dm-2 
Sensors:   Message: No sensors data was found. Is sensors configured? 
Info:      Processes: 202 Uptime: 5d 14h 25m Memory: 5.82 GiB used: 1.31 GiB (22.6%) Init: systemd runlevel: 5 Compilers: 
           gcc: 7.4.0 clang: 6.0.0-1ubuntu2 Shell: bash v: 4.4.20 inxi: 3.0.32 

[gm10]

You need to find the process leaking memory and forcing all others to get killed to make room for it. Probably also in the logs, otherwise just open up System Monitor and see which one keeps growing.

[sebbo]

Thanks.

Okay. I did some more research and i saw a couple of "Web Content" Processes that were in the list of processes that are dumped into whatever log dmesg ist showing.

I attached some more dmesg output.

I googled what this "Web Content" processes are about and I read it is due to having lots of tabs openend. Well yes, I'm a tab-hoarder, I admit it.

I will keep firefox closed over night and see if that helps. If yes, i can debug further into this direction.

Edit: 3Kb above max file size
https://pastebin.com/8bMBhZmQ

[ColdBootII]

Hi,

I'd suggest that you try the same use case with Chromium and "The Great Suspender" extension installed as I've read that addons of similar type for Firefox, are not very efficient.

[sebbo]

So.

I quit firefox before leaving the system overnight.

I came back and still OOM Killer was active last night

I still have my old Mint 19.1 VM and compared settings. I noticed some difference in GraphicsController - no idea whether this is the reason. I had n't had this problem with the old VM, so i'll check if that changes something.

[Spearmint2]

I see swap entries mentioned, and wondering if that might be part of the problem? Try it with swap turned off before suspending, then open immediately again to check any effect and if all is well, suspend and see if the same problem continues.

Code: Select all

swapoff -a

You can on waking, turn swap back on.

Code: Select all

swapon -a

Might also run a memtest option during boot.

[gm10]

Edit: 3Kb above max file size
https://pastebin.com/8bMBhZmQ

Ok, according to that you ran out of swap space and your X-server was taking up nearly a GB of RAM and got killed. No, that's not a normal amount of RAM for Xorg.

I quit firefox before leaving the system overnight.

I came back and still OOM Killer was active last night

I still have my old Mint 19.1 VM and compared settings. I noticed some difference in GraphicsController - no idea whether this is the reason. I had n't had this problem with the old VM, so i'll check if that changes something.

Did you check the logs, did Xorg get killed again using up a lot of RAM? Also that Firefox and in particular its WebExtensions were really not running?

It is quite possibly a problem with the graphics driver as you mentioned, but it could also be another process making improper usage of X. You could keep monitoring Xorg, and if you see the memory grow, check its memory map it see what makes it allocate all that memory:

Code: Select all

sudo pmap -x $(pgrep Xorg)|less

[sebbo]

Yes, it was again XOrg that was killed.

I will check back tomorrow and see what "evidence" i can catch after logging in.

The VM is actually not suspended. It is just the Screen that is locked.

I leave the host (laptop) running over night and it also just locks the screen.

Thanks for your help! Really appreciated!

[Spearmint2]

Well, there's your problem!!! All kind of process are just going to keep doing their thing in the background.

[gm10]

By the way, completely forgot to mention, I saw a lot of ssh processes in that log, is that normal and expected while you're not even using the system? I'd look at those, first, in particular if each of them connects to your X server there's a very good chance they are the culprits.

[Spearmint2]

By the way, completely forgot to mention, I saw a lot of ssh processes in that log, is that normal and expected while you're not even using the system? I'd look at those, first, in particular if each of them connects to your X server there's a very good chance they are the culprits.

There's an idea. He might be leaving the wifi (& bluetooth) on, and it's set to connect to any open signal, and even if not, it will keep looking for the connect it uses. That can drain a battery.

[sebbo]

Update.

Just came into the office and the XOrg session was NOT killed this time.

I closed Firefox and Thunderbird over night and as i wrote yesterday, i had changed the graphics driver setting in the VirtualBox setting to match the settings of my OldVM that didn't have this problem (but was a mint 19.1 machine).

So it's either FF+TB or the graphics card. I will leave firefox running over the weekend and see what happens.

@Spearmint2
Sure, leaving Applications running will make them do their work... but it had worked fine in the past, so I think it should work fine now.

@gm10
regarding the SSH sessions:
yes, i use ssh extensively. I keep running outgoing ssh sessions and forget to close them.
Also i have autossh running, so that the VM will always hold open a connection to a server that i can use to reverse-ssh into my Virtual Machine at work, if i need it. (It's better than port forwarding imho...)
I'm using incoming X11 on ssh very rarely. I'm doing X11vnc outgoing to another machine almost daily though, but i usually close this connection.

@Spearmint2
To be honest I do not see any connection between running out of memory and auto connecting to BT/Wifi that will drain the battery?
The Host (a laptop) is connected via ethernet most of the time, and even if it is connected to wifi, it is connected to power all the time.

I'll report back latest on monday!

Thanks for all of your help and suggestions!

[sebbo]

So... Monday... back to work

I kept Firefox open over the weekend (+ Gajim). Apart from a Gajim Error message, everything was as i left it on Friday.

I guess it was the Graphic-Card Setting in VirtualBox. I will monitor this for a few days and then update my initial post with this information as the likely solution.

Thanks again to everyone that helped so far!

[sebbo]

Thursday now...

I have gone back to my behaviour of keeping everything open when i leave office.

Luckily, the OOM Killer has not come back since i changed this parameter in the VirtualBox Settings for this Virtual Machine.

I think it is safe to assume that this was the reason for this - if not i'll be back

Graphics Controller must be set to VBoxVGA

I have no idea if that is (not) the default. I think before i had the setting on VMSVGA (i don't know why and how, i usually do not touch this...)

[gm10]

Thanks for the info. VMSVGA is the default for Linux hosts. Hadn't heard about issues with it but as long as the switch works for you that's all that matters. Note that the VBoxVGA driver you've got now is deprecated and will eventually be removed though.