Linux Mint 19.2 Session gets OOM-killed overnight

Forum rules
Before you post please read how to get help
Post Reply
sebbo
Level 2
Level 2
Posts: 54
Joined: Wed Mar 18, 2015 5:00 am
Location: DE

Linux Mint 19.2 Session gets OOM-killed overnight

Post by sebbo » Wed Sep 11, 2019 1:08 am

Good morning,

since the upgrade to Mint 19.2 XFCE this is really bothering me.

When i leave my Mint running in a virtualbox over night (Host is also a Linux Mint), the session gets killed somehow.

That means when i log in, i see an empty desktop and all my running apps are gone.

When i run dmesg i see the following:

Code: Select all

[53069.217854] [25530]     0 25530    25228      311   233472        0             0 cupsd
[53069.217855] [25531]     0 25531    75881      380   344064        0             0 cups-browsed
[53069.217857] Out of memory: Kill process 1290 (Xorg) score 595 or sacrifice child
[53069.217872] Killed process 1290 (Xorg) total-vm:4818228kB, anon-rss:4167676kB, file-rss:0kB, shmem-rss:43308kB
[53069.589804] oom_reaper: reaped process 1290 (Xorg), now anon-rss:0kB, file-rss:0kB, shmem-rss:20824kB
[53070.046109] vgdrvLinuxIOCtlSlow: copy_to_user(0x7ffc772f6e90,,0x48); uCmd=0xc0485607!
[53071.381084] vgdrvLinuxIOCtlSlow: copy_to_user(0x7f755c179b20,,0x58); uCmd=0xc0585607!
I have already deactivated the OOM Killer, following this post https://ubuntuforums.org/showthread.php?t=2197016 that basically says to run

Code: Select all

sysctl vm.overcommit_memory=2
echo "vm.overcommit_memory=2" >> /etc/sysctl.conf 
However... I undid this change pretty fast, as i had the feeling that this will simply make applications freeze and crash.

I can't be the only one experiencing something like this, am I?

I haven't found anything useful.. anyone has an idea how to tackle this?

THanks

Code: Select all

System:    Host: sebastian-mint192 Kernel: 4.15.0-60-generic x86_64 bits: 64 compiler: gcc v: 7.4.0 Desktop: Xfce 4.12.3 
           Distro: Linux Mint 19.2 Tina base: Ubuntu 18.04 bionic 
Machine:   Type: Virtualbox System: innotek product: VirtualBox v: 1.2 serial: <filter> 
           Mobo: Oracle model: VirtualBox v: 1.2 serial: <filter> BIOS: innotek v: VirtualBox date: 12/01/2006 
Battery:   ID-1: BAT0 charge: 50.0 Wh condition: 50.0/50.0 Wh (100%) model: innotek 1 status: Full 
CPU:       Topology: Dual Core model: Intel Core i5-7300U bits: 64 type: MCP arch: Kaby Lake rev: 9 L2 cache: 3072 KiB 
           flags: lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 bogomips: 10847 
           Speed: 2712 MHz min/max: N/A Core speeds (MHz): 1: 2712 2: 2712 
Graphics:  Device-1: VMware SVGA II Adapter driver: vmwgfx v: 2.14.0.0 bus ID: 00:02.0 
           Display: x11 server: X.Org 1.19.6 driver: vmware unloaded: fbdev,modesetting,vesa resolution: 1736x886~60Hz 
           OpenGL: renderer: llvmpipe (LLVM 8.0 256 bits) v: 3.3 Mesa 19.0.8 direct render: Yes 
Audio:     Device-1: Intel 82801AA AC97 Audio vendor: Dell driver: snd_intel8x0 v: kernel bus ID: 00:05.0 
           Sound Server: ALSA v: k4.15.0-60-generic 
Network:   Device-1: Intel 82540EM Gigabit Ethernet driver: e1000 v: 7.3.21-k8-NAPI port: d020 bus ID: 00:03.0 
           IF: enp0s3 state: up speed: 1000 Mbps duplex: full mac: <filter> 
           Device-2: Intel 82371AB/EB/MB PIIX4 ACPI type: network bridge driver: piix4_smbus v: N/A port: d200 bus ID: 00:07.0 
           Device-3: Intel 82540EM Gigabit Ethernet driver: e1000 v: 7.3.21-k8-NAPI port: d240 bus ID: 00:08.0 
           IF: enp0s8 state: up speed: 1000 Mbps duplex: full mac: <filter> 
Drives:    Local Storage: total: 40.43 GiB used: 24.52 GiB (60.7%) 
           ID-1: /dev/sda vendor: VirtualBox model: VBOX HARDDISK size: 40.43 GiB 
Partition: ID-1: / size: 37.86 GiB used: 24.16 GiB (63.8%) fs: ext4 dev: /dev/dm-1 
           ID-2: /boot size: 703.5 MiB used: 291.8 MiB (41.5%) fs: ext4 dev: /dev/sda1 
           ID-3: swap-1 size: 976.0 MiB used: 77.7 MiB (8.0%) fs: swap dev: /dev/dm-2 
Sensors:   Message: No sensors data was found. Is sensors configured? 
Info:      Processes: 202 Uptime: 5d 14h 25m Memory: 5.82 GiB used: 1.31 GiB (22.6%) Init: systemd runlevel: 5 Compilers: 
           gcc: 7.4.0 clang: 6.0.0-1ubuntu2 Shell: bash v: 4.4.20 inxi: 3.0.32 


gm10
Level 18
Level 18
Posts: 8692
Joined: Thu Jun 21, 2018 5:11 pm

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by gm10 » Wed Sep 11, 2019 2:43 am

You need to find the process leaking memory and forcing all others to get killed to make room for it. Probably also in the logs, otherwise just open up System Monitor and see which one keeps growing.
Tune up your LM 19.x: ppa:gm10/linuxmint-tools

sebbo
Level 2
Level 2
Posts: 54
Joined: Wed Mar 18, 2015 5:00 am
Location: DE

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by sebbo » Wed Sep 11, 2019 3:29 am

Thanks.

Okay. I did some more research and i saw a couple of "Web Content" Processes that were in the list of processes that are dumped into whatever log dmesg ist showing.

I attached some more dmesg output.

I googled what this "Web Content" processes are about and I read it is due to having lots of tabs openend. Well yes, I'm a tab-hoarder, I admit it.

I will keep firefox closed over night and see if that helps. If yes, i can debug further into this direction.

Edit: 3Kb above max file size :roll:
https://pastebin.com/8bMBhZmQ

ColdBootII
Level 4
Level 4
Posts: 434
Joined: Wed Aug 02, 2017 8:19 am

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by ColdBootII » Wed Sep 11, 2019 6:24 am

Hi,

I'd suggest that you try the same use case with Chromium and "The Great Suspender" extension installed as I've read that addons of similar type for Firefox, are not very efficient.

sebbo
Level 2
Level 2
Posts: 54
Joined: Wed Mar 18, 2015 5:00 am
Location: DE

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by sebbo » Thu Sep 12, 2019 1:55 am

So.

I quit firefox before leaving the system overnight.

I came back and still OOM Killer was active last night :(

I still have my old Mint 19.1 VM and compared settings. I noticed some difference in GraphicsController - no idea whether this is the reason. I had n't had this problem with the old VM, so i'll check if that changes something.

User avatar
Spearmint2
Level 16
Level 16
Posts: 6116
Joined: Sat May 04, 2013 1:41 pm
Location: Maryland, USA

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by Spearmint2 » Thu Sep 12, 2019 2:57 am

I see swap entries mentioned, and wondering if that might be part of the problem? Try it with swap turned off before suspending, then open immediately again to check any effect and if all is well, suspend and see if the same problem continues.

Code: Select all

swapoff -a
You can on waking, turn swap back on.

Code: Select all

swapon -a
Might also run a memtest option during boot.
All things go better with Mint. Mint julep, mint jelly, mint gum, candy mints, pillow mints, peppermint, chocolate mints, spearmint,....

gm10
Level 18
Level 18
Posts: 8692
Joined: Thu Jun 21, 2018 5:11 pm

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by gm10 » Thu Sep 12, 2019 4:00 am

sebbo wrote:
Wed Sep 11, 2019 3:29 am
Edit: 3Kb above max file size :roll:
https://pastebin.com/8bMBhZmQ
Ok, according to that you ran out of swap space and your X-server was taking up nearly a GB of RAM and got killed. No, that's not a normal amount of RAM for Xorg.
sebbo wrote:
Thu Sep 12, 2019 1:55 am
I quit firefox before leaving the system overnight.

I came back and still OOM Killer was active last night :(

I still have my old Mint 19.1 VM and compared settings. I noticed some difference in GraphicsController - no idea whether this is the reason. I had n't had this problem with the old VM, so i'll check if that changes something.
Did you check the logs, did Xorg get killed again using up a lot of RAM? Also that Firefox and in particular its WebExtensions were really not running?

It is quite possibly a problem with the graphics driver as you mentioned, but it could also be another process making improper usage of X. You could keep monitoring Xorg, and if you see the memory grow, check its memory map it see what makes it allocate all that memory:

Code: Select all

sudo pmap -x $(pgrep Xorg)|less
Tune up your LM 19.x: ppa:gm10/linuxmint-tools

sebbo
Level 2
Level 2
Posts: 54
Joined: Wed Mar 18, 2015 5:00 am
Location: DE

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by sebbo » Thu Sep 12, 2019 5:23 am

Yes, it was again XOrg that was killed.

I will check back tomorrow and see what "evidence" i can catch after logging in.

The VM is actually not suspended. It is just the Screen that is locked.

I leave the host (laptop) running over night and it also just locks the screen.

Thanks for your help! Really appreciated! :D

User avatar
Spearmint2
Level 16
Level 16
Posts: 6116
Joined: Sat May 04, 2013 1:41 pm
Location: Maryland, USA

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by Spearmint2 » Thu Sep 12, 2019 5:30 am

Well, there's your problem!!! All kind of process are just going to keep doing their thing in the background.
All things go better with Mint. Mint julep, mint jelly, mint gum, candy mints, pillow mints, peppermint, chocolate mints, spearmint,....

gm10
Level 18
Level 18
Posts: 8692
Joined: Thu Jun 21, 2018 5:11 pm

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by gm10 » Thu Sep 12, 2019 5:36 am

By the way, completely forgot to mention, I saw a lot of ssh processes in that log, is that normal and expected while you're not even using the system? I'd look at those, first, in particular if each of them connects to your X server there's a very good chance they are the culprits.
Tune up your LM 19.x: ppa:gm10/linuxmint-tools

User avatar
Spearmint2
Level 16
Level 16
Posts: 6116
Joined: Sat May 04, 2013 1:41 pm
Location: Maryland, USA

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by Spearmint2 » Thu Sep 12, 2019 5:45 am

gm10 wrote:
Thu Sep 12, 2019 5:36 am
By the way, completely forgot to mention, I saw a lot of ssh processes in that log, is that normal and expected while you're not even using the system? I'd look at those, first, in particular if each of them connects to your X server there's a very good chance they are the culprits.
There's an idea. He might be leaving the wifi (& bluetooth) on, and it's set to connect to any open signal, and even if not, it will keep looking for the connect it uses. That can drain a battery.
All things go better with Mint. Mint julep, mint jelly, mint gum, candy mints, pillow mints, peppermint, chocolate mints, spearmint,....

sebbo
Level 2
Level 2
Posts: 54
Joined: Wed Mar 18, 2015 5:00 am
Location: DE

Re: Linux Mint 19.2 Session gets OOM-killed overnight

Post by sebbo » Fri Sep 13, 2019 1:07 am

Update.

Just came into the office and the XOrg session was NOT killed this time.

I closed Firefox and Thunderbird over night and as i wrote yesterday, i had changed the graphics driver setting in the VirtualBox setting to match the settings of my OldVM that didn't have this problem (but was a mint 19.1 machine).

So it's either FF+TB or the graphics card. I will leave firefox running over the weekend and see what happens.

@Spearmint2
Sure, leaving Applications running will make them do their work... but it had worked fine in the past, so I think it should work fine now.

@gm10
regarding the SSH sessions:
yes, i use ssh extensively. I keep running outgoing ssh sessions and forget to close them.
Also i have autossh running, so that the VM will always hold open a connection to a server that i can use to reverse-ssh into my Virtual Machine at work, if i need it. (It's better than port forwarding imho...)
I'm using incoming X11 on ssh very rarely. I'm doing X11vnc outgoing to another machine almost daily though, but i usually close this connection.

@Spearmint2
To be honest I do not see any connection between running out of memory and auto connecting to BT/Wifi that will drain the battery?
The Host (a laptop) is connected via ethernet most of the time, and even if it is connected to wifi, it is connected to power all the time.

I'll report back latest on monday!

Thanks for all of your help and suggestions!

Post Reply

Return to “Others”