HOW-TO: get a speed gain by using zswap

[smurphos]

The solution is to develop a special kernel for desktop use to handle memory and swap in another way that are more desktop friendly. and that will not happen as Linux desktop is just a little tiny niche of users and it's not worth the time to develop two different kernels.

The solution is already there - it's called zram and does away with storage based swap entirely. Instead the system compresses RAM to a RAM based virtual swap - if that then fills beyond the parameters set the OOM killer kicks in. I know Pjotr doesn't recommend it based on some testing experience a good few years ago, but I've used it for a couple of years now trouble free on both memory poor and memory OK hardware - it's the only thing that makes the memory poor hardware usable, e.g I can run Manjaro Cinnamon on an old 2GB RAM chromebook with reasonable performance, no complete freezes albeit it can lag a little bit whilst the (also underpowered) CPU is busy compressing/decompressing RAM. My main 8GB RAM laptop just works, always.

It is now the default out of the box set up for some distros, most notably Fedora. ChromeOS (which is just a modded Gentoo) has always used it.

[Menard]

I want to backup Pjotr post and statement.

System partitions should never be filled not even in windows..
Data storage partitions those don't matter in the same way, as it only effect reed/write times to/from/between those drives.
a filled system disk will effect the system performance in more ways then one. That goes for both Linux and Windows.

'
Edit.
The reason you Menard and all else that experiences this to get a total freeze when your memory is used to 99 to 100% is called swapdeath or I/O trashing.
This is a kind of "bug" in the linux kernel that was first discovered in 1997 or if it was 1998. They never solved that problem as it only effect low end computers. Servers has a lot more memory and faster drives to handle this phenomenon.
So if you start your terminal and monitor your disk activity when you reaches 99/100% memory and starts sudo iotop -d 2 -P then you will see how the swap I/O is on 99% usage even that your cpu is idle. This is what causes the system to freeze as disk I/O has maximum priority in the system.
If you temporary disable swap and your system reach 100% the system will not freeze up as the OOM killer will kick in and kill that program that is the cause. With swap enabled OOM killer will not kick in as the swap is consider as more usable memory.

To solve this problem the linux kernel developers just using the word... add more RAM as it is cheap.
Linux kernel is primary a server system and a server system does not use RAM in the same way as a desktop does.
The solution is to develop a special kernel for desktop use to handle memory and swap in another way that are more desktop friendly. and that will not happen as Linux desktop is just a little tiny niche of users and it's not worth the time to develop two different kernels.

OK, thanks a lot

Yes but it is not really a problem for me because as I wrote, it was in a special stress test but normally with that alarm script I use, and the zswap, it should not happen again
But is there something to do when it happens ? I wonder if instead of trying ctrl+alt+del I ought to try to start a terminal and xkill and put the x on the application that do this, I think I have some seconds before the total freeze, often it is enough to close Chrome normally with a 2 to 54 minutes delay

NB : I remember that with Win XP it was quite the same behaviour if without Swap and also if RAM at 100% it ended with a system shutdown

[Marie SWE]

The solution is to develop a special kernel for desktop use to handle memory and swap in another way that are more desktop friendly. and that will not happen as Linux desktop is just a little tiny niche of users and it's not worth the time to develop two different kernels.

The solution is already there - it's called zram and does away with storage based swap entirely. Instead the system compresses RAM to a RAM based virtual swap - if that then fills beyond the parameters set the OOM killer kicks in. I know Pjotr doesn't recommend it based on some testing experience a good few years ago, but I've used it for a couple of years now trouble free on both memory poor and memory OK hardware - it's the only thing that makes the memory poor hardware usable, e.g I can run Manjaro Cinnamon on an old 2GB RAM chromebook with reasonable performance, no complete freezes albeit it can lag a little bit whilst the (also underpowered) CPU is busy compressing/decompressing RAM. My main 8GB RAM laptop just works, always.

It is now the default out of the box set up for some distros, most notably Fedora. ChromeOS (which is just a modded Gentoo) has always used it.

zram is good I'm with you on that one. but sadly is mostly a new type of bandage on an open wound. (i will explain my expression why.)

I have made an pretty extensive research on the subject. I started do some research in 2019 and I post some of it on a swedish computer forums in 2020 to get some feedback of my thoughts and i named it Swap Of Death as it sounded a bit more then just swapdeath as Linus torvald and they called it. (so if you can handle swedish I can give you the link. It's to much to translate to english as it's a thread like this.. and i don't think google translate can do the job, as i and other sweds use the swedish language a bit loosely on forums. )
my research has been a little on and off under this three years as it is just a free time thing. I wish i had more time to it and one or two programmer that could handle kernel programming i could collaborate with as i do have some ideas that would be interesting to test.
The big core problem is how the kernel handles memory and what and and what not to swap out. Some of the freeze problem with a big swap data is because the operating system starts to swap out some of the desktop environment. On a server a desktop environment isn't a ting as all linux user know after a year or two on linux desktop. This is one of the things why windows is superior then linux on handling memory overdrive with a big swap data file. Windows by design never swaps out the desktop environment, that's why you can notice windows to be slow when overloaded but it's never totally freezes up like linux does. Then we have the linux disk I/O trashing thing, and that is a thing of it's own but mostly happens with swap, but you can get it in other situations too. when you get a disk I/O overload on 99% but the bittransfer isn't anywhere near the disk max reed/write speed. you can have 99% I/O on so low data transfer as 800KB/s (HDD)

This will become a five page post if i go in to everything.
I still do the research a little now and then when i have time and want to spend time at my computer. I have also worked at some bandages to the open wound, but i don't have programing skills, that is my biggest problem. So some swedish programmers would be nice to get to know for this subject as my english isn't the best... as when discussing deep things like this, it is a tiny-tiny bit of advantage if you can speak relatively quickly and freely without the need for a dictionary for certain words.

So if you smurphos is in to this thing perhaps we can collaborate on it now and then?
It would be nice to be the ones that solves the 90's bug.

[Marie SWE]

I want to backup Pjotr post and statement.

System partitions should never be filled not even in windows..
Data storage partitions those don't matter in the same way, as it only effect reed/write times to/from/between those drives.
a filled system disk will effect the system performance in more ways then one. That goes for both Linux and Windows.

'
Edit.
The reason you Menard and all else that experiences this to get a total freeze when your memory is used to 99 to 100% is called swapdeath or I/O trashing.
This is a kind of "bug" in the linux kernel that was first discovered in 1997 or if it was 1998. They never solved that problem as it only effect low end computers. Servers has a lot more memory and faster drives to handle this phenomenon.
So if you start your terminal and monitor your disk activity when you reaches 99/100% memory and starts sudo iotop -d 2 -P then you will see how the swap I/O is on 99% usage even that your cpu is idle. This is what causes the system to freeze as disk I/O has maximum priority in the system.
If you temporary disable swap and your system reach 100% the system will not freeze up as the OOM killer will kick in and kill that program that is the cause. With swap enabled OOM killer will not kick in as the swap is consider as more usable memory.

To solve this problem the linux kernel developers just using the word... add more RAM as it is cheap.
Linux kernel is primary a server system and a server system does not use RAM in the same way as a desktop does.
The solution is to develop a special kernel for desktop use to handle memory and swap in another way that are more desktop friendly. and that will not happen as Linux desktop is just a little tiny niche of users and it's not worth the time to develop two different kernels.

OK, thanks a lot

Yes but it is not really a problem for me because as I wrote, it was in a special stress test but normally with that alarm script I use, and the zswap, it should not happen again
But is there something to do when it happens ? I wonder if instead of trying ctrl+alt+del I ought to try to start a terminal and xkill and put the x on the application that do this, I think I have some seconds before the total freeze, often it is enough to close Chrome normally with a 2 to 54 minutes delay

NB : I remember that with Win XP it was quite the same behaviour if without Swap and also if RAM at 100% it ended with a system shutdown

I have worked on a few things to prevent the computer from freezing with some success. The problem is that I have not thoroughly tested the consequences for how it affects the rest of the system, as it is about changing I/O priorities and other kernel parameters and so on.
So i will not post this, as i don't want anyone to do this things on there daily driver machines and then they get big problems or system crashes. It's one thing to test in a lab environment and an other thing if those ho just copy/paste from forums do it on there daily driver systems in hope it solves there problem

[Marie SWE]

I want to add to my long post
This phenomenon swapdeath is an low end computer problem with low RAM and HDD spinndisks
As in the 90's this phenomenon was about low end computers with 32-64MB RAM(i don't remember the exact number of ram as my notes in on an other computer) and IDE hard disks.... in the 00's when Sata-I and Sata-II came and computers start to have GB Ram the phenomenon disappeared for a few years.. then Linux started to be bigger and bigger and memory-hungry programs begun to appear and the phenomenon came back again as the "bug" is still there. Now when computers start to have SSD Nvme and 32-64GB ram.. the phenomenon is almost gone again...
But the "bug" is still there and it will appear again in the future then Linux grows and programs grows as in the 90's.
I use the word Bug loosely as its both a bug but also a server design thing.. This phenomenon do hit low end servers too, but most common in the desktop world.

So you ho have this problem, use an SSD for the system or just to your swap partition and it will become less of a problem as a temporary workaround.

.

.

Edit 2022-07-06 time 00:44 swedish timezone

As I got this swapdeath or I/O trashing now tonight and my old laptop just freeze up for a bit over 30minutes.. I manage to get a printscreen between it just hacked and the mouse was moveable a centimeter every 10 second
HDD led was not blinking it was a solid green color, but the actual disk read/write wasn't high.
So this printscreen can show how it looks like when it happens. I/O is 99.99% and it makes the computer freeze up..
Just a view for new users that wondering what it is.
.

[smurphos]

The IO thing is definitely an issue - the only freezes I have suffered in the last couple of years are related to a Cinnamon/CJS bug exposed by Cinnamenu's home search feature which if it kicks in spams xsession-errors with something like 20,000 + lines of repeated error message per second. That level of write (even on a fast NVME SSD) does completely freeze up the system, requiring an xserver kill to recover, but I don't think that's a kernel issue - more that X's logging hasn't got any rate protection built in, and CJS shouldn't have the bug in the first place that causes it to repeat the same error ad-nauseam at sub millisecond intervals without just aborting whatever its trying to do.

[Marie SWE]

'
wow, you are really fast, Smurphos... typing 20000 lines a second. (just joking)

That kind of data transfer can choke any disk even on large raid setups as the cpu often is faster then the disk transfer speed. But the system it self shouldn't freeze with maximum disk transfer if the cpu and memory still has capacity left. so that is the I/O thing.
Memory leaks is a nice thing that often is triggers swapdeath/I/O trashing. We all love the nice memory leak bugs.
I do have a dedicated lab low end laptop I don't use anymore(first gen i5 2GB ram and a slow HDD) for this phenomenon.. I still do some tests on it now and then, but that was almost 6month ago I labbed on it the last time. So it a really low-low-low priority project I'm work on. and it is a bit over my head, so I still learning as I go.
But over the head projects it is a good way to learn how Linux really works. I do feel as I'm young again and back in the late 80s first years of the 90s when I learned how ms-dos and windows worked... It's both fun, but also extremely frustrating, as I am use to solve new things within minutes in the MS-world. but in the Linux-world is months instead of minutes.

As for me I don't use Nmve/SSD as HDD still is cheaper/GB and has a longer lifecycle in writing to disk... So on my desktops I use small SSD's for swap only as a workaround for swapdeath phenomenon. My laptops only has HDD and I put the swap partition in the beginning of the disk as it has a bit higher data-transfer-rate than at the end of the disk. But swapdeath is a fact on them sometimes.
Boot time is never an issue for me, as I start my computers and then I go and get my coffee and make some phone calls.. so when I'm back in 5-10-20minutes the computers is up and running... So 5second boot or 5minutes boot.. I never notice the difference.
But mostly I use suspend to disk on my desktops and suspend to ram on my laptops(except when traveling then I set my laptops to suspend to disk)

[Menard]

I ve read your interesting conclusions, it is allways interesting to understand how things happen
These times I am more on evaluating my last hard drive, a second 2.5" L200 Toshiba, so I ll make a post about this Gnome-disk weird benchmark tool
and also I was searching if I could lower the power consumption using the hibernation, I could try? I say "try" because I don't know exactly how may happen this data writing on a hard drive and perhaps it will be too long for my espectation ... but let's try and for this I wonder if this zswap that I activated, supports the hibernation : ?????

Notice that with my hard drives, my conclusion is that the transfer speed decreases significativly only after the middle of the disk, so I don't try to use the first GB

[Menard]

How-to with explanations on my website:
https://easylinuxtipsproject.blogspot.c ... html#ID1.2
(item 1.2)

On my machines, I was agreeably surprised by the speed gain that I noticed. Have fun!

Can you tell me if zswap but also zram are compatible with the hibernation ?

[Pjotr]

How-to with explanations on my website:
https://easylinuxtipsproject.blogspot.c ... html#ID1.2
(item 1.2)

On my machines, I was agreeably surprised by the speed gain that I noticed. Have fun!

Can you tell me if zswap but also zram are compatible with the hibernation ?

Don't know; I don't ever use hibernation myself.... For what it's worth: my first thought is that it would likely be problematic or even impossible. You might Google a bit for it, in order to find out.

[Marie SWE]

I use zswap and hibernation.. and it works without problems
I have never used zram though.

[smurphos]

You can't hibernate with ZRAM only, but I believe you can have a set up with a low priority physical swap primarily for hibernation, and a high priority ZRAM swap for actual swapping.

[antikythera]

@Pjotr - suggestion for your tutorial, tweak the command to update the initramfs for all installed kernels, not just the running one which is what yours does at present.

sudo update-initramfs -u -k all

also, for LMDE5 lz4 and z3fold don't seem to be supported by the kernels. I get this output despite following your tutorial to the letter and double-checking the results in the files concerned. And yes I did update-grub and issue the command above to update the initramfs files for all installed kernels. libs for lz4 and z3 are installed by default.

[ 2.656759] zswap: loaded using pool lzo/zbud

Code: Select all

# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
z3fold
lz4

Code: Select all

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=lz4"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

EDIT - zswap is configured somewhere else or there's a policy preventing it working. I rolled back to the timeshift snapshot taken earlier this evening before I started trying this out and it's still showing the same zswap: loaded using pool lzo/zbud output.

In fact it is not being activated:

Code: Select all

grep -R . /sys/module/zswap/parameters
/sys/module/zswap/parameters/same_filled_pages_enabled:Y
/sys/module/zswap/parameters/enabled:N
/sys/module/zswap/parameters/max_pool_percent:20
/sys/module/zswap/parameters/compressor:lzo
/sys/module/zswap/parameters/zpool:zbud
/sys/module/zswap/parameters/accept_threshold_percent:9

Half tempted to switch the old Athlon back to Mint 20.3 but to be honest it has 16GB DDR4-3200 (albeit on a single channel controller) and SSD for storage so zswap wouldn't make too much difference. kernel 5.4 would for experimental-amdgpu support again though, it has been removed in kernel 5.9 and above.

[Menard]

Thanks Piotr, Marie, Smurphos I am gointg to mind about your 3 advices

I want to try hibernation because it can reduce daily power consumption ... when I get out for a while
Don't tell me that I could shutdown, because if I shut down I cannot know if someone accessed my data physically ... so 3 solutions :
1 - to let the system started
2 - hibernation
3 - to survey with for example my webcam but Linux ... too complicated, I can do it with XP but I must reboot the machine

[Pjotr]

@Pjotr - suggestion for your tutorial, tweak the command to update the initramfs for all installed kernels, not just the running one which is what yours does at present.

sudo update-initramfs -u -k all

Good tip; thanks.

Furthermore, I'm testing Mint 21 at the moment (I'm one of the prerelease iso testers). In Mint 21, there's no need to add lz4 to the initramfs by hand, so it'll load automatically when told to do so by Grub. Only z3fold still needs to be added to the initramfs.

As I'm already modifying my how-to's for Mint 21, this particular how-to of mine will change as well, in order to take that into account.

[Menard]

On Linux Mint 20.3 Cinnamon it is ok with the default tutorial, I ve just upgraded the kernel to 5.15 and it stays like before

Code: Select all

$ dmesg | grep zswap
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-33-generic root=UUID=02f0eb31-8cc0-4461-86dd-27320a907bb0 ro quiet splash zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=lz4
[    0.031468] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-33-generic root=UUID=02f0eb31-8cc0-4461-86dd-27320a907bb0 ro quiet splash zswap.enabled=1 zswap.zpool=z3fold zswap.compressor=lz4
[    0.850104] zswap: loaded using pool lz4/z3fold

[Marie SWE]

I did an edit in my post above viewtopic.php?p=2186698#p2186698 to just show new users how swapdeath can look like.

[Menard]

I ve read that something could help : it s that we can set the priority of I/O disk by disk ... do yo know it ?

[Marie SWE]

I ve read that something could help : it s that we can set the priority of I/O disk by disk ... do yo know it ?

this is a really complex question.
We are talking about changes on kernel level that will effect the whole system.. one wrong and you kill the system.. so if you going to modify disk I/O settings.. use a VM or a lab-computer.
I have research and still does I/O trashing or swapdeath for a while back, little now and then.. it is a looong term project as it have low priority in my life so to speak.
You can find a lot of information of how the kernel works at https://docs.kernel.org/index.html

If you do find a way before I do, I will just be happy.

Edit
I did a swedish post 2020 but it contains some of the links i gather then.. I do have a lot more links and i mean a lot of them in another computer i use for research
But some of this link may be of interest... My post is in swedish, but all links is in english
https://www.sweclockers.com/forum/post/18807752

[Menard]

Hello, The time I want to spend on these machines is allways limited in theory, in practice I often have nothing more interesting to do

Can someone tell me if he, she, uses double session work ? and with zram or zswap ? It means for me switching sessions, and switching back after logging out the last one