Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Chat about Linux in general
Post Reply
User avatar
michael louwe
Level 9
Level 9
Posts: 2881
Joined: Sun Sep 11, 2016 11:18 pm

Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by michael louwe » Sat Nov 17, 2018 5:31 am

https://www.phoronix.com/scan.php?page= ... sect&num=1 - Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower - 16 Nov 2018
An intentional kernel change for better mitigating against Spectre is the latest reason why the Intel Linux performance is much slower. That change is "STIBP" for cross-hyperthread Spectre mitigation on Intel processors. STIBP is the Single Thread Indirect Branch Predictors (STIBP) allows for preventing cross-hyperthread control of decisions that are made by indirect branch predictors.
A patched computer that is much slower. Your choice.

ud6

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by ud6 » Sat Nov 17, 2018 5:46 am

michael louwe wrote:
Sat Nov 17, 2018 5:31 am
A patched computer that is much slower. Your choice.
A 10-15% reduction in speed is significant, though it seems many would argue that Intel were actually trying to improve speed whilst sacrificing security, so not so much a slow down as just not using a speed up which was inappropriately designed..

https://m.slashdot.org/story/345352

User avatar
catweazel
Level 17
Level 17
Posts: 7767
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by catweazel » Sat Nov 17, 2018 6:02 am

ud6 wrote:
Sat Nov 17, 2018 5:46 am
A 10-15% reduction in speed is significant
Only on paper. The human eye needs about 20% difference before a change in the speed of some thing is detected. The problem is, you now know that 4.20 will be 10-15% slower so, by dint of sheer self-fulfilling prophecy, you'll notice it.
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

ud6

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by ud6 » Sat Nov 17, 2018 6:26 am

catweazel wrote:
Sat Nov 17, 2018 6:02 am
..Only on paper. The human eye needs about 20% difference before a change in the speed of some thing is detected...
Maybe if I'm only waiting for a film to buffer or transferring a short csv file.. but if I was doing a 7 year medical degree I think my eye would notice 8 months difference, or if I was rendering a film the difference between 4 hours and 4 hrs 36 mins to me is quite a difference. Not that I'm a speed freak, but I am impatient :lol: :lol:

User avatar
catweazel
Level 17
Level 17
Posts: 7767
Joined: Fri Oct 12, 2012 9:44 pm
Location: Australian Antarctic Territory

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by catweazel » Sat Nov 17, 2018 6:55 am

ud6 wrote:
Sat Nov 17, 2018 6:26 am
catweazel wrote:
Sat Nov 17, 2018 6:02 am
..Only on paper. The human eye needs about 20% difference before a change in the speed of some thing is detected...
Maybe if I'm only waiting for a film to buffer or transferring a short csv file.. but if I was doing a 7 year medical degree I think my eye would notice 8 months difference, or if I was rendering a film the difference between 4 hours and 4 hrs 36 mins to me is quite a difference. Not that I'm a speed freak, but I am impatient :lol: :lol:
I seriously doubt it. If I was running a 4-hour + task I'd be off out smoking my pipe and listening to the birds. Anyway, the last person to be able to detect a speed difference significantly below 10% was British racing driver, Stirling Moss. He had an autonomic forward and backward saccade angular velocity exceeding that of monkeys, around 1100 degrees per second.

I just thought I'd slip that little bit of history into the conversation :)
¡uʍop ǝpısdn sı buıɥʇʎɹǝʌǝ os ɐıןɐɹʇsnɐ ɯoɹɟ ɯ,ı

User avatar
michael louwe
Level 9
Level 9
Posts: 2881
Joined: Sun Sep 11, 2016 11:18 pm

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by michael louwe » Sat Nov 17, 2018 7:23 am

ud6 wrote:
Sat Nov 17, 2018 5:46 am
A 10-15% reduction in speed is significant, though it seems many would argue that Intel were actually trying to improve speed whilst sacrificing security, so not so much a slow down as just not using a speed up which was inappropriately designed..
.
This latest Linux 4.20 testing endeavor started out with seeing the Intel Core i9 performance pulling back in many synthetic and real-world tests. This ranged from Rodinia scientific OpenMP tests taking 30% longer to Java-based DaCapo tests taking up to ~50% more time to complete to code compilation tests taking measurably longer to lower PostgreSQL database server performance to longer Blender3D rendering times. ...

With seeing clear performance regressions on a number of systems when running the latest Linux 4.20 code, and especially with being able to reproduce it on high-core-count hardware (thus significantly cutting down the kernel build times), this morning I kicked off the kernel bisecting process to see why this new kernel is causing many workloads to run so much slower than Linux 4.19.
Maybe, a new and patched Intel system with Linux kernel 4.20 or higher will be running like a tortoise when under heavy workloads, eg Cloud servers.

With so many Meltdown and Spectre holes or bugs in the CPU, maybe, it's more worthwhile for Intel to get rid of all her Speculative Execution and Branch Prediction stuffs and redesign her new CPUs from zero, ie restart from the CPU design of the 1990s, like some people who like to buy Vintage clothes. ...
https://www.zdnet.com/article/researche ... e-attacks/ - Researchers discover seven new Meltdown and Spectre attacks - Experiments showed that processors from AMD, ARM, and Intel are affected. - By Catalin Cimpanu for Zero Day | November 14, 2018
Last edited by michael louwe on Sat Nov 17, 2018 7:42 am, edited 2 times in total.

ud6

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by ud6 » Sat Nov 17, 2018 7:30 am

catweazel wrote:
Sat Nov 17, 2018 6:55 am
I seriously doubt it. If I was running a 4-hour + task I'd be off out smoking my pipe and listening to the birds. Anyway, the last person to be able to detect a speed difference significantly below 10% was British racing driver, Stirling Moss. He had an autonomic forward and backward saccade angular velocity exceeding that of monkeys, around 1100 degrees per second.

I just thought I'd slip that little bit of history into the conversation :)
Interesting. Maybe I should take up pipe smoking too.
michael louwe wrote:
Sat Nov 17, 2018 7:23 am
This ranged from Rodinia scientific OpenMP tests taking 30% longer to Java-based DaCapo tests taking up to ~50% more time to complete to code compilation tests taking measurably longer to lower PostgreSQL database server performance to longer Blender3D rendering times. ...
Eek. Rendering is what I most need processing speed for. Maybe do the old fashioned thing of setting up various render jobs over night.. or start smoking a pipe.

Hoser Rob
Level 12
Level 12
Posts: 4449
Joined: Sat Dec 15, 2012 8:57 am

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by Hoser Rob » Sat Nov 17, 2018 10:06 am

If it really works I'd happily take a 15-20% speed reductiuon for the security. Getting hacked would be a lot more time consuming.

User avatar
thx-1138
Level 6
Level 6
Posts: 1269
Joined: Fri Mar 10, 2017 12:15 pm
Location: Athens, Greece

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by thx-1138 » Sat Nov 17, 2018 11:58 am

#1) ...we're not using 4.20 yet: quite far away from such actually.
Someone uses such, his/her own (potential problem &) responsibility.

#2) ...do i really need to mention that someone doesn't have to enable every mitigation in the world along with the kitchen sink?
Read carefully, be patient, read again, don't jump to 'conclusions' or be impressed by catchy headlines, ...
then make your decisions wisely & according to your needs.
Else, just go with the defaults, and don't complain (what for, it's totally redudant - there's no-one listening).

#3) ...compared to the original mitigations back in January / February (4.14.x, backports to previous kernels etc),
tons of work & optimizations have taken place in order to avoid slowdowns.
For the vast majority of normal 'desktop' users, the overall slow-down effect should be barely noticeable with 4.17 / 4.18 - if at all:
4.18 literally flies here, especially compared to the 4.13 series used back in January.
On 'ancient' kernel series though, it's rather questionable if such optimizations have (or can be) backported.
Of course, you don't have to take a random guy's word for the above:
you can simply just go through & read numerous previous articles in the very exact above benchmarking site.

#4) ...merely following the comments in the aforementioned article: https://lkml.org/lkml/2018/11/17/248
Ie. monsieur...Intel already provided some patches to avoid the stress, & discusses them with monsieur...Suse
(who, in turn, was the one that provided the always&anywhere stibp-enabled patch).

And that #4 specifically, is pretty much the only thing which is not (semi-irrelevant) Saturday evening chit-chatting (and / or fud). :wink:

gm10
Level 13
Level 13
Posts: 4533
Joined: Thu Jun 21, 2018 5:11 pm

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by gm10 » Sat Nov 17, 2018 12:04 pm

michael louwe wrote:
Sat Nov 17, 2018 7:23 am
ie restart from the CPU design of the 1990s, like some people who like to buy Vintage clothes. ...
You mean back when the FPU wasn't counting correctly? Sounds like fun, I'm in! :)

On a more serious note, don't forget Spectre/Meltdown isn't Intel-specific.
michael louwe wrote:
Sat Nov 17, 2018 7:23 am
https://www.zdnet.com/article/researche ... e-attacks/ - Researchers discover seven new Meltdown and Spectre attacks - Experiments showed that processors from AMD, ARM, and Intel are affected. - By Catalin Cimpanu for Zero Day | November 14, 2018
Yes, but these are already handled by existing mitigations.

rambo919
Level 3
Level 3
Posts: 183
Joined: Wed May 22, 2013 3:11 pm

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by rambo919 » Sun Nov 18, 2018 1:27 am

The arguments for humans "not being able to detect speed loss" are fundamentally flawed.... it assumes that only a few tasks are running and only one requires a lot of power.... and that expensive faster chips are the norm. As soon as many high power processes are run (particularly if many of them are virtualized) a 10% speed difference starts becoming quite obvious. But also on a slower chip almost ANY speed difference is relatively easily detectable.

It's simmilar to people assuming Virtualbox is fast because they only use it on machines that have a lot of RAM and small swap drives... for some reason Virtualbox prioritizes swap usage and things get very very very slow half the time on a machine with only 4GB of RAM even though only 2GB of RAM was given to the VM in the first place because basically ONLY the swap gets used with most RAM being untouched.

Other than the similarity of assumptions causing problems I specifically mention virtualization because many (most?) of the intel security flaw "fixes" directly impact hardware virtualization specifically.

User avatar
michael louwe
Level 9
Level 9
Posts: 2881
Joined: Sun Sep 11, 2016 11:18 pm

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by michael louwe » Mon Nov 19, 2018 7:52 am

https://www.phoronix.com/scan.php?page= ... tibp&num=1 - The Spectre/Meltdown Performance Impact On Linux 4.20, Decimating Benchmarks With New STIBP Overhead - Written by Michael Larabel in Software on 17 November 2018
https://www.phoronix.com/scan.php?page= ... BP-Comment - Linus Torvalds Comments On STIBP & He's Not Happy - STIBP Default Will End Up Changing - Written by Michael Larabel in Linux Kernel on 19 November 2018

User avatar
Pjotr
Level 20
Level 20
Posts: 11053
Joined: Mon Mar 07, 2011 10:18 am
Location: The Netherlands (Holland)
Contact:

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by Pjotr » Mon Nov 19, 2018 7:59 am

michael louwe wrote:
Mon Nov 19, 2018 7:52 am
https://www.phoronix.com/scan.php?page= ... tibp&num=1 - The Spectre/Meltdown Performance Impact On Linux 4.20, Decimating Benchmarks With New STIBP Overhead - Written by Michael Larabel in Software on 17 November 2018
https://www.phoronix.com/scan.php?page= ... BP-Comment - Linus Torvalds Comments On STIBP & He's Not Happy - STIBP Default Will End Up Changing - Written by Michael Larabel in Linux Kernel on 19 November 2018
You have to love Linus Torvalds:
I think we should use the same logic as for L1TF: we default to something that doesn't kill performance. Warn once about it, and let the crazy people say "I'd rather take a 50% performance hit than worry about a theoretical issue".
Trust Linus Torvalds to bring common sense back into kernel design. I'm glad he's the boss of the kernel team. :mrgreen:
Tip: 10 things to do after installing Linux Mint 19 Tara
Keep your Linux Mint healthy: Avoid these 10 fatal mistakes
Twitter: twitter.com/easylinuxtips
All in all, horse sense simply makes sense.

User avatar
michael louwe
Level 9
Level 9
Posts: 2881
Joined: Sun Sep 11, 2016 11:18 pm

Re: Bisected: The Unfortunate Reason Linux 4.20 Is Running Slower

Post by michael louwe » Tue Dec 04, 2018 7:48 am

https://www.phoronix.com/scan.php?page= ... tant-Fixes - Linux 4.19.7 Getting Revised STIBP Code, Important PCIe Fix For Radeon GPUs - Written by Michael Larabel in Linux Kernel on 4 December 2018
- The improved upon STIBP (Single Thread Indirect Branch Predictor) code. After the original STIBP code was back-ported to 4.19, it was quickly reverted due to the large performance impact. Fortunately, being merged to mainline Linux 4.20 Git a few days ago and quickly coming to the 4.19 stable tree is the improved STIBP code where by default this cross-HT Spectre V2 mitigation is only opt-in via prctl and by default for SECCOMP processes. So all is good on the STIBP front for the upcoming Linux 4.19.7.

Post Reply

Return to “Chat about Linux”