Linux 5.5 Seeing Some Wild Swings In Performance - Improvements But Also Regressions

[perpetually high]

Linuxxx I really can't thank you enough, acpi-cpufreq performance is most definitely faster than both intel_pstate performance and intel-cpufreq performance.

The RedHat low latency guide you mentioned has some great stuff in there, and the command "sudo turbostat sleep 5" and others give insight to why it is faster like you mentioned.

And I say this as someone that compiles their own tweaked out kernel (-O3, -march=native, preemptive low-latency kernel, 1000 MHz, and many other tweaks) and this one change has certainly improved the perceived latency of the machine in just normal day-to-day stuff that may not show in benchmarks. It's as if every move I make on the desktop is getting the full maxed-out performance, if that makes sense.

Thanks again!

[aufkrawall]

At this occasion: The reported high idle clocks without intel-pstate=powersave are completely out of reality too. The CPU (at least Skylake and newer) internally clocks down without the OS knowing.

[Linuxxx]

When I disable pstate on my notebook I lose my Turbo Boost range that is pretty high. With pstate my CPU can scale between 800Mhz and 3.2Ghz. With acpi_cpufreq it is only 800Mhz - 2Ghz. When running on all cores the CPU normally operates at 2.9Ghz. So I would lose 900Mhz of processing power.

So it really depends on your CPU. My desktop CPU only has a Turbo Boost of 200Mhz (3.2 -> 3.4) but only with single thread operations. So acpi_cpufreq might be better here.

You can also use cpupower to set a fixed minimum and maximum frequency. So it does not scale too much.

Obviously You only looked at the reported numbers by acpi-cpufreq without digging any further, so to counter any possible upcoming FUD, let me first of all state this very clearly:

No, you are not losing the CPU's Turbo-Boost capability when utilising acpi-cpufreq! On the contrary, just like the CPU is still able to enter its deepest sleep state, acpi-cpufreq will request and also get the boost clocks from your Intel CPU!

With that out of the way, let me explain the numbers game:

When executing

sudo cpupower -c all frequency-info

you correctly noticed that only the base clocks are shown with acpi-cpufreq.
However, when looking more closely, you will also notice that the highest base clock will be shown twice as a possible speed-step, e.g. 3.10 Ghz in the case of my i5-3350P.
The first one of these is actually the boost state; it's basically just the honest way of reporting the feature, since how much boost you get out of your Intel CPU is dependant on external factors and thus entirely up to the processors firmware/microcode!

So, how can we nevertheless observe with acpi-cpufreq that we are actually getting the Intel turbo-boost we unfortunately paid for?
Here's the trick:

cpupower -c all frequency-info

Notice something?
Omitting the "sudo" command will lead to 'cpupower' being unable to query the hardware directly for the current CPU frequency, so it will revert to asking the Linux kernel for the correct value.
And voila! Depending on the load and the selected governor, you will see that your nowadays crippled Intel CPU is indeed still able to hit its advertised turbo speeds! (But probably only until a vulnerability is found for that feature, too!)

[Volta]

Kudos to Intel for doing that. The company really does great things for Linux (and as far as I know they extensively use Linux for developing new CPU architectures).

They're for sure better software than hardware company. When comes to 5.5 performance gains and regressions it was expected, because there are some important changes in scheduler or low level stuff. It was described in one of the Ingo's commits during a merge window.

[-MacNuke-]

Seriously Intel, are you doing this on purpose?

When I disable pstate on my notebook I lose my Turbo Boost range that is pretty high. With pstate my CPU can scale between 800Mhz and 3.2Ghz. With acpi_cpufreq it is only 800Mhz - 2Ghz. When running on all cores the CPU normally operates at 2.9Ghz. So I would lose 900Mhz of processing power.

So it really depends on your CPU. My desktop CPU only has a Turbo Boost of 200Mhz (3.2 -> 3.4) but only with single thread operations. So acpi_cpufreq might be better here.

You can also use cpupower to set a fixed minimum and maximum frequency. So it does not scale too much.

[nuetzel]

Looks like Intel runs something like that, see https://lkml.org/lkml/2019/11/26/748

Actually this robot reports a lot of various regressions along with git commits which caused them.

Kudos to Intel for doing that. The company really does great things for Linux (and as far as I know they extensively use Linux for developing new CPU architectures).

FWIW

phoronix-test-suite.glmark2.0.score -23.7% regression

is FIXED (Was a WC pat_memtype_list regression.)

https://lkml.org/lkml/2019/12/1/109

[tildearrow]

I know how You feel...

However, we as Intel victims are not completely helpless:

Recently, I have found out why passing

intel_pstate=disable

to the kernel leads to such drastic improvements in performance & latency:

From the RedHat Low-Latency tuning guide:


So all along it was Intel's IDLE driver that was the culprit, not the P-State driver!
This finally explains why Intel systems always felt so sluggish, even though I was always using the Performance governor while also setting the so-called Perf-Bias to zero for maximum performance-bias!

Seriously Intel, are you doing this on purpose?

I haven't used the P-State frequency driver in years because I feel it's just a big lie.
powersave does nothing to save power, and performance does nothing to help with performance!

[birdie]

Wow. Epic swings. That is just rubbish. Performance regression on directory basis?
How hard can it be? Run on a farm. Compile on every commit, build against a few targets, vote for a suite, run it and dump a feedback on the developer of the commit. Priority to specific directories in the kernel. arch/*, block, kernel, mm, fs.

Looks like Intel runs something like that, see https://lkml.org/lkml/2019/11/26/748

Actually this robot reports a lot of various regressions along with git commits which caused them.

Kudos to Intel for doing that. The company really does great things for Linux (and as far as I know they extensively use Linux for developing new CPU architectures).

[aphysically]

Right some tests are ~50% faster, others ~50% slower. I am working on bisecting the biggest ones right now.

there was something in the mailing list about a performance bisect honeypot in the last week

[pyler]

Bad news for linus