Linux 5.15 Hit By Some Early Performance Regressions But Quickly Reverted

[phoronix]

Phoronix: Linux 5.15 Hit By Some Early Performance Regressions But Quickly Reverted

In addition to Linus Torvalds dealing with the -Werror fallout, separately in kernel land there were also some significant performance regressions introduced during the Linux 5.15 that led to Linus reverting some of the changes...

https://www.phoronix.com/scan.php?page=news_item&px=Linux-5.15-Memcg-Regressions

[tildearrow]

Too bad LinuxBenchmarking is no longer a thing...

[M@GOid]

Considering the frequency of performance regressions that still slips trough "stable" releases, I wonder how much the big companies invest on Linux QA. I mean, if your earnings depend on server performance, it is in your best interest that a patch from another company don't mess with your business.

[caligula]

Considering the frequency of performance regressions that still slips trough "stable" releases, I wonder how much the big companies invest on Linux QA. I mean, if your earnings depend on server performance, it is in your best interest that a patch from another company don't mess with your business.

They might have their internal patch sets to maintain the expected level of performance. It's actually beneficial for them if the other companies use the standard mainline kernels without knowing anything about these regressions.

[mdedetrich]

They might have their internal patch sets to maintain the expected level of performance. It's actually beneficial for them if the other companies use the standard mainline kernels without knowing anything about these regressions.

Could you explain why?

To me the most idiomatic way of handling this is to run such benchmarks against such commits (i.e. the commits which claim to improve or alter performance) before they get merged.

Otherwise it creates all of this unnecessary patch revert ceremony

[caligula]

Could you explain why?

Cloud providers A, B, and C use the Linux kernel. A knows that a regression X occurs in kernel 5.15. They patch it internally. X goes undetected for companies B and C. Their stack runs slower. A wins market share.

[Serafean]

They might have their internal patch sets to maintain the expected level of performance. It's actually beneficial for them if the other companies use the standard mainline kernels without knowing anything about these regressions.

I know a company that does keep an internal patchset. It also became frozen on a specific version of linux due to said patchset. They have been discussing trying to catch up to mainline for a few years now.

[pmorph]

Considering the frequency of performance regressions that still slips trough "stable" releases, I wonder how much the big companies invest on Linux QA. I mean, if your earnings depend on server performance, it is in your best interest that a patch from another company don't mess with your business.

I would think most big companies don't work that much with the latest stable kernels? They probably use older kernels (with patches), and run performance tests over the whole product.

[M@GOid]

I would think most big companies don't work that much with the latest stable kernels? They probably use older kernels (with patches), and run performance tests over the whole product.

That is a cloud provider type situation. I talk more about a company that sells products that run on Linux, like Intel, IBM for hardware, or Oracle for software. If they product start to under-perform compared to the competition because of a kernel bug, their sales people will suffer.

Even in the case of Oracle, they eventually will have to upgrade the kernel of their distro sooner or later.

[stormcrow]

I would think most big companies don't work that much with the latest stable kernels? They probably use older kernels (with patches), and run performance tests over the whole product.

Yeah, but that can lead to Serafean 's noted consequence. That can have rather drastic consequences if you're not agile enough to keep on top of the (very) fast changing landscape we're now in. Exploits are being probed and utilized within minutes of public disclosure. Distros tend to be on the inside loop of errata disclosures because of exposure - people know who to contact. Who's going to tell company Y that their internally customized database cluster is now vulnerable to a severity 9.8 CVE RCE chain ...oops... it just got pwned through a pivoted BEC just 5 minutes ago. Now a TB of customer data is for sale on the dark web, the company has lost access to half its servers, and lawyers are lining up to sue for negligence.