Linux 5.0 To Linux 5.9 Kernel Benchmarking Was A Bumpy Ride With New Regressions

[Luke]

When out of all huge multibillion dollar companies (IBM, Intel, Microsoft et al) and individuals working on the kernel, Phoronix (which is run by a single individual) finds mission-critical regressions which cut performance by half, it speaks volumes about how Linux is being developed.<clip>

Indeed. Some of those mulitbillion dollar companies ought to pony up some money for this particular test. IBM, Intel, and AMD all could afford to run this test themselves (the latter two could probably do it quickly with clusters of a few dozen nodes of their best chips) but apparently chose not to. They should be doing this sort of testing on an ongoing basis (rc-version by rc-version) but are choosing not to. Now Michael steps up to the plate and saves them as well as volunteer kernel devs MONTHS of work chasing regressions.

I understand without prior offer and acceptance no legal contract exists, but if these huge businesses ever want to get this kind of help again they ought to help fund the kitchen behind the table at which they are gorging themselves.

[a-zander]

Excellent work here Michael ! Maybe you should have a virtual tip jar for these types of deep dive reports.

[Michael]

Excellent work here Michael ! Maybe you should have a virtual tip jar for these types of deep dive reports.

PayPal tip link is mentioned on the first page of this article.

[a-zander]

PayPal tip link is mentioned on the first page of this article.

And that link has now been used by me. Cheers! Seriously, this is the type of content that made me a subscriber in the first place.

[Michael]

Indeed. Some of those mulitbillion dollar companies ought to pony up some money for this particular test. IBM, Intel, and AMD all could afford to run this test themselves (the latter two could probably do it quickly with clusters of a few dozen nodes of their best chips) but apparently chose not to. They should be doing this sort of testing on an ongoing basis (rc-version by rc-version) but are choosing not to. Now Michael steps up to the plate and saves them as well as volunteer kernel devs MONTHS of work chasing regressions.

I understand without prior offer and acceptance no legal contract exists, but if these huge businesses ever want to get this kind of help again they ought to help fund the kitchen behind the table at which they are gorging themselves.

I used to run daily benchmarks of Linux Git in many performance tests [ http://linuxbenchmarking.com/?daily-...x-kernel-tests ] so could have spotted the regressions sooner. But abandoned those efforts unfortunately due to lack of funding due to power/cooling costs and capacity issues amongst my other tests. Could run lots of daily benchmarks if only there weren't such obstacles.

[gfunk]

No gaming benchmarks? would be nice to see one native game and one wine/proton game to see which kernel is best for performance

[indepe]

For workloads where Linux 5.9 is slower, Jens recommends the software use IOSQE_ASYNC for offloading of requests or by making use of multiple rings. In the context of FIO, Jens yesterday added this change that allows setting IOSQE_ASYNC for offloading of every N requests.

So force_async=1 results in the previous behavior? I haven't used io_uring yet, but I plan to at a certain point... it seems increasingly difficult to keep track of all the evolving details.

Kudos for the great article and research!

[digitalsin]

How come Microsoft and other Unix vendors (Oracle/IBM/QNX) and even open source community projects (FreeBSD/OpenBSD/NetBSD) don't have such longstanding regressions? I'm pretty sure the Linux kernel has an order of magnitude more backing/support/development money than all the BSDs combined.

Also, I do understand that regressions happen, it's quite usual with software development. What escapes me is when we have regressions which linger on for years.

Maybe because Linux is a "moving target". Also probably a factor is the size of it [it's large]. As far as Microsoft they have so many years of manpower behind them at any given moment.

[_Alex_]

If one had the resources, it could probably be easy to test thousands of patches over the last few years and find something like a top-10 or top-100 list of regressors and then assign it as a top priority for fixing.

Then fix these (or most of them) and have something like +30% - +50% performance for the linux kernel

(It might sound a lot but it's actually conservative when a single patch can do damage like 10-20-30%.)

[smitty3268]

Maybe because Linux is a "moving target". Also probably a factor is the size of it [it's large]. As far as Microsoft they have so many years of manpower behind them at any given moment.

MS has been getting roasted for all the regressions they have been shipping lately. It's not like 10 years ago when they were spending a year testing all the code they released, they've moved onto a rapid development and release cycle and are now running into the exact same problems everyone else does with that. Plus they fired a major part of their QA employees.

One of their updates even started deleting files. That makes a little performance issue like this seem like nothing in comparison.