Linux 5.0 To Linux 5.9 Kernel Benchmarking Was A Bumpy Ride With New Regressions

[leipero]

Yeah I wouldn't bother bisecting kernel or any large code-base these days with less than 64 cores.... Would have done my dual 7742 2P server if it were not busy with other work.

I feel little bit of the sarcasm here . But seriously, those compile times are insane, without localmodconfig you would need as many threads as possible .

[leipero]

When out of all huge multibillion dollar companies (IBM, Intel, Microsoft et al) and individuals working on the kernel, Phoronix (which is run by a single individual) finds mission-critical regressions which cut performance by half, it speaks volumes about how Linux is being developed. People are arguing if it's ready for the desktop, meanwhile the core part of Linux is being sabotaged all the fucking time. It's quite sad really. In a perfect world we would see performance increases with each release, instead the fastest kernel out of the ten which have been tested, is 5.2 which was released over a year ago. That's appalling.

To be fair, it's the rc kernel that have regressions, regardless, it's very useful to note them.

[Michael]

I feel little bit of the sarcasm here . But seriously, those compile times are insane, without localmodconfig you would need as many threads as possible .

Not really sarcastic.... When bisecting multiple kernel regressions would be too time consuming not using the highest end core count systems when they are available. This testing already took about a week as is with all the kernel benchmarks and then the bisecting process and still rather time consuming and ultimately probably not profitable in the end in terms of the traffic / new subscribers or tips to justify the effort.

[ferry]

When out of all huge multibillion dollar companies (IBM, Intel, Microsoft et al) and individuals working on the kernel, Phoronix (which is run by a single individual) finds mission-critical regressions which cut performance by half....

Inside each multibilion dollar company, you will find a division and inside that division a department and inside that department a group and inside that group there will be a person doing real work, fixing stuff and generally making peoples lives better, while the rest are busy have Zoom meetings.

...In a perfect world we would see performance increases with each release...

Performance isn't everything. For most tasks on a desktop a100% performance increase would go unnoticed. A fixed bug will certainly be noticed. And a $#%#% application that would understand in which directory a want to save my new file so that I don't need to traverse 10 subdirectories, would save me huge amounts of time, even with a 50% performance decrease.

IOW usability trumps performance (except for HPC where usability == performance)

[leipero]

Not really sarcastic.... When bisecting multiple kernel regressions would be too time consuming not using the highest end core count systems when they are available. This testing already took about a week as is with all the kernel benchmarks and then the bisecting process and still rather time consuming and ultimately probably not profitable in the end in terms of the traffic / new subscribers or tips to justify the effort.

I can relate to that, trust me. Few weeks back I got the idea to contribute results running PTS, esitmated time was about 10+ hours . I understand that some tests need time, but you may want to consider "trimming time" whenever is possible while not affecting results (maybe a script that would re-calculate by % on the OB - if possible, to save old results in usable fashion). That being said, I'm still commited to run full test suite at some point.
As for the other issue, it is what it is, hence why the above suggestion might make sense.

[onlyLinuxLuvUBack]

Something doesn't make sense to me.

If you set buffered=1 & direct=0 in fio that number I nearly get , so buffered has a limit.

If you set buffered=0 & direct=1 in fio that number I get 3.56 Million Iops with nvme m2 ssds.

How come with your expensive nvme drive you get nearly the same ball park numbers if direct or not ?
does that drive support multiple namespaces? Did you use namespace=1 ?

[christianb]

Well done, Michael! This sort of work is what convinced me to get premium.

[Charlie68]

When out of all huge multibillion dollar companies (IBM, Intel, Microsoft et al) and individuals working on the kernel, Phoronix (which is run by a single individual) finds mission-critical regressions which cut performance by half, it speaks volumes about how Linux is being developed. People are arguing if it's ready for the desktop, meanwhile the core part of Linux is being sabotaged all the fucking time. It's quite sad really. In a perfect world we would see performance increases with each release, instead the fastest kernel out of the ten which have been tested, is 5.2 which was released over a year ago. That's appalling.

Unfortunately regressions are part of development, they shouldn't happen, but sometimes they do, but I don't think it's Microsoft's fault or anyone else's, it happens when you're dealing with code as complex as a kernel can be.

[birdie]

Unfortunately regressions are part of development, they shouldn't happen, but sometimes they do, but I don't think it's Microsoft's fault or anyone else's, it happens when you're dealing with code as complex as a kernel can be.

How come Microsoft and other Unix vendors (Oracle/IBM/QNX) and even open source community projects (FreeBSD/OpenBSD/NetBSD) don't have such longstanding regressions? I'm pretty sure the Linux kernel has an order of magnitude more backing/support/development money than all the BSDs combined.

Also, I do understand that regressions happen, it's quite usual with software development. What escapes me is when we have regressions which linger on for years.

[jetwalsh]

Michael,

Outstanding work! Simply brilliant, in-depth analysis. Bravo!