Linux 6.1 Lands Revert For "Huge Performance Regressions" From Three Lines Of Code
Ahead of the Linux 6.1-rc8 kernel that Linus Torvalds is expected to issue shortly rather than going straight to Linux 6.1 stable, a revert for a small change leading to "huge performance regressions" in select areas has fortunately been caught and reverted.
For the Linux 6.1 merge window was a memory management change to align larger anonymous mappings to THP boundaries. The commit reasoned:
But it turned out that the three lines of code changed ended up regressing the Linux 6.1 kernel performance in some areas.
Intel's kernel testing found a 95% regression in one of the will-it-scale benchmarks that was traced back to that change. That benchmark isn't particularly real-world but it does raise concerns over what other (real-world) user-space workloads may also be impacted too that haven't yet been tested on the Linux 6.1 development builds.
Linux developer Nathan Chancellor also reported a big regression in kernel build times from that original change. For building out all of his kernel builds on a Threadripper 3990X workstation, it went from 2 hours and 20 minutes to then over 3 hours... Or for a x86_64 allmodconfig build, from 318 seconds to 406 seconds. Quite a significant slowdown in kernel build times, in addition to the run-time performance hit noted by Intel testing. Meanwhile the original change had helped with GCC-based kernel build times by making it about 2% faster.
Thus for now Linus Torvalds decided to revert the 3 line patch to deal with what he calls the "huge performance regressions" until this situation can be better sorted out to avoid the significant regressions.
For the Linux 6.1 merge window was a memory management change to align larger anonymous mappings to THP boundaries. The commit reasoned:
"Align larger anonymous memory mappings on THP boundaries by going through thp_get_unmapped_area if THPs are enabled for the current process.
With this patch, larger anonymous mappings are now THP aligned. When a malloc library allocates a 2MB or larger arena, that arena can now be mapped with THPs right from the start, which can result in better TLB hit rates and execution time."
But it turned out that the three lines of code changed ended up regressing the Linux 6.1 kernel performance in some areas.
Intel's kernel testing found a 95% regression in one of the will-it-scale benchmarks that was traced back to that change. That benchmark isn't particularly real-world but it does raise concerns over what other (real-world) user-space workloads may also be impacted too that haven't yet been tested on the Linux 6.1 development builds.
Linux developer Nathan Chancellor also reported a big regression in kernel build times from that original change. For building out all of his kernel builds on a Threadripper 3990X workstation, it went from 2 hours and 20 minutes to then over 3 hours... Or for a x86_64 allmodconfig build, from 318 seconds to 406 seconds. Quite a significant slowdown in kernel build times, in addition to the run-time performance hit noted by Intel testing. Meanwhile the original change had helped with GCC-based kernel build times by making it about 2% faster.
Thus for now Linus Torvalds decided to revert the 3 line patch to deal with what he calls the "huge performance regressions" until this situation can be better sorted out to avoid the significant regressions.
11 Comments