VMware Hits A Nasty Performance Regression With Linux 5.13

Written by Michael Larabel in Virtualization on 30 July 2021 at 09:00 AM EDT. 12 Comments
VIRTUALIZATION
VMware has found the Linux 5.13 kernel that was released as stable one month ago has led to a serious performance regression for their ESXi enterprise hypervisor.

VMware found that there is a big performance regression with Linux 5.13 under their ESXi software. They found that ESXi compute workloads could be affected by up to 3x while networking workloads were at only a 40% regression. Their compute tasks were as simple as Linux kernel compile times that were severely impacted.

VMware engineers found that this very significant performance regression came from a scheduler change... While the patch in question "sched: Move SCHED_DEBUG sysctl to debugfs" sounds rather mundane, it was found to ultimately impact the default value of sched_wakeup_granularity_ns.

Changing that default scheduler wakeup granularity from 15ms to 4ms (and the minimum granularity from 10ms to 3ms) is causing compute and network workloads to be significantly hurt at least under ESXi. The belief is these new lower values by default is causing more preemption and involuntary switches.

The details can be found via this mailing list post. VMware is now seeking input on why this default change occurred, whether it was even intentional, and if it can be changed back to the prior default given the severe performance impact they are seeing on this latest stable kernel build.

Outside of their ESXi hypervisor, I haven't heard of Linux 5.13 causing any severe performance impact nor have I seen such across my testing on many different systems albeit not focused on virtualized environments.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week