VMware Hits A Nasty Performance Regression With Linux 5.13

VMware has found the Linux 5.13 kernel that was released as stable one month ago has led to a serious performance regression for their ESXi enterprise hypervisor.

VMware found that there is a big performance regression with Linux 5.13 under their ESXi software. They found that ESXi compute workloads could be affected by up to 3x while networking workloads were at only a 40% regression. Their compute tasks were as simple as Linux kernel compile times that were severely impacted.

VMware engineers found that this very significant performance regression came from a scheduler change... While the patch in question "sched: Move SCHED_DEBUG sysctl to debugfs" sounds rather mundane, it was found to ultimately impact the default value of sched_wakeup_granularity_ns.

Changing that default scheduler wakeup granularity from 15ms to 4ms (and the minimum granularity from 10ms to 3ms) is causing compute and network workloads to be significantly hurt at least under ESXi. The belief is these new lower values by default is causing more preemption and involuntary switches.

The details can be found via this mailing list post. VMware is now seeking input on why this default change occurred, whether it was even intentional, and if it can be changed back to the prior default given the severe performance impact they are seeing on this latest stable kernel build.

Outside of their ESXi hypervisor, I haven't heard of Linux 5.13 causing any severe performance impact nor have I seen such across my testing on many different systems albeit not focused on virtualized environments.