Looking At Why Linux 5.0 Is Running Slower For Apache & PostgreSQL On Some Systems

Last week I reported on some slowdowns when running on the Linux 5.0 development kernel for both Intel and AMD systems. As a few days passed and the regression didn't seem to be figured out and addressed by upstream, and several inquiries from Phoronix readers, I spent some time looking at some of the slowdowns encountered when running on this bleeding-edge code.

The slowdowns when encountered so far on a few different systems were some of the most sizable regressions since the Linux 4.14 to 4.15 transition when Spectre and Meltdown mitigations began rolling out. But with the 5.0 regressions, they haven't been across the board and range from a few percent to about 10% or so.

With not being massive slowdowns to quickly and trivially rule out any fluctuation/noise and also not occurring for a large number of tests, it wasn't quick to have it bisected, but with Linux 5.0 set to be released this weekend I decided to devote some time to having it bisected with the Phoronix Test Suite testing framework.

For my purposes, I was looking at the slowdowns occurring with the Apache web server (HTTPD) with testing via Siege and the PostgreSQL database server with pgbench. The Sockperf socket performance benchmark also seems to fall in the same boat, but haven't had the time yet to look into all of the 5.0 cases of slower performance to see if there are multiple regressions at play or not... Well, at least for the network/socket-related workloads.


First, thanks to AMD for having sent out the Dell PowerEdge R7425 server a few months back. This server with dual EPYC 7601 processors for a combined 64 cores / 128 threads currently remains the most powerful server I have available locally and with 512GB of RAM and twenty SATA 3.0 SSDs in RAID can make light work out of building the Linux kernel and related tasks. The Dell PowerEdge R7425 worked out great and sharply speeds up the process of bisecting this Linux kernel regression.

From the bisecting, the preliminary cause appears to be net: allow binding socket in a VRF when there's an unbound socket. Still running some follow-up tests and reverting to confirm, but appears to be the source at least of the Apache / PostgreSQL slowdowns spotted.

That work is part of this series that originally explained, "There is no easy way currently for applications that want to receive packets in the default VRF to be isolated from packets arriving in VRFs, which makes using VRF-unaware applications in a VRF-aware system a potential security risk. So change the inet socket lookup to avoid packets arriving on a device enslaved to an l3mdev from matching unbound sockets by removing the wildcard for non sk_bound_dev_if and instead relying on check against the secondary device index, which will be 0 when the input device is not enslaved to an l3mdev and so match against an unbound socket and not match when the input device is enslaved. Change the socket binding to take the l3mdev into account to allow an unbound socket to not conflict sockets bound to an l3mdev given the datapath isolation now guaranteed." Though a developer there argued that it's not an actual security risk.

I am not into the networking area that closely, so I'll just leave these initial findings here and update when I've wrapped up some additional tests with any other findings.