Linux 5.0 To Linux 5.9 Kernel Benchmarking Was A Bumpy Ride With New Regressions

First up was wondering why the IO_uring performance was so poor on the Linux 5.9 kernel compared to prior releases when using the same FIO build and the same FIO configuration throughout.

Thanks to the Phoronix Test Suite for making it easy to reproduce and layering on top of Git bisect for speeding up the kernel bisection and the kernel build speeds of the EPYC 7702, the commit was soon uncovered.

The commit leading to the lower random read performance with IO_uring was tracked down to this two-line patch over mm: allow read-ahead with IOCB_NOWAIT set.

When asking Jens Axboe about this patch that he authored and maintainer of the Linux block subsystem and IO_uring lead developer, he provided some interesting insight over the performance drop. And it's actually expected.

Jens explained that Linux 5.9 is indeed handling buffered I/O differently now with IO_uring. Prior to Linux 5.9, buffered reads would spawn extra threads while that is no longer the case with 5.9+. With the new behavior in Linux 5.9, the latency should improve while the throughput can be impacted if it previously benefited from spreading the load over the multiple threads. Basically there is now more efficient async buffered I/O but can cause a regression particularly on larger systems where previously were using more threads.

For workloads where Linux 5.9 is slower, Jens recommends the software use IOSQE_ASYNC for offloading of requests or by making use of multiple rings. In the context of FIO, Jens yesterday added this change that allows setting IOSQE_ASYNC for offloading of every N requests.

In showing of more efficient usage from user-space, off Linux 5.9 I then carried some new tests with FIO Git for benchmarking as well as using this brand new force_async option to offload every four requests to the async helper.

Indeed when using the updated FIO and the new option for adapting to the Linux 5.9 behavior, it's possible to see much better performance.

IO_uring is very exciting but with it still being adopted / relatively new, hopefully any major real-world users of it will quickly adapt their behavior to be friendly towards Linux 5.9 and beyond.

So that explains the change seen on Linux 5.9 with FIO, but what about that big Apache HTTPD drop?