IOCB_NOWAIT For Linux Pipes Yields 10~23x Performance Improvement

In wanting to avoid waiting for pipes via the IOCB_NOWAIT option in order to further enhance IO_uring performance, Jens Axboe has implemented said functionality and in a simple test is seeing 10x to 23x performance improvements.

"One thing that's always been a bit slower than I'd like with io_uring is dealing with pipes. They don't support IOCB_NOWAIT, and hence we need to punt them to io-wq for handling. This series adds support for FMODE_NOWAIT to pipes," began one of Axboe's latest patch series. But it's more exciting with how that message ended:

Curious on how big of a difference this makes, I wrote a small benchmark that simply opens 128 pipes and then does 256 rounds of reading and writing to them. This was run 10 times, discarding the first run as it's always a bit slower. Before the patch:

Avg: 262.52 msec
Stdev: 2.12 msec
Min: 261.07 msec
Max 267.91 msec

and after the patch:

Avg: 24.14 msec
Stdev: 9.61 msec
Min: 17.84 msec
Max: 43.75 msec

or about a 10x improvement in performance (and efficiency).

I ran the patches through the ltp pipe and splice tests, no regressions observed.

While a 10x improvement is exciting enough, Axboe then followed up with hitting a 23x improvement!

The above test was for a pipe being empty when the read is issued, if the test is changed to have data when, then it looks even better:

Before:

Avg: 249.24 msec
Stdev: 0.20 msec
Min: 248.96 msec
Max: 249.53 msec

After:

Avg: 10.86 msec
Stdev: 0.91 msec
Min: 10.02 msec
Max: 12.67 msec

or about a 23x improvement.

Hopefully this FMODE_NOWAIT support for pipes will be ready for releasing with the Linux 6.4 kernel this summer.