NVMeTCP Offload Bits Coming For Linux 5.14 To Lower CPU Utilization, Better Latency

[phoronix]

Phoronix: NVMeTCP Offload Bits Coming For Linux 5.14 To Lower CPU Utilization, Better Latency

Adding to other networking changes queuing up for the upcoming Linux 5.14 cycle, NVMeTCP Offload has begun landing into "net-next" ahead of this next kernel merge window...

https://www.phoronix.com/scan.php?page=news_item&px=NVMeTCP-Offload-Linux-5.14

[microcode]

Cool. I guess this is the sort of thing WD is looking to enable with their programmable storage stuff.

[IntrusionCM]

"With this implementation on top of the Marvell qedn driver (using the Marvell FastLinQ NIC), we were able to demonstrate the following CPU utilization improvement: On AMD EPYC 7402, 2.80GHz, 28 cores:
- For 16K queued read IOs, 16jobs, 4qd (50Gbps line rate): Improved the CPU utilization from 15.1% with NVMeTCP SW to 4.7% with NVMeTCP offload. On Intel(R) Xeon(R) Gold 5122 CPU, 3.60GHz, 16 cores: - For 512K queued read IOs, 16jobs, 4qd (25Gbps line rate): Improved the CPU utilization from 16.3% with NVMeTCP SW to 1.1% with NVMeTCP offload. In addition, we were able to demonstrate the following latency improvement: - For 200K read IOPS (16 jobs, 16 qd, with fio rate limiter): Improved the average latency from 105 usec with NVMeTCP SW to 39 usec with NVMeTCP offload. Improved the 99.99 tail latency from 570 usec with NVMeTCP SW to 91 usec with NVMeTCP offload."

The news message shortens sadly the text too much. The details are essential - as otherwise the wrong impression might be given that Intel > AMD. Which isn' true

[Ipkh]

Reading the specific details, the Intel system had the largest improvement. Although I agree that both systems substantially improved.

[S.Pam]

From the link :


On AMD EPYC 7402, 2.80GHz, 28 cores:
For 16K queued read IOs, 16jobs, 4qd (50Gbps line rate):
Improved the CPU utilization from 15.1% with NVMeTCP SW to 4.7% with NVMeTCP offload.

On Intel(R) Xeon(R) Gold 5122 CPU, 3.60GHz, 16 cores:
For 512K queued read IOs, 16jobs, 4qd (25Gbps line rate):
Improved the CPU utilization from 16.3% with NVMeTCP SW to 1.1% with NVMeTCP offload.

So we're testing also different network hardware.

In any case. Cool improvement.