Linux 6.13 RDMA Changes Headlined By NVIDIA's New Data Placement Ordering Feature

Written by Michael Larabel in Hardware on 26 November 2024 at 06:00 AM EST. 3 Comments
HARDWARE
The RDMA subsystem updates were sent out last Friday for the ongoing Linux 6.13 kernel cycle. Most notable with the RDMA updates is the NVIDIA Mellanox "MLX5" network driver introducing a new Data Direct Placement (DDP) feature to further help with performance.

As covered a few weeks back on Phoronix, the NVIDIA-Mellanox Data Direct Placement (DDP) feature is a novel data placement ordering feature for enhancing performance. The DDP feature allows for the user to consume data out of order from RDMA write and send operations while the completion ordering remains in order. With Data Direct Placement the intent is on achieving higher bandwidth by permitting responders to receive packets out of order. Paired with the multi-plane support this can net big bandwidth wins for the data path.

NVIDIA MLX5 adapter


It will be interesting to see the DDP performance advantages in practice but so far on the prior patches I haven't seen any numbers to quantify it from NVIDIA. The patches sum up DDP as:
"This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed."

Aside from the MLX5 data placement ordering feature, the Broadcom BNXT driver adds some minor features/optimizations and various other enhancements within the subsystem. More details within this pull request.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week