Linux 6.13 RDMA Changes Headlined By NVIDIA's New Data Placement Ordering Feature

The RDMA subsystem updates were sent out last Friday for the ongoing Linux 6.13 kernel cycle. Most notable with the RDMA updates is the NVIDIA Mellanox "MLX5" network driver introducing a new Data Direct Placement (DDP) feature to further help with performance.

As covered a few weeks back on Phoronix, the NVIDIA-Mellanox Data Direct Placement (DDP) feature is a novel data placement ordering feature for enhancing performance. The DDP feature allows for the user to consume data out of order from RDMA write and send operations while the completion ordering remains in order. With Data Direct Placement the intent is on achieving higher bandwidth by permitting responders to receive packets out of order. Paired with the multi-plane support this can net big bandwidth wins for the data path.

It will be interesting to see the DDP performance advantages in practice but so far on the prior patches I haven't seen any numbers to quantify it from NVIDIA. The patches sum up DDP as:

"This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed."

Aside from the MLX5 data placement ordering feature, the Broadcom BNXT driver adds some minor features/optimizations and various other enhancements within the subsystem. More details within this pull request.