RDMA Changes For Linux 5.12 Add DMA-BUF Support For Peer-To-Peer Transfers With GPUs

The changes within the remote direct memory access (RDMA) subsystem for Linux 5.12 are deemed "quite small" but there is one interesting addition courtesy of Intel.

Covered last year on Phoronix was the Intel-led work on DMA-BUF support for RDMA for supporting peer-to-peer transactions over PCI Express between RDMA-enabled NICs and other PCIe devices, such as graphics cards. That peer-to-peer support via DMA-BUF with a focus on GPU support is what's coming in Linux 5.12.

The RDMA pull for Linux 5.12 includes the support for creating an RDMA memory region from a DMA-BUF ID to allow for PCIe peer-to-peer transfers with GPU video memory. This work will presumably play a role with Intel's forthcoming Xe HP/HPC offerings in more efficient transfers to/from RDMA devices.

The RDMA peer-to-peer support should yield better performance than going through system memory and avoiding excess copies. There is already P2PDMA and Heterogeneous Memory Management (HMM) within the kernel for dealing with similar use-cases while this new addition to the RDMA code for DMA-BUF importing using device memory should be a simpler alternative, doesn't require page structure, and more easily adapt to existing GPU drivers. The usage for now may be a bit limited though until the GPU drivers allow pinning device memory via the DMA-BUF interface, as Intel engineers noted in the original patches.

The RDMA pull for Linux 5.12 also includes the usual assortment of driver updates, code clean-up, support for more than 255 ports on an RDMA device, and other changes.