Experimental Linux Patches Allow User-Space Peer-To-Peer DMA Between NVMe Drives

[phoronix]

Phoronix: Experimental Linux Patches Allow User-Space Peer-To-Peer DMA Between NVMe Drives

A set of Linux kernel patches posted on friday allow peer-to-peer DMA (P2PDMA) transfers between NVMe drives using existing O_DIRECT operations or the NVMe pass-through interface from user-space...

http://www.phoronix.com/scan.php?page=news_item&px=Linux-RFC-P2PDMA-NVMe-Drives

[Zan Lynx]

Pretty sweet.

Can the code for NVMe direct to GPU be far behind? I've already seen code implementations for Linux but they were hacky.

Then of course we'd need 3D libraries to support textures and geometry on NVMe, automatic MIPMAP generation and automatic triangle mesh downsampling. Then we'd have all the parts to work just like a Playstation 5.

[torsionbar28]

A set of Linux kernel patches posted on friday allow peer-to-peer DMA (P2PDMA) transfers between NVMe drives

This seems like more of an enterprise use case, I don't see this benefiting end users much or at all. Even moving bulk data between two PCIe Gen 4 SSD's, my Ryzen 5 3600 doesn't seem to be breaking a sweat.

[Zan Lynx]

This seems like more of an enterprise use case, I don't see this benefiting end users much or at all. Even moving bulk data between two PCIe Gen 4 SSD's, my Ryzen 5 3600 doesn't seem to be breaking a sweat.

With several drives going all out, your system can run out of RAM bandwidth pretty easily. But yeah, probably more of an issue for big EPYC systems with several of those "ruler" SSDs.

[kylew77]

What is this for? Does this help software raid performance to be higher when using NVMe drives?

[torsionbar28]

What is this for? Does this help software raid performance to be higher when using NVMe drives?

P2P DMA transfers means data moving from one NVMe drive directly to another NVMe drive without first going through host memory. As for software RAID, no, data is not transfered between drives in a RAID array, except during a rebuild.

[CochainComplex]

If I remember correctly once Linus from Linus Tech tips has build a storage server which bottlenecked the epyc bandwidth. AFAIK he also struggled with Linux - in a server use case this might have ended in performance issues drops. But it was worst case situation to really push the epic. I took them some very FAST raid arrays of NVMe. This is some vuage remembering - take this info with some grain of salt. Most likely it's about such situations.

Edit: it was this one https://youtu.be/xWjOh0Ph8uM

[Ladis]

Direct DMA transfer between devices, avoiding copying into RAM and talking to CPU, is used also in the new consoles so the GPU can stream graphics data faster and without slowing the CPU. E.g. Sony says the game will need only data for 1 second of travelling through the game instead of 10 seconds what we have on the current consoles and PC (that also means you will need significantly more VRAM on PC than on the new consoles to get their performace at that visual quality - that's why RTX 3070 with its 8 GB RAM is DOA). PC will receive this feature (MS calls it DirectStorage), but it will take months to get from beta to the final version of Windows 10, only newest cards will support it, and only few games will support it (as 99% of PC gamers will not have the hardware for that).

PS: This DirectStorage feature is also for GPU compute servers as some types of computations work on a large set of data and you want to connect many GPUs to one server CPU (from my brother working there, all new GPGPU servers use EPYC as the I/O chiplet doing this job is one root domain and Intel Xeons are divided in multiple root domains - means you can't have a shared NVMe drive for all GPUs; the previous solution was a card working as the I/O for GPUs and NVMe, but that has a slow connection to CPU for modern usage).

[kylew77]

P2P DMA transfers means data moving from one NVMe drive directly to another NVMe drive without first going through host memory. As for software RAID, no, data is not transfered between drives in a RAID array, except during a rebuild.

Thank you kindly for that explanation.