Linux Getting Driver Work To Support Tesla V100 NVLink GPUs On High-End POWER9 Servers
The V100 Volta GPUs on these POWER9 servers aren't just conventional PCIe cards plugged in but connected via NVLink and allow for coherent memory and NPU/ATS support on the POWER9 CPU. IBM has been leading the Linux kernel work to allow for the unmodified NVIDIA POWER driver to work on this hardware.
It's a set of 20 patches affecting the kernel's PowerPC and VFIO code for allowing this Tesla V100 GPU support to happen on the likes of the IBM POWER9 Witherspoon servers. The patch series goes on to explain in more technical detail:
POWER9 Witherspoon machines come with 4 or 6 V100 GPUs which are not pluggable PCIe devices but still have PCIe links which are used for config space and MMIO. In addition to that the GPUs have 6 NVLinks which are connected to other GPUs and the POWER9 CPU. POWER9 chips have a special unit on a die called an NPU which is an NVLink2 host bus adapter with p2p connections to 2 to 3 GPUs, 3 or 2 NVLinks to each. These systems also support ATS (address translation services) which is a part of the NVLink2 protocol. Such GPUs also share on-board RAM (16GB or 32GB) to the system via the same NVLink2 so a CPU has cache-coherent access to a GPU RAM.
This exports GPU RAM to the userspace as a new VFIO device region. This preregisters the new memory as device memory as it might be used for DMA. This inserts pfns from the fault handler as the GPU memory is not onlined until the vendor driver is loaded and trained the NVLinks so doing this earlier causes low level errors which we fence in the firmware so it does not hurt the host system but still better be avoided; for the same reason this does not map GPU RAM into the host kernel (usual thing for emulated access otherwise).
More details in the patches. While these high-end POWER9 + Volta HPC servers are out of reach for most, it's fun to see this hardware used by some of the leading supercomputers getting closer to actually working off a mainline kernel. Though as these patches are still under review, don't count on seeing the work merged for the imminent Linux 4.21 cycle.
3 Comments
