The Linux Kernel Preparing To Take Advantage Of The Intel DSA / ENQCMD In Sapphire Rapids

Expected with next year's Intel Sapphire Rapids Xeon CPUs is the Intel DSA as the Data Streaming Accelerator for high performance data movement and transformation operations. Since the end of 2019 there have been Linux patches surfacing for bringing up the DSA support and now as we roll into 2021 the Linux kernel looks to begin making use of the new capabilities.

With the Intel DSA block there are new instructions with ENQCMD/ENQCMDS for submitting work descriptors along with the PASID instruction for the Process Address Space ID. We've seen various Linux driver work around the Data Streaming Accelerator as well as supporting these new instructions in the open-source compiler.

Queued now as part of an x86/pasid branch for possible introduction with the Linux 5.10 cycle kicking off next week is supporting ENQCMD (or rather, ENQCMDS as the supervisor mode version) within the kernel.

This branch enumerates the ENQCMD/ENQCMDS instructions and related bits along with PASID. It culminates with adding an ENQCMDS wrapper around that new instruction.

That wrapper is intended to be picked up by other kernel code moving forward for handling work submission within the kernel when running on a DSA-enabled processor. Once the kernel begins making use of the new path on supported hardware, it should provide some measurable performance/efficiency benefits. As explained via that wrapper commit:

Currently, the MOVDIR64B instruction is used to atomically submit 64-byte work descriptors to devices. Although it can encounter errors like device queue full, command not accepted, device not ready, etc when writing to a device MMIO, MOVDIR64B can not report back on errors from the device itself. This means that MOVDIR64B users need to separately interact with a device to see if a descriptor was successfully queued, which slows down device interactions.

ENQCMD and ENQCMDS also atomically submit 64-byte work descriptors to devices. But, they *can* report back errors directly from the device, such as if the device was busy, or device not enabled or does not support the command. This immediate feedback from the submission instruction itself reduces the number of interactions with the device and can greatly increase efficiency.

This new documentation that is also part of the x86/PASID branch provides a high-level overview of the new Intel Shared Virtual Addressing (SVA) capabilities with ENQCMD.

Sapphire Rapids CPUs likely won't be out until around the end of 2021 so there still is several release cycles to go for the kernel to begin making use of ENQCMDS.