Linux Disables Intel's New ENQCMD Feature Since Kernel Code "Broken Beyond Repair"

For two years now we have seen various open-source patches around ENQCMD instructions for the code compilers and the Linux kernel. This Enqueue Stores support is part of Intel's Data Streaming Accelerator coming with Xeon Scalable "Sapphire Rapids". But even with Sapphire Rapids not launching until later this year, the already merged Linux kernel code is force-disabling the ENQCMD instructions support as the current kernel support is deemed "broken beyond repair" and will need to be reworked.

In recent months the Linux kernel bits have been coming together for Intel's DSA (Data Streaming Accelerator) and handling of the ENQCMD instructions along with related kernel work like on the XSAVES supervisor states.

ENQCMD allows atomically submitting a work descriptor to a device and further spelled out via the kernel documentation and Intel's PRM.

While the code has been part of the mainline kernel, sent in today as part of x86/urgent fixes for Linux 5.13 is now force-disabling of the ENQCMD/ENQCMDS feature code in the kernel.

The patch force-disabling the ENQCMD feature support explains, "While digesting the XSAVE-related horrors which got introduced with the supervisor/user split, the recent addition of ENQCMD-related functionality got on the radar and turned out to be similarly broken... [When invoked via a SMP function call from the IOMMU/SMV code] is broken beyond repair. The comment in the code claims that it is safe to invoke this in an IPI, but that's just wishful thinking...The offending commit does not cleanly revert, but it's sufficient to force disable X86_FEATURE_ENQCMD and to remove the broken update_pasid() code to make this dysfunctional all over the place. Anything more complex would require more surgery and none of the related functions outside of the x86 core code are blatantly wrong, so removing those would be overkill. As nothing enables the PASID bit in the IA32_XSS MSR yet, which is required to make this actually work, this cannot result in a regression except for related out of tree train-wrecks, but they are broken already today."

The patch sent in as part of the x86/urgent fixes for Linux 5.13 summed it up as, "Force-disable ENQCMD[S] instructions support and remove update_pasid() because of insufficient protection against FPU state modification in an interrupt context, among other xstate horrors which are being addressed at the moment. This one limits the fallout until proper enablement."

It's important to note that here the issue is with the kernel (software) code being broken around ENQCMD/ENQCMDS rather than the yet to be released hardware. The code submitted by Intel for enabling the support turned out to be not up to scratch while having already worked its way into the mainline kernel and now needing to be disabled until it can be reworked.

We'll see how long it takes before the kernel code is reworked that the ENQCMD feature is re-enabled. Sapphire Rapids is reported to be shipping later this calendar year but given the kernel release cadence and new feature code being limited to the merge window periods, we'll see how the stars align if the Data Streaming Accelerator support will be ready to go at-launch with the mainline kernel or if now perhaps there will be some delays with Intel's normally first-rate Linux support at launch.

The code comment disabling ENQCMD simply puts it as "Force disable because it's broken beyond repair." This was sent in as a patch for the current Linux 5.13 merge window but has also been copied for stable, so will also be disabled for current Linux kernel stable series too in the coming days.