Intel Preparing New "Staging" Feature For Better Handling CPU Microcode Updates

Intel is preparing a new staging feature for better handling of CPU microcode updates. Initial patches for the Linux kernel are out for discussion in enabling this feature that can yield around a 40% reduction in latency during the CPU microcode updating process.

This Intel staging microcode feature is intended to address the increasing latency spikes that happen when late-loading CPU microcode on running systems. Due to the complexities involved in applying CPU microcode updates on a running system and needing to temporarily stop cores and then invalidate/flushing caches, it can negatively impact running workloads. Late-loading CPU microcode can be considered "dangerous" but in the name of reducing system downtime, it's still a practice particularly on servers.

Sent out today from Intel Linux engineers is enabling a new "Intel Staging" feature. The staging feature allows for processing more of the CPU microcode update without bringing down CPU cores until hitting the actual activation process. Rather than loading, validating, and activating all in one-go that adds to the latency for CPU halts, the staging feature moves everything but the activation step out of the critical path.

The RFC patches on this Intel staging microcode update explain:

"As microcode images have increased in size, a corresponding rise in load latency has become inevitable. This latency spike significantly impacts late loading, which remains in use despite the cautions highlighted in the documentation. The issue is especially critical for continuously running workloads and virtual machines, where excessive delays can lead to timeouts.

== Staging for Latency Reduction ==

Currently, writing to MSR_IA32_UCODE_WRITE triggers the entire update process -- loading, validating, and activation -- all of which contribute to the latency during CPU halt. The staging feature mitigates this by refactoring all but the activation step out of the critical path, allowing CPUs to continue serving workloads while staging takes place."

This Intel staging feature isn't a pure software solution but requires support on the part of the processor. Intel engineers report preliminary testing on a "pre-production system" showed around a 40% reduction in latency during the CPU microcode late-loading process. There is also room for further performance optimizations with this Intel staging feature.

The developers didn't elaborate what pre-production platform was used for this Intel staging feature testing, but given the timing and other ongoing Linux enablement work, my bet would be on this feature premiering next year with Intel Xeon Diamond Rapids servers.