Benchmarking The Work-In-Progress Spectre/STIBP Code On The Way For Linux 4.20

Due to the performance fallout from the introduction of STIBP for cross-hyperthread Spectre Variant Two mitigation in the Linux 4.20 kernel, the stable Linux kernels are reverting those patches after originally being quickly back-ported to those branches. For Linux 4.20, STIBP on by default for all processes remain in place until the revised STIBP code is ready for merging that is still expected to happen before the stable Linux 4.20 debut in about one month's time. Here are some initial benchmarks of those preliminary improvements.

The improved STIBP heuristics have gone through several rounds of public review on the kernel mailing list in recent weeks. The patches are now queued in tip/tip.git's new WIP.x86/pti Git branch. With that work-in-progress code now in Thomas Gleixner's tree, I figured I would run some performance tests of that code to see if there was anything unexpected and to verify the restoration of the performance drops that could be easily in double digit losses.

Among the work carried out by the work-in-progress code includes:

- Disabling STIBP (Single Thread Indirect Branch Predictors) when enhanced IBRS (Indirect Branch Restricted Speculation) is used since otherwise it's redundant while only hurting performance.

- Better tracking and reporting when SMT (Simultaneous Multi Threading) is actually active.

- A new command-line option for controlling indirect branch speculation for user-space. The option is spectre_v2_user= with values of on/off/auto. The default auto behavior is unconditionally disabled for now.

- Most importantly for correcting the recent performance drop is supporting per-task indirect branch speculation control so STIBP can be controlled per-process rather than globally being enabled. As part of that, adding PR_SPEC_INDIRECT_BRANCH option for the PR_GET_SPECULATION_CTRL and PR_SET_SPECULATION_CTRL prctls for fine-grained controls of STIBP as well as IBPB. This also adds a spectre_v2_user=prctl option to the above kernel command line parameter. This prctl addition now allows processes that are potentially subject to untrusted code, offer sandboxing, or other cases where you would want to protect against a Spectre V2 cross-hyperthread attack, can opt into this protection.

- Enabling STIBP on all SECCOMP threads by default unless they explicitly opt-out. SECCOMP was already enabling SSBD (Speculative Store Bypass Disable) by default for Spectre V4. This makes sense with SECCOMP being for "secure computing" to already restrict system call access among other restrictions. SECCOMP is used (or optionally used) by Docker, VSFTPD, OpenSSH, Firefox, Flatpak, Snappy, QEMU, Chrome, LXD, and other security sensitive programs/daemons. I.e. applications/daemons with sandboxed processes or potentially untrusted user code are already taking extra measures with SECCOMP so it makes sense to apply STIBP.

The updated Spectre tunables are outlined by the kernel-parameters.txt. What's important though for end-users just trying to deal with their Linux desktop without regressing performance is the default change: rather than enabling STIBP everywhere, by default with this code the cross-hyperthread Spectre V2 mitigation is only applied to processes requesting it via the prctl function as well as to all SECCOMP threads.

From an Intel Core i9 7960X system I ran benchmarks of the Linux 4.19.0 kernel, Linux 4.20 Git as of today where STIBP is still applied for all processes, and then a fresh kernel build using the WIP.x86/pti tree. This is mostly to verify the performance of the benchmarks is restored and there being no unexpected surprises considering how STIBP landed in the first place without communicating the significant performance impact and a lot of the x86 speculation code being touched by these patches.