An Early Look At The L1 Terminal Fault "L1TF" Performance Impact On Virtual Machines

Written by Michael Larabel in Software on 15 August 2018 at 09:45 AM EDT. Page 1 of 3. 15 Comments.

Yesterday the latest speculative execution vulnerability was disclosed that was akin to Meltdown and is dubbed the L1 Terminal Fault, or "L1TF" for short. Here are some very early benchmarks of the performance impact of the L1TF mitigation on the Linux virtual machine performance when testing the various levels of mitigation as well as the unpatched system performance prior to this vulnerability coming to light.

As soon as the L1TF patches hit the Linux kernel Git tree, which ended up being a few minutes prior to the embargoed organizations from publishing their information on this speculative execution side channel attack, I was already building out the latest Linux kernel Git code for running some benchmarks. These are just the start of the tests while benchmarks on other systems are still being carried out and being revised as I continue researching more information on L1TF.

With having started the kernel build and testing quickly upon finding out about the L1 Terminal Fault, my testing began with an Intel Core i7 8086K given it was in a rack under my desk and available for testing. With L1TF's exposure really being to virtual machines, testing Intel Xeon obviously makes more sense and those are among the systems now being tested under Linux with the latest mitigation techniques. But I continued on testing the i7-8086K anyhow from the position as what home/workstation users might see who are firing up their own virtual machines. Granted, in that scenario with likely not having untrusted users on those virtual machines, the exposure in the real-world to being affected by L1TF is quite small. Anyhow, Intel Xeon tests are currently being done and this is just some preliminary performance data.

Prior to the virtual machine tests, I did run some basic benchmarks on this system from the bare metal / host state just to verify the default mitigation now applied to the mainline Linux kernel did not affect the performance. Indeed, there were no unexpected changes of some quick Linux 4.18 to Linux 4.19 Git kernel performance on this i7-8086K box with Intel 120GB NVMe SSD and 16GB of RAM with the presence of PTE Inversion as a basic litmus test.

So with that it was onwards to the VM testing to see the performance impact. The scenarios tested for this initial look at the L1TF mitigated performance included:

- The unpatched/prior performance on this system/VM without any mitigation for the L1 Terminal Fault while obviously still having KPTI, __user pointer sanitization, and full generic Retpoline IBPB IBRS_FW and SSBD. None of those other Spectre/Meltdown mitigation features were changed throughout this testing but just looking at the L1TF impact.

- The default/out-of-the-box performance on the latest kernel with L1TF mitigation. The default behavior is l1tf=flush whereby SMT/HT is still left enabled but for hypervisor mitigation it's set to conditional L1D flushing. The conditional L1D flushing is done on VMENTER only when there is code between VMEXIT and VMENTER that could lead host memory.

- Sticking to the default (l1tf=flush) control mitigation but now forcing the L1D cache flush on every VMENTER rather than being done conditionally. This change can be done from the kernel command-line with the kvm-intel.vmentry_l1d_flush=always parameter.

- Testing while the host's kernel had l1tf=full applied for providing all available mitigations. This full protection includes disabling SMT (Hyper Threading) and unconditional L1D flushing. SMT/HT though should really only need to be disabled by this full protect if running untrusted guests/VMs on the system as obviously disabling SMT/HT will lead to a more profound performance impact.

More information on the L1 Terminal Fault tunables for the Linux kernel can be found via the parameter documentation.

During this initial testing, the KVM virtual machine had 10 vCPUs on this i7-8086K system and access to 8GB of the system's 16GB of DDR4-3000 memory. There were no other active processes or VMs running on this system during the benchmarking process with Ubuntu 18.04.1 LTS on both the host and guest VMs.

This first round (of likely several rounds) of L1TF Linux benchmarking was carried out in a fully-automated and reproducible manner using the open-source Phoronix Test Suite benchmarking software.

Update: To note, no microcode changes/updates were made to the systems under test for this article, just testing/comparing the kernel patches.

Related Articles