Intel SGX Enclaves Were Prone To Crashes On Linux Under Heavy Memory Pressure

Intel's Software Guard Extensions (SGX) as security-related extensions to their processors that allow for protected memory enclaves has had a rather bouncy journey. Intel continues supporting SGX on their latest Xeon processors but on the client side have been deprecated since 11th Gen Core. Over the years SGX has been found vulnerable to various attacks from speculative execution exploits to Plundervolt. It also turns out under Linux until now was also open to crashing under memory pressure.

Queued up as part of the SGX changes for Linux 5.19 is addressing the possibility of the SGX support crashing when under heavy memory pressure. Dave Hansen of Intel explained in the SGX updates for v5.19:

A set of patches to prevent crashes in SGX enclaves under heavy memory pressure:

SGX uses normal RAM allocated from special shmem files as backing storage when it runs out of SGX memory (EPC). The code was overly aggressive when freeing shmem pages and was inadvertently freeing perfectly good data. This resulted in failures in the SGX instructions used to swap data back into SGX memory.

The "good" news is that it's difficult to trigger this behavior on the mainline Linux kernel and likely how the problem lasted so long. Intel noticed the issue when testing their latest out-of-tree patches for "SGX2" and then when investigating that discovered that the mainline code is also vulnerable albeit less likely to be encountered.

The SGX updates for Linux 5.19 fix this issue by being more careful about truncating pages out of the backing storage and the marking of dirty pages.