ZFS On Linux Has Figured Out A Way To Restore SIMD Support On Linux 5.0+

Those running ZFS On Linux (ZoL) on post-5.0 (and pre-5.0 supported LTS releases) have seen big performance hits to the ZFS encryption performance in particular. That came due to upstream breaking an interface used by ZFS On Linux and admittedly not caring about ZoL due to it being an out-of-tree user. But now several kernel releases later, a workaround has been devised.

Some Linux distributions have resorted to reverting the kernel patch that stopped exporting the kernel FPU begin/restore functions used by ZoL for tapping vector-based (SSE/AVX) algorithms. But now ZFS On Linux itself has figured out a solution to restore said SIMD support on these recent kernel releases.

With this ZoL commit made last week, SIMD compatibility should be working now on recent kernels and thus restoring the better performance.

Restore the SIMD optimization for 4.19.38 LTS, 4.14.120 LTS, and 5.0 and newer kernels. This is accomplished by leveraging the fact that by definition dedicated kernel threads never need to concern themselves with saving and restoring the user FPU state. Therefore, they may use the FPU as long as we can guarantee user tasks always restore their FPU state before context switching back to user space.

For the 5.0 and 5.1 kernels disabling preemption and local interrupts is sufficient to allow the FPU to be used. All non-kernel threads will restore the preserved user FPU state.

For 5.2 and latter kernels the user FPU state restoration will be skipped if the kernel determines the registers have not changed. Therefore, for these kernels we need to perform the additional step of saving and restoring the FPU registers. Invalidating the per-cpu global tracking the FPU state would force a restore but that functionality is private to the core x86 FPU implementation and unavailable.

In practice, restricting SIMD to kernel threads is not a major restriction for ZFS. The vast majority of SIMD operations are already performed by the IO pipeline. The remaining cases are relatively infrequent and can be handled by the generic code without significant impact.

I'll work to have out some new ZFS On Linux benchmarks soon.