Intel QAT Adapted For Zstd To Provide Big Performance/Efficiency Wins

While Intel has maintained the QATzip open-source compression library for demonstrating data compression using QuickAssist Technology (QAT) with DEFLATE/LZ4/LZ4s, Intel has also been working on QAT'ed Zstd for achieving some sizable victories in performance and power efficiency.

On Wednesday Intel announced their work on supporting Zstandard compression with QuickAssist Technology. This QAT integration for Zstd builds off the Zstd 1.5.4 support for allowing an external implementation of a sequence producer to be inserted into the Zstd pipeline.

Zstd with the Intel QAT plug-in can achieve some astounding victories according to Intel's numbers:

"This enables the utilization of Intel® QuickAssist Technology (Intel® QAT), which can deliver up to 3.2x better throughput, 3.8x reduction in P99 latency, and 3.3x better performance per watt when compared to zstd for compression. With these improvements, it's expected that Intel QAT will open new breakthrough use cases where compression can now be leveraged for workloads where it would not have been feasible previously.

Intel QAT will be an external sequence producer for zstd, improving performance while exposing the functionality to applications through the familiar zstd interface."

Intel also shared this slide, among others, but do note the differing in Zstd compression levels with their benchmarks:

Currently this QAT Zstd support is maintained via the QAT-ZSTD-Plugin code on GitHub.

Intel provided more details on this QAT Zstd integration via this Intel.com community post. Being a big fan of Zstd and with the enormous traction around this compression algorithm in the open-source community, I look forward to running some QAT-Zstd benchmarks on Sapphire Rapids... Stay tuned as time allows over the next few weeks.