Intel QAT Zstd Plugin v0.1 Released For Speeding Up Zstandard Compression

[phoronix]

Phoronix: Intel QAT Zstd Plugin v0.1 Released For Speeding Up Zstandard Compression

As a big win for Intel's QuickAssist Technology (QAT) accelerator found as an option with Sapphire Rapids processors and prior QAT hardware, there's been a QAT Zstd plug-in to provide big performance/efficiency benefits. Version 0.1 of that plug-in was released today...

https://www.phoronix.com/news/Intel-QAT-Zstd-0.1

[ddriver]

Intel sure knows how to provide good clear data. What's better than showcasing performance advantages at completely different compression sittings.

Also, kudos on focusing on a proprietary hardware solution rather than developing something that can target generic compute interfaces.

[Anux]

Intel sure knows how to provide good clear data. What's better than showcasing performance advantages at completely different compression sittings.

Also, kudos on focusing on a proprietary hardware solution rather than developing something that can target generic compute interfaces.

Their levels are completely different from zstd because they use a different (obviously) implementation. The actual compression lies between level 4 and 5 of zstd.

I wonder if an FPGA is used for this feature. And didn't Zen4 have an integrated FPGA? Is there any documentation on how to use Zens FPGA under Linux, i'd like to get my hands wet, something like QAT would be a nice improvement if you use compression in the fs.

[skeevy420]

Their levels are completely different from zstd because they use a different (obviously) implementation. The actual compression lies between level 4 and 5 of zstd.

I wonder if an FPGA is used for this feature. And didn't Zen4 have an integrated FPGA? Is there any documentation on how to use Zens FPGA under Linux, i'd like to get my hands wet, something like QAT would be a nice improvement if you use compression in the fs.

Yeah, that shit threw me off the first time discussing this.

[ktecho]

kudos on focusing on a proprietary hardware solution rather than developing something that can target generic compute interfaces.

What? The entire point of this library is being able to use the accelerators that they bundle inside the CPU.

[markg85]

So this is:

1. Vendor locked to intel QAD.
2. different compression levels making it tricky at best to compare it against "vanilla zstd"
3. (assuming, as i think i read it in the previous post about this) not a compatible bitstream. Aka, "qad zstd" files won't open with "vanilla zstd".

I see like literally no reason why anyone would even want to use this. You'd be locking yourself in the intel platform. Good for intel, not for you.
Even if my above assumption (point 3) is wrong, it would still be intel-only and therefore ironically not scalable in terms of platforms.

So why does this exist?
As it's useless.

I'm guessing it only exists as a promotional means for intel QAD, not to actually be used.

[ddriver]

Their levels are completely different from zstd because they use a different (obviously) implementation. The actual compression lies between level 4 and 5 of zstd.

Your point being? The software version definitively supports compression level 9. And it is a safe bet that the accelerated version supports levels other than 9.

There is nothing preventing them from having charted test runs with the same compression level.

It is just stupid to showcase performance difference while doing vastly different workloads when they could have easily tested with the same input parameters.

I am not saying "deceitful" just "stupid" here. They are probably at a disadvantage in terms of showcasing acceleration here, because higher compression rates become increasingly slower.

But doing it as they did gives no clear idea on the performance boost people can expect. Its not informative., it appears to target the crowd that understands "compare two numbers" absent any "understanding of the numbers".

What? The entire point of this library is being able to use the accelerators that they bundle inside the CPU.

I though the point was to provide acceleration.

But also, imagine that zstd's point, rather than providing compression, was to provide compression exclusively on something proprietary to meta, now that wouldn't be overly useful to the world, now would it?

I presume it is not some hand written binary blob, they could demonstrate good spirit by releasing the verilog or vhdl or whatever they are using, so the acceleration can be ported to other hardware. Spirit of FOSS and all.

[abu_shawarib]

Your point being? The software version definitively supports compression level 9. And it is a safe bet that the accelerated version supports levels other than 9.

There is nothing preventing them from having charted test runs with the same compression level.

If the compression ratio is not the same, then that comparison is useless and misleading.

[patstew]

Your point being? The software version definitively supports compression level 9. And it is a safe bet that the accelerated version supports levels other than 9.

There is nothing preventing them from having charted test runs with the same compression level.

It is just stupid to showcase performance difference while doing vastly different workloads when they could have easily tested with the same input parameters.

I am not saying "deceitful" just "stupid" here. They are probably at a disadvantage in terms of showcasing acceleration here, because higher compression rates become increasingly slower.

But doing it as they did gives no clear idea on the performance boost people can expect. Its not informative., it appears to target the crowd that understands "compare two numbers" absent any "understanding of the numbers".

The point is facebook-ZSTD level 9 is not the same as QAT-ZSTD level 9. The levels do not have a specific meaning, it's just an implementatipon specific tradeoff between compression ratio and speed, and those settings give the closest match in compression ratio. You might as well complain that gzip and winzip give different results when set to some number.

[Anux]

There is nothing preventing them from having charted test runs with the same compression level.

Only that there is no same compression level.