Intel QAT Zstd Plugin v0.1 Released For Speeding Up Zstandard Compression

[stormcrow]

So this is:

1. Vendor locked to intel QAD.
2. different compression levels making it tricky at best to compare it against "vanilla zstd"
3. (assuming, as i think i read it in the previous post about this) not a compatible bitstream. Aka, "qad zstd" files won't open with "vanilla zstd".

I see like literally no reason why anyone would even want to use this. You'd be locking yourself in the intel platform. Good for intel, not for you.
Even if my above assumption (point 3) is wrong, it would still be intel-only and therefore ironically not scalable in terms of platforms.

So why does this exist?
As it's useless.

I'm guessing it only exists as a promotional means for intel QAD, not to actually be used.

None of these are strikes against Intel because the people that are likely to use this already only use Intel CPUs, it's just a question whether or not their servers have a QAT coprocessor or not. The people developing this aren't going to be using it to serve websites anyway. It'll be used for internal compression.

Sorry to break it to you, but the corporate world has long been a largely Intel only shop with a smaller few using AMD. Clients and IoT are more diverse, but the server world is Intel dominated.

[geerge]

Possibly they've chosen level naming based on the similar ranges of parameters they work with as a search space. Meaning intel level 9 may more selectively/smartly search what standard level 9 does, but has comparable compression to standard level 5 (which is why it's the comparison) at a quicker pace.

Or the level naming is completely arbitrary. Either way the compression levels could be defined as letters of the greek alphabet for all that it matters, git v1.0 is incomparable to ls v1.0 is incomparable to firefox v1.0.

[markg85]

None of these are strikes against Intel because the people that are likely to use this already only use Intel CPUs, it's just a question whether or not their servers have a QAT coprocessor or not. The people developing this aren't going to be using it to serve websites anyway. It'll be used for internal compression.

Sorry to break it to you, but the corporate world has long been a largely Intel only shop with a smaller few using AMD. Clients and IoT are more diverse, but the server world is Intel dominated.

Perhaps my more liberate views of using open source and these days using AMD too cloud my view of the corporate intel dominance. To me it feels dangerous to use something that is locked to a certain vendor. You're left at the mercy of that vendor and are in trouble when that vendor decides to change things around (like ex. increasing prices massively or just dropping the thing you were using).

[geerge]

Perhaps my more liberate views of using open source and these days using AMD too cloud my view of the corporate intel dominance. To me it feels dangerous to use something that is locked to a certain vendor. You're left at the mercy of that vendor and are in trouble when that vendor decides to change things around (like ex. increasing prices massively or just dropping the thing you were using).

They are complying with an open standard spec aka zstd, so what's the problem?

[partcyborg]

Sorry to break it to you, but the corporate world has long been a largely Intel only shop with a smaller few using AMD. Clients and IoT are more diverse, but the server world is Intel dominated.

This isn't even remotely true. What do you work for Intel? Otherwise quit with the corporate shilling

[mangeek]

So why does this exist?
As it's useless.

I have thousands of servers generating terabytes of highly-compressible logs. If I can reduce the storage impact by putting part of the CPU that's not used for anything else to work (the servers aren't using QAT for much else), then this could be a huge win, to the tune of tens of thousands of dollars a year. Plus, those compressed logs will be much faster and less impactful on the I/O fabric to retrieve. Saving I/O on logging means everything gets faster.

Also, looks like the streams are accessible without needing QAT hardware, there's no lock-in, just a benefit for the architecture. That sort of thing has been around forever, and I remember people like you complaining about MMX, AltiVec, and 3DNow. Why wouldn't I want Quake to run faster because it can take advantage of some hardware accelerators on my CPU?