Announcement

Collapse
No announcement yet.

Intel QAT Zstd Plugin v0.1 Released For Speeding Up Zstandard Compression

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Intel QAT Zstd Plugin v0.1 Released For Speeding Up Zstandard Compression

    Phoronix: Intel QAT Zstd Plugin v0.1 Released For Speeding Up Zstandard Compression

    As a big win for Intel's QuickAssist Technology (QAT) accelerator found as an option with Sapphire Rapids processors and prior QAT hardware, there's been a QAT Zstd plug-in to provide big performance/efficiency benefits. Version 0.1 of that plug-in was released today...

    Phoronix, Linux Hardware Reviews, Linux hardware benchmarks, Linux server benchmarks, Linux benchmarking, Desktop Linux, Linux performance, Open Source graphics, Linux How To, Ubuntu benchmarks, Ubuntu hardware, Phoronix Test Suite

  • #2
    Intel sure knows how to provide good clear data. What's better than showcasing performance advantages at completely different compression sittings.

    Also, kudos on focusing on a proprietary hardware solution rather than developing something that can target generic compute interfaces.

    Comment


    • #3
      Originally posted by ddriver View Post
      Intel sure knows how to provide good clear data. What's better than showcasing performance advantages at completely different compression sittings.

      Also, kudos on focusing on a proprietary hardware solution rather than developing something that can target generic compute interfaces.
      Their levels are completely different from zstd because they use a different (obviously) implementation. The actual compression lies between level 4 and 5 of zstd.

      I wonder if an FPGA is used for this feature. And didn't Zen4 have an integrated FPGA? Is there any documentation on how to use Zens FPGA under Linux, i'd like to get my hands wet, something like QAT would be a nice improvement if you use compression in the fs.

      Comment


      • #4
        Originally posted by Anux View Post
        Their levels are completely different from zstd because they use a different (obviously) implementation. The actual compression lies between level 4 and 5 of zstd.

        I wonder if an FPGA is used for this feature. And didn't Zen4 have an integrated FPGA? Is there any documentation on how to use Zens FPGA under Linux, i'd like to get my hands wet, something like QAT would be a nice improvement if you use compression in the fs.
        Yeah, that shit threw me off the first time discussing this.

        Comment


        • #5
          Originally posted by ddriver View Post
          kudos on focusing on a proprietary hardware solution rather than developing something that can target generic compute interfaces.
          What? The entire point of this library is being able to use the accelerators that they bundle inside the CPU.

          Comment


          • #6
            So this is:
            1. Vendor locked to intel QAD.
            2. different compression levels making it tricky at best to compare it against "vanilla zstd"
            3. (assuming, as i think i read it in the previous post about this) not a compatible bitstream. Aka, "qad zstd" files won't open with "vanilla zstd".
            I see like literally no reason why anyone would even want to use this. You'd be locking yourself in the intel platform. Good for intel, not for you.
            Even if my above assumption (point 3) is wrong, it would still be intel-only and therefore ironically not scalable in terms of platforms.

            So why does this exist?
            As it's useless.

            I'm guessing it only exists as a promotional means for intel QAD, not to actually be used.

            Comment


            • #7
              Originally posted by Anux View Post
              Their levels are completely different from zstd because they use a different (obviously) implementation. The actual compression lies between level 4 and 5 of zstd.
              Your point being? The software version definitively supports compression level 9. And it is a safe bet that the accelerated version supports levels other than 9.

              There is nothing preventing them from having charted test runs with the same compression level.

              It is just stupid to showcase performance difference while doing vastly different workloads when they could have easily tested with the same input parameters.

              I am not saying "deceitful" just "stupid" here. They are probably at a disadvantage in terms of showcasing acceleration here, because higher compression rates become increasingly slower.

              But doing it as they did gives no clear idea on the performance boost people can expect. Its not informative., it appears to target the crowd that understands "compare two numbers" absent any "understanding of the numbers".

              Originally posted by ktecho View Post

              What? The entire point of this library is being able to use the accelerators that they bundle inside the CPU.
              ‚Äč

              I though the point was to provide acceleration.

              But also, imagine that zstd's point, rather than providing compression, was to provide compression exclusively on something proprietary to meta, now that wouldn't be overly useful to the world, now would it?

              I presume it is not some hand written binary blob, they could demonstrate good spirit by releasing the verilog or vhdl or whatever they are using, so the acceleration can be ported to other hardware. Spirit of FOSS and all.
              Last edited by ddriver; 08 September 2023, 12:12 PM.

              Comment


              • #8
                Originally posted by ddriver View Post

                Your point being? The software version definitively supports compression level 9. And it is a safe bet that the accelerated version supports levels other than 9.

                There is nothing preventing them from having charted test runs with the same compression level.
                If the compression ratio is not the same, then that comparison is useless and misleading.

                Comment


                • #9
                  Originally posted by ddriver View Post

                  Your point being? The software version definitively supports compression level 9. And it is a safe bet that the accelerated version supports levels other than 9.

                  There is nothing preventing them from having charted test runs with the same compression level.

                  It is just stupid to showcase performance difference while doing vastly different workloads when they could have easily tested with the same input parameters.

                  I am not saying "deceitful" just "stupid" here. They are probably at a disadvantage in terms of showcasing acceleration here, because higher compression rates become increasingly slower.

                  But doing it as they did gives no clear idea on the performance boost people can expect. Its not informative., it appears to target the crowd that understands "compare two numbers" absent any "understanding of the numbers".
                  The point is facebook-ZSTD level 9 is not the same as QAT-ZSTD level 9. The levels do not have a specific meaning, it's just an implementatipon specific tradeoff between compression ratio and speed, and those settings give the closest match in compression ratio. You might as well complain that gzip and winzip give different results when set to some number.

                  Comment


                  • #10
                    Originally posted by ddriver View Post
                    There is nothing preventing them from having charted test runs with the same compression level.
                    Only that there is no same compression level.

                    Comment

                    Working...
                    X