Announcement

Collapse
No announcement yet.

A ZSTD-Compressed Linux Kernel Could Be Up Next

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • A ZSTD-Compressed Linux Kernel Could Be Up Next

    Phoronix: A ZSTD-Compressed Linux Kernel Could Be Up Next

    The Facebook-developed Zstd compression algorithm was added to the Linux 4.14 kernel and hooked up for Btrfs/Squashfs file-system compression. Support is now being worked on so Zstandard can be used for compressing/decompressing the Linux kernel image...

    http://www.phoronix.com/scan.php?pag...x-Kernel-Image

  • #2
    I think this already covers the topic
    https://events.linuxfoundation.org/s...ojp13_klee.pdf

    Comment


    • #3
      I use xz compressed kernels and mine are 6.3MB with all firmware compiled in. I don't have any modules, just the bits I need compiled in.

      Comment


      • #4
        It's going to be another option barely anyone will choose. An XZ compressed kernel decompresses within milliseconds on modern x86.

        They should drop BZIP2 from it as it's likely the least used compression algorithm throughout the kernel anyway. LZO could have been dropped with LZ4, because there is barely any difference between the two and LZ4 being the faster at decompression. LZ4 itself can be dropped with ZSTD and so could GZIP. Then it makes sense to have ZSTD. It's only bloating the kernel options now. The least they should do is to declare some of the options as obsolete.
        Last edited by sdack; 10-12-2017, 06:28 AM.

        Comment


        • #5
          Originally posted by caligula View Post
          I think this already covers the topic
          https://events.linuxfoundation.org/s...ojp13_klee.pdf
          I thought a while about the use of LZ4, but the speed gain is really marginal. First of all does the speed become irrelevant the faster a CPU is. And for a small, embedded device can the size of a kernel image be of more importance than it's boot-up speed. A device then doesn't have to boot every time you use it, but can also make use of hibernate and suspend functions to avoid the entire boot process. So this whole argument around LZ4 for the kernel image seems to be more theoretical than it's actually practical.
          Last edited by sdack; 10-11-2017, 11:41 AM.

          Comment


          • #6
            Originally posted by sdack View Post
            It's going to be another option barely anyone will choose. An XZ compressed kernel decompresses within milliseconds on modern x86.

            They should drop BZIP2 from it as it's likely the least used compression algorithm throughout the kernel anyway. LZO could have been dropped with LZ4, because there is barely any difference between the two and LZ4 being the faster at decompression. LZ4 itself can be dropped with ZSTD and so could GZIP. Then it make sense to have ZSTD. It's only bloating the kernel options now. The least they should do is to declare some of the options as obsolete.
            I would say it doesn't completely remove the need for LZ4. Those decompression speeds are still > 3x faster than ZSTD. Otherwise agreed.

            The results for this algorithm are very impressive. Given that BTRFS still only supports lzo and zlib, I can't wait to have transparent access to this algorithm. I'll need to run a long balance operation on all my drives though!

            Comment


            • #7
              Originally posted by thelongdivider View Post
              I would say it doesn't completely remove the need for LZ4. Those decompression speeds are still > 3x faster than ZSTD.
              AFAIK can ZSTD be used as a replacement for LZ4 and features about 22 different compression settings. As such would it be nice to be able to choose the setting with the algorithm.

              I do agree however that there will be people who find it more attractive than XZ, but I doubt the kernel image will grow as fast in size as CPUs grow in speed and thus will it only be a matter of time until everyone wants to use XZ and at it's highest settings (which I don't think is currently the default for when one chooses XZ for kernel image and the initramdisk).
              Last edited by sdack; 10-11-2017, 11:49 AM.

              Comment


              • #8
                Originally posted by sdack View Post
                It's going to be another option barely anyone will choose. An XZ compressed kernel decompresses within milliseconds on modern x86.
                Depends. It's mostly relevant to very old hardware, but OpenWrt\LEDE partially dropped 4mb EEPROM support in multiple devices due to size constraints not letting LUCI (the web-ui) to fit in. So, a better compression for the kernel just might make the difference. Also, just speculating here, but the compression used for the filesystem by the kernel might also be determined by the same code so that could really help. Well, assuming it's the same code.

                Comment


                • #9
                  Does anyone compress the kernel with zoplfi? Seems like you could get at least a few extra percent with maybe a few hundred iterations.

                  As for a way to test decompression speed, it's kinda two numbers that matter isn't it?
                  1) How much time is spent reading the kernel from the non-volatile storage?
                  2) How much time is spent decompressing the kernel?

                  Add these two together and you get a total score on one machine, use them independently to scale your results to machines with different CPU speeds or different storage speeds.

                  Comment


                  • #10
                    Originally posted by c117152 View Post
                    So, a better compression for the kernel just might make the difference.
                    ZSTD compresses worse than XZ at higher levels, and for kernel images in embedded you want best compression, period. Even a total shit CPU is going to decompress it in a few seconds, and if it gives you more space for fitting the firmware, then it's worth it.

                    Comment

                    Working...
                    X