Announcement

Collapse
No announcement yet.

Updated Zstd Planned For Linux 5.16 With Better Performance

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #21
    Originally posted by sinepgib View Post

    Sorry, I assumed you were not using an initrd because you didn't mention it. I _think_ Debian uses a compressed one but with gzip, plus all possible modules by default rather than the minimum to boot. My current initramfs is about 7MiBs as well. But yeah, the image size is similar.
    Everything I've seen online suggests gzip, but in my 'config-5.14.0-1-amd64' I have 'CONFIG_KERNEL_XZ=y'. I don't remember changing it manually so that's a bit weird.

    Code:
    CONFIG_HAVE_KERNEL_GZIP=y
    CONFIG_HAVE_KERNEL_BZIP2=y
    CONFIG_HAVE_KERNEL_LZMA=y
    CONFIG_HAVE_KERNEL_XZ=y
    CONFIG_HAVE_KERNEL_LZO=y
    CONFIG_HAVE_KERNEL_LZ4=y
    CONFIG_HAVE_KERNEL_ZSTD=y
    # CONFIG_KERNEL_GZIP is not set
    # CONFIG_KERNEL_BZIP2 is not set
    # CONFIG_KERNEL_LZMA is not set
    CONFIG_KERNEL_XZ=y
    # CONFIG_KERNEL_LZO is not set
    # CONFIG_KERNEL_LZ4 is not set
    # CONFIG_KERNEL_ZSTD is not set
    Seen some talk on the mailing lists that they are considering moving to zstd. They too say they _think_ they are using gzip by default.

    Comment


    • #22
      Originally posted by Brisse View Post

      Everything I've seen online suggests gzip, but in my 'config-5.14.0-1-amd64' I have 'CONFIG_KERNEL_XZ=y'. I don't remember changing it manually so that's a bit weird.

      Code:
      CONFIG_KERNEL_XZ=y
      Seen some talk on the mailing lists that they are considering moving to zstd. They too say they _think_ they are using gzip by default.
      Note this line is about the kernel compression, not the initrd. They need not use the same algorithm.

      Comment


      • #23
        Originally posted by Brisse View Post
        Is the kernel images getting a lot bigger with each new release? I'm on Debian testing, and recently it pushed a 5.14 based kernel instead of the previous 5.10 and now that I got a second update to 5.14, it won't fit in my /boot partition without purging the old one first

        CONFIG_KERNEL_XZ=y
        df -h
        ...
        /dev/sda2 237M 95M 131M 42% /boot

        GNOME Disks says it's 256MiB total and 141MiB free, but whatever. That's after purging the old backup kernel and having just one left.
        Yes, default kernel images are getting bigger. This is actually not a drawback. It just means that default kernels ship with more drivers/functionality built in and/or optimizations turned on.

        However, I was also in a similar situation and was only able to upgrade the kernel by temporarily mounting more space to /boot (use the following command only if you understand the implication that if you mess it up, you won't have any working kernel on /boot, and know how to avoid it)
        Code:
        mount -t tmpfs none /boot
        I eventually conceded and re-installed the system with a 1GB /boot.
        I think this is now the default /boot size in Fedora 34 and Debian Bullseye anyway.
        Last edited by mppix; 05 October 2021, 08:32 PM.

        Comment


        • #24
          Originally posted by RedEyed View Post
          I use zstd to package my tool, with zstd it has better compress ratio and MUCH faster decompression speed (it's crucial during installation) compared to lz4
          Citation needed. lz4 basically runs at IO speed. zstd very much does not.
          Until your post, I've never seen anyone claim, anywhere, ever, that zstd is EVER faster than lz4, let alone "MUCH" so.

          Comment


          • #25
            Originally posted by Brisse View Post

            Everything I've seen online suggests gzip, but in my 'config-5.14.0-1-amd64' I have 'CONFIG_KERNEL_XZ=y'. I don't remember changing it manually so that's a bit weird.

            Code:
            CONFIG_HAVE_KERNEL_GZIP=y
            CONFIG_HAVE_KERNEL_BZIP2=y
            CONFIG_HAVE_KERNEL_LZMA=y
            CONFIG_HAVE_KERNEL_XZ=y
            CONFIG_HAVE_KERNEL_LZO=y
            CONFIG_HAVE_KERNEL_LZ4=y
            CONFIG_HAVE_KERNEL_ZSTD=y
            # CONFIG_KERNEL_GZIP is not set
            # CONFIG_KERNEL_BZIP2 is not set
            # CONFIG_KERNEL_LZMA is not set
            CONFIG_KERNEL_XZ=y
            # CONFIG_KERNEL_LZO is not set
            # CONFIG_KERNEL_LZ4 is not set
            # CONFIG_KERNEL_ZSTD is not set
            Seen some talk on the mailing lists that they are considering moving to zstd. They too say they _think_ they are using gzip by default.
            For initramfs You'd look in either /etc/initramfs-tools/initramfs.conf or /etc/initramfs-tools/conf.d/nameit.

            I'd bet You have COMPRESS=gzip (replacable with COMPRESS=xz). You can also play with MODULES setting (e.g. change most with dep).

            Comment


            • #26
              Originally posted by arQon View Post

              Citation needed. lz4 basically runs at IO speed. zstd very much does not.
              Until your post, I've never seen anyone claim, anywhere, ever, that zstd is EVER faster than lz4, let alone "MUCH" so.
              Fair note.

              In my case, I need high compression ratio and lz4 can't achieve it.

              In sake of your request I did benchmarks.
              You're partially right, lz4 decompression is a bit faster , but zstd wins in all other benchmarks

              Here are my results:


              My rig is:
              SSD: Samsung SSD 870 EVO 500 GB
              CPU: Intel Core I7-9800X 3.80 GHz
              Last edited by RedEyed; 06 October 2021, 05:49 AM.

              Comment


              • #27
                Originally posted by barti_ddu View Post

                For initramfs You'd look in either /etc/initramfs-tools/initramfs.conf or /etc/initramfs-tools/conf.d/nameit.

                I'd bet You have COMPRESS=gzip (replacable with COMPRESS=xz). You can also play with MODULES setting (e.g. change most with dep).
                Thank you. Can confirm it says COMPRESS=gzip. Not going to touch any of the settings for now but maybe I'll dive into the documentation a bit and then consider some changes.

                Edit:
                gzip, MODULES=most 73.6MiB
                gzip, MODULES=dep 42.0MiB
                xz, MODULES=dep 25.0MiB
                Last edited by Brisse; 06 October 2021, 07:18 AM.

                Comment


                • #28
                  Originally posted by Brisse View Post
                  gzip, MODULES=most 73.6MiB
                  gzip, MODULES=dep 42.0MiB
                  xz, MODULES=dep 25.0MiB
                  It looks better, but still like a lot of excessive modules (or firmware, or both) in the initrd. Mine is ~12M gzipped, 7.6M xzipped (stock Debian 5.14 kernel, I'm on testing too). Didn't do anything special, however, I've blacklisted fs that I don't use (like jfs, minix, etc.). Not sure if it does have impact on initrd though.

                  BTW, You can easily inspect the contents of initrd image in Debian with lsinitramfs:
                  Code:
                  # lsinitramfs -l /boot/initrd.img-5.14.0-1-amd64

                  Comment


                  • #29
                    Great news. Since there are 5.14 patches I'm hoping it'll get backported to 5.15.

                    On top of this, there are significant performance improvements coming down the line in the next zstd release, and the new automated update patch generation will allow us to pull them easily.
                    To me, that's the best part of this announcement. Performance improvements are great; not having to wait so long for them is better. AFAICT, the only thing BTRFS is missing over ZFS in regards to Zstd is fast support...which can equal LZ4 in speeds.

                    I don't know how to do this, but I was thinking and since Zstd basically decompresses just as fast regardless of level "compress-idle:X and compress-force-idle:X" mount options would be nice. Whenever the system is idle and doing nothing it would compress stuff to level X. We could use "compress-force:2,compress-force-idle:15" so the system is nice and fast when we're doing stuff and can compress higher when we're not.

                    Comment


                    • #30
                      Originally posted by arQon View Post

                      Citation needed. lz4 basically runs at IO speed. zstd very much does not.
                      Until your post, I've never seen anyone claim, anywhere, ever, that zstd is EVER faster than lz4, let alone "MUCH" so.
                      Note running at IO speed with a much larger file can mean actual decompression takes longer than one that doesn't match IO speed but leads to a much smaller file. At least that's the effect on my boot time with compressing the initramfs with zstd and lz4 on a slow computer (SSD mounted on SATA-II netting about 300MiB/s, Atom N270 CPU). At the end of the day, cat would be fastest to decompress if we left IO time out of the equation :shrug:

                      Comment

                      Working...
                      X