Updated Zstd Planned For Linux 5.16 With Better Performance

[ms178]

This patchset is one of those examples where I ask myself why upstreaming takes so long. Lack of interest, lack of reviewers, technical reasons - or all of it combined?

[sinepgib]

This patchset is one of those examples where I ask myself why upstreaming takes so long. Lack of interest, lack of reviewers, technical reasons - or all of it combined?

Lack of reviewers and dependence of upstream zstandard on the licb (it uses malloc internally) that makes updating it involved. There were plans to extract that and allow custom allocators, but AFAIK it's not yet finished.

[Brisse]

Is the kernel images getting a lot bigger with each new release? I'm on Debian testing, and recently it pushed a 5.14 based kernel instead of the previous 5.10 and now that I got a second update to 5.14, it won't fit in my /boot partition without purging the old one first

CONFIG_KERNEL_XZ=y
df -h
...
/dev/sda2 237M 95M 131M 42% /boot

GNOME Disks says it's 256MiB total and 141MiB free, but whatever. That's after purging the old backup kernel and having just one left.

[RedEyed]

Fingers crossed. zstd has been a game changer for me personally, and would love to see it get better

Yes, same to me.
I use zstd to package my tool, with zstd it has better compress ratio and MUCH faster decompression speed (it's crucial during installation) compared to lz4

UPDATE 1:
decompression speed similar to lz4

[molletts]

df -h
...
/dev/sda2 237M 95M 131M 42% /boot

GNOME Disks says it's 256MiB total and 141MiB free, but whatever. That's after purging the old backup kernel and having just one left.

Wow, I was just thinking that my kernel images were getting a little bloated.

$ df -h
...
/dev/sda1                  15M   12M  2.0M  86% /boot
total 9695
lrwxrwxrwx 1 root root       1 Oct 23 2010 boot -> .
drwxr-xr-x 5 root root    1024 Jun 8 2016 grub
-rw-r--r-- 1 root root 3198416 Aug 19 19:30 kernel-5.13.11-gentoo
-rw-r--r-- 1 root root 3313360 Sep 16 18:25 kernel-5.14.5-gentoo
-rw-r--r-- 1 root root 3401072 Oct 4 10:20 kernel-5.14.9-gentoo
drwx------ 2 root root   12288 Oct 23 2010 lost+found
$ ls -l /boot

They have got a little bigger with time, but the difference between the 5.14.5 and 5.14.9 ones is purely because I added zstd support to it try out with f2fs on a small USB device. I'll double the size of /boot when I next move the installation to new storage. I'm already using 31MB (the odd MB gets the second partition onto a nice OCD-pleasing power-of-two sector number) on more recent builds.

[RedEyed]

Wow, I was just thinking that my kernel images were getting a little bloated.

$ df -h
...
/dev/sda1 15M 12M 2.0M 86% /boot
total 9695
lrwxrwxrwx 1 root root 1 Oct 23 2010 boot -> .
drwxr-xr-x 5 root root 1024 Jun 8 2016 grub
-rw-r--r-- 1 root root 3198416 Aug 19 19:30 kernel-5.13.11-gentoo
-rw-r--r-- 1 root root 3313360 Sep 16 18:25 kernel-5.14.5-gentoo
-rw-r--r-- 1 root root 3401072 Oct 4 10:20 kernel-5.14.9-gentoo
drwx------ 2 root root 12288 Oct 23 2010 lost+found
$ ls -l /boot

They have got a little bigger with time, but the difference between the 5.14.5 and 5.14.9 ones is purely because I added zstd support to it try out with f2fs on a small USB device. I'll double the size of /boot when I next move the installation to new storage. I'm already using 31MB (the odd MB gets the second partition onto a nice OCD-pleasing power-of-two sector number) on more recent builds.

Always used 1GB of boot part (on desktop/servers not IoT)

[Brisse]

Always used 1GB of boot part (on desktop/servers not IoT)

Yeah, I wonder if 256MiB is getting too cramped. This Debian install was originally Debian 9 which I reconfigured to follow unstable for many years, now testing after the recent Debian 11 release. Out of curiosity I checked a laptop with a recent install of vanilla Debian 11 and it's still the exact same partition layout. Also checked a VM with Fedora 34, and they are using 1,1GiB as default.

[sinepgib]

I'm on Debian testing

Seeing on Arch the default image is ~7MiB, I'd say Debian testing is carrying _a lot_ of debug info in the image. Or maybe even packing the headers inside the image, which is an option.

[Brisse]

Seeing on Arch the default image is ~7MiB, I'd say Debian testing is carrying _a lot_ of debug info in the image. Or maybe even packing the headers inside the image, which is an option.

Yes, I figured from molletts post that Gentoo (in their case) clearly does something very differently from Debian and that we are probably comparing apples and oranges.

$ ls /boot
config-5.14.0-1-amd64
initrd.img-5.14.0-1-amd64    #73.6MiB
vmlinuz-5.14.0-1-amd64    #7.2MiB
efi lost+found
grub System.map-5.14.0-1-amd64

I added respective file sizes behind comments (#) on the files that matter. The others are minuscule in comparison. I'm no expert on how this works specifically, but I'm guessing the initrd* one is an uncompressed image, while the vmlinuz* one is the compressed image. Notice it's size is similar to what you're seeing on Arch. I'm going to go out on a limb here, and guess that Arch and Gentoo stores the uncompressed image somewhere else.

Edit: Reading on Wikipedia that it is sometimes stored in

/boot

and other times in

/

Yay, I learned something today

[sinepgib]

Yes, I figured from molletts post that Gentoo (in their case) clearly does something very differently from Debian and that we are probably comparing apples and oranges.

$ ls /boot
config-5.14.0-1-amd64
initrd.img-5.14.0-1-amd64 #73.6MiB
vmlinuz-5.14.0-1-amd64 #7.2MiB
efi lost+found
grub System.map-5.14.0-1-amd64

I added respective file sizes behind comments (#) on the files that matter. The others are minuscule in comparison. I'm no expert on how this works specifically, but I'm guessing the initrd* one is an uncompressed image, while the vmlinuz* one is the compressed image. Notice it's size is similar to what you're seeing on Arch. I'm going to go out on a limb here, and guess that Arch and Gentoo stores the uncompressed image somewhere else.

Edit: Reading on Wikipedia that it is sometimes stored in

/boot

and other times in

/

Yay, I learned something today

Sorry, I assumed you were not using an initrd because you didn't mention it. I _think_ Debian uses a compressed one but with gzip, plus all possible modules by default rather than the minimum to boot. My current initramfs is about 7MiBs as well. But yeah, the image size is similar.