Updated Zstd Planned For Linux 5.16 With Better Performance

[geearf]

No it's not changing cache usage. My system has 64GB of memory for example, and currently I can see it is using 21GB for caching purposes with 'free -m', but the actual memory usage of running applications is 8.2GB rather than the usual 15GB. I honestly have no idea what it does differently than official or Ubuntu kernels. Have you tried installing and running it? I'm curious if you see the same benefits on your machines. (and what OS/kernel you normally run)

I have not tried it, maybe I should, but my RAM usage is quite inconsistent so without running something like PTS I wouldn't really know if it was faring better.
If not cache, maybe something to do with zswap/zram? I am really curious but can't think of anything else that could impact it.

[krzyzowiec]

How is the patchset achieving lower memory usage? Is it about using less for cache?

No it's not changing cache usage. My system has 64GB of memory for example, and currently I can see it is using 21GB for caching purposes with 'free -m', but the actual memory usage of running applications is 8.2GB rather than the usual 15GB. I honestly have no idea what it does differently than official or Ubuntu kernels. Have you tried installing and running it? I'm curious if you see the same benefits on your machines. (and what OS/kernel you normally run)

[geearf]

You should really consider running Xanmod kernel if you aren't already. It literally cut my memory usage in half, from 15GB to 7GB on my desktop, and from around 7GB down to 4GB on my laptop. It made my laptop much more usable (8GB max memory).

How is the patchset achieving lower memory usage? Is it about using less for cache?

[Quackdoc]

You should really consider running Xanmod kernel if you aren't already. It literally cut my memory usage in half, from 15GB to 7GB on my desktop, and from around 7GB down to 4GB on my laptop. It made my laptop much more usable (8GB max memory).

Ill give it a try sometime, Currently using zenkernel, and it seems to work well, interesting to see what it will do on my system

[arQon]

Note running at IO speed with a much larger file can mean actual decompression takes longer than one that doesn't match IO speed but leads to a much smaller file.

Theoretically true, sure, but in practice almost never unless you're using absolutely bottom-tier storage like an SD card / USB 2 / etc. Even on spinning rust it's probably a wash.

> At least that's the effect on my boot time with compressing the initramfs with zstd and lz4 on a slow computer (SSD mounted on SATA-II netting about 300MiB/s, Atom N270 CPU).

I don't have anything QUITE that weak to compare against, but it seems very unlikely that a weak Atom can provide throughput of > 300MB/s on any nontrivial decompressor, even given has significant RAM use and/or per-use init overhead. (Which is a perfectly valid tradeoff to make, and very much a "good" one for initramfs specifically, given its size and the knowledge that all the RAM in the machine is currently available).

On my (quite a lot faster) Atom, initramfs creation is slower than dirt, and a constant annoyance. I'd rather just have it be uncompressed, but keep forgetting to change it. Thanks for the reminder.

[arQon]

You're partially right

How gracious of you to *almost* admit to being completely wrong...

zstd it has ... MUCH faster decompression speed compared to lz4

[krzyzowiec]

Forgive my brevity, am on mobile and little sleep. I use zstd for both zram and storage compression on a 64gb 4gb ram dualcore celeron.

lz4 while faster has lower compression ratio. almost a difference of 2.1x in lz4, and 2.8x on zstd, while it sounds small, when you need as much space as you can get, it makes a difference.

for me, zstd enabled me to once again use the device for daily usage, webbrowsing, typing and music simultaneously. as now even basic research can saturate low ram.

You should really consider running Xanmod kernel if you aren't already. It literally cut my memory usage in half, from 15GB to 7GB on my desktop, and from around 7GB down to 4GB on my laptop. It made my laptop much more usable (8GB max memory).

[FPScholten]

The Xanmod kernel has incorporated these zstd patches for quite some time now.

[sinepgib]

Citation needed. lz4 basically runs at IO speed. zstd very much does not.
Until your post, I've never seen anyone claim, anywhere, ever, that zstd is EVER faster than lz4, let alone "MUCH" so.

Note running at IO speed with a much larger file can mean actual decompression takes longer than one that doesn't match IO speed but leads to a much smaller file. At least that's the effect on my boot time with compressing the initramfs with zstd and lz4 on a slow computer (SSD mounted on SATA-II netting about 300MiB/s, Atom N270 CPU). At the end of the day, cat would be fastest to decompress if we left IO time out of the equation :shrug:

[skeevy420]

Great news. Since there are 5.14 patches I'm hoping it'll get backported to 5.15.

On top of this, there are significant performance improvements coming down the line in the next zstd release, and the new automated update patch generation will allow us to pull them easily.

To me, that's the best part of this announcement. Performance improvements are great; not having to wait so long for them is better. AFAICT, the only thing BTRFS is missing over ZFS in regards to Zstd is fast support...which can equal LZ4 in speeds.

I don't know how to do this, but I was thinking and since Zstd basically decompresses just as fast regardless of level "compress-idle:X and compress-force-idle:X" mount options would be nice. Whenever the system is idle and doing nothing it would compress stuff to level X. We could use "compress-force:2,compress-force-idle:15" so the system is nice and fast when we're doing stuff and can compress higher when we're not.