Btrfs Zstd Support Coming To Linux 4.14

[zanny]

In the kernel land, lzo is often pretty boring and a wrong option. I'm hoping that btrfs adopts more of those compression formats. There already was a patch for lz4, but it didn't get accepted. I wonder why..

Thats the point of zstd, no? It has similar perf to gzip but faster comp and decomp times like lzo. There were many articles circulating 2 years ago about legitimate issues using lz4 as a generic block device wide compression algorithm when those patches first surfaced, it wasn't some conspiracy.

[peppercats]

Don't get me wrong, I love btrfs, but I'm curious why zstd is your defining feature? Btrfs has had two other compression formats for years now. And more importantly, zstd is NOT a compression formats that ZFS even supports.

lzo and deflate are crap compared to lz4/zstd(who are from the same author, btw)

[jacob]

I really doubt btrfs is going to magically leapfrog ZFS just by changing compression algorithm. Zfs has been consistently outdoing btrfs in almost every benchmark I've seen for many years.

It must be a very different set from benchmarks from those I've seen.

[timofonic]

It must be a very different set from benchmarks from those I've seen.

Can you both put URLs instead of writing empty words?

[roelandjansen]

Can you both put URLs instead of writing empty words?

there are tests where BTRFS outperforms ZFS a lot. And the other way around too. The following is a link that describes testing, methodology et al and can be repeated -- all is documented in this university-thing.

https://www.diva-portal.org/smash/ge...FULLTEXT01.pdf

And this is also an interesting read:

https://www.slideshare.net/fuzzycz/postgresql-on-ext4-xfs-btrfs-and-zfs

(and I am wondering why you only said that to the second poster; the initial poster did not really come up with hard details either?)

[sdack]

lzo and deflate are crap compared to lz4/zstd(who are from the same author, btw)

lzo still compresses faster than lz4, only lz4 is faster at decompression. lzo still has it's uses especially when used in transparent compression where users are often not aware of it and thus a fast compression still matters.

Adding support for zstd to btrfs or any other software is good news. Only file compression on the file system level itself has never reached a wide-spread use, which makes the news a curiosity piece. People either compress their files with designated tools such as those used for video-, image- and audio compression, or they choose general compression tools such as xz, 7-Zip, WinRAR and ZIP for their high compression, or in case of the ZIP format, for their availability. Compression on the file system level or only on the transport level doesn't provide the same advantages as when it's done on the application level. So I want to see news about zstd being integrated into git and gcc for example, while I could care less about btrfs.

[Stellarwind]

http://www.ilsistemista.net/index.php/virtualization/47-zfs-btrfs-xfs-ext4-and-lvm-with-kvm-a-storage-performance-comparison.html?limitstart=0

[sdack]

http://www.ilsistemista.net/index.ph...l?limitstart=0

That's a bit old. Any further back in time and you will trigger GODWIN's law.

[scineram]

I heard somewhere zfs was interested in adding zstd as an option. Not sure if there's a timeline or any active work on that though.

https://reviews.freebsd.org/D11124

[DrYak]

For the record, Zstd is developped by Yann Collet, the guy behind LZ4.

Will it be smart to switch to zstd from lzo?

According to Zstd's own author, it plays in a different field.
LZOP (and LZ4) (and Snappy) are all designed to be as fast as possible algorithm while still providing some compression.

Zstd aims to the same category as GZIP (average compression / average speed), while beating it a bit (much faster and/or a tiny bit better compression).

It's definitely not in the category of stronger compressor as XZ/LZMA, QPak, etc. (better compression/sometime even best achievable compression, at the cost of massive speed drop)

So the question is, what do you look for :

- Compress as fast as possible ? Nope, keep LZO (and hope LZ4 gets implemented). Specially if your storage media is fast and can keep feeding LZO.
(Of course if your mass storage is tape... that you access remotely... over a 56k modem...)

- Compress as well as possible without hitting the performance too much ?
Then definitely test ZStd, it's supposed to have better performance than Gzip. (can compress faster for the samecompression ratio, or can compress a tiny bit better at similar speed).
So if your main reason of using LZO instead of GZIP is "Gzip is to slow", you should definitely give Zstd a try.

Zfs has been consistently outdoing btrfs in almost every benchmark I've seen for many years.

...in the specific cases where Zfs basically automatically activates its "nocow" options - which would be the recommendations given by the BTRFS wiki too.

i.e.: in some cases - like databases or virtual machines - very huge files see tons of manipulations.
a "cow" system like btrfs or zfs would be generating tons of indirections each time it does a cow-transaction on writes.
the huge files ends-up being a huge labyrinth of fragmentation.

ZFS is able to detect these special case and shut down "COW", forcing all the files to be written "inplace" - thus exchanging performance for stability(*).

BTRFS doesn't do this (but still has an options to do auto-defrag and at least reduce the amount of fragmentation), in order not to take stability-reducing decision without the user's command(*).

(*) : in practice in these 2 specific classes of cases - database and virtual-machine - the "nocow" route doesn't actually affect them much.
COW is actually redundant in this case, because each case has its own internal mechanism to handle data loss and BTRFS' and ZFS' COW would come redundant.
Database usually use a separate journal of transactions (so in case of power loss, they can replay the transaction to rebuild a clean data base).
Virtual Machine have whatever mechanism is used by the filesystem running on top of the VM image (...well except when the OS itself is something stupid coming out of microsoft and uses asinine formats like FAT or exFAT).

So in practice, ZFS isn't taking much danger by automating the task.

But similar performance could be achieved with BTRFS by simply making the huge files with "+C" extended attribute (or mount with "nocow" option for sake of simplicity)
But nobody cares to benchmark it this way.

----

Adendum :
Now there's another case of such big files : Torrents.
If you torrent a big movie files, that file is also going to see tons of small random writes scattered across the file.
(data isn't written linearly while being downloaded from a single source - like a web download.
Instead, data is written random as the various "chunks" are arriving out of order from all the peers into the network)
(in theory - I know in practice there are some torrent client which try first to get/or give more prioprity to the first part of the file, so that it can start playing early. Achieving something looking remotely like a web-linear download so you can whatch a movie before it has completely been downloaded - i think to remember that popcorn is such a program)

I haven't tested, but according to docs, ZFS should also see "tons of scattered small random writes" situation and trigger "nocow" on the torrent.
And in *that case* it would be a mistake.
Torrent have *no mechanism* to correct for corruption. They can detect it (checking checksums) but not correcting it.
If you suddenly pull the power-plug while a torrent is downloading, the result on disk will be corrupted, but lacking any mechanism (i.e.: unlike a database' transaction journal) it can only see the result using the checksum, and restart again downloading the corrupted chunks.

On the other hand, BTRFS can only take "auto-defrag" decision on its own in this case. If you don't manually "chattr +C" the empty file before you start downloading, nothing else will happen, the file will keep been written as COW. If you pull power-plug midway, BTRFS will be able to automatically recover the last known good copy, thank to cow.