Originally posted by caligula
View Post
Announcement
Collapse
No announcement yet.
Btrfs Zstd Support Coming To Linux 4.14
Collapse
X
-
-
Originally posted by doublez13 View Post
Don't get me wrong, I love btrfs, but I'm curious why zstd is your defining feature? Btrfs has had two other compression formats for years now. And more importantly, zstd is NOT a compression formats that ZFS even supports.
Comment
-
Originally posted by ormaaj View Post
I really doubt btrfs is going to magically leapfrog ZFS just by changing compression algorithm. Zfs has been consistently outdoing btrfs in almost every benchmark I've seen for many years.
Comment
-
Originally posted by timofonic View Post
Can you both put URLs instead of writing empty words?
https://www.diva-portal.org/smash/ge...FULLTEXT01.pdf
And this is also an interesting read:
(and I am wondering why you only said that to the second poster; the initial poster did not really come up with hard details either?)Last edited by roelandjansen; 02 September 2017, 06:19 AM.
- Likes 1
Comment
-
Originally posted by peppercats View Postlzo and deflate are crap compared to lz4/zstd(who are from the same author, btw)
Adding support for zstd to btrfs or any other software is good news. Only file compression on the file system level itself has never reached a wide-spread use, which makes the news a curiosity piece. People either compress their files with designated tools such as those used for video-, image- and audio compression, or they choose general compression tools such as xz, 7-Zip, WinRAR and ZIP for their high compression, or in case of the ZIP format, for their availability. Compression on the file system level or only on the transport level doesn't provide the same advantages as when it's done on the application level. So I want to see news about zstd being integrated into git and gcc for example, while I could care less about btrfs.
Comment
-
Originally posted by ormaaj View PostI heard somewhere zfs was interested in adding zstd as an option. Not sure if there's a timeline or any active work on that though.
- Likes 1
Comment
-
For the record, Zstd is developped by Yann Collet, the guy behind LZ4.
Originally posted by geearf View PostWill it be smart to switch to zstd from lzo?
LZOP (and LZ4) (and Snappy) are all designed to be as fast as possible algorithm while still providing some compression.
Zstd aims to the same category as GZIP (average compression / average speed), while beating it a bit (much faster and/or a tiny bit better compression).
It's definitely not in the category of stronger compressor as XZ/LZMA, QPak, etc. (better compression/sometime even best achievable compression, at the cost of massive speed drop)
So the question is, what do you look for :
- Compress as fast as possible ? Nope, keep LZO (and hope LZ4 gets implemented). Specially if your storage media is fast and can keep feeding LZO.
(Of course if your mass storage is tape... that you access remotely... over a 56k modem...)
- Compress as well as possible without hitting the performance too much ?
Then definitely test ZStd, it's supposed to have better performance than Gzip. (can compress faster for the samecompression ratio, or can compress a tiny bit better at similar speed).
So if your main reason of using LZO instead of GZIP is "Gzip is to slow", you should definitely give Zstd a try.
Originally posted by ormaaj View PostZfs has been consistently outdoing btrfs in almost every benchmark I've seen for many years.
i.e.: in some cases - like databases or virtual machines - very huge files see tons of manipulations.
a "cow" system like btrfs or zfs would be generating tons of indirections each time it does a cow-transaction on writes.
the huge files ends-up being a huge labyrinth of fragmentation.
ZFS is able to detect these special case and shut down "COW", forcing all the files to be written "inplace" - thus exchanging performance for stability(*).
BTRFS doesn't do this (but still has an options to do auto-defrag and at least reduce the amount of fragmentation), in order not to take stability-reducing decision without the user's command(*).
(*) : in practice in these 2 specific classes of cases - database and virtual-machine - the "nocow" route doesn't actually affect them much.
COW is actually redundant in this case, because each case has its own internal mechanism to handle data loss and BTRFS' and ZFS' COW would come redundant.
Database usually use a separate journal of transactions (so in case of power loss, they can replay the transaction to rebuild a clean data base).
Virtual Machine have whatever mechanism is used by the filesystem running on top of the VM image (...well except when the OS itself is something stupid coming out of microsoft and uses asinine formats like FAT or exFAT).
So in practice, ZFS isn't taking much danger by automating the task.
But similar performance could be achieved with BTRFS by simply making the huge files with "+C" extended attribute (or mount with "nocow" option for sake of simplicity)
But nobody cares to benchmark it this way.
----
Adendum :
Now there's another case of such big files : Torrents.
If you torrent a big movie files, that file is also going to see tons of small random writes scattered across the file.
(data isn't written linearly while being downloaded from a single source - like a web download.
Instead, data is written random as the various "chunks" are arriving out of order from all the peers into the network)
(in theory - I know in practice there are some torrent client which try first to get/or give more prioprity to the first part of the file, so that it can start playing early. Achieving something looking remotely like a web-linear download so you can whatch a movie before it has completely been downloaded - i think to remember that popcorn is such a program)
I haven't tested, but according to docs, ZFS should also see "tons of scattered small random writes" situation and trigger "nocow" on the torrent.
And in *that case* it would be a mistake.
Torrent have *no mechanism* to correct for corruption. They can detect it (checking checksums) but not correcting it.
If you suddenly pull the power-plug while a torrent is downloading, the result on disk will be corrupted, but lacking any mechanism (i.e.: unlike a database' transaction journal) it can only see the result using the checksum, and restart again downloading the corrupted chunks.
On the other hand, BTRFS can only take "auto-defrag" decision on its own in this case. If you don't manually "chattr +C" the empty file before you start downloading, nothing else will happen, the file will keep been written as COW. If you pull power-plug midway, BTRFS will be able to automatically recover the last known good copy, thank to cow.
Last edited by DrYak; 02 September 2017, 11:04 AM.
- Likes 2
Comment
Comment