Zstd 1.5.1 Released With Even More Performance Improvements

[piorunz]

I never use Facebook nor Instagram or any of their products and consider them very evil (and fatal for teenagers), but Zstd is their only good product.

Btrfs is a total failure.

Congratulations, Facebook = META. You did something good, finally

Complete nonsense. If Btrfs is total failure, thousands of computers wouldn't be running it. I run it. Just this month, my 512 GB NVMe drive (Crucial CT500P2SSD8) gave Btrfs checksum errors in dmesg log. I located the file. It was unreadable, corrupted write by NVMe drive. I couldn't copy or "cat" it. I was able to determine where this file was from (it was a system library), and reinstall the package. Bit rot on NVMe drive. Now, this NVMe fails SMART long test. Very bad. I would never know if not for Btrfs. I have ECC DDR4 RAM on this machine (Ryzen build).
99% of people don't even know this is happening on their computers, because they don't have checksumming file system. I will never revert to ext4, I want data integrity.

[caligula]

Congratulations, Facebook = META. You did something good, finally

FB didn't do this. Zstd was invented before FB came along.

[useless]

I consider critical stuff like a filesystem ought to work nicely without tuning

What? You're telling us that we shouldn't tune filesystems for our purposes? Name one concise example of a filesystem that doesn't require tuning to 'work nicely' in every workload (which is what you're implying).

with sane defaults and all available features working nicely. If tuning and avoiding features is mandatory for a harmless experience, it's not good at all.

Ever heard of 'experimental features'? Another case of RTFM? As if any feature doesn't come with some kind of tradeoff (whichever filesystem).

[sdack]

Complete nonsense. If Btrfs is total failure, thousands of computers wouldn't be running it. I run it. Just this month, my 512 GB NVMe drive (Crucial CT500P2SSD8) gave Btrfs checksum errors in dmesg log. I located the file. It was unreadable, corrupted write by NVMe drive. I couldn't copy or "cat" it. I was able to determine where this file was from (it was a system library), and reinstall the package. Bit rot on NVMe drive. Now, this NVMe fails SMART long test. Very bad. I would never know if not for Btrfs. I have ECC DDR4 RAM on this machine (Ryzen build).
99% of people don't even know this is happening on their computers, because they don't have checksumming file system. I will never revert to ext4, I want data integrity.

While I appreciate it when people speak out about BTRFS and keep pointing out its usefulness, do I disagree with the idea of checksumming being some kind of life saver. Checksums do not fix anything, only redundancy does. And when your point truly is that you would otherwise not have noticed it then the data was obviously not very valuable in the first place, and all that checksumming did for you, in this case, was to give you extra work. The 99% of people you mention are quite happy regardless of whether their file system does or does not support checksums. People just do not care as much as you may think. If anything is it the history of BTRFS that has made its users more careful. Not saying this is a bad thing, but please do not overrate the checksum feature.

[microcode]

FB didn't do this. Zstd was invented before FB came along.

No; zstd was released in 2015, and was developed by Yann Collet, a Facebook employee.

[sinepgib]

And when your point truly is that you would otherwise not have noticed it then the data was obviously not very valuable in the first place, and all that checksumming did for you, in this case, was to give you extra work.

You clearly don't keep backups then. You don't access them often, but you definitely want to know if they got corrupted. That said, yes, you keep redundant backups for fixing up corruption. But you find out if a given one is corrupted before switching to the other copy. Checksums help with that.

[useless]

While I appreciate it when people speak out about BTRFS and keep pointing out its usefulness, do I disagree with the idea of checksumming being some kind of life saver. Checksums do not fix anything, only redundancy does. And when your point truly is that you would otherwise not have noticed it then the data was obviously not very valuable in the first place, and all that checksumming did for you, in this case, was to give you extra work.

As someone who had to deal with bad hardware and firmware bugs in hard drives, I truly appreciate filesystem check-summing. The filesystem does it for you so you don't need to roll your own to prevent backup propagation. It gives no extra work since it's transparent at both ends of send/receive (and, of course, 'in transit').

The 99% of people you mention are quite happy regardless of whether their file system does or does not support checksums. People just do not care as much as you may think. If anything is it the history of BTRFS that has made its users more careful. Not saying this is a bad thing, but please do not overrate the checksum feature.

That's because they don't know any better. With btrfs being the default filesystem for workstation and regular desktop PCs they will start to notice corruption. 'People' are learning that hard drives sometimes sucks, bad, specially consumer oriented ones. If anything, having automatic and trouble-free checksumming is an understatement.

[DrYak]

No; zstd was released in 2015, and was developed by Yann Collet, a Facebook employee.

Cyann got hired at Facebook in summer 2015, basically for his past work on compression (including also releasing LZ4, initially in 2011).

Most of the work on Zstd has been done while Cyann was still employed at Orange, he was publicly documenting the development on his blog.

The development started way before when Cyann though about adding an entropy stage to his pure-dictionnary based approache in previous LZ4 (initially he went for Huffman, then the whole compression scene was shaken by Duda's tANS, FSE became a thing, and Cyann incorporated them into his work, which eventually evolved into Zstd, and got an official release in January 2015).

Eventually in June 2015 Facebook hired him because their datacenters would benefint from ultra-fast compression, and thus they needed to make sure that the Zstd author was on their payroll (his LinkedIn entry litteraly list the further development and maintenance of LZ4 and Zstd as his purpose at Facebook).

For those who are into data compression Zstrd is "that thing that Cyann wrote", not "a product by Zuckerberg's Meta"

BTRFS followed basically a similar path, being the brainchild of Chris Mason who moved from SuSE (working on ReiserFS) to Oracle and started working on that filesystem at that time. Eventually down the line 6 years Facebook hired him, for the same obvious strategic needs in their datacenters.
(Not that being initially an Oracle opensource product doesn't come with its own stigma)

I am quite sure that Bcachefs will follow the same path, currently known for being the brainchild of Kent Overstreet, but probably that by 2030 when finally this filesystem reaches "features maturity", Kent will be hired by some big company with strategic interest in their datacenters, and suddenly people with short memory will start criticize it as being some product by evil Microsoft/IBM/Amazon/whatever (depending on who will snatch Kent).

[skeevy420]

What? You're telling us that we shouldn't tune filesystems for our purposes? Name one concise example of a filesystem that doesn't require tuning to 'work nicely' in every workload (which is what you're implying).



Ever heard of 'experimental features'? Another case of RTFM? As if any feature doesn't come with some kind of tradeoff (whichever filesystem).

Ext2

Everyone else already said all the good comments I came to make.

[DrYak]

If Btrfs is total failure, thousands of computers wouldn't be running it. I run it.

I run it, too. Including on my smartphone (Sailfish OS).

Just this month, my 512 GB NVMe drive (Crucial CT500P2SSD8) gave Btrfs checksum errors in dmesg log. I located the file. It was unreadable, corrupted write by NVMe drive. I couldn't copy or "cat" it.

Note that the normal behaviour of the BTRS kernel driver is to outright block access to extent that are corrupted according to their checksum, instead of letting you retrieve the damaged/bit-flipped block
(There are ways around such as relying instead on the CLI's "btrfs restore" might help better).

Bit rot on NVMe drive. Now, this NVMe fails SMART long test. Very bad. I would never know if not for Btrfs.

Although I too run "btrfs scrub" periodically to detect such problems (and correct them if a usable RAID1 copy exists), I would like to point out that it is also possible to periodically run both short (I do nightly) and long (used to be weekly on all drives, but now I only run them monthly on mechanical HDD, given that I have a weekly btrfs scrub) using the smartd service that comes with smarttools.

This periodic long-test comes with a few advantages on top of the scrubs:
- done entirely within the drive, doesn't stress the IO nor keeps the kernel busy (which would be useful for storage on a less fast bus, e.g.: USB attached storage on single board computers)
- (though it will still slow down the seeks on mechanical HDDs).
- it can detect problems in unused space (wheres btrfs will obviously scrub only allocated chunks).
- can detect problems in layers between btrfs and the storage (e.g.: RAID5/6 is still considered unstable on BTRFS, so on the machine that use such RAID mode, I am simply piling a BTRFS in single mode on top of an MDADM RAID6. Some errors corrected by the raid layer will be silent in the scrub report).