Zstd Compression Under Review For OpenZFS

[Ornias1993]

Ornias here (yeah the same from Github, if you want to verify throw me an email).
Yes, i'm annoyed easily and about a lot of things. I'm not going to dive into every one of them to defend myself. It irrelevant for ZSTD on ZFS.

Firstoff some perspective: I've gotten promises in DECEMBER 2019 from the leadership they would review it "next month" due to the holidays. This was after we spend 3 months to get the previous work from brainslayer and Allen (Jude) review ready and tested.

Januari: No review, No comment, pinged about it, no review, no comment
Februari: No review, No comment, pinged about it, no review, no comment
March: : No review, No comment, pinged about it, no review, no comment
April? You guessed it

Here comes May: Leadership is having a meeting without any of the ZSTD maintainers, Allan throws up some lies about it (and him restructuring it, which he isn't actually doing nor is he involved anymore), the leadership believes everything and makes some statements that clearly show they have barely even looked at the PR description(s).

May 2.0: Behlendorf actually reviews it a bit with some feedback... He wanted it moved and compacted. (If only he either documented the folder structure or given us this simple hint in like... October-Januari... But okey, thats a review.)
He also made the statement making some time to review next week...

Today: Fixed all his feedback, updated ZSTD to the new version, compacted the library into one file, removed the header magic from zstd after review from zstd folks (who actually did review our work multiple times by now, much appreciated)... etc... Still waiting on actuall code review (the previous one was just his "first glance" review according to behlendorf himself) or a response delaying his review.

Why are we worked up about it?
Are we mad it isn't being reviewed? Not at all.
Are we mad because they don't like it? Not at all.
Are we mad because it isn't going fast enough? Not at all.

We are mad, because we are trying time and time again to bring this under their attention and all we get are false promises. We all KNOW they are busy people and don't expect feedback or review. Hell we don't even expect them to like it. We just want then to communicate about they plans (or lack thereoff).

A quick: "Hey guys we are kinda busy, not going to make it this month... sorry, we need to delay till next month." would be awesome.
Or even a "Guys, we appreciate the effort... but we don't really like it... sorry..." would be great. So we can focus our attention elsewhere.

Getting back into coding a PR every few months is not fun or easy btw...


What I tried to make clear with this:
Us complaining about the leadership isn't about them not-liking it or the delays. It's about not communicating about what the heck they want (or don't want) from us.

Now ontopic:
I read a comment stating they might not find it "enterprise grade", well they could've said so.
Our solution has been reviewed by ZSTD folks multiple times by now and the compression side seems to be fine.
It has also been tested by at least 2 people in production already for 6 months (without many issues), has undergone frequent performance testing and stresstesting.

The only thing we REALLY need is review, either from the leadership or any of you.
We need people from the ZFS side to think what they think needs tweaking.

So, even without leadership activity: Show your support and review if your can... Or try it out for yourselves, Our performance test procedure is linked, ping brainslayer for the stresstests or makeup your own tests

[SystemCrasher]

Is there a good reason why compression support is not added to the VFS or another generic layer instead of each individual FS?

In case of e.g. Linux zstd algo itself is split to its library, as well as many other algos do. However, exact details on how particular filesystem deals with blocks allocation, how exacrly it puts on-disk structures, what they contain, why and how exactly compression fits all that is up to particular filesystem. Idea to put it to VFS implies filesystems used some unified approach that could be factored out. Which isn't a case to begin with.

[geearf]

ZFS very tightly integrated and basically bypasses the "normal" Linux layers for doing this stuff. It spans logical volume management, file system, deduplication, compression, encryption, data integrity, multi-level caching and "RAID" management. Some view this tight integration as ZFS' strength while others find it a weakness.
It does use shared stuff when it's allowed to (some functionality in Linux is exposed only to GPL-licensed modules which ZFS is not) like encryption routines for example.

I think it's good for some stuff to , if not at the FS-level, at least be understood by the FS. For instance, having RAID management allows the FS to auto-recover from a potential issue, unlike when using mdadm. But I am not convinced about the same for compression.
But yeah, maybe for something like ZFS it wouldn't make sense to share too much since it started non-natively.

Thanks!

[numacross]

Is there a good reason why compression support is not added to the VFS or another generic layer instead of each individual FS?

ZFS very tightly integrated and basically bypasses the "normal" Linux layers for doing this stuff. It spans logical volume management, file system, deduplication, compression, encryption, data integrity, multi-level caching and "RAID" management. Some view this tight integration as ZFS' strength while others find it a weakness.
It does use shared stuff when it's allowed to (some functionality in Linux is exposed only to GPL-licensed modules which ZFS is not) like encryption routines for example.

[geearf]

Is there a good reason why compression support is not added to the VFS or another generic layer instead of each individual FS?

[Terrablit]

Ornias1993 appears to be perpetually angry and overly aggressive about a lot of things for seemingly no reason. This is the same guy who out of nowhere started flaming Jens Axboe over nothing [1][2] (Jens is the Linux block maintainer and author of fio). I would take what he is complaining/raging about with a HUGE grain of salt.

[1] https://github.com/axboe/fio/issues/891
[2] https://github.com/axboe/fio/issues/892

It's not just him, though. c0d3z3r0 actually made the PR and he deleted it out of frustration before restoring it because it had work from others, too. Brainslayer from DD-WRT also commented that it seemed like someone didn't want zstd support. Just one guy, maybe. But three on the same PR?

Well in Open ZFS defense remember this is used to handle data in Enterprise sectors and even something that could work nice with a home/SOHO setup may not work properly on those monster storage systems and also take into account is not easy to find reviewers that can actually understand what they are reviewing at this level which is the same problem the kernel have in some subsystems.

It's been reviewed by a lot of people already. There's a lot of people collaborating on the same issue. I mean, the guy from dd-wrt is contributing patches. They wrote tests for everything. And OpenZFS has been getting a lot of updates. As well as merging ZoL and BSD into the project. There's simultaneous work for things like persistent l2arc. It's not like ZFS development is slow. It's just that some things don't get attention. Like the lz4 compression that sorely needs updating, but is painful because they customized it way back when. Which is why zstd is using the upstream zstd library.

At the same time, this has been going on for like 3 years. They were asked to squash all the commits, but they ignored that and broke them out, and then, years later, were asked to be in a discussion to break it out even more. There's a lot of interest in the project. And the compression algorithm is manually selected, and supposedly relatively simple. There would be people willing to run tests on this for a while. But just look at those minutes in the context of several years and you can see that it's beyond low-priority. They don't want it. They mentioned that it needs updating post-l2arc, but that'd already been done by then. There's comments about l2arc hardcoding the compression algorithm in there which are being handwaved by others on the OpenZFS project. And rebasing wouldn't have needed doing if any of the nearly-dozen PRs from over the years had been accepted.

If it needs some time in staging, that's one thing. If it needs reviews, review it so it goes away already. But don't just tout the risks of new code and refuse to even look at it while other things get merged and keep pushing it back. Not every feature runs up against a wall like this. Somehow it's just zstd that has to be absolutely perfect before it makes it in while other features are sliding in and perpetually moving the goalposts. If you can't articulate a reason why, you need to talk about the real reason you don't want it or stop blocking.

The problem isn't stability. The problem is 3+ years of people working on this, although the current crew may have been on it for less. If you've got a feature that's been pending that long with people actively doing work, you're shitting on them if you don't review and accept or cut it loose.

[jrch2k8]

It's pretty frustrating to see this languish for so long while the OpenZFS leadership wanks off to its limbo death rattle. If you're not going to actively participate or manage, get out of the way. They didn't even have the stones to talk about it in the PR or a review, but sent it back for a nebulous "review" to waste more time. Now most of the original contributors are on the cusp of quitting after wasting this much time.

Leadership that blocks advertised/promoted features from inclusion for years without any concrete feedback or participation should just don their pointy-haired-boss dunce caps and fuck off back to clown town already.

Well in Open ZFS defense remember this is used to handle data in Enterprise sectors and even something that could work nice with a home/SOHO setup may not work properly on those monster storage systems and also take into account is not easy to find reviewers that can actually understand what they are reviewing at this level which is the same problem the kernel have in some subsystems.

[Space Heater]

Wow, the PRs speak a horrible story (though one-sided?) about OpenZFS leadership... Is this just an isolated case or are there more examples?

Ornias1993 appears to be perpetually angry and overly aggressive about a lot of things for seemingly no reason. This is the same guy who out of nowhere started flaming Jens Axboe over nothing [1][2] (Jens is the Linux block maintainer and author of fio). I would take what he is complaining/raging about with a HUGE grain of salt.

[1] https://github.com/axboe/fio/issues/891
[2] https://github.com/axboe/fio/issues/892

[Terrablit]

It's pretty frustrating to see this languish for so long while the OpenZFS leadership wanks off to its limbo death rattle. If you're not going to actively participate or manage, get out of the way. They didn't even have the stones to talk about it in the PR or a review, but sent it back for a nebulous "review" to waste more time. Now most of the original contributors are on the cusp of quitting after wasting this much time.

Leadership that blocks advertised/promoted features from inclusion for years without any concrete feedback or participation should just don their pointy-haired-boss dunce caps and fuck off back to clown town already.

[starshipeleven]

The first PR was in 2018 as well. I'm a bit surprised though that the same guy that presented working ZSTD-compresion on FreeBSD's ZFS in 2017 seems to now bikeshed over the review in 2020?

If it isn't made by him he does not want to see it, I guess.