OpenZFS Lands A Very Nice Performance Optimization

[dragorth]

A few years ago I tried a desktop BSD distro called PC-BSD which used ZFS by default and the filesystem used literal gigabytes of ram. Idk, maybe these days things have improved in that regard, so please enlighten me if yes.



I know it's cool to hate on Btrfs to the point of making hyperbolic statements like these, but I've been using Btrfs for about 2 years now with zero stability issues whatsoever and as an average user, filesystem compression alone is a reason for me to use it.

The filesystem uses about 1GB of memory for every TB of HDD space. This is that advanced cache that was mentioned earlier. This boils down to philosophies, and ZFS has options that can make that cache extremely fast and save you lots of disk space, if you are running the right workload.

[satadru]

The filesystem uses about 1GB of memory for every TB of HDD space.

That's only if you have deduplication enabled...

[Developer12]

Nice fairy tales:

https://storytime.ivysaur.me/posts/why-not-zfs/

lol. *insert badly written attack website*

Let's run down the list shall we:

>out of tree
Doesn't matter. See my comment you quoted.

>slow encryption
ZFS quickly went back to using the denied instructions, simply from a different execution context where the newly-GPL'd setup and teardown functions aren't required. Pretty sure this also applies to encryption. Code churn, but whatever. Big picture, the kernel can't really afford to break "all filesystems" or "all users of SIMD." Otherwise, why even allow non-GPL modules in the first place?

>"rigid"
ZFS is structured as a stripe across various virtual storage devices, each with their own redundancy. Sorry it doesn't allow for arbitrary device-on-device stacking. ¯\_(ツ)_/¯

>Can't add/remove from a raidz
The patchset for doing so is waiting for the next big release. Removal is a tricky problem and at this point will require additional future work, but it's best to move progressively when restructuring core raidz code. Wouldn't want to end up with a terminally-broken raid implementation now would we?

>raidz is slow
So use a stripe of mirrors (RAID10) then. Raidz is for space efficiency, not speed.

>file-based raid is slow
Sorry, re-silvering and scrubbing are slow. Scrub periodically.
A delightful cherrypick. The fact mdadm is faster on SMR drives is because it's doing resilvering in the dumbest way possible. ZFS only needs to resilver the area that's actually being used, making it overwhelmingly faster on drives that aren't garbage SMR.

>looses to ext4 in speed
Doesn't loose your data though. The fact is that ZFS has to do more work to protect data integrity. No free lunch.

>performance degrades with low free space
Much improved. This is one of the many performance pathologies that have largely gone away over time. The less said about what happens to BTRFS when it runs out of space, the better. As for using ext4 and *NTFS* as a comparison, see above.

>layering violation aka "it's both a filesystem and a volume manager
Yeah, and it does the job of both better because of it (see: massive speedup from only resilvering area used by actual files).
Don't worry, the code is internally layered. If you want a different filesystem, write one against ZFS' internal object store.

>no reflink
It's a CoW filesystem making this less than mandatory. (dedupe makes this completely unnesisary, but is unrecomended as it often isn't worth the performance hit) Nonetheless, according to the long-running issue for this, one of the required bits of infra looks like it was ready to land in 2021. I'll differ the state of this work to someone on the front lines of ZFS development.

>dedupe is a memory hog
Don't use it then. It's not recommended because the performance hit often isn't worth it. CoW works fine without it.

>dedupe is sync not async
See above.

>ARC uses a lot of memory
As is it's job. It's a cache. By default it's configured to use half your ram. A couple years ago some work landed to make ARC memory reclaim essentially instant, so this is now a non-issue. If you still need lots of ARC, install an L2ARC device which as of a year or two ago now also remains hot across reboots.

>"buggy because there are ~380 issues still open on the github"
A meaningless metric (how many are real bugs? how many haven't already been fixed?) that lacks any sense of scale. See my comment you quoted for the real-world difference in reliability, backed by extensive test coverage.

>no disk checking tool
Disk checking tools are a bad idea. Actually. An ounce of prevention is worth a pound of cure.
ZFS checks everything whenever you access it, as well as during a scrub which you're supposed to run regularly.
Trying to recover a known-bad filesystem post-corruption via amputation (eg. ext4 and every other filesystem without the ability to repair data) is living in a state of sin. Lacking data checksums makes disk checking tools further useless, but I digress.

[Developer12]

I know it's cool to hate on Btrfs to the point of making hyperbolic statements like these...

Fair enough.

but I've been using Btrfs for about 2 years now with zero stability issues whatsoever and as an average user, filesystem compression alone is a reason for me to use it.

Then keep using it if it works for you. My point is there's no reason to not use ZFS for any new installations. Fwiw, ZFS also supports equivalent compression in various fun fruity flavors.

[Danny3]

1.) Is factually way better
2.) Is not really incompatible but more about linux devs being unsure about CDDL compatibility in case of some troll

2.) Why don't just change the licence to GPL to avoid that the Linux devs are usure to solve this uncertainty?

I know you didn't mean it like that, but the way you that said implies that Ext4 and BTRFS have license issues.

Yes, this is the one.

They use it for the same reason some people pick NVIDIA over AMD -- it's better for their use-cases. I wouldn't pick NVIDIA over AMD, but some people would. I would pick OpenZFS because it allows mixing and matching of settings to a very fine degree that no other file system allows. You can pick OpenZFS because it is one tool where as with built-in you need three tools: LUKS, LVM, and BTRFS. That said, LUKS+OpenZFS is a bit better if you're worried about a local hacker trying to figure out what is on a disk based on metadata. It's still less layers for more features.

Anyhoo, OpenZFS is also crossplatform between Linux, FreeBSD, and Solaris and eventually with macOS and Windows. What are the next best cross platform file systems? NTFS? Vfat?

Just because it's the "wrong" kind of open source doesn't negate that it's open source.

Thanks for pointing out my confusing comment and for the explanations!
But isn't ExFAT pretty cross-platform, especially between Linux and Windows?

Because:
- it's just a "problem" of license. CDDL is OSI approved, it's opensource. And other people integrate easily OpenZFS with Linux for you (TrueNAS, Proxmox, for instance). So if it's not in Linux, whatever...
- there is no competion on Linux against OpenZFS. The best one would be Stratis I guess, which is pretty young (I never tested it)... Ext4 is just a simple filesystem. Btrfs, lol, it can't even handle a simple RAID5 safely...

Ok, but then, if the license is open source already, what's the big deal about going one little step further and make it GPL2 so that's exactly the same as the kernel and avoid any uncertainty?

[Developer12]

2.) Why don't just change the licence to GPL to avoid that the Linux devs are usure to solve this uncertainty?
...
Thanks for pointing out my confusing comment and for the explanations!
But isn't ExFAT pretty cross-platform, especially between Linux and Windows?
...
Ok, but then, if the license is open source already, what's the big deal about going one little step further and make it GPL2 so that's exactly the same as the kernel and avoid any uncertainty?

Can't change the licence because the rights to the original code are still held by Sun, now Oracle. Oracle can't make it closed source again, but the OpenZFS devs can't make it GPL either. It's a legal stalemate.

Oracle eventually (through their own actions, not a decree) relicenced dtrace from CDDL to GPL by using it with the linux kernel. They currently ship their own ZFS appliances (a continuation of fishworks at Sun) with a proprietary fork of ZFS, running on Solaris instead. For as long as those boxes remain profitable they don't really have any reason to change things, even though they haven't been able to hinder widespread adoption of OpenZFS.

[NobodyXu]

>no reflink
It's a CoW filesystem making this less than mandatory. (dedupe makes this completely unnesisary, but is unrecomended as it often isn't worth the performance hit) Nonetheless, according to the long-running issue for this, one of the required bits of infra looks like it was ready to land in 2021. I'll differ the state of this work to someone on the front lines of ZFS development.

TBF, I found the article very misleading and most of them being totally wrong.

But this point about lack of reflink isn't wrong.
The use case including steam/wine which takes advantages of reflink to reduce space usage.
It can also be used by other tools which just needs to copy a file from one location to another to reduce space usage.

Unless all you care is modifying one file, otherwise your point is moot, CoW fs does not make this less needed and it's not an excuse for CoW to not support this.

[NobodyXu]

Fair enough.

Then keep using it if it works for you. My point is there's no reason to not use ZFS for any new installations. Fwiw, ZFS also supports equivalent compression in various fun fruity flavors.

Not everybody needs RAID 5/6, sometimes even a single device with DUP profile is suffice.
The point is Btrfs is in-tree, with out-of-the-box support and you don't need to worry about external kernel module or installing it as most distros already includes it in the kernel and you just need to install some tools if not found (btrfs-tools).

It provides a lot of useful features (checksum, compression, DUP profile for single disk, offline deduplication, O(1) snapshot, reflink) compared to ext4 and others (though xfs does support reflink) and if I really have more disks, I can simply build a RAID 1 or RAID 10, not everybody needs RAID 5/6/z-{1, 2, 3} since I don't have that much data and don't have that many disks.

It does need more improvement on its performance, better btrfs-send protocol, RAID 1 optimization, fix its RAID 5/6 for whoever wants to use it, maybe supports more disks with RAID 10, etc, but in its current state is very usable for me and a lot of people, so saying that everybody should use ZFS for new installations is so wrong since not everybody is trying to build a NAS with 5 or more disks.

[toves]

Never did understand why ZFS on Linux is some kind of religious war. The licensing concerns are fair enough but its still your choice - there are a few storage appliances that have chosen yes. From a technical perspective why not just give it a try? Most stable Linux distros can grab the kernel models and user space from a repository. You don't even need to allocate disks - vdevs can be anything - lvm logical volumes are fine - I have used 10 2Tb files on a NFS volume exported from a OneFS appliance as vdevs admittedly not in anger but did work reasonably well.

I suspect some antipathy derives from user space tools being fairly different from the mkfs/mount&c style. I think Sun perhaps was also responsible to some extent when ZFS was introduced (Solaris) many of its features competed with existing add-on products (Veritas?, Solstice?) - a point that was possibly obfuscated to some extent by Sun marketing types at the time. In any case there a few former Solaris sysadmins that are anaphylactic when ZFS is mentioned.

[itoffshore]

Is this the one with the license problems that will never be accepted in the Linux kernel?
When you have EXT4 and BTRFS, I don't understand why would anyone use a file system that doesn't properly support the Linux kernel having a compatible license.

I use both ZFS & btrfs together.

ZFS is just a more reliable choice for data. Your data will never become inaccessible if the filesystem becomes full (as has happened to me on 2 occasions with btrfs). DKMS is mature & updating the ZFS modules & utils is mostly never an issue (at least on Arch) - & never an issue on Ubuntu. Encrypted ZFS without needing LUKS is nice & NFS on ZFS is simple.

Btrfs is a good choice for a root filesystem & I've used it for this purpose for 6-7 years with zero issues. System snapshots are a nice fallback but I've only ever needed them once or twice in all this time. For / & /var btrfs is reliable enough with default settings. With good use of filesystem quotas btrfs is probably reliable for data (ZFS automatically reserves a % to prevent a no op situation)