Linus Torvalds Doesn't Recommend Using ZFS On Linux

[oiaohm]

Those github setup guides look fun.. but you know I think I'd just like to do:
Step 1. Install FreeBSD
Step 2. You already have atomic upgrades enabled, there is no step two.

Swap FreeBSD for RHEL/Centos Atomic Host or Silverblue. Yes step 2 stays the same and you have atomic upgrades enabled outbox no matter the file system you choose.

I don't think FreeBSD has atomic upgrades if choose some other file system other than ZFS on install. Yet the ones I listed you can choose many different file systems and still have atomic upgrades.

I don't think ZFS will ever catch up on raw performance.

[k1e0x]

Do note I write Btrfs/LVM not pure Btrfs. LVM can do the perfect roll back snapshots under the file system. That quote from manual was pure Btrfs solution.



Basically its telling that you are tunnelled visioned.

https://www.projectatomic.io/docs/os-updates/ Yes ostree from project atomic is also registering after 1 command in the boot loader.

That because the XFS/ostree one is not called snapshots. Its atomic upgrades https://ostree.readthedocs.io/en/lat...omic-upgrades/. "cloning boot environments" does not have to equal snapshots.

Please note these atomic upgrades can function without relink just not as disc effective.

https://ostree.readthedocs.io/en/lat...ting-existing/

What is different here is you are creating the snapshots above the file system using the VFS layer todo the bending. Really do you want your boot environment fully writable?

ostree is slower with the relink system.

Ostree/project atomic is clone able and transportable to other systems that are not using the same file system type.


There are three places you can do snapshoting.

1) Block layer. LVM
2) VFS layer (mount namespaces under Linux)
3) File system.

Snapshotting to be disc space effective just need a cow of some from. LVM contains a CoW. XFS contains a CoW by relink. Of course ZFS and Btrfs contain a CoW.

k1e0x I guess the concept of a VFS layer snapshotting never crossed your mind.

Wow that sounds like a lot of limitations... well don't worry.. Linux will catch up someday. The technology to do this is only a decade old.. and still not in RHEL.. shame..

Those github setup guides look fun.. but you know I think I'd just like to do:
Step 1. Install FreeBSD
Step 2. You already have atomic upgrades enabled, there is no step two.

[oiaohm]

OpenSUSE has pretty much the cutting edge of btrfs integration. From their manual.[/URL]

Do note I write Btrfs/LVM not pure Btrfs. LVM can do the perfect roll back snapshots under the file system. That quote from manual was pure Btrfs solution.

I did a search for XFS writable snapshots... Google said "Did you mean ZFS writable snapshots?" I think that's telling..

On FreeBSD it takes one command to add a boot environment you can select from the boot loader. No packages, no config changes. It's writable, closable and transportable to another system.

Basically its telling that you are tunnelled visioned.

https://www.projectatomic.io/docs/os-updates/ Yes ostree from project atomic is also registering after 1 command in the boot loader.

That because the XFS/ostree one is not called snapshots. Its atomic upgrades https://ostree.readthedocs.io/en/lat...omic-upgrades/. "cloning boot environments" does not have to equal snapshots.

Please note these atomic upgrades can function without relink just not as disc effective.

https://ostree.readthedocs.io/en/lat...ting-existing/

What is different here is you are creating the snapshots above the file system using the VFS layer todo the bending. Really do you want your boot environment fully writable?

ostree is slower without the relink system but will work anyhow.

Ostree/project atomic is clone able and transportable to other systems that are not using the same file system type.


There are three places you can do snapshoting.

1) Block layer. LVM
2) VFS layer (mount namespaces under Linux)
3) File system.

Snapshotting to be disc space effective just need a cow of some from. LVM contains a CoW. XFS contains a CoW by relink. Of course ZFS and Btrfs contain a CoW.

k1e0x I guess the concept of a VFS layer snapshotting never crossed your mind.

[k1e0x]

You really need to stop that claim now. ostree supports XFS reflink. Ostree used in combination with XFS with relink gives this exact same feature of cloning boot environments quickly with disc space effectiveness and created checksums to locate damage. Technically any file system that supports relink could be used.

So there are three file systems under Linux that can do boot environments. ZFS, Btrfs/LVM and Ostree/XFS. Ostree/XFS you see in Fedora project silverblue.

Really ZFS does not have as many advantages as one would think other than its per block protection. XFS being a part CoW file system with reflink means with software like ostree on top it can pull large number of the ZFS tricks but with higher overall performance.

OpenSUSE has pretty much the cutting edge of btrfs integration. From their manual.

3.3.3 Limitations
A complete system rollback, restoring the complete system to the identical state as it was in when a snapshot was taken, is not possible.

There is a whole list of things that it can't restore (including the kernel) This isn't true on FreeBSD it can restore everything to the way it was before.

I did a search for XFS writable snapshots... Google said "Did you mean ZFS writable snapshots?" I think that's telling..

Linux does not really have this feature in anything I've seen.. I don't see it in any distros and even there is a way to do it it'd def not as clean or well implemented. It probably also does a copy so it's slow as hell.

On FreeBSD it takes one command to add a boot environment you can select from the boot loader. No packages, no config changes. no waiting for anything to copy. It's writable, closable and transportable to another system and works even on an encrypted disk. The jail integration is just the cherry on top.. that's *two* commands.

Use.. maybe your system upgrade went sideways and you need to get the system back up. Restore the old environment and boot production then send the failed update system to a dev system and boot the environment in a jail. After you fix and test it you can send it back.

[oiaohm]

3. You don't know what you're talking about. COW alone has nothing to do with bit-rot or uncorrectable errors. You're thinking of block checksums, and yes, they are good. COW provides other features such as snapshots, cloning and boot environments. Boot environments are pretty cool.. maybe Linux should get on that... oh wait.. ZFS is the only Linux file system that does it and we can't have *that*.

Check this out.. FreeBSD 12 has a new command for boot environments..
https://www.freebsd.org/cgi/man.cgi?query=bectl

bectl create [email protected]
bectl jail [email protected]

You just cloned your running OS into a writable, bootable, virtual environment and spawned a shell in it and it was instant.

You really need to stop that claim now. ostree supports XFS reflink. Ostree used in combination with XFS with relink gives this exact same feature of cloning boot environments quickly with disc space effectiveness and created checksums to locate damage. Technically any file system that supports relink could be used.

So there are three file systems under Linux that can do boot environments. ZFS, Btrfs/LVM and Ostree/XFS. Ostree/XFS you see in Fedora project silverblue.

Really ZFS does not have as many advantages as one would think other than its per block protection. XFS being a part CoW file system with reflink means with software like ostree on top it can pull large number of the ZFS tricks but with higher overall performance.

[k1e0x]

Either my writing skills have massively deteriorated or your reading skills (intentionally or unintentionally) are lacking.

1. ZFS on Oracle (Be that Solaris or Unbreakable Linux) != ZOL.
One is *supported* by a major enterprise-grade OS supplier the other, is, well, not.
Comparing the two is humorous, at best.
Most of the ZFS on Linux users don't bother to pay the huge sums of money required to get proper support.
And if it breaks (see 3) they will get to keep all the pieces.

2. ext4 *is* support by enterprise-grade OS supplier (Be that RHEL, SUSE or Ubuntu).
Good luck trying to get support for your RHEL data cluster if you use anything ZFS.
Same goes for any other enterprise grade Linux distribution (beyond Oracle, that is, see 1).

3. Copy on write my protect you from single-bit-curroption. It *does* not protect you from bugs.
Having just witnessed a 0.8PB cluster (w/ ZFS) go up the flames, I'm not that impressed.

Again, if you use ZFS on a support platform, good for you.
If you are stupid enough to use ZOL on an unsupported platform thinking that copy-on-write might save your ass, think again.

- Gilboa

1. I have never seen Oracle ZFS storage appliance used in the enterprise personally, I know they exist I've just never seen them. Only Sun Microsystem's before Solaris version 10. (Pool version 28~) Generally it's used on FreeBSD storage clusters that are secondary storage to NetApp, EMC or DDN. (around 32-64 spindles) You can get paid commercial ZFS (and ZoL) support from both FreeBSD 3rd parties and Canonical. Ubuntu 19.10 has ZFS on root in the installer and so will 20.04-LTS

2. All I got to say about ext4 is hope you like fsck.

3. You don't know what you're talking about. COW alone has nothing to do with bit-rot or uncorrectable errors. You're thinking of block checksums, and yes, they are good. COW provides other features such as snapshots, cloning and boot environments. Boot environments are pretty cool.. maybe Linux should get on that... oh wait.. ZFS is the only Linux file system that does it and we can't have *that*.

Check this out.. FreeBSD 12 has a new command for boot environments..
https://www.freebsd.org/cgi/man.cgi?query=bectl

bectl create [email protected]
bectl jail [email protected]

You just cloned your running OS into a writable, bootable, virtual environment and spawned a shell in it and it was instant.

[gilboa]

I can assure you ZoL *IS* used in enterprise. I've used ZFS in enterprise since ~2006 on Solaris, FreeBSD and Linux. It is pretty much the fix for ransomware.

The CDDL nor the GPL have restrictions on usage, only on distribution. You can and always have been able to do whatever you want on your own systems.

I posted this before. 100 PB - ZoL Implementation. Storage of that size actually isn't that unique for ZFS (typically more Solaris and FreeBSD but ZoL is getting more popular.) Most enterprises don't share information on their storage platforms. Since that one is a quite impressive research system they do have details posted about it.
https://medium.com/codedotgov/oss-sp...x-6596fca6e5f6

"unlike ext4/EXFS, when ZOL breaks (and I've seen it break badly in production environment, including very recently) you get to keep all the pieces." -- seriously? You're going to make the claim ext4 is *better* at data integrity than ZFS? No.. It's not. Not in any universe. Like Linus you don't seem familiar with ZFS.. that is ok.. but unlike Linus you shouldn't go making claims when you don't have the basic facts.

FYI, ZFS is a copy on write, always consistent file system with a checksum on every data block.
EXT4 is... not. It is a journaled, extent based file system that updates blocks in place making it not always consistent. Do you like fsck? Or data blocks with silent corrupt files? EXT4 is for when you want a small but present possibility of your pictures looking like this and never knowing till you open it.

Either my writing skills have massively deteriorated or your reading skills (intentionally or unintentionally) are lacking.

1. ZFS on Oracle (Be that Solaris or Unbreakable Linux) != ZOL.
One is *supported* by a major enterprise-grade OS supplier the other, is, well, not.
Comparing the two is humorous, at best.
Most of the ZFS on Linux users don't bother to pay the huge sums of money required to get proper support.
And if it breaks (see 3) they will get to keep all the pieces.

2. ext4 *is* support by enterprise-grade OS supplier (Be that RHEL, SUSE or Ubuntu).
Good luck trying to get support for your RHEL data cluster if you use anything ZFS.
Same goes for any other enterprise grade Linux distribution (beyond Oracle, that is, see 1).

3. Copy on write my protect you from single-bit-curroption. It *does* not protect you from bugs.
Having just witnessed a 0.8PB cluster (w/ ZFS) go up the flames, I'm not that impressed.

Again, if you use ZFS on a support platform, good for you.
If you are stupid enough to use ZOL on an unsupported platform thinking that copy-on-write might save your ass, think again.

- Gilboa

[oiaohm]

k? I'm sure he is. It's odd you talk about the "XFS guy" .. where as ZFS is a community of paid developers, is only one person working on improving XFS? That is no good.

https://lwn.net/Articles/796847/
There is the XFS project lead who you see in that video laying out a plan then multi different personal including Oracle staff bring it into reality.

I did watch this on the XFS design and plans.
Teaching an old dog new tricks. https://www.youtube.com/watch?v=wG8FUvSGROw

So you have watched it.

Talking about loopback devices cp and tar... and showing scripts that implement this.. it's ugly.

That was only the prototype. iomap complete no loopback devices in play

He also has a small mistake in that ZFS does not cache the file "bash" from 1000 clones.

That was not refering ZFS this is your expertise getting in the way its referring the issue btrfs and it interaction with the VFS layer above it. It also something that could happen with namespaces as well with ZFS.

The ARC only works on blocks, 1000 clones of bash if unmodified share the same blocks. The ARC also isn't a LRU cache like most other filesystems. It's why benchmarking ZFS isn't really fair if you defeat it's cache, because the ARC is integral to it's use and it actually works, unlike LRU.

iomap work is to bring ARC like block management to all Linux kernel supported filesystems in time. Also provide a route so this declone at block layer is not undo by something at the VFS layer. Something you forget Linux kernel memory management above ZFS when running on Linux is also a LRU. Some of the work of iomap is to fix the Linux kernel memory system.

I hope ARC is ready to handle 2Meg blocks and other horrible caused by the up coming Linux kernel large pages.

[k1e0x]

Are you sure. You thought Linux blocking fpu access for ZFS was bad. Now think what you are going todo if you wake up in a world that all Linux file systems have to be iomap based.

XFS developer is up-to something that should have you ZFS users and developers kind of worried.

There is a slight performance hit on kernels above 5.0 nothing in enterprise uses 5.0 that I'm aware of most recent stuff is like 4.19-LTS. I haven't noticed any real performance hit.. It preforms well 5.0 and above depending on your pool layout.

k? I'm sure he is. It's odd you talk about the "XFS guy" .. where as ZFS is a community of paid developers, is only one person working on improving XFS? That is no good.

I did watch this on the XFS design and plans.
Teaching an old dog new tricks. https://www.youtube.com/watch?v=wG8FUvSGROw

And I have some thoughts on this. In the early part when he's explaining "cow filesystems" He's talking directly about ZFS's design (leaf-to-root) and not about other implementation and he sounds to me like he has a lot of respect for that design. Part of the magic of ZFS is when it walks up the tree to update it's pointers it also adds it's checksums, it's snapshot references and other info all in one go. (I think this is rather elegant personally, Sun used a "design negative" in theory to add a lot of positive features in a seamless way.. almost like "well.. since you're there anyway.." and in the end it ends up being a performance boost as opposed to other methods trying to implement the same thing)

Then he seems sad and lists his problems going into replicating that feature set and really seems to be grasping as straws. Talking about loopback devices cp and tar... and showing scripts that implement this.. it's ugly. He also has a small mistake in that ZFS does not cache the file "bash" from 1000 clones. The ARC only works on blocks, 1000 clones of bash if unmodified share the same blocks. The ARC also isn't a LRU cache like most other filesystems. It's why benchmarking ZFS isn't really fair if you defeat it's cache, because the ARC is integral to it's use and it actually works, unlike LRU.

...You know I'm left with the feeling here.. That it's sad seeing really smart people try to jump over hurdles and work so hard to do this stuff. I really think he should take the old dog out and shoot it.. and if he thinks ZFS is a no go due to license.. alright, fine if you have that opinion ok.. but instead of all this stuff he should start working on bcachefs or HAMMER2. He needs a new design because when you start jumping to the point of a loopback device.. you might as well just start over. You sort of are anyhow and anything you come up with is going to be a hack.

Idk tho.. maybe I should be "worried" for totally vague reasons. Sorry.. just really not "worried" after watching that. just sad for him.

[oiaohm]

The rest of us who actually have work to do already have one.

Are you sure. You thought Linux blocking fpu access for ZFS was bad. Now think what you are going todo if you wake up in a world that all Linux file systems have to be iomap based.

XFS developer is up-to something that should have you ZFS users and developers kind of worried.