OpenZFS 2.1.8 Released With Linux 6.1~6.2 Compatibility Updates, Bug Fixes

[Developer12]

Use Btrfs for such a use-case on Linux. If your system is bogged, Btrfs driver lives in your kernel, while ZFS is out-of-kernel. I believe Suze offers something you described... not sure though (I'm on Arch). If you insist on ZFS (for good reasons) consider OmniOS or OpenIndiana who have similar features as you describe. Those OpenSolaris-derived systems have ZFS in-kernel.

In kernel or out of kernel doesn't matter here either.

A kernel is useless without an initramfs anyway, which has your shell and other utilities. The solution is simply to include a ZFS module in the initramfs, just as you would include a driver module for a storage controller. Somehow people forget that a lot of the modules for their system already live there.

[ryao]

ryao,

Sorry for bothering you but I have a question that you could maybe answer. Is OpenZFS going to implement something equivalent to DirectStorage or WTF ever the PS5 uses to load games fast off an NVMe? Some of these new and upcoming games with ridiculous ram requirements have me wondering if OpenZFS will have a solution for that. Maybe using an 100+GB NVMe used as a ZIL+L2ARC?

It is not a bother.

DirectStorage is a Microsoft library for loading compressed assets into GPU memory efficiently. It does that through existing system APIs, so there is nothing new for a filesystem to implement in order to support for it. The descriptions that most websites give about the GPU cutting the CPU out of the storage stack with DirectStorage are untrue and as far as I have heard from the proton developers, the Windows DirectStorage library works fine in Wine. In fact, one of the Proton developers (Josh) uses ZFS, so I imagine if it were not working on ZFS, I would have heard about that it from him.

As for what the PS5 uses, it appears to use a special microcontroller below UFS that does compression/decompression. The magic is all in that microcontroller and the use of unified memory. There is not much for the filesystem to do there, since it is oblivious to all of this.

That said, I realized a few years ago that games were already doing asset streaming. A few games that already do asset streaming are Fortnite (from before the UE5 switch), Zelda BoTW and Star Citizen. If you manage to teleport somewhere in the game that it does not expect, you will probably notice assets suddenly popping into view (which is really jarring in star citizen, since walking out of an elevator can make you feel that you just walked into space). Newer games will just be doing more of this and supporting the bandwidth needed to make it work is more of a matter for hardware than for the filesystem.

As far as what the filesystem can do to help here, there is the general performance work that we are already doing, but I do not expect it to influence games very much. Game developers will likely begin asynchronous assert loads in the background at least half a second early, and that will likely be enough to support a wide assortment of storage configurations. In such a situation, no one is going to care how the filesystem did things as long as they are loaded in time.

[q66_]

This is because you have not understood the fundamental failure. ZFS being non mainline kernel module has to be in the initramfs. So when the ZFS package updates for the module so does the initramfs for all installed kernels including the backup kernels. Yes mainline file system you have multi kernels with different versions of your file system drivers that are in fact developed in isolation to each other.

in chimera, which the OP on first page was asking about, this is almost never going to be a problem in practice for several reasons:

1) the distribution kernels come accompanied with packages having prebuilt zfs modules for said kernel (so in that way it's not much different from any other module that needs to be included in initramfs, which includes most in-tree filesystems)
2) custom kernels that need zfs built from source still partially mitigate the problem, because chimera does not use DKMS; instead, a custom (and more robust) system for kernel module management is in place; you can have kernels with binary ZFS and kernels with source ZFS in one system (because kernels that have binary ZFS have special suppression files installed, which prevent ZFS from being managed in source form for that kernel)
3) the kernel packaging in the system has automatic backups on kernel updates; the backed up kernels are still bootable, but the hooks are written in a way that none of the packaging will touch the kernel's modules or initramfs, so if you update your kernel, your older kernel will still be available, effectively frozen in time, until you delete it (which means you can restore that way)

and... even if all that fails and you still end up with unbootable system: you can simply boot the live media and restore your system from there (as they come with ZFS support out of box)

that said, ZFS is still just one of the options to choose for the root filesystem, so nobody is forced into it (it's the user's choice to make)

[oiaohm]

A kernel is useless without an initramfs anyway, which has your shell and other utilities.

https://blog.fpmurphy.com/2013/08/boot-linux-without-an-initramfs-2.html

Pure falsehood that linux is useless without initramfs Developer12 and has been for basically forever.

There are cases where initramfs is forced. Using LVM to have LVM tools does force initramfs. If you are using native btrfs or ext4 or xfs without LVM it is possible to boot system without initramfs.

Also it possible that initramfs is not it own file on disc but instead has been embedded into kernel. Yes some distributions do this for their rescue kernels There is a problem here of course OpenZFS is a incompatible license so this option is off the table. Now this comes a bigger problem as described latter based on how Linux distributions manage initramfs updating.

1) the distribution kernels come accompanied with packages having prebuilt zfs modules for said kernel (so in that way it's not much different from any other module that needs to be included in initramfs, which includes most in-tree filesystems)

Most in-tree file systems are not factor here. Recommend distribution installation file systems the modules are normally built into the kernel images. If you have done that not using LVM then the system can be booted without the initramfs with most distributions.

Yes ext4, btrfs and xfs with distributions that support these as boot partitions end up built into kernel most commonly.

Not all modules that you put in initramfs are created equal this will come clear latter.

2) custom kernels that need zfs built from source still partially mitigate the problem, because chimera does not use DKMS; instead, a custom (and more robust) system for kernel module management is in place; you can have kernels with binary ZFS and kernels with source ZFS in one system (because kernels that have binary ZFS have special suppression files installed, which prevent ZFS from being managed in source form for that kernel))

This is just extra failure points.

3) the kernel packaging in the system has automatic backups on kernel updates; the backed up kernels are still bootable, but the hooks are written in a way that none of the packaging will touch the kernel's modules or initramfs, so if you update your kernel, your older kernel will still be available, effectively frozen in time, until you delete it (which means you can restore that way)

I did not say the problem happened with kernel updates. Something causes regeneration of all the initramfs that can be there are no backup initramfs copies made.

Why did I say all initramfs that can be. It is possible for initramfs to be embedded in the kernel image. Of course this contains the problem that all contents have to be license compatible with the Linux kernel. Yes some distributions have rescue kernels where those particularly have fixed time frame initramfs built into them but this is not possible with ZFS due to license. Yes fixed time frame initramfs build into kernel is possible with lvm usage because its tools are license compadible.

Not being Linux kernel compatible prevents a fail safe option.

Fail safe options
1) No Initramfs boot. This has limitation like no LVM as well.
2) Initramfs embedded in kernel image so this is not updated with packages updating initramfs files. Not possible with ZFS due to OpenZFS license issue. But any linux mainline feature/file system this is legally doable.

You have missed to have older initramfs still around after updates effecting initramfs the initramfs had to have been embedded in the kernel image by most Linux distribution designs. This is the very thing OpenZFS is incompatible with due to license.

initramfs management is weakness in Linux Distribution designs. OpenZFS being out of tree file system is depending on this weak point and then OpenZFS license removes option to keep around old initramfs image. This is why OpenZFS commonly turns out badly for Linux users that you can see in many distributions issue lists.

Yes the initramfs management design of the Linux Distributions was designed around the concept there would be rescue Linux kernels provided by the distribution with emergency initramfs embedded. Yes this is fault and a half. Initramfs management has not been updated for not having rescue kernels this results in items altering initramfs bricking systems on those distributions. Rescue linux kernels historically depend on everything in initramfs being Linux kernel license compatible because the initramfs would be made part of the kernel image.

and... even if all that fails and you still end up with unbootable system: you can simply boot the live media and restore your system from there (as they come with ZFS support out of box)

So you are saying a person has to carry their device around with back up live media all the time.

that said, ZFS is still just one of the options to choose for the root filesystem, so nobody is forced into it (it's the user's choice to make)

That does not make it a good option. Also many distributions don't provide ZFS as a root filesystem option and many if they do like nixos they don't recommend using it.

Is ubuntu initramfs management really suitable to be using OpenZFS as boot or root partition the answer is in fact no it not. Yes the management pattern of initramfs is the old one that your safe guard is rescue kernels with snapshots in time of the initramfs built in. Yes failure of initramfs resulting in having to get to live media to fix is in fact a symptom of a core problem in initramfs management. There are actions that can make you risk of being on the bad side of this higher.

Yes you might now claim this is not OpenZFS fault. Issue here where have been the OpenZFS people pushing for the initramfs management to be fixed up so that users wanting to run root/boot ZFS are not walking themselves into trouble. There is way too much of you can just use live media to fix this. Yes this is like car with one tyre that is tread less fine as long as the road stays dry problem rain has to happen.

OpenZFS developers and users have not noticed:
1) How few things touch the initramfs.
2) As the number of items causing alterations to initramfs go up reported cases of bricked system with Linux distributions go up.
3) Have not noticed that the initramfs management design has fundamental limitations.
4) Initramfs management design mandate rescue kernels with embedded initramfs that distributions no longer do so massively increasing risk when altering initramfs.
5) OpenZFS is not compatible with embedded in kernel initramfs that is something that is required with the current initramfs management design if you follow it to be durable.

OpenZFS has integration problems like it or not. Some of these integration problems happen to run into common linux distribution flawed design..

[q66_]

...

considering that the majority of distribution kernels are *not* capable of booting without initramfs kernel modules (and a rescue kernel with embedded initramfs is something practically nobody does) even without zfs makes pretty much everything you said a moot point

[oiaohm]

considering that the majority of distribution kernels are *not* capable of booting without initramfs kernel modules (and a rescue kernel with embedded initramfs is something practically nobody does) even without zfs makes pretty much everything you said a moot point

This statement of yours is not as true as what you would think. Its not black and white answer. I am sitting in debian testing with a ext4 root partition using GPT partition table. All the initrd contains is CPU microcode. My system boots ok without the CPU microcode from the initrd. Yes it will proceed to pick the microcode up from the root partition once it gets that far. Yes I could have a install using xfs or btrfs installed not using LVM and have my initrd be the same where it totally optional with debian.

Of course if this system was debian using LVM then the initrd would be more complex.

Please note its not just debian that able todo this but the Ubuntu kernel as well and all the distributions based of these kernels. Then your fedora and SUSE kernels are also able to fire up without initrd.

Majority of linux distribution kernels are in fact able to boot without initramfs/initrd. The limitation its not all configurations and there is some risk of issues due to late loading of cpu microcode. Using LVM that makes you have initrd with more in it and increased risk of failure. OpenZFS increased risk of failure same reason.

I really don't know where the idea that the initramfs has to have kernel modules thinking that is not the common. LVM kernel module with debian is built into the kernel. LVM initramfs/initrd is the userspace tools to configure up everything.

OpenZFS is the most common case for initramfs/initrd to contain kernel module.

q66 you really need to open up your initrd files and take a look inside and notice that normally kernel modules are not in there. Distributions build the core file systems into the kernel image itself. Yes lsmod still lists them as modules even if they were built into the kernel image.

q66 there was a change in 2011 when the Linux kernel got direct processing of UUID numbers for partitions this resulted in the initrd coming way simpler because before that change UUID and label processing use to have to be a script thing in the initrd/initramfs. Yes my debian initrd here does not even have a shell it is completely tool less.

q66 the claim that distributions are not capable of booting without initramfs that disappeared after 2011.
https://lore.kernel.org/lkml/[email protected]/
Yes the disappearance of need for initramfs for distribution kernel to find root partition and fire up system happens because of Chrome OS work.

q66 the lie that distrobution kernels cannot boot without initramfs needs to stop that not been the case for majority of distributions for a decade now for many install configurations. LVM lot of ways like the UUID work to make the kernel be able directly handle that case should happen its a defect that to use LVM that you need initramfs.

The reality other than CPU microcode majority of the time of the initramfs contains anything its normally sign of missing feature in the Linux kernel(like LVM setup) or very hack work around like OpenZFS.

https://docs.kernel.org/admin-guide/...r/dm-init.html there are a lot of things you can bring up these days by the Linux kernel command line.

Yes even LVM you can with Linux kernel alone without initrd/initramfs and the right command line get the system booted. Lot of cases not safe because you need userspace tools to perform the safety checks.

q66 see the big difference here. XFS, ext4, btrfs and lvm snapshots if the initrd is dead means to get into your system is not gone. Yes even encrypted partitions you don't in fact need initrd to bring those up.

Linux kernel is way more functional. Its a rare case that you find mainline kernel drivers in the initrd/initramfs.

[Velocity]

From afar i can see three camps:

The first camp are those who discovered ZFS and simply love it! The few disadvantages and quirks that ZFS has are easily circumvented if you like ZFS enough. Their users are keen to promote the advantages of ZFS to others.

The second camp is the GPL camp who feel endangered somehow that something non-GPL and even incompatible-with-GPL (for binary distribution) could become so popular as to threaten the hegemony of the GPL ecosystem of Linux operated systems. It appears to me they perceive it as a threat that needs to be dealt with. Even Linux's second-in-command appears suspect in a controversial anti-ZFS move by removing functions in the kernel that ZFS relied on. The details do not matter. I sense distrust and unwillingness to make all parties happy. If you like Btrfs why would you intentionally want to cripple ZFS for other users? I sense bitterness and negative emotions. I can recall the same kind of discussions about LLVM/Clang and perhaps the FreeBSD OS in general, also on this forum in the past. Supremacy might indeed be the best word to describe it.

The third camp is simply pragmatic, users who do not know what to use and some things they hear about ZFS are good, while counter arguments may sway them away from trying out ZFS altogether. I think this group deserves more attention because i feel it is sad if ZFS could be beneficial to their situation. Likewise, ZFS fans may over-emphasize the potential benefits of ZFS while in the specific user's situation these wouldn't be crucial and the user might be better off using ext4 or btrfs instead.

My own feelings: we should respect each other's differences and should welcome multiple philosophies and approaches. That is what i like about the open source world: that i have choice! I don't like systemd? Then i won't use it. I don't like snaps? Idem dito. Well.. Ubuntu played tricks on me, but yeah.

Anyone fiercely against ZFS or any other technology should understand that for other users it serves them just fine. What is good for you may not be the best choice for others. That is why we have supermarkets so we can choose ourselves the food we like, instead of having the same carrots being delivered to your doorstep every day. Celebrate choice - it is the cornerstone of a free and open source software community!

[oiaohm]

The first camp are those who discovered ZFS and simply love it! The few disadvantages and quirks that ZFS has are easily circumvented if you like ZFS enough. Their users are keen to promote the advantages of ZFS to others.

These also have the habit of over looking the flaws as well. Like the repeating case of initramfs/initrd failures. And the legal risks.

The second camp is the GPL camp who feel endangered somehow that something non-GPL and even incompatible-with-GPL (for binary distribution) could become so popular as to threaten the hegemony of the GPL ecosystem of Linux operated systems. It appears to me they perceive it as a threat that needs to be dealt with. Even Linux's second-in-command appears suspect in a controversial anti-ZFS move by removing functions in the kernel that ZFS relied on. The details do not matter. I sense distrust and unwillingness to make all parties happy. If you like Btrfs why would you intentionally want to cripple ZFS for other users? I sense bitterness and negative emotions. I can recall the same kind of discussions about LLVM/Clang and perhaps the FreeBSD OS in general, also on this forum in the past. Supremacy might indeed be the best word to describe it.

https://www.fsf.org/licensing/zfs-and-linux The reality is every time CDDL and GPLv2 have been properly reviewed they are incompatible licenses. This incompatibility limits how a GPLv2 and CDDL work can interact with each other. . Next is Linux kernel has stacks of patented technologies that are licensed to be used with GPLv2 works.
EXPORT_SYMBOL_GPL thing predates OpenZFS existing yes 2005 for the EXPORE_SYMBOL_GPL thing. The idea that OpenZFS was being special singled out is not the case and was not the first case either.

Symbols being converted to EXPORT_SYMBOL_GPL its fairly critical to understand the why. Some cases can be horrible. The functions OpenZFS was using happened to be modified to be faster using a method that is in fact patented and only licensed to be used under the Linux kernel GPLv2 license. This had happened before back in 2006 with niswrapper as well does make things tricky when it does happen. OpenZFS/ZFS people thought this had be anti-ZFS or anti-third not GPL not that there is a legal problem that will keep on happening. Having a kernel module for Linux that is not GPLv2 or GPLv2 compatible license at times you are going to have the nightmare that some function will be tagged that you cannot use it from time to time. The way functions are coded in the Linux kernel change and what patents they use change with these changes and this results in legal usage change.

https://lwn.net/Articles/114839/ there is another problem with CDDL the wording of 6.4. Lets say one of the patent holders who license patents to the Linux kernel as GPLv2 who decide to sue the distributor who you got Linux kernel and OpenZFS there is risk of odd termination because patent holder could be a Linux kernel contributor so you and your distribution have got licensed code from them so the CDDL license terminates because you are one of the attacking parties distributors or got your copy from them.

So EXPORT_SYMBOL_GPL can be pain but bad news here its require pain. But the pain if it not put on something patent protected and legal case starts is way worse for the CDDL licensed module as many users could have their right to use the CDDL terminated.

.Yes you have missed a critical camp. Not anti-ZFS as such but the legal camp. The legal camp say CDDL kernel module interfacing with the Linux kernel should expect problems every so often due to the license difference and the way different patents have been licensed to the Linux kernel for fee less usage. Some of this is going to be changes Linux kernel developers have to-do that break the CDDL module but also prevent legal actions that could open a really bad Pandora box.

Yes admit the legal camp exists is not something those pushing ZFS want to accept is the case. Yes accepting this is the case also means accepting that ABI/API breakage from the Linux kernel from time to time is going to be status normal because OpenZFS is CDDL,. A patent case coming from a party who licensed something to the linux kernel who does not use ZFS could result in the worse out come for many OpenZFS users if the EXPORT_SYMBOL_GPL stuff is not done. So software breakage and developer time cost vs people legally not being able to use OpenZFS if something blows up. Choose you poison here.

The third camp is simply pragmatic, users who do not know what to use and some things they hear about ZFS are good, while counter arguments may sway them away from trying out ZFS altogether. I think this group deserves more attention because i feel it is sad if ZFS could be beneficial to their situation. Likewise, ZFS fans may over-emphasize the potential benefits of ZFS while in the specific user's situation these wouldn't be crucial and the user might be better off using ext4 or btrfs instead.

You see it with ZFS fans over and over again where they have not going systematically looking for defects. Yes just like q66 and developer12 here making claim that Linux useless without initramfs to start this is a pure false claim. Majority of Linux todays distribution installs can start up with default kernel with initramfs missing. False claim means they don't have to look at we do boot and root ZFS we are now forcing a breaking point in initramfs usage that we don't tell users to mitigate. Yes special effort of making recovery kernel entry in grub that has made a copy of the initramfs out side the normal distribution update system so when the distribution updating breaks the initramfs users is not locked out of their system. Why is this mitigation not required with default ext4 and btrfs and xfs because you can boot into those without initramfs at all.

History method when Linux booting did depend on initramfs that over 10 years ago was to make recovery kernels with initramfs/initrd embedded so distribution updating could not alter this copy of initramfs/initrd so countering initrd/initramfs breakage by updates of course this is not going to work with OpenZFS CDDL license due to it being GPLv2 incompatible. There are places where OpenZFS has skipped out on doing the correct integration work and documenting hang on this is a problem that need to be mitigated. Yes new mitigation method need for the initrd/initramfs problem is need all because of the CDDL license problem means the prior solution of mainline kernel expand features so that kernel alone can start system or embedded(initramfs/initrd) into kernel cannot legally work.

This is not the only place OpenZFS happens to have bad integration but the initramfs/initrd is one that should have been fixed and should have been spotted. Yes myth that Linux kernel needs initramfs/initrd to boot prevented seeing that failures in initrd should not be user locked out of system to the point of needing external media to solve the problem if there is there is lack of migitation against this problem.

[skeevy420]

It is not a bother.

DirectStorage is a Microsoft library for loading compressed assets into GPU memory efficiently. It does that through existing system APIs, so there is nothing new for a filesystem to implement in order to support for it. The descriptions that most websites give about the GPU cutting the CPU out of the storage stack with DirectStorage are untrue and as far as I have heard from the proton developers, the Windows DirectStorage library works fine in Wine. In fact, one of the Proton developers (Josh) uses ZFS, so I imagine if it were not working on ZFS, I would have heard about that it from him.

As for what the PS5 uses, it appears to use a special microcontroller below UFS that does compression/decompression. The magic is all in that microcontroller and the use of unified memory. There is not much for the filesystem to do there, since it is oblivious to all of this.

That said, I realized a few years ago that games were already doing asset streaming. A few games that already do asset streaming are Fortnite (from before the UE5 switch), Zelda BoTW and Star Citizen. If you manage to teleport somewhere in the game that it does not expect, you will probably notice assets suddenly popping into view (which is really jarring in star citizen, since walking out of an elevator can make you feel that you just walked into space). Newer games will just be doing more of this and supporting the bandwidth needed to make it work is more of a matter for hardware than for the filesystem.

As far as what the filesystem can do to help here, there is the general performance work that we are already doing, but I do not expect it to influence games very much. Game developers will likely begin asynchronous assert loads in the background at least half a second early, and that will likely be enough to support a wide assortment of storage configurations. In such a situation, no one is going to care how the filesystem did things as long as they are loaded in time.

Not sure why, but I always thought there was something going on that the FS level combined with an NVMe whenever I read about DirectStorage/PS5 in the past. Thanks for clearing that up for me.

Two last questions. Since it seems like a lot of this gaming stuff comes down to having fast enough storage, say I added a 512GB NVMe as a L2ARC to my 3x4TB (8TB) RAIDZ that contains games and random crap, is there a way to ensure a specific game and prefix would be there instead of random crap or would I just have to play a game once or twice to warm up the cache? Would there be much of a point, performance or other, in using 5-10% of that 512GB L2ARC NVMe as a ZIL/SLOG?

[phoenix_rizzen]

I am curious to know what people think about using OpenZFS as root system. How does it compare to BTRFS? I am quite curios to try Chimera Linux (musl libc, LLVM/Clang system compiler with libc++ and FreeBSD userland tools), which include OpenZFS out-of-the-box.

I just wonder if I should try and go "all in" and also try to set it up on ZFS

Been using ZFS for root filesystems at home and work ever since it was first supported for booting in FreeBSD ... 8? Maybe 9? (For data pools, we've been using it since ZFSv6 hit FreeBSD 7, booting off gmirror setups.) For servers at work (running FreeBSD), it's separate root and data pools, using a single mirror vdev (on SSDs) for the root pool, and multiple raidz2 vdevs (SATA HDs) for the data pool. At home, it's a mirror vdev (NVMe) in the root pool. and mirror vdevs (SATA HDs) in the data pool.

At home, it started as a FreeBSD system, but I switched to Ubuntu 22.04 last year to get better Plex support/features. Data pool carried over without issues.

With Linux, you just have to make sure your OpenZFS, kernel, and GRUB (or other boot loader) packages stay in sync; which the distro package team should take care of for you. Just don't go installing random kernel packages or compiling your own without putting in the due diligence to make sure it works.

Best way to get useful experience with a piece of software is to use it daily. What better way to learn about ZFS than to use it on your local system? Just be sure to have a working backups system, as you're bound to mess things up a few times while you get used to the command syntax. And that goes for any filesystem you're using for the first time. I've screwed up ext*, XFS, and JFS filesystems many times in the past before I understood how the mkfs/fsync commands worked.