Ubuntu 24.04 Supports Easy Installation Of OpenZFS Root File-System With Encryption

[Classical]

Why are people using Ubuntu? Void is the distro to use with ZFS.

Void takes very long when it builds ZFS for the kernel.

Alpine Linux is the fastest and most reliable ZFS implementation in the Linux world. It completely installs an new ZFS version in 3 seconds.

ZFS also works fine on Calculate Linux and Gentoo.

ALT Linux has a new ZFS implementation and ROSA Linux has an old ZFS version.

A few other Linux systems also offer ZFS: https://openzfs.github.io/openzfs-do...ted/index.html

Btrfs is kind of a failed project, this is why Red Hat abandoned Btrfs.

I will say ZFS and Hammer2 are the 2 most reliable file systems under heavy load.

But if you don't believe me, even Red Hat admitted that Btrfs is not suitable for professional usage.

It has performance an reliability issues in most RAID modes.

[pWe00Iri3e7Z9lHOX2Qx]

It is not.



The only thing that use LUKS is a small zdev. No ZPOOL use a LUKS encrypted device.



It has not. You are right. The ZFS installation on Ubuntu 24.04 (tried yesterday on a VM) is the same as it was on 22.04. Same LUKS-encrypted keystore zdev. Same limitations.



Not sure. But the kernel and initrd are not encypted. You can use everything LUKS supports. But it will happen in the initrd, not in grub.

The way Ubuntu set up ZFS with encryption is the following:

• 2 zpool: a small one for grub boot environment (not sure why do you need this...), a main one for everything else;
• the installer will use 1 disk, partitioning it in 3(no partition is LUKS-encrypted): one partition for bootloader(bios)/esp(uefi), one small partition as a boot zpool vdev, one big partition as a main/root zpool vdev;
• the boot zpool is not encrypted
• the main zpool contains a small zdev called

  keystore

  that is LUKS-encrypted and contains a small ext4-formatted file system with a single key file
• the root filesystem in the main zpool (and all other ones created by the Ubuntu installer) are encrypted using native zfs encryption with the key saved in the keystore

The boot process is the following:

• grub boots from the boot zpool, unlocking the encryption is done by scripts in the initramfs
• initramfs scripts import the main zpool
• initramfs scripts decrypt

  /dev/zd0

  into

  /dev/mapper/keystore-rpool

• initramfs scripts mount

  /dev/mapper/keystore-rpool

  at

  /run/keystore/POOLNAME/

  and load the key
• the root filesystem is mounted

IMHO I think Ubuntu developers came up to this idea to overcome the limitations of zfs native encryption: with a LUKS-encrypted keystore you can support every LUKS supports(passwords, TPM2, tokens, ....) and still have a zfs native encrypted sytem.

Yup. To illustrate further...

nuke@nuke-Standard-PC-Q35-ICH9-2009:~$ zfs get encryption,keylocation
NAME                                              PROPERTY     VALUE                                  SOURCE
bpool                                             encryption   off                                    default
bpool                                             keylocation  none                                   default
bpool/BOOT                                        encryption   off                                    default
bpool/BOOT                                        keylocation  none                                   default
bpool/BOOT/ubuntu_3dw9v9                          encryption   off                                    default
bpool/BOOT/ubuntu_3dw9v9                          keylocation  none                                   default
rpool                                             encryption   aes-256-gcm                            -
rpool                                             keylocation  file:///run/keystore/rpool/system.key  local
rpool/ROOT                                        encryption   aes-256-gcm                            -
rpool/ROOT                                        keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9                          encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9                          keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/srv                      encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/srv                      keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/usr                      encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/usr                      keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/usr/local                encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/usr/local                keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var                      encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var                      keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/games                encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/games                keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/lib                  encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/lib                  keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/lib/AccountsService  encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/lib/AccountsService  keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/lib/NetworkManager   encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/lib/NetworkManager   keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/lib/apt              encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/lib/apt              keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/lib/dpkg             encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/lib/dpkg             keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/log                  encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/log                  keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/mail                 encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/mail                 keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/snap                 encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/snap                 keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/spool                encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/spool                keylocation  none                                   default
rpool/ROOT/ubuntu_3dw9v9/var/www                  encryption   aes-256-gcm                            -
rpool/ROOT/ubuntu_3dw9v9/var/www                  keylocation  none                                   default
rpool/keystore                                    encryption   off                                    default
rpool/keystore                                    keylocation  none                                   default

[royce]

It is not.



The only thing that use LUKS is a small zdev. No zpool use a LUKS encrypted partition.



It has not changed. You are right. The ZFS installation on Ubuntu 24.04 (tried yesterday on a VM) is the same as it was on 22.04. Same LUKS-encrypted keystore zdev. Same limitations.



Not sure. But the kernel and initrd are not encypted. You can use everything LUKS supports. But it will happen in the initrd, not in grub.

The way Ubuntu set up ZFS with encryption is the following:

• 2 zpool: a small one for grub boot environment (not sure why do you need this...), a main one for everything else;
• the installer will use 1 disk, partitioning it in 3 (no partition is LUKS-encrypted): one tiny partition for bootloader(bios)/esp(uefi), one small partition as a boot zpool vdev, one big partition as a main/root zpool vdev;
• the boot zpool is not encrypted
• the main zpool contains a small zdev called keystore that is LUKS-encrypted and contains a small ext4-formatted file system with a single key file
• the root filesystem in the main zpool (and all its children created by the Ubuntu installer) are encrypted using native zfs encryption with the key saved in the keystore

The boot process is the following:

• grub boots from the boot zpool, unlocking the encryption is done by scripts in the initramfs
• initramfs scripts import the main zpool
• initramfs scripts decrypt /dev/zd0 into /dev/mapper/keystore-POOLNAME
• initramfs scripts mount /dev/mapper/keystore-POOLNAME at /run/keystore/POOLNAME/ and load the key
• the root filesystem and all children are mounted

IMHO I think Ubuntu developers came up with this idea to overcome the limitations of zfs native encryption: with a LUKS-encrypted keystore you can support everything LUKS supports(passwords, TPM2, tokens, ....) and still have a zfs native encrypted system.

Thank you for confirming. Canonical should really consider looking into zfsbootmenu, a lot of that complexity is moot if you just cut grub out of the equation, including zsys, although I'm not sure what can be done about tpm2 integration

[pWe00Iri3e7Z9lHOX2Qx]

Also something worth mentioning, in the past you could just edit a config (think it was for zsys) in the live installer environment and set up whatever ZFS options you wanted there and the installer would blindly obey. In one of the last releases, the file was still the same in the old installer versions, but had changed / moved in the newer flutter based installers. When I have some time I'll poke around and see where the installation defaults are coming from in 24.04.

[kengreen]

ZFS (and BTRFS and BcacheFS) benefits are easy creation of striped, mirror or other RAID topology. How easy is it to create a mirror on Ubuntu 24.04 after creating the single disk ZFS? Is it just a matter of copying the partitions from one disk to the other, then attaching the bpool and rpool partitions and allowing resilvering to finish?

[Paradigm Shifter]

Nice to see that support isn't dropped. Admit I was a little worried. I use ext4 for root because it's a lot simpler to set up, but ZFS for data arrays (SSD and HDD ZFS RAID)...

[RedEyed]

I used zfs on rpi4 for external USB 6 TB drive and almost lost my data.

After 3 years of using it, the pool got into unrecoverable error. I noticed it, when drive was unmounted, and dmesg showed zfs backtrace.


I scrapped pool many times, which takes about 1 week (for some reasons, speed of scrapping was about 300 kb/s) and nothing happened (zfs kernel module crash).

Then I set the pool to the read only mode, and started copying data with rsync to new drive with ext4.

During the copy process, I constantly got zfs module kernel crashes.

As I used rsync, I run it over and over again and figured out, that crash happens only in case, when it tries to copy one file (some movie)
Tried to remove it: kernel panic, so I just excluded it from rsync

Finally, I **mostly** recovered my data

Maybe the problem is not in zfs, maybe it's USB storage driver, or not very tested arm pathway, but now I'm afraid of touching zfs.

PS: I formatted drive to ext4, tested it with long SMART, no errors

[Agno]

How easy is it to create a mirror on Ubuntu 24.04 after creating the single disk ZFS? Is it just a matter of copying the partitions from one disk to the other, then attaching the bpool and rpool partitions and allowing resilvering to finish?

IIRC you don't need to copy anything. Just attach you second disk or partition to the desired zpool: the vdev will be turned into a mirror and the resilvering process will start, maybe also a scrub will start(not sure) after the resilvering. IIRC you are done with one command + waiting.

ZFS (and BTRFS and BcacheFS) benefits are easy creation of striped, mirror or other RAID topology.

I agree these are huge benefits (even if I wonder who trusts BTRFS for "other RAID topology" anymore). But I think there are lots of benefit beside those that require multiple block devices. Native snapshot, encryption(zfs and bcachefs) and compression, children-filesystems/subvolume, send/receive (zfs and btrfs), etc.. are pretty useful for single disk system too (think of a laptop or a usb drive). Unfortunately I had problem when I tried zfs without redundancy. I like zfs features but it looks like the single-device use case is not well supported. For a USB drive I went with btrfs over LUKS, which I still don't trust but my data are also elsewhere.

I used zfs on rpi4 for external USB 6 TB drive and almost lost my data.

I had a similar experience. Single device zfs with no redudant vdev is not well supported. I tried zfs on root back with ubuntu 22.04. After a kernel panic (unrelated to zfs) the pool could not be imported due to some metadata being corrupted. I was able to import it read-only, mount most filesystems and read all the snapshot. I didn't lost any data besides some logs. ZFS relies a lot on vdev-level redundancy and it becomes very annoying when even a little piece of data is corrupted (which will eventually happen even on a healthy device) unless it has a way to fix them.

[LinuxNoob]

IIRC you don't need to copy anything. Just attach you second disk or partition to the desired zpool: the vdev will be turned into a mirror and the resilvering process will start, maybe also a scrub will start(not sure) after the resilvering. IIRC you are done with one command + waiting.

But this would not allow booting if the first drive failed, correct?

[S.Pam]

But this would not allow booting if the first drive failed, correct?

EFI does not support raid of any kind, unless you have a hardware raid that provides UEFI support.