A Quick Look At EXT4 vs. ZFS Performance On Ubuntu 19.10 With An NVMe SSD

[guglovich]

phoronix Interesting test. I would like an update for 2022.

[S.Pam]

Not a checksumming thing like with ZFS and Btrfs... do those scrub? I guess they must. How else to you prevent rot?

They verify checksums on each read. If there is a mismatch they will recover a good copy from dup or raid. Scrub simply detects faults before you need the files, especially when data is not in raid. That way you can use your backups. Scrub will also help you detect disks that are going bad and starts to corrupt data.

[henrik]

For me, checksumming and bit rot protection in zfs or btrfs is of no value. They don't protect against accidental data modifictionby myself, or by a virus/malware or whatever. The only sane thing to do is to regularly compare differences in your dataset (which you care about) and manually decide if you need to restore from backup. There is a great little tool called "integrit" which can do this for you. With this, you can skip the advanced file systems and still get better protection for valuable data. This method is not fancy but sure a lot safer than trusting the file system to protect you from bad things happening ABOVE the file system layer...

[cynic]

Uh, this test demonstrates a fundamental misunderstanding of what ZFS is, and what it is good at.

Why anyone would install ZFS on a single drive in any configuration is beyond me. What it excels at is redundant RAID-like configurations.

Any comparison between zfs and any other file system should be against hardware RAID or another software RAID configuration.

ZFS (and btrfs) can detect silent data corruption, even if it cannot fix it in a single drive configuration. If you care about your data integrity, this is quite a useful feature.

Oh, and let's stop talking about BTRFS like it is a viable option. BTRFS is dead. Red Hat deprecated it, it is losing support left and right, it is going nowhere. Dead as a doornail.

btrfs is very far from being dead. It is actively developed by, among the others, Oracle, SuSE and Facebook. Just take a look at the developer mailing list.
Redhat deprecating it on RHEL surely is not good, but it's not dramatic either.

[mattlach]

Uh, this test demonstrates a fundamental misunderstanding of what ZFS is, and what it is good at.

Why anyone would install ZFS on a single drive in any configuration is beyond me. What it excels at is redundant RAID-like configurations.

Any comparison between zfs and any other file system should be against hardware RAID or another software RAID configuration.

The right tools for the right job.

Oh, and let's stop talking about BTRFS like it is a viable option. BTRFS is dead. Red Hat deprecated it, it is losing support left and right, it is going nowhere. Dead as a doornail.

[S.Pam]

not to sure why you are bench-marking ext4 vs zfs with a single disk ? or why btrfs was not in the mix as its closer to zfs features

Even on single disks I choose to use to use Btrfs because of several reasons.
* checksums
* snapshots for easy backup and revision management
* compression
* easy and advanced volume management

Of course ZFS has similar/same features if you like ZFS better.

So it isn't exactly fair to benchmark these against ext4, xfs and the like if you need these more advanced features.

[MasterCATZ]

not to sure why you are bench-marking ext4 vs zfs with a single disk ? or why btrfs was not in the mix as its closer to zfs features

[deusexmachina]

I wonder if one could build a box like this: https://www.youtube.com/watch?v=N0qtu5NXhuQ
That gets the same 5GBps that Linus demonstrated, but running ZFS! Would it still need hardware raid cards? If so, then which hardware raid cards for doing "software raid" like ZFS?

[deusexmachina]

I can give you some base rules(and i have bunch more in older posts in other threads as well):

Hardware:

• if possible always prefer ECC over fast ram, all technicalities aside it protects you from getting trash written from RAM to Pools.
• ZFS use a lot of RAM by default(unless fine tuned), so if you don't wanna spend hours with arcstat just set you minimum RAM to 32GB for a regular home usage
• If you wanna encrypt get a CPU at least modern enough for 4 cores with AES-NI, Zen based CPU's are great choices.
• If you plan to use NVME in as disk not as ZIL/SLOG drives, set you minimum requirement to at least ThreadRipper or X299 if you love to waste money. ZFS can handle RAID on NVME on any system and won't require any sort of BIOS or dongle extras but regular desktop boards lacks PCI-e bandwith, so you will always be limited to 1 drive speed or worse depending the mobo.
• Never use ZFS on RAID 0 or with a single drive because basically you will get all the downsides of CoW with literally 0 benefit.

Generals:

• know you data, depending on your data the performance can be great or unbearable trash.
• not all properties on a pool are for you, just use what you need.
• never ever ever use ZFS on a bare pool, volumes exist for a reason and if you don't use them then you should consider why are you using ZFS to start with.

Basic Optimization:

• Compression is a great tool but a deceptive one, if you have a volume with lots of office files or highly compressible files(let say your documents folder ) compression will work great and boost you transfer rate because you save lots of bandwidth but if you have lots of non-compressible files like videos or already compressed files enabling compression on that volume will skyrocket you latency with 0 bandwidth savings, aka you are wasting CPU cycles for no reason.
• Deduplication is a nice feature but it requires huge amounts of RAM/CPU and can make latency spike harshly if misused, so keep in mind it should only be used on volumes with lots of small files that you know can be redundant and compressible(like a samba share for people to save office files or the likes) or a volume with big binaries that you know have lot in common like ISOs, Virtual Machine Drives(in case you share several instances of the same OS),etc.
• Large Dnodes should be use on auto always unless you have a very specific reason not to(like Solaris compatibility).
• Atime=off Relatime=on, this one don't need much explanation.
• recordsize is a tricky one, my rule is 16k for certain databases, 128k for general bunch of small compressible files and 1M for volumes where most of your files are bigger than 1M(like videos/iso, etc), if you don't do this right you will get low performance and/or very high fragmentation
• Sync goes always standard unless you really know what you are doing.
• xattr=sa, acltype=posixacl and aclinherit=(up to you) worked great for me through the years but as always check the documentation first.
• Encryption require testing because depending on you data/compression/recordsize could be great or a slow dog, so do some tests before blindly go and start encrypting.

FAT WARNING:

• Never use ZFS on RAID 0 or with a single drive because basically you will get all the downsides of CoW with literally 0 benefit.
• Most changes made to a Volume affect only new/modified files be careful to make most of your changes before adding data to Volumes/Pools/etc.
• A lot of fine tuning can be done through kernel module parameters as well but is way more complex than a simple post can handle.

Very useful! Is there someway to @copy the phoronix.com authors so next ZFS tests we can have everyone in the comments saying, "ya totally fair config, thanks!" Looking forward to the next ZFS raidz2 benchmarks on the top SSDs vs ext4 & XFS.

[DrYak]

RAID 5/50/6/60 is still very nuclear on BTRFS, it may work or it may eat you data and kill you kitten

Anything using BTRFS' RAID5 or 6 is still considered officially NOT STABLE and SHOULD NOT be used in anything but testing environment. (I don't dare to use them)
For now you're limited to stacking it a top a MD RAID5/6 (that's what I use instead if I want erasure coding)
Though in that setup BTRFS can detect bitrot (checksum missmatch) but not attempt to fix it (it won't get access to the redundant strips).

Is very slow on big LUNs specially on PostgreSQL(last tried with PG10/lin5.1), at least compared to ZFS but it may be related to the first issue since i didn't test with RAID1(which i think is the strongest one on BTRFS atm)
{...}
BTRFS and virtualization are not very good friends.

Both boil down to BTRFS being CoW and thus not very good at big numbers of random write in large files. (You can add large torrent files as a third exemple).
Work-around consist of tagging these large files a no COW ("chattr +C" on them before starting to write into the files. Or alternatively touch/chattr a new file and then pipe over data from the old file), which will cause BTRFS to perform the write operations in-place, and will delegate the integrity on whatever that system uses (on the database's log journal, the VM's own mechanism used by the guest OS, and torrent's tree of hashes).
auto-defrag can also help alleviate a tiny bit, by coalescing multiple neighboring writes into a single one.

Also RAID5/6 leads to expensive read-modify-write cycles when only a fraction of a whole stripe is added ( <- my experience on write intense situations).

BcacheFS is interesting because - by leveraging it's tiered storage - it can (according to Kent) condense those writes into *new stripes* (thus append-only write cycles, not the dreaded read-modify-write).

In general i believe BTRFS lacks flexibility in the volume/snapshot department as well but this may be subjective depending on what you do.

I would find interesting if you could go into details (e.g.: about which scenario you can't accomplish with BTRFS).

I don't think it works ok on NVME(as with several drives), i can't prove it since the logs say nothing but on nvme i just noticed some services get random huge latency spikes and the only difference is ZFS vs BTRFS but well i may be wrong(i also didn't bother much to go the extra mile i just nuked the server and went back to ZFS to test another stuff <-- was a test server ofc).

Haven't got much experience with NVMe. Though there isn't a fundamental reason why BTRFS shouldn't work with them, except maybe the usual low-performance stuff of BTRFS getting in the way (random writes and eventually maze of pointers to traverse until finding the correct extent in a heavily fragmented file, read-modify-write cycles if you dare to RAID5, the need to absolutely checksum everything meaning you're not operating at maximum bandwidth, etc.)

Don't get me wrong, i do believe for desktop/small servers BTRFS is sufficient and stable enough, specially considering is a lot simpler than ZFS and give similar features but for Enterprise stuff or really important stuff i do believe ZFS is without peers,

Just as an example of this way of thinking: I run BTRFS on my workstation, the admins deploy ZFS on the HPC here.

or a workload i couldn't find a way to optimize the living bejesus out of it

This is probably one of the main advantage of ZFS: it's extremely tunable.

BTRFS is quite limited from that point of view (locally turning of CoW on certain files, autodefrag, and that's about it).

damn, even today i have a client with an old server that has lost in the last 10 years 23 of its original 24 hard drives(is implied i've been replacing the damaged drives for new ones and resilvering, of course) but have never lost a bit of data.

Well, on the other hand, *any* erasure coding could handle such scenario, down to even your garden variety of MD RAID5/6.
(Caveat aside about BTRFS's own Raid5/6 not being stable)