Linux 4.11 File-System Tests: EXT4, F2FS, XFS & Btrfs

[DrYak]

I actually had problems with that. Jolla Phone reported plenty of space available, while I was really out of space in BTRFS. I had to go to command line root and run some BTRFS commands to get it fixed.

There's actually a graphical interface for "btrfs-maintenance", and even a systemd's .timer to periodically rebalance.
Though I don't k exactly where it is. It might be a 3rd party openrepos package.
I actually use the command line interface : "btrfs-balancer balance".
You really need to balance monthly~weekly to guaranty that you'll not reach "enospc" error.

EDIT: There is one. Hmm.... Jolla should *definitely* implement something similar on their base installation.

But I have to admit that Jolla has done a few sub-optimal choices in the way they've handled btrfs:

- They use "dup" for the metadata RAID mode. Means each block group of metadata is written twice on the media.
It used to make sense with spinning rust media (if there a bad sector in one copy of the HDD, you can still use the alternate copy elsewhere).
It's useless on modern flash media (the flash translation layer groups writes together to help the wear-levelling. Thus boths copies will end up on the same flash erase-block. So either both will be fried, or both will be okay).
Modern btrfs recommand "single" mode instead for SSDs, but apparently back when the phones where shipped, it wasn't yet by default in mkfs.btrfs

You can change it post facto with :

btrfs balance start -v -f -mconvert=single /

That remove this waste (for flash) of space.

- They use separate metadata and data block group. (as if you used 2 different partitions handled by LVM)
Most of the problems arise when there's plenty of free space on the data block groups. But to do an action, btrfs needs to allocation a new blockgroup (e.g.: needs more metadata space), and there are no free block group left.
(as if you needed to lvresize your metadata partition, but there's no extent free left on the volumegroup, because all is assigned to the data partition).

The problem is that data block groups are allocated in stripes of 1GiB (and metadata is allocated in stripes of 256MiB).
A small flash drive like Jolla 1's 12GiB root partition is quick to get allocated for data block group (only up to 12 data groups available), and then you already ran out of block groups to allocate for metadata.

Modern BTRFS has a "mixed mode" where both data and metadata are written on the same block group. So if you need to write more metadata, you don't need to actually allocation another 256 MiB of metadata block groups, you can actually write on the free space left on the other data block groups.
This is currently recommanded by btrfs for small flashmedia with only around a dozen of GiB.

But it was just then introduced with the 3.2 series of kernel. So probably given that it wasn't yet tested enough back then, Jolla decided to stay with the more usual classic approach.

And it still remained flaky and unstable until I did a full factory reset/reformat.

That is really weird...
Did you get srcub errors ?

I've tried using BTRFS on my work desktop VM for a while. Compression and checksuming were nice features, but it played really badly with low disk space situations.

again, you need to regularly balance it.
openSUSE provides a "btrfs-maintenance" package that gives you a cron job for scrubbing (check for errors), balancing (free block groups), and TRIM (if you VM can use it for sparse disk images)

Overall, now I'm a bit careful with using BTRFS as I'm not certain it will behave in stable & predictable manner.

you either need to understand how the whole btrfs stacks works, specially the block group allocation (the equivalent of LVM in a classic stack) and how it interacts with the filesystem.
or you need to give it to something else to worry (btrfs-balancer, btrfs-maintenance) and carry the maintenance for you (either scripts that you call manually - i do this on my phone once in a while on the phone) (or have a cron job / a systemd .timer do it for you - i do this on my suse machines, and have my own home grown script for debian)

with such my expercience with btrfs has been worry-free.

[coder111]

I actually had problems with that. Jolla Phone reported plenty of space available, while I was really out of space in BTRFS. I had to go to command line root and run some BTRFS commands to get it fixed. And it still remained flaky and unstable until I did a full factory reset/reformat.

I've tried using BTRFS on my work desktop VM for a while. Compression and checksuming were nice features, but it played really badly with low disk space situations. Never got around to trying snapshots or backups/replication.

Overall, now I'm a bit careful with using BTRFS as I'm not certain it will behave in stable & predictable manner.

It is actually the default on Jolla's Sailfish OS (a smartphone, no less).
It is also the default on some flavours of Suse, because their snapper technology extensively uses snapshots to make checkpoints.
(both provide tools to help around the weird maintenance of btrfs)

).

[DrYak]

I feel bad for btrfs. It will never be the default file system for desktop usage with those numbers. EXT4 still seems to be the way to go.

It is actually the default on Jolla's Sailfish OS (a smartphone, no less).
It is also the default on some flavours of Suse, because their snapper technology extensively uses snapshots to make checkpoints.
(both provide tools to help around the weird maintenance of btrfs)

And regarding numbers :
BTRFS is an entirely different class of filesystem with ton of features (checksums, snapshotting litterally for free, built-in volume management). And some of those features get in the way of performance.
(SQLite on BTRFS is clearly an example given how CoW works and how it doesn't like frequent inplace rewrites - like databases, virtual images, etc. Which, by the way, is typically the kind of worlkload where you would NOT be interested in CoW and snapshotting, because these are the system which have their own crash recovery and consistency checks in place - databases' journals and the filesystem running inside the VM).

I'm more surprisd how F2FS manage to maintain good performance (it's a log-structured file system).
But maybe it implements facilities to detect this kind of situations (ZFS is a CoW system that does exactly that, and recently BTRFS has added an auto-defrag for this exact purpose.

Michael Is it somehow possible to also benchmark bcachefs (http://bcachefs.org/)?

Yup, specially given that it aims for the same kind of featureset as BTRFS and ZFS (subvolumes, snapshoting, checksuming, on the fly compression, volume management, including various flavours of RAID).
Even if currently it's still missing core features (snapshotting is still not there yet, and duplication/RAID1 recovery need to be stress tested). It might be still interesting to see how it handles compared to the rest of the crop.

A file system comparison can never be complete without the "killer" filesystem.

At some point in time, kernel dev should face reality and rename "Reiser4" as "ShishkinFS".

(Though given the direction of evolution of filesystems :

- ReiserFS won't see any more development, and I think EXT4 and XFS already have overtaken it in terms of features. (I'm not sure anymore if it's still competitive for huge trees of small files. e.g.: compiling kernels)
And this kind of previous-generation filesystem (journaled filesystems) isn't the rage anymore.

- Reiser4 is playing in the same field as the current gen of filesystems (BTRFS, ZFS, and eventually BCACHEFS once Kent manages to finish it). But despite all the efforts of Shishkin, I don't know if he'll manage to get on par with filesystems that have strong backing like e.g.: BTRFS (Facebook).
Also, I don't know if it'll manage to match all the modern features too - e.g.: it can have some form of CoW activated (a.k.a "Write anywhere journalling"), but I don't think it supports snapshots
).

[M@yeulC]

Is XFS an acronym? What does it stand for?

Btrfs looks pretty good, I think. Copy on write, checksums, etc. slow it down quite a bit, but those are valuable feaures. Plus, there is actually a performance gain when you copy files on your hard disk, which is a rather frequent use case, IMO.

[Azrael5]

it would be useful to use F2fs to install linux operating systems into 3.0 or 3.1 USB flash memory. I assume it should be the default file system over NVMe too.

[treba]

Well, the decision for/against a next-gen filesystem should imo not be made on the ground of raw performance. I use BTRFS and ZFS alot these days because their features totally outweigh the performance penalty for me, especially checksumming (I'd never use a big NAS without it anymore), snapshots, compression and deduplication.
Still good to see BTRFS getting more competitive on performance grounds. But performance is barely problem I think.

What's worth to note you benchmarked 4.9 with deadline scheduler and 4.10, 4.11 witch CFQ.

[stiiixy]

I am using XFS for large file storage for video and music and EXT4 for the smaller stuff, as they both handled those respective use cases better for my simple set up. I'm well overdue however for another ring of tests, especially when BTRFS brings out a more stable R5/6.

[Marc Driftmeyer]

The average Linux User doesn't touch any of these products you're benchmarking against their respective filesystems. I suppose you can only work with what's available but benchmarking GNOME, KDE, and their main applications using the following filesystems would have far greater value.

[nuetzel]

Maybe the regression stems from this:
"mm, page_alloc: only use per-cpu allocator for irq-safe requests"
So it was reverted.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d34b0733b452ca3ef1dee36436dab08b8aa6a85c

Please retry with 4.11-rc8...