EXT4 Lands A Nice Performance Improvement For Appending To Delalloc Files

[linuxgeex]

So basically a log file optimization?

Trivially, yes. But it's not as trivial as it sounds, since there's lots of applications using log-structured file formats to get log-structured filesystem performance benefits from non-log-structured filesystems, and/or their use-case involves constant incremental data being stored without too much worry around running out of disk space. An early example of this still in use today would be the mbox file format. SQLite is an example of one you're using a lot today, as it's used by Firefox, Google Chrome, and Safari to maintain session state for tabs, as well as cookies and the Web Storage API, and SQLite is getting (IMHO) over-adopted as a bloated per-app registry in place of more robust and user-readable config files. Databases in general end up acting like logs even where they aren't designed from the ground up to get the best benefits from being logs.

[phuclv]

When I first came here, this was all swamp. Everyone said I was daft to build an extent based file system on a swamp, but I built in all the same, just to show them. It sank into the swamp. So I built a second one. That sank into the swamp. So I built a third. That burned down, fell over, then sank into the swamp. But the fourth one stayed up. And that's what you're going to get, Lad, the strongest extent based file system in all of England.

ext2 and ext3 don't support extents at all. The "ext" stands for extended and only until the fourth extended file system the extent feature were added

[geerge]

How do you use de-duplication? The only way I'm aware of it being used is in snapshots.

As I've mentioned before, I view checksums as the main value that I derive from BTRFS, on my personal machines.

https://btrfs.readthedocs.io/en/latest/Deduplication.html

[coder]

If FS-level compression/dedupe/redundancy

How do you use de-duplication? The only way I'm aware of it being used is in snapshots.

As I've mentioned before, I view checksums as the main value that I derive from BTRFS, on my personal machines.

[geerge]

There are multiple other options but I see no reason to try another file system as ext4, and 2, 3 before that, have all been very reliable and fast for my workstations. Compiling the kernel, for example, is never limited by the file system as CPU load is always close to 100%. I don't think Abaqus or GIMP will be any faster with another file system either. It is good to see improvements are still being done on such a mature software.

If FS-level compression/dedupe/redundancy doesn't get your motor running then there is no need. I wish BTRFS could do extension-based compression because then it would be much less painful to do high compression on things that can compress. The current state of play is that unless you force compression it'll turn off compression once enough incompressible data has been found, not so useful for tar/iso/archives that can mix arbitrarily.

[mrg666]

There are multiple other options but I see no reason to try another file system as ext4, and 2, 3 before that, have all been very reliable and fast for my workstations. Compiling the kernel, for example, is never limited by the file system as CPU load is always close to 100%. I don't think Abaqus or GIMP will be any faster with another file system either. It is good to see improvements are still being done on such a mature software.

[coder]

Another aspect to consider is recoverability. I can't track down the reference, but I recall reading XFS was best used with a UPS, where as ext4 was generally OK without one (or at least remained recoverable via fsck). That was years ago, so it may well be XFS has improved since then.

That ext4 is mature and quite recoverable was further backed up by papers such as Shehbaz Jaffer, Stathis Maneas, Andy Hwang, and Bianca Schroeder, Evaluating File System Reliability on Solid State Drives (https://www.usenix.org/conference/at...ntation/jaffer). Their paper only considered btrfs, ext2 and F2FS, but the recoverability of ext4 turned out to be quite robust.

Used how? In the early days of EXT4, I remember performance with barriers enabled sucked, so the normal thing was to mount it with the nobarrier option. That surely had to impact recoverability, however.

I believe the main reason we use XFS, at my job, is for combating filesystem fragmentation. If you leave about 5% to 10% of free space, that's plenty for XFS to ensure fragmentation doesn't become too bad. On mechanical hard disks, fragmentation poses a serious performance issue.

This evening, while searching for the above reference, I also found this paper: Anthony Rebello, Yuvraj Patel, Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, Can Applications Recover from fsync Failures? (https://www.usenix.org/conference/at...tation/rebello). I've only briefly scanned it, it seems quite interesting.

If fsync() or fdatasync() don't work as advertised, I consider it a filesystem bug. If the caller fails to handle an error, such as EINTR, in a robust way, then the application code is at fault.

My interest is purely from the perspective of a desktop user with about 2 TB of for /home on a single SSD, no UPS, and root on a separate smaller SSD, rotating media used for on and offline backups only. A lot may have changed since I last examined the issue, but I've stuck with ext4 because of reliability, not speed. By reliability, I mean a fsck has always managed to return me to a mountable file system, at which point it's easy to make comparisons with the various backups.

I like BTRFS for its checksums.

At my job, we make use of BTRFS' ability to do atomic snapshots - when making backups, we first make a snapshot and then backup that. This has two advantage:

1. minimizing the chance of inconsistencies, due to the backup process reading different files at different times.
2. letting you know exactly what was backed up, as a consequence, enabling you to verify the backup!

For relatively low-turnover data, snapshots also serve as a type of backup that protects against mostly user error (e.g. "whoops, deleted wrong file/directory!"). We use hourly snapshots, on a departmental fileserver, for this purpose.

[sbivol]

it may well be XFS has improved since then.

There were a few attempts at fixing XFS, starting with this pull. I don't know if the fundamental problem of writing metadata before data was ever solved in XFS, this is what allowed it to claim being faster than EXT4. Linus himself described writeback as idiotic, adding "It literally does everything the wrong way around - writing data later than the metadata that points to it. Whoever came up with that solution was a moron. No ifs, buts, or maybes about it".

I can say from experience that XFS in RHEL 8 does not have this particular flaw where any system reset causes data loss. I haven't bothered with investigating the writeback data loss scenarios since I gave up on XFS completely and would never use it no matter how much FUD Red Hat continues to peddle about EXT4.

I've stuck with ext4 because of reliability, not speed.

Then you're lucky since you also got the best speed possible in this scenario.

[szymon_g]

Can someone explain me the reason to use EXT4, when XFS is faster and has more features?

Does xfs support FS level encryption? What about enlarging the partition without losing the data?

[digitaltrails]

Can someone explain me the reason to use EXT4, when XFS is faster and has more features?

Another aspect to consider is recoverability. I can't track down the reference, but I recall reading XFS was best used with a UPS, where as ext4 was generally OK without one (or at least remained recoverable via fsck). That was years ago, so it may well be XFS has improved since then.

That ext4 is mature and quite recoverable was further backed up by papers such as Shehbaz Jaffer, Stathis Maneas, Andy Hwang, and Bianca Schroeder, Evaluating File System Reliability on Solid State Drives (https://www.usenix.org/conference/at...ntation/jaffer). Their paper only considered btrfs, ext2 and F2FS, but the recoverability of ext4 turned out to be quite robust.

This evening, while searching for the above reference, I also found this paper: Anthony Rebello, Yuvraj Patel, Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, Can Applications Recover from fsync Failures? (https://www.usenix.org/conference/at...tation/rebello). I've only briefly scanned it, it seems quite interesting.

My interest is purely from the perspective of a desktop user with about 2 TB of for /home on a single SSD, no UPS, and root on a separate smaller SSD, rotating media used for on and offline backups only. A lot may have changed since I last examined the issue, but I've stuck with ext4 because of reliability, not speed. By reliability, I mean a fsck has always managed to return me to a mountable file system, at which point it's easy to make comparisons with the various backups.