XFS With Linux 6.9 Brings Online Repair Improvements

[oleid]

I'm running XFS for more than 20 years, on tens of TBs of storage, and I haven't found yet any corruption.

Some time ago I had issues with some files being zeroed after a crash. That's when I switched to ext4. I guess it could be 15 years ago. But I've been told that this bug is long fixed.

[muncrief]

gbcox

How should we update the Kernel without removing the old version?
We provide proper support for EVERY major update of the kernel and zfs module. Yes, upstream dont offically support it yet, due ONE missing PR, but we have pulled this into our zfs module and maintaining an own branch.

This has been also tested from several people with different configurations. Same for the NixOS People, which use the CachyOS Kernel.
We really care about zfs support, specially when the major version does change and we go through a long time of testing and reporting also bugs to zfs.

If you are using zfs-dkms, then you need to do on your own, but the cachyos configuration does use the precompiled zfs module from our side, which is all time compatible.
If you want to use an older kernel, youll find it in your cache.

Edit:
And if you really want to downgrade the kernel to version X, you can all time to this via compiling it on your own.
Just use an older commit and then set your options and compile your kernel.
https://github.com/CachyOS/linux-cachyos

Aha! Thank you so much ptr1337. So the problem was pilot error, I was indeed using zfs-dkms because I didn't realize Cachyos made sure their own ZFS modules were compatible with the latest kernel and the lts kernel.

So I uninstalled zfs-dkms, and then installed linux-cachyos-zfs and linux-cachyos-lts-zfs (and also cachyos-zfs though it just seems to be a meta package for linux-cachyos-zfs), and reinstalled linux-cachyos and linux-cachyos-lts for good measure, and everything works like a charm!

I still don't like the idea of Cachyos releasing the .0 version of the kernel instead of waiting for .1, but after everything I've been through if there's no way to avoid it I can live with it.

In any case this is why I asked my questions here, because I knew someone would have the answer and save me a lot of wasted time and probable errors. Thanks again for your help.

[gbcox]

One uses btrfs if one needs compression or subvolumes. Xfs provides neither. So if you don't need any of those features, there is little reason for btrfs, there are faster file systems.

Yep... I stopped using RAID when I went to XFS and I don't use compression.

[muncrief]

You don't even mention the filesystem you're using now, so it's very hard to understand _what_ the issue you're having is. More likely is hardware (lack of ECC? something else?) but it can also be software, if you're running something exotic.

I'm running XFS for more than 20 years, on tens of TBs of storage, and I haven't found yet any corruption. Probably there are bits flipped in some files, though, because of the following:

But note that XFS is *not* designed to catch user data corruption. The recent work is for improving *metadata* health.

I used ext4 up until about 8 months ago when I switched to OpenZFS. And there were no persistent hardware related errors, just silent data corruption because so much of my data is ancient and eventually silent corruption will happen. But everything is good now, I just wasn't using Cachyos correctly.

But I swear, this is the first mistake or error I have ever made!

Oh wait ... never mind ... my engineering life, and life in general, is full of them

[aviallon]

I used ext4 up until about 8 months ago when I switched to OpenZFS. And there were no persistent hardware related errors, just silent data corruption because so much of my data is ancient and eventually silent corruption will happen. But everything is good now, I just wasn't using Cachyos correctly.

But I swear, this is the first mistake or error I have ever made!

Oh wait ... never mind ... my engineering life, and life in general, is full of them

Don't forget to enable scrubbing. Another famous Linus did this fatal mistake.

[muncrief]

Don't forget to enable scrubbing. Another famous Linus did this fatal mistake.

Thanks for the reminder aviallon. I scrub my pools manually every month right now because the default systemd scrub timers/services had problems with multiple pools, especially the large ones. I also had errors if I shut down or rebooted my systems without pausing the scrubs first, even though it's supposed to be okay. But now that everything else has been taken care of and I'm sure I'm going to keep OpenZFS I'll create my own systemd services that coordinate the scrubs and make sure scrubs are paused and resumed gracefully during reboots and shutdowns. I'll also add automated emails to let me know the results.

[iustinp]

Some time ago I had issues with some files being zeroed after a crash. That's when I switched to ext4. I guess it could be 15 years ago. But I've been told that this bug is long fixed.

Oh, that. I think I have two or three such files in the year that this was a problem. I was running hardware raid with battery backup, so maybe that’s why I had few issues - the files were temporary files, recently written, not important ones. But yes, this was about 12-15 years ago.

[CommunityMember]

Still can't shrink it. Will we ever get that?

It will depend on when you send in your PR that implements that. When can we expect it?

The vast majority of XFS users never shrink their pool. The majority of the developers on the xfs project either directly work for, or are paid from funding provided by, organizations that never shrink their pool, and those organizations are not likely going to ever resource something they do not want. If you want it, do it (or find someone to pay to do it).

[tildearrow]

I'm enjoying Michaels random article pictures.

That's what happens when the topics are so abstract that cannot be described with a picture at all.

[pong]

I'm not exactly offering the most nuanced opinion here, but AFAICT with many kinds of storage media you basically MUST "periodically"
read / verify / rewrite data blocks to keep the storage integrity high and avoid "bit rot". Magnetic storage suffers bit-rot due to
magnetic fields (environmental and adjacent / nearby written other-data) coupled with thermal energy interacting to slowly depolarize / rotate etc. stored data's magnetic fields over time. In FLASH / SSD type cells basically they operate on stored charge on a capacitor (IIRC) and eventually
leakage / diffusion of charge and ionizing radiation effects (environmental) and thermal factors cause the degradation of the stored multi-bit cells so they may stare with a value of 16, slowly degrade to 15.7, .... etc until you've lost an entire bit or more.

So you kind of "have" to have a data store protected by a reliable integrity checksum / hash / parity redundancy / error correction scheme to be able to get your data back despite some bit errors and/or at least determine IF the read-back data is very probably accurate vs. what was written.

And then after N months / years of "rotting" you should "scrub" the medium or every portion of it incrementally reading all contained data, writing that data "freshly" to another empty formatted chunk , then (sooner or later) erasing the original chunk where the old data was to recycle it for future re-use. And if you detect stored errors in the process you use parity / backup / whatever to recover that chunk and proceed on.

AFAICT almost nothing commonly available / affordable (DVD, CD, HDD, SSD, many magnetic tapes, ...), is going to do well just storing data on it
and then 5-10+ years later coming back and expecting there has been zero bit-rot. That may be over/underestimating the archival reliability of
particular cases but the point is you EVENTUALLY have to do scrubbing / refreshing for most media and the more you value the data the more
you should assume there could have been partial or whole storage failures and you should detect / recover from those "promptly" before other
mechanisms / events make it more likely you'll have multiple failures over time, lost backup / redundancy, etc.

Of course merely spinning up storage, reading / verifying it, and re-writing it costs a toll on its life-time / reliability so you don't want to 100% scrub
the archive daily or weekly or monthly probably but some happy-medium frequency just enough to ensure reliability but not enough to "burn out"
the storage's "active life time" / TBW rating / spin-up-down count / MTBF etc. unduly prematurely vs. NN years.

If you have a mirroring based backup it is simpler to recover a wholly or partially lost "drive" but you're "wasting" at least / more than 50% of your storage. So then there's EDAC / ECC / parity based redundancy where you assume that out of N drives K of them could totally fail and you'd like to
be able to recover so you somehow store K drives worth of redundant information spread over the N drives so you can't call it a backup of the
entire data volume but you assume 1-2-...K drives may partially or entirely fail and you can still usually recover from that if that's less than the
N total array drives and the other drives can be assumed to work reliably during recovery and reforming a new redundant array.

Really we're pretty "lucky" if bit-rot is the main failure mode to avoid. In practice due to quality defects or accidents / environmental disasters (flood, fire, ...) it's not uncommon for entire drives to just die wholly at any random time so that's something you have to tolerate in addition to the longer
term "prevent bit rot" scrubbing / rewriting.

...11 TB of data accumulated over 4 decades, and was plagued for years by silent data corruption. Though I have both local and cloud backups of everything, if an issue wasn't discovered for 2 or 3 or 4 years, or more, at times it was impossible to find and restore the uncorrupted data.