Bcachefs File-System Pull Request Submitted For Linux 6.5

[Anux]

You would be surprised to know how many failures I had due to sata cables or controllers...

How many then? I had more than 2000 PC's going through my hands, none ever had a SATA cable problem. Powersupply and hard disks are the typical failures and then comes a lot of low quality plastic switches on not so cheap bussines cases ^^ and a defect board or RAM module here and there.

And it doesn't make much sense, those cables get barely moved and a mouse cable is under much higher stress and doesn't fail regulary either. If those cables aren't defect from the start I don't have any explanation how they would degrade over time, maybe extremly humid environements (but then everything else would also be affected and not just those cables).

[darkbasic]

How many then?

I think it's three sata cables and four sata controllers in total: not many but enough to not trust them blindly anymore. To be honest one of the sata cables had been slightly trimmed at the connector in order to fit so I wouldn't count that one. Also many of these failures (the non-catastrophic ones) wouldn't have been discovered if it wasn't for zfs.

[kreijack]

This is a pretty stark contrast to how ZFS works, where you have redundant metadata structures and they're all individually checksummed so you can fall back to a good copy. You can actually restore it fully, not just say "this is damaged, remove it to become consistent again" or "we don't know where this folder goes anymore, move it to lost+found."

This is what does BTRFS too. The [meta]data is checksummed, and if the checksum doesn't match then another copy is looked for good data. If this matches, then the good data is copied over the bad one.

However there are other cases where the the metadata is damaged by (e.g.) bit flip in ram. And in these cases, if the bitflip happened before the checksum what described above doesn't work. Same if the bitflip happens after the checksum but before the data writing..

However for performance reasons some data structure are copied several times. For example for each directory entry there are multiple copies ordered by different criteria: in btrfs there is a set of directory entries ordered by the hash of the name (to faster the access on the basis of the name) and by directory index (to faster the directory traverse)...
Also the free space accounting is tracked in two (or even more) ways, in order to speed up the access depending by the action.

A fsck could implement some heuristics to rebuild the correct data on the basis of other data. And this would help even in case of single (no raid) filesystem.

So there are cases where a fsck helps.

[Knghtbrd]

Really want to see bcachefs ready for general use, but at least the bcachefs pages claim there's a few things that just aren't done yet, and they're things that matter. Then again btrfs also has things that aren't done and kind of matter. It might just need to be added to the kernel at some point with things still needing work, but I don't know if now's quite the sime.

One man shows with large egos usually wind up going off the rails sooner or later. It'd suck of the biggest barrier to bcachefs adoption winds up being not at all a technical problem, but rather entirely an ego thing.

[DrYak]

Btrfs' own Raid1 and Raid10. I am fearing Raid5/6 is not stable so I am not using it.

Same here, I am also strictly following the "RAID5/6 not considered stable" status from BTRFS' docs.
For the server RAID6 is handled by mdadm, and BTRFS is running on top. As BTRFS is COW it has a "mostly append-only" property that reduces the typical write amplification of parity RAID.
But this setup cannot leverage the checksums to guide the RAID recovery (unlike Btrfs' build in raid). But luckily it's RAID6, so even with 1 dead drive there's still 1 extra parity remaining.
This setup has also survived multiple drive failure+upgrades (taking the oportunity that a drive has failed to replace with a larger drive, like you did). By some strange coincidence, I've never managed to upgrade a drive before failure (the couple of times I've detected early signs of incoming failure in SMART logs, by chance (by unluck?) the replacement drive happened to be out of stock and only arrive with several months delay by which time the old drive started to fail. Funniest was the last one. As it was the last, I decided to replace preemptively. Turns out the drive model I wanted wasn't produced anymore, and after a while until the shop sorted it out with the supplier, I had to order a different (larger) model, by the time that drive arrived... click of death started).

Everwhere I use BTRFS with SSD (either SATA mSATA or NVMe) with RAID1 for metadata, and RAID0 (System and other easy to readownload) or RAID1 (home) for data.
(This even include a mini server build with raspberry pi and two mSATA to USB bridges).
Moving/replacing/upgrading drives has been a breeze with BTRFS while using the system.

With one single exception: after finishing moving the last partition and removing the LVM volume group that was used to partition the old storage... SystemD panicked, because it suddenly did not see the old device name (containing the VG. I should have used UUID instead in my fstab, I know...) though that something was broken and kicked me out of my logged session.

But SystemD and device path shennigans aside, everything has been a breeze.

Main PC is regularly (like once a month) dropping one drive, probably I need to replace some SATA cable finally, instead of just re-seating them.

Had that on an old~ish workstation I use. Changing cable didn't fix anything (even going for SATA cable with the retension clip). Getting rid of the spinning rust actually got rid of the problem.

Not to mention ECC RAM has saved my bacon as well, more than once.
Modern computing and linuxing is amazing, I love it.

I use ECC on my big server, plan to use it in my next workstation, but never noticed any error recovery. Though haven't been checking very often.

Yup, modern Linux computing is great.
(And modern Steam on Linux is great once work is finished).

[Developer12]

It's also fully checksummed and fully copy-on-write like ZFS (except for files flagged with special attributes).
So basically the same: you don't try to make sense of half borked data, you use the checksum to determine which copy of the copy-on-write is the correct one.

Though in the specific case of BTRFS, there exist some fsck tools (that where written for some compliance) but they are not the first choice to use in case of accidents.

Sadly being copy-on-write doesn't in any way imply keeping multiple copies of metadata around from which to repair from.