Bcachefs Multi-Device Users Should Avoid Linux 6.7: "A Really Horific Bug"

[S.Pam]

I have had numerous single disk btrfs filesystems destroyed by a power event. None were recoverable beyond running btrfs recover and dumping about 30% of the data on the fs to another drive

Here's the problem. Faulty hardware that plainly lies about supporting barriers.

[Old Grouch]

I would recommend, ZFS.

Individual recommendations are anecdote, not data.

There needs to be objective testing (waves at Michael) of filesystems over the long term (stops waving at Michael) to generate good data on the behaviour of filesystems. Some people have a use-case which is benign, and the filesystem copes well with their use, so they are a 'happy bunny': others can have a pathological use-case that exposes nasty bugs or unwanted behaviour, so they are unhappy. In essence, unless you are part of a long-term file-system test project, any statement you make is pretty much 'it works for me' or 'it doesn't work for me', which is an almost completely empty statement.

What we do know is that the average user uses a file system on hardware built down to a price to meet a notional bit-error rate, with power supplies of unknown quality, in an unknown electromagnetic environment, with no ECC. So what is needed are robust files systems that cope with running on fallible hardware, and don't make the assumption of zero-BER (among other things). We need filesystems that make recovery from corruption and failure easy, assuming they detect it in the first place. So yes, ZFS, btrfs and bcachefs look like good candidates, but the devil is in the details.

[cynic]

Individual recommendations are anecdote, not data.

There needs to be objective testing (waves at Michael) of filesystems over the long term (stops waving at Michael) to generate good data on the behaviour of filesystems. Some people have a use-case which is benign, and the filesystem copes well with their use, so they are a 'happy bunny': others can have a pathological use-case that exposes nasty bugs or unwanted behaviour, so they are unhappy.

this is not "anedoctical".
if a usecase triggers a bug, then it's a bug.

[fitzie]

Because he changes the fs as it progress'es so the upgrade tool is necessary. Reverting changes to the upgrade tool defeats the point of it exisiting.

I mean why the f**k are you questioning how he is solving the bug. He solved it in ~one week, not 8 years (unlike some other filesystems) and while his language his harsh its understandable in this case its a very critical bug. I personally maintain open source software that is used in mission critical scenarios (i.e. bank payment systems) that has LTS branches and such bugs are backported basically without question, it shouldn't be an argument.

I'm not questioning how he's "solving the bug". I'm questioning why he gave such poor notice to his users. he waited five days after coming up with a fix to tell anyone about the bug, so I don't believe it's so critical. even in the bug report he doesn't even mention where people can find the fix. and yeah, if tools triggered the bug, then wouldn't telling people what version of tools inform them to exactly what versions are safe and which aren't (so they can make decisions on when and if to upgrade kernels)? this isn't just harsh language, this is really just chaotic behavior.


and I'll say this for the last time. I've donated to bcachefs patreon for many years because I had good hope for the project, and I still do. this is why i'm kinda shocked about how kent is letting everyone down with his bizarre behavior. his is capable of being pleasant and rational but is too often not acting that way, and his defenders don't accept that this will lead to his own frustration and thereby negatively effect bcachefs's chances of success.

[Quackdoc]

Individual recommendations are anecdote, not data.

There needs to be objective testing (waves at Michael) of filesystems over the long term (stops waving at Michael) to generate good data on the behaviour of filesystems. Some people have a use-case which is benign, and the filesystem copes well with their use, so they are a 'happy bunny': others can have a pathological use-case that exposes nasty bugs or unwanted behaviour, so they are unhappy. In essence, unless you are part of a long-term file-system test project, any statement you make is pretty much 'it works for me' or 'it doesn't work for me', which is an almost completely empty statement.

What we do know is that the average user uses a file system on hardware built down to a price to meet a notional bit-error rate, with power supplies of unknown quality, in an unknown electromagnetic environment, with no ECC. So what is needed are robust files systems that cope with running on fallible hardware, and don't make the assumption of zero-BER (among other things). We need filesystems that make recovery from corruption and failure easy, assuming they detect it in the first place. So yes, ZFS, btrfs and bcachefs look like good candidates, but the devil is in the details.

data is just a collection of anecdotes. without any anecdotes we have no data.

[Old Grouch]

data is just a collection of anecdotes. without any anecdotes we have no data.

The plural of anecdote is not data.

I don't want to go into the detail of generation and testing of hypotheses, and all the background work necessary in any research following a scientific method. Anecdotes can generate ideas suitable for further testing, and whole swathes of academia rely on qualitative methods, but a collection of anecdotes, no matter how large, are not informative. Processing according to pre-formulated criteria is necessary to transform the collection into data from which information might possibly be derived.

It's a nice quip, but wrong. Sorry.

[Old Grouch]

this is not "anedoctical".
if a usecase triggers a bug, then it's a bug.

A bug is not a recommendation. It might be a fact.

"I recommend the chicken.", is a shared opinion.
"The chicken is contaminated with salmonella.", is a bug.
"The chicken tastes bad to me, I think it is rotten!", is an opinion that might indicate a true fact.

Recommendations without some form of backing data are pretty much valueless.

[mdedetrich]

I'm not questioning how he's "solving the bug". I'm questioning why he gave such poor notice to his users.

I'm sorry to break it to you but giving notice and then solving the bug within a week is the opposite of poor notice unless its a high level CVE. Let me also remind you that bcachefs is officially marked as EXPERIMENTAL in the kernel, in other words as a user if you are creating this filesystem you know what you are signing up for (i.e. it will have bugs like this).

I actually maintain a large open source project that is used in mission critical usecases and the only thing that's poor about this is the stonewalling refusal of putting it into 6.7 LTS, that's what would actually help users.

Like this is the problem, your incessant "can someone please think of the children" is ignoring all context and nuance.

[cynic]

A bug is not a recommendation. It might be a fact.

"I recommend the chicken.", is a shared opinion.
"The chicken is contaminated with salmonella.", is a bug.
"The chicken tastes bad to me, I think it is rotten!", is an opinion that might indicate a true fact.

Recommendations without some form of backing data are pretty much valueless.

a usecase that trigger a bug IS actual data.

[Old Grouch]

a usecase that trigger a bug IS actual data.

Of course, yes. Are we in violent agreement? Or is one of us missing something?