Bcachefs Hopes To Remove "EXPERIMENTAL" Flag In The Next Year

[DumbFsck]

Imagine running a 100TB+ storage with a experimental filesystem...

Seems reasonable. 100TB is right around what I wouldn't mind having duplicated just for testing purposes (so in addition to backups and all other kinds of duplication).

Like, how much does it cost right now? 10k EUR in (used) Kioxia 3.8TB sas drives that you can do >5 DWPD easily...

And if you're down for spinning rust, a tenth of the price...

[EmanuC]

openzfs recently went on and silently corrupted data for quite some time. Such has not happened yet on bcachefs.

Are you sure about that?

Ok, I tried to boot into a Fedora Rawhide ISO with kernel 6.8-rc7 to see if it would help. Looks like now I can't even try to do anything. Both dmesg and bcachefsck give this error:

bch2_dev_in_fs() Split brain detected between /dev/sdd1 and /dev/sda1: /dev/sda1 believes seq of /dev/sdd1 to be 49, but /dev/sdd1 has 58 Not using /dev/sdd1 bch2_dev_in_fs() Split brain detected between /dev/sdc and /dev/sda1: /dev/sda1 believes seq of /dev/sdc to be 49, but /dev/sdc has 58 Not using /dev/sdc bch2_dev_in_fs() Split brain detected between /dev/sdb and /dev/sda1: /dev/sda1 believes seq of /dev/sdb to be 49, but /dev/sdb has 58 Not using /dev/sdb bch2_fs_open() bch_fs_open err opening /dev/sda1: insufficient_devices_to_start





I guess my previous attempt to run kernel fsck messed up the FS. I guess this FS is actually gone now ... oh well.

So much for "the filesystem that won't eat your data"

https://github.com/koverstreet/bcachefs/issues/656#issuecomment-1984324914

[flower]

Are you sure about that?


https://github.com/koverstreet/bcach...ent-1984324914

it sounds like the fs was mountable in the end without lost data.
yes it is a bug. but bcachefs hasnt eat any data (or silently corrupted it).

[EmanuC]

it sounds like the fs was mountable in the end without lost data.
yes it is a bug. but bcachefs hasnt eat any data (or silently corrupted it).

Read it again carefully. I quoted the comment from the user who lost their data and had to format, but it seems like you ignored that comment.

I guess my previous attempt to run kernel fsck messed up the FS. I guess this FS is actually gone now ... oh well.

So much for "the filesystem that won't eat your data"

I can no longer even attempt to mount the fs unless i do -o very_degraded and try with only one device, because of the "split brain" errors. I think my attempt at running kernel fsck butchered the FS and now it refuses to use multiple devices.

Would it still be useful to try to debug it in that state?

I have not reformatted yet, I still have the broken FS. I could try to get on IRC tonight (for me in EU, by your posting times above I guess you are in a NA timezone) to help debug if it would be of any use. Otherwise, I will reformat tonight and reinstall my OS.

Another user:

Sorry, I was in a bit of a rush so I reformatted and restored backups. The rebuilt environment is exactly the same though, so there's a chance that the problem will reappear if we're lucky.

Another user:

However, after fixing, there are several files with 0 bytes (but with latest modified time, thus they deceived `rsync` unless i use option `-c` for checksum), and some other files with the correct size but just garbage data (because they are just yml config file, but after fsck'ed it has a repeating non-ASCII chars) ==> basically data loss, "it still ate my data".

Another user:

Data Corruption


Then it really started going downhill. Right at the top of the bcachefs.org homepage it proudly proclaims

The COW filesystem for Linux that won’t eat your data.

Unfortunately my experience was not in agreement with this claim. Within a day of use some bcachefs task got in an infinite loop and froze the system. I had to force a shutdown and then the system wouldn’t boot up again. I plugged in a monitor and found the following error:

[flower]

Read it again carefully. I quoted the comment from the user who lost their data and had to format, but it seems like you ignored that comment.

well if someone formats before trying to rescue the data no one can help.
anyway: every fs has bugs. i am sure bcachefs will have a data eaten bug too. i am just not convinced that one has been discovered yet.

a silent data corruption (as with zfs) is the worst case anyway. everything else can be solved with backups. luckily i wasnt affected (i still use zfs on my nas but plan to switch to bcachefs in the future)

[DumbFsck]

Read it again carefully. I quoted the comment from the user who lost their data and had to format, but it seems like you ignored that comment.






Another user:


Another user:


Another user:

It seems you didn't read the whole discussion. Everyone who was there with the same issue recovered their data.

Even the fellow who re-formatted, when they had the same issue again later, was able to recover everything.

The blog post is from much earlier than bcachefs being added to the kernel.


The only "damning" link of yours is the reddit post, but we will never know if it was recoverable or not.



From what you sent of github, it really does seem it doesn't eat data. It might corrupt everything, but then is able to recover - unless the user reformats before a fix is up.

[Britoid]

You meant XFS in the Stratis part. Red Hat fucking hates OpenZFS because it makes Stratis look like a bad joke.

Bcachefs still has a lot of work to do to be anywhere near the robustness and usefulness of OpenZFS.

What makes you think they hate OpenZFS?

RHEL doesn't need it, it's provides no value to the intended use case of RHEL. The only advantage is the tooling (LVM/LG CLI is horrible), which is what Stratis is designed to help with. XFS is a far more trusted filesystem.

A non-free filesystem poses no threat to Red Hat. Fedora uses btrfs because the non-Red Hat community wanted it, as CoW, snapshots, compression etc are valuable for desktops. But it has little chance of being adopted in RHEL.

[Britoid]

Seems reasonable. 100TB is right around what I wouldn't mind having duplicated just for testing purposes (so in addition to backups and all other kinds of duplication).

Like, how much does it cost right now? 10k EUR in (used) Kioxia 3.8TB sas drives that you can do >5 DWPD easily...

And if you're down for spinning rust, a tenth of the price...

I think a larger file system is unlikely to run into as many bugs, as it's likely just to be many writes and not many deletes.

[Daktyl198]

It's honestly wild to me that such an unstable and in-development filesystem is even a part of the kernel to begin with. It should be developed out-of-tree until it's at least halfway stable and nearing 1.0 in terms of features.

[cutterjohn]

no, there are a few people: https://evilpiepirate.org/git/bcachefs.git/log/

that's ridiculous the VAST majority ARE FROM kent!

It's ajoke that he thinks that he can remove the experimental flag from bcachefs next year when, e.g. Ubuntu still have the very tried and true openZFS still under experimental ffs! Simply because a handful of loudmouths don't care for the license! WTF?!

I've been in linux since pretty much the beginning FFS! And the GPL fucktards are really irking me nowadays! Oh it's not gplv3 it's not OSS, etc. It's fscking TIRESOME FFS! What's worse is that they will circle jerk instead of doing anything useful like oh say suggesting an alternative....

Then again personally I find zero problems with permissive licenses nor non GPL more restrictive licenses...


...and then let us not even start on things like CLA or CDDL(ZFS 'problem') but I digress this community has become WHOLLY TOXIC, justy lok as this fscking forum FFS!

ENOUGH IS ENOUGH ALREADY!

[EDIT]

..and linux since the beginning I mean that I read Linus' fscking post on comp.os.minix in fscking 1991 I had similar desires but I was on mac hardware at that time, using sunos at university... I built an x86 AMD machine the next year and hey had loads of fun bootstrapping linux from a trillion(OK I exaggerate slightly) floppies! You whiners today make me want to puke!

[/EDIT]

[EDIT2]

I am VERY sorry for the VERY OFFTOPIC comment at the end of the day. It is ONE, if not my firstmost pet peeve with most forums where topics RAPIDLY digress from the original subject but reading the forums here again... and seeing how often the topic diverges away into something entirely else rather than moving it elsewhere... yeah, and I am guilt of that myself here, but I will NOT remove the post.

I too wish that I were in Munich and having way too much beer, but alas and alack...

[/EDIT2]