Bcachefs Working Towards Online Fsck, Faster Mount Times

[starshipeleven]

Such comment was bound to be made, maybe I should've linked this in advance:

https://forums.freenas.org/index.php?threads%2Fsingle-drive-zfs.35515%2F#post-216140

Teaser:

Well, the CTO of iXsystems said something like "single disk ZFS is so pointless it's actually worse than not using ZFS".

And on the desktop you rarely use pairs of disks. Especially in laptops.

FYI: You can do a RAID1 with 2 partitions on the same drive.

If you are on a SSD the performance does not tank that much, and you will have bitrot protection.

But yeah, that's one of the reasons I'm using Btrfs and not ZFS.

[starshipeleven]

I have tried btrfs three times. Once in 2013, once in 2015 (one of those was a Fedora install), and once a month ago when it ate several terabytes of data while in (12TB + 12TB mirror) RAID10. That should never happen. I luckily had a backup on some older hard drives formatted with XFS. In 2013 I had it as the root partition and one day it would not boot and the partition was borked. In 2015 I had major performance issues related to it that continued to slow down the system until it no longer functioned.

So I will never trust btrfs. I wasted 2 weeks trying to recover it and the best I got was cp to some spare hard drives and see what fails and what doesn't. Every time someone says "Oh, well that was true in YYYY, but now it's so good!" I can't believe them.

A filesystem meant to replace ZFS/ext4/XFS, or whatever, shouldn't have unrecoverable data loss with something like RAID10, ever. I can't stress how incredibly incompetent it is to have a filesystem that was started about 9 years ago and still has these issues. It's great that you weren't burnt, but when I expect that thing to be rock solid and I almost lost wedding footage among other things which I would NEVER be able to just redownload or recompile or reinstall.

Could you mention what distro you used in the last test? I'd like to avoid it like the plague.

[danboid]

The big draws with ZFS for me are its non-expensive snapshots and boot environments. Last I heard, performance under BTRFS tanks after a few snapshots are made. Is this still the case?

I didn't like the way the Antergos installer handled ZFS (it relied on a ext4 /boot last time I tried it) and I'd rather use regular Arch instead of the Antergos repos which is why I created ALEZ, the Arch Linux Easy ZFS installer:

https://github.com/danboid/ALEZ

Arcan sounds like a much better thought out replacement for X than Wayland to me, and it seems to be further ahead in some respects despite not getting anywhere near as much attention or funding as Wayland. I like that there is only one implementation rather than it merely being a protocol for a start. Per display DPI settings is what I miss from crusty ol' X. Arcan has this.

https://arcan-fe.com/about/

[blackiwid]

I have tried btrfs three times. Once in 2013, once in 2015 (one of those was a Fedora install), and once a month ago when it ate several terabytes of data while in (12TB + 12TB mirror) RAID10. That should never happen. I luckily had a backup on some older hard drives formatted with XFS. In 2013 I had it as the root partition and one day it would not boot and the partition was borked. In 2015 I had major performance issues related to it that continued to slow down the system until it no longer functioned.

So I will never trust btrfs. I wasted 2 weeks trying to recover it and the best I got was cp to some spare hard drives and see what fails and what doesn't. Every time someone says "Oh, well that was true in YYYY, but now it's so good!" I can't believe them.

The development of Btrfs began in 2007, and since August 2014 the file system's on-disk format has been marked as stable.

So you used a filesystem that was marked as unstable and lost data and then complain here about it? The problem sits on your end before the monitor.

Here as example a data loss on the first google page of xfs in 2009 maybe there are more recent ones I don't care:

http://xfs.9218.n7.nabble.com/xfs-data-loss-td27930.html

That filesystem was released 1994 and got integrated 2001 into the linux kernel. And compared to the (real?) CoW filesystems it's very simple has no raid features as example.
But you expect a filesystem which development startet 6 years ago which is that complex and not be marked as stable to work without problems?

So you might say ZFS worked probably at that point true but it was releasen in solaris 10 2006 which means it had already developed for a few years so the devolpement startet probably 2003 or even earlier than that.

I am pretty sure it ate some data in solaris 10 too. Heck ZFS has big bugs till this year:

https://www.theregister.co.uk/2018/04/10/zfs_on_linux_data_loss_fixed/

[profoundWHALE]

The development of Btrfs began in 2007, and since August 2014 the file system's on-disk format has been marked as stable.

So you used a filesystem that was marked as unstable and lost data and then complain here about it? The problem sits on your end before the monitor.

Since you can't read, I didn't bother to read the rest of your post. I was using BTRFS on a RAID10 system which borked itself a month ago, as in, the year 2018. I had been using it for quite some time and it appeared stable when it randomly destroyed data, and I'm not taking about new writes. Random data was just... lost. Drives are less than one year old AND in RAID10 so they should NEVER have had that happen.

[profoundWHALE]

Could you mention what distro you used in the last test? I'd like to avoid it like the plague.

It was either Ubuntu or Solus and I can't remember which. Edit: It was probably Solus, as I was using Solus when I was looking at replacements for BTRFS.

[profoundWHALE]

Care to share some details? How many disks , what kind of setup? Post to the mailing list? etc etc... I would love to dig into this a bit.

I had 4 6TB HGST 7200RPM HDDs configured with RAID10 through BTRFS's own software raid thing. It was automounted at boot with fstab and hosted on NFS and Samba.

At one point I either tried to copy, or open, or move, or rename a file and I got an input/output error. Uh oh. Scrub failed due to errors and fsck didn't find anything to fix. I tried to recover the files with the super block and tree rebuilding commands but I ended up wasting about 1.5 days per run of any of those commands.

Eventually it got to the point where I had to find some spare hard drives and copy only what I really needed (I can redownload my steam library) to both save time and because I don't have that many spare hard drives. The cp command would fail on whatever file was borked and I'd just have to hope that I had backups.

[DrYak]

FYI: You can do a RAID1 with 2 partitions on the same drive..

Yes, intheory. (BTRFS even has a mode for that called "dup" that does it on the same partition, and it's the default for metadata stored on spinning rust).

If you are on a SSD the performance does not tank that much, and you will have bitrot protection.

Nooo.

Any flash controller chip, in its flash translation layer code, will try to group writes together.
If you write 2 copies of something at the same time, there's a high chance they'll get written next to each other in flash. (Maybe even in the same erase group).

So you get some (very limited) protection against single bit flips (something that ECC should also protect you from anyway, so anything better than cheap consumer SD cards should have bitrot resiliance) (though some manufacturer like Transcend offer ECC even on some consumer, non-industral SD cards).
But you won't get much portection against an erase group getting toaster on the flash chips. Chances both your original and your copy where stored on the same group and you lost both at the same time.

That's the reason why BTRFS defaults to "single" instead of "dup" for metadata on non-rotational media.

At one point I either tried to copy, or open, or move, or rename a file and I got an input/output error. Uh oh. Scrub failed due to errors and fsck didn't find anything to fix. I tried to recover the files with the super block and tree rebuilding commands but I ended up wasting about 1.5 days per run of any of those commands.

You ran fsck, and you tried rebuilding trees. There are your problem. On a COW (like Btrfs, Zfs, BcacheFS, etc.) or Log structured (UDF, F2FS, etc.) you don't. Never. Only as a last resort.
You first try to fall back on an older version of your filesystem (the "super block" was a correct way to try). If that ones doesn't manage to access your data, the next best things is to use "btrfs restore" to recover the files.
Only once you have stored copy of you file may you start fucking up your partition with "--repair".

Eventually it got to the point where I had to find some spare hard drives and copy only what I really needed (I can redownload my steam library) to both save time and because I don't have that many spare hard drives. The cp command would fail on whatever file was borked and I'd just have to hope that I had backups.
[/QUOTE]

[profoundWHALE]

You ran fsck, and you tried rebuilding trees. There are your problem. On a COW (like Btrfs, Zfs, BcacheFS, etc.) or Log structured (UDF, F2FS, etc.) you don't. Never. Only as a last resort.

I already said that there was a major issue (input/output error). It was impossible to do any recovery of previous superblocks because it would error out. I could not scrub because it would error out. I suspected a hard drive failure, but then I remembered that I had redundancy, so I tried to check for a good file that could be transferred, but that errored out. I did my research before attempting any repairs and only did fsck once I was sure that I would get no further, and it was still a waste of time.

Either way, my point stands: this never should have happened on a filesystem that is designed in a way so that it should never happen unless there is some catastrophic failure which would kill anything.

Also, I had an unexpected shutdown while transferring everything to Bcachefs and ran fsck, and everything was fine. There's no reason why that should destroy data.