CoreOS Moves From Btrfs To EXT4 + OverlayFS

[Kano]

Maybe btrfs is more easy to recover if no compression is used, I don't use it without. Did you ever enable lzo compression?

[chrisb]

Actually, you don't seem to know what happens with BTRFS. "Deleting stuff" doesn't free up space.

You were taking about snapshots. When you delete a snapshot, all of the snapshot data (delta from original data) and the metadata is deleted, and that space is available for reuse. Additionally, if you delete data and it is the last copy of that data (no more references from other sub volumes/snapshots) then the space will immediately be available for reuse. Snapshots should be deleted with the btrfs tool and not rm (using rm to make changes to files under a snapshot root will increase the used metadata size and probably the delta size too).

[darkbasic]

"Why not give XFS a try?"

I used it for quite a lot in my webserver because of per-directory quotas. After some time it started having such low performance that I had to switch to ext4 and get rid of quotas. No more thanks.

[balouba]

using it as well on several machines for about 2 years here. did run into an issue a few times: when the fs is improperly umounted (power failure, killing a vm, pulling out a usb drive, etc. stuff ppl shouldnt do but do all the time regardless) in some rare cases btrfs would corrupt files. ive a feeling its the one bug most other ppl hit.
Its fixed in recent versions though.

Id also note that while ext4 received a lot of testing and real world use, overlayfs didnt get all that much. openzfs either.

[jimbohale]

using it as well on several machines for about 2 years here. did run into an issue a few times: when the fs is improperly umounted (power failure, killing a vm, pulling out a usb drive, etc. stuff ppl shouldnt do but do all the time regardless) in some rare cases btrfs would corrupt files. ive a feeling its the one bug most other ppl hit.
Its fixed in recent versions though.

Id also note that while ext4 received a lot of testing and real world use, overlayfs didnt get all that much. openzfs either.

overlayfs is a lot less complicated than something like btrfs. I suspect that they encountered a few show stopping bugs in btrfs like kernel panics, I've encountered a few of those myself or data corruption bugs but those were when it was marked as experimental so I don't hold it against it. ext4 is a tried and true filesystem, so is xfs (much more so than ext4 actually but lacking drive shrinking). I suspect they want to give it a few more years, which I agree with considering CoreOS's target audience of enterprise.

Facebook could probably manage to run minix with their development team, so the argument of it being good enough for facebook isn't quite a valid one IMO. They have a huge development team and a HUGE systems administration team. Much larger than most teams, and much more oriented towards low-level hacking than a lot of other teams are going to be. That said, I suspect that they don't use it for things like database storage where integrity is a bit more crucial (edit: just googled it, they are trialing it, not necessarily running it in main production).

[Nelson]

Is there any science on the root of the problems? All the popular COW filesystems seem to have problems when very low on disk space. There is also the autodefrag mount option which is supposed to fix the meta data rebalance and slowdown issues. Is there a more detailed dump of information? Were these problems happening on "normal" systems or were they tightly constrained on disk space or what?

[dee.]

This is what happens when sheeple

Wow. Did you just use the word "sheeple" unironically?

Wow.

[Slartifartblast]

Just the fact that I could fill up my drive and not being able to restore the space without adding another device to the pool is a pretty crucial flaw. The distro was a OpenSUSE factory snapshot, so, definitely a rolling distro from the start.
Happened on two machines, at around the same time. I probably have to look over them again and rebalance them before they get both get f'ed up again.

I haven't had time to figure out how to make it stop taking snapshots all the fucking time, so I just delete them for now.

I ran into this little bit of fun when Opensuse Factory first came out and it's a royal pain in the arse to kill it properly, at least I found out about it while testing it as a VM and swiftly made my mind up to stick with good old EXT4 while booting BTRFS into touch.

See here

https://forums.opensuse.org/showthread.php/501164-Btrfs-Why

[xeekei]

overlayfs is a lot less complicated than something like btrfs. I suspect that they encountered a few show stopping bugs in btrfs like kernel panics, I've encountered a few of those myself or data corruption bugs but those were when it was marked as experimental so I don't hold it against it. ext4 is a tried and true filesystem, so is xfs (much more so than ext4 actually but lacking drive shrinking). I suspect they want to give it a few more years, which I agree with considering CoreOS's target audience of enterprise.

Facebook could probably manage to run minix with their development team, so the argument of it being good enough for facebook isn't quite a valid one IMO. They have a huge development team and a HUGE systems administration team. Much larger than most teams, and much more oriented towards low-level hacking than a lot of other teams are going to be. That said, I suspect that they don't use it for things like database storage where integrity is a bit more crucial (edit: just googled it, they are trialing it, not necessarily running it in main production).

Facebook only fairly recently started heavily testing Btrfs. I think its quality and development will certainly pick up speed from here. While I hate Facebook the site, and its privacy issues, I love what Facebook brings to Linux. They also are doing some work on the network stack to make it rival FreeBSD, an area our cousin OS has been better at.

[rdnetto]

My OpenSUSE install picked btrfs by default so I thought I'd go with it.
A few weeks later I'm stuck trying to remove data from the drive since it filled up in no-time with tons and tons of fucking snapshots.
Had to get a hold of a usb stick and format that to btrfs and add it so that I could REMOVE DATA (since removing files doesn't actually free up space, you need to rebalance the metadata manually )

Why would a rolling distro use this by default, when users have to manually watch out for their drive filling up and manually run rebalancing and deletion of snapshots every now and then? All through a few dozen terminal commands.

If you are administrating larger systems I'm sure btrfs could be really useful, but as the default for a desktop? No fucking way.

I think the real problem you have is how OpenSUSE configures btrfs.
Btrfs snapshots are great for rolling release distros, because they can provide an easy way to undo changes in case an update breaks everything. (Which does happen every so often, depending on how bleeding edge your distro is.) I use that exact setup, and it works brilliantly for me.

However, it has some important caveats that are basically the result of the snapshot applying to the entire subvolume:

• /home needs to be a separate subvolume, or it'll be included in the snapshots.
• The snapshots need to be automatically deleted after a few months
• If there are any databases, etc. stored on the rootfs, they also need to be separate subvolumes.

I do agree that needing to add storage in order to be able to delete files is really counter-intuitive though. The btrfs developers should probably set the defaults to create some reserved space on btrfs volumes specifically for improving this corner case.

What do you expect it to do? When your drive is full, the only options are to add more space or delete stuff. Obviously it can't just automatically delete your data... So if you fill the drive, you will need to either add more space or manually delete stuff.

The biggest gotcha associated with snapshots, IMO, is that deleting files won't necessarily free up space - the only reliable way to do that is to delete old snapshots. If you store a very large file, then realize you need to delete after a month of snapshots, the only way to free up the space associated with it is to not only delete it, but delete all the associated snapshots as well. For this reason, I have an XFS partition (a btrfs subvol would also have worked, but I wanted to use an SSD for it) mounted at ~/scratch for various temporary files.

Btrfs has some great features, but it's easy to shoot yourself in the foot with it. I think it's a bad idea for distros to implement things like automatic snapshots, because they have implications that the user really needs to know about - just google 'btrfs enospc' to see how many people have had similar to experiences to Micket.

Is there any science on the root of the problems? All the popular COW filesystems seem to have problems when very low on disk space. There is also the autodefrag mount option which is supposed to fix the meta data rebalance and slowdown issues. Is there a more detailed dump of information? Were these problems happening on "normal" systems or were they tightly constrained on disk space or what?

It's well known that btrfs runs into a lot of issues when low on disk space, since it needs to create a copy whenever it does anything. This compounded by the following:

• automatic snapshots can fill up your disk quite easily
• snapshots make freeing up space occupied by large, old files difficult
• plenty people use an SSD with btrfs for /, which means there isn't that much space in the first place
• figuring out how much free space you actually have is quite difficult under btrfs. For example, there's no way to determine how much space is used by a subvolume or a snapshot - you can only access stats on a per-volume level.

The general consensus seems to be that btrfs is fine, as long as you never let the disk usage grow beyond 90%. (Adding reserved space would essentially formalize that, though people would likely complain about the loss of space.) But I can definitely see why someone who could use ext4+overlayfs instead would avoid it - the complexity and problems associated with these features aren't negligible. I use btrfs all of my systems, but I would never suggest its use in an enterprise setting where you want to be able to forget it's even there.