Btrfs Async Buffered Writes Slated For Linux 6.1 - 2x Throughput Improvement

[coder]

It doesn't immediately mean the use case is useful for all comers, especially people that are using it on commodity hardware. That last kicker is why ext4fs is 'over engineered' to a fault because of all of the corner case, gotchas, and outright screwiness of PC/consumer grade hardware.

IIRC, ext4 is "over-engineered" to the point that a lot of those reliability features aren't actually used on commodity hardware (e.g. barriers and ordered-data mode), because they tank performance too badly.

You might just dismiss that as a problem caused by hardware manufacturers, but the practical matter is that all hardware is capable of causing hiccups even with extremely good ECC from the RAM to the CPU out to the IO buses.

BTRFS' checksums are meant to cover those cases, but I'd point out that both SATA and PCIe have a basic checksum feature. So, it's mainly memory errors and faulty drives that you should worry about.

I find it laughable to suggest ext4 is a more stable alternative for low-quality hardware, when it has no data checksums. From what I'm reading, even its metadata isn't completely checksummed.

BTRFS has historically been poor at anything Facebook itself doesn't use which has lead to data loss over the years.

That sounds like one of those "stupid ... broad sweeping statements" of which you spoke so highly. Please, cut the BS.

openSUSE has been using BTRFS as the default root filesystem for like the past 5 years, already.

[coder]

yes they run btrfs, but not on bare metal.

Do their employees have workstations or laptops? What filesystem(s) do you suppose those use?

[zexelon]

Do their employees have workstations or laptops? What filesystem(s) do you suppose those use?

I am unclear if this was supposed to be a rhetorical question or perhaps something holding up a sarcasm sign?

But in any event, I have no idea what file system any employee would use, and quite frankly don't care. Their data is 100% expendable. They should never have any corporate code, business intelligence or the like on their local file system. If you are any mid-sized or larger company and allow this, you are doing it wrong... very very wrong regardless of the file system your employee is using.

[ATLief]

Are these "async buffered writes" relevant for normal users? Do they happen during normal operation or is this some exotic corner case which is used only very rarely?

Admittedly I’m not an expert in anything, so this might be all wrong. But my understanding is that basically all IO on Linux is async buffered. Applications would have to explicitly specify otherwise and it hurts performance for little benefit, so most things stick with the default. The “barrier” feature (which is enabled by default despite slightly hurting performance) was needed specifically because of buffering.

So hopefully this speedup will be something that almost everyone benefits from almost all of the time.

[RahulSundaram]

Originally you said this:

Just as a heads up, almost every btrfs discussion uses Facebook as some champion for deploying btrfs. Facebook DOES NOT actually use btrfs except in some edge case experimental setups. They did do a deep dive into it, but all indications now are that it is dropped.

Now you say this

I could totally see Meta/FB still supporting btrfs heavily, but only for their use case in containers.

Your statements don't agree with each other at all especially since FB relies on containers heavily.

[zexelon]

Originally you said this:



Now you say this



Your statements don't agree with each other at all especially since FB relies on containers heavily.

Agreed, and based on the information provided by skeevy420 above. The premise as discussed in the rest of the post you "cherry picked" from is referring to bare metal.

At the end of the day, I dont actually personally care to argue this much further as its starting to go in circles and as I mentioned above I am not a Meta/FB employee so take it all with a grain of salt.

The premise of the statement that started this whole circular arguing is that people hold up FB as some reason to install a distro on a btrfs, but that is wrong. Facebook does not run btrfs on bare metal, they run xfs under Tectonic. Based again on the information supplied above it looks like they use btrfs in containers, but this is a very specialized use case and not one that regular users are going to run into.

All of that said to bring this thread back on point, its awesome to see improvement in btrfs, but there are still a lot of use cases (i.e. the ones not supported by Meta/FB) where it seems to have bugs (i.e. RAID 5/6 that is still a use option but is known to cause data loss).

[jacob]

man, it's getting hard to recognize trolls from genuine Rust fanboys!

Trolls and fanboys have always been one and the same, whether it's about Rust it anything else. Intelligent proponents of any technology have a genuine understanding of its strengths and they know where it has tangible benefits and where it doesn't. They can ridicule trolls/fanboys just as easily as the self-proclaimed "experts" who "know" that whatever they have been using for the past 20 years has no issues and needs no replacement.

PS: using Rust for a new FS designed from scratch might be sensible. Rewriting something as complex as btrfs from scratch in another language is a terrible idea.

[skeevy420]

Hey thanks for the link! This is not exactly the same application though, yes they run btrfs, but not on bare metal. From everything that I have seen its in their containers only. Based on that my guess is they are running btrfs in its most basic mode possible (i.e. single virtual block dev, no pseudo RAID, etc.).

My interpretation of this thread so far was in reference to bare metal and as far as I know the vast majority of Meta (aka Facebook) is run on xfs backing Tectonic. This is where they store all those btrfs containers from what I understand.

Now take all of this with a grain of salt... I am not a Meta employee! Just a huge fan of Tectonic and what they built there.

I could totally see Meta/FB still supporting btrfs heavily, but only for their use case in containers.

Do you have anything that actually supports your container view? So far all you've given is a link to a cluster filesystem running on top of XFS for shared data. That doesn't prove or disprove what they're using as their root file system nor does it prove or disprove your container position. Who is to say that they're not using BTRFS for root and XFS+Tectonic for shared data drives? That seems to be either already be the case or one of their goals based on all the data I've seen.

Last year Phoronix reported on Facebook using CentOS for their servers and Fedora for their desktops and BTRFS root was one of the deciding factors in regards to Fedora. Facebook is also part of the CentOS Hyperscale SIG that maintains a BTRFS enabled kernel, COW enabled DNF, and other things of that nature. They're literally backing BTRFS on CentOS. Are they backing a BTRFS root? Hell if I know. All the stuff I come across in regards to their CentOS SIG and BTRFS is generic, nothing root specific. All the root stuff is around Fedora...but it isn't a stretch to think they'd want all the BTRFS features like snapshots and whatnot for their CentOS servers.

I wish I gave enough of a shit to run their either their GNOME or KDE installer long enough to see if they offer BTRFS as a root file system for CentOS Stream (since BTRFS isn't an officially supported CentOS file system). Granted those are nearly a year old so who knows how much has changed in the meantime or even if BTRFS root was one of their goals with those releases. As a curious person it'd be nice to see those ISOs get some updates and for the Hyperscale SIG to clarify their stance on BTRFS root.

[F.Ultra]

Are these "async buffered writes" relevant for normal users? Do they happen during normal operation or is this some exotic corner case which is used only very rarely?

This is for performing writes using io_uring so we have to wait for the every day applications to start using io_uring before we will see the benefit from this.

[zexelon]

Do you have anything that actually supports your container view? So far all you've given is a link to a cluster filesystem running on top of XFS for shared data. That doesn't prove or disprove what they're using as their root file system nor does it prove or disprove your container position. Who is to say that they're not using BTRFS for root and XFS+Tectonic for shared data drives? That seems to be either already be the case or one of their goals based on all the data I've seen.

Last year Phoronix reported on Facebook using CentOS for their servers and Fedora for their desktops and BTRFS root was one of the deciding factors in regards to Fedora. Facebook is also part of the CentOS Hyperscale SIG that maintains a BTRFS enabled kernel, COW enabled DNF, and other things of that nature. They're literally backing BTRFS on CentOS. Are they backing a BTRFS root? Hell if I know. All the stuff I come across in regards to their CentOS SIG and BTRFS is generic, nothing root specific. All the root stuff is around Fedora...but it isn't a stretch to think they'd want all the BTRFS features like snapshots and whatnot for their CentOS servers.

I wish I gave enough of a shit to run their either their GNOME or KDE installer long enough to see if they offer BTRFS as a root file system for CentOS Stream (since BTRFS isn't an officially supported CentOS file system). Granted those are nearly a year old so who knows how much has changed in the meantime or even if BTRFS root was one of their goals with those releases. As a curious person it'd be nice to see those ISOs get some updates and for the Hyperscale SIG to clarify their stance on BTRFS root.

If you read the full white paper it would help.