Bcachefs File-System Is Working On Going Upstream In The Linux Kernel

[jacob]

My impression, by reading what he publishes here and there is that his gut feeling is that he has tackled a lot of hardcore problems as part of his formely "tiered caching" / nowadays "copy management groups", and hope to be able to leverage all the work from BCache and BCacheFS to implement cheaply snapshots on top (just as BTRFS leverages their specific flavour of B-trees to manage the COW and Snapshotting).

My gut feeling is that there could still be unforeseen hidden gnomes lurking somewhere in there, ready to jump and bite...
{...}
And maybe, if kent discovers that he won't be able to bring snapshotting to the current BcacheFS, he'll develops a BCacheFS2 with what's need for a working snapshotting.

(My gut feeling is that this exactly what is going to happen)

Kent certainly knows the BCache layer better than I do so I wish him that it works out the way he hopes. The point is though that BCache has never been designed for this kind of functionality, and he admits that he hasn't really started working on that part yet. I'm not predicting that it's going to fail, just saying that I'm curious to see it because BTRFS aside, we saw in the past a number of projects with wild claims and an implementation that was "almost there" and this is precisely the moment where things always went pear shaped. And yes, it would be great if in this case it actually works, and by basing is design on BCache he's at least trying an approach that the others didn't use.

[DrYak]

Where are you getting that encryption is finished?
The status page at https://www.patreon.com/bcachefs says, "Encryption: Not yet started"

But on the other hand, the webpage mentions it's just merged in.
(I have the impression that there is no official centralized reliable status page, but you have to gather together several tools.)
(Whatever, we count on him to make a good FS, not to be brilliant at PR/communication/webmastering)

I kind of share this feeling. Someone pointed out in another thread that the thing about btrfs is that they tackled all the hardest problems first. An it's true that it had snapshots and replication basically from day 1 - maybe variously buggy, but it was there. Whereas with Bcachefs, the hardest work remains to be done and he as good as says so himself. {...} The real "fun" comes once you go beyond that, especially with COW exposed to users (e.g. snapshots). I won't be surprised if once he aims to get there, he discovers that it's suddenly a lot harder to remain robust and keep good performance{...} because put simply, this is where Frankenstein lives.

My impression, by reading what he publishes here and there is that his gut feeling is that he has tackled a lot of hardcore problems as part of his formely "tiered caching" / nowadays "copy management groups", and hope to be able to leverage all the work from BCache and BCacheFS to implement cheaply snapshots on top (just as BTRFS leverages their specific flavour of B-trees to manage the COW and Snapshotting).

My gut feeling is that there could still be unforeseen hidden gnomes lurking somewhere in there, ready to jump and bite...

On the other hand, I kind of see the point of people who where pushing Kent to upstream his file system :
From the mailing list posts, it seems that some people though : "Well, it's already a functional file system at this stage and could already be useful for some", so let's release the current beast as a "in between gens" FS.
(The caching offered by the storage group management is useful as a form of hierarchical storage management)

And maybe, if kent discovers that he won't be able to bring snapshotting to the current BcacheFS, he'll develops a BCacheFS2 with what's need for a working snapshotting.

(My gut feeling is that this exactly what is going to happen)

[oiaohm]

stratis is just (not written yet) easier interface to existing obsolete functionality

So by you hardware raid controllers are obsolete. There are very big reasons to want a solution that works well with device mapper lead one is hardware acceleration of raid operations..

btrfs should in time work well behaved.
it is impossible to work well with system device mapper system if your definition of well includes "not being crazy slow"

You must be refering to software raid under device mapper because btrfs raid is no where close to the hardware raids under device mapper. Also the software raid in device mapper is not always slower than the btrfs raid.

"Data-only CoW limits the functionality that XFS can provide" quote from article

Is that limitation harmful. XFS functionality is limit but this means XFS has limited paths to-do particular functionality. There is more than 1 way to do what ZFS and BTRFS do and the other paths have not been fully explored.

it can't. well, when you do that, you get zfs or btrfs, because only fs knows about files

This was a line I was expecting. Ever heard of a object raid. So if the file system is telling the raid system under it what is objects then raid does not really need to know about files. Basically file system drivers need to care about file system meta data. Raid part only need to know how you wish the data stored object information can achieve this. Setting raid type by device is horrible. A proper object raid can have multi raid types inside the 1 partition.

The reality gets nastier having an object raid controller would allow the raid control to checksum block of data without having to transfer all that data to the cpu.

the xfs path is "we have old design and have to make it work somehow"

Yes its old design but is a well tested design.

it is not like xfs has any choice. its design predates idea of combining device management with fs. it is ridiculous we are seriously discussing filesystem which can't change its size

In fact xfs can grow size without issues. Horrible process you can shrink it.

http://xfs.org/index.php/Shrinking_Support

There is a list of features that need to be implemented to shrink the big problem is the first item on list. In fact being able to subvolume will help xfs support shrinking.

Last thing you want is sitting on a device mapper cow storage/lvm over allocation and getting a ENOSPC while attempt to shrink and this happens to be after you have already over written some data.

Lot of file systems support shrinking but in cow storage or lvm over allocation or other form of over allocate they fail badly when the ENOSPC happens. Problem here is currently the ENOSPC message will turn up when the buffers are attempted to be flushed to storage. So this issue nukes btrfs, zfs.... every file system that currently support shrinking if they happen to be sitting on one of these back ends as you most likely get ENOSPC message after you have already overwritten data in the resize process and blocks of data have been thrown into the void. XFS quality control is what has prevented XFS from having shrinking.

Please note when XFS was made it was made with XLV. So the idea of file system working device management first appears in the XFS/XLV combination. XLV also had object raid. It also explains why XFS does not have data checksum feature as that was a feature of the XLV as well. Turns out zfs and btrfs are not that feature great when put head to head with the old XFS/XLV combination. The current XFS is still a shadow of it past self and XFS cannot catch up with it past self until the layers its running on top of are fixed.

[pal666]

patreon does not have to be his only source of income. There are alternatives like liberapay, paypal, etc...
He may have enough money left to have enough wiggle with a little extra coming from patrons.
I read somewhere he may had a sponsor, but not sure anymore.

he may have army of minions in his castle, but he doesn't. we have just one part-time dev with ridiculous claims not backed by anything but his patreon page

[pal666]

Depends. A full-time bus driver makes about $1800-$2400/€1800-€2100 at quite some transit companies (there are exceptions, like MTA in NYC where you earn a lot more). So 1754 is close to full-time in that job.

well, you can't have more than one full-time job, so if he already has irl job, then second job is not full-time

[geearf]

patreon does not have to be his only source of income. There are alternatives like liberapay, paypal, etc...
He may have enough money left to have enough wiggle with a little extra coming from patrons.
I read somewhere he may had a sponsor, but not sure anymore.

[Vistaus]

if he does have irl job, then 1754 surely can't be a full-time job, it can only be part-time

Depends. A full-time bus driver makes about $1800-$2400/€1800-€2100 at quite some transit companies (there are exceptions, like MTA in NYC where you earn a lot more). So 1754 is close to full-time in that job. And as a freelancer, it can also be full-time depending on the job. But I agree that he probably has a higher earning job.

[pal666]

Stratis is a stop gap from description "devicemapper subsystem, and the XFS filesystem".

stratis is just (not written yet) easier interface to existing obsolete functionality

XFS should in time work well behaved with the system device mapper system.

btrfs should in time work well behaved.
it is impossible to work well with system device mapper system if your definition of well includes "not being crazy slow"

https://lwn.net/Articles/747633/

This new tricks of XFS gets very interesting on the device management side.

"Data-only CoW limits the functionality that XFS can provide" quote from article

Can device mapper subsystem be modified to work with files instead of devices as well.

it can't. well, when you do that, you get zfs or btrfs, because only fs knows about files

The XFS path is can we make file system and device management work cooperatively with each other so we don't have multi copies of device management.

the xfs path is "we have old design and have to make it work somehow"

The issue here is XFS treats itself as only part of the solution and attempt to avoid implementing everything itself where it can.

it is not like xfs has any choice. its design predates idea of combining device management with fs. it is ridiculous we are seriously discussing filesystem which can't change its size

[jacob]

Yes, he makes big promises that one day, once it's implemented, they will by fast and lightweight and nice.

I kind of share this feeling. Someone pointed out in another thread that the thing about btrfs is that they tackled all the hardest problems first. An it's true that it had snapshots and replication basically from day 1 - maybe variously buggy, but it was there. Whereas with Bcachefs, the hardest work remains to be done and he as good as says so himself. Frankly, developing a filesystem that is functionally on par with ext4 or xfs is not *that* hard. The real "fun" comes once you go beyond that, especially with COW exposed to users (e.g. snapshots). I won't be surprised if once he aims to get there, he discovers that it's suddenly a lot harder to remain robust and keep good performance and that there is, in fact, much less merit in his bashing of btrfs than he thinks, because put simply, this is where Frankenstein lives. This is really not meant as a skepticism or critique against Overstreet: he's doing a great job and if he manages to pull it off, it would be great for the entire community. But the fact is that there were similar projects before that went 90% of the way, like Bcachefs today, only to discover that the remaining 10% were basically unfeasible with what they had built thus far. Tux3 is one example that comes to mind.

[oiaohm]

its "method" is integration of fs with device management, which is not wrong. and last time i checked xfs had nothing to compete with it

Stratis is a stop gap from description "devicemapper subsystem, and the XFS filesystem".

I mentioned this as a stop gap.

https://en.wikipedia.org/wiki/XFS#Snapshots

XLV is simple to forget had existed. So XFS should in time work well behaved with the system device mapper system.

https://lwn.net/Articles/747633/

This new tricks of XFS gets very interesting on the device management side.

If XFS file system can be modified to straight up mount using a file instead of device while able still able to use device this makes you ask another question. Can device mapper subsystem be modified to work with files instead of devices as well.

This opens up a very different path. If device mapper doing raid is working on files in a generic way you end up with like ZFS and BTRFS device management but it generic so you could have a file level raid with multi different file system under it.

Yes the idea of integrating device management with file system maybe right. But making device management part of file system might be wrong. The XFS path is can we make file system and device management work cooperatively with each other so we don't have multi copies of device management.

Really software raid under file system and software raid above/part of file system have to do most of the same things.

technically speaking you will have to duplicate metadata of whole filesystem atomically during each snapshot. i guess if you can do it atomically, then you have cow. so they can't. and it will take space

XFS has Data Copy On Write. This is like how ext3cow does it. Ext3cow does COW between block layer and upper file system parts.

You don't need COW structures in the metadata if you do it from the data/block layer as XFS is lining up to-do and how Ext3cow did it.

Non Cow metadata in snapshots has been is quite usable its simple to forget working cooperatively with the device management XFS has had the block layer providing COW so having functionality of a full copy on write file system without being a copy on write file system. There have been issues with ENOSPC what has made using block level COW not recommend again the new tricks for XFS is addressing this issue.

The issue here is XFS treats itself as only part of the solution and attempt to avoid implementing everything itself where it can.

Its going to be interesting to see how the XFS changes play out remember the changes will not just be limited to XFS we should expect to see changes in the device mapper as well. Yes XFS developers work on issues in the device mapper in the Linux kernel.