Announcement

Collapse
No announcement yet.

Bcachefs File-System Is Working On Going Upstream In The Linux Kernel

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Bcachefs File-System Is Working On Going Upstream In The Linux Kernel

    Phoronix: Bcachefs File-System Is Working On Going Upstream In The Linux Kernel

    Kent Overstreet who has been spending the past few years working on the Bcachefs file-system born out of the BCache block cache technology is now starting work on upstreaming the code to the mainline kernel...

    http://www.phoronix.com/scan.php?pag...Upstream-Start

  • jacob
    replied
    Originally posted by DrYak View Post
    My impression, by reading what he publishes here and there is that his gut feeling is that he has tackled a lot of hardcore problems as part of his formely "tiered caching" / nowadays "copy management groups", and hope to be able to leverage all the work from BCache and BCacheFS to implement cheaply snapshots on top (just as BTRFS leverages their specific flavour of B-trees to manage the COW and Snapshotting).

    My gut feeling is that there could still be unforeseen hidden gnomes lurking somewhere in there, ready to jump and bite...
    {...}
    And maybe, if kent discovers that he won't be able to bring snapshotting to the current BcacheFS, he'll develops a BCacheFS2 with what's need for a working snapshotting.

    (My gut feeling is that this exactly what is going to happen)
    Kent certainly knows the BCache layer better than I do so I wish him that it works out the way he hopes. The point is though that BCache has never been designed for this kind of functionality, and he admits that he hasn't really started working on that part yet. I'm not predicting that it's going to fail, just saying that I'm curious to see it because BTRFS aside, we saw in the past a number of projects with wild claims and an implementation that was "almost there" and this is precisely the moment where things always went pear shaped. And yes, it would be great if in this case it actually works, and by basing is design on BCache he's at least trying an approach that the others didn't use.

    Leave a comment:


  • DrYak
    replied
    Originally posted by jwilliams View Post
    Where are you getting that encryption is finished?
    The status page at https://www.patreon.com/bcachefs says, "Encryption: Not yet started"
    But on the other hand, the webpage mentions it's just merged in.
    (I have the impression that there is no official centralized reliable status page, but you have to gather together several tools.)
    (Whatever, we count on him to make a good FS, not to be brilliant at PR/communication/webmastering)

    Originally posted by jacob View Post
    I kind of share this feeling. Someone pointed out in another thread that the thing about btrfs is that they tackled all the hardest problems first. An it's true that it had snapshots and replication basically from day 1 - maybe variously buggy, but it was there. Whereas with Bcachefs, the hardest work remains to be done and he as good as says so himself. {...} The real "fun" comes once you go beyond that, especially with COW exposed to users (e.g. snapshots). I won't be surprised if once he aims to get there, he discovers that it's suddenly a lot harder to remain robust and keep good performance{...} because put simply, this is where Frankenstein lives.
    My impression, by reading what he publishes here and there is that his gut feeling is that he has tackled a lot of hardcore problems as part of his formely "tiered caching" / nowadays "copy management groups", and hope to be able to leverage all the work from BCache and BCacheFS to implement cheaply snapshots on top (just as BTRFS leverages their specific flavour of B-trees to manage the COW and Snapshotting).

    My gut feeling is that there could still be unforeseen hidden gnomes lurking somewhere in there, ready to jump and bite...

    On the other hand, I kind of see the point of people who where pushing Kent to upstream his file system :
    From the mailing list posts, it seems that some people though : "Well, it's already a functional file system at this stage and could already be useful for some", so let's release the current beast as a "in between gens" FS.
    (The caching offered by the storage group management is useful as a form of hierarchical storage management)

    And maybe, if kent discovers that he won't be able to bring snapshotting to the current BcacheFS, he'll develops a BCacheFS2 with what's need for a working snapshotting.

    (My gut feeling is that this exactly what is going to happen)



    Leave a comment:


  • oiaohm
    replied
    Originally posted by pal666 View Post
    stratis is just (not written yet) easier interface to existing obsolete functionality
    So by you hardware raid controllers are obsolete. There are very big reasons to want a solution that works well with device mapper lead one is hardware acceleration of raid operations..

    Originally posted by pal666 View Post
    btrfs should in time work well behaved.
    it is impossible to work well with system device mapper system if your definition of well includes "not being crazy slow"
    You must be refering to software raid under device mapper because btrfs raid is no where close to the hardware raids under device mapper. Also the software raid in device mapper is not always slower than the btrfs raid.

    Originally posted by pal666 View Post
    "Data-only CoW limits the functionality that XFS can provide" quote from article
    Is that limitation harmful. XFS functionality is limit but this means XFS has limited paths to-do particular functionality. There is more than 1 way to do what ZFS and BTRFS do and the other paths have not been fully explored.

    Originally posted by pal666 View Post
    it can't. well, when you do that, you get zfs or btrfs, because only fs knows about files
    This was a line I was expecting. Ever heard of a object raid. So if the file system is telling the raid system under it what is objects then raid does not really need to know about files. Basically file system drivers need to care about file system meta data. Raid part only need to know how you wish the data stored object information can achieve this. Setting raid type by device is horrible. A proper object raid can have multi raid types inside the 1 partition.

    The reality gets nastier having an object raid controller would allow the raid control to checksum block of data without having to transfer all that data to the cpu.

    Originally posted by pal666 View Post
    the xfs path is "we have old design and have to make it work somehow"
    Yes its old design but is a well tested design.

    Originally posted by pal666 View Post
    it is not like xfs has any choice. its design predates idea of combining device management with fs. it is ridiculous we are seriously discussing filesystem which can't change its size
    In fact xfs can grow size without issues. Horrible process you can shrink it.
    http://xfs.org/index.php/Shrinking_Support
    There is a list of features that need to be implemented to shrink the big problem is the first item on list. In fact being able to subvolume will help xfs support shrinking.

    Last thing you want is sitting on a device mapper cow storage/lvm over allocation and getting a ENOSPC while attempt to shrink and this happens to be after you have already over written some data.

    Lot of file systems support shrinking but in cow storage or lvm over allocation or other form of over allocate they fail badly when the ENOSPC happens. Problem here is currently the ENOSPC message will turn up when the buffers are attempted to be flushed to storage. So this issue nukes btrfs, zfs.... every file system that currently support shrinking if they happen to be sitting on one of these back ends as you most likely get ENOSPC message after you have already overwritten data in the resize process and blocks of data have been thrown into the void. XFS quality control is what has prevented XFS from having shrinking.

    Please note when XFS was made it was made with XLV. So the idea of file system working device management first appears in the XFS/XLV combination. XLV also had object raid. It also explains why XFS does not have data checksum feature as that was a feature of the XLV as well. Turns out zfs and btrfs are not that feature great when put head to head with the old XFS/XLV combination. The current XFS is still a shadow of it past self and XFS cannot catch up with it past self until the layers its running on top of are fixed.

    Leave a comment:


  • pal666
    replied
    Originally posted by geearf View Post
    patreon does not have to be his only source of income. There are alternatives like liberapay, paypal, etc...
    He may have enough money left to have enough wiggle with a little extra coming from patrons.
    I read somewhere he may had a sponsor, but not sure anymore.
    he may have army of minions in his castle, but he doesn't. we have just one part-time dev with ridiculous claims not backed by anything but his patreon page

    Leave a comment:


  • pal666
    replied
    Originally posted by Vistaus View Post
    Depends. A full-time bus driver makes about $1800-$2400/€1800-€2100 at quite some transit companies (there are exceptions, like MTA in NYC where you earn a lot more). So 1754 is close to full-time in that job.
    well, you can't have more than one full-time job, so if he already has irl job, then second job is not full-time

    Leave a comment:


  • geearf
    replied
    patreon does not have to be his only source of income. There are alternatives like liberapay, paypal, etc...
    He may have enough money left to have enough wiggle with a little extra coming from patrons.
    I read somewhere he may had a sponsor, but not sure anymore.

    Leave a comment:


  • Vistaus
    replied
    Originally posted by pal666 View Post
    if he does have irl job, then 1754 surely can't be a full-time job, it can only be part-time
    Depends. A full-time bus driver makes about $1800-$2400/€1800-€2100 at quite some transit companies (there are exceptions, like MTA in NYC where you earn a lot more). So 1754 is close to full-time in that job. And as a freelancer, it can also be full-time depending on the job. But I agree that he probably has a higher earning job.
    Last edited by Vistaus; 05-11-2018, 11:24 AM.

    Leave a comment:


  • pal666
    replied
    Originally posted by oiaohm View Post
    Stratis is a stop gap from description "devicemapper subsystem, and the XFS filesystem".
    stratis is just (not written yet) easier interface to existing obsolete functionality
    Originally posted by oiaohm View Post
    XFS should in time work well behaved with the system device mapper system.
    btrfs should in time work well behaved.
    it is impossible to work well with system device mapper system if your definition of well includes "not being crazy slow"
    Originally posted by oiaohm View Post
    https://lwn.net/Articles/747633/
    This new tricks of XFS gets very interesting on the device management side.
    "Data-only CoW limits the functionality that XFS can provide" quote from article
    Originally posted by oiaohm View Post
    Can device mapper subsystem be modified to work with files instead of devices as well.
    it can't. well, when you do that, you get zfs or btrfs, because only fs knows about files
    Originally posted by oiaohm View Post
    The XFS path is can we make file system and device management work cooperatively with each other so we don't have multi copies of device management.
    the xfs path is "we have old design and have to make it work somehow"
    Originally posted by oiaohm View Post
    The issue here is XFS treats itself as only part of the solution and attempt to avoid implementing everything itself where it can.
    it is not like xfs has any choice. its design predates idea of combining device management with fs. it is ridiculous we are seriously discussing filesystem which can't change its size
    Last edited by pal666; 05-11-2018, 02:55 PM.

    Leave a comment:


  • jacob
    replied
    Originally posted by DrYak View Post
    Yes, he makes big promises that one day, once it's implemented, they will by fast and lightweight and nice.
    I kind of share this feeling. Someone pointed out in another thread that the thing about btrfs is that they tackled all the hardest problems first. An it's true that it had snapshots and replication basically from day 1 - maybe variously buggy, but it was there. Whereas with Bcachefs, the hardest work remains to be done and he as good as says so himself. Frankly, developing a filesystem that is functionally on par with ext4 or xfs is not *that* hard. The real "fun" comes once you go beyond that, especially with COW exposed to users (e.g. snapshots). I won't be surprised if once he aims to get there, he discovers that it's suddenly a lot harder to remain robust and keep good performance and that there is, in fact, much less merit in his bashing of btrfs than he thinks, because put simply, this is where Frankenstein lives. This is really not meant as a skepticism or critique against Overstreet: he's doing a great job and if he manages to pull it off, it would be great for the entire community. But the fact is that there were similar projects before that went 90% of the way, like Bcachefs today, only to discover that the remaining 10% were basically unfeasible with what they had built thus far. Tux3 is one example that comes to mind.

    Leave a comment:

Working...
X