Facebook To Trial Btrfs Deployments

[phoronix]

Phoronix: Facebook To Trial Btrfs Deployments

While it should come as no surprise given that Facebook recently hired multiple Btrfs file-system developers, but they will be trying out the Btrfs file-system within real-world deployments at the social networking company...

http://www.phoronix.com/vr.php?view=MTY0NDk

[xeekei]

Even if everything goes smoothly and deployments snowball from here, it sure took its sweet time. I hope Btrfs will be a worthy contender to ZFS.

[deploylinux]

All great, but where are the specifics?

I check btrfs out every six months or so and have yet to be convinced it is stable, and there are constant updates to the latest kernels so it is obviously very much still a work in progress....so I'm curious which kernel version and userspace environment facebook feels is stable enough for production deployment?

[liam]

I check btrfs out every six months or so and have yet to be convinced it is stable, and there are constant updates to the latest kernels so it is obviously very much still a work in progress....so I'm curious which kernel version and userspace environment facebook feels is stable enough for production deployment?

According to the lwn article, facebook carries a few different kernels, with the latest being 3.10 (with 60 patches, iirc). I'd imagine that's what they'd use, if not something even more recent. Of more concern, though, is the problem with cfq and stable pages (that is, they're performance hit). There's some interesting discussion in the pgsql article as well regarding general io buffering issues.

[chinoto]

[...]there are constant updates to the latest kernels so it is obviously very much still a work in progress....[...]

Now I'm not saying you're necessarily wrong, but it could be the most stable thing in the world and they're just working on making it faster. Compared to ext4, it is rather slow (hardly noticeable on an HDD though).

[jwilliams]

Now I'm not saying you're necessarily wrong, but it could be the most stable thing in the world...

Besides being not necessarily wrong, he is also absolutely not wrong.

Even still, there are serious issues with btrfs that make it unsuitable as a drop-in replacement for ext4. Just this week there was a discussion about how btrfs can get into a state where there is plenty of free space but chunks are not automatically reclaimed after files are deleted, which can result in out of space on either data or metadata, even though there is plenty of space overall. There was another issue on the email list discussed recently (I just skimmed it) that may or may not be related, but something with balance was not working the way some users expected, and the solution required some manual tweaking. And I still see posts every few weeks from someone whose btrfs filesystem has become corrupted and cannot be mounted, for whatever reason (this sort of thing is not supposed to happen to a production filesystem, especially a COW one)

The project management for btrfs continues to be poor. I think if there were a decent project manager, the remaining issues that are keeping btrfs from being production ready would be rapidly prioritized and fixed. But the project seems to wander aimlessly, never reaching production quality.

[kernelOfTruth]

Besides being not necessarily wrong, he is also absolutely not wrong.

Even still, there are serious issues with btrfs that make it unsuitable as a drop-in replacement for ext4. Just this week there was a discussion about how btrfs can get into a state where there is plenty of free space but chunks are not automatically reclaimed after files are deleted, which can result in out of space on either data or metadata, even though there is plenty of space overall. There was another issue on the email list discussed recently (I just skimmed it) that may or may not be related, but something with balance was not working the way some users expected, and the solution required some manual tweaking. And I still see posts every few weeks from someone whose btrfs filesystem has become corrupted and cannot be mounted, for whatever reason (this sort of thing is not supposed to happen to a production filesystem, especially a COW one)

The project management for btrfs continues to be poor. I think if there were a decent project manager, the remaining issues that are keeping btrfs from being production ready would be rapidly prioritized and fixed. But the project seems to wander aimlessly, never reaching production quality.

nice summary !

even though I'm using it on the system partition - it out of a sudden yesterday showed a file with checksum missing (probably hardware: hdd (new - is that even possible ? I checked it with badblocks and smartctl, etc. etc.) , processor or RAM issue; could be a btrfs-issue though). Luckily it wasn't any important file

[ 5397.363295] BTRFS info (device dm-3): no csum found for inode 87317 start 112476160
[ 5397.363303] BTRFS info (device dm-3): no csum found for inode 87317 start 112480256
[ 5397.363306] BTRFS info (device dm-3): no csum found for inode 87317 start 112484352
[ 5397.363310] BTRFS info (device dm-3): no csum found for inode 87317 start 112488448
[ 5397.363313] BTRFS info (device dm-3): no csum found for inode 87317 start 112492544
[ 5397.363317] BTRFS info (device dm-3): no csum found for inode 87317 start 112496640
[ 5397.363320] BTRFS info (device dm-3): no csum found for inode 87317 start 112500736
[ 5397.363324] BTRFS info (device dm-3): no csum found for inode 87317 start 112504832
[ 5397.363327] BTRFS info (device dm-3): no csum found for inode 87317 start 112508928
[ 5397.363330] BTRFS info (device dm-3): no csum found for inode 87317 start 112513024
[ 5397.363334] BTRFS info (device dm-3): no csum found for inode 87317 start 112517120
[ 5397.363337] BTRFS info (device dm-3): no csum found for inode 87317 start 112521216
[ 5397.363341] BTRFS info (device dm-3): no csum found for inode 87317 start 112525312
[ 5397.363344] BTRFS info (device dm-3): no csum found for inode 87317 start 112529408
[ 5397.363348] BTRFS info (device dm-3): no csum found for inode 87317 start 112533504
[ 5397.363351] BTRFS info (device dm-3): no csum found for inode 87317 start 112537600
[ 5397.461709] BTRFS info (device dm-3): csum failed ino 87317 off 112476160 csum 488911438 expected csum 0
[ 5397.461716] BTRFS info (device dm-3): csum failed ino 87317 off 112480256 csum 1298834631 expected csum 0
[ 5397.461720] BTRFS info (device dm-3): csum failed ino 87317 off 112484352 csum 2766330694 expected csum 0
[ 5397.461723] BTRFS info (device dm-3): csum failed ino 87317 off 112488448 csum 1279344126 expected csum 0
[ 5397.461727] BTRFS info (device dm-3): csum failed ino 87317 off 112492544 csum 1791079649 expected csum 0
[ 5397.461731] BTRFS info (device dm-3): csum failed ino 87317 off 112496640 csum 1582102096 expected csum 0
[ 5397.461735] BTRFS info (device dm-3): csum failed ino 87317 off 112500736 csum 2464632511 expected csum 0
[ 5397.461738] BTRFS info (device dm-3): csum failed ino 87317 off 112504832 csum 2137084469 expected csum 0
[ 5397.461742] BTRFS info (device dm-3): csum failed ino 87317 off 112508928 csum 2535423305 expected csum 0
[ 5397.461746] BTRFS info (device dm-3): csum failed ino 87317 off 112513024 csum 1594280384 expected csum 0

find / -mount -inum 87317
/var/lib/clamav/main.cld

afaik got this also happening in the past - so this leaves this impression of btrfs randomly corrupting files on its own for me (of course could be wrong since it also detected a harddrive/partition that is on the verge of becoming broken - showing some more badblocks, internal errors); mentioned high memory usage (a memory leak ? on the mailing list), RAID handling when removing and re-adding a harddrive

and of course this issue with ENOSPEC/free space fragmentation and/or non-reclaim of free space

this might be a side-effect of Btrfs' COW nature but at this late (?) point of development it slowly should be tackled and must not need an manual (especially advanced user) intervention to continue to be usable

if development continues like that it will again take years until the round edges are smoothened out

it has come a long way and improved a lot but there's still the feeling that fundamental mechanisms and parts are missing underneath

[zxy_thf]

Even if everything goes smoothly and deployments snowball from here, it sure took its sweet time. I hope Btrfs will be a worthy contender to ZFS.

I'm afraid it'll take years before btrfs catches up ZFS's features

[benmoran]

I'm afraid it'll take years before btrfs catches up ZFS's features

It already has many features (such as volume management) that ZFS doesn't have.
What features in particular are you talking about? Personally I'm wating for proper RAID5/6 code, but other
than that I can't think of anything lacking. (Besides maybe support for raw block device export, which I don't
think is even planned).

[kernelOfTruth]

It already has many features (such as volume management) that ZFS doesn't have.
What features in particular are you talking about? Personally I'm wating for proper RAID5/6 code, but other
than that I can't think of anything lacking. (Besides maybe support for raw block device export, which I don't
think is even planned).

local copies ?

copies=3

lz4 compression ?

more advanced checksums than crc32c (e.g. sha256) ?

easy to use / user-friendly expert tools ?

trivial errors or "features" that are not showing during routine checks ? (e.g. the checksum missing for space_cache)

an inode_cache that doesn't take minutes to regenerate after having shut down the "links" browser and in the process slowing down the system to a crawl ?

actual NO slowing down during heavy i/o writes of the *entire* system compared to ext4 ? (on dm-crypt)

stability ? (already got corrupted files on /home; after a few crashes (hardlocks) the /home directory got that much messed up that the kernel would panic or couldn't access /home anymore)

these are all working features that are currently available on ZFSonLinux

this is not to badmouth Btrfs - rather a rant since I'm personally interested in a 2nd drop-in filesystem that could also be used on /home instead of ZFS (which doesn't support realtime kernels yet and suspend-to-disk, probably suspend-to-ram neither) - especially on the laptop to fully use energy-saving and time-saving features while maintaining maximum data integrity & -safety