Announcement

Collapse
No announcement yet.

Facebook To Trial Btrfs Deployments

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Facebook To Trial Btrfs Deployments

    Phoronix: Facebook To Trial Btrfs Deployments

    While it should come as no surprise given that Facebook recently hired multiple Btrfs file-system developers, but they will be trying out the Btrfs file-system within real-world deployments at the social networking company...

    http://www.phoronix.com/vr.php?view=MTY0NDk

  • #2
    Even if everything goes smoothly and deployments snowball from here, it sure took its sweet time. I hope Btrfs will be a worthy contender to ZFS.

    Comment


    • #3
      All great, but where are the specifics?

      I check btrfs out every six months or so and have yet to be convinced it is stable, and there are constant updates to the latest kernels so it is obviously very much still a work in progress....so I'm curious which kernel version and userspace environment facebook feels is stable enough for production deployment?

      Comment


      • #4
        Originally posted by deploylinux View Post
        I check btrfs out every six months or so and have yet to be convinced it is stable, and there are constant updates to the latest kernels so it is obviously very much still a work in progress....so I'm curious which kernel version and userspace environment facebook feels is stable enough for production deployment?
        According to the lwn article, facebook carries a few different kernels, with the latest being 3.10 (with 60 patches, iirc). I'd imagine that's what they'd use, if not something even more recent. Of more concern, though, is the problem with cfq and stable pages (that is, they're performance hit). There's some interesting discussion in the pgsql article as well regarding general io buffering issues.

        Comment


        • #5
          Originally posted by deploylinux View Post
          [...]there are constant updates to the latest kernels so it is obviously very much still a work in progress....[...]
          Now I'm not saying you're necessarily wrong, but it could be the most stable thing in the world and they're just working on making it faster. Compared to ext4, it is rather slow (hardly noticeable on an HDD though).

          Comment


          • #6
            Originally posted by chinoto View Post
            Now I'm not saying you're necessarily wrong, but it could be the most stable thing in the world...
            Besides being not necessarily wrong, he is also absolutely not wrong.

            Even still, there are serious issues with btrfs that make it unsuitable as a drop-in replacement for ext4. Just this week there was a discussion about how btrfs can get into a state where there is plenty of free space but chunks are not automatically reclaimed after files are deleted, which can result in out of space on either data or metadata, even though there is plenty of space overall. There was another issue on the email list discussed recently (I just skimmed it) that may or may not be related, but something with balance was not working the way some users expected, and the solution required some manual tweaking. And I still see posts every few weeks from someone whose btrfs filesystem has become corrupted and cannot be mounted, for whatever reason (this sort of thing is not supposed to happen to a production filesystem, especially a COW one)

            The project management for btrfs continues to be poor. I think if there were a decent project manager, the remaining issues that are keeping btrfs from being production ready would be rapidly prioritized and fixed. But the project seems to wander aimlessly, never reaching production quality.

            Comment


            • #7
              Originally posted by jwilliams View Post
              Besides being not necessarily wrong, he is also absolutely not wrong.

              Even still, there are serious issues with btrfs that make it unsuitable as a drop-in replacement for ext4. Just this week there was a discussion about how btrfs can get into a state where there is plenty of free space but chunks are not automatically reclaimed after files are deleted, which can result in out of space on either data or metadata, even though there is plenty of space overall. There was another issue on the email list discussed recently (I just skimmed it) that may or may not be related, but something with balance was not working the way some users expected, and the solution required some manual tweaking. And I still see posts every few weeks from someone whose btrfs filesystem has become corrupted and cannot be mounted, for whatever reason (this sort of thing is not supposed to happen to a production filesystem, especially a COW one)

              The project management for btrfs continues to be poor. I think if there were a decent project manager, the remaining issues that are keeping btrfs from being production ready would be rapidly prioritized and fixed. But the project seems to wander aimlessly, never reaching production quality.
              nice summary !

              even though I'm using it on the system partition - it out of a sudden yesterday showed a file with checksum missing (probably hardware: hdd (new - is that even possible ? I checked it with badblocks and smartctl, etc. etc.) , processor or RAM issue; could be a btrfs-issue though). Luckily it wasn't any important file

              [ 5397.363295] BTRFS info (device dm-3): no csum found for inode 87317 start 112476160
              [ 5397.363303] BTRFS info (device dm-3): no csum found for inode 87317 start 112480256
              [ 5397.363306] BTRFS info (device dm-3): no csum found for inode 87317 start 112484352
              [ 5397.363310] BTRFS info (device dm-3): no csum found for inode 87317 start 112488448
              [ 5397.363313] BTRFS info (device dm-3): no csum found for inode 87317 start 112492544
              [ 5397.363317] BTRFS info (device dm-3): no csum found for inode 87317 start 112496640
              [ 5397.363320] BTRFS info (device dm-3): no csum found for inode 87317 start 112500736
              [ 5397.363324] BTRFS info (device dm-3): no csum found for inode 87317 start 112504832
              [ 5397.363327] BTRFS info (device dm-3): no csum found for inode 87317 start 112508928
              [ 5397.363330] BTRFS info (device dm-3): no csum found for inode 87317 start 112513024
              [ 5397.363334] BTRFS info (device dm-3): no csum found for inode 87317 start 112517120
              [ 5397.363337] BTRFS info (device dm-3): no csum found for inode 87317 start 112521216
              [ 5397.363341] BTRFS info (device dm-3): no csum found for inode 87317 start 112525312
              [ 5397.363344] BTRFS info (device dm-3): no csum found for inode 87317 start 112529408
              [ 5397.363348] BTRFS info (device dm-3): no csum found for inode 87317 start 112533504
              [ 5397.363351] BTRFS info (device dm-3): no csum found for inode 87317 start 112537600
              [ 5397.461709] BTRFS info (device dm-3): csum failed ino 87317 off 112476160 csum 488911438 expected csum 0
              [ 5397.461716] BTRFS info (device dm-3): csum failed ino 87317 off 112480256 csum 1298834631 expected csum 0
              [ 5397.461720] BTRFS info (device dm-3): csum failed ino 87317 off 112484352 csum 2766330694 expected csum 0
              [ 5397.461723] BTRFS info (device dm-3): csum failed ino 87317 off 112488448 csum 1279344126 expected csum 0
              [ 5397.461727] BTRFS info (device dm-3): csum failed ino 87317 off 112492544 csum 1791079649 expected csum 0
              [ 5397.461731] BTRFS info (device dm-3): csum failed ino 87317 off 112496640 csum 1582102096 expected csum 0
              [ 5397.461735] BTRFS info (device dm-3): csum failed ino 87317 off 112500736 csum 2464632511 expected csum 0
              [ 5397.461738] BTRFS info (device dm-3): csum failed ino 87317 off 112504832 csum 2137084469 expected csum 0
              [ 5397.461742] BTRFS info (device dm-3): csum failed ino 87317 off 112508928 csum 2535423305 expected csum 0
              [ 5397.461746] BTRFS info (device dm-3): csum failed ino 87317 off 112513024 csum 1594280384 expected csum 0

              find / -mount -inum 87317
              /var/lib/clamav/main.cld
              afaik got this also happening in the past - so this leaves this impression of btrfs randomly corrupting files on its own for me (of course could be wrong since it also detected a harddrive/partition that is on the verge of becoming broken - showing some more badblocks, internal errors); mentioned high memory usage (a memory leak ? on the mailing list), RAID handling when removing and re-adding a harddrive

              and of course this issue with ENOSPEC/free space fragmentation and/or non-reclaim of free space

              this might be a side-effect of Btrfs' COW nature but at this late (?) point of development it slowly should be tackled and must not need an manual (especially advanced user) intervention to continue to be usable

              if development continues like that it will again take years until the round edges are smoothened out

              it has come a long way and improved a lot but there's still the feeling that fundamental mechanisms and parts are missing underneath

              Comment


              • #8
                Originally posted by xeekei View Post
                Even if everything goes smoothly and deployments snowball from here, it sure took its sweet time. I hope Btrfs will be a worthy contender to ZFS.
                I'm afraid it'll take years before btrfs catches up ZFS's features

                Comment


                • #9
                  Originally posted by zxy_thf View Post
                  I'm afraid it'll take years before btrfs catches up ZFS's features
                  It already has many features (such as volume management) that ZFS doesn't have.
                  What features in particular are you talking about? Personally I'm wating for proper RAID5/6 code, but other
                  than that I can't think of anything lacking. (Besides maybe support for raw block device export, which I don't
                  think is even planned).

                  Comment


                  • #10
                    Originally posted by benmoran View Post
                    It already has many features (such as volume management) that ZFS doesn't have.
                    What features in particular are you talking about? Personally I'm wating for proper RAID5/6 code, but other
                    than that I can't think of anything lacking. (Besides maybe support for raw block device export, which I don't
                    think is even planned).
                    local copies ?

                    copies=3

                    lz4 compression ?

                    more advanced checksums than crc32c (e.g. sha256) ?

                    easy to use / user-friendly expert tools ?

                    trivial errors or "features" that are not showing during routine checks ? (e.g. the checksum missing for space_cache)

                    an inode_cache that doesn't take minutes to regenerate after having shut down the "links" browser and in the process slowing down the system to a crawl ?

                    actual NO slowing down during heavy i/o writes of the *entire* system compared to ext4 ? (on dm-crypt)

                    stability ? (already got corrupted files on /home; after a few crashes (hardlocks) the /home directory got that much messed up that the kernel would panic or couldn't access /home anymore)

                    these are all working features that are currently available on ZFSonLinux

                    this is not to badmouth Btrfs - rather a rant since I'm personally interested in a 2nd drop-in filesystem that could also be used on /home instead of ZFS (which doesn't support realtime kernels yet and suspend-to-disk, probably suspend-to-ram neither) - especially on the laptop to fully use energy-saving and time-saving features while maintaining maximum data integrity & -safety

                    Comment


                    • #11
                      Originally posted by benmoran View Post
                      It already has many features (such as volume management) that ZFS doesn't have.
                      What features in particular are you talking about? Personally I'm wating for proper RAID5/6 code, but other
                      than that I can't think of anything lacking. (Besides maybe support for raw block device export, which I don't
                      think is even planned).
                      What do you mean by 'volume management'? ZFS has datasets which are similar to 'volumes' but far, far more flexible, as well as ZVols, which are exactly as they sound like: volumes.

                      Comment


                      • #12
                        I haven't had any major problems with Btrfs despite running it on several production systems over the last few years... The only thing I'm still waiting to happen is bcache support, which should be landing in the kernel that is about to be released.

                        Comment


                        • #13
                          Originally posted by kernelOfTruth View Post
                          afaik got this also happening in the past - so this leaves this impression of btrfs randomly corrupting files on its own for me (of course could be wrong since it also detected a harddrive/partition that is on the verge of becoming broken - showing some more badblocks, internal errors); mentioned high memory usage (a memory leak ? on the mailing list), RAID handling when removing and re-adding a harddrive
                          I also have clamav on two machines with btrfs and this has not hapenned yet. It only OOPS-ed during boot on a 32bit UP machine (probably a race).

                          Originally posted by kernelOfTruth View Post
                          and of course this issue with ENOSPEC/free space fragmentation and/or non-reclaim of free space
                          Had that too. If you retry again, the copy might work. That is indeed odd and weird.

                          Originally posted by kernelOfTruth View Post
                          this might be a side-effect of Btrfs' COW nature but at this late (?) point of development it slowly should be tackled and must not need an manual (especially advanced user) intervention to continue to be usable

                          if development continues like that it will again take years until the round edges are smoothened out

                          it has come a long way and improved a lot but there's still the feeling that fundamental mechanisms and parts are missing underneath
                          Altough it's nice to read about the technical details underlying these problems, it's indeed not going to cut it for production use.

                          Luckily I also migrated all of my machines to lvm, so moving back to ext4 or tux3 should not be much of a problem.

                          Comment


                          • #14
                            richACLs?

                            I am using btrfs on my laptop. My Samba-Server is using reiserfs on a raid disk.

                            The only thing I miss at btrfs is richacls-support, but i think that feature will never come to btrfs.

                            Comment


                            • #15
                              Originally posted by kernelOfTruth View Post
                              local copies ?

                              copies=3

                              lz4 compression ?

                              more advanced checksums than crc32c (e.g. sha256) ?

                              easy to use / user-friendly expert tools ?

                              trivial errors or "features" that are not showing during routine checks ? (e.g. the checksum missing for space_cache)
                              an inode_cache that doesn't take minutes to regenerate after having shut down the "links" browser and in the process slowing down the system to a crawl ?

                              actual NO slowing down during heavy i/o writes of the *entire* system compared to ext4 ? (on dm-crypt)

                              stability ? (already got corrupted files on /home; after a few crashes (hardlocks) the /home directory got that much messed up that the kernel would panic or couldn't access /home anymore)

                              these are all working features that are currently available on ZFSonLinux

                              this is not to badmouth Btrfs - rather a rant since I'm personally interested in a 2nd drop-in filesystem that could also be used on /home instead of ZFS (which doesn't support realtime kernels yet and suspend-to-disk, probably suspend-to-ram neither) - especially on the laptop to fully use energy-saving and time-saving features while maintaining maximum data integrity & -safety
                              Some valid points, and some opinion in there.

                              If I remember correctly, the "copies=3" (or more) work is something that should show up after raid 5/6 scrubbing support is ready.

                              The lz4 compression seems to be really wanted by some, but I get the impression that not many people are interested enough to work on it. I don't use this kind of file system for space savings personally, so I am one of those that is happy with lzo.

                              I can't comment on the dm-crypt stuff, since I don't use it. Valid complaint though, as it seems like built-in encryption is not being considered.

                              Af far as stability and corruption, I've only had issues once with the known balance bug on a recent kernel. I've personally used btrfs on many machines at home and in the lab, some of which are hard reset many many times. I've not had any corruption in a dozen file systems. I know people have issues, but with recent code I consider it quite stable.

                              Some valid points, but I don't think it's fair to say it will take years to catch up to ZFS' features - because some are not planned to be included at all.

                              Comment

                              Working...
                              X