Announcement

Collapse
No announcement yet.

Linux 5.4 EXT4 / XFS / Btrfs RAID Performance On Four HDDs

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #11
    Originally posted by kgardas View Post
    with md underneah, btrfs is not able to perform seal-healing in case of bad blocks. Which from my point of view is the killer feature of both ZFS and btrfs. So yes, in this case using btrfs is useless.
    Not quite correct - BTRFS *with* DUP profile for data and metadata can correct corrupt data even if layered on top of MD.
    (you can also mix and match more exotic setups e.g BTRFS RAID1 or even RAID10 on top of multiple MDRAID5 groups).
    And even if you stay with the default configuration BTRFS still does actually check the data it returns (due to checksums) and since it has (by default) a DUPE of the metadata then at least the filesystem structures should self-heal. This is more than what you get if you put ext4 or xfs on top of md. ext4 and/or xfs will just return data corrupted or not., but both ext4 and xfs (as far as I know) is more tolerant of metadata corruption than BTRFS is- well if both copies of the metadata is broken anyway...

    Originally posted by PuckPoltergeist View Post
    Maybe yes, maybe no. With a faulty device, btrfs will detect errors and will try to correct it with the dupe. But it has no knowledge about the RAID and the RAID-layout. It doesn't know how to find dupes on non-error devices. This will induce new error-scenarios. I don't know about btrfs and caches, but because of this, ZFS must not run hard- or software RAID. Correctness is only guaranteed when ZFS has full control about all caches.
    Indeed. Even if BTRFS try to locate copies (DUP profile) far apart it is no guarantee that the MD layer or any other layer (including a hdd/sdd) will actually physically locate the offsets far apart. Logical and physical addresses can be quite different.
    Regarding caches on BTRFS as well, most of the horror stories on the mailing list seems to be from more interesting setups with BTRFS on top of bcache/lvm or weird hardware that don't implement barriers correctly so it would seem that BTRFS is just as sensitive as ZFS if not more so.


    http://www.dirtcellar.net

    Comment


    • #12
      The point of btrfs and zfs built in raid is to be more efficient, flexible and safer. So yea, running on top of mdraid could be a bad choice unless you understand the complications.

      Comment


      • #13
        Originally posted by kgardas View Post

        with md underneah, btrfs is not able to perform seal-healing in case of bad blocks. Which from my point of view is the killer feature of both ZFS and btrfs. So yes, in this case using btrfs is useless.
        I'd also add that using md precludes you the ability of adding/removing storage of any size while online and the ability to change raid type on the fly.

        Comment


        • #14
          Originally posted by loganj View Post
          i cure myself of Seagate a few years ago. i had 2 hdds of 3 tb each bought at 1 year difference. and both crashed at 1 year difference. i was lucky that both still had warranty. and these drivers was mostly for movies or storage. i mostly do only read after both of them were full. i didn't even bother to delete what i have on it once it was full. its strange to have hdd failure from 90% of usage time for reading. and to make it clear none of them were use for 2-3 hours/day. probably not even 1 min/week sometimes
          The 3 TB gen of hard drives (from all manufacturers, but Seagate had it worst IIRC) was really bad, though I don't remember why. It was a bit over my head.

          Comment


          • #15
            I have worked extensively with BTRFS and tested it on both Ubuntu and openSUSE/SLES and the Ubuntu's implementation is way slower than SUSE's (no surprise as they are a BTRFS contributor and use it by default).
            Maybe Michael can give a try the same test on both Ubuntu & SLES/openSUSE (all defaults) ...

            Comment


            • #16
              Originally posted by waxhead View Post
              Not quite correct - BTRFS *with* DUP profile for data and metadata can correct corrupt data even if layered on top of MD.
              (you can also mix and match more exotic setups e.g BTRFS RAID1 or even RAID10 on top of multiple MDRAID5 groups).
              And even if you stay with the default configuration BTRFS still does actually check the data it returns (due to checksums) and since it has (by default) a DUPE of the metadata then at least the filesystem structures should self-heal. This is more than what you get if you put ext4 or xfs on top of md. ext4 and/or xfs will just return data corrupted or not., but both ext4 and xfs (as far as I know) is more tolerant of metadata corruption than BTRFS is- well if both copies of the metadata is broken anyway...
              Oh, I would never ever think about a possibility someone is able to put btrfs/zfs on md and use data/metadata duplication to preserve self healing of fs. OK, let's see what you get if you:

              (i) use btrfs/zfs with data/metadata duplication on md RAID1. So basically from installed space 2x X drive, you will get 1x X drive in RAID1 and yet, you will get 1/2 X drive for btrfs/zfs data duplication, so you are on 1/4 of drive capacity. What you get is self-healing of fs.

              (ii) use btrfs/zfs with data/metadata duplication on md RAID0. You get 2x X drive which is then 1x X drive in fs capacity. You falsely get fs self-healing capability since if any of two drive fail, you will loose half of the filesystem depending on how fs layes data on drives. You don't know for sure in this case.

              However, if you use pure btrfs/zfs RAID1 on 2 drives, you get 1x X drive capacity but you get self-healing for sure even if one drive fails. It's a property of the fs implementation here. No play on luck in case of MD RAID0 + btrfs/zfs dup failure here...

              So to me, still pure btrfs/zfs looks way much more capable using their own RAID than to be used on top of MD RAID.

              Comment


              • #17
                Originally posted by kgardas View Post

                Oh, I would never ever think about a possibility someone is able to put btrfs/zfs on md and use data/metadata duplication to preserve self healing of fs. OK, let's see what you get if you:

                (i) use btrfs/zfs with data/metadata duplication on md RAID1. So basically from installed space 2x X drive, you will get 1x X drive in RAID1 and yet, you will get 1/2 X drive for btrfs/zfs data duplication, so you are on 1/4 of drive capacity. What you get is self-healing of fs.

                (ii) use btrfs/zfs with data/metadata duplication on md RAID0. You get 2x X drive which is then 1x X drive in fs capacity. You falsely get fs self-healing capability since if any of two drive fail, you will loose half of the filesystem depending on how fs layes data on drives. You don't know for sure in this case.

                However, if you use pure btrfs/zfs RAID1 on 2 drives, you get 1x X drive capacity but you get self-healing for sure even if one drive fails. It's a property of the fs implementation here. No play on luck in case of MD RAID0 + btrfs/zfs dup failure here...

                So to me, still pure btrfs/zfs looks way much more capable using their own RAID than to be used on top of MD RAID.
                on MDRAID0 you will loose your entire filesystem just for the record, anyway - my point was that if Phoronix does a benchmark like this then at least include BTRFS in the same setup as the other filesystems e.g. on top of MD. You want to compare eggs to eggs right?. Now BTRFS is in the fortunate position that it has its own "RAID" like features, but that is the bacon and while it is very interesting to see how a pure BTRFS solution compares with other filesystems layered on top it is not always the best comparison.

                Using BTRFS' "RAID" capabillitis maximize disk usage, but there are plenty of reasons why you want BTRFS on top of MD. For example you can run BTRFS mirror on top of two large MD RAID0 arrays for performance and some redundancy. You can also run BTRFS "RAID"1 or "RAID"10 on top of larger MD RAID5 or RAID6 arrays which should be a pretty "fail-safe" solution at least in theory. Max storage capacity is not always what you need



                http://www.dirtcellar.net

                Comment


                • #18
                  Originally posted by waxhead View Post
                  my point was that if Phoronix does a benchmark like this then at least include BTRFS in the same setup as the other filesystems e.g. on top of MD. You want to compare eggs to eggs right?.
                  I don't agree. It's not how you normally use these new filesystems.

                  Your analogy is akin to comparing ostrich with another bird and judging their ability to run, ignoring that one of them can fly.

                  Comment


                  • #19
                    I think a RAID 5 set of tests would have been helpful here. It appears that btrfs did better when striping was in use (RAID 0) than when mirroring was was used (RAID 1). A hybrid of the two (RAID 10) ended up in the middle. Having RAID 5 would give a another data point to see if btrfs was inherently better with striped data.

                    Comment


                    • #20
                      Originally posted by Spam View Post

                      I don't agree. It's not how you normally use these new filesystems.

                      Your analogy is akin to comparing ostrich with another bird and judging their ability to run, ignoring that one of them can fly.
                      Fair enough , but even without self healing BTRFS have a few other tricks up it's sleeve such as snapshots, compression and the fact that you would get a error message if your file has silently corrupted is still something that (as far as I know) neither xfs or ext4 can offer in such a setup. Not having self-healing "enabled" may not be such a problem for something that actually not happen that often, and if you have good tested backups! So yeah , I understand what you mean by people normally not using the filesystem like that , but I still think it would be a good idea to test btrfs the same way as the others - but hey, that's me!

                      http://www.dirtcellar.net

                      Comment

                      Working...
                      X