Btrfs RAID HDD Testing On Ubuntu Linux 14.10

[phoronix]

Phoronix: Btrfs RAID HDD Testing On Ubuntu Linux 14.10

With the Btrfs file-system continuing to stabilize while still adding more functionality and is generating continued interest from more Linux distributions and other open-source projects, I've found it time to run some fresh Btrfs RAID benchmarks to see how the next-generation Linux file-system is performing with its built-in RAID handling.

http://www.phoronix.com/vr.php?view=21049

[oleid]

Any idea, why writing to RAID 1 is faster than writing to a single disk?

[nanonyme]

Any idea, why writing to RAID 1 is faster than writing to a single disk?

Just a guess but a smart file system could be balancing writes between the two disks and syncing the two disk delayed taking advantage of parallel IO

[FishB8]

I don't think these results show much that isn't expected.

What would be more interesting is a comparison of how BTRFS raid stacks up to other raid solutions. So something like:

Btrfs Raid 0
vs.
Btrfs Raid 0 + SSD bcache (or dm-cache)
vs.
Kernel Soft (mdadm) Raid 0
vs.
Mid-level LSI hardware Raid 0

[Isedonde]

Just a guess but a smart file system could be balancing writes between the two disks and syncing the two disk delayed taking advantage of parallel IO

I'm not really experienced with RAID.

I was hoping that a write request wouldn't finish before the data is stored on both disks to ensure redundancy at all times. I'd expect RAID1 to be a little slower than single disk mode in that case. So maybe this is not the case with Btrfs and syncing to the other disk only happens after the successful write to one disk? This sounds a little dangerous, but maybe it's a nice trade-off for better speed.

Also, why is random read faster using RAID0 compared to RAID1? AFAICT, when using RAID1, it should be possible to read 50% of requests from disk 1, and 50% of requests from disk 2 (not taking seek time etc. into account) because both disks store exactly the same data. So it should be about twice as fast as single-disk mode. Using RAID0 should be a little slower, because the file system can't choose which disk to read from. It should still be faster than single-disk mode though, because some requests can probably be distributed between disks and therefore be run in parallel.

Are the Btrfs RAID1 algorithms adding a lot of overhead so RAID0 is faster than RAID1 when reading?

[jrch2k8]

I'm not really experienced with RAID.

I was hoping that a write request wouldn't finish before the data is stored on both disks to ensure redundancy at all times. I'd expect RAID1 to be a little slower than single disk mode in that case. So maybe this is not the case with Btrfs and syncing to the other disk only happens after the successful write to one disk? This sounds a little dangerous, but maybe it's a nice trade-off for better speed.

Also, why is random read faster using RAID0 compared to RAID1? AFAICT, when using RAID1, it should be possible to read 50% of requests from disk 1, and 50% of requests from disk 2 (not taking seek time etc. into account) because both disks store exactly the same data. So it should be about twice as fast as single-disk mode. Using RAID0 should be a little slower, because the file system can't choose which disk to read from. It should still be faster than single-disk mode though, because some requests can probably be distributed between disks and therefore be run in parallel.

Are the Btrfs RAID1 algorithms adding a lot of overhead so RAID0 is faster than RAID1 when reading?

most filesystems and RAID setups don't need to carry much metadata(XFS, Ext4, NTFS, etc.), so basically you gain almost linear performance on RAID 1,10,01 etc. but ZFS and BTRFS have to handle data and metadata, checksum, sub-volumes pointers, etc and the actual data, the mindfuck issue here is BTRFS and ZFS also scale almost linearly in performance but benchmarks only measure data commits/reads but not the metadata's, so in raw throughput it looks slower but in reality is writing/reading more data and syncing the metadata in both disk(you can select in BTRFS to raid1,10,etc. only the data and not the metadata(<--dangerous) and get basically the expected benchmark result).

In the case of Raid 0 both disks have independent metadata and data and there is no need for synchronisation or both ways checksums, etc. so in benchmarks look really fast (<-- RAID 0 is very insecure and is not meant for storage of sensitive data), obviously the same applies for ZFS

[phred14]

most filesystems and RAID setups don't need to carry much metadata(XFS, Ext4, NTFS, etc.), so basically you gain almost linear performance on RAID 1,10,01 etc. but ZFS and BTRFS have to handle data and metadata, checksum, sub-volumes pointers, etc and the actual data, the mindfuck issue here is BTRFS and ZFS also scale almost linearly in performance but benchmarks only measure data commits/reads but not the metadata's, so in raw throughput it looks slower but in reality is writing/reading more data and syncing the metadata in both disk(you can select in BTRFS to raid1,10,etc. only the data and not the metadata(<--dangerous) and get basically the expected benchmark result).

In the case of Raid 0 both disks have independent metadata and data and there is no need for synchronisation or both ways checksums, etc. so in benchmarks look really fast (<-- RAID 0 is very insecure and is not meant for storage of sensitive data), obviously the same applies for ZFS

Some time back I saw the suggestion that if you're using SSD to enhance performance of existing drives (as opposed to just running from the SSD) you're better off using it as an Ext4 external journal than as a cache. With mdadm and RAID1 read performance was great - the dual-spindle advantages of RAID0 plus on multi-threaded workloads one drive could seek to one thread and the other drive to the other. In other words, RAID1 could outperform RAID0 for multithreaded reads. However RAID1 still needs to write to both drives, which is why I assumed it might not have looked as good as I would have hoped.

Can btrfs have external metadata, and how much does that help its performance? (This assumes that metadata operations are significant, as you suggest.)

[jrch2k8]

Some time back I saw the suggestion that if you're using SSD to enhance performance of existing drives (as opposed to just running from the SSD) you're better off using it as an Ext4 external journal than as a cache. With mdadm and RAID1 read performance was great - the dual-spindle advantages of RAID0 plus on multi-threaded workloads one drive could seek to one thread and the other drive to the other. In other words, RAID1 could outperform RAID0 for multithreaded reads. However RAID1 still needs to write to both drives, which is why I assumed it might not have looked as good as I would have hoped.

Can btrfs have external metadata, and how much does that help its performance? (This assumes that metadata operations are significant, as you suggest.)

i don't think a btrfs RAID can handle non btrfs metadata(ext4 and familily simply don't have enough metadata to be compatible) and mdadm is not compatible with btrfs raids(or ZFS) as far as i know.

The issue with BTRFS and ZFS is they are not designed for speed but for trust,flexibility and advanced storage features, so what i do is use the actual tool for the job, it means if i need speed i have a volume in XFS and when i need to store it and be damn sure the data is always in a correct state i have/move it in/to BTRFS/ZFS.

my point Ext4/XFS will always be faster than BTRFS/ZFS by design and BTRFS/ZFS will always be the feature full trusty place to put your data in by design even if it means lose some raw throughput

[schmidtbag]

I'm surprised RAID1 read performance was so low. I'm also surprised how often RAID 0 writing was so low, but with BTRFS I can't say I'm too surprised.

[phred14]

i don't think a btrfs RAID can handle non btrfs metadata(ext4 and familily simply don't have enough metadata to be compatible) and mdadm is not compatible with btrfs raids(or ZFS) as far as i know.

I was suggesting having btrfs put it's metadata on a separate partition on a separate spindle. Not to say that a different partition on that spindle isn't used for something else, but hopefully it will still be better than having the metadata on the same spindle - if it can be done.