Linux 5.4 EXT4 / XFS / Btrfs RAID Performance On Four HDDs

[gilboa]

With 4 drives it is possible to get RAID5, would be possible to test it is as well?

I second that.
At least in my experience, MDRAID5 is faster than MDRAID10.

Gilboa

[gilboa]

i cure myself of Seagate a few years ago. i had 2 hdds of 3 tb each bought at 1 year difference. and both crashed at 1 year difference. i was lucky that both still had warranty. and these drivers was mostly for movies or storage. i mostly do only read after both of them were full. i didn't even bother to delete what i have on it once it was full. its strange to have hdd failure from 90% of usage time for reading. and to make it clear none of them were use for 2-3 hours/day. probably not even 1 min/week sometimes

I've got well over 100 Seagate drives, mostly enterprise ones (2-8TB), some over 5 year old, and I rarely have more than 2-3 drive replacements a year.

BTW big chunk of them are used in a large 24x7 cluster running ovirt (Vmware ESX replacement).

You sample is far too small to make any meaningful conclusions.

Gilboa

[gadnet]

thanks for your insight waxhead !

[waxhead]

this is rather strange to see some reading test with 10x perf loss from raid 1 to raid 10.
I dont see how any reads could be 10x slower in raid10 as it seems to me that the worst case scenario a raid 10 is as slow as raid1 and in best case it is 2x faster no ?

Well if you have a stripe that is 64 bytes long , and you have to interleave this across 4x disks you get 16 bytes pr. disk.
Now if you are reading tons of files that are less than 16 bytes long for example only one disk needs to be read, but you still need to read back the entire stripe and therefore leaving the disks busy reading the stripe (Raid10) instead of being free for parallel reads (raid1) from other programs/threads.

Also since physical addresses do not necessarily match logical ones reading the strip can never be faster than the slowest disks (number of seeks or the longest seek). So if stripe data is scattered on one disks the other disks has to wait for it.

if this explains this benchmarks can probably be discussed until there are no free harddisks space in the world to hold the transcript of the discussion , but your setup needs to be designed for the workload you want, so it all depends on use.

PS! regarding your comment about atime , if I remember correctly lazytime (which is posix compliant) was supposed to be imeplemented on the VFS layer , but so far it seems to be pretty filesystem specific. BTRFS for example have relatime by default and I think I remember a discussion some years ago where it was decided that BTRFS would not benefit from lazytime compared to relatime - that may have changed of course...

[gadnet]

Pretty good improvements on Btrfs part compared to a year ago.

I think you should run these benchmarks with noatime as default though. atime hurts performance, especially for COW filesystems, and is not really used by anything these days, unless you use mutt.

i really wonder if ANYONE use atime anyway apart for hardcore mutt users yes but i was thinking it was default mount option now on all distros, the atime cost is really too much for what it provides in return.

[gadnet]

this is rather strange to see some reading test with 10x perf loss from raid 1 to raid 10.
I dont see how any reads could be 10x slower in raid10 as it seems to me that the worst case scenario a raid 10 is as slow as raid1 and in best case it is 2x faster no ?

[S.Pam]

You're right there. How about testing both options then . I love the flexibility with Btrfs. Both with snapshots and the whole volume management thing, but also the ability to hot add/remove disks live without any downtime or unmounting.

[waxhead]

I don't agree. It's not how you normally use these new filesystems.

Your analogy is akin to comparing ostrich with another bird and judging their ability to run, ignoring that one of them can fly.

Fair enough , but even without self healing BTRFS have a few other tricks up it's sleeve such as snapshots, compression and the fact that you would get a error message if your file has silently corrupted is still something that (as far as I know) neither xfs or ext4 can offer in such a setup. Not having self-healing "enabled" may not be such a problem for something that actually not happen that often, and if you have good tested backups! So yeah , I understand what you mean by people normally not using the filesystem like that , but I still think it would be a good idea to test btrfs the same way as the others - but hey, that's me!

[StuffOfInterest]

I think a RAID 5 set of tests would have been helpful here. It appears that btrfs did better when striping was in use (RAID 0) than when mirroring was was used (RAID 1). A hybrid of the two (RAID 10) ended up in the middle. Having RAID 5 would give a another data point to see if btrfs was inherently better with striped data.

[S.Pam]

my point was that if Phoronix does a benchmark like this then at least include BTRFS in the same setup as the other filesystems e.g. on top of MD. You want to compare eggs to eggs right?.

I don't agree. It's not how you normally use these new filesystems.

Your analogy is akin to comparing ostrich with another bird and judging their ability to run, ignoring that one of them can fly.