Btrfs RAID 0/1/5/6/10 Benchmarks On Linux 4.12

[starshipeleven]

Better question are there any known bugs in btrfs that eat your data? Did they managed to fix them all or there are some left.

RAID 5/6 still eats your data the same as before (there is still the so called "RAID5 write hole", and parity isn't checksummed), there still are some bugs that may force your fs to go read-only if using RAID1/10, but there are fixes for that that should get in 4.13 or 4.14 (making btrfs aware of devices connected/disconnected).
Quotas aren't reliable.

[Creak]

I may be wrong but I think that Btrfs' RAID implementation replicates individual files, not entire volumes as LVM does, so in theory it's more flexible. In practice of course, if you want Ext4 with RAID you can only use LVM and if you want Btrfs with RAID, you must use its own RAID support (I don't remember why but there is some reason why it should not be used on top of LVM).

BTW, Btrfs' other features (snapshots, subvolumes, CoW etc.) are EXTREMELY addictive. I also thought I didn't really need them but after having tried using them once, I can't imagine living without them. It's not even just for servers - I'm talking about a laptop used primarily as a development machine and day-to-day usage - so beware! ;-)

Interesting... I mainly thought that Btrfs was intended for servers. I can see how snapshots can be interesting (that would be the main feature that could interest me in Btrfs), but can you give me some examples of a day-to-day usage of subvolumes, CoW, etc. ? Get me excited!

[bhuisgen]

I've lost 4TB of data with a btrfs RAID1 array last month. Nothing wrong with my hard drives but 1 btrfs leaf was corrupted (kernel 4.9). So there is nothing to repair (in fact btrfs repair segfault) so you replace the faulty disk and you loose everything. So it's funny to see these benchmarks trying to compare shitty fs and stable fs for performance... To destroy your data btrfs is the big winner

[starshipeleven]

Interesting... I mainly thought that Btrfs was intended for servers. I can see how snapshots can be interesting (that would be the main feature that could interest me in Btrfs), but can you give me some examples of a day-to-day usage of subvolumes, CoW, etc. ? Get me excited!

I use deduplication heavily, it allows me to make dozens of copies of the same projects (each with minor modifications/tests/something) and will use up only a fraction more space than it actually would if I copied them whole.

If you copy something it's automatically deduplicated (i.e. it's just a link to the same data blocks), if you copy from outside you need to run offline btrfs deduplication tools to find what can be deduplicated.

[kreijack]

RAID 5/6 still eats your data the same as before (there is still the so called "RAID5 write hole", and parity isn't checksummed), there still are some bugs that may force your fs to go read-only if using RAID1/10, but there are fixes for that that should get in 4.13 or 4.14 (making btrfs aware of devices connected/disconnected).
Quotas aren't reliable.

In the 4.11 and 4.12 some patches address most (all) of the known bugs related to the parity rebuilding. The BTRFS crews are looking to some systematic tests about raid5/6. So the things are going a lot better regarding raid5/6. Of course these patches are recent, so we need some months to be sure that all the bugs are properly addressed.

Regarding the RAID5 write-hole, yes BTRFS has it, as mdraid subsystem until few months ago. So it is not so terrific. Of course if you want to use raid5/6 you need a battery backup-ed power supply (but this is not BTRFS specific. It also is true for a harware-raid without battery backup).
Regarding the fact that the parity is not checksummed: which would be the problem.? The (meta)data is checksummed, so adding a further checksum doesn't add any value.

It is true that the device disconnect/removing still has problem.

[kreijack]

Is the integrated RAID feature of Btrfs better than the one from lvm?
(just focusing on Btrfs vs lvm, not Btrfs vs Ext4 since I'm pretty sure I don't need all the features of Btrfs)

If you have some form of redundancy, in case of corruption of the data BTRFS (and ZFS) is capable to rebuild the corrupted data from the good copy. This feature is not available nor with mdadm, nor with LVM (or device mapper).

On the other side, BTRFS is still immature in handling a device disappear.

[starshipeleven]

In the 4.11 and 4.12 some patches address most (all) of the known bugs related to the parity rebuilding.

Which is still only some of the issues, and only on 4.12, not on 4.11. As I said it's getting better, but it's still not safe.

Regarding the RAID5 write-hole, yes BTRFS has it, as mdraid subsystem until few months ago. So it is not so terrific. Of course if you want to use raid5/6 you need a battery backup-ed power supply (but this is not BTRFS specific. It also is true for a harware-raid without battery backup).

Well, mdraid isn't a COW filesystem that is supposed to be immune to unclean shutdowns (in single-disk/RAID1/10 it is).

Regarding the fact that the parity is not checksummed: which would be the problem.? The (meta)data is checksummed, so adding a further checksum doesn't add any value.

That if parity isn't checksummed then the guarantee that my data isn't corrupted goes out of the window.

Note that metadata is the filesystem's infrastructure, while data is my stuff.

[PuckPoltergeist]

RAID 5/6 still eats your data the same as before (there is still the so called "RAID5 write hole", and parity isn't checksummed),

I doubt, btrfs will get checksums for parity. At least not with this design, cause this breaks semantic layers. Also I don't see the benefit from checksumming parity. Why should anyone do this?

[starshipeleven]

I doubt, btrfs will get checksums for parity. At least not with this design, cause this breaks semantic layers. Also I don't see the benefit from checksumming parity. Why should anyone do this?

Because without checksums then how can you be sure that parity isn't corrupted?
In its RAID1 everything is checksummed for example, not just one of the two copies.

[PuckPoltergeist]

Because without checksums then how can you be sure that parity isn't corrupted?
In its RAID1 everything is checksummed for example, not just one of the two copies.

Are you sure the disk-stripes are checksumed? Doesn't sound reasonable, cause this will be already a layer-break.

How I could sure, parity isn't corrupted? The same way, I'm sure, the data-stripes aren't. Simple example with RAID5 with three disks:

The data is reconstructed from the two data-stripes. If one data-stripe isn't available (cause disk failure), it's reconstructed from one data-stripe + parity. If one of them got corrupted, the reconstructed data will be corrupt. But this will be covered by the checksum from the file/metadata. Any corruption will be uncovered by the checksums on filesystem level. So why adding more checksums on device level?