4-Disk Btrfs Native RAID Performance On Linux 4.10

[starshipeleven]

That makes sense, thanks. Does it affect burst transfer performance though?

Can you clarify? What you mean?

[jacob]

Can you clarify? What you mean?

Let's say the FS wants to transfer a number of logically contiguous blocks (like an extent, for example). Normally it would occur as a single DMA operation, in burst mode. But if the physical blocks are scattered around, would that affect the transfer speed and/or max number of blocks transferred per request?

[stiiixy]

As far as I am aware, ZFS on Linux was still a FUSE-vased implementation, and would therefore not yield proper results compared to a native Solaris or at least BSD system.

And why has no one here mentioned the one critical point about RAID1? Its only as fast as the slowest drive. You can have 100 drives, but it still only reads at that slowest drives speed. You also only have that smallest drives capacity limit as well. But, you'd have redundancy across 100 drives. You only start to get speed from RAID1 if you pair it with 0, and that is spread across the number of 0 arrays, but increases risk of lost data. Let's say we turned the 100 drives in to RAID10 with 50 drives in RAID1, the next fifty in RAID1, then we can put them in to a RAID0 array for twice the speed. Similarly with four arrays of 25 drives.

Going by my testing the benefits of BTRFS simply weren't there compared to md on spinning rust (no SSDs). It was slow in RAID10, and 5 was slower and 6 was abyssmal. Native/MD was far superior with a choice of FS to boot, giving both speed and redundancy. If it takes less time on my drive to do that same thing, then I've won, as there's less wear and tear being performed on the systems resources. Since the BTRFS 5/6 storage issue popped up, I had to stay well clear, as we're using several drive arrays for primary storage and didn't feel like having to risk losing any of them and testing all the archived data all the time to be sure things worked.

And I want BTRFS to be the FS it was promising. Just not yet for my needs.

[Zucca]

heavy caching is done by page cache, zfs uses much more ram due to obsolete design

Hm. Obsolote? Which "brach" of ZFS? OpenZFS, OracleZFS? (Are there more?)
l also wonder how much (if any) there is performance gain/loss between those.

I, personally have been using btrfs only, as it seems quite user friendly AND btrfs does not care a bit about different disk sizes and how many disk you give to it.
For example I used 5xSSD setup a while ago. I think there was three different sized disks. btrfs managed to utilize around 90% of the space on RAID1 (and briefly on RAID5 and RAID6 too). I've head that ZFS on the other hand is more picky (enteprise users don't really care about that)... But still flexible when compared to "regular raid".

[starshipeleven]

Let's say the FS wants to transfer a number of logically contiguous blocks (like an extent, for example). Normally it would occur as a single DMA operation, in burst mode. But if the physical blocks are scattered around, would that affect the transfer speed and/or max number of blocks transferred per request?

Nope. Flash chips are in the "RAM" category, they are "random access memory", fragmentation does not affect performance (as long as there is enough free space for new writes, SSD controllers also do scans on their own to "defrag" and compact their flash level to leave enough free space without telling anything to the OS, but fragmentation per-se isn't an issue).
Hard drives are of course sequential access memory, that's a very high-tech version of a gramophone after all.

The main limitation of flash is read/write speed, a flash chip isn't terribly fast on its own.
SSD controllers give you far more speed than the average usb flashdrive because they actively fragment the writes you do on different flash chips, making a "RAID0" of sorts (some also have caches and other stuff on different faster chips and whatever).

[starshipeleven]

As far as I am aware, ZFS on Linux was still a FUSE-vased implementation, and would therefore not yield proper results compared to a native Solaris or at least BSD system.

Wrong, there is an out-of-tree kernel module, and Ubuntu is using this. https://github.com/zfsonlinux/zfs

And why has no one here mentioned the one critical point about RAID1?

Because btrfs RAID1 is not like that?

You also only have that smallest drives capacity limit as well. But, you'd have redundancy across 100 drives.

Again, btrfs RAID1 is not like that, you can have a RAID1 with x1 1TiB and x2 512GiB drives and it will still have 1 TiB of capacity, as long as it can split the stuff on two different drives it's happy.
Also btrfs RAID1 has no way to increase the amount of redundancy, all drives you add increase capacity, not redundancy.

Going by my testing the benefits of BTRFS simply weren't there compared to md on spinning rust (no SSDs).

As said to others, btrfs's raid was not meant to compete with plain block-level RAID, but to allow btrfs to work safely on multiple drives.

The fact that you post bullshit like "the benefits of btrfs simply weren't there compared to md" you clearly show you don't fucking need any of the many features btrfs offers vs any other filesystem, so the issue is only on your side that you chose the wrong setup.

Just not yet for my needs.

Given the stated needs, btrfs will never be the fs for you. If you don't give a fuck about checksums, snapshots, CoW, deduplication, array flexibility as said above (you can also convert array types live) and only need raw speed, you will never need to switch from ext4/xfs on mdadm raid. It's unlikely that raw speed of btrfs will ever match that of plain block-level raid with far simpler filesystems.

[starshipeleven]

Hm. Obsolote? Which "brach" of ZFS? OpenZFS, OracleZFS? (Are there more?)

He is talking about core ZFS design concept, not the slightly different branches.

[pal666]

sudo apt install zfs

sudo: apt: command not found

btw, linux is https://git.kernel.org/cgit/linux/ke...inux.git/tree/

[pal666]

Hm. Obsolote? Which "brach" of ZFS? OpenZFS, OracleZFS? (Are there more?)

yes. all of them.
zfs was designed before invention of cow btrees. so zfs designers sacrificed btrees for cow.

I've head that ZFS on the other hand is more picky

i've heard it can't change filesystem size

[SystemCrasher]

Hm. Obsolote? Which "brach" of ZFS? OpenZFS, OracleZFS? (Are there more?)

I guess he refers to the fact ZFS had its own memory management, not really integrated with rest of linux kernel memory mangement. OTOH btrfs is properly integrated with Linux kernel from the very start. So btrfs behaves pretty much like any other Linux filesystem and does not hogs much memory, nor one have to fiddle with cache size tuning on low mem systems. It could even be used on small pi sized SBC things, Linux Sunxi ppl are quite serious on using btrfs to create tiny NAS-like things, etc. Not sure if someone managed to do something about this weird memory management in ZFS. I don't care anyway since I'm not going to use foreign modules, especially for filesystems. Especially for boot filesystem, etc. Easy to guess why.

l also wonder how much (if any) there is performance gain/loss between those.

Then I guess you could run benchmarks? Though who cares of e.g. proprietary Oracle solaris? Granted recent news, who would use Oracle Solaris anyway? They have to be really crazy persons.

RAID6 too). I've head that ZFS on the other hand is more picky (enteprise users don't really care about that)... But still flexible when compared to "regular raid".

Btrfs does RAID quite smartass way. If you've got 5 storages and want mirror, it comes down to "put me 2 copies of blocks on different devces". Interesting part is the fact pairs of devices aren't sent in stone and its runtime decision instead, the only limit is there should be at least some 2 devices with some free space on them. They do not have to be equal size, etc. So if one uses mirror and got 5 similar devices they could expect approx 2.5 x device size capacity. Getting a bit more complicated to compute if devices are inequal, hopefully it gives idea why question like "how many free space I have left" isn't very trivial to asnswer for btrfs. It would keep storing mirrored blocks as far as it could find 2 different devices with some free space, and these do not have to be same pair of devices all the time.