Btrfs Gets Big Changes, Features In Linux 3.14 Kernel

[mercutio]

You can keep repeating yourself as much as you want, it won't change anything. There's not a shred of proof anyone would actually notice a difference by adding lz4.

Show me blind testing user studies where people are suddenly surprised by the speed difference and maybe i'll change my mind.

i don't think it needs to be proven that lz4 is faster. it's obviously faster. showing speed differences by user testing is going to make more difference on things like multiuser systems, slower systems, fast disk systems, cellphones etc. it also uses less cpu.

it could be argued that most cpus are 4 to 8 core now, and that speed doesn't really matter - and in that instance there are other alternatives too like zhuff:

http://fastcompression.blogspot.com/p/zhuff.html

which is by the same author as lz4. the cool thing about lz4 is it's not really slower than doing no compression at all, and on space starved ssd's etc, it could become a happy default.

when the next generation of ssd's come it should be a bit more obvious. on ssd raid it's already obvious when copying files over fast network connections. but i realise most people only have gigabit atm.

the main thing is it's pending for ages, and it seems to have somehow missed being included. it'll also need to be included in grub support etc to be applicable to most users, and being included in mainstream is going to make that a lot easier. in zfs it's easy to see it outperforming lzjb but lzo isn't lzjb.

for some reason btrfs on systems i've tried it on often seem to lag on disk writes afterwards with high cpu usage. i don't know if it's compression or normal background activities, but it's quite noticably intrusive enough so that on my main server i'm using ext4 on raid for /, and zfs for data.

[mercutio]

You're doing it wrong then...


(I use ext4)

with bonnie zfs with lz4 compression is about 2.5x faster on reads than btrfs with lzo compression on i7-4770 cpu with 3xsamsung 840 ssd in raid 10 far=2 layout (which gives 3 disk read performance). but bonnie is heavily compressible.

[mercutio]

In the workloads btrfs would use? I doubt it.

Again, like i said the first time and you conveniently snipped out of your reply: It's not likely to ever be noticed by anyone in real life, unless they are specifically staring at benchmark times.

There's lots of actual features that are far more important.

true, i'd like to see 3 disk raid 10 support, but i haven't seen any work started on that. and i think lz4 support would benefit more users. i'd ilke to see btrfs with lz4 compression start appearing on cellphones for instance.

[jwilliams]

true, i'd like to see 3 disk raid 10 support, but i haven't seen any work started on that. and i think lz4 support would benefit more users. i'd ilke to see btrfs with lz4 compression start appearing on cellphones for instance.

Also, the number of developer hours needed to add LZ4 to btrfs is very much lower than that needed for getting btrfs RAID working well with such capabilities.

I agree that LZ4 would be great on systems with lower-power CPUs and limited storage space, like cell phones.

[smitty3268]

i don't think it needs to be proven that lz4 is faster. it's obviously faster. showing speed differences by user testing is going to make more difference on things like multiuser systems, slower systems, fast disk systems, cellphones etc. it also uses less cpu.

it could be argued that most cpus are 4 to 8 core now, and that speed doesn't really matter - and in that instance there are other alternatives too like zhuff:

http://fastcompression.blogspot.com/p/zhuff.html

which is by the same author as lz4. the cool thing about lz4 is it's not really slower than doing no compression at all, and on space starved ssd's etc, it could become a happy default.

when the next generation of ssd's come it should be a bit more obvious. on ssd raid it's already obvious when copying files over fast network connections. but i realise most people only have gigabit atm.

the main thing is it's pending for ages, and it seems to have somehow missed being included. it'll also need to be included in grub support etc to be applicable to most users, and being included in mainstream is going to make that a lot easier. in zfs it's easy to see it outperforming lzjb but lzo isn't lzjb.

for some reason btrfs on systems i've tried it on often seem to lag on disk writes afterwards with high cpu usage. i don't know if it's compression or normal background activities, but it's quite noticably intrusive enough so that on my main server i'm using ext4 on raid for /, and zfs for data.

All of which i agree with. LZ4 would be nice to have.

I just don't agree with the "OMG, LZ4 would be amazing and they aren't doing it yet so they're awful people and horrible developers" mindset. There are a lot of other things that i think should take precedence first. When someone has the time, go ahead and hook it up and that'll be nice.

In fact, it should be something simple enough that a new volunteer unfamiliar with btrfs should be able to make it happen pretty simply. Maybe that's why none of the core developers are picking it up, because it's a nice entry-level task for someone new to come along and play with.

[jwilliams]

I just don't agree with the "OMG, LZ4 would be amazing and they aren't doing it yet so they're awful people and horrible developers" mindset.

You're hilarious. First you make false statements demonstrating that you do not know what you are talking about. Then when you realize your mistake, you inflate a huge straw man and try to brazen it out.

The fact is that Chris Mason is not good at project management for a large project of this type. He does not prioritize tasks, he does not manage developer time well, and he does not efficiently balance costs and benefits. No one said he is a "horrible developer", let alone an "awful person". That is just your straw man, trying to divert attention from your mistake.

The cost of a few developer hours of time is minimal, and the benefits of adding LZ4 significantly outweigh the costs. The time required is so small that it is absurd to claim that other things should take precedence. This is a similar idea to the two-minute rule from David Allen's "Getting Things Done". In time management, if something comes up that takes less than two minutes, just do it now. A similar idea applies to project management. If something with significant benefits can be done in a few developer hours, then it should be done immediately.

[smitty3268]

You're hilarious. First you make false statements demonstrating that you do not know what you are talking about. Then when you realize your mistake, you inflate a huge straw man and try to brazen it out.

Learn to read, please. It will help you out in life. I've been 100% consistent the whole time. Everyone knows that one compression is "better" than the other. The question is whether it's enough to be noticeable by the average person, and how important it is in comparison to other features. I don't know how many times i've said that, but it's been the same thing in every single post.

The fact is that Chris Mason is not good at project management for a large project of this type. He does not prioritize tasks, he does not manage developer time well, and he does not efficiently balance costs and benefits. No one said he is a "horrible developer", let alone an "awful person". That is just your straw man, trying to divert attention from your mistake.

Ok, perhaps that was an exaggeration. However, your initial post:

Just another symptom of the poor project management for btrfs. That is the sort of project that could be completed relatively quickly and provide some significant benefits.

is pretty clearly hinting in that direction. And i think you're wrong. Why should he bother with such a trivial feature when someone else can take care of the easy stuff on the side, while he focuses on the more difficult and important core features?

The cost of a few developer hours of time is minimal, and the benefits of adding LZ4 significantly outweigh the costs. The time required is so small that it is absurd to claim that other things should take precedence. This is a similar idea to the two-minute rule from David Allen's "Getting Things Done". In time management, if something comes up that takes less than two minutes, just do it now. A similar idea applies to project management. If something with significant benefits can be done in a few developer hours, then it should be done immediately.

Again, this all comes down to you thinking it's a "significant" benefits. I disagree, and you've yet to show any reason i might be wrong.

[smitty3268]

I agree that LZ4 would be great on systems with lower-power CPUs and limited storage space, like cell phones.

While this is certainly the target device where LZ4 would show the most improvement, who actually wants to use btrfs on a cell phone? No one is even discussing that, because no one wants all the overhead such a feature rich filesystem would provide on a phone. FSs such as F2FS are what are interesting there.

[jwilliams]

While this is certainly the target device where LZ4 would show the most improvement, who actually wants to use btrfs on a cell phone? No one is even discussing that, because no one wants all the overhead such a feature rich filesystem would provide on a phone.

Wrong again. You seem to make a habit of stating incorrect things, and then you tell me that I need to learn how to read. Heh. You need to learn how to write, or at least how to avoid making incorrect statements.

[mercutio]

Learn to read, please. It will help you out in life. I've been 100% consistent the whole time. Everyone knows that one compression is "better" than the other. The question is whether it's enough to be noticeable by the average person, and how important it is in comparison to other features. I don't know how many times i've said that, but it's been the same thing in every single post.


Ok, perhaps that was an exaggeration. However, your initial post: is pretty clearly hinting in that direction. And i think you're wrong. Why should he bother with such a trivial feature when someone else can take care of the easy stuff on the side, while he focuses on the more difficult and important core features?



Again, this all comes down to you thinking it's a "significant" benefits. I disagree, and you've yet to show any reason i might be wrong.

the difference isn't minor, as a general idea, here's fsbench on a i7-4770 cpu with 1 thread:
 # ./fsbench -b8192 -t1 lz4 zlib lzo /src/text/dickens 
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
LZ4                                     r97          
    7282840 (x 1.400)      363 MB/s 1859 MB/s       103e6  105e6
zlib                                    1.2.8        6
    4670489 (x 2.182)     34.0 MB/s  219 MB/s        18e6   19e6
LZO                                     2.06         1x1
    6969959 (x 1.462)      329 MB/s  468 MB/s       104e6  100e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

and with 8 threads:
# ./fsbench -b8192 -t8 lz4 zlib lzo /src/text/dickens  
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
LZ4                                     r97          
    7282840 (x 1.400)     1979 MB/s 7951 MB/s       564e6  566e6
zlib                                    1.2.8        6
    4670489 (x 2.182)      164 MB/s 1159 MB/s        89e6   94e6
LZO                                     2.06         1x1
    6969959 (x 1.462)     1848 MB/s 2483 MB/s       584e6  588e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

it's quite obvious from those numbers, that decompression speed with lzo on a single thread is slower than a ssd, and lz4 is significantly faster than a ssd. and on older cpus the same should hold true too. i used 8k block size because that's more akin to what file systems use, and is the standard block size of ssd's these days. i have no idea what file systems are doing for matching to block sizes of hard-disks/ssds/raid arrays when using compression atm. they may not be ideal still.