Btrfs To Ship Multiple Performance Improvements In The Next Linux Kernel

[ssokolow]

What world I am living in you ask?

Note that it says "The Netherlands". You never want to start an argument about the world by generalizing research on a single country.

[aht0]

Indeed. But that's OK. I have never been been fond of that "or any later version" clause. I also release my own software strictly as GPLv2 only. In part because I still can't wrap my mind about what the GPLv3 actually means in terms of its legal foundations and consequences, and also because I don't like the idea that someone will at some point rewrite the licence for my software in ways that I can't predict.

It's your call.

Of course everything can always be attributed to luck, but initially FreeBSD was really more advanced than Linux, and yet the big players preferred investing massively into then-toy OS Linux (in the case of IBM, it was literally billions of $$$) than contributing to FreeBSD which, in theory, could have been ready for prime time quicker and for cheaper thanks to its headstart. One can forever speculate why it was so, but FWIW my explanation is that contributing stuff like XFS, LVM, RCU etc. to BSD means effectively giving it for free to your competitors to use in their proprietary products, that compete directly with your own. Contributing the same to Linux is safe from this point of view, courtesy to the copyleft a.k.a. viral nature of the GPL.

If you had bothered to check, you'd notice that IBM started it's support for Linux back in 1999. Oct 1998 FreeBSD 3.0-RELEASE with initial SMP support was released. Which was quite problematic. By the time of 4.0-RELEASE which had fixed these issues (Spring 2000), train had passed.

You understand licenses quite wrong. BSD license does not denounce ownership/authorship, it's just giving user more freedom to do with the software as they please. Author remains author and he/she could sue you, if you removed the relevant authorship-headers from sources. You could fork it and relicense your fork under GPL but the original author has to remain that.

The desktop remains the great failure for Linux, that's clear. As for the rest, which planet do you live on? Windows holds something like 35% of the overall server market and is heavily concentrated mainly on SMBs. The rest is pretty much all Linux (with a very very VERY few notable exceptions). And there is also a lot more to computing than desktop and servers. Last time I looked it up (~ 2 weeks ago), some 67% of all cloud deployments were Linux-based. All 500 of the current Top500 run Linux. More than 75% of all mobile devices in the world run Linux. In the IoT world, there is basically Arduino, MIPS and ARM, and the latter two are virtually all Linux (if you search hard enough you may be able to find the odd NetBSD here or there, but it's statistically irrelevant).
That's a definition of success lots of people would dream of.

bah.. 75% mobiles run Android, not Linux. Android is just using Linux kernel and highly modified one at that. Following that same chain of logic one could claim that Android is also BSD because equally important OS component (It's C library) has BSD origins.

What world I am living in you ask?

[GreenReaper]

And, of course, Facebook uses BTRFS.

...largely in single mode, as I understand it. Also Facebook probably has node-level redundancy going on. It might be more more suitable for systems with hardware RAID if you're looking for redundancy.

[caligula]

This is great and all, but I don't think most desktop users notice much difference in filesystems, I certainly don't. The difference between a HDD and an SSD is night and day, but once you've made that leap I don't think there's much in it. I recently went from a SATA 3 SSD to an NVME m.2 drive for reasons unrelated to performance, I didn't notice anything. I use BTRFS, I doubt I'd notice any difference if I went with EXT4, or whatever the performance leader is.

Still, that's just my use case, I'm sure under different workloads, especially in the server space, this is particularly useful. And it's nice that it's getting development attention.

If your workload actually stresses the disk, your NVMe drive might throttle after few seconds due to overheating which closes the gap between SATA SSD and NVMe. This is quite likely if the drive is powerful and doesn't have a heatsink installed (it won't unless you bought one). Now, why a typical Linux user won't even notice this is the basic file operations might make extensive use of Linux disk cache, which works really nicely both for writes and reads. For example, my system typically uses around 2-3 gigabytes of RAM as a disk cache after starting up the desktop, browser, and other basic applications. A large part of my whole distribution is now loaded in RAM. It would work fast even off of an USB 1.0 drive. When I do some work, the tools don't read whole files, they use random access and seek only relevant parts. The difference is more obvious when actually doing something terribly I/O bound like having a big ass server or moving 500+ gigabytes of stuff around. When you operate the system manually, I'd argue that in many cases the tasks just don't generate enough IOPS to saturate the SATA SSD.

[Nille]

kreijack lets think about this scenario. Raid 6, 5 Disks and one Disk failed. Parity on one disk is invalid. How does btrfs fix this problem now? How does it know which parity is invalid? Does the restore check it that the file was invalid with parity A and test again with parity B?

This is one of my concerns.

[kreijack]

Not sure what you mean by "checksums for parity against bitrot"? Btrfs has had checksumming for reliable data and metadata for years now.

parity was not checksummed in the past

BTRFS doesn't have parity checksummed. From a reliability point of view, there is no gain to have the parity checksummed. Even in the case that both the data and the parity are corrupted, the current BTRFS checksum infrastructure is good enough to detect the problem.
The only advantage having a parity checksummed is the speed to detect a possible corruption. However, again, the parity checksummed is not needed from a reliability point of view.

[waxhead]

Congratulations! You're the third guy to answer and completely fail to understand the meaning of AUTOMATIC. Automation is the complete opposite of having to manage everything by hand. OP did not ask how to disable COW, he asked how to make it automatic. Implying he already knew how to do it manually.

Yup, thanks for noticing. All good things comes in threes I guess. My mistake, and I hereby nominate myself to the ignorant-ego-bastard-of-the-week award. I only read the first page so there you go

Anyway , doing this automatic has to be done on the application level. And even then it is sort of "against" what the user expect of his/her/it's? filesystem.

The best "automatic" method is to create a separate subvolume, set that nocow and store your things in there. Ergo , a isolated part of the filesystem having nocow attribute set. On the other hand you *might* want to consider other filesystems for your use case , but then again you loose the flexibility of BTRFS anyway. Or another option is to enable the autodefrag option that sort of does the trick for some stuff.

[Michael_S]

(2)Bunch of random incidents aided Linux equally or more than it's license. Linux took off in popularity when FreeBSD was implementing SMP and did at first shitty job. Then FreeBSD's users (it was used far more than Linux back then) migrated to Linux because it happened to be ready and accessible alternative (no OpenSolaris yet). Later times, additional factors aiding Linux were Oracle closing OpenSolaris after buying Sun and Google opting to use Linux kernel for it's new embedded OS Android. Without all of it, FreeBSD or OpenSolaris could easily be in the same position Linux has nowadays. Just mostly luck IMHO.

Where GPL in fact aided and served it's purpose was with Linksys court case - suddenly people could have access to sources for their routers - would not have been possible with BSD license.

Also, define success - Linux has a few percents market on desktop and still less far smaller market share in servers than Windows, except for web servers where it indeed rules the roost.

I disagree on the license part. Many of the important contributions to Linux came from corporate-backed developers even in the late 1990s and early 2000s. It was popular then, but at the time it didn't dominate servers like it does today. The only reason many of those contributions went upstream was GPLv2. FreeBSD will never catch up because companies like Apple and Sony that make extensive use of it have no interest in pushing back key in-house enhancements that might allow it to compete with their own products.

Copyleft is free-as-in-freedom, BSD/MIT/Apache is free-as-in-labor. And of course today companies are violating the terms of the GPL or using firmware and associated tricks to avoid releasing their code while still using Linux.

And Android explains why Linux is popular as the kernel on mobile operating systems. But even without Android, Linux conquered the server space.

[garegin]

Not every app no, only ones that don't work well with COW. Only the application itself can know if it doesn't work well with COW, and only the application knows if it is safe to disable COW.

That’s what i had in mind.
i wonder how APFS or ReFS does it

[Spazturtle]

That's just retarded. Obviously there's no sense patching every single app that works with files to support a filesystem-level feature. It has to be a system service at a bare minimum if it cannot be built into the kernel.

Not every app no, only ones that don't work well with COW. Only the application itself can know if it doesn't work well with COW, and only the application knows if it is safe to disable COW.