Benchmarking ZFS On FreeBSD vs. EXT4 & Btrfs On Linux

[LasseKongo]

I find this hard to realize, because trollman attacks the essence of Open Source and defends anti Open Source company. It's enough to read some comments here or at some mailing lists, osnews to realize the guy isn't emotionally stable. It's like sun crap makes him to feel good.

Believe it or not but it does happen, maybe because there is one thing Mr. Kebbaert hates more than Linux, and that is Microsoft which are much more hostile to open source than Sun and maybe even Oracle. We have to see about Oracle though, it looks like they are doing their best to destroy all the open products they got when buying Sun.

I had high hopes myself that Opensolaris would be used to bring the whole Solaris platform in to the 21?st century by providing both good stability, a nice filesystem but also more modern look for desktops, bettet package management etc, but Oracle seems to think different.

When i built my last fileserver I waited and waited for the new Opensolaris release (I wanted dedup), but I eventually gave up and used Linux instead, currently with XFS, but will hopefully move to Btrfs once it is deemed stable.

[LasseKongo]

Posting papers from 2008, 2006... is misleading in computer world. The information is basically not valid anymore.

Well, returning to thread main question. About performance, there is a guy on btrfs mailing list that tried btrfs on high-end server with 16 multiple solid state disks. First he do the test with 2.6.32

ZFS BtrFS
1 SSD 256 MiByte/s 256 MiByte/s
2 SSDs 505 MiByte/s 504 MiByte/s
3 SSDs 736 MiByte/s 756 MiByte/s
4 SSDs 952 MiByte/s 916 MiByte/s
5 SSDs 1226 MiByte/s 986 MiByte/s
6 SSDs 1450 MiByte/s 978 MiByte/s
8 SSDs 1653 MiByte/s 932 MiByte/s
16 SSDs 2750 MiByte/s 919 MiByte/s

Then he switch to 2.6.35 , which has better optimizations, mainly direct I/O and hardware checksum check.

Reference figures:
16* single disk (theoretical limit): 4092 MiByte/s
fio data layer tests (achievable limit): 3250 MiByte/s
ZFS performance: 2505 MiByte/s

BtrFS figures:
IOzone on 2.6.32: 919 MiByte/s
fio btrfs tests on 2.6.35: 1460 MiByte/s
IOzone on 2.6.35 with crc32c: 1250 MiByte/s
IOzone on 2.6.35 with crc32c_intel: 1629 MiByte/s
IOzone on 2.6.35, using -o nodatasum: 1955 MiByte/s

The thread:

http://www.mail-archive.com/linux-bt.../msg05689.html

Nice find, interesting figures. Seems like ZFS has the upper hand when using a larger number of drives, but I think we will se Btrfs closing this gap with new releases, there was a significant jump from 2.6.32 to 2.6.35.
Lets just hope SSDs drop in price so I can afford 16 of them

[kraftman]

Believe it or not but it does happen, maybe because there is one thing Mr. Kebbaert hates more than Linux, and that is Microsoft which are much more hostile to open source than Sun and maybe even Oracle. We have to see about Oracle though, it looks like they are doing their best to destroy all the open products they got when buying Sun.

I had high hopes myself that Opensolaris would be used to bring the whole Solaris platform in to the 21?st century by providing both good stability, a nice filesystem but also more modern look for desktops, bettet package management etc, but Oracle seems to think different.

When i built my last fileserver I waited and waited for the new Opensolaris release (I wanted dedup), but I eventually gave up and used Linux instead, currently with XFS, but will hopefully move to Btrfs once it is deemed stable.

I also thought Oracle will do more good for the community, but it seems they're not better then Sun, maybe even worse. At least they support btrfs development, but on the other side they attack Red Hat and promote 'their' Linux which is based on RHEL.

@Jimbo

Posting papers from 2008, 2006... is misleading in computer world. The information is basically not valid anymore.

Well, returning to thread main question. About performance, there is a guy on btrfs mailing list that tried btrfs on high-end server with 16 multiple solid state disks. First he do the test with 2.6.32

There are some concerns on the mailing list you have provided about the way someone benchmarked btrfs, but it's great to see such comparison!

[bewst]

just look at the ZFS server I linked to.

Would you mind re-posting that link? I can't find it.

Thanks.

[kraftman]

Solaris doesn't scale Kebb, it scales only up to 64CPUs (next year it should scale up to 128CPUs) while Linux scales up to 4096CPUs right now. I bet neither you nor me can prove which scales better up to 64CPUs. Linux is far more superior, but Solaris has ZFS which is its advantage right now.

http://www.h-online.com/open/news/item/Oracle-outlines-Solaris-11-says-little-of-OpenSolaris-1053998.html

A focus on scalability would also be incorporated in preparation for the next generation of hardware, like the 128 core, 16,384 thread system with 64TB of memory that Oracle is currently developing. Solaris 11 is expected around the second half of 2011.

[kebabbert]

@Trollman

You like quoting Linus a lot, so here you go:

So, Linus says Linux code is better FUDer.

Again, it is about credibility. I have told you, if you want to show something bad about Solaris, then you have to give credible links, such as research papers, Solaris developers, etc and other credible links. You can not give links to Solaris competitors - that is simply not credible. That is the reason I quote Linus when I want to show something negative about Linux. Or when I quote Linux scaling experts, when I want to show how bad Linux scales. I would avoid quoting Solaris experts, that is simply not credible when we talk about Linux crappy code.

So, when you quote Linus Torvalds, who said "But I am in active competition with Sun, so I hope Solaris dies" - do you really think that is credible when he says "Linux has better code than Solaris"?

[kebabbert]

Kebabbert is as usual viciously defending everything Sun/Solaris.

This is not true. I have attacked Sun in IDG.SE forums when they did something bad.

Same story in the Swedish forums where he also hangs out, always claiming that he is not posting any FUD and always backing up his claims with links to "research papers", some which are of questionable value (biased) or too old.

At least I post some credible links. Others here do not. What would you prefer, me to make up things and lie, like the Linux guys here confessed they do, or would you like me to post links to academic papers?

For Christmas 2008 he was even invited to the the Sun Sweden Christmas party for being such a good henchman, run this link through the translator if you want:

http://www.idg.se/2.1085/1.202161/staende-ovationer-nar-kebabbert-dok-upp-pa-suns-julfest

Translation of the heading if you don?t bother to read the rest: Standing ovations when Kebabbert showed up at the Sun Christmas party.

This is true and nothing I try to hide or conceal. I was a Sun supporter, but now I am supporting Oracle instead. I bet there are are some Linux supporters here? If Oracle does something bad, I will attack Oracle. As I have attacked Sun earlier.

Although he also defends open source in general (which I also advocate), I find it hard to trust someone that is so completely biased to a vendor that he automaticly assumes that any competing products are inferior without really testing them, or reading up on them, like in the beginning of this thread when he didn?t even know Btrfs was fully checksumming all data & metadata, one of the basic design goals.

Maybe I have read things about BTRFS? Maybe I know about research on data corruption? Maybe I am just questioning if the BTRFS team will really succeed with providing data safety? I have read lots on data safety, even research papers. Researchers show it is very difficult to really achieve data safety, I have posted several research papers on this. I KNOW this is HARD. If you really think it is only a matter of adding some checksums here and there in BTRFS, you have not followed the debate on data safety for years (as I have). That is the reason I am sceptical, I do have much research to fall back upon, to justify this scepticism. It is not just some downtalk from my side. I am basing my scepticism on earlier research, not on lies or Air. But I would love to be proved wrong regarding BTRFS. Competition is good.

In short, you have not followed the data safety research for years, and you would be naive if you trust BTRFS to give data safety. Of course the goal of XFS and JFS and Raid5 is also to give data safety. It is outspoken design goal. But, guess what? THEY DONT SUCCEED GIVING DATA SAFETY. What does that tell you? That "I am biased against BTRFS"? Or that "I have read much research papers and therefore I am sceptical because I know much about data corruption"?

I also have to comment on the "Btrfs has only one full time developer" issue, for once we don?t know how many part time developers there are, if it was 50 people working 50% I think that would be worth more than 1 full timer ? So even if it is just one full time developer, it doesn?t necessarily mean it is low on resources.

Of course you are right here. That is not the question. I have never questioned if there are many developers on BTRFS. I am only asking, "if Oracle is serious with BTRFS, why dont they assign a large team in-house at Oracle"? Oracle has recently killed off OpenSolaris distro, and rename it to Solaris 11 Express. They want total control over the products they sell.

You know, there are lots of ZFS bashers here. I am just trying to bash BTRFS back. There are lots of "muahahaha ZFS is sloooow", "ZFS is dead!", "BTRFS is a ZFS killer" etc. To get some balance, I just posted some valid concerns about BTRFS, if you read research papers. Do you read research papers?

Also if you look at recent kernel changelogs since a while back you will see that there are people committing more code than the supposed only full timer Chris Mason. In the latest 2.6.35 kernel there were 13 submitted patches, none of them were authored by Mason.
See http://kernelnewbies.org/LinuxChanges

Good for BTRFS. Yes, I mean it. Competetion is good, it forces ZFS and BTRFS to be better. And us customers wins!

I can agree though that it would be really interesting to see a ZFS/Btrfs comparison on some real heavy workloads with a lot of drives, but I can?t really find anything useful on the web.

I agree that heavy workloads are interesting, because ZFS is for Enterprise storage servers, not desktop. I am really not interested in performance, as I told you earlier. But to stop all those ZFS bashers, I just told them that ZFS also can achieve extreme performance if there is a need. "Muahhaha in your face".

Since Btrfs is from what I know not really even near a v1.0 there will probably be a lot of room for various optimizations still

I know BTRFS is not ready production. That is the reason I say BTRFS lags behind ZFS as of NOW. BTRFS is inferior AS OF NOW because BTRFS is in beta stage. Maybe in the future BTRFS will be better than ZFS, but I doubt that - this is my personal guess. ZFS team will not rest.

but I?m pretty confident that it will be a very safe filesystem once it reaches production,

This is I my main concern. Why do you think this? Have you read one single research papers on this, at all? If you had been more knowledgeable on this, you would also doubt. Just like me. I am not trying to downtalk BTRFS, data corruption is a just real concern. If you ask researchers on data corruption I bet good money they would laugh if you said that a "solution is safe". I am not saying ZFS is without problems or bugs, but I am saying that initial research shows that ZFS is safe. Research also shows that all those "robust filesystems" such as XFS, JFS, etc are not safe. Wouldnt it be good to listen to researchers?

I know how bad Linux scales, but Linux kernel devs says it scales well. Why would I trust one single guy Chris Mason who designs BTRFS to be safe?

probably in a year or two, and if it combines that with good performance ZFS will have a tough competitor.

Competition is good. I want ZFS and BTRFS to be the best they can be. I will sit in the middle and get good technology, for free!

I?ll keep tracking this thread, I guess there is more FUD to come

If you suggest I lie, then please point out my lies. If you suggest (like Kraftman) I downtalk Linux, then your definition of FUD is not correct. Sure, I downtalk Linux, but I do not lie about Linux.

Otherwise, please dont call me a FUDer, because then it is you that do FUD about me. Just like the MS people at IDG.SE. I have asked them for years to point out ANY lies from me. To quote a lie. They have never ever succeeded. But I have proved that they lie.

[kebabbert]

Posting papers from 2008, 2006... is misleading in computer world. The information is basically not valid anymore.

But at least I am posting some credible links. I do not make up things, nor lie as Linux guys confessed they do. What would you prefer, me to lie as them, or post to research papers that are 2 years old?

Well, returning to thread main question. About performance, there is a guy on btrfs mailing list that tried btrfs on high-end server with 16 multiple solid state disks. First he do the test with 2.6.32

ZFS BtrFS
1 SSD 256 MiByte/s 256 MiByte/s
2 SSDs 505 MiByte/s 504 MiByte/s
3 SSDs 736 MiByte/s 756 MiByte/s
4 SSDs 952 MiByte/s 916 MiByte/s
5 SSDs 1226 MiByte/s 986 MiByte/s
6 SSDs 1450 MiByte/s 978 MiByte/s
8 SSDs 1653 MiByte/s 932 MiByte/s
16 SSDs 2750 MiByte/s 919 MiByte/s

Then he switch to 2.6.35 , which has better optimizations, mainly direct I/O and hardware checksum check.

Reference figures:
16* single disk (theoretical limit): 4092 MiByte/s
fio data layer tests (achievable limit): 3250 MiByte/s
ZFS performance: 2505 MiByte/s

BtrFS figures:
IOzone on 2.6.32: 919 MiByte/s
fio btrfs tests on 2.6.35: 1460 MiByte/s
IOzone on 2.6.35 with crc32c: 1250 MiByte/s
IOzone on 2.6.35 with crc32c_intel: 1629 MiByte/s
IOzone on 2.6.35, using -o nodatasum: 1955 MiByte/s

The thread:

http://www.mail-archive.com/linux-bt.../msg05689.html

Ok, so what are you trying to prove here? That ZFS is slower? Ok, it may be slower, according to this test. It is not really important to me, who is fastest. Which data is safest? That is the thing I care about. If research shows BTRFS to be safer than ZFS, then I switch. If BTRFS is faster, I dont care. I am only saying that it is possible to reach extreme performance with ZFS

But I would prefer a comparison with ZFS on a Solaris machine, and not on a FreeBSD machine as your post shows. Your post uses FreeBSD and ZFS. The FreeBSD port of ZFS has not been without problems, and ZFS in FreeBSD always lags behind Solaris version.

And also, your post uses only 16 drives. That is nothing when we talk about Enterprise storage halls. You dont see such tiny machines in Enterprise halls. ZFS is for Enterprise halls, with many discs. Then it excels. It may be slow on desktop. That is ok with me. I dont mind if ZFS is slow on desktop, as long as my data are safe.

[smitty3268]

Ok, so what are you trying to prove here? That ZFS is slower? Ok, it may be slower, according to this test. It is not really important to me, who is fastest.

And also, your post uses only 16 drives. That is nothing when we talk about Enterprise storage halls. You dont see such tiny machines in Enterprise halls. ZFS is for Enterprise halls, with many discs. Then it excels. It may be slow on desktop. That is ok with me. I dont mind if ZFS is slow on desktop, as long as my data are safe.

Ah, kebabbert - are you even reading the links posted? That test showed that ZFS was faster than BTRFS, once you added in enough disks, but that with a few tweaks and a newer kernel BTRFS was quickly catching up.

Also, how many disks are required to be in a machine before it counts as "enterprise" in your mind? 16 high-end SSDs sounds like quite a bit to me, they probably have the capabilities of 3 times that many traditional HDDs. More than that is quickly becoming very, very, specialized and marginal market.

[smitty3268]

I am only asking, "if Oracle is serious with BTRFS, why dont they assign a large team in-house at Oracle"?

Because they don't need one?

Sun was the sole developer of ZFS, so they had to devote large amounts of resources to it's development. BTRFS @ Oracle was a completely different situation. Oracle took the lead in developing it, but they were also able to draw upon resources from Red Hat, IBM, and many other places throughout the community, making a larger in-house team by Oracle unnecessary.