Announcement

Collapse
No announcement yet.

Benchmarking ZFS On FreeBSD vs. EXT4 & Btrfs On Linux

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • HPC servers are simple to build. SMP servers are difficult to build. I am claiming HPC can not be SMP.

    Ok, the discussion continues here:
    http://phoronix.com/forums/showthrea...508#post238508
    Last edited by kebabbert; 11-14-2011, 08:40 AM.

    Comment


    • Originally posted by kebabbert View Post
      Kraftmans comparisons:
      -----------------------
      You first compared a 2.4GHz Linux server to an old 800 MHz SPARC Solaris server. We see that Linux server is 3x faster here. 2.4GHz / 0.8Ghz = 3.

      Now you compare the latest 10-core Intel Westmere-EX, the fastest x86 cpu in the world, to an old AMD Opteron with 6-cores. SAP benchmarks show that the Westmere-EX is 3x faster than the AMD Opteron.

      Do you think these are fair comparisons? (As you said earlier: yes, these are fair).
      When Kebbaberts compares different systems it's good, but when others do this it's bad?


      Kebabberts comparisons:
      -----------------------
      I compare the SAP benchmarks from the same year. I do not compare the latest server to an several year old server.

      Linux server has an advantage here, it has faster cpus, and faster memory sticks. Solaris is at disadvantage here, it uses slower hardware. And who has the highest cpu utilization? Who gets the highest benchmarks? You tell me.
      You compare more powerful hardware to much less expensive one with less amount of RAM, slower memory sticks and slower database. Is this fair? Even slowlaris that wastes CPU cycles, because of its bloat can win benchmark when it's running on much better hardware. That CPU utilization and much lower score just means slowlaris is bloated, don't you think? Do you have something to backup your claims regarding Westmere-EX giving Linux huge advantage on 40core machine compared to 48core one? As a single CPU comparison it mayb be three times faster in some things, but do you have something to backup your claims that it will still have such advantage on 40core machine?

      Regarding if Bonwick FUDs or not about Linux scales bad on SMP servers, I asked you to post a link to a SMP server. You pointed to the SGI Altix server.

      Here is another Linux server. It has 4.096 cores, up to 8.192 cores. It also runs a single Linux kernel image. Just as the SGI Altix server.
      http://www.theregister.co.uk/2011/09..._amd_opterons/

      A programmer writes:




      The SGI Altix server that goes up to many thousands of cores:
      http://www.sgi.com/products/servers/altix/uv/


      http://www.hpcprojects.com/products/...product_id=941

      Thus, you are wrong again. The SGI Altix server is HPC, just as I said all the time.
      Being HPC doesn't mean it's not SMP. Try again, because it's SMP system.

      http://www.pcworld.com/businesscente...ll_street.html

      "The trading shops saw that the lowest-latency solutions would only be possible with Linux," Lameter said. "The older Unixes couldn't move as fast as Linux did."
      The wallstreet migration to Linux proves only Linux is fast enough and that's why they abandoned solaris. It's nothing amazing in this decision since Linux isn't bloated the way solaris is. It seems having 30% slower binaries is too much for critical workloads and that's why they replaced solaris. Damn shame.

      There are Linux SMP servers on the market, they typically have 4cpus, 6cpus or 8cpus. Then there are Linux HPC servers running a single Linux image, they typically have 4.096 cores or even more. There are not Linux servers in between. Either they have 4-8 cpus, or they have thousands of cores. Why are there no Linux servers with 16 cpus? Because such servers are SMP servers.
      Just lies. You saw 64CPUs Big Tux.

      Now, please post links to a Linux SMP server. Not a 4 cpu Linux server. But one with 16 cpus, or 32 cpus. Let me tell you: there does not exist such SMP Linux servers on the market. For a reason: Linux can not handle 16 cpus on SMP machines. Nor can Linux handle 32 cpus.
      I gave it you few times. Just google for Big Tux. It was 64CPUs (not cores) machine and afaik it has been upgraded lately.

      So, once more I ask you to post links to a Linux SMP server which has 16 or 32 cpus. You will not find any such links. Bonwick was right all the time; Linux scales bad on SMP servers. Up to 6-8 cpus are the maximum. On HPC servers everybody knows that Linux scales well.
      Kebbabert's lying as usual. Like I said, Big Tux.

      The Linux server is using 3x faster cpus. Of course Linux is faster. If they compared Windows on a 3x faster server, I promise you that Windows would be faster than Linux, too.
      The slowlaris server with double amount of RAM running n much more expensive hardware with faster database is simply faster. Even Windows would be faster on such hardware, I guess. However, I can't promise that.

      Thanks. At least I do not FUD, as you do. You confessed you FUD. This proves you FUD a lot.
      You FUD and lie like it was proven.

      What is your point? Linux does great speed ups, because it is so slow. What is your point? That Linux is slow?
      No, it's fast and becomes even faster. The point is it's being faster and faster in general.

      The benchmarks show that Solaris is faster.
      The benchmarks show Linux is faster.

      This is funny. You reject this ZFS vs BTRFS benchmarks, because ZFS used many disks and therefore scaled better. Your objection? That BTRFS is unstable, that is why you reject BTRFS.
      I don't reject this test, because ZFS used many disk, but because it was tested against unstable btrfs.

      Now, do you remember when Phoronix benchmarked OpenSolaris vs Linux, and Linux won? But OpenSolaris was the alfa version of Solaris 11 (OpenSolaris was not even beta). Why do think it is fair to compare unstable OpenSolaris to Linux? Why do you think it is not fair to compare ZFS to unstable BTRFS?
      In this case it should be fair, because solaris has very few devs compared to Linux, so it can't be so fast and stable thus it doesn't really matter if you benchmark unstable or stable slowlaris edition. Afaik Phoronix benchmarked "stable" (open)slowlaris editions, too.

      When Linux wins, it is fair and good. When Solaris and ZFS wins, it is unfair. Why is this? One rule book for Linux, and another rule book for others?
      That's your domain.

      Again, the Linux server used 3x faster CPUs.
      I don't have your point. It used less cores. Is it a fair comparison when slowlaris has hardware advantage? You were showing me slowlaris that used double amount of RAM, faster DB and much more expensive hardware. Is that fair?

      I missed this. The lastest SAP benchmark used 4 cpus. That is not good scaling. Linux did not scale crap out of Solaris, by using 4 cpus. Let me tell you, if Linux used 2 cpus, then I suspect Linux would have good cpu utilization.
      Linux doesn't waste CPU cycles for bloat thus it's faster even with less utilization.

      Ted Tso said that Linux kernel devs did not have access to as much as 32 core servers earlier. Ted Tso says the opposite of what you say, he agrees with me.
      Surprisingly there are kernel devs that has access to much bigger machines. In example SGI sent patches for Linux and there are other devs that has access to 48core systems and more. Ted didn't agree with you. Maybe he just didn't have access to big servers once, but he improved performance later. That's a fact. IBM, Fujitsu, HP, Oracle have access to big servers and they're working actively on the Linux kernel.

      You wont stop FUDing?
      I won't stop replying to your FUD. Will I get a price, too? Kebb, what do you think? Is there anyone so generous as sun was? ;>

      Comment


      • Originally posted by kebabbert View Post
        HPC servers are simple to build. SMP servers are difficult to build. I am claiming HPC can not be SMP.

        Ok, the discussion continues here:
        http://phoronix.com/forums/showthrea...508#post238508
        What does HPCwire means saying
        SGI Altix SMP System?

        Comment


        • I will answer in this thread
          http://phoronix.com/forums/showthrea...508#post238508

          Comment


          • Originally posted by kebabbert View Post
            Blah, blah, blah, blah...
            Ok, the discussion continues here:
            http://phoronix.com/forums/showthrea...508#post238508
            Thank God, take the e-Peen flashing elsewhere folks.
            No one gives a shit how big your e-Peens are.
            Last edited by jalyst; 11-21-2011, 10:16 AM.

            Comment


            • Errors are a fact of life.

              Originally posted by kebabbert View Post
              I havent looked at the benchmarks, but I am convinced you are correct.

              But instead of benching performance, if we benched data safety, I am sure ZFS destroys them all. BTRFS is not ready yet. I doubt it offers good data protection, read my long post above.

              Performance is not that important. Is your data safe? No? Then it is your choice to use a fast but data corrupting filesystem! I prefer to use a safe filesystem.
              That's just it -- different situations call for different tools. I read about people using ZFS for home media servers and the only ting I can think of is "overkill." Personally, I don't care if one bit gets flipped in a movie file. Seriously, one pixel, one color off, in one frame. I do believe that corruption happens more often than people think, but with GUI OS's, there's a huge chance that one pixel in a Windows Icon is going to be off. There's just so much non-critical space on a hard drive these days.

              Now, in the production server side, I could see where my work doesn't want a bit flipped in the source code and they are willing to pay the performance penalty and associated costs for ECC memory, etc. But they are sill stuck with NTFS, unless they want to triple IT spending and go with a SAN solution.

              But 100% data integrity doesn't seem possible at a reasonable price point, if you look as CERN's data. Even with ZFS and ECC, other subsystems can induce errors. I suppose you could go the three systems route and only trust trust results that two out of three agree upon but who would pay for that except for a few fringe cases? Certainly not my home media server.

              Comment


              • Originally posted by ldillon View Post
                That's just it -- different situations call for different tools. I read about people using ZFS for home media servers and the only ting I can think of is "overkill." .
                You wouldn't say that when your BTRFS or MDRAID-6 setup just utterly trashed itself.

                I benchmarked things - and then I started deliberately breaking things. ZFS survives incidents which would have you reaching for the backups in any other circumstance. On top of that it's quite fast when laid out correctly.

                It's not about bit flipping. It's about durability.

                Comment


                • Originally posted by stoatwblr View Post
                  You wouldn't say that when your BTRFS or MDRAID-6 setup just utterly trashed itself.

                  I benchmarked things - and then I started deliberately breaking things. ZFS survives incidents which would have you reaching for the backups in any other circumstance. On top of that it's quite fast when laid out correctly.

                  It's not about bit flipping. It's about durability.
                  I see that you have some pretty strong feelings about this. My point was I would't use RAID in the first place (and I've read reports of people's ZFS setups failing too).

                  I shy away from RAID, etc, in home-usage scenarios and even in some smaller work environments. I've seen a few cases where people screw up things (lose data) not understanding RAID (thinking it's a backup solution) or replacing the wrong disk (on a non-server style enclosure with the blinking lights). For the home user, the data doesn't change much and they are better off with two (or more) separate disks that do a rsync (or robocopy) every 24 hours or so. This provides a backup (something many forget) and has a much lower level of complexity. The through-put of a single disk is usually enough to saturate gigabit Ethernet, so performance is generally not a concern. The odds of moving the disks to another system or OS in case a motherboard fails is also much greater (especially versus a "fake-raid" setup). You're not going to temporarily move your sexy ZFS setup to your Linux or Windows box if you have a hardware failure, whereas you can mount a disk or two.

                  RAID, ZFS and other enterprise solutions have their place, to be sure, but that place isn't your home media server unless you really know what you are doing (and even then I think people do it for bragging rights or to increase their "geek creed".) RAID is a high-availability solution and that's a solution looking for a problem in most homes. If setting up an array is how someone wants to spend their time, I'm OK with that, but if you want simple, quick to implement and reliable, it's better to avoid complexity. The only thing you're loosing is HA (and a lot of headaches).

                  Comment


                  • Originally posted by ldillon View Post
                    That's just it -- <SNIP>
                    Please don't necro in this thread, let it die already, you're posting long after the discussion died, not cool to revive it again.
                    There was a separate thread started anyway, + there's other active (or more recent) related threads nowadays, just search the forum.
                    Last edited by jalyst; 10-01-2012, 01:50 AM.

                    Comment


                    • Originally posted by Michael View Post
                      Most desktop users though that get excited about hearing about ZFS + Linux possibilities are running such single drive setups though, so this testing is aimed at them (like most Phoronix articles towards desktop users), and not those enterprise installations.
                      I'd be surprised if that were the case at all.

                      Personally I run a FreeBSD based SAN and I'd prefered ZFS on Linux which is why the stories are mildly interesting, not because I'm also a Linux desktop user.

                      Comment


                      • Originally posted by ldillon
                        I see that you have some pretty strong feelings about this. My point was I would't use RAID in the first place (and I've read reports of people's ZFS setups failing too).
                        Everything fails, the advantages of technologies is how they handle failure. If you're running a ZRAID and two disks drop, well you're screwed. The same goes with a RAID|2 with 3 disks dropping. I doubt its been failing much beyond user error though, but I accept its not outwith the realms of possibilities.

                        Originally posted by ldillon
                        I shy away from RAID, etc, in home-usage scenarios and even in some smaller work environments. I've seen a few cases where people screw up things (lose data) not understanding RAID (thinking it's a backup solution) or replacing the wrong disk (on a non-server style enclosure with the blinking lights). For the home user, the data doesn't change much and they are better off with two (or more) separate disks that do a rsync (or robocopy) every 24 hours or so. This provides a backup (something many forget) and has a much lower level of complexity. The through-put of a single disk is usually enough to saturate gigabit Ethernet, so performance is generally not a concern. The odds of moving the disks to another system or OS in case a motherboard fails is also much greater (especially versus a "fake-raid" setup). You're not going to temporarily move your sexy ZFS setup to your Linux or Windows box if you have a hardware failure, whereas you can mount a disk or two.
                        You're correct in the sense that RAID is not a replacement for backup. Critically RAID doesn't protect against catastrophic failure such as a fire, massive power failure, or hardware fault killing numerour drives. The problem is, neither does your solution. What are you going to do with your "copy data every 24 hours backup solution" when your single disk fails fairly regularly and you keep eating up to 24 hours in lost productivity? Using raid is about minimising your exposure to this, not completely negating it. Its about spending a bit more money and time to save a lot more money and time in the long run with disk failures. Similarly, what are you going to do when a power surge takes out all the drives in your device at once? Where are your backups then? The problem with statements like "RAID is not a backup", is you should have a fair enough of what is a backup, should not suggest something that isn't a viable backup and you certainly should not suggest something that is inferior to any decent RAID solution.

                        Also FreeBSD is free, thus importing your data to another system should be pretty easy assuming you know how and you have the capacity to do so. Even if you don't have another server, and you can't power the server you do have down (this would be fairly stupid by the way), a virtual machine will see you though. Of course, even if you don't know how, the disaster recovery plan that any business should have, and you should have been testing periodically, will of course see you through.

                        Originally posted by ldillon
                        RAID, ZFS and other enterprise solutions have their place, to be sure, but that place isn't your home media server unless you really know what you are doing (and even then I think people do it for bragging rights or to increase their "geek creed".) RAID is a high-availability solution and that's a solution looking for a problem in most homes. If setting up an array is how someone wants to spend their time, I'm OK with that, but if you want simple, quick to implement and reliable, it's better to avoid complexity. The only thing you're loosing is HA (and a lot of headaches).
                        I actually run RAIDZ2 at home with cheap SATA drives. I don't do it because of geek cred, I do it because when a disk failed the other day, all I had to do was bend down, pull the bad disk*, push in a replacement, and add it to the pool. All without any downtime. Course, thats not my backup for important data if the house goes on fire, but its convenient nonetheless.

                        * actually I had the same problem you mentiond above. My n40l doesn't have any blinking lights to tell me the disk failed, so I didn't actually know which one to pull. You know what I did? I read the manual.

                        Comment


                        • Everything is a tradeoff

                          ownagefool, (I'm not going to quote for brevity's sake)

                          (When I write "ZFS" below, I mean "any RAID solution." I'm not picking on ZFS.)

                          My comments weren't geared toward someone with your advanced technical skills so much as the average home user. I don't think the "average home user" has 24 hours of productivity in the average day. They might have, on average, a couple of new files that are important, now that most people are using web-based email.

                          No, having the second disk in the same PC isn't the idea situation. It's the cost-effective solution. You could put a second disk in a second PC, or in another building, etc, but I contend that verses the hassles, potential pitfalls, cost and complexity of other solutions, it's a good tradeoff for most home users. (unless you are a consultant that gets paid by the hour, then the more complex the better </kidding>) Less prone to problems than an external drive and a much better than nothing, which is what most people have.

                          The point about "swapability" was that it doesn't take a re-install to just put the second drive in another (Linux or Windows) PC that a regular user is more likely to have around. Most people have more than one PC these days, but they are in use. Sure, you could reinstall the ZFS server if you have spare, unused hardware, but most home users don't and couldn't figure it out. But they could probably manage to move a drive (or I could talk them through it over the phone).

                          "What are you going to do when a power surge takes out all the drives in your device at once?"
                          So you are assuming that the home user with the ZFS setup doesn't back up to disk and has a tape backup (or something)?
                          If we are comparing apples to apples, the ZFS guy has a spare disk(s) that he backs up to and that gets take out in your power surge, along with the ZFS array. (As a side bet, I'd wager that half the people running ZFS at home don't have any backup of the array at all because ZFS is so uber-reliable) In an ideal world we'd all do off-site backups but the average home user shouldn't be expect to do that. Sure, your could back up "to the cloud" as a secondary backup, but that (apples to apples) would work with the two disk setup or the ZFS setup.

                          Have we talked about cost yet? If the use can only afford two disks, what solution would you suggest for the same amount of money?

                          Comment


                          • Originally posted by ldillon View Post
                            ownagefool, (I'm not going to quote for brevity's sake)

                            (When I write "ZFS" below, I mean "any RAID solution." I'm not picking on ZFS.)

                            My comments weren't geared toward someone with your advanced technical skills so much as the average home user. I don't think the "average home user" has 24 hours of productivity in the average day. They might have, on average, a couple of new files that are important, now that most people are using web-based email.
                            Idillon, I mean no disrespect, however you extended your comments to "some small businesses" and made some incorrect statements. You may actually know the difference, or at least know enough to learn more when you need to know, but other people reading what you said may get the wrong idea, so I felt it appropriate to put them correct or at least get you to put them in context. If you were talking purely about home users, I may not have even bothered.

                            Originally posted by ldillon View Post
                            No, having the second disk in the same PC isn't the idea situation. It's the cost-effective solution. You could put a second disk in a second PC, or in another building, etc, but I contend that verses the hassles, potential pitfalls, cost and complexity of other solutions, it's a good tradeoff for most home users. (unless you are a consultant that gets paid by the hour, then the more complex the better </kidding>) Less prone to problems than an external drive and a much better than nothing, which is what most people have.
                            Its only cost-effective in the sense that if costs less and doesn't effectively protect your data. If your data is important to you, then you need a proper backup solution. As a purely home user, its likely your data may not be important at all, then of course flying by the seat of your pants is completely OK. If the home user maintains their data is important when you do not believe is really is, quote them the real cost of a proper solution. If they back off, thats fine. Their data wasn't that important afterall.

                            Originally posted by ldillon View Post
                            "What are you going to do when a power surge takes out all the drives in your device at once?"
                            So you are assuming that the home user with the ZFS setup doesn't back up to disk and has a tape backup (or something)?
                            If their data is really worth backing up, then its worth doing correctly. If its not really worth backing up, you can do whatever you like to make them feel warm and fuzzy, but if they actually NEED their data, you need to go with full disclosure.

                            Originally posted by ldillon View Post
                            If we are comparing apples to apples, the ZFS guy has a spare disk(s) that he backs up to and that gets take out in your power surge, along with the ZFS array. (As a side bet, I'd wager that half the people running ZFS at home don't have any backup of the array at all because ZFS is so uber-reliable) In an ideal world we'd all do off-site backups but the average home user shouldn't be expect to do that. Sure, your could back up "to the cloud" as a secondary backup, but that (apples to apples) would work with the two disk setup or the ZFS setup.
                            Yep, you're 100% correct. A ZFS array is NOT a backup either. If you're not running a backup and your data is important, then you deserve a face palm. The more important, the bigger the face palm.

                            Also, full disclosure, I don't actually backup 99% of my ZFS based NAS. My files, outwith my code (I work at home and thats backed up) none of my files are really important. I have the NAS mostly as a home server for convenience, and being mostly trust worthy is good enough for me (when I lose all my files, it'll be a bummer, but its not like I'll lose any money or productivity).

                            Originally posted by ldillon View Post
                            Have we talked about cost yet? If the use can only afford two disks, what solution would you suggest for the same amount of money?
                            1. If their data isn't important (and lets be honest, most peoples data isn't important) but their uptime is then I'd just leave them with the straight up RAID1 and make them a restore disk at a push.
                            2. If teir data is important, but their uptime less so, then something like bacula (or the cloud if you're into that sorta thing) to do offsite backups (assuming they have a decent net connection).
                            3. If both are important, then combine the first two options.
                            4. If neither is important, I'd do nothing (maybe a restore disk, this at least will stop them calling you when their disk eats itself).

                            If you're taking it seriously, you're gonna want to run some checks to make sure the RAID array is clean and that it'll email errors (and label the disks!). You'll also want to make sure those backups and working. You can do that on the backup server, but you'll probably want to do a bit of work to check whether the box is up because you dont want it screaming about no backups on a PC that hasn't been turned on in 3 days. As always, the amount of effort depends on the value of the data.


                            On that point I started my career at a webhosting provider on the phones no less. Obviously we dealt with servers falling over on occasion. You'd get people calling up (on the cheapest package no less) screaming down the phone they were losing $20k per hour because their site was down. The answer was generally always the same, "why on earth are you paying for a best efforts shared pacakge at $20 a year when you're losing several times more of the cost of a proper solution".
                            Last edited by ownagefool; 10-01-2012, 09:12 PM.

                            Comment

                            Working...
                            X