Announcement

Collapse
No announcement yet.

Fedora, Debian, FreeBSD, OpenBSD, OpenSolaris Benchmarks

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Fedora, Debian, FreeBSD, OpenBSD, OpenSolaris Benchmarks

    Phoronix: Fedora, Debian, FreeBSD, OpenBSD, OpenSolaris Benchmarks

    Last week we published the first Debian GNU/kFreeBSD benchmarks that compared the 32-bit and 64-bit performance of this Debian port -- that straps the FreeBSD kernel underneath a Debian GNU user-land -- to Debian GNU/Linux. We have now extended that comparison to put many other operating systems in a direct performance comparison to these Debian GNU/Linux and Debian GNU/kFreeBSD snapshots of 6.0 Squeeze to Fedora 12, FreeBSD 7.2, FreeBSD 8.0, OpenBSD 4.6, and OpenSolaris 2009.06.

    http://www.phoronix.com/vr.php?view=14533

  • indezign
    replied
    Well after a long period of time, we now have FreeBSD 8.1 RC2 with high-performance ZFS. I wonder how it competes with Fedora 13 using EXT4?

    Leave a comment:


  • kraftman
    replied
    Originally posted by kebabbert View Post
    I agree that to be able to draw a full conclusion, we need to know more details and debug the SAP system and see exactly why Linux has problems with cpu utilization on almost identical hardware.
    It's not almost identical hardware and software is different too.

    No one objects when Linux is faster ("ah, such a good and fair benchmark".
    Another lie.

    But when Linux is slower on, e.g., SAP benhcmarks, there are people questioning the SAP benchmark.
    A sane person can't say Linux scales worse basing on SAP papers you posted and this is one of the reasons you're an idiot to me.

    ("no, this can not be right, something is wrong with the benchmark")
    I see there's really something wrong with you.

    I think this is not really fair, but hey, since when has Linux supporters been fair? They compare 800 MHz SPARC to 2.4GHz Intel Core Duo and thinks that is fair. If I question that, they insist it is fair. Only one year later, they MAY admit it was not fair. But maybe not.
    This is exactly what you're doing here. You're showing two different papers with two different servers and you say Solaris scales better and you don't want to admit it's not fair. You're also accusing others exactly what you're doing.

    Linus lags behind - demand a full investigation with full debug information and dont accept this benchmark. "Something must be wrong, I refuse to accept this stupid stupid stupid SAP benchmark."
    This "stupid stupid stupid SAP benchmark" didn't benchmark Linux and Solaris, but two different servers. Phoronix benchmarks systems using their stock configurations (usually) and this is quite fair. Of course you can install newer packages, tweak some things etc. and results will be different. You will accept defaults are tested or you can cry like a baby.

    I still think Linux supporters are not fair, but that is the way life is: not fair. Nothing to do, just accept it.
    It's you who's not fair. You gave Bonwick (a SUN developer) PR talk to backup your claims, two different SAP papers which you treat like an OS benchmark etc.

    You said in one of your comments something like: Linux is known it doesn't scale well. I can say Solaris is known of being slow thus the name Slowlaris.

    Leave a comment:


  • kraftman
    replied
    Originally posted by kebabbert View Post
    You claim that I FUD. But everything I claim, is backed up by relevant links and certified benchmarks and white papers in an academic way - then it is not FUD. On the other hand, YOU, have not provided any relevant backup for your claims. And you claim that I lie, but do not/can not point out my lies.
    I won't even bother replying to your other bull. You're still lying and FUDing, because I pointed you were lying and FUDing and you're saying now you backed up everything you said by "relevant links and certified benchmarks...", but you didn't backup things I pointed and which are lies and FUD. Every time you say "But everything I claim, is backed up by relevant links" is another lie. I consider you didn't provided any relevant backup for your claims and I was repeating this since some time.

    Leave a comment:


  • F.Ultra
    replied
    No, all that I said was that I had no trouble to create software that scaled to 16 cores,
    A little explanation perhaps could be in order. When first reading the SAP benchmarks I didn't notice that they used 8 6-core CPUs so I erroneously saw 16 cores used and not 48, which is why I thought that my experience was relevant.

    Leave a comment:


  • F.Ultra
    replied
    Maybe I misunderstood you, but you claimed that you had no scalability problems on Linux up to 16 cores (4 cpus) for your special parallell workload and therefore you doubt Linux has scalability problems with many more cpus? I thought this was a kind of extrapolating?
    No, all that I said was that I had no trouble to create software that scaled to 16 cores, I did never say anything about Linux with more cpus. My only point was that creating scalable software on Linux or Solaris might be completely different tasks.
    but you dont rule out that Linux can have scalability problems on SAP
    No I don't, we clearly see lower performance from HP, this cannot be denied. It would be interesting however to see the Solaris benchmark retested on the HP hardware with the MaxDB, and also the reverse. Not to prove a point, more that it would be interesting to see what would happen.
    Maybe you should write to Phoronix and question all Linux vs OpenSolaris vs FreeBSD benchmarks? They should stop, because you want Phoronix to debug and explain exactly why one distro is slower?
    Please don't be this silly. It was you who posted SAP benchmarks as proof that Linux does not scale well, I have never posted any benchmark result anywhere claiming that Solaris or any other OS is bad, so please do not go down that road, it's just silly.

    There is zealots from every camp doing just that, but these people are idiots and we do not have to drag ourselves down to their level?

    Leave a comment:


  • F.Ultra
    replied
    I gave the numbers from the SAP benchmark a closer look.

    HP is performing 20.46% worse than Sun. While HP has a CPU with 7.69% higher clockrate compared with Sun the utilization is 12.12% lower, if we combine these two numbers we get 19.81% total lower CPU utilization for HP compared with Sun, which matches the lower performance.

    Also HP is using 19.76% less SAP users than Sun.

    Thus HP is having less CPU utilization due to it performing less number of transactions with a faster CPU so that is why the CPU utilization is lower.

    In essence: HP is feeding their system with roughly 20% less traffic and is getting roughly 20% less performance, this is close to 1:1.

    The question is why HP decided to feed with 20% less users than Sun, it could very well be that Linux on DL785 and SAP scales worse than Sun so that they would get even less performance with 10000 users or it could be any other reason. We simply do not know, we have to ask the HP team why the chose 8022 users instead of 10000.

    This all depends upon how SAP is programmed. Let's say that they use malloc() in the hot code path inside a critical section, here GNU libc is unfortunately not fully optimized which is why we Linuxprogrammers who create software that has to scale avoids doing just that. Solaris probably does not have this particular issue since it uses another libc (the BSDs for example does not have this particular problem). If this is the case it is no wonder that SAP works better on Solaris than on Linux. Calling malloc() in the hot code path is a deadly sin anyways

    Leave a comment:


  • kebabbert
    replied
    Originally posted by F.Ultra View Post
    Yes it is, and that is why nobody here has done that.
    Maybe I misunderstood you, but you claimed that you had no scalability problems on Linux up to 16 cores (4 cpus) for your special parallell workload and therefore you doubt Linux has scalability problems with many more cpus? I thought this was a kind of extrapolating?

    Never mind. We both agree that Solaris is better for SAP for this very setup (and we saw that Solaris easily handled more clients than linux). We dont agree why we see this higher CPU utilization on Solaris, but you dont rule out that Linux can have scalability problems on SAP:
    Originally posted by F.Ultra View Post
    4. SAP is deadlocking semaphores on Linux but not on Solaris for whatever reason.
    I agree we dont know why, maybe the SAP user had a head ache and turned on wrong parameters, or whatever. My interpretation is simply that Solaris lives up to it's old reputation of scaling well, and Linux lives up to it's old reputation of scaling not so well. Occam's razor. But that is MY interpretation, and your interpretation is different.

    I agree that to be able to draw a full conclusion, we need to know more details and debug the SAP system and see exactly why Linux has problems with cpu utilization on almost identical hardware.

    Maybe you should write to Phoronix and question all Linux vs OpenSolaris vs FreeBSD benchmarks? They should stop, because you want Phoronix to debug and explain exactly why one distro is slower? But then, if Phoronix found out the problem, they could just correct the source code so the system runs faster. But this takes good kernel programming skills to do - so it is maybe unfair to request this. But it is not unfair to request this, when Linux lags behind, it seems. No one objects when Linux is faster ("ah, such a good and fair benchmark". But when Linux is slower on, e.g., SAP benhcmarks, there are people questioning the SAP benchmark ("no, this can not be right, something is wrong with the benchmark"). I think this is not really fair, but hey, since when has Linux supporters been fair? They compare 800 MHz SPARC to 2.4GHz Intel Core Duo and thinks that is fair. If I question that, they insist it is fair. Only one year later, they MAY admit it was not fair. But maybe not.

    Linux wins a benchmark - no problem. Everything is fine.

    Linus lags behind - demand a full investigation with full debug information and dont accept this benchmark. "Something must be wrong, I refuse to accept this stupid stupid stupid SAP benchmark."

    I still think Linux supporters are not fair, but that is the way life is: not fair. Nothing to do, just accept it.

    Let us agree that we disagree and conclude this debate.

    Leave a comment:


  • F.Ultra
    replied
    I do not agree on this. I suspect you have not studied complexity theory for parallell computers. There are many problems that are P-complete.
    Of course there are many, I just said that it did not apply to most, and with most I meant the kind of problems that affect most types of software in the IT industry; webbservers, databases, stock exchange trading systems, games etc.
    I think it is wrong to extrapolate from 4 cpus up to hundreds of CPUs
    Yes it is, and that is why nobody here has done that.
    To me, the price is not important.
    And we are not talking about you, we are talking about HP, their PR department and why they chose to enter with the setup they did. What IBM does or does not do is highly irrelevant since we are talking about HP.
    For the benchmark, I suspect it is designed to not be DataBase bound. That would be a bad benchmark. Most probably the speed of the RAM is important, not the DB.
    I agree with you, but that does not mean that the DB cannot affect the outcome of the benchmark. Even if the DB is only used once per 100k transactions in the benchmark it could still impact the numbers in the way that we see.

    Because what we see is that the CPU utilization is not 99%, and this can be because of a number of reasons:

    1. There is not enough incoming requests to max out the server. It could be that the number of SAP clients used is too small, that they have congested the network etc. This would fit nice with the fact that the HP is using CPUs that is 200MHz stronger.

    2. The CPU is waiting for I/O, either there is writes/reads to disk or the DB is holding the CPU back.

    3. HP is displaying the userspace CPU utilization and is not including the kernel CPU utilization.

    4. SAP is deadlocking semaphores on Linux but not on Solaris for whatever reason.

    We simply do not know why, unless we perform the benchmark ourselves and take some significant time to debug it we cannot say why HO got the result they got.

    All I can say is that there are very few things involved with scaling to CPUs within the kernel that can lower the CPU utilization, in 99% of cases where you see low CPU utilization it is either due to being I/O bound or not having enough requests to process.

    Leave a comment:


  • kebabbert
    replied
    Originally posted by F.Ultra View Post
    No I don't disagree at all, what I am trying to tell you is exactly what you wrote yourself in the text that I have quoted. The only thing that your posted benchmarks tells is that SAP runs better on Solaris on a Sunfire than Linux on a HP DL785.
    Ok, glad we can agree on something at last!


    Originally posted by F.Ultra View Post
    If you know what you are doing, most things are easy to parallelize,
    I do not agree on this. I suspect you have not studied complexity theory for parallell computers. There are many problems that are P-complete.


    Originally posted by F.Ultra View Post
    and even if they are not it's not often the kernel that stands in your way. I wrote about 16 cores because that is the biggest hardware that I have needed so far (only have to process a few million TPS), so I had first hand experience with that number.
    I think it is wrong to extrapolate from 4 cpus up to hundreds of CPUs. Microsoft has tried to scale for decades, and still dont succeed.


    Originally posted by F.Ultra View Post
    This is completely besides the point, what I wanted you to notice was that HP chose to enter the benchmark with a much cheaper solution (they have hardware matching and surpassing the Sunfire when it comes to price), and the question is why. HP is the only ones that knows this.
    To me, the price is not important. IBM has very expensive gear. Any modern x86 CPU is 5-10x faster than the fastest Mainframe CPU. You need 16 Nehalem-EX cpus to match the biggest IBM Mainframe with 64 CPUs. Both give 400Mips/cpu, the difference is that Nehalem-EX gives that under software emulation which is a factor 5-10x slower. The biggest mainframe costs maybe 20 million USD? Read about this on wikipedia on article "Herkules Mainframe emulator"

    I have Armani clothes that cost much more than ordinary clothes. Price doesnt tell you everything. I think we should study the components, are they equivalent?

    HP and Sun entered this benchmark with almost identical hardware. HP used slightly faster components, yes.


    Originally posted by F.Ultra View Post
    Even if the DB is not used heavily it can still have an impact on the numbers, especially since we are talking about close to a million TPS. Trust me, I write this kind of software.
    For the benchmark, I suspect it is designed to not be DataBase bound. That would be a bad benchmark. Most probably the speed of the RAM is important, not the DB.

    I have no reason not to trust you, but I can email SAP and ask about the benchmark if you wish. OTOH, maybe you should trust me, as I have double master; one in Comp Sci (specialization: algo theory and discrete math) and another Master in Math (specialization: pure math/financial math). The Math Master includes a B Sc in Math. The Comp Sci Master includes a B Sc in Comp Sci: two B Sc and two Masters. I work at one of the most famous well known financial companies, with extreme demands on our largest systems. Maybe you are using it.

    Leave a comment:

Working...
X