Announcement

Collapse
No announcement yet.

FreeBSD 8.0 vs. Ubuntu 9.10 Benchmarks

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #46
    Originally posted by energyman View Post
    blabla, and there they are, the FreeBSD apologists. Always claiming FreeBSD is faster.
    Few are claiming FreeBSD is faster. Most of the comments I see here claim the test was poorly done (and fundamentally unfair.)

    GCC 4.4 includes support for processor features like SSSE3, SSE4.1, SSE4.2, and more that are not present in GCC 4.2 (which is used by FreeBSD.)
    Due to licensing issues, the newer GCC can't be used in the base OS.
    The performer of this benchmark didn't say they compiled everything with GCC 4.4 for a apples to apples test.

    The documentation for compiling things with GCC 4.4 are here:
    http://www.freebsd.org/doc/en/articl...c/article.html

    If you want an honest test where it is proved Ubuntu is faster, you should perform a fair test. If you wish to "just show FreeBSD slower" that can be done by running the benchmark in an unfair way. Like was done by this test.

    Comment


    • #47
      Originally posted by kraftman View Post
      They benchmarked distro using Ext3 in writeback mode and distro using Ext4 in ordered (or some other). Distro using Ext3 and writeback was faster in *SQL and distro using Ext4 and ordered was slower.
      Yes that is clear. One thing that keeps come up again and again is that there is an assumption that most users will have sufficient knowledge or care enough to switch between filesystems or tweak between options.


      In the same way, most Linux-heads abstract XP vs Vista vs Win7 as "fast", "dog-slow", "mostly better", they don't go down into that level of details. For a Linux head they would look at the top level "is it faster" and go with XP or wait for Win7 if they _had_ to use a Microsoft OS.

      This is no different. If someone is looking at writing an application using something like SQLite, they generally either won't have complete access to the system (hosted server or end-user system). Consequently their view of supporting different OSes will be based on a high level
      "yeah, Ubuntu-9.04 is faster, I'd recommend you my app on this",
      rather than

      "yeah, if you reconfigure the filesystem by modifying the kernel options with this string and changing your fstab entries, you will get good performance with Ubuntu 8-10."


      If you have ever supported software, you know that you will *always* go for the first comment and push people away from 8-10. It is all about domain's of expertise and variables that you can control. Most people simply won't get down to modifying those options on a system. Their view outside _their_ domain of expertise is mostly a black box.

      The key driver for Michael's performance reviews and comparisons is to drive awareness of system differences. It is bare and simple facts. It is the upstream maintainers who are making decisions on behalf of users. If they have done their jobs right, they are balancing the experience of the end user, the safety of the data and so on. Some users tune, the majority will not and inherit the maintainers view of how things should work. There are many cases of Michael's testing having beneficial results ultimately for the community when the numbers are groked and a correct decision made.

      My standard response here is that if there is an accessible collection of directions for changing the configuration then you may have a strong argument for changing the testing. People who live and breathe filesystems assume that everybody knows how to tune the filesystem, but those who don't need to have their hands held very tightly.

      Do you know of any accessible guides for tuning filesystems?

      Matt
      Last edited by mtippett; 09-28-2009, 04:08 PM.

      Comment


      • #48
        Originally posted by mtippett View Post
        Do you know of any accessible guides for tuning filesystems?
        For newbies or for you? I don't know any. I don't even do it myself, because defaults are enough for me.

        @Risner

        GCC 4.4 includes support for processor features like SSSE3, SSE4.1, SSE4.2, and more that are not present in GCC 4.2 (which is used by FreeBSD.)
        Aren't some flags have to be specified to use such features? And does opteron supports all of them? :>
        Last edited by kraftman; 09-28-2009, 04:15 PM.

        Comment


        • #49
          Originally posted by kraftman View Post
          For newbies or for you? I don't know any. I don't even do it myself, because defaults are enough for me.
          Accessible was used carefully . For FS tuning, I consider my self as newbie (with a deeper technical understanding).

          The point is that
          "defaults + accessible tuning guide" = argument for "fairer" comparison
          where as
          "defaults + tuning as a black art" ~~ "defaults"
          for the majority of users.

          I don't completely buy into "fairer" because "fairness" is a function of level of expertise, effort in configuration, particularly intent for the results . That is why Phoronix for the most part relies on the upstream's "Best Effort" of what they believe is production worthy.

          (To be fair, I updated my reply slightly between our exchanges .

          Comment


          • #50
            Originally posted by kraftman View Post
            Aren't some flags have to be specified to use such features? And does opteron supports all of them? :>
            Yes -O is used to enable them (they are enabled by default if you use the default of O2) and yes AMD support those features:
            http://www.amd.com/us-en/Processors/...l?redir=SWAB01

            Comment


            • #51
              Originally posted by mtippett View Post
              Accessible was used carefully . For FS tuning, I consider my self as newbie (with a deeper technical understanding).

              The point is that
              "defaults + accessible tuning guide" = argument for "fairer" comparison
              where as
              "defaults + tuning as a black art" ~~ "defaults"
              for the majority of users.

              I don't completely buy into "fairer" because "fairness" is a function of level of expertise, effort in configuration, particularly intent for the results . That is why Phoronix for the most part relies on the upstream's "Best Effort" of what they believe is production worthy.

              (To be fair, I updated my reply slightly between our exchanges .
              I believe I've got your point Your examples can be quite relative IMHO, because black art can be something normal for some people However, it will be the best to see final releases comparison, because what we see now isn't something what end user will use yet To make a completely fair test, which many people expect, ignoring what devs consider being the best for end users someone should set both systems to something like this:

              tune them for maximum performance while both remain identically stable same time, data is safe and there are identical apps versions used. However, doing such comparison is probably impossible.

              "defaults + accessible tuning guide" = argument for "fairer" comparison are IMHO:

              Ubuntu: using ext4 in writeback mode, turning off debugging (if enabled),

              FeeBSD: turning off debugging, using same GCC version as Ubuntu.


              @Risner

              Thank you.
              Last edited by kraftman; 09-29-2009, 04:05 AM.

              Comment


              • #52
                FreeBSD 7.2 kernel with and without debugging options

                OK, just to show how big the difference in performance can be between a kernel with and one without the debugging options, I ran a buildworld on FreeBSD 7.2 (not 8.0, because it's the only one I have).

                I am not sure what are the options used in 8.0 RC, but I used:

                options KDB
                options DDB
                options INVARIANTS
                options INVARIANT_SUPPORT
                options WITNESS
                options DEBUG_LOCKS
                options DEBUG_VFS_LOCKS
                options DIAGNOSTIC

                The machine is a Dual Core 2 at 2GHz with 1GB RAM and a 5400RPM SATA HDD.
                The command used: make -j4 buildworld, after a make clean, of course. -j4 means 4 parallel jobs.

                The results are:
                - without debugging options: 32 minutes and 10 seconds
                - with debugging options: 41 minutes and 55 seconds.
                That's about 30% more time for the one with debugging options on.

                Comment


                • #53
                  Originally posted by clau View Post
                  The results are:
                  - without debugging options: 32 minutes and 10 seconds
                  - with debugging options: 41 minutes and 55 seconds.
                  That's about 30% more time for the one with debugging options on.
                  Nice improvement in this. There also some debugging options enabled in Ubuntu, but I wonder if they have meaningful impact on performance.

                  Comment


                  • #54
                    Originally posted by kraftman View Post
                    Nice improvement in this. There also some debugging options enabled in Ubuntu, but I wonder if they have meaningful impact on performance.
                    We'll see soon, when both are stable

                    Comment


                    • #55
                      Originally posted by clau View Post

                      ...

                      The results are:
                      - without debugging options: 32 minutes and 10 seconds
                      - with debugging options: 41 minutes and 55 seconds.
                      That's about 30% more time for the one with debugging options on.
                      Yes, I agree that turning on all those options will have a meaningful impact on performance. Effectively you have given an upper and lower bound of what the performance could be like.

                      The debugability of the system is a sliding scale with performance at the other end, the assertion is that there is debug options placed on the kernel means that FreeBSD-8.0 will site somewhere on that debugability scale. Where it actually sits will take someone with sufficient FreeBSD-Fu to show how to help move from supposition on the options to fact and place us at a quantified point on the scale.

                      Excuse my ASCII art..

                      [Debugability]-------=|=-----------------[Performance]

                      There also exists a scale for file-system integrity, integrity at one end, performance at the other. Most distributions choose to push the default options up towards integrity, but as the openSUSE "filesystem barrier" benchmarking demonstrated, you can easily push it to far and lose a lot of performance.

                      [Data Integrity]-------=|=-----------------[Performance]

                      In almost all of these scales (security, memory efficiency, etc) the intent is to always find a balance.

                      So I agree that there may be some debug options turned on within the development versions, but I strongly feel that the FreeBSD maintainers would still ensure the slider is at a balanced mid-way point, rather than the debugability at all costs - which appears to be your benchmark numbers.

                      Anyone with the sufficient FreeBSD-8.0-fu to help?

                      Matt

                      Comment


                      • #56
                        Originally posted by Apopas View Post
                        We'll see soon, when both are stable
                        So, there were a bunch of messages (including from myself) about this, without actually checking it. I just installed FreeBSD 8.0RC1 in Qemu, and AFAICT the debugging options aren't enabled.
                        Therefore, I don't expect big changes in the RELEASE, unless the benchmarking will be done by using a newer GCC version (8.0 has version 4.2.1, although a newer version can be used from ports).

                        Comment


                        • #57
                          Originally posted by clau View Post
                          So, there were a bunch of messages (including from myself) about this, without actually checking it. I just installed FreeBSD 8.0RC1 in Qemu, and AFAICT the debugging options aren't enabled.
                          Therefore, I don't expect big changes in the RELEASE, unless the benchmarking will be done by using a newer GCC version (8.0 has version 4.2.1, although a newer version can be used from ports).
                          Thanks, getting close... FreeBSD-Fu is strong in this one. Anyone care to confirm and expand on clau's research?

                          Again, just like Solaris, availability of Sun Studio which is faster doesn't change the fact that Solaris by default will build 32 bit binaries with a slower version of gcc. For most users they won't know what is happening except for Solaris is feeling slow.

                          I also stumbled across

                          http://forums.freebsd.org/showthread.php?p=43021

                          even the FreeBSD community doesn't *really* know what is turned on.

                          Which goes back to one of my other comments that I frequently make. If there isn't an accessible (well written/easy to follow) guide about how to change things for the better, the black art of tuning is held with specialist or the maintainers.

                          It is the maintainers who are making the tuning decisions, consequently the benchmark approach of "out of the box" is still valid and still valuable.

                          Regards,

                          Matthew

                          Comment


                          • #58
                            Originally posted by mtippett View Post
                            So I agree that there may be some debug options turned on within the development versions, but I strongly feel that the FreeBSD maintainers would still ensure the slider is at a balanced mid-way point, rather than the debugability at all costs - which appears to be your benchmark numbers.
                            Prerelease versions are not for production use, therefore the performance is less important, while finding stability issues is much more important.
                            Besides, a FreeBSD developer can always compile a kernel with whatever options he needs, whenever he needs it.

                            Comment


                            • #59
                              Oh, but there is alot of FreeBSD documentation. One just needs to read. For instance:

                              The Handbook:
                              http://www.freebsd.org/doc/en/books/handbook/

                              Configuration and Tuning (from Handbook):
                              http://www.freebsd.org/doc/en/books/...ig-tuning.html

                              Kernel Debugging (from Handbook):
                              http://www.freebsd.org/doc/en/books/...rneldebug.html

                              Developer's handbook:
                              http://www.freebsd.org/doc/en_US.ISO...pers-handbook/

                              The thing is I didn't know when exactly the debugging options are removed from the default kernel, in in RC or in RELEASE versions.

                              Nevertheless, I suppose this subject is closed, since 8.0RC1 seems to have the debugging options disabled.

                              Comment


                              • #60
                                Originally posted by clau View Post
                                Prerelease versions are not for production use, therefore the performance is less important, while finding stability issues is much more important.
                                Besides, a FreeBSD developer can always compile a kernel with whatever options he needs, whenever he needs it.
                                Yes, I agree. The slider moves around throughout the development cycle.

                                BTW, slashdot has FreeBSD-FU..

                                http://linux.slashdot.org/comments.p...d&pid=29564961

                                In particular

                                Originally posted by slashdot-OMG
                                Anyone actually familiar with the FreeBSD development and release process would know that a release candidate has a considerable amount of debugging options turned on.

                                On Sep-10, most debugging was disabled: http://lists.freebsd.org/pipermail/s...er/013399.html [freebsd.org]

                                On Sep-17, there was the last commit before 8.0-RC1: http://lists.freebsd.org/pipermail/s...er/013645.html [freebsd.org]

                                Anyone familiar with the FreeBSD development and release process would know that there are no fixed rules rules when certain stuff happens and there are no sweeping changes like turning off debugging between a late RC and the actual release. (Other debugging stuff like kernel and module symbols are kept for the release.)
                                So let's reset the comments at the start of the thread, and have a discussion based on fact . New thread, or let's talk impact and cause .

                                Regards,

                                Matthew

                                Comment

                                Working...
                                X