Announcement

Collapse
No announcement yet.

Defining The Phoronix Test Suite

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Defining The Phoronix Test Suite

    Phoronix: Defining The Phoronix Test Suite

    It's going on four years that we have been providing benchmarks under Linux at Phoronix, and through this time the process has evolved as more applications have come about that gauge Linux system performance and meet our criteria for use in hardware reviews. While our internal process is continually refined, we have received requests for creating a Linux benchmarking manifest so that interested parties are able to easily reproduce our tests for comparative purposes and know everything that's involved. With that said, we are now working to establish the Phoronix Test Suite that will consist both of detailed documentation defining our hardware benchmarking protocols and free software that automates much of this work.

    http://www.phoronix.com/vr.php?view=11813

  • #2
    really, really great idea. i'm going to think about several components for such a benchmark, as soon as i've such results, i'm going to write them here

    Comment


    • #3
      what licence will this be - gpl?

      Comment


      • #4
        Originally posted by hmmm View Post
        what licence will this be - gpl?

        GPLv3 likely (chance it could be MIT/X11 but most likely GPL)
        Michael Larabel
        http://www.michaellarabel.com/

        Comment


        • #5
          Way to go, this is a great idea. I'm sure this will go far to create a standard benchmarking tool for Linux.

          Comment


          • #6
            Are there any free games in there? OpenArena, Cube2, Nexuiz (shameless plug)? What about e.g. oggenc?

            Comment


            • #7
              oggenc is in there.

              SavageX: Last I checked out the Nexuiz benchmarking capabilities, I wasn't too impressed to be honest. Have they improved as of late? Last time I looked into it was about a year ago. If you're able to shed anymore light on it I'd be more than willing to investigate it further.
              Michael Larabel
              http://www.michaellarabel.com/

              Comment


              • #8
                Originally posted by Michael View Post
                SavageX: Last I checked out the Nexuiz benchmarking capabilities, I wasn't too impressed to be honest. Have they improved as of late? Last time I looked into it was about a year ago. If you're able to shed anymore light on it I'd be more than willing to investigate it further.
                As basically all Quake-engine based games it has the nice timedemo command. Try e.g. "timedemo demos/demo1". I think it should even work directly at game startup: "./nexuiz-linux-whatever-binary +timedemo demos/demo1"

                Or you can create a benchmark config file (e.g. "benchmark.cfg" in the Nexuiz "data" directory):

                timedemo/demo1
                timedemo/demo1
                timedemo/demo1
                timedemo/demo2
                timedemo/demo2
                timedemo/demo2

                and do a "./nexuiz-linux-bla-bla +exec benchmark.cfg" to automate testing.

                Each run will print to console (ingame and to stdout) statistics like min/avg/max framerates.

                We'll release a new version of Nexuiz hopefully later this month (see http://emptyset.endoftheinternet.org...nexuiz/builds/ for SVN builds) which will introduce (optional) rendering features like reflective water with refraction - this will make sure even fast cards are kept busy ;-)

                (Plus we finally have a much nicer menu ;-) )

                Comment


                • #9
                  Where did you find RAMspeed? I can't locate it anywhere, google doesn't find it, nor is it in Debian's repos..

                  Comment


                  • #10
                    Originally posted by curaga View Post
                    Where did you find RAMspeed? I can't locate it anywhere, google doesn't find it, nor is it in Debian's repos..


                    http://www.alasir.com/software/ramspeed/
                    Michael Larabel
                    http://www.michaellarabel.com/

                    Comment


                    • #11
                      Awesome. I've been using Hardinfo for benchmarking, but their benchmarks are a bit weird (author said they'll need to be re-designed - but maybe they could just include yours. Hardinfo also has a central server so you can compare your benchmarks to other setups anonymously).

                      Comment


                      • #12
                        I suggest you a crossplatform test. Not all the benchmark - better if they are -, but at least a part of it.

                        If the same test can be run over windows / Linux / MacOS / and others with a single number score, It would be better.

                        Then the driver development across the time and systems would be better, at least to watch.

                        For example a KDE4 test (it is cross-platform), and some similar as VLC tests for videos, Quake tests for games etc.

                        Opengl test would be a great tool to compare between drivers and systems.

                        We will find that, best sysmark - only windows - results may be not best with Linux or Mac OS.

                        And perhaps in a future some systems under Linux score high than under windows with driver improvements, especially for games.

                        I should put office tests with openoffice, browser tests with Firefox and opera, and of course on that tests Linux and MacOS should win windows, HD and RAM ones probably too. The lack are graphic drivers, and ATI and NVIDIA are working well now opening their drivers, but it is not enough.

                        I wish this benchmark become a world reference one for non directx benchmarks for every OS.
                        I would like to see your scores at any other computer analitics web page, as anandtech, tom's hardware, extremetech, 4gamers and similars.
                        To address this question of multi-platform compatibility, the Phoronix Test Suite software is compatible with Solaris and I'd anticipate it even working on the *BSDs.

                        However, the Phoronix Test Suite specification will be specifically designed and tested for Linux. Sure, you can apply the same testing practices to Windows or any other OS for that matter, but the chosen benchmarks and accompanying documentation will be all geared for Linux testing.
                        Michael Larabel
                        http://www.michaellarabel.com/

                        Comment


                        • #13
                          OpenGl benchmakr

                          Hi Michael,

                          i got a new 8800gt this week and i needed some benchmark to compare it with my 7800gt. And i found a wonderfu OpenGl Benchmark (very gpu heavy, shader heavy).

                          Its called "Fur" from ozone3d and its freeware:

                          http://www.ozone3d.net/benchmarks/fur/

                          Its made for windows, but it runs 100% and with 0%, and i really mean 0% performance loss in wine-0.9.55.

                          My windows score: 2339
                          My wine score: 2344

                          I think this benchmark would be great for the testsuite.

                          What do you think ?

                          Best regards,
                          Christian

                          Comment


                          • #14
                            Originally posted by christian_frank View Post
                            Hi Michael,

                            i got a new 8800gt this week and i needed some benchmark to compare it with my 7800gt. And i found a wonderfu OpenGl Benchmark (very gpu heavy, shader heavy).

                            Its called "Fur" from ozone3d and its freeware:

                            http://www.ozone3d.net/benchmarks/fur/

                            Its made for windows, but it runs 100% and with 0%, and i really mean 0% performance loss in wine-0.9.55.

                            My windows score: 2339
                            My wine score: 2344

                            I think this benchmark would be great for the testsuite.

                            What do you think ?

                            Best regards,
                            Christian
                            Hi Christian,

                            Thanks for the link regarding Fur. However, at this point no WINE-dependent benchmarks will be included with the Phoronix Test Suite specification itself. It could be supported by the PTS software though. Right now with new WINE releases coming out every two weeks, it's quickly changing and really wouldn't present itself as a good candidate in everyday benchmarking until it settles down in the future.

                            Then relying upon a Windows test in a Linux test suite would also need to be thought about and debated.
                            Michael Larabel
                            http://www.michaellarabel.com/

                            Comment


                            • #15
                              hm, perhaps sauerbraten2 is the way to go? http://en.wikipedia.org/wiki/Sauerbr...8video_game%29
                              at least for graphic tests/benchmarks, this game seems to be a good choice, it uses actual techniques such as hdr-rendering etc.
                              the licence is also gpl-compatible. don't know if they do have a benchmark or a timedemo built-in, but you could ask the devs. it is written using sdl, so it is platform independent. have a look at it

                              another idea is to use superpi. this benchmark is really popular all around the world^^. of course it is primary for cpus, but better than nothing. you could calculate the first 20mio decimals with it and measure the time..

                              Comment

                              Working...
                              X