Announcement

Collapse
No announcement yet.

Phoronix Test Suite 9.2 Released For Open-Source, Cross-Platform Benchmarking

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Phoronix Test Suite 9.2 Released For Open-Source, Cross-Platform Benchmarking

    Phoronix: Phoronix Test Suite 9.2 Released For Open-Source, Cross-Platform Benchmarking

    Phoronix Test Suite 9.2-Hurdal is available today as the newest quarterly feature release to the Phoronix Test Suite for automated, cross-platform and open-source benchmarking...

    http://www.phoronix.com/scan.php?pag...Test-Suite-9.2

  • #2
    First and foremost, love Matt and Phoronix, doing great work! (everyone should send money!)

    OpenBenchmarking and PTS are excellent. However, the problem is with "benchmarking". Benchmarking is great as it allows you to compare "things". The problem is that PTS moves and moves rapidly invalidating prior run benchmarks as now the test suite itself becomes the "apple" vs. "orange" problem. Anyway, just one of those frustrating things.

    Want to see the difference between today's "X" vs. yesterday's "Y"? Due to the ever changing PTS, this isn't easy unless you happen to own both "X"" and "Y" or you find you can still get the older versions to run on today's "X" (arguably a problem as well as PTS is often just following the rapid pace of development of an underlying test).

    So... awesome, but somtimes frustrating. Works best when I'm comparing things I own where I can create unified "apples" to "apples" testing (though painful when you have to run a long test again and again and again as it keeps moving forward).

    There's no silver bullet here, just an observation.

    Comment


    • #3
      Originally posted by cjcox View Post
      First and foremost, love Matt and Phoronix, doing great work! (everyone should send money!)

      OpenBenchmarking and PTS are excellent. However, the problem is with "benchmarking". Benchmarking is great as it allows you to compare "things". The problem is that PTS moves and moves rapidly invalidating prior run benchmarks as now the test suite itself becomes the "apple" vs. "orange" problem. Anyway, just one of those frustrating things.

      Want to see the difference between today's "X" vs. yesterday's "Y"? Due to the ever changing PTS, this isn't easy unless you happen to own both "X"" and "Y" or you find you can still get the older versions to run on today's "X" (arguably a problem as well as PTS is often just following the rapid pace of development of an underlying test).

      So... awesome, but somtimes frustrating. Works best when I'm comparing things I own where I can create unified "apples" to "apples" testing (though painful when you have to run a long test again and again and again as it keeps moving forward).

      There's no silver bullet here, just an observation.
      My name is Michael. The Phoronix Test Suite tests themselves are versioned separate (not to mention distributed separately as well as long as you have an Internet connection) from the Phoronix Test Suite and supports version binding of said tests by default as part of test suites and result files, etc for ensuring only tests that are comparable versions are actually compared.. All of the PTS changes are generally done to the non-testing / execution code path...
      Michael Larabel
      http://www.michaellarabel.com/

      Comment


      • #4
        Sorry about the name thing (I meant Michael!!!).... yes. I understand that openbenchmarking knows what it can compare, it's just that it can't compare most of the time because things are always changing. So, lots of data sort of trapped in time (non-comparable). Sure, you can preserve a PTS version, but sometimes the changes are required to get things to run on "newer" Linux, etc. Again, there's no great answer for this... unless you're saying that PTS old versions are supported for a long time (ensured to work as platforms become more modern). I would think "no" though.

        Comment


        • #5
          Originally posted by cjcox View Post
          There's no silver bullet here, just an observation.
          It's a good point, though. Maybe quarterly releases are too much? Maybe 1-2 releases per year, with only compatibility patches, in the interim.

          Comment


          • #6
            Originally posted by coder View Post
            It's a good point, though. Maybe quarterly releases are too much? Maybe 1-2 releases per year, with only compatibility patches, in the interim.
            Has anyone seen any difference in performance between PTS releases? I have not. I would suspect the only cases where PTS version differences would impact the performance would be if the system has less than 1GB of RAM or so under very heavy memory pressure. Most code changes / feature additions these days tend to be just on hardware/software detection or in result analysis after testing is complete, none of which obviously happen while test execution is happening.
            Michael Larabel
            http://www.michaellarabel.com/

            Comment


            • #7
              Again, not a performance issue, just lack of ability to "compare" with a vast db (unless you're comparing to something very very recent). Again, maybe I'm the only one that finds it interesting (?).

              Comment


              • #8
                Originally posted by cjcox View Post
                Again, not a performance issue, just lack of ability to "compare" with a vast db (unless you're comparing to something very very recent). Again, maybe I'm the only one that finds it interesting (?).
                The test profile versions are all versioned independently and running any of them will lock it in the metadata. And multiple versions of a given test can be installed on the same system. If you find some random test on OpenBenchmarking.org and you go 'phoronix-test-suite benchmark the-ob-id', it will fetch the same exact test profile versions as used in that original test case. So I am not sure I understand what you are saying about 'comparing to something very very recent'

                All test profiles are developed independent of PTS test releases (sans the rare case where a test depends upon a new PTS feature in which case it specifies minimum version requirements) https://github.com/phoronix-test-suite/test-profiles
                Michael Larabel
                http://www.michaellarabel.com/

                Comment


                • #9
                  If I want to compare with A, I have to run test version X. All of A, tests version X,Y.Z. Want to compare with B, I have run test version G. All of B, G,H,Z. Want to compare with C, I have to run test version Q, etc... etc...

                  I suppose it depends on perspective if this is a problem or not.

                  Edit: My point is that chances are, if I run a set of tests, it matches with nothing to compare with in the db (80-90+% of the time?). You almost have to "target" specifically what you want to test against.
                  Last edited by cjcox; 12-03-2019, 08:37 PM.

                  Comment


                  • #10
                    Originally posted by cjcox View Post
                    If I want to compare with A, I have to run test version X. All of A, tests version X,Y.Z. Want to compare with B, I have run test version G. All of B, G,H,Z. Want to compare with C, I have to run test version Q, etc... etc...

                    I suppose it depends on perspective if this is a problem or not.

                    Edit: My point is that chances are, if I run a set of tests, it matches with nothing to compare with in the db (80-90+% of the time?). You almost have to "target" specifically what you want to test against.
                    Ahhh so you are referring to it from that perspective, which is separate from PTS releases/versions. Yes, for simple end-users wanting to compare their performance to others on a web, that can be a pain/challenge particularly due to the vast spectrum of tests, versions of those tests, and all of the test options. Many different possible combinations. For that perspective there isn't a easy solution for an open-source solution that seeks by nature to offer a diverse collection of tests beyond working more on 'recommended' test suites (set of test profiles, test suites can version lock to specific test profile versions) and encouraging users to run said suites.
                    Michael Larabel
                    http://www.michaellarabel.com/

                    Comment

                    Working...
                    X