Phoronix Test Suite 9.2 Released For Open-Source, Cross-Platform Benchmarking

[coder]

There's no silver bullet here, just an observation.

It's a good point, though. Maybe quarterly releases are too much? Maybe 1-2 releases per year, with only compatibility patches, in the interim.

[cjcox]

Sorry about the name thing (I meant Michael!!!).... yes. I understand that openbenchmarking knows what it can compare, it's just that it can't compare most of the time because things are always changing. So, lots of data sort of trapped in time (non-comparable). Sure, you can preserve a PTS version, but sometimes the changes are required to get things to run on "newer" Linux, etc. Again, there's no great answer for this... unless you're saying that PTS old versions are supported for a long time (ensured to work as platforms become more modern). I would think "no" though.

[Michael]

First and foremost, love Matt and Phoronix, doing great work! (everyone should send money!)

OpenBenchmarking and PTS are excellent. However, the problem is with "benchmarking". Benchmarking is great as it allows you to compare "things". The problem is that PTS moves and moves rapidly invalidating prior run benchmarks as now the test suite itself becomes the "apple" vs. "orange" problem. Anyway, just one of those frustrating things.

Want to see the difference between today's "X" vs. yesterday's "Y"? Due to the ever changing PTS, this isn't easy unless you happen to own both "X"" and "Y" or you find you can still get the older versions to run on today's "X" (arguably a problem as well as PTS is often just following the rapid pace of development of an underlying test).

So... awesome, but somtimes frustrating. Works best when I'm comparing things I own where I can create unified "apples" to "apples" testing (though painful when you have to run a long test again and again and again as it keeps moving forward).

There's no silver bullet here, just an observation.

My name is Michael. The Phoronix Test Suite tests themselves are versioned separate (not to mention distributed separately as well as long as you have an Internet connection) from the Phoronix Test Suite and supports version binding of said tests by default as part of test suites and result files, etc for ensuring only tests that are comparable versions are actually compared.. All of the PTS changes are generally done to the non-testing / execution code path...

[cjcox]

First and foremost, love Matt and Phoronix, doing great work! (everyone should send money!)

OpenBenchmarking and PTS are excellent. However, the problem is with "benchmarking". Benchmarking is great as it allows you to compare "things". The problem is that PTS moves and moves rapidly invalidating prior run benchmarks as now the test suite itself becomes the "apple" vs. "orange" problem. Anyway, just one of those frustrating things.

Want to see the difference between today's "X" vs. yesterday's "Y"? Due to the ever changing PTS, this isn't easy unless you happen to own both "X"" and "Y" or you find you can still get the older versions to run on today's "X" (arguably a problem as well as PTS is often just following the rapid pace of development of an underlying test).

So... awesome, but somtimes frustrating. Works best when I'm comparing things I own where I can create unified "apples" to "apples" testing (though painful when you have to run a long test again and again and again as it keeps moving forward).

There's no silver bullet here, just an observation.

[phoronix]

Phoronix: Phoronix Test Suite 9.2 Released For Open-Source, Cross-Platform Benchmarking

Phoronix Test Suite 9.2-Hurdal is available today as the newest quarterly feature release to the Phoronix Test Suite for automated, cross-platform and open-source benchmarking...

http://www.phoronix.com/scan.php?page=news_item&px=Phoronix-Test-Suite-9.2