Announcement

Collapse
No announcement yet.

Phoronix Test Suite 9.2 Released For Open-Source, Cross-Platform Benchmarking

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • boxie
    replied
    Originally posted by coder View Post
    What about a set of workstation/HPC tests? I think the scientific & simulation tests don't belong in either the server or desktop category. Server tests should be things like database, file serving, and web serving.

    Probably put code compilation into the workstation category? Or maybe put it in desktop, under the assumption that most workstations would also run the desktop tests?


    OpenCL-based tests should be included in both desktop & workstation categories. Server installs tend to be more minimal, and they tend to lack GPUs.

    Also, instead of "light" and "heavy", what about "quick" and "full"?
    Workstation tests could go into desktop-heavy
    At least 1 ooencl test in -light

    The idea of -light is to quickly run. If I can bench my system and have it done in 20 minutes that's cool. 1 test per subsystem kinda thing

    -heavy gives you a more complete look with multiple tests per subsystem.

    Leave a comment:


  • coder
    replied
    Originally posted by boxie View Post
    Then maybe break the recommended tests up into something like?

    2019-desktop-light
    2019-desktop-heavy
    2019-server-light
    2019-server-heavy

    with differences being stuff like - Desktop suites have games / production tests (darktable opencl etc).
    What about a set of workstation/HPC tests? I think the scientific & simulation tests don't belong in either the server or desktop category. Server tests should be things like database, file serving, and web serving.

    Probably put code compilation into the workstation category? Or maybe put it in desktop, under the assumption that most workstations would also run the desktop tests?

    Originally posted by boxie View Post
    I feel that with a dedicated test suite like this you could also push distros in given direction. e.g. - making sure OpenCL is up and running by default.
    OpenCL-based tests should be included in both desktop & workstation categories. Server installs tend to be more minimal, and they tend to lack GPUs.

    Also, instead of "light" and "heavy", what about "quick" and "full"?

    Leave a comment:


  • boxie
    replied
    Originally posted by Michael View Post

    Thanks, most (or even all, besides UI work) is already in place. But even then it's still somewhat difficult in providing the recommended tests simply due to people running PTS on everything from Raspberry Pi to hundreds of core CPUs. Granted, the use-case for following such feedback will likely fall to the smaller core count / enthusiast/desktop range. WIll do some thinking and play with at least a recommendation from that perspective.

    Right now if running PTS benchmark without any options it shows:
    Then maybe break the recommended tests up into something like?

    2019-desktop-light
    2019-desktop-heavy
    2019-server-light
    2019-server-heavy

    with differences being stuff like - Desktop suites have games / production tests (darktable opencl etc).

    I feel that with a dedicated test suite like this you could also push distros in given direction. e.g. - making sure OpenCL is up and running by default.

    Leave a comment:


  • Michael
    replied
    Originally posted by boxie View Post

    I guess you can drive that a little more. A line at the end of the usage output for a standard <year> comparison test suite. maybe?

    Recommended First Benchmark Runs:
    phoronix-test-suite benchmark pts/standard-long-2019
    phoronix-test-suite benchmark pts/standard-short-2019

    which have a nice selection of appropriate tests (cpu/memory/gpu/primary storage) for us plebs to compare our systems against "all year". (and also to look back and see how good our new shiny rigs are over past years)

    you could then have an openbenchmarking.org page just dedicated to those scores. so we can see various stats

    I know that it would be a bunch of work initially, but I could see you using these things as a standard comparison for systems all year round (potential time saving?)
    Thanks, most (or even all, besides UI work) is already in place. But even then it's still somewhat difficult in providing the recommended tests simply due to people running PTS on everything from Raspberry Pi to hundreds of core CPUs. Granted, the use-case for following such feedback will likely fall to the smaller core count / enthusiast/desktop range. WIll do some thinking and play with at least a recommendation from that perspective.

    Right now if running PTS benchmark without any options it shows:

    ./phoronix-test-suite benchmark


    [PROBLEM] Phoronix Test Suite Argument Missing

    CORRECT SYNTAX:
    phoronix-test-suite benchmark [Test | Suite | OpenBenchmarking ID | Test Result] ...


    Recently Saved Test Results:
    xxxx

    New + Updated Tests:
    libgav1 dav1d build2 blender

    Recent OpenBenchmarking.org Results From This IP:
    xxxx

    Leave a comment:


  • boxie
    replied
    Originally posted by Michael View Post

    Ahhh so you are referring to it from that perspective, which is separate from PTS releases/versions. Yes, for simple end-users wanting to compare their performance to others on a web, that can be a pain/challenge particularly due to the vast spectrum of tests, versions of those tests, and all of the test options. Many different possible combinations. For that perspective there isn't a easy solution for an open-source solution that seeks by nature to offer a diverse collection of tests beyond working more on 'recommended' test suites (set of test profiles, test suites can version lock to specific test profile versions) and encouraging users to run said suites.
    I guess you can drive that a little more. A line at the end of the usage output for a standard <year> comparison test suite. maybe?

    Recommended First Benchmark Runs:
    phoronix-test-suite benchmark pts/standard-long-2019
    phoronix-test-suite benchmark pts/standard-short-2019

    which have a nice selection of appropriate tests (cpu/memory/gpu/primary storage) for us plebs to compare our systems against "all year". (and also to look back and see how good our new shiny rigs are over past years)

    you could then have an openbenchmarking.org page just dedicated to those scores. so we can see various stats

    I know that it would be a bunch of work initially, but I could see you using these things as a standard comparison for systems all year round (potential time saving?)

    Leave a comment:


  • Michael
    replied
    Originally posted by cjcox View Post
    If I want to compare with A, I have to run test version X. All of A, tests version X,Y.Z. Want to compare with B, I have run test version G. All of B, G,H,Z. Want to compare with C, I have to run test version Q, etc... etc...

    I suppose it depends on perspective if this is a problem or not.

    Edit: My point is that chances are, if I run a set of tests, it matches with nothing to compare with in the db (80-90+% of the time?). You almost have to "target" specifically what you want to test against.
    Ahhh so you are referring to it from that perspective, which is separate from PTS releases/versions. Yes, for simple end-users wanting to compare their performance to others on a web, that can be a pain/challenge particularly due to the vast spectrum of tests, versions of those tests, and all of the test options. Many different possible combinations. For that perspective there isn't a easy solution for an open-source solution that seeks by nature to offer a diverse collection of tests beyond working more on 'recommended' test suites (set of test profiles, test suites can version lock to specific test profile versions) and encouraging users to run said suites.

    Leave a comment:


  • cjcox
    replied
    If I want to compare with A, I have to run test version X. All of A, tests version X,Y.Z. Want to compare with B, I have run test version G. All of B, G,H,Z. Want to compare with C, I have to run test version Q, etc... etc...

    I suppose it depends on perspective if this is a problem or not.

    Edit: My point is that chances are, if I run a set of tests, it matches with nothing to compare with in the db (80-90+% of the time?). You almost have to "target" specifically what you want to test against.
    Last edited by cjcox; 12-03-2019, 08:37 PM.

    Leave a comment:


  • Michael
    replied
    Originally posted by cjcox View Post
    Again, not a performance issue, just lack of ability to "compare" with a vast db (unless you're comparing to something very very recent). Again, maybe I'm the only one that finds it interesting (?).
    The test profile versions are all versioned independently and running any of them will lock it in the metadata. And multiple versions of a given test can be installed on the same system. If you find some random test on OpenBenchmarking.org and you go 'phoronix-test-suite benchmark the-ob-id', it will fetch the same exact test profile versions as used in that original test case. So I am not sure I understand what you are saying about 'comparing to something very very recent'

    All test profiles are developed independent of PTS test releases (sans the rare case where a test depends upon a new PTS feature in which case it specifies minimum version requirements) https://github.com/phoronix-test-suite/test-profiles

    Leave a comment:


  • cjcox
    replied
    Again, not a performance issue, just lack of ability to "compare" with a vast db (unless you're comparing to something very very recent). Again, maybe I'm the only one that finds it interesting (?).

    Leave a comment:


  • Michael
    replied
    Originally posted by coder View Post
    It's a good point, though. Maybe quarterly releases are too much? Maybe 1-2 releases per year, with only compatibility patches, in the interim.
    Has anyone seen any difference in performance between PTS releases? I have not. I would suspect the only cases where PTS version differences would impact the performance would be if the system has less than 1GB of RAM or so under very heavy memory pressure. Most code changes / feature additions these days tend to be just on hardware/software detection or in result analysis after testing is complete, none of which obviously happen while test execution is happening.

    Leave a comment:

Working...
X