Announcement

Collapse
No announcement yet.

Benchmark API

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Benchmark API

    Good time of day Michael,

    I'm currently working on a reproducible, Nix-based (meaning, binary-reproducible) Haskell compilation benchmark, and I'm wondering -- what is the PTS API for test inputs/runtime/results?

    From my side:​​​​​
    • Can easily consume/produce JSON
    • Runtime-wise - my phases are:
      1. preparation (setting up env with dependencies)
      2. warmup runs (filling kernel caches etc.)
      3. actual benchmark iterations
      4. result collection & emission

  • #2
    If you have a test bash script that runs your benchmark, from there I can easily show how to spin it into a test profile. It's incredibly easy but don't have much in the way of formal documentation at the moment.
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      > If you have a test bash script that runs your benchmark

      There it is: https://github.com/deepfire/benchmar...bench/bench.sh

      One problem is that the "measure" phase still includes some small "evaluation" overhead, which doesn't exactly belong to benchmarking, but that's hard to avoid, until I know about the conventions on how to pass state between the warmup and benchmark phases.
      Last edited by deepfire; 06-01-2020, 10:58 PM.

      Comment


      • #4
        Originally posted by Michael View Post
        If you have a test bash script that runs your benchmark, from there I can easily show how to spin it into a test profile. It's incredibly easy but don't have much in the way of formal documentation at the moment.
        So, I've used the `build-gcc` benchmark in the `test-profiles` repository on Github as inspiration, and here's an RFC version:

        https://github.com/phoronix-test-sui...files/pull/150

        Note that, as mentioned in the PR, you'll need to install Nix separately -- this is an unfortunate requirement that can't really be avoided.

        The upside is reproducibility for the benchmark's environment, though.
        Last edited by deepfire; 06-03-2020, 06:05 PM.

        Comment

        Working...
        X