Announcement

Collapse
No announcement yet.

Phoronix Test Suite 6.6 M2 Preps More Open-Source Benchmarking Improvements

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Phoronix Test Suite 6.6 M2 Preps More Open-Source Benchmarking Improvements

    Phoronix: Phoronix Test Suite 6.6 M2 Preps More Open-Source Benchmarking Improvements

    The second development release of Phoronix Test Suite 6.6-Loppa is now available for your cross-platform, open-source benchmarking needs...

    http://www.phoronix.com/scan.php?pag...t-Suite-6.6-M2

  • #2
    Hi Michael,

    Have you already checked out my research on metrics to benchmark jitter? :
    https://www.phoronix.com/forums/foru...554#post878554
    https://www.phoronix.com/forums/foru...624#post883624

    It would be very nice if one of these could be included in one of the upcoming releases of the Phoronix Test Suite.
    I can help you with the implementation, but I'd first like to get some feedback on it (if you have time of course).

    Comment


    • #3
      Originally posted by Eliasvan View Post
      Hi Michael,

      Have you already checked out my research on metrics to benchmark jitter? :
      https://www.phoronix.com/forums/foru...554#post878554
      https://www.phoronix.com/forums/foru...624#post883624

      It would be very nice if one of these could be included in one of the upcoming releases of the Phoronix Test Suite.
      I can help you with the implementation, but I'd first like to get some feedback on it (if you have time of course).
      PTS makes use of what information is exposed via the given game. For games that can provide constant frame-time/FPS measurements, it tends to offer that and graph all of that information. However, most games just offer average (and min/max). So unless more games begin making it possible to parse the additional information to make those other outputs possible, I'm not sure there is much to do from the PTS side... Or am I missing something here? I assume most of those Windows sites from those links are just using FRAPS.
      Michael Larabel
      http://www.michaellarabel.com/

      Comment


      • #4
        Originally posted by Michael View Post

        PTS makes use of what information is exposed via the given game. For games that can provide constant frame-time/FPS measurements, it tends to offer that and graph all of that information. However, most games just offer average (and min/max). So unless more games begin making it possible to parse the additional information to make those other outputs possible, I'm not sure there is much to do from the PTS side... Or am I missing something here? I assume most of those Windows sites from those links are just using FRAPS.
        Thanks a lot for your reply!

        I agree with you that PTS shouldn't do too much post-processing, and leave this responsibility to the app.
        It's also true that most games don't provide detailed frame-time measurements,
        and therefore I agree that it isn't worth the effort to do anything on the PTS side.

        But...
        What if practically every game could offer these measurements in the same way, and also do the post-processing such that PTS only needs to parse the resulting jitter metric as one output value?
        You probably know what I'm referring to: libframetime[*]

        However, currently there are some problems with using libframetime to achieve reproducible and comparable benchmark results:
        • not all games support to benchmark right away, such that the first few frames must be left out; this amount of left-out frames could be non-deterministic, making it harder to compare results of different runs
        • results are only valid for comparison when the game engine renders independent of local time, i.e., it should use a fixed simulation step / delta game-time between each frame

        Let me try to clarify the last point with the following simplified pseudo code:

        Conventional game loop (without VSync of course), not suitable for benchmarking:
        Code:
            gameTime = gameTimeEpoch
            
            previousTime = time()
            while True:
                currentTime = time()
                deltaTime = currentTime - previousTime
                
                gameTime += deltaTime
                render(scene(gameTime))
                previousTime = currentTime
        Proper benchmarking game loop, yielding reproducible and comparable benchmark results:
        Code:
            gameTime = gameTimeEpoch
            
            while True:
                deltaTime = 1.0 / 60    # as if game was rendered @ 60 fps
                
                gameTime += deltaTime
                render(scene(gameTime))
        As you can see for the proper benchmarking game loop, the sequence of all gameTime values during a run is deterministic, always the same, so the frames should be visually identical between different runs.

        Let's suppose we're not allowed to modify the source code of the game to implement a proper benchmarking game loop,
        then we can just create a hook for time() (and render()) to accomplish this for (nearly) every game:
        Code:
            time_orig = time        # backup of real time() function
            render_orig = render    # backup of real render() function (actually glXSwapBuffers())
            
            currentTime = time_orig()
            def time():
                return currentTime
            
            def render():
                render_orig()
                currentTime += 1.0 / 60    # as if game was rendered @ 60 fps
        I've tested this approach on the following games, with success:
        • gl-117
        • 0ad
        • supertuxkart
        • flightgear
        • xonotic (localhost of course)

        Visual proof can be found here.

        If I would add this functionality to libframetime, and create the script to generate the jitter metric, and maybe also create some test profiles,
        would you possibly reconsider your decision and sometimes feature the new jitter metric in some of your future articles?

        .[*]: a new post-processing script can be created to use the frame-times file as input and output the jitter metric, e.g., something like this awk script
        Last edited by Eliasvan; 08-14-2016, 07:12 PM. Reason: Bypass post approval

        Comment

        Working...
        X