The Importance Of Benchmark Automation & Why I Hate Running Linux Games Manually

[Michael]

Michael, as a young indie game dev shop, how exactly would you like something like "./mygame --benchmark" to behave? I'm thinking about this from the context of Skyfire, the game we are developing, which I would love to release on Linux first (all prototypes are available for free on Linux, Mac, and Windows): https://agileperception.com/skyfire We're using Unreal Engine 4, if that makes any difference.

My biggest obstacle on Linux, currently, is that I don't own a Linux desktop. We have Linux servers, naturally, but nothing with a real graphics card. So I cross-package a Linux build (native packaging for UE4 requires a real graphics card) from a Windows VM on my Mac Pro workstation.

Basically just a way to be able to automatically fire up the game (and ideally CLI switches or at least some config file I can modify, to specify the resolution and other settings), run the test without needing any interaction, and to then dump the results to either the standard output or a file.

[liam]

Michael, as a young indie game dev shop, how exactly would you like something like "./mygame --benchmark" to behave? I'm thinking about this from the context of Skyfire, the game we are developing, which I would love to release on Linux first (all prototypes are available for free on Linux, Mac, and Windows): https://agileperception.com/skyfire We're using Unreal Engine 4, if that makes any difference.

My biggest obstacle on Linux, currently, is that I don't own a Linux desktop. We have Linux servers, naturally, but nothing with a real graphics card. So I cross-package a Linux build (native packaging for UE4 requires a real graphics card) from a Windows VM on my Mac Pro workstation.

You realize that you can easily install Linux to your Mac?
Either through a VM or dual boot. No destruction to your current desktop required

[liam]

I can understand both sides of the argument. On Michael's side it really boils down to manpower.
On the other side of the spectrum, the following [H]ardOCP article should be a must-read for anyone interested in hardware benchmarks IMO. Canned benchmarks have the inherent risk of merely benchmarking benchmark-"optimized" drivers.

Ironically enough, OpenBenchmarking should be a great platform to crowd-source benchmarks. Sure, the hardware configurations vary wildly, but with enough data, one should be able to extrapolate results, correct?

In the Time section he said:

...but this is actually not even close to being the most important reason for test/benchmark automation...

And then goes on to eight other reasons that don't have to do with manpower.

[CleanCut]

Basically just a way to be able to automatically fire up the game (and ideally CLI switches or at least some config file I can modify, to specify the resolution and other settings), run the test without needing any interaction, and to then dump the results to either the standard output or a file.

What kind of information would you like in the results?

[curaga]

What kind of information would you like in the results?

Most profiles have min/avg/max fps, some also expose the per-frame frame times.

[CleanCut]

Most profiles have min/avg/max fps, some also expose the per-frame frame times.

Great, thanks!

[PsynoKhi0]

In the Time section he said:


And then goes on to eight other reasons that don't have to do with manpower.

And maybe I should stop glossing over articles... The problem with canned benchmarks still stands though. Though now it might even infect hardware design too. Case in point, the gtx 1080.

[liam]

And maybe I should stop glossing over articles... The problem with canned benchmarks still stands though. Though now it might even infect hardware design too. Case in point, the gtx 1080.

I wouldn't argue with that, but the problems that he outlined exist everywhere.
Apitrace seems like the way to go (and he uses it) as that lets you reliably reproduce an arbitrary source of gameplay, but apitrace has requirements that make it a non-optimal solution (you really need the full call stack as well).