The Importance Of Benchmark Automation & Why I Hate Running Linux Games Manually

[mcallegari]

I partly agree with sheepdestroyer, but I also think that synthetic performance references give you the idea of what the metal can do.
For example, let's say open arena does 100fps, then you start a Steam game and you get 15fps. You would know that the game engine sucks and you need either to get a refund or wait for a fix
Comparison is good in both ways.

Anyway, @micheal: have you considered using Google before posting such long articles with all your colorful number of justifications for what others do or don't do ?
After a 30 seconds research, I got into this: https://glxosd.nickguletskii.com/
and this: https://github.com/ValveSoftware/voglperf
Maybe it's worth your precious time to look into it and a) make phoronix users happy and b) stop complaining

[curaga]

Anyway, @micheal: have you considered using Google before posting such long articles with all your colorful number of justifications for what others do or don't do ? After a 30 seconds research, I got into this: https://glxosd.nickguletskii.com/ and this: https://github.com/ValveSoftware/voglperf Maybe it's worth your precious time to look into it and a) make phoronix users happy and b) stop complaining

He knows, and he already has better tools for those purposes (libframetime, apitrace). Being able to capture timings is only useful if the game has an automated benchmark but doesn't output numbers; using a GL trace completely disregards the cpu load, making invalid results.

[PsynoKhi0]

I partly agree with sheepdestroyer, but I also think that synthetic performance references give you the idea of what the metal can do.
For example, let's say open arena does 100fps, then you start a Steam game and you get 15fps. You would know that the game engine sucks and you need either to get a refund or wait for a fix
Comparison is good in both ways.

Anyway, @micheal: have you considered using Google before posting such long articles with all your colorful number of justifications for what others do or don't do ?
After a 30 seconds research, I got into this: https://glxosd.nickguletskii.com/
and this: https://github.com/ValveSoftware/voglperf
Maybe it's worth your precious time to look into it and a) make phoronix users happy and b) stop complaining

I can understand both sides of the argument. On Michael's side it really boils down to manpower.
On the other side of the spectrum, the following [H]ardOCP article should be a must-read for anyone interested in hardware benchmarks IMO. Canned benchmarks have the inherent risk of merely benchmarking benchmark-"optimized" drivers.

Ironically enough, OpenBenchmarking should be a great platform to crowd-source benchmarks. Sure, the hardware configurations vary wildly, but with enough data, one should be able to extrapolate results, correct?

[philips]

The Phoronix Test Suite also supports the test profiles being able to skip the data over say the very first run, etc.

How exactly is this done? Is something done automatically in case the variance is too high for the first run, or something you need to set manually?
In any case, I'm glad this feature exists since is very handy for some tests.

[boltronics]

Considering the quality of their ports no wonder they don't want to port said functionality

I came here to post this too, although I wasn't going to be as harsh.

I think it boils down to Feral and other game porters not wanting anyone to perform direct comparisons against the Windows builds. This is especially true when a game has the AMD logo at the start or some such, when the game may not even run correctly without Nvidia hardware.

[philips]

Anyway, @micheal: have you considered using Google before posting such long articles with all your colorful number of justifications for what others do or don't do ?
After a 30 seconds research, I got into this: https://glxosd.nickguletskii.com/
and this: https://github.com/ValveSoftware/voglperf
Maybe it's worth your precious time to look into it and a) make phoronix users happy and b) stop complaining

He's already done this.
Check the Atlantis Substance benchmark to see how he turned a simple demo in to a full benchmark (frametimes graphs included):

http://openbenchmarking.org/innhold/bb9afb827dc1dd8be71cf79bdf3d169355c5fdcf

[Mike Frett]

Have you tried to contact Feral to see if they'd work with you on this? They're usually very responsive to the linux community.

Yet not responsive enough to care about making their Games work on AMD Hardware. =p

[Michael]

Anyway, @micheal: have you considered using Google before posting such long articles with all your colorful number of justifications for what others do or don't do ?
After a 30 seconds research, I got into this: https://glxosd.nickguletskii.com/
and this: https://github.com/ValveSoftware/voglperf
Maybe it's worth your precious time to look into it and a) make phoronix users happy and b) stop complaining

Besides what everyone else pointed out, with regards to voglperf, the glxosd is irrelevant. It's just showing sensors on the screen, which is still inaccurate if you think that's how you should look at the temperature data, etc. PTS logs all of the data and with its automation censures the start/stop logging time are always the same, etc.

[bulletxt]

Maybe I don't want to know how many FPS I'll get in Tomb Raider. Maybe I want to know the relative performance of the Nvidia blob vs Nouveau. Or Nouveau vs AMDGPU. Or Ubuntu vs Fedora. Or the last kernel/mesa vs git kernel/mesa. Where else on the web do you get a more comprehensive view than on Phoronix?

And I get an idea of real life performance by looking at the relative performance of the stuff Michael tests against what I've seen before. I don't need to know that my card will get an average of 56FPS on Tomb Raider, I can look at the performance on something Xonotic and I know roughly the relative performance of that test against more modern games. It doesn't have to be exact.

Bullshit.

[CleanCut]

Michael, as a young indie game dev shop, how exactly would you like something like "./mygame --benchmark" to behave? I'm thinking about this from the context of Skyfire, the game we are developing, which I would love to release on Linux first (all prototypes are available for free on Linux, Mac, and Windows): https://agileperception.com/skyfire We're using Unreal Engine 4, if that makes any difference.

My biggest obstacle on Linux, currently, is that I don't own a Linux desktop. We have Linux servers, naturally, but nothing with a real graphics card. So I cross-package a Linux build (native packaging for UE4 requires a real graphics card) from a Windows VM on my Mac Pro workstation.