Dota 2 Test broken

[Michael]

Dont know if thats intended behavior but it seems that the new test isnt used on old benchmark presets.
I have many datapoints from old mesa versions and want to compare to the current one. But pts uses everytime 1.2.2, only if I create a new preset its using 1.2.6.

That's intended to ensure past results are 1:1 comparable with new runs, but in your old results you can simply sed/find-and-replace the version with the new version.

[Strunkenbold]

Dont know if thats intended behavior but it seems that the new test isnt used on old benchmark presets.
I have many datapoints from old mesa versions and want to compare to the current one. But pts uses everytime 1.2.2, only if I create a new preset its using 1.2.6.

[Strunkenbold]

Thx for fixing this. Now I can continue my regression testing.

[Michael]

Apologies for the delay, I've now been able to test it out. Indeed, the latest Steam updates to Dota 2 tweak the output of the FPS information from the game... The test profile has now been adjusted to account for the latest dota2. This should be fixed by the latest test profile update, pts/dota2-1.2.5.

[bruized]

I noticed the same thing. Based on the csv output that number is the frame variability. I'm not sure how the results parser works to tell whether this is a bug but the fix is to edit the results-definition.xml of the test and remove UNKNOWN i.e. :

<OutputTemplate>pts2.dem,UNKNOWN,#_RESULT_#, 17.5, 30.1,3840,2160 (...)

changed to

<OutputTemplate>pts2.dem,,#_RESULT_#, 17.5, 30.1,3840,2160 (...)

[Strunkenbold]

I think some recent game update broke how pts received the numbers from the benchmark.

Before I got results between 110-120 fps now its much lower.
When you watch the benchmark you can clearly see that fps is still in that range. So there is no system regression.

Resolution: 1920 x 1080 - Renderer: Vulkan:
14.8
15.4
14.4

Looks almost like its returning minimal fps now instead of average.