Mesa 18.0/18.1/18.2 RadeonSI + RADV Benchmark Comparison With Radeon RX 580 / R9 Fury / RX Vega 64

[stalkerg]

I totally agree with @oooverclocker we should also see min fps for some percentile.

[Michael]

I like to thank Michael for this nice test but I am concerned that when we compare gaming performance the same way Windows-related testers do we might get the same FPS-targeted drivers that deliver a worse gaming experience. It would be fantastic when we could decide to measure only the frame time and how much it varies. We should honor those who make our video games more smooth and more enjoyable instead of creating wrong inducements with these FPS counts that don't show much about the smoothness. To be able to compete we need console-like performance and stability. Otherwise there is no reason to prefer Linux over Windows for gaming and Linux consoles will not be successful.
And finally Linux can't become successful on the desktop without these preconditions.

PTS generally grabs whatever pertinent information is exposed by the games under test... While some on Windows use FRAPS and other popular programs, there isn't any good/always-working/universal program for obtaining the same information on Windows... PTS has tried using libframetime previously for some games to provide frame-time information but it led to it not working with some OpenGL games, some interaction issues under other cases, etc. Plus there isn't any universal tool for leveraging the Vulkan tests either. Wish more games would nicely expose it but as you can see in tests of Rise of the Tomb Raider and friends where it exposes a lot, PTS happily captures it.

[Michael]

I totally agree with @oooverclocker we should also see min fps for some percentile.

See my other post just now.

[stalkerg]

See my other post just now.

I understand your concerns but we still need to do something, because currently we just have wrong results for analytic. AMD developers have reached hardware limits and most work will be do in other fields, just not interesting to look into same results.

[Strunkenbold]

Just wondering if the steam shader cache was active? This is would explain why 18.2 has serious performance regressions. Simply as dev versions dont get precompiled shaders from Steam...
In general, was the mesa shader cache cleaned after each run? The test compared 18.0 between different kernels. But if the cache wasnt cleaned after the first run, the second run just used the pre compiled shaders (which Steam did not already delivered) and it looks like kernel 4.17 gave some mild improvements which were in reality just because of the cached shaders.

When I do vulkan testing with dota2, the first run is always running like a slideshow. PTS doesnt exclude the first run from the result. In the end, the results are actually not compareable. You always have to make two runs from the same mesa version to get compareable results.

The stuttering in the first run is actually something which is unacceptable but I fear that this is something which cant be fixed. This means bad frame times and low min fps forever.

[Strunkenbold]

Unfortunately I can confirm corrupt rendering with an RX 580 8GiB in Deus Ex.
It was a wonderful year 2017 for RadeonSI but the performance improvements seem to slowly reach the limit with RadeonSI and Polaris and instead of spending a lot of effort to improve frame rates further I have to suggest putting a bit more significance to stability, smooth frametimes and compatibility. I don't know how often it's the case but I think when there are games that break stuff or that need fixes that make RadeonSI slower / delivering more inconsistent frame rates it might be more and more important to contact the developers instead of fixing it in RadeonSI - if that is the case for some regressions.

There a 5 different card generations and hundreds of games out there and always at least 3 different versions of LLVM. Mesa radeon team consists of two people. Sure they have piglit but my feeling is that this is not enough to guarantee a hassle free experience.
Especially since they dont fix long standing bugs anymore. I would still encourage you to report your findings to freedesktop or here:

https://www.gamingonlinux.com/wiki/Games_broken_on_Mesa

[oooverclocker]

PTS has tried using libframetime previously for some games to provide frame-time information but it led to it not working with some OpenGL games, some interaction issues under other cases, etc. Plus there isn't any universal tool for leveraging the Vulkan tests either. Wish more games would nicely expose it but as you can see in tests of Rise of the Tomb Raider and friends where it exposes a lot, PTS happily captures it.

Thanks for the extensive answer. I see, there is lack of infrastructure everywhere around the drivers. Important data such as frame time must be reliably obtainable in a unified way. The community has to find a way to address this.

[Strunkenbold]

When I do vulkan testing with dota2, the first run is always running like a slideshow. PTS doesnt exclude the first run from the result. In the end, the results are actually not compareable. You always have to make two runs from the same mesa version to get compareable results.

Would be great to get an answer, anyway here are some numbers:

First run, heavy stuttering due shader compiling.
Resolution: 1920 x 1080 - Renderer: Vulkan:
102.5
121.6
119.3
119.9
120.9
120

Average: 117.37 Frames Per Second
Deviation: 6.24%

Same settings, just ran again. Shader Cache is fully working.
Resolution: 1920 x 1080 - Renderer: Vulkan:
122
122.3
122.2

Average: 122.17 Frames Per Second
Deviation: 0.13%

So you can see, even as pts executed the first test 6 times instead of 3, the difference between the results is quite big.
Thats why I ask again, was the steam shader cache active? Got the shader cache deleted before those numbers were produced?