NVIDIA vs. AMD OpenGL & Vulkan Benchmarks With Valve's Dota 2

[dang_valve]

This is Dan, one of the developers of the Source 2 Vulkan render backend at Valve. I'm trying to replicate the performance results here. I downloaded the pts.dem file from http://www.phoronix-test-suite.com/b...ts-dota2-2.zip and I believe I am using a similar command-line based on http://openbenchmarking.org/innhold/...a1110a2808cf00 .

Command-line:
[-vulkan | -gl ] +timedemoquit pts2 +demo_quitafterplayback 1 +cl_showfps 1 +fps_max 0 -novconsole -fullscreen +timedemo_start 1 +timedemo_end 1000

My system is Intel i7-3770k NVIDIA TITAN X and I tested using the 367.18 driver on Ubuntu x64 15.10 4.2.0.27. I don't have a 4k monitor, but I see:

• Vulkan: 138.8fps GL: 139.6fps (2560x1440 Best Looking)
• Vulkan: 149.7fps GL: 142.0fps (800x600 Best Looking)

I need to test at 4k to compare results closer, but some thoughts from me:

• I have some suspicion that perhaps the game/dota/cfg/video.txt settings used for each of these tests was not identical. Dota 2 will reconfigure settings automatically when there is a new PCI deviceID/vendorID detected and that would skew results massively if the settings are not the same. You could add -autoconfig_level 3 to your command-line for example to test Best Looking settings and not have Dota 2 autodetect. If you are not verifying the settings are the same between GPU changes then that could explain why the results are so strange.
• We have spent most of the effort improving perf for CPU limited situations so if you could include some lower res scores (still at Best Looking) that would be good.
• I do not expect (and have not been able to replicate) such large GPU perf drops so I'm not sure yet why I'm not replicating it. I would have expected GPU perf to be close between Vulkan + GL. If the video settings did not match between the GL and Vulkan tests that might explain it.
• Your timedemo is a fair test, but it is fairly low in terms of activity. The more particles/models/etc. that are on-screen during the timedemo the better I expect Vulkan to do since that is when we tend to be come renderthread CPU bound on the other APIs (GL/DX9/DX11)
• In CPU limited situations I am seeing very large gains on AMD Vulkan vs GL.

[Kano]

It seems that Dota 2 needs much more VRAM, it does not start with 1 GB VRAM at 1920x1200 and your cards seem to have got only 2 GB VRAM that can not run the game at 4k. I think the 4 GB models should be able to use Vulkan at this res. Maybe monitor VRAM use - with different res. The GTX 980 Ti has much more VRAM than the GTX 980 and is therefore faster, more interesting is the R9 285 with 2 GB VRAM, compared to NVIDIA my guess is that AMD uses some kind of compression - maybe implict active vs by default off.

@Dang_valve

A card with 12 GB VRAM does logically not hit VRAM issues at your tested res, use some more common cards please...

[humbug]

Michael; did you opt into the steam client beta as well? In the release notes valve says to do that in order to ensure optimal vulkan performance.

[pinguinpc]

This is some dota 2 opengl and vulkan tests


System Specs Used in Both Tests

Nvidia Drivers 367.18 (run package from nvidia homepage)

Xubuntu 16.04 64Bit - Kernel 4.4.0-23 generic (ubuntu mainline) - CPUFreq: Performance

CPU: INTEL Pentium G3258 (Haswell 22nm) 4.1Ghz + Artic Cooling Alpine 11 Plus

MEM: 8GB DDR3 1333 (2x4) Patriot value (dual channel: 21.3 gb/s)

GPU: Zotac Nvidia Geforce GT630 (GK208 28nm: 384 Shaders / 8 ROPS) Zone Edition Passive Cooling 2GB DDR3 1800Mhz 64Bit (14.4Gb/s)

MAINBOARD: MSI H81M E33


OpenGL

Vulkan

Concluded test can say this: opengl client dont use gpu at max but leave enough gpu for dont give troubles when nvenc stay recording at 48fps

But on vulkan side gpu shows more higher values 99 and 100% in some time periods and this dont allow use nvenc for recording at 48fps, only possible use 30fps without problems*

*If gpu have 99 or 100% of use nvenc down use and if this situation appears video recording is damaged because nvenc skip fps

Respect performance as others said performance is similar in some situations less or more: with much objects in scene is more notorious or with some explosions

Without forget vulkan stay stable on test (no crash) and 48fps nvenc recording reduce performance around 10 to 15fps depending scene

[liam]

Seems people here really aren't paying attention... Did you not read the page regarding CPU usage? Sure, he's using a high resolution, but Vulkan is already proving it's worth. The drop in CPU usage, even at 4K, is hugely significant. If you have a 1080p display and any of the GPUs that were tested, you don't need Vulkan.

I noticed the lower cpu usage, but, from a glance, the numbers (cpu/fps) looked close enough to explain things.
I agree with the others than trying this test with a lower resolution is worthwhile.
On that note, I'd like to put this idea out there:
SINCE EVERYONE LOVES TO COMPLAIN ABOUT MICHAEL'S TESTS, WHY NOT ACTUALLY WRITE SOME PLUGINS TO PTS THAT YOU'D FIND USEFUL.
iT'S ALL OPEN SOURCE SO IT'S NOT AS THOUGH YOU'RE SIMPLY GIVING MICHAEL THIS GREAT ECONOMIC BENEFIT (THOUGH, HOPEFULLY, HE WOULD BENEFIT FROM IT AS WELL BECAUSE THERE'S NO ONE ELSE OUT THERE THAT ACTUALLY REGULARLY TESTS LINUX).

i now resume normal font
BTW, I'm actually serious about this. Adding actually useful plugins to pts (or whatever test platform) would be genuinely useful. For the most part, the kernel developers simply have different needs than desktop users, and without shaming data, or a giant patch set backed by Intel that fixes all these kernel/drm issues, they just won't get fixed*.

*any time soon

[stiiixy]

I noticed the lower cpu usage, but, from a glance, the numbers (cpu/fps) looked close enough to explain things.
I agree with the others than trying this test with a lower resolution is worthwhile.
On that note, I'd like to put this idea out there:
SINCE EVERYONE LOVES TO COMPLAIN ABOUT MICHAEL'S TESTS, WHY NOT ACTUALLY WRITE SOME PLUGINS TO PTS THAT YOU'D FIND USEFUL.
iT'S ALL OPEN SOURCE SO IT'S NOT AS THOUGH YOU'RE SIMPLY GIVING MICHAEL THIS GREAT ECONOMIC BENEFIT (THOUGH, HOPEFULLY, HE WOULD BENEFIT FROM IT AS WELL BECAUSE THERE'S NO ONE ELSE OUT THERE THAT ACTUALLY REGULARLY TESTS LINUX).

i now resume normal font
BTW, I'm actually serious about this. Adding actually useful plugins to pts (or whatever test platform) would be genuinely useful. For the most part, the kernel developers simply have different needs than desktop users, and without shaming data, or a giant patch set backed by Intel that fixes all these kernel/drm issues, they just won't get fixed*.

*any time soon

I agree =D

And there's certainy lots for Michael to take away from the feedback regardless, especially since a Valve employee is posting.

Has there ever been a rapport wiht Valve since Michael visited? Why aren't they sharing resource enough to build PTS in to a damn fine testing engine for at least Valve games. The feedback to both party's, and benefit's to Valve along wiht the community would be worth their weight in gold.

[philips]

[*]Your timedemo is a fair test, but it is fairly low in terms of activity. The more particles/models/etc. that are on-screen during the timedemo the better I expect Vulkan to do since that is when we tend to be come renderthread CPU bound on the other APIs (GL/DX9/DX11)[*]In CPU limited situations I am seeing very large gains on AMD Vulkan vs GL.

Can you (or anyone else) upload somewhere a relevant timedemo so we (and Michael) can use it?

[shanefagan]

I have a R9 285 and I have between 80FPS and 95FPS at 1080p. These results look pretty off compared to what I got on my machine. I ran into a bug which reduced performance but minimum that I got before that bug was 75FPS and that was in a bigger team fight which had a lot of effects. Also it has to be noted that CPU usage and power consumption overall was lower and performance matched OpenGL on my machine so the results were predictable but more stable.

[bug77]

WTF Michael, the whole point about Vulkan are CPU-bound scenarios, why did you test only 4K which is the less interesting resolution?

Well, at no point has Vulkn been advertised to lower CPU driver overhead at the expense of 4k performance, so testing at 4k is very much relevant.

[curaga]

There are no AMD numbers due to the AMDGPU-PRO driver not currently exposing the GPU utilization in any meaningful way.

I believe radeontop works with AMDGPU as well, though I have no such cards to test it. Can anyone running AMDGPU try it?