GL_vs_VK: A Micro-Benchmark Looking At The Overhead Of OpenGL vs. Vulkan APIs

[RippeR37]

Hey there, author of these benchmarks here.

First of all, I would like to point out that these tests are in no way comprehensive over both APIs. These are designed as small benchmarks simulating only part of a scene (some aspect/algorithm used like shadow mapping) and some specific synthetic scenarios like purely CPU-bound work (+ as many draw calls as you can push) with very low GPU workload. These tests doesn't even touch subjects like on-fly dynamic allocations with my custom memory manager or other aspects that must be taken care of by bigger applications.
So results of these shouldn't be taken entirely out of given above context.

Furthermore, I've added a disclaimer at the top of README file in repository that states that these tests were done using only basic features of both APIs, as testing the limits of both was not in the scope of my research. Nothing there comes even close to techniques known as AZDO and I didn't do any special optimization. I wanted to test APIs overhead and some new features (like push constants) and not the fact that with very big effort (and possibly using some vendor-specific extensions) you can achieve similar performance in both APIs.

At last, while it's nice see such comprehensive tests done on Linux, I want to add that my tests on Windows platform shows slightly different picture. While in 1st test the difference is big in favor of Vulkan (which is to be expected (!) as in it's default configuration it's entirely CPU-bound on draw calls), and the second one mainly too (as again, it revolves around many draw calls for small objects), I actually get worse performance in Vulkan and no benefit of multithreaded work there then with OpenGL version. I'd say that it might be either drivers (which means that Linux drivers sucks, as I get a lot worse performance on Linux then on Windows in this test with OpenGL API) or I might have done something wrong (though I've read code 3x now and I don't see possibilities for optimization). Also if I rig first test to use heavier objects then performance gets pretty equal between each version, with GL version being slightly faster).

So in the end - please take it with a grain of salt and please verify code to see how it's made or test it on your HW. You can get noticeable different results.

[RippeR37]

Hey there, author of these benchmarks here.

First of all, I would like to point out that these tests are in no way comprehensive over both APIs. These are designed as small benchmarks simulating only part of a scene (some aspect/algorithm used like shadow mapping) and some specific synthetic scenarios like purely CPU-bound work (+ as many draw calls as you can push) with very low GPU workload. These tests doesn't even touch subjects like on-fly dynamic allocations with my custom memory manager or other aspects that must be taken care of by bigger applications.
So results of these shouldn't be taken entirely out of given above context.

Furthermore, I've added a disclaimer at the top of README file in repository that states that these tests were done using only basic features of both APIs, as testing the limits of both was not in the scope of my research. Nothing there comes even close to techniques known as AZDO and I didn't do any special optimization. I wanted to test APIs overhead and some new features (like push constants) and not the fact that with very big effort (and possibly using some vendor-specific extensions) you can achieve similar performance in both APIs.

At last, while it's nice see such comprehensive tests done on Linux, I want to add that my tests on Windows platform shows slightly different picture. While in 1st test the difference is big in favor of Vulkan (which is to be expected (!) as in it's default configuration it's entirely CPU-bound on draw calls), and the second one mainly too (as again, it revolves around many drawcalls for small objects), I actually get worse performance in Vulkan and no benefit of multithreaded work there then with OpenGL version. I'd say that it might be either drivers (which means that Linux drivers sucks, as I get a lot worse performance on Linux then on Windows in this test with OpenGL API) or I might have done something wrong (though I've read code 3x now and I don't see possibilities for optimization). Also if I rig first test to use heavier objects then performance gets pretty equal between each version, with GL version being slightly faster).

So in the end - please take it with a grain of salt and please verify code to see how it's made or test it on your HW. You can get noticeable different results.

[RippeR37]

BTW. As for minimum/maximum frame rates - I wouldn't count on them that much, as currently they are computed as min/max frame time on CPU taken between consecutive:
- buffer swaps (in GL)
- vkQueuePresntKHR() calls in VK

I think good idea could be to take samples from some time window and computing min/max/avg of these time windows instead.
Currently averages are computed as number or presented frames between start and end of benchmark divided by benchmark duration so it should be good.

[starshipeleven]

It doesn't look right. Vulkan was supposed to be a thin layer that would put the full power of the hardware in the hands of the application. Yet AMD still trails, even it has the horse power. Intel also seems to punch way above their weight. So maybe these benchmarks need a few more iterations?

The test looks at CPU overhead, not at performance. The graphics are easy enough as it's not supposed to stress GPU, it's spamming stuff on CPU side.

[gurv]

What? The tests were designed to be limited by the CPU. That was the whole point.

Exactly, furthermore results are similar to what I see on my PC: radeonsi + amdgpu lags behind nvidia by about 20% in cpu overhead.
I was also surprised to see amdgpu have its importance as far as cpu overhead is concerned: I get significantly worse performance with amd-staging kernel 4.11 vs mainline 4.12

[NihilMomentum]

It doesn't look right. Vulkan was supposed to be a thin layer that would put the full power of the hardware in the hands of the application. Yet AMD still trails, even it has the horse power. Intel also seems to punch way above their weight. So maybe these benchmarks need a few more iterations?
Still, thanks for adding them Michael.

Look again at the tests, the 780Ti is beating the 1080Ti. This, like most situations nowadays, is probably the fault of the application itself, for the most part.

[aufkrawall]

Wanna throw in my CPU benchmark results of Serious Sam Fusion 2017 Beta with Vulkan:
Here on a GTX 1070, Vulkan on Linux is massively faster in CPU bound scenarios (aka really, really low resolution and graphics effects) than Vulkan on Windows. It was something like ~250fps vs. 350fps. I triple checked because I couldn't believe the result, but it seems that the numbers were valid.
Now question is if that's caused by the game or by the driver/OS.

[ua=42]

It doesn't look right. Vulkan was supposed to be a thin layer that would put the full power of the hardware in the hands of the application. Yet AMD still trails, even it has the horse power. Intel also seems to punch way above their weight. So maybe these benchmarks need a few more iterations?
Still, thanks for adding them Michael.

Remember, they are still working on the vulkan driver on linux for radeon. It is know to not be as fast as it should.

[cj.wijtmans]

Nothing there comes even close to techniques known as AZDO and I didn't do any special optimization.

Ah i am sure some readers were expecting me to comment about this. The usual opengl AZDO argument. AZDO/multithreading should come close to vulkan.

[Sidicas]

Too many games that can do OpenGL rendering, but are poorly optimized. Granted there are games like Left4Dead 2 and Team Fortress 2 that run fantastic on OpenGL. It seems a lot of game devs dont have the expertise or the time to build an OpenGL game that runs at even 70% the performance of Direct3D. Need to make it easier to optimize and find why games like pcsx2 lose so much framerate with OpenGL as opposed to Direct3D. The gap is even bigger comparing OpenGL to Vulcan.