Tweet
(edit / TLDR): Antialiasing is disabled on Openbenchmarking (but it shouldn't be), and also Tessellation:Normal is used instead of Tessellation:Extreme.
As I have said in another thread, I have noticed a problem with the Heaven 4.0 benchmark on openbenchmarking.org .
The benchmark settings that openbenchmarking.org is using are non-standard, non-comparable to other results on the Internet, and those settings do not produce "sufficiently fair" results across architectures and generations of GPUs. There also a few other issues. I really like the Heaven benchmark otherwise, so I thought I could help there.
Here I give better settings for the Heaven 4.0 benchmark, for anyone who would like to use them.
"Heaven 4.0 Extreme FHD" setting is the most widely used one on the Internet, with results on multiple websites; it is what "people want" to test. Note that this is a non-standard setting, it matches the "Extreme" preset, but uses a higher resolution.
A few notes/comments/instructions:
- The command line executes a 64-bit version of Heaven. To execute a 32-bit version (it produces the same results) , replace ./heaven_x64" with ./heaven_x86; also you will probably need to set: export LD_LIBRARY_PATH=./x86:$LD_LIBRARY_PATH
- "windowed" mode is used instead of "fullscreen". This is important for several reasons. First, when using "fullscreen", Heaven benchmark may automatically switch to a lower or higher resolution if the requested resolution is unavailable. Second, it allows for benchmarking even some resolutions above the monitor's native resolution. Finally, the results are usually identical (or very close) to the "fullscreen" mode.
- sound is set to null (I think it is more appropriate that way), but the results are no different when sound is enabled.
- extern_plugin ",GPUMonitor" -- this can also be ommited. Users usually like it, so I suggest keeping it.
- in the given command lines, renderer is set to OpenGL; adjust as required (opengl / direct3d9 / direct3d11)
- There are three APIs that Heaven 4.0 can use on Windows: DirectX 11, DirectX 9, and OpenGL. Those settings are also valid for WinE translation.
- There is a native Heaven 4.0 benchmark for Linux, I suspect that it produces up to 5% higher scores on most video cards. The native Heaven benchmark should be included, and all 7 versions of Heaven should be clearly disambiguated: LinuxNative64/OpenGL, WinE/OpenGL, WinE/D3D11, WinE/D3D9, Windows/OpenGL, Windows/D3D11, Windows/D3D9. The difference between LinuxNative64/OpenGL and Windows/OpenGL is sometimes minor (if the drivers are good), and sometimes huge. The difference between Windows/OpenGL and Windows/D3D11 ranges from 0% to 7% better results for D3D11.
- If multiple runs of Heaven are performed, the best one should be selected as the correct result. The slower runs can only be a result of anomalies, for example if some other application is running in the background. When there are no other applications running, then the results of Heaven are quite consistent, and there is virtually no need to perform multiple runs. Another possible reason for getting a lower result is if the GPU is too hot at the start, but in most cases Heaven does not cause the GPU to exceed the temperature limits (except maybe on some laptops).
- (note for users) About GPU memory requirements: If more GPU memory is available than required, it will not affect the final score in any way; most new GPUs have plenty of RAM for running Heaven. To lower the memory requirements (for old GPUs with 1 GB memory or less), disable the 3D accelerated "Aero" compositor on Windows. On Linux, use a CPU-based compositor (like Xfce). Heaven does not tolerate too small amounts of video memory; the framerate decreases drastically with just a 1% reduction in available video RAM. DirectX renderers will use 6% less memory than specified; native OpenGL renderer matches the specified value.
Somewhat related note: the difference in scores between cards of the same manufacturer (i.e. ASUS, MSI, Sapphire, etc...) with the same GPU depend on the clock setting, and are usually not much different than differences between manufacturers. So I suggest that the manufacturer name should be removed when aggregating the results (i.e. agregate based on the GPU only, not based on manufacturer and GPU). That should result in more reliable results, for more different GPUs.
Perhaps I will also write some additional info about the Heaven benchmark for the users, if I get some spare time.
As I have said in another thread, I have noticed a problem with the Heaven 4.0 benchmark on openbenchmarking.org .
The benchmark settings that openbenchmarking.org is using are non-standard, non-comparable to other results on the Internet, and those settings do not produce "sufficiently fair" results across architectures and generations of GPUs. There also a few other issues. I really like the Heaven benchmark otherwise, so I thought I could help there.
Here I give better settings for the Heaven 4.0 benchmark, for anyone who would like to use them.
"Heaven 4.0 Extreme FHD" setting is the most widely used one on the Internet, with results on multiple websites; it is what "people want" to test. Note that this is a non-standard setting, it matches the "Extreme" preset, but uses a higher resolution.
Code:
Name: Heaven 4.0 Extreme FHD (1920x1080) APIs: DirectX 11 or OpenGL Settings: 1920x1080, Quality:Ultra, Tessellation:Extreme, 8xAA GPU memory requirements: 1120 MB, plus approx. 130 MB for a 3D-accelerated desktop compositor Command line: ./heaven_x64 -project_name "Heaven" -data_path "../" -engine_config "../data/heaven_4.0.cfg" -system_script "heaven/unigine.cpp" -sound_app null -video_app opengl -video_multisample 3 -video_fullscreen 0 -video_mode 6 -video_vsync 0 -extern_define ",RELEASE,LANGUAGE_EN,QUALITY_ULTRA,TESSELLATION_EXTREME" -extern_plugin ",GPUMonitor"
Code:
Name: Heaven 4.0 Preset Extreme (1600x900) Note: this is a standard preset from the benchmark APIs: DirectX 11 or OpenGL (DirectX 11 is preset standard on Windows, OpenGL on Linux) Settings: 1600x900, Quality:Ultra, Tessellation:Extreme, 8xAA GPU memory requirements: 970 MB, plus approx. 130 MB for a 3D-accelerated desktop compositor Command line: ./heaven_x64 -project_name "Heaven" -data_path "../" -engine_config "../data/heaven_4.0.cfg" -system_script "heaven/unigine.cpp" -sound_app null -video_app opengl -video_multisample 3 -video_fullscreen 0 -video_mode 5 -video_vsync 0 -extern_define ",RELEASE,LANGUAGE_EN,QUALITY_ULTRA,TESSELLATION_EXTREME" -extern_plugin ",GPUMonitor"
A few notes/comments/instructions:
- The command line executes a 64-bit version of Heaven. To execute a 32-bit version (it produces the same results) , replace ./heaven_x64" with ./heaven_x86; also you will probably need to set: export LD_LIBRARY_PATH=./x86:$LD_LIBRARY_PATH
- "windowed" mode is used instead of "fullscreen". This is important for several reasons. First, when using "fullscreen", Heaven benchmark may automatically switch to a lower or higher resolution if the requested resolution is unavailable. Second, it allows for benchmarking even some resolutions above the monitor's native resolution. Finally, the results are usually identical (or very close) to the "fullscreen" mode.
- sound is set to null (I think it is more appropriate that way), but the results are no different when sound is enabled.
- extern_plugin ",GPUMonitor" -- this can also be ommited. Users usually like it, so I suggest keeping it.
- in the given command lines, renderer is set to OpenGL; adjust as required (opengl / direct3d9 / direct3d11)
- There are three APIs that Heaven 4.0 can use on Windows: DirectX 11, DirectX 9, and OpenGL. Those settings are also valid for WinE translation.
- There is a native Heaven 4.0 benchmark for Linux, I suspect that it produces up to 5% higher scores on most video cards. The native Heaven benchmark should be included, and all 7 versions of Heaven should be clearly disambiguated: LinuxNative64/OpenGL, WinE/OpenGL, WinE/D3D11, WinE/D3D9, Windows/OpenGL, Windows/D3D11, Windows/D3D9. The difference between LinuxNative64/OpenGL and Windows/OpenGL is sometimes minor (if the drivers are good), and sometimes huge. The difference between Windows/OpenGL and Windows/D3D11 ranges from 0% to 7% better results for D3D11.
- If multiple runs of Heaven are performed, the best one should be selected as the correct result. The slower runs can only be a result of anomalies, for example if some other application is running in the background. When there are no other applications running, then the results of Heaven are quite consistent, and there is virtually no need to perform multiple runs. Another possible reason for getting a lower result is if the GPU is too hot at the start, but in most cases Heaven does not cause the GPU to exceed the temperature limits (except maybe on some laptops).
- (note for users) About GPU memory requirements: If more GPU memory is available than required, it will not affect the final score in any way; most new GPUs have plenty of RAM for running Heaven. To lower the memory requirements (for old GPUs with 1 GB memory or less), disable the 3D accelerated "Aero" compositor on Windows. On Linux, use a CPU-based compositor (like Xfce). Heaven does not tolerate too small amounts of video memory; the framerate decreases drastically with just a 1% reduction in available video RAM. DirectX renderers will use 6% less memory than specified; native OpenGL renderer matches the specified value.
Code:
Name: Heaven 4.0 Preset Basic (1280x720) Note: this is a standard preset from the benchmark APIs: All APIs, but D3D9 is the standard in the preset, OpenGL is standard on Linux Settings: 1280x720, Quality:Medium, Tessellation:Disabled, 2xAA GPU memory requirements: 600 MB, plus approx. 130 MB for a 3D-accelerated desktop compositor Command line: ./heaven_x64 -project_name "Heaven" -data_path "../" -engine_config "../data/heaven_4.0.cfg" -system_script "heaven/unigine.cpp" -sound_app null -video_app opengl -video_multisample 1 -video_fullscreen 0 -video_mode 3 -video_vsync 0 -extern_define ",RELEASE,LANGUAGE_EN,QUALITY_MEDIUM,TESSELLATION_DISABLED" -extern_plugin ",GPUMonitor"
Code:
Name: Heaven 4.0 Low (1024x576) Note: this setting is for old GPUs with less than 512 MB of video-RAM. APIs: All, but D3D9 or OpenGL is recommended for compatibility reasons Settings: 1024x576, Quality:Low, Tessellation:Disabled, AA off GPU memory requirements: 295 MB, plus approx. 130 MB for a 3D-accelerated desktop compositor Command line: ./heaven_x64 -project_name "Heaven" -data_path "../" -engine_config "../data/heaven_4.0.cfg" -system_script "heaven/unigine.cpp" -sound_app null -video_app opengl -video_multisample 0 -video_fullscreen 0 -video_mode 2 -video_vsync 0 -extern_define ",RELEASE,LANGUAGE_EN,QUALITY_LOW,TESSELLATION_DISABLED" -extern_plugin ",GPUMonitor"
Somewhat related note: the difference in scores between cards of the same manufacturer (i.e. ASUS, MSI, Sapphire, etc...) with the same GPU depend on the clock setting, and are usually not much different than differences between manufacturers. So I suggest that the manufacturer name should be removed when aggregating the results (i.e. agregate based on the GPU only, not based on manufacturer and GPU). That should result in more reliable results, for more different GPUs.
Perhaps I will also write some additional info about the Heaven benchmark for the users, if I get some spare time.
Comment