Better Wine Benchmarking This Summer For Windows Programs On Linux

Written by Michael Larabel in WINE on 14 June 2018 at 03:33 PM EDT. 36 Comments
WINE
Improvements are underway for Wine benchmarking to help ensure this open-source program for running Windows apps/games on Linux and other operating systems continues without introducing performance regressions and for being able to quantitatively verify expected performance improvements around its Direct3D/OpenGL/Vulkan code-paths, among other areas of Wine.

One of the interesting GSoC 2018 summer projects sponsored by Google this summer is on automated game benchmarks for Wine. In particular, Direct3D game benchmarks with a hope of helping the Wine project move forward especially for having results over performance improvements between Wine revisions. The work is being done by "Moihack" and mentored by Wine veterans Stefan Dösinger and Aaryaman Vasishta.

As part of that Google Summer of Code project, the Phoronix Test Suite is being used for facilitating the benchmarking process with the student working on new Wine-centered test profiles to run in a fully-automated and reproducible manner. He's uploaded some of his work via Moihack on OpenBenchmarking.org so others can easily utilize the tests too and there also Stefan's test profiles.

Stefan Dösinger years ago at CodeWeavers previously worked on automated Wine benchmarks using the Phoronix Test Suite and that effort has now been revived thanks to GSoC 2018. During the earlier efforts, the Phoronix Test Suite was running within Wine itself while now it's been decided to punt it outside of Wine and then to have the test profiles call into Wine for running the Windows binaries. Among other reasons, this will also be helpful for better hardware/software detection, being able to leverage the Phoronix Test Suite's system sensor reporting during benchmark runs, Phoromatic integration with auto power-on/off/WoL support and systemd integration, etc.


In Phoronix Test Suite Git now for helping this effort is now the WINE_VERSION= environment variable recognition to point to the desired Wine binary for querying the versioning information for reporting to the system results table. Another new feature is the USE_WINE= environment variable that will auto-load the new use_wine module.

What this use_wine module does is when running on Linux (or macOS) systems, rather than using the Linux version of test profiles, the module will dynamically hook-in and use the Windows versions of tests, but not before the test profile is auto-converted to call into Wine for carrying out the benchmarks. The Wine binary/command specified by the USE_WINE will be used, in case you have multiple Wine/Wine-Staging/CrossOver builds on the same system and wish to compare them. This Wine module I have been working on is so far working for most of the dozens of Windows test profiles available via the Phoronix Test Suite. Thanks to the highly structured test profiles, it's working out quite well and further showing the versatility of the Phoronix Test Suite.

Thanks to the Wine improvements with PTS, I'll certainly be running some fresh Wine/Wine-Staging/DXVK and more benchmarks soon on Phoronix. It will also be interesting to see how far the Wine benchmarking advances by the end of GSoC wrapping up.

I do happen to have some extra boxes as well I could devote to daily/bi-daily/weekly benchmarking at LinuxBenchmarking.com of Wine similar to what I do with other open-source projects, especially now that the Phoronix Test Suite will run outside of Wine to make the Phoromatic integration nicer. Unfortunately the Wine daily build PPA no longer is maintained nor does there seem to be any other daily/bi-daily package repositories for Ubuntu or other tier-one Linux distributions that I am aware of... If anyone has any ideas, feel free to share so then I could also be providing these automated routine Wine benchmarks. I'd much prefer having a public package archive so that others could pull from the same binaries for comparison/reproducing and to eliminate any possible questions by others over the Wine build configuration, etc.

Any other Wine benchmarking ideas? Feel free to share in the forums.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week