RFC: Polishing Up The Result Graphs

[Michael]

When a card cannot be tested due to testfailure still include it in the results graph - just show it as failed

What would be the value in that?

[Lltp]

First, I would love to see some histograms or even better: probability density plots, especially for FPS related things whose distributions tend not to be normal at all (the reason why FPS average is usually not representative). I can ellaborate on that if needed.

Then, you could also apply a smoothing function to line graphs like power graphs. A simple gaussian or rectangular smoothening would be enough. The reason is that people reading your article care about the trend, not individual and noisy spikes.

While I'm talking about power graphs, there is a big problem with your power measurements: what you really want is an energy consumption. Basically, if I transpose what you do in the realm of school, it's like giving much harder exams to the best students and far easier ones to the others; ultimately concluding that the best students did very bad at the exam compared to the others. To solve this, I would suggest either to work at constant framerate and look at the power consumption (or better, its integral over time: the energy) OR to calculate watt/FPS (not as an average but for each single point) OR to work at constant number of frames per test and calculate the energy (which again, better represents the concept behind "average power consumption").

Finally I would be glad if you could do something to display full graphs on phones. On mine these are usually cropped and I need to ask for the desktop version of the website to get the graphs displayed in totality.

[uid313]

Google have some nice colors here.

https://www.google.com/design/spec/style/color.html

[Michael]

First, I would love to see some histograms or even better: probability density plots, especially for FPS related things whose distributions tend not to be normal at all (the reason why FPS average is usually not representative). I can ellaborate on that if needed.

Unfortunately, 90%+ of the games only expose average FPS and if I'm lucky will expose min/max too, but less than a handful expose the per-frame timing information in a useful manner for consumption. (And libframetime in all my trials still doesn't work always well.)

[andrei_me]

What would be the value in that?

Graph consistency between tests, if a r600g card is the 3rd hardware to appear on the graph in OpenArena test, it will be in the 3rd to appear in Unigine Valley test, even if it doesn't working, allowing one to skim the graphs and get a fast overview of the results


Also, one can't discern if the card was tested and failed or if it was tested at all

[GreatEmerald]

One general thing about the graphs that had me very confused until recently is how to get to the OpenBenchmarking result from the graphs. For one, the box icon is really not obvious; putting a text label next to it saying where it takes you would improve it a lot. For another, shift-clicking or middle-clicking the box opens... the box icon, not the OpenBenchmarking site, in a new tab.

[geearf]

I still think a summary graph would be good where you would get scores from each test and put them together to get a final score.
When you test drivers, sometimes we get some better performance in some games and worse in others, the summary graph would help see if we're on the positive or negative side as a whole.

[duby229]

One general thing about the graphs that had me very confused until recently is how to get to the OpenBenchmarking result from the graphs. For one, the box icon is really not obvious; putting a text label next to it saying where it takes you would improve it a lot. For another, shift-clicking or middle-clicking the box opens... the box icon, not the OpenBenchmarking site, in a new tab.

I second this.

[Michael]

I still think a summary graph would be good where you would get scores from each test and put them together to get a final score.
When you test drivers, sometimes we get some better performance in some games and worse in others, the summary graph would help see if we're on the positive or negative side as a whole.

There is summary graphs on OpenBenchmarking.org, but won't be in the articles, since I need as everyone as possible clicking through the articles to actually break even on all the testing that goes into each article....

[boxie]

in the SVG graph images, the PTS logo and the base.png is a data uri for a PNG - surely this would be better as SVG vectors