Quote Originally Posted by Kivada View Post
Send me the gear and I'll write the article. I'm not going to test my old stuff. I'll do Linux vs windows on the same hardware with overclock and underclock/volt results, power draw, noise levels, heat output etc. since those are the kinds of things that allot of people find relevant and will become more relevant with people looking to switch from Windows to SteamOS. If you want I'll even do dry ice and LN2 testing for shits and giggles.
1) By not testing the old stuff you're only going to test the latest and greatest, which gives absolutely no comparison for the people who are shopping around looking to replace their existing graphics cards. Such as the people who are looking to replace their old 4770 with something new in the same price range that they bought that card for 5 years ago (e.g. my brother in law). If you can look at the performance difference between a 4770 and a 6850 on the same graph, that's good information to have.

2) Broke high school and college students are the large majority of people who are overclocking their gear. Anyone in the business world would NEVER over/under-clock a machine that they depended on. Guess which group has more money to spend on hardware (and is worth advertising to).

3) Any time you get into benchmarking overclocked/underclocked systems, you are by definition testing on a configuration that is not supported by the manufacturer, and likely not reproducible by others... Why would you ever bother, if you're attempting to be taken seriously?

4) Power draw, thermals, and noise levels are useful... I'll give you that one.

5) Even if Michael's not testing the latest and greatest games out there, he's still giving benchmarks of relative improvements in the drivers supporting the various GPUs that are available. I know that TF2 was basically unplayable a year ago on my 6850. I know that the driver has gotten much better by looking at the benchmarks on this site and seeing the relative improvements in most games. Therefore, I know that it's probably worth trying TF2 again on my system now.

Michael's limited to mostly using automated benchmark which are reproducible, and can be run with a minimum of user involvement. PTS is also designed so that you can run the whole thing via an ssh tunnel to a test farm in a server room if needed (something that those paying commercial customers are after, and also has come in useful for me a few times). If more games start shipping with automated benchmarking modes, then you'll see more games with PTS profiles that get run in articles like this. Manually running through levels in games w/ FRAPS to record frame rates CAN be vaguely reproducible, but every time you see it in a windows GPU review, there's usually a large disclaimer about how they believe the numbers are reproducible, but they can't guarantee it.

Recording and replaying api traces would give you reproducible benchmarks with regards to how fast your CPU, GPU and driver can render frames, but it won't necessarily guarantee a useful framerate number that you can then use to go shopping for components. Yes, PTS is currently limited to either free/oss games which are not the latest and greatest, or commercial games which can be difficult to test in an automated manner... but given the resource constraints at hand, it's pretty damn good.