Tweet
Interesting discussion overall, with some good points all around.
I'm thinking that indeed it makes the most sense to select a distribution based on what you're benchmarking. If you're benchmarking games, use SteamOS or Gears on Gallium. If you're benchmarking new hardware, use Arch or Tumbleweed. If you're benchmarking compiler/filesystem settings... Hm, well, that's harder to decide. Who is the target audience? Probably not regular users, and Arch makes the most sense then; maybe Fedora and Clear Linux. Same with compute: it will be mostly used on CentOS, but you hardly ever want to benchmark old versions, so Fedora makes the most sense. If you're benchmarking desktop environments, Ubuntu makes the most sense, since only it has (actual) support for Unity. If you're benchmarking the evolution of performance over time, some Ubuntu spin (Linux Mint perhaps) makes the most sense, since it has LTS releases that are still supported and a semiannual release cycle.
It's true that for reproducibility, rolling releases may not be optimal, but then PTS already reports all the versions of relevant software. The next best thing would be a distro with a fast release cycle and new packages; Fedora, maybe.
Overall there is not a whole lot of point in benchmarking Ubuntu by default, since it's not really the target audience for most of the benchmarks. It seems that Michael likes to use the PPA system for testing latest Mesa, but then Gears on Gallium/Tumbleweed is a better point of reference. Of course, for the most part there is not a whole lot of difference in performance anyway.
I'm thinking that indeed it makes the most sense to select a distribution based on what you're benchmarking. If you're benchmarking games, use SteamOS or Gears on Gallium. If you're benchmarking new hardware, use Arch or Tumbleweed. If you're benchmarking compiler/filesystem settings... Hm, well, that's harder to decide. Who is the target audience? Probably not regular users, and Arch makes the most sense then; maybe Fedora and Clear Linux. Same with compute: it will be mostly used on CentOS, but you hardly ever want to benchmark old versions, so Fedora makes the most sense. If you're benchmarking desktop environments, Ubuntu makes the most sense, since only it has (actual) support for Unity. If you're benchmarking the evolution of performance over time, some Ubuntu spin (Linux Mint perhaps) makes the most sense, since it has LTS releases that are still supported and a semiannual release cycle.
It's true that for reproducibility, rolling releases may not be optimal, but then PTS already reports all the versions of relevant software. The next best thing would be a distro with a fast release cycle and new packages; Fedora, maybe.
Overall there is not a whole lot of point in benchmarking Ubuntu by default, since it's not really the target audience for most of the benchmarks. It seems that Michael likes to use the PPA system for testing latest Mesa, but then Gears on Gallium/Tumbleweed is a better point of reference. Of course, for the most part there is not a whole lot of difference in performance anyway.
Comment