Ubuntu Benchmarking Workloads + Clouds

Besides all the other Ubuntu 15.10 chatter happening this week, which is heavily centered around Unity 8 / Ubuntu Convergence and Snappy topics, there was also talk of "Ubuntu Benchmarking Workloads" in the Ubuntu Cloud track.

The talk was mainly aimed at running performance benchmarks using Juju and Ubuntu in any popular cloud deployment environments.

For those that missed it, a few weeks ago Canonical developers began contributing JuJu support to the Phoronix Test Suite, which is one of the key components to the current Ubuntu Cloud benchmarking strategy. There was also an Apache Cassandra test and other simple JuJu charms used for benchmarking today as part of their new "actions" feature.

How Linux benchmarking of servers and other equipment looked to me years ago...

Long story short, the JuJu charm makes for easy and straightforward, reproducible benchmarks. However, if you're an existing Phoronix Test Suite user, there isn't really anything new brought to the table -- only the easy start-up of new cloud instances for the task, which can be easily scripted by most users/services anyhow. In terms of the rest, it could all easily be achieved by the Phoronix Test Suite 6+ years ago with benchmarking computers half-way around the world and building benchmark farms with Phoromatic, which since then Phoromatic has undergone several significant changes and PTS has continued to evolve with new and significant quarterly updates.

What Does Linux Benchmarking Look Like?... Back In 2009.

A 6+ year old feature of Phoromatic and OpenBenchmarking.org is a feature mentioned during this session that they're working on: a web UI for centrally managing the Juju benchmarks and collecting of the results. Much of it is even around 8+ years old when factoring in the old Phoronix Global that was the pre-OpenBenchmarking.org platform. I would bet that with their forthcoming web UI they'll work to tie it into the Canonical Landscape component, or at least that would be most logical so at least they're commercializing off of it.

Besides the bit of NIH, it's mostly the same issues I had ranted to Jorge Castro years ago at SCALE with regard to Canonical working on Ubuntu benchmarking... For years now each group/team inside Canonical has seemed to work on their own approach to Ubuntu benchmarks, have heard about too many different ones and forks over the many years of UDS/UOS, etc. Up to now I thought that "CheckBox" was the primary Ubuntu QA/test software heard of (which also has some PTS methods) but this latest work on JuJu benchmarking doesn't even seem to incorporate CheckBox or vise-versa. While I have no idea if it's still ongoing, there's been XMir performance measurement tests of (X)Mir using PTS and other independent tests. On the Linaro side there was also playing around with PTS and LAVA for doing their benchmarks over the years. These are just some of the more recent "Ubuntu benchmarking" items that come to mind (and that I could easily Google to confirm today) that I've heard about over the years that are non-centralized, fragmented, and seem to come and go. There's many more that want to go back through old UDS/UOS notes over the years.

Honestly the common denominator I've found over the years from these different Ubuntu/Canonical benchmarking projects is that: 1.) they all use Ubuntu. 2.) They all rely upon the Phoronix Test Suite in some manner for carrying out different performance tests. The fragmentation seems horrible. None of these Ubuntu-led projects even seem to standardize yet on a common format for storing of data: meanwhile, the Phoronix Test Suite XML has been stable for years and the versioning via OpenBenchmarking.org ensures that if you want to run a test comparison today against an old result from years ago ensures the same exact test parameters are utilized.

So long story short, you can now easily run Phoronix-Test-Suite benchmarks via JuJu and Canonical is working towards some of their own benchmarking components to try to better ease Ubuntu Cloud benchmarking. They also have some own JuJu tests like Apache Cassandra.

Benchmarking from bare metal to cloud, regardless of Linux distribution.

For those looking towards all of the automated, reproducible Linux benchmarks that you want just not on Ubuntu but any Linux distribution or even OS X / Solaris / BSD, visit Phoronix-Test-Suite.com. If you want to right now centrally manage many test systems (bare metal), collect all of their results, etc, you can do all of that trivially via Phoromatic, which is built into the latest versions of PTS to allow for straight-forward deployments. You can optionally upload your results to OpenBenchmarking.org to centrally see all public results, run side-by-side comparisons directly, and much more. Making your own test is also quite easy and very worthwhile. OpenBenchmarking.org has over 500 test profiles and 100 test suites and is used by companies around the globe.

The basement Linux server room being run constantly for reference and verification purposes of PTS/OB that's up to around 60 benchmarking systems and constantly stressing upstream, open-source benchmarking software.

Those wanting to watch the Ubuntu Cloud benchmarking video from UOS can find it embedded below.