Mac OS X 10.6.3 Packs Some Regressions

Yesterday we published Mac OS X 10.6.2 vs. Ubuntu 10.04 benchmarks. Overall, both the latest Mac OS X and Ubuntu Linux operating systems were competitive with one another, but there were a few strong points to each OS. As luck would have it, Apple finally introduced the Mac OS X 10.6.3 update that brought a variety of maintenance updates to Snow Leopard. There were a few rumored performance-related changes as well as addressing "compatibility issues with OpenGL-based applications" that was supposed to mean graphics driver updates, so we decided to run our automated tests atop Mac OS X 10.6.3 to look for any differences.

The same Mac Mini from yesterday's tests was used in the Mac OS X 10.6.2 performance article, which included an Intel Core 2 Duo P7350 processor clocked at 2.50GHz, 1GB of DDR3 system memory, a 120GB Fujitsu MHZ2120BH G1 HDD, and NVIDIA GeForce 9400 graphics. Mac OS X 10.6.3 bumps the kernel version from 10.2.0 to 10.3.0 and it takes the X.Org Server from 1.4.2-apple45 to 1.4.2-apple53. Through the Phoronix Test Suite, we ran the same set of tests as we did for yesterday's article. However, there were not many changes to note. In fact, the changes came down to OpenGL regressions. In our computational and compression tests along with other areas like ray-tracing, video playback, and cryptography, the performance results between Mac OS X 10.6.2 and 10.6.3 were virtually identical. Even most of our OpenGL gaming tests performed the same between these two Snow Leopard point releases. Below is an example from our run with OpenArena where the numbers are almost the same (but a very small drop in frame-rate with the lower resolutions).

One of the areas though where the Mac OS X 10.6.3 results became interesting was with Warsow.

As you can see, there is quite the disparity in test results between Mac OS X 10.6.2 and 10.6.3. With the lower resolutions, the delta was much more significant and in fact made this open-source game unplayable under Mac OS X 10.6.3 with the Mac Mini using NVIDIA GeForce 9400 graphics. What is also signifying a problem with Apple's 10.6.3 update is the frame-rate increasing at higher resolutions rather than decreasing as it should. At 1920 x 1080, Warsow was actually faster than when it was running at 800 x 600. For those unfamiliar with the Phoronix Test Suite, at each resolution the game is tested three times for accuracy (or more, if the deviation between the runs is considered significantly different).

With X-Plane 9 it is another example of the Mac OS X 10.6.3 operating system regressing. The frame-rate at 1920 x 1080 dropped by nearly 40% with this OpenGL test using the newest Apple maintenance update.

Certainly, these performance drops were not what we were expecting when Apple mentioned, "address compatibility issues with OpenGL-based applications", but this is the case at least with the NVIDIA graphics between the 10.6.2 and 10.6.3 releases. In some tests like Urban Terror and OpenArena there were not these massive losses. Fortunately for Apple (and anyone else), via the Phoronix Test Suite it should be as easy as running phoronix-test-suite batch-benchmark warsow xplane9 for reproducing these performance regressions. Beyond that, there were not any other areas we tested where the performance was significantly different from Mac OS X 10.6.2, which means Ubuntu 10.04 LTS still should be running neck-and-neck with this latest Apple operating system. Next up though is seeing Microsoft's Windows 7 can compete with these other operating systems.