EXT4 Lets Us Down, There Goes Our R600/700 Mesa Tests

For the past several days benchmarks have been going on a plethora of ATI Radeon HD 2000/3000/4000 (R600/700 generation) graphics cards as well as some of the older Radeon X1000 (R500) hardware for reference. All of this testing has been done with the current open-source ATI driver stack with Mesa to show where the performance is at for the H1'2010 Linux distributions. At our disposal is quite a collection of graphics cards, but to much dismay this article has been postponed as the testing process had to be restarted from scratch.

Why? The test results have been lost. The only time this has occurred previously was due to disk drive failure, but this time it's thanks to EXT4 in the Linux 2.6.32 kernel. Originally the EXT4 file-system was a nice evolutionary upgrade to EXT3 (and an incremental solution until Btrfs was ready) that provided some major performance boosts. Though as the adoption of EXT4 increased, it was found that the file-system wasn't always reliable but data loss was somewhat common. This was happening due to delayed allocation with data not being written to the disk before the system went down or crashed, but due to EXT4's design the file being touched would be left empty rather than holding the original file contents.

In kernel releases since the EXT4 data loss problems were widely noted, improvements to EXT4 have been made to improve the integrity of the file-system and making it safer for general desktop usage. These improvements to better preserve data have come at a large performance cost in some tests and the speed of EXT4 continues to degrade even with EXT4 in the Linux 2.6.32 kernel losing speed. However, at least with the default mount options, the EXT4 file-system still is not always reliable.

When testing out the Radeon HD 2400PRO on Linux it locked up with the ioquake3-powered World of Padman game, which was being run through the Phoronix Test Suite. All graphics cards tested until getting to the HD 2400PRO had run fine, but the system locked up and were unable to change to a virtual terminal or do anything besides rebooting the system.

When the system was back up, to much surprise and disappointment, all of the test results were gone. However, it was just not the test results file that was being read to and written from, but all of the test results and files contained within the active directory (~/.phoronix-test-suite/test-results/ati/). From the Phoronix Test Suite XML files to all of the test logs and system logs were no longer present that were found within a sub-directory. Many of these files were not even read from or written to on the most recent mount of the EXT4 file-system. The Phoronix Test Suite even keeps backup copies of the XML test results from previous runs in a separate file to fend off data loss problems like that, but alas they were stored in the same directory. Again, most of the files were not touched since several reboots ago. We're still not sure how the entire directory got lost and no writes should have been pending while the actual OpenGL game was running, but that's what happened and we're now investigating as this issue should have never happened.

As a result, the ATI Radeon Mesa testing had to be restarted from the beginning and so it will be at least another week before these R500/600/700 numbers are published. The total number of cards to be tested may also be cut down. To fend off this scenario from taking place again, support will be added to Phoromatic for an option of remotely syncing test results with an account even if the tests weren't part of the Phoromatic test queue. Therefore if EXT4 decides to let us down again, we'll at least have a copy remotely -- so will you, if you use Phoromatic.

With the Radeon HD 2400PRO we will also be working to reproduce this crash in World of Padman (usually easy with the Mesa drivers) to see if we can make EXT4 again lose an entire directory worth of data. This time though there will be plenty of documentation and video recordings. It's becoming more tempting to switch to XFS, fall-back to EXT3, or just eagerly await the stabilization of Btrfs.