Stability of Flexible IO Tester
Great work on Phoronoix! I'm really enjoying using it.
I'm trying to use Phoronix to compare performance characteristics of different Linux kernels. In order to ensure that my comparison isn't just looking at noise I planned to run universe-cli several times, and take an average for each kernel.
In general this is working quite well. I've only eye-balled the data, but for most tests, the high and low scores are within better than 5% of each other.
Unfortunately, I've found that the times I get from the Flexible IO Tester benchmark vary by a factor of 2, even on the same kernel. The data is below.
Run 1: 15.28 seconds
Run 2: 21.85 "
Run 3: 11.56 "
Run 4: 18.33 "
Run 5: 11.72 "
As you can see, the variance is extremely high; and I suspect that we're seeing more noise than signal. All of these times were collected using identical hardware, and an identical kernel. To rule out interference from a background process, I ran these in single user mode.
Can anyone help explain what this test is measuring? Can it be reworked to give more stable results?
Here is information on fio in detail: http://freshmeat.net/projects/fio/
That specific test profile isn't too commonly used by Phoronix.
If you notice variance in other tests, you can tamper with the TimesToRun XML tag within the test profile located in pts/test-profiles/<name>.xml if it takes more runs to get a better average. Other tests can also be tweaked differently.
As your testing continues, any patches are certainly welcome.