Test run number does not correlate to deviation

[mateusz_bl]

Hi

I've been running Redis benchmark recently and I've found out that there's something weird with the Run count. From what I found out, Run count is usually set to 3, unless the deviation is high - then the test is repeated (no more than X times).

In my case usually each Redis test (like GET, SET) is executed 15 times (so I guess that's the maximum Run count). That would be OK if I didn't notice that sometimes the test is executed less than 15 times but with a high deviation as well. Two examples:

1. Test with pretty low (3.77%) deviation; executed 15 times:

Redis 6.0.9:
pts/redis-1.3.1 [Test: LPOP]
Test 1 of 5
Estimated Trial Run Count: 3
Estimated Test Run-Time: 1 Minute
Estimated Time To Completion: 5 Minutes [05:08 PDT]
Started Run 1 @ 05:03:35
...
Started Run 15 @ 05:08:38 *

Test: LPOP:
1352482.5
1527436.75
1544173
1518607.75
1440539.38
1547039.5
1436016
1464175.75
1523002.12
1563477.25
1511287.88
1499724
1501050.75
1541787.25
1550682.62

Average: 1501432.17 Requests Per Second
Deviation: 3.77%
Samples: 15

2. Test with high (7.43%) deviation; executed only 12 times:

Redis 6.0.9:
pts/redis-1.3.1 [Test: GET]
Test 4 of 5
Estimated Trial Run Count: 3
Estimated Test Run-Time: 1 Minute
Estimated Time To Completion: 2 Minutes [16:22 UTC]
Started Run 1 @ 16:21:07
...
Started Run 12 @ 16:24:43 *

Test: GET:
2108841.75
1970863.25
2158901.5
1975906.75
2194447.25
2176292.5
2145050
2106593.5
2074728.5
1732501.75
1774308
2074258.38

Average: 2041057.76 Requests Per Second
Deviation: 7.43%
Samples: 12

My question is: why in the first example the test was repeated so many times if in the second example (with higher deviation) the test stopped after 12 runs? How is this calculated?

I was using Phoronix Test Suite v10.2.2. I didn't change any default configuration variables (like FORCE_TIMES_TO_RUN). And I didn't create user-config.xml file as well.

Thanks!

[Michael]

Short (time) running tests will run up to 5x the default run count if necessary to obtain a more accurate result. However, after 3~4x it will re-evaluate the deviation. If the deviation isn't tightening or the results otherwise appearing sporadic where it's unlikely to provide a more accurate result, it will bail out early. This is likely what you are seeing with the run is it exiting early since it looks like it can't obtain a more accurate result.