pts/dbench vs system/dbench

[tytso]

I'm trying to figure out why dbench wasn't run in my local/fs-benchmark test suite. It was working previously; but now it's being silently skipped.

While trying to figure out why, I noticed this discrepancy:

root@pts-perf:~# phoronix-test-suite list-installed-tests | grep dbench
pts/dbench-1.0.0 - Dbench
pts/dbench-1.0.1 - Dbench
root@pts-perf:~# phoronix-test-suite list-available-tests | grep dbench
system/dbench - Dbench Disk

Did the dbench test suite change from pts/dbench to system/dbench? I note this isn't true with, say, fs-mark:

root@pts-perf:~# phoronix-test-suite list-installed-tests | grep fs-mark
pts/fs-mark-1.0.2 - FS-Mark
root@pts-perf:~# phoronix-test-suite list-available-tests | grep fs-mark
pts/fs-mark - FS-Mark Disk

Could this have something to do with the fact that dbench is marked as deprecated?

Where else should I look to understand why dbench was skipped despite the fact that it's in the suite definition file.

# cat test-suites/local/fs-benchmarks/suite-definition.xml
...
<Execute>
<Test>pts/dbench</Test>
<Arguments>1</Arguments>
<Description>1 Clients</Description>
</Execute>
<Execute>
<Test>pts/dbench</Test>
<Arguments>12</Arguments>
<Description>12 Clients</Description>
</Execute>
...

This is unchanged from the pts/disk suite definition file, and as I said, this was working before, and isn't working now, and I'm trying to figure out why.

Thanks!!

[Michael]

Hmmm, if you run `phoronix-test-suite info <suite name>` does it show up?

The difference between pts/dbench and system/dbench are that the "system/" tests tend to rely on the system/package-manager-provided packages/software under test rather than building from source. Though that shouldn't be interfering with anything in the pts/* namespace for instance and you could even run both from the same suite.

If dbench shows up in the info output mentioned above, does it run fine if you do: phoronix-test-suite debug-run pts/dbench ?

[tytso]

Hmm, so one the problems is that the first time I ran it, the first hint of trouble was when I ran "phoronix-test-suite batch-run fs-benmarks" and it printed "Test 1 of 9". When I let it finished, I realized it had skipped the two dbench benchmarks. (Note: local/fs-benchmark started as a fork of pts/disk, but I had removed the FIO benchmarks, since those benchmarks essentially never changed across many kernel versions, and in the case of large I/O workloads, especially large Direct I/O workloads, it was more about measuring the underlying disk speeds and not the file system overhead, which on file systems like xfs and ext4, tend to be closer to zero especially for DIO.) Previously, fs-benchmark would run 11 tests, so it would say, "Test 1 of 11"... "Test 11 of 11".

I tried using "pts info pts/dbench", and it showed the test successfully. I then ran "pts info local/fs-benchmark" and it reported:

Phoronix Test Suite v9.8.0
FS Benchmarks

Suite Description: File system benchmarks

Run Identifier: local/fs-benchmarks
Suite Version: 1.0.0
Maintainer: Theodore Ts'o
Status:
Suite Type: Disk
Unique Tests: 5
Contained Tests:
SQLite
FS-Mark Test: 1000 Files, 1MB Size
FS-Mark Test: 1000 Files, 1MB Size, No Sync/FSync
FS-Mark Test: 5000 Files, 1MB Size, 4 Threads
FS-Mark Test: 4000 Files, 32 Sub Dirs, 1MB Size
Compile Bench Test: Initial Create
Compile Bench Test: Compile
Compile Bench Test: Read Compiled Tree
Dbench 1 Clients
Dbench 12 Clients
PostMark
18 Tests / 5 Unique Tests

18 Tests? Huh? Then when I tried running batch-run fs-benchmarks, it reported:

FS-Mark 3.3:
pts/fs-mark-1.0.2 [Test: 1000 Files, 1MB Size]
Test 1 of 10
^^^^^^^

Huh? It was earlier saying Test of 1 of 9, and it ran 9 tests. (When it should have been 11.) How it's saying 10 tests? So I typed control C, and did a few more phoronix-test-suite info commands, and then when I tried the batch-run again, it said, running "Test 1 of 11". So there is some kind of Heisinbug going on here.

For now, I've created a local/fs-bench suite where I use system/dbench instead of pts/dbench, and I've also forked pts/fs-mark to create a local/fs-mark which runs fs-mark using both 32k files as well as 1MB files. The reason why I want a smaller file size is I want things to be more obvious when there are improvements in the metadata overhead, as opposed to data writes which are just measuring disk speeds, and not file system overhead. I'll try to switch back to pts/dbench but the problem around 3 hours to do a full benchmark run, and dbench happens near the end of the suite. It would be nice if there was a mode where it prints the tests that it would have run, since apparently the number of tests reported by "pts info <test-suite>" seems to have at best a loose connection to the number of tests *actually* run when "pts batch-run <test-suite>" is executed.

[Michael]

Sorry for the delay, tytso finally finding the time again to look at it.

Regarding the pts/dbench issue in the suite, that is happening since the test was marked as 'deprecated' and so the current behavior is to not include deprecated tests in suites, but cleaning that up with the next PTS release... But in any case since dbench is still being used, I just updated the test profile to mark it no longer as deprecated. So once your system(s) see pts/dbench-1.0.3 it won't appear as deprecated and thus should work fine.

Regarding your forked fs-mark test profile, I'd be happy to carry any patches like that upstream if desired to simplify your workflow and improve the upstream test.

And then for the 'tests' count, it basically comes down to some unclear verbage and some cases historical artifacts:

Originally PTS would only allow for 1 test output/result per test being run, that was improved years ago for tests that may report both say throughput and latency at the same time. So PTS properly deals with one test run outputting multiple test results and some areas of PTS are reporting the 'Tests' as in the tests being run (knowing that particular test is providing multiple results) while other areas are reporting the total count / total combinations in full. So internally it's all being handled correctly and formerly when everything was 1 test run = 1 result was easy in terms of saying 'Test XYZ count', but for lack of clear verbage in some areas that is why right now it may appear to be a discrepency in the test count being reported. I'll try to clear up the language and make the count being shown more uniform or clear to the user at least.