pts/server brings many failed tests, but why?

[Edur]

Hi,

I want to test different hardware servers under different hypervisors (VMware, XCP-ng, Proxmox and so on).

On my fresh installed Ubuntu 20.04 and Phoronix 9.6.0 as root with all dependencies installed the pts/server suite brings many failed tests at the end. I still do not understand why.

The following tests failed to properly run:

- pts/blogbench-1.1.0
- pts/apache-siege-1.0.4: Concurrent Users: 200
- pts/apache-siege-1.0.4: Concurrent Users: 250
- pts/mysqlslap-1.1.0: Clients: 64
- pts/mysqlslap-1.1.0: Clients: 256
- pts/pgbench-1.9.1: Scaling: Buffer Test - Test: Normal Load - Mode: Read Write
- pts/pgbench-1.9.1: Scaling: Buffer Test - Test: Normal Load - Mode: Read Only
- pts/pgbench-1.9.1: Scaling: Buffer Test - Test: Heavy Contention - Mode: Read Write
- pts/pgbench-1.9.1: Scaling: Buffer Test - Test: Heavy Contention - Mode: Read Only
- pts/mcperf-1.3.0: Method: Get - Connections: 1
- pts/mcperf-1.3.0: Method: Get - Connections: 4
- pts/mcperf-1.3.0: Method: Get - Connections: 16
- pts/mcperf-1.3.0: Method: Get - Connections: 32
- pts/mcperf-1.3.0: Method: Get - Connections: 64
- pts/mcperf-1.3.0: Method: Get - Connections: 128
- pts/mcperf-1.3.0: Method: Get - Connections: 256
- pts/mcperf-1.3.0: Method: Set - Connections: 1
- pts/mcperf-1.3.0: Method: Set - Connections: 4
- pts/mcperf-1.3.0: Method: Set - Connections: 16
- pts/mcperf-1.3.0: Method: Set - Connections: 32
- pts/mcperf-1.3.0: Method: Set - Connections: 64
- pts/mcperf-1.3.0: Method: Set - Connections: 128
- pts/mcperf-1.3.0: Method: Set - Connections: 256
- pts/mcperf-1.3.0: Method: Delete - Connections: 1
- pts/mcperf-1.3.0: Method: Delete - Connections: 4
- pts/mcperf-1.3.0: Method: Delete - Connections: 16
- pts/mcperf-1.3.0: Method: Delete - Connections: 32
- pts/mcperf-1.3.0: Method: Delete - Connections: 64
- pts/mcperf-1.3.0: Method: Delete - Connections: 128
- pts/mcperf-1.3.0: Method: Delete - Connections: 256
- pts/mcperf-1.3.0: Method: Add - Connections: 1
- pts/mcperf-1.3.0: Method: Add - Connections: 4
- pts/mcperf-1.3.0: Method: Add - Connections: 16
- pts/mcperf-1.3.0: Method: Add - Connections: 32
- pts/mcperf-1.3.0: Method: Add - Connections: 64
- pts/mcperf-1.3.0: Method: Add - Connections: 128
- pts/mcperf-1.3.0: Method: Add - Connections: 256
- pts/mcperf-1.3.0: Method: Replace - Connections: 1
- pts/mcperf-1.3.0: Method: Replace - Connections: 4
- pts/mcperf-1.3.0: Method: Replace - Connections: 16
- pts/mcperf-1.3.0: Method: Replace - Connections: 32
- pts/mcperf-1.3.0: Method: Replace - Connections: 64
- pts/mcperf-1.3.0: Method: Replace - Connections: 128
- pts/mcperf-1.3.0: Method: Replace - Connections: 256
- pts/mcperf-1.3.0: Method: Append - Connections: 1
- pts/mcperf-1.3.0: Method: Append - Connections: 4
- pts/mcperf-1.3.0: Method: Append - Connections: 16
- pts/mcperf-1.3.0: Method: Append - Connections: 32
- pts/mcperf-1.3.0: Method: Append - Connections: 64
- pts/mcperf-1.3.0: Method: Append - Connections: 128
- pts/mcperf-1.3.0: Method: Append - Connections: 256
- pts/mcperf-1.3.0: Method: Prepend - Connections: 1
- pts/mcperf-1.3.0: Method: Prepend - Connections: 4
- pts/mcperf-1.3.0: Method: Prepend - Connections: 16
- pts/mcperf-1.3.0: Method: Prepend - Connections: 32
- pts/mcperf-1.3.0: Method: Prepend - Connections: 64
- pts/mcperf-1.3.0: Method: Prepend - Connections: 128
- pts/mcperf-1.3.0: Method: Prepend - Connections: 256
- pts/cassandra-1.0.3: Test: Reads
- pts/cassandra-1.0.3: Test: Mixed 1:1
- pts/cassandra-1.0.3: Test: Mixed 1:3
- pts/hbase-1.0.1: Test: Random Write - Clients: 1
- pts/hbase-1.0.1: Test: Random Write - Clients: 4
- pts/hbase-1.0.1: Test: Random Write - Clients: 16
- pts/hbase-1.0.1: Test: Random Write - Clients: 32
- pts/hbase-1.0.1: Test: Random Write - Clients: 64
- pts/hbase-1.0.1: Test: Random Write - Clients: 128
- pts/hbase-1.0.1: Test: Random Write - Clients: 256
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 1
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 4
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 16
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 32
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 64
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 128
- pts/hbase-1.0.1: Test: Async Random Write - Clients: 256
- pts/hbase-1.0.1: Test: Random Read - Clients: 1
- pts/hbase-1.0.1: Test: Random Read - Clients: 4
- pts/hbase-1.0.1: Test: Random Read - Clients: 16
- pts/hbase-1.0.1: Test: Random Read - Clients: 32
- pts/hbase-1.0.1: Test: Random Read - Clients: 64
- pts/hbase-1.0.1: Test: Random Read - Clients: 128
- pts/hbase-1.0.1: Test: Random Read - Clients: 256
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 1
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 4
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 16
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 32
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 64
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 128
- pts/hbase-1.0.1: Test: Async Random Read - Clients: 256
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 1
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 4
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 16
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 32
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 64
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 128
- pts/hbase-1.0.1: Test: Sequential Write - Clients: 256
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 1
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 4
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 16
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 32
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 64
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 128
- pts/hbase-1.0.1: Test: Sequential Read - Clients: 256
- pts/hbase-1.0.1: Test: Increment - Clients: 1
- pts/hbase-1.0.1: Test: Increment - Clients: 4
- pts/hbase-1.0.1: Test: Increment - Clients: 16
- pts/hbase-1.0.1: Test: Increment - Clients: 32
- pts/hbase-1.0.1: Test: Increment - Clients: 64
- pts/hbase-1.0.1: Test: Increment - Clients: 128
- pts/hbase-1.0.1: Test: Increment - Clients: 256

Detailed debug-benchmark example for pts/server (for the blogbench test run)

========================================
BlogBench (Run 1 of 1)
========================================


Test Run Command: cd /var/lib/phoronix-test-suite/installed-tests/pts/blogbench-1.1.0/ && ./blogbench 2>&1


Frequency = 10 secs
Scratch dir = [/var/lib/phoronix-test-suite/installed-tests/pts/blogbench-1.1.0//scratch]
Spawning 3 writers...
Spawning 1 rewriters...
Spawning 5 commenters...
Spawning 100 readers...
Benchmarking for 30 iterations.
The test will run during 5 minutes.

Nb blogs R articles W articles R pictures W pictures R comments W comments
765 2109482 39563 1437440 38788 903026 72176
1148 2162055 21919 1477712 20942 984356 43925
1512 2170516 21159 1487922 19839 1008404 42751
1877 2226106 20899 1529475 19853 1061128 42545
2224 2154031 20598 1496055 19788 994654 35185
2582 2091998 21829 1448854 19210 976793 40336
2955 2144429 21517 1484065 19644 988148 37027
3330 2146309 22660 1496238 21006 992295 33169
3683 2097889 20887 1457541 20149 948628 36808
4049 2141985 21076 1497831 19696 959197 36795
4419 2076129 21376 1459892 20898 953305 41333
4762 2145670 20573 1505589 18778 979357 35926
5074 4229033 19803 1464216 17469 964373 37089
5348 1766341 16842 1244530 16389 812179 33149
5631 1470204 16403 1033168 15385 688086 32955
5799 978613 11487 690249 10459 472871 41676
5920 862194 8894 607487 7616 419768 38092
6021 760746 8249 534361 6474 388973 33196
6127 694702 7773 487914 6744 357874 33512
6228 657251 8122 463625 6399 344213 31464
6313 588556 6707 413481 5869 310963 29057
6396 566554 6543 397936 5113 308895 27252
6484 553720 7090 387925 5332 298897 30501
6556 513290 5602 359542 4527 283012 29249
6629 494587 5818 348222 5044 275289 26033
6706 476339 6004 336231 4762 277005 25950
6777 415253 5598 294945 4877 242901 22606
6845 437169 5408 310719 4529 252197 23289
6924 444836 6009 316203 4901 253103 25789
6995 393655 5853 279927 4537 229770 22105

Final score for writes: 6995
Final score for reads : 795953


Failed Initial Check For Matching: not in READ


Failed Initial Check For Matching: not in WRITE


The test run did not produce a result.


Log File At: /var/lib/phoronix-test-suite/installed-tests/pts/blogbench-1.1.0/blogbench-1.1.0-1590262609-1.log

The "result lines" are correct and matches with "/var/lib/phoronix-test-suite/test-profiles/pts/blogbench-1.1.0/results-definition.xml" I cannot understand "Failed Initial Check For Matching".

phoronix-test-suite debug-benchmark pts/blogbench-1.1.0

brings this output, slightly different (errors also here, but still results)

# phoronix-test-suite debug-benchmark pts/blogbench-1.1.0


Evaluating External Test Dependencies .................................................. .................................................. .................................................. .........................................

Phoronix Test Suite v9.6.0

Installed: pts/blogbench-1.1.0


BlogBench 1.1:
pts/blogbench-1.1.0
Disk Test Configuration
1: Read
2: Write
3: Test All Options
** Multiple items can be selected, delimit by a comma. **
Test: 3


========================================
Phoronix Test Suite v9.6.0
System Information
========================================


PROCESSOR: 8 x Intel Xeon E5-2620 v4
Core Count: 8
Extensions: SSE 4.2 + AVX2 + AVX + RDRAND + FSGSBASE
Cache Size: 20480 KB
Microcode: 0xb000038

GRAPHICS: Cirrus Logic GD 5446
Screen: 1024x768

MOTHERBOARD: Xen HVM domU v4.13
BIOS Version: 4.13
Chipset: Intel 440FX 82441FX PMC

MEMORY: 1 x 16380 MB RAM

DISK: 49GB
File-System: ext4
Mount Options: relatime rw
Disk Scheduler: MQ-DEADLINE

OPERATING SYSTEM: Ubuntu 20.04
Kernel: 5.4.0-31-generic (x86_64)
Compiler: GCC 9.3.0
System Layer: Xen HVM domU 4.13
Security: itlb_multihit: KVM: Vulnerable
+ l1tf: Mitigation of PTE Inversion
+ mds: Mitigation of Clear buffers; SMT Host state unknown
+ meltdown: Mitigation of PTI
+ spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp
+ spectre_v1: Mitigation of usercopy/swapgs barriers and __user pointer sanitization
+ spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: disabled RSB filling
+ tsx_async_abort: Mitigation of Clear buffers; SMT Host state unknown


========================================
BlogBench (Run 1 of 1)
========================================


Test Run Command: cd /var/lib/phoronix-test-suite/installed-tests/pts/blogbench-1.1.0/ && ./blogbench READ 2>&1

########################################
BlogBench:
Test: Read

777600 Final Score

Average: 777600 Final Score
########################################


Result compared to 799 OpenBenchmarking.org samples since 4 December 2018; median result: 514314. Box plot of samples:
[---------##############*#######!#####*######*##*########### ####################----*----------------------*-*---------*----------------------------------| ]
This Result (60th Percentile): 777600 ^
ADATA SX8200PNP: 727149 ^ 8 x 300GB H106030SDSUN300G: 1360222 ^ 4 x 2048GB MR9460-16i: 1930331 ^
^ Samsung SSD 970 EVO Plus: 606079 Samsung SSD 970 PRO: 1768747 ^
^ INTEL SSDPEKNW512G8: 382654 INTEL SSDPED1D280GA: 1723568 ^

Failed Initial Check For Matching: WRITE not in READ


Result Key: #_RESULT_#


Template Line: Final score for writes: #_RESULT_#


Result Parsing Search Key: "writes"


Result Line: Final score for writes: 7153


Test Result Parser Returning: 7153


########################################
BlogBench:
Test: Write

7153 Final Score

Average: 7153 Final Score
########################################


Result compared to 797 OpenBenchmarking.org samples since 4 December 2018; median result: 3631. Box plot of samples:
[|-*---*---*-##!##############*###############################* #################*#####*####*#---------------------------------------------------*-------------------------*------*-*----------------------------------------| ]
^ This Result (57th Percentile): 7153
^ 2 x 32GB SATA SSD: 2575 SSDSCKJB120G7R: 14978 ^ Samsung SSD 970 EVO: 22087 ^ Samsung SSD 970 PRO: 34899 ^ INTEL SSDPED1D280GA: 43456 ^
^ INTEL SSDSC2KF25: 1793 TOSHIBA-RD400: 20845 ^ Force MP600: 42771 ^
^ QEMU HDD: 815 INTEL SSDPE2ME800G4: 19332 ^ INTEL SSDPED1D960GAY: 41153 ^

Other tests such as mysql etc. seem to have their own problems such as the daemon not starting or its socketfile not being found under /tmp/..., again interestingly the test works when called directly and manually.

Is there any documentation that can help you a bit more with debugging?

[Michael]

The blogbench issue should be fixed next time you run `phoronix-test-suite openbenchmarking-refresh` and then run the server suite, it wasn't passing some needed arguments from the test suite definition.

For other tests that work when called outside of the server suite, would need to know which ones you were able to reproduce that behavior for as opposed to not being able to run at all. All of these tests at least on my side run fine on Ubuntu 20.04. Note that some tests like mysqlslap may have issues running as root due to their default configuration in the name of security.