Phoromatic: Run arbitary benchmark fails: [PROBLEM] Communication with server failed.

[markus.leitold]

Hi everybody,

we just came accross this weird behaviour:

We are running a phoromatic server (/phoronix-test-suite/phoronix-test-suite start-phoromatic-server) and at another machine a phoromatic client (/phoronix-test-suite/phoronix-test-suite phoromatic.connect serverFQDN:31042/6S5X8A)

These are the config XMLs:
Server:

<?xml version="1.0"?>
<!--Phoronix Test Suite v10.8.3-->
<PhoronixTestSuite>
  <Options>
    <OpenBenchmarking>
      <AnonymousUsageReporting>FALSE</AnonymousUsageReporting>
      <IndexCacheTTL>3</IndexCacheTTL>
      <AlwaysUploadSystemLogs>FALSE</AlwaysUploadSystemLogs>
      <AllowResultUploadsToOpenBenchmarking>FALSE</AllowResultUploadsToOpenBenchmarking>
    </OpenBenchmarking>
    <General>
      <DefaultBrowser></DefaultBrowser>
      <UsePhodeviCache>FALSE</UsePhodeviCache>
      <DefaultDisplayMode>DEFAULT</DefaultDisplayMode>
      <PhoromaticServers></PhoromaticServers>
      <ColoredConsole>AUTO</ColoredConsole>
    </General>
    <Modules>
      <AutoLoadModules>toggle_screensaver, update_checker, perf_tips, ob_auto_compare, load_dynamic_result_viewer</AutoLoadModules>
    </Modules>
    <Installation>
      <RemoveDownloadFiles>FALSE</RemoveDownloadFiles>
      <SearchMediaForCache>TRUE</SearchMediaForCache>
      <SymLinkFilesFromCache>FALSE</SymLinkFilesFromCache>
      <PromptForDownloadMirror>FALSE</PromptForDownloadMirror>
      <EnvironmentDirectory>~/.phoronix-test-suite/installed-tests/</EnvironmentDirectory>
      <CacheDirectory>~/.phoronix-test-suite/download-cache/</CacheDirectory>
    </Installation>
    <Testing>
      <SaveSystemLogs>TRUE</SaveSystemLogs>
      <SaveInstallationLogs>TRUE</SaveInstallationLogs>
      <SaveTestLogs>TRUE</SaveTestLogs>
      <SleepTimeBetweenTests>6</SleepTimeBetweenTests>
      <RemoveTestInstallOnCompletion>FALSE</RemoveTestInstallOnCompletion>
      <ResultsDirectory>~/.phoronix-test-suite/test-results/</ResultsDirectory>
      <AlwaysUploadResultsToOpenBenchmarking>FALSE</AlwaysUploadResultsToOpenBenchmarking>
      <AutoSortRunQueue>TRUE</AutoSortRunQueue>
      <ShowPostRunStatistics>TRUE</ShowPostRunStatistics>
    </Testing>
    <TestResultValidation>
      <DynamicRunCount>TRUE</DynamicRunCount>
      <LimitDynamicToTestLength>20</LimitDynamicToTestLength>
      <StandardDeviationThreshold>2.5</StandardDeviationThreshold>
      <ExportResultsTo></ExportResultsTo>
      <MinimalTestTime>2</MinimalTestTime>
      <DropNoisyResults>FALSE</DropNoisyResults>
    </TestResultValidation>
    <ResultViewer>
      <WebPort>RANDOM</WebPort>
      <LimitAccessToLocalHost>TRUE</LimitAccessToLocalHost>
      <AccessKey></AccessKey>
      <AllowSavingResultChanges>TRUE</AllowSavingResultChanges>
      <AllowDeletingResults>TRUE</AllowDeletingResults>
    </ResultViewer>
    <BatchMode>
      <SaveResults>TRUE</SaveResults>
      <OpenBrowser>FALSE</OpenBrowser>
      <UploadResults>TRUE</UploadResults>
      <PromptForTestIdentifier>TRUE</PromptForTestIdentifier>
      <PromptForTestDescription>TRUE</PromptForTestDescription>
      <PromptSaveName>TRUE</PromptSaveName>
      <RunAllTestCombinations>TRUE</RunAllTestCombinations>
      <Configured>FALSE</Configured>
    </BatchMode>
    <Networking>
      <NoInternetCommunication>TRUE</NoInternetCommunication>
      <NoNetworkCommunication>FALSE</NoNetworkCommunication>
      <Timeout>20</Timeout>
      <ProxyAddress></ProxyAddress>
      <ProxyPort></ProxyPort>
      <ProxyUser></ProxyUser>
      <ProxyPassword></ProxyPassword>
    </Networking>
    <Server>
      <RemoteAccessPort>31042</RemoteAccessPort>
      <Password></Password>
      <WebSocketPort>31043</WebSocketPort>
      <AdvertiseServiceZeroConf>TRUE</AdvertiseServiceZeroConf>
      <AdvertiseServiceOpenBenchmarkRelay>FALSE</AdvertiseServiceOpenBenchmarkRelay>
      <PhoromaticStorage>~/.phoronix-test-suite/phoromatic/</PhoromaticStorage>
    </Server>
  </Options>
</PhoronixTestSuite>

Client:

<?xml version="1.0"?>
<!--Phoronix Test Suite v10.8.3-->
<PhoronixTestSuite>
  <Options>
    <OpenBenchmarking>
      <AnonymousUsageReporting>FALSE</AnonymousUsageReporting>
      <IndexCacheTTL>3</IndexCacheTTL>
      <AlwaysUploadSystemLogs>FALSE</AlwaysUploadSystemLogs>
      <AllowResultUploadsToOpenBenchmarking>FALSE</AllowResultUploadsToOpenBenchmarking>
    </OpenBenchmarking>
    <General>
      <DefaultBrowser></DefaultBrowser>
      <UsePhodeviCache>FALSE</UsePhodeviCache>
      <DefaultDisplayMode>DEFAULT</DefaultDisplayMode>
      <PhoromaticServers>serverFQDN:31042</PhoromaticServers>
      <ColoredConsole>AUTO</ColoredConsole>
    </General>
    <Modules>
      <AutoLoadModules>toggle_screensaver, update_checker, perf_tips, ob_auto_compare, load_dynamic_result_viewer</AutoLoadModules>
    </Modules>
    <Installation>
      <RemoveDownloadFiles>FALSE</RemoveDownloadFiles>
      <SearchMediaForCache>TRUE</SearchMediaForCache>
      <SymLinkFilesFromCache>FALSE</SymLinkFilesFromCache>
      <PromptForDownloadMirror>FALSE</PromptForDownloadMirror>
      <EnvironmentDirectory>~/.phoronix-test-suite/installed-tests/</EnvironmentDirectory>
      <CacheDirectory>~/.phoronix-test-suite/download-cache/</CacheDirectory>
    </Installation>
    <Testing>
      <SaveSystemLogs>TRUE</SaveSystemLogs>
      <SaveInstallationLogs>TRUE</SaveInstallationLogs>
      <SaveTestLogs>TRUE</SaveTestLogs>
      <SleepTimeBetweenTests>6</SleepTimeBetweenTests>
      <RemoveTestInstallOnCompletion>FALSE</RemoveTestInstallOnCompletion>
      <ResultsDirectory>~/.phoronix-test-suite/test-results/</ResultsDirectory>
      <AlwaysUploadResultsToOpenBenchmarking>FALSE</AlwaysUploadResultsToOpenBenchmarking>
      <AutoSortRunQueue>TRUE</AutoSortRunQueue>
      <ShowPostRunStatistics>TRUE</ShowPostRunStatistics>
    </Testing>
    <TestResultValidation>
      <DynamicRunCount>TRUE</DynamicRunCount>
      <LimitDynamicToTestLength>20</LimitDynamicToTestLength>
      <StandardDeviationThreshold>2.5</StandardDeviationThreshold>
      <ExportResultsTo></ExportResultsTo>
      <MinimalTestTime>2</MinimalTestTime>
      <DropNoisyResults>FALSE</DropNoisyResults>
    </TestResultValidation>
    <ResultViewer>
      <WebPort>RANDOM</WebPort>
      <LimitAccessToLocalHost>TRUE</LimitAccessToLocalHost>
      <AccessKey></AccessKey>
      <AllowSavingResultChanges>TRUE</AllowSavingResultChanges>
      <AllowDeletingResults>TRUE</AllowDeletingResults>
    </ResultViewer>
    <BatchMode>
      <SaveResults>TRUE</SaveResults>
      <OpenBrowser>FALSE</OpenBrowser>
      <UploadResults>TRUE</UploadResults>
      <PromptForTestIdentifier>TRUE</PromptForTestIdentifier>
      <PromptForTestDescription>TRUE</PromptForTestDescription>
      <PromptSaveName>TRUE</PromptSaveName>
      <RunAllTestCombinations>TRUE</RunAllTestCombinations>
      <Configured>FALSE</Configured>
    </BatchMode>
    <Networking>
      <NoInternetCommunication>TRUE</NoInternetCommunication>
      <NoNetworkCommunication>FALSE</NoNetworkCommunication>
      <Timeout>20</Timeout>
      <ProxyAddress></ProxyAddress>
      <ProxyPort></ProxyPort>
      <ProxyUser></ProxyUser>
      <ProxyPassword></ProxyPassword>
    </Networking>
    <Server>
      <RemoteAccessPort>31042</RemoteAccessPort>
      <Password></Password>
      <WebSocketPort>31043</WebSocketPort>
      <AdvertiseServiceZeroConf>TRUE</AdvertiseServiceZeroConf>
      <AdvertiseServiceOpenBenchmarkRelay>TRUE</AdvertiseServiceOpenBenchmarkRelay>
      <PhoromaticStorage>~/.phoronix-test-suite/phoromatic/</PhoromaticStorage>
    </Server>
  </Options>
</PhoronixTestSuite>

After starting the client it is successfully registered at the server, the client output sais it is "Idling, waiting for task assignment...."

Now we want to run a single shot benchmark. Therefore we use "Testing / Run a Benchmark" from the menu, we fill out the form and click at "Run Benchmark"
After this the benchmark ticket is created, which can also be found in the sqlite DB:

SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
sqlite> select * from phoromatic_benchmark_tickets;
6S5X8A|926830|1708005398|DBench|.SYSTEM|zbs-dbench-1.0.0||1|ptsadmin|2024-02-15 13:56:38|8EIMB||​

But after waiting for 1-2 minutes no test is started at the client but it just gives this output:
grafik.png

Both ports web and websocket are reachable from the client. Checked this with a simple telnet.

At server side this can be found in phoromatic.log:
"8EIMB - needs to benchmark ticket for DBench"

Why is the server complaining the ticket? The Client ID (8EIMB) is correct.

At this point the Server GUI is quit laggy, at client side the "[PROBLEM]" - Entries are repeating endlessly.

Any idea what is going wrong here? Is there a way to get more debug output from server / client?

BTW: Just allowing one single attachment / screenshot in this forum makes describing a problem much harder...

Thx in advance
Markus

[markus.leitold]

OK finally I found the root cause by myself during hours of debugging the pts code ... a combination of multiple things:

Important in this case: The phoromatic server runs in an air-gapped environment, so no internet communication is available at all. (For this, we set "NoInternetCommunication" to "TRUE").

When it comes to the point that tests should be started via "Run Benchmark Test", this is what happens server-side:
-> Look at "IndexCacheTTL" in the user config... If this is exceeded, refresh the index cache
--> First weird thing / bug: Despite "NoInternetCommunication" is set, the server tries to refresh the cache, which for us fails. In the meantime the request from the client is timed out, the "[PROBLEM]" - Entries occur, causing the client to repeat the request, the server again fails to refresh the cache .... endless loop...

Then I thought to simply set "IndexCacheTTL" to "0" (which was "3" before, look above), as regarding the code should lead to skip the server refreshing the OBO Index Cache.
But then multiple bugs hit concerning the way PHP handles "0" in control structures like "if" or the empty - function. In the end, the setted "0" for "IndexCacheTTL" never was respected, so the server again tried to refresh caches, which fails like described.

I made a few dirty hacks to the code and was able to make the server respect the setted "0" and now tests are running again.
Will file a detailed bug report for this here: https://github.com/phoronix-test-sui...t-suite/issues