systemd script does not honor custom ports for server

[amerather]

I am new to Phoronix Test Suite. We run our services in public cloud. We typically build the service as a deb package and bake it into AMI (contains Ubuntu 16 kernel and other packages). We then launch instance (system) with this AMI. Deb package contains all instruction on installing, configuring and starting the service. Also, service is launched as part of auto-scaling group, that means in case of hardware failure, new server with our service is launched. No human intervention is required.

I want to set up both client and server phoronix test suite as a service. I want to perform following tasks:

• When phoronix test suite is installed the first time, It prompts user for: "Do you agree to these terms and wish to proceed (Y/n): y". How do I set user options so that it won't be prompted.
• I have customized "user-config.xml" with custom ports and user options. When starting the server using systemctl it uses default ports and does not take into account changes made in user-config.xml file, located in ~/.phoronix-test-suite. However, when start the daemon from command line: $ phoronix-test-suite start-phoromatic-server, it works fine and use custom ports.
• Since clients come and go, I want to have an automated way to clean up clients on the server so that the server won't unnecessarily poll clients. What files are maintained on server for clients.

Appreciate for any help that you provide.

Amer.

[Michael]

I am new to Phoronix Test Suite. We run our services in public cloud. We typically build the service as a deb package and bake it into AMI (contains Ubuntu 16 kernel and other packages). We then launch instance (system) with this AMI. Deb package contains all instruction on installing, configuring and starting the service. Also, service is launched as part of auto-scaling group, that means in case of hardware failure, new server with our service is launched. No human intervention is required.

I want to set up both client and server phoronix test suite as a service. I want to perform following tasks:

• When phoronix test suite is installed the first time, It prompts user for: "Do you agree to these terms and wish to proceed (Y/n): y". How do I set user options so that it won't be prompted.
• I have customized "user-config.xml" with custom ports and user options. When starting the server using systemctl it uses default ports and does not take into account changes made in user-config.xml file, located in ~/.phoronix-test-suite. However, when start the daemon from command line: $ phoronix-test-suite start-phoromatic-server, it works fine and use custom ports.
• Since clients come and go, I want to have an automated way to clean up clients on the server so that the server won't unnecessarily poll clients. What files are maintained on server for clients.

Appreciate for any help that you provide.

Amer.

1. If you are running it as root, it won't prompt you for that. Or are you launching the services as a user?

2. When running via systemctl, it is /etc/phoronix-test-suite.xml you need to modify due to its privileges.

3. Clients are logged in the Phoromatic Server via a SQLite database.

[amerather]

Thanks for your reply. Yes, it is working as you described. I have few more questions:

• To see if the server starts on the new machine with same account information, I tar up the files: phoromatic.db and accounts directory from the working server and copy it to new server. I restarted the server, but it did not recognized the account setting saved in phoromatic.db (SQLite) and requested to create a new account. What other files are used to keep the server state so that one continue to use the same account ID and keep the old results on new server. If it does not work, I may consider keeping server state on NFS mounted filesystem so that it can be available on new server.
• I see there is a Linux perf module provided in directory: /usr/share/phoronix-test-suite/pts-core/modules. I want to run it during the benchmark. Document states that "LINUX_PERF=1" should be set. Where do I set it?
• Similar question about system monitoring while tests are running. Do we have any module that capture /proc stats similar to the one reported by vmstat, mpstat, pidstat, iostat etc..
• When developing a new benchmark, which directory one should use to install it. Should it go into the same directory where other benchmarks files are installed.

Thanks again for your help.
Amer.

[Michael]

Thanks for your reply. Yes, it is working as you described. I have few more questions:

• To see if the server starts on the new machine with same account information, I tar up the files: phoromatic.db and accounts directory from the working server and copy it to new server. I restarted the server, but it did not recognized the account setting saved in phoromatic.db (SQLite) and requested to create a new account. What other files are used to keep the server state so that one continue to use the same account ID and keep the old results on new server. If it does not work, I may consider keeping server state on NFS mounted filesystem so that it can be available on new server.
• I see there is a Linux perf module provided in directory: /usr/share/phoronix-test-suite/pts-core/modules. I want to run it during the benchmark. Document states that "LINUX_PERF=1" should be set. Where do I set it?
• Similar question about system monitoring while tests are running. Do we have any module that capture /proc stats similar to the one reported by vmstat, mpstat, pidstat, iostat etc..
• When developing a new benchmark, which directory one should use to install it. Should it go into the same directory where other benchmarks files are installed.

Thanks again for your help.
Amer.

1. The phoromatic.db and phoromatic directory is all you should need to transport to another server. Can you provide more information on the error or whatever you ran into?

2. The LINUX_PERF=1 environment variable would need to be set on the client systems prior to the Phoronix Test Suite process starting up. There currently isn't an option to expose this via the Phoromatic Server for clients.

3. Some of those sensors are currently exposed via setting the MONITOR=all environment variable. Patches are welcome for other sensors / log capturing or could talk about some custom engineering support if desired for it and any other features you might want.

4. To get started with a new test profile development, I'd recommend start by modifying an existing test in ~/.phoronix-test-suite/test-profiles/pts/ and copy it to ~/.phoronix-test-suite/test-profiles/local/ to begin test development of a new profile.

[amerather]

Thanks Michael! I will look into server configuration to find what could be missing in phoromatic.db and account directory.
Linux perf will also be useful to capture hot stack traces during cpu benchmarks. perf output can be processed by Brendan's FlameGraph to generate SVG file for visualization. Learn more about it at http://www.brendangregg.com/.

I am currently evaluating Phoronix Test Suite as potential benchmark framework to automate testing new AMI (Ubuntu Kernel) builds. I am planning to add few more benchmarks for Numa memory latency and bandwidth. Questions:

• Do we have any benchmarks available for Tomcat webapp?
• Do you have Jenkins plugin to automate running Test suite after every build? I want to monitor AMI GIT commit as a trigger to start benchmark.

Once I have manager approval, I will spend more time to build additional test suites and contribute to community.

Thanks,
Amer.

[Michael]

Thanks Michael! I will look into server configuration to find what could be missing in phoromatic.db and account directory.
Linux perf will also be useful to capture hot stack traces during cpu benchmarks. perf output can be processed by Brendan's FlameGraph to generate SVG file for visualization. Learn more about it at http://www.brendangregg.com/.

I am currently evaluating Phoronix Test Suite as potential benchmark framework to automate testing new AMI (Ubuntu Kernel) builds. I am planning to add few more benchmarks for Numa memory latency and bandwidth. Questions:

• Do we have any benchmarks available for Tomcat webapp?
• Do you have Jenkins plugin to automate running Test suite after every build? I want to monitor AMI GIT commit as a trigger to start benchmark.

Once I have manager approval, I will spend more time to build additional test suites and contribute to community.

Thanks,
Amer.

Nothing for Tomcat currently available but should presumably not be too difficult to add, I don't have experience with Tomcat myself.

With Phoromatic it's easy to have it fire off a test schedule to execute whenever you ping/pull from a special URL (it's mentioned on the test schedules page), so that way it's pretty easy to automate with Jenkins or to write a few lines of bash script to check if a new Git commit is present, then wget that URL, and just set that on a timer or whatever.

[amerather]

Thanks Michael for all your help. I am slowly making progress. I decided to use NFS as a common directory for both phoronix client and server binaries and data. I also setup all the required links to make sure everything: binaries, results and test suites are all saved in NFS directory. I also setup clients to run test suites on their own at launch time can be triggered manually by the server. Results will be saved in date-stamped directory. Since all clients are using a common phoronix-test-suite.xml file and saving results in common place, I want to name "results" directory to match client's IP address or hostname that one can set via environment variable. Is there way to specify environment variable in phoronix-test-suite.xml file to allow substitution at the time client reads it?

If there is a better approach to achieve this task, please let me know.

Thanks,
Amer.

[Michael]

Thanks Michael for all your help. I am slowly making progress. I decided to use NFS as a common directory for both phoronix client and server binaries and data. I also setup all the required links to make sure everything: binaries, results and test suites are all saved in NFS directory. I also setup clients to run test suites on their own at launch time can be triggered manually by the server. Results will be saved in date-stamped directory. Since all clients are using a common phoronix-test-suite.xml file and saving results in common place, I want to name "results" directory to match client's IP address or hostname that one can set via environment variable. Is there way to specify environment variable in phoronix-test-suite.xml file to allow substitution at the time client reads it?

If there is a better approach to achieve this task, please let me know.

Thanks,
Amer.

See the Phoronix Test Suite HTML/PDF documentation for environment variables, there are a number of options including specifying of the test results identifier when in batch benchmarking mode, so from there you can feed it whatever you want.

[amerather]

Michael,

I read the documents suggested and updated a sample test file (test.sh) below to run sample tests. Tests run fine, but I don't see environment variables have any effect. I continue to see timestamped directory even after doing "export " these variables in test.sh file. All these modules are loaded from /etc/phoronix-test-suite.xml file, as recommended.
"
..
<Modules>
<LoadModules>toggle_screensaver,graphics_event_checker,update_c hecker,linux_perf,system_monitor,perf_per_dollar,r esult_notifier,html_results_export,pushover_net</LoadModules>
</Modules>

When I start the test, I get message:

"
The result_notifier module for providing external hooks has been loaded.
Your Pushover.net user key must be passed via the PUSHOVER_NET_USER environment variable.
Unloading system monitor: No Supported Sensors Selected To Monitor
"
I placed the key that I received after registering at pushover.net site. I also have "MONITOR=all" set in the script.

-----test.sh----
# Set environment variables
TEST_RESULTS_NAME="cpu-benchmark"
TEST_RESULTS_IDENTIFIER=$EC2_INSTANCE_TYPE
TEST_RESULTS_DESCRIPTION="This is a cpu Test"
EXPORT_RESULTS_HTML_EMAIL_TO="[email protected] "
COST_PERF_PER_DOLLAR="100"
PUSHOVER_NET_USER="userkey"

export PUSHOVER_NET_USER
export EXPORT_RESULTS_HTML_EMAIL_TO
export TEST_RESULTS_NAME
export TEST_RESULTS_IDENTIFIER
export COST_PERF_PER_DOLLAR
export TEST_RESULTS_DESCRIPTION

#export LINUX_PERF=1 # Not required as it will be loaded from xml file
MONITOR=all
PERFORMANCE_PER_WATT=1
EXPORT_RESULTS_HTML_FILE_TO="/efs/amibench/html-results"

#sudo phoronix-test-suite batch-install Suite pts/compress-pbzip2-1.5.0 pts/compress-7zip-1.6.2 pts/encode-mp3-1.5.0 pts/x264-2.0.0 pts/ffmpeg-2.5.0 pts/openssl-1.9.0 pts/gcrypt-1.0.3 pts/smallpt-1.0.2 pts/stream-1.3.1

#sudo phoronix-test-suite batch-run Suite pts/compress-7zip-1.6.2 pts/encode-mp3-1.5.0 pts/x264-2.0.0 pts/ffmpeg-2.5.0 pts/openssl-1.9.0 pts/gcrypt-1.0.3 pts/smallpt-1.0.2 pts/stream-1.3.1 pts/compress-pbzip2-1.5.0
sudo phoronix-test-suite batch-run Suite pts/encode-mp3-1.5.0
sudo phoronix-test-suite batch-run Suite pts/x264-2.0.0
sudo phoronix-test-suite batch-run Suite pts/ffmpeg-2.5.0
sudo phoronix-test-suite batch-run Suite pts/openssl-1.9.0
sudo phoronix-test-suite batch-run Suite pts/gcrypt-1.0.3

-----/etc/phoronix-test-suite.xml --

<?xml version="1.0"?>
<!--Phoronix Test Suite v6.6.1-->
<?xml-stylesheet type="text/xsl" href="xsl/pts-user-config-viewer.xsl"?>
<PhoronixTestSuite>
<Options>
<OpenBenchmarking>
<AnonymousUsageReporting>FALSE</AnonymousUsageReporting>
<IndexCacheTTL>3</IndexCacheTTL>
<AlwaysUploadSystemLogs>FALSE</AlwaysUploadSystemLogs>
</OpenBenchmarking>
<General>
<DefaultBrowser></DefaultBrowser>
<UsePhodeviCache>TRUE</UsePhodeviCache>
<DefaultDisplayMode>DEFAULT</DefaultDisplayMode>
<PhoromaticServers></PhoromaticServers>
<FullOutput>FALSE</FullOutput>
</General>
<Modules>
<LoadModules>toggle_screensaver,graphics_event_che cker,update_checker,linux_perf,system_monitor,perf _per_dollar,result_notifier,html_results_export,pu shover_net</LoadModules>
</Modules>
<Installation>
<RemoveDownloadFiles>FALSE</RemoveDownloadFiles>
<SearchMediaForCache>TRUE</SearchMediaForCache>
<SymLinkFilesFromCache>FALSE</SymLinkFilesFromCache>
<PromptForDownloadMirror>FALSE</PromptForDownloadMirror>
<EnvironmentDirectory>/efs/amibench/Test-Suite/</EnvironmentDirectory>
<CacheDirectory>/efs/amibench/Test-Suite/download-cache/</CacheDirectory>
</Installation>
<Testing>
<SaveSystemLogs>TRUE</SaveSystemLogs>
<SaveInstallationLogs>FALSE</SaveInstallationLogs>
<SaveTestLogs>FALSE</SaveTestLogs>
<RemoveTestInstallOnCompletion></RemoveTestInstallOnCompletion>
<ResultsDirectory>/efs/amibench/Test-Results</ResultsDirectory>
<AlwaysUploadResultsToOpenBenchmarking>FALSE</AlwaysUploadResultsToOpenBenchmarking>
<AutoSortRunQueue>TRUE</AutoSortRunQueue>
</Testing>
<TestResultValidation>
<DynamicRunCount>TRUE</DynamicRunCount>
<LimitDynamicToTestLength>20</LimitDynamicToTestLength>
<StandardDeviationThreshold>3.50</StandardDeviationThreshold>
<ExportResultsTo></ExportResultsTo>
<MinimalTestTime>3</MinimalTestTime>
</TestResultValidation>
<BatchMode>
<SaveResults>TRUE</SaveResults>
<OpenBrowser>FALSE</OpenBrowser>
<UploadResults>FALSE</UploadResults>
<PromptForTestIdentifier>FALSE</PromptForTestIdentifier>
<PromptForTestDescription>FALSE</PromptForTestDescription>
<PromptSaveName>FALSE</PromptSaveName>
<RunAllTestCombinations>TRUE</RunAllTestCombinations>
<Configured>TRUE</Configured>
</BatchMode>
<Networking>
<NoInternetCommunication>FALSE</NoInternetCommunication>
<NoNetworkCommunication>FALSE</NoNetworkCommunication>
<Timeout>20</Timeout>
<ProxyAddress></ProxyAddress>
<ProxyPort></ProxyPort>
</Networking>
<Server>
<RemoteAccessPort>7403</RemoteAccessPort>
<Password></Password>
<WebSocketPort>7401</WebSocketPort>
<AdvertiseServiceZeroConf>TRUE</AdvertiseServiceZeroConf>
<PhoromaticStorage>~/efs/amibench/phoromaticStorage/</PhoromaticStorage>
</Server>
</Options>
</PhoronixTestSuite>

[amerather]

Michael,

As suggested, I reviewed the docs for environment variables. I loaded the required modules in phoronix-test-suite.xml and applied associated and other environment variables in test script below. Tests run fine, but environment variables do not take any affect. I get few messages when test starts:

"
The result_notifier module for providing external hooks has been loaded.
Your Pushover.net user key must be passed via the PUSHOVER_NET_USER environment variable.
Unloading system monitor: No Supported Sensors Selected To Monitor
"
I registered at the pushover.net and set the user key given. Also setting MONITOR=all.


---test.sh---
# Set environment variables
TEST_RESULTS_NAME="cpu-benchmark"
TEST_RESULTS_IDENTIFIER=$EC2_INSTANCE_TYPE
TEST_RESULTS_DESCRIPTION="This is a cpu Test"
EXPORT_RESULTS_HTML_EMAIL_TO="[email protected] "
COST_PERF_PER_DOLLAR="100"
PUSHOVER_NET_USER="ujhfdbh9iy6bzqfcyvwi8js2iphcnc"
export PUSHOVER_NET_USER
export EXPORT_RESULTS_HTML_EMAIL_TO
export TEST_RESULTS_NAME
export TEST_RESULTS_IDENTIFIER
export COST_PERF_PER_DOLLAR
export TEST_RESULTS_DESCRIPTION

# Set module options
#export LINUX_PERF=1 # Not required as it will be loaded from xml file
MONITOR=all
PERFORMANCE_PER_WATT=1
EXPORT_RESULTS_HTML_FILE_TO="/efs/amibench/html-results"

#sudo phoronix-test-suite batch-install Suite pts/compress-pbzip2-1.5.0 pts/compress-7zip-1.6.2 pts/encode-mp3-1.5.0 pts/x264-2.0.0 pts/ffmpeg-2.5.0 pts/openssl-1.9.0 pts/gcrypt-1.0.3 pts/smallpt-1.0.2 pts/stream-1.3.1
#sudo phoronix-test-suite batch-run Suite pts/compress-7zip-1.6.2 pts/encode-mp3-1.5.0 pts/x264-2.0.0 pts/ffmpeg-2.5.0 pts/openssl-1.9.0 pts/gcrypt-1.0.3 pts/smallpt-1.0.2 pts/stream-1.3.1 pts/compress-pbzip2-1.5.0
sudo phoronix-test-suite batch-run Suite pts/encode-mp3-1.5.0
sudo phoronix-test-suite batch-run Suite pts/x264-2.0.0
sudo phoronix-test-suite batch-run Suite pts/ffmpeg-2.5.0