How to write a testsuite to compare different builds of same software?

[f18m]

Hi,
I discovered about PTS by reading the following article:

https://www.phoronix.com/scan.php?page=news_item&px=Glibc-2.26-Redis-Test

In that article Redis is compared against 2 different glibc versions and with jemalloc enabled.
I would like to reproduce that test locally (actually my target would be to add a test of Redis using tcmalloc!)... I managed to install the testsuite and run
phoronix-test-suite benchmark 1708052-TY-REDISMALL91

But I don't get a nice comparison of "Redis with glibc 2.26" vs "Redis with glibc 2.25"... how can I achieve that?

Thanks!
Francesco

[Michael]

phoronix-test-suite benchmark 1708052-TY-REDISMALL91

<do whatever changes you want to glibc>

phoronix-test-suite force-install 1708052-TY-REDISMALL91 (if wanting to trigger recompile)
phoronix-test-suite benchmark 1708052-TY-REDISMALL91

Or I'm not sure I understand your question?

[f18m]

Ok now that I look more closely at the XML file of that test I realize that to obtain the results with different glibc version no gcc compile trick was used, but instead entirely different OS versions were used!

So yes: I can do the steps you propose to modify glibc settings and re-run the suite but I was looking for something totally automatic that comes up with the performance profiles with the 2 different glibc versions. I.e. something that downloads and compiles v2.25, and then links redis against that, runs the test and then starts from scratch again with version 2.26.

Perhaps this kind of test is too much... but maybe it's easier to run in a completely-automatic fashion a test jemalloc vs tcmalloc ?
To use tcmalloc you just need to do:

LD_PRELOAD="/usr/lib/libtcmalloc.so" ./your_benchmark_utility.

Similarly for jemalloc I could do:
LD_PRELOAD=`jemalloc-config --libdir`/libjemalloc.so.`jemalloc-config --revision` ./your_benchmark_utility Do you think it's possible to write a PTS xml that runs Redis tests with those 2 different LD_PRELOAD env vars? Thanks!

[Michael]

Ok now that I look more closely at the XML file of that test I realize that to obtain the results with different glibc version no gcc compile trick was used, but instead entirely different OS versions were used!

So yes: I can do the steps you propose to modify glibc settings and re-run the suite but I was looking for something totally automatic that comes up with the performance profiles with the 2 different glibc versions. I.e. something that downloads and compiles v2.25, and then links redis against that, runs the test and then starts from scratch again with version 2.26.

Perhaps this kind of test is too much... but maybe it's easier to run in a completely-automatic fashion a test jemalloc vs tcmalloc ?
To use tcmalloc you just need to do:

LD_PRELOAD="/usr/lib/libtcmalloc.so" ./your_benchmark_utility.

Similarly for jemalloc I could do:
LD_PRELOAD=`jemalloc-config --libdir`/libjemalloc.so.`jemalloc-config --revision` ./your_benchmark_utility Do you think it's possible to write a PTS xml that runs Redis tests with those 2 different LD_PRELOAD env vars? Thanks!

If you are on latest PTS release, there shouldn't be any need to modify the tests. There is a TEST_EXEC_PREPEND environment variable for setting anything prior to calling the test. So if you do something like TEST_EXEC_PREPEND="LD_PRELOAD=xxx" ./phoronix-test-suite <command> it should work.

[f18m]

If you are on latest PTS release, there shouldn't be any need to modify the tests.

I have downloadeed and installed version 7.6.. is that ok?

There is a TEST_EXEC_PREPEND environment variable for setting anything prior to calling the test. So if you do something like TEST_EXEC_PREPEND="LD_PRELOAD=xxx" ./phoronix-test-suite <command> it should work.

Yeah, ok thanks, good to know.
Anyway I would like to play with PTS XMLs myself (if I find the syntax is flexible enough I would like to reuse the framework for testing my own software -- I'm a c++ developer)... in my mind I can just change the install.sh file I can see at https://openbenchmarking.org/innhold...af87c6565b983c, right?

Also is there a quick guide to write PTS XMLs? I couldn't find any..

Thanks again!

[Michael]

I have downloadeed and installed version 7.6.. is that ok?


Yeah, ok thanks, good to know.
Anyway I would like to play with PTS XMLs myself (if I find the syntax is flexible enough I would like to reuse the framework for testing my own software -- I'm a c++ developer)... in my mind I can just change the install.sh file I can see at https://openbenchmarking.org/innhold...af87c6565b983c, right?

Also is there a quick guide to write PTS XMLs? I couldn't find any..

Thanks again!

7.6 should be fine.

You should find that the PTS test profile architecture is quite extensible and has been used for over hundreds of upstream tests/programs. If any extensions are needed, I am happy to add it.

Yes, it's easiest to just start by modifying existing tests. As they are installed onto your computer they are in ~/.phoronix-test-suite/test-profiles/ or /var/lib/phoronix-test-suite/ if root. Then just modify as you go is the easiest.

There unfortunately isn't any recent documentation on creating tests. I hope by the 8.0 release (June) to have some fresh documentation, but if you run into any questions when looking at other tests or modifying, feel free to post and happy to respond with guide/info on achieving what you want. Usually I'm quick at responding here.

[f18m]

Perfect thanks.
I will look into the XML and Bash script and see if I manage to do what I need.

Thanks