Announcement

Collapse
No announcement yet.

How to write a testsuite to compare different builds of same software?

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • How to write a testsuite to compare different builds of same software?

    Hi,
    I discovered about PTS by reading the following article:
    https://www.phoronix.com/scan.php?pa....26-Redis-Test

    In that article Redis is compared against 2 different glibc versions and with jemalloc enabled.
    I would like to reproduce that test locally (actually my target would be to add a test of Redis using tcmalloc!)... I managed to install the testsuite and run
    phoronix-test-suite benchmark 1708052-TY-REDISMALL91

    But I don't get a nice comparison of "Redis with glibc 2.26" vs "Redis with glibc 2.25"... how can I achieve that?

    Thanks!
    Francesco

  • #2
    phoronix-test-suite benchmark 1708052-TY-REDISMALL91

    <do whatever changes you want to glibc>

    phoronix-test-suite force-install 1708052-TY-REDISMALL91 (if wanting to trigger recompile)
    phoronix-test-suite benchmark 1708052-TY-REDISMALL91


    Or I'm not sure I understand your question?
    Michael Larabel
    http://www.michaellarabel.com/

    Comment


    • #3
      Ok now that I look more closely at the XML file of that test I realize that to obtain the results with different glibc version no gcc compile trick was used, but instead entirely different OS versions were used!

      So yes: I can do the steps you propose to modify glibc settings and re-run the suite but I was looking for something totally automatic that comes up with the performance profiles with the 2 different glibc versions. I.e. something that downloads and compiles v2.25, and then links redis against that, runs the test and then starts from scratch again with version 2.26.

      Perhaps this kind of test is too much... but maybe it's easier to run in a completely-automatic fashion a test jemalloc vs tcmalloc ?
      To use tcmalloc you just need to do:

      LD_PRELOAD="/usr/lib/libtcmalloc.so" ./your_benchmark_utility.

      Similarly for jemalloc I could do:
      LD_PRELOAD=`jemalloc-config --libdir`/libjemalloc.so.`jemalloc-config --revision` ./your_benchmark_utility Do you think it's possible to write a PTS xml that runs Redis tests with those 2 different LD_PRELOAD env vars? Thanks!

      Comment


      • #4
        Originally posted by f18m View Post
        Ok now that I look more closely at the XML file of that test I realize that to obtain the results with different glibc version no gcc compile trick was used, but instead entirely different OS versions were used!

        So yes: I can do the steps you propose to modify glibc settings and re-run the suite but I was looking for something totally automatic that comes up with the performance profiles with the 2 different glibc versions. I.e. something that downloads and compiles v2.25, and then links redis against that, runs the test and then starts from scratch again with version 2.26.

        Perhaps this kind of test is too much... but maybe it's easier to run in a completely-automatic fashion a test jemalloc vs tcmalloc ?
        To use tcmalloc you just need to do:

        LD_PRELOAD="/usr/lib/libtcmalloc.so" ./your_benchmark_utility.

        Similarly for jemalloc I could do:
        LD_PRELOAD=`jemalloc-config --libdir`/libjemalloc.so.`jemalloc-config --revision` ./your_benchmark_utility Do you think it's possible to write a PTS xml that runs Redis tests with those 2 different LD_PRELOAD env vars? Thanks!
        If you are on latest PTS release, there shouldn't be any need to modify the tests. There is a TEST_EXEC_PREPEND environment variable for setting anything prior to calling the test. So if you do something like TEST_EXEC_PREPEND="LD_PRELOAD=xxx" ./phoronix-test-suite <command> it should work.
        Michael Larabel
        http://www.michaellarabel.com/

        Comment


        • #5
          If you are on latest PTS release, there shouldn't be any need to modify the tests.
          I have downloadeed and installed version 7.6.. is that ok?

          There is a TEST_EXEC_PREPEND environment variable for setting anything prior to calling the test. So if you do something like TEST_EXEC_PREPEND="LD_PRELOAD=xxx" ./phoronix-test-suite <command> it should work.
          Yeah, ok thanks, good to know.
          Anyway I would like to play with PTS XMLs myself (if I find the syntax is flexible enough I would like to reuse the framework for testing my own software -- I'm a c++ developer)... in my mind I can just change the install.sh file I can see at https://openbenchmarking.org/innhold...af87c6565b983c, right?

          Also is there a quick guide to write PTS XMLs? I couldn't find any..

          Thanks again!

          Comment


          • #6
            Originally posted by f18m View Post
            I have downloadeed and installed version 7.6.. is that ok?


            Yeah, ok thanks, good to know.
            Anyway I would like to play with PTS XMLs myself (if I find the syntax is flexible enough I would like to reuse the framework for testing my own software -- I'm a c++ developer)... in my mind I can just change the install.sh file I can see at https://openbenchmarking.org/innhold...af87c6565b983c, right?

            Also is there a quick guide to write PTS XMLs? I couldn't find any..

            Thanks again!
            7.6 should be fine.

            You should find that the PTS test profile architecture is quite extensible and has been used for over hundreds of upstream tests/programs. If any extensions are needed, I am happy to add it.

            Yes, it's easiest to just start by modifying existing tests. As they are installed onto your computer they are in ~/.phoronix-test-suite/test-profiles/ or /var/lib/phoronix-test-suite/ if root. Then just modify as you go is the easiest.

            There unfortunately isn't any recent documentation on creating tests. I hope by the 8.0 release (June) to have some fresh documentation, but if you run into any questions when looking at other tests or modifying, feel free to post and happy to respond with guide/info on achieving what you want. Usually I'm quick at responding here.
            Michael Larabel
            http://www.michaellarabel.com/

            Comment


            • #7
              Perfect thanks.
              I will look into the XML and Bash script and see if I manage to do what I need.

              Thanks

              Comment

              Working...
              X