Announcement

Collapse
No announcement yet.

Parsing Multi-result test without test arguments

Collapse
X
Collapse
 
  • Page of 1
  • Filter
  • Time
  • Show
Clear All
new posts
Previous template Next
  • #1

    Parsing Multi-result test without test arguments

    Hi, I'm trying to run another benchmarking software from PTS, parse its output and report it as a result in PTS.
    If my results were dependent on test arguments, I could use
    <ResultKey>PTS_USER_SET_ARGUMENTS</ResultKey> like in CompileBench or
    <MatchToTestArguments>#WRITE</MatchToTestArguments> like in IOZone.

    My current solution is a results parser section like this
    <ResultsParser>
    <AppendToArgumentsDescription>Integer Math (MOps./Sec.)</AppendToArgumentsDescription>
    <LineHint>This Computer</LineHint>
    <OutputTemplate>This Computer #_RESULT_# 7008.4 29.1 329.8 12359.7 1740.1 691.8 7416.2 1735.5 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 9377.4 N/A N/A N/A N/A 2789.1</OutputTemplate>
    </ResultsParser>
    <ResultsParser>
    <AppendToArgumentsDescription>Floating Point Math (MOps./Sec.)</AppendToArgumentsDescription>
    <LineHint>This Computer</LineHint>
    <OutputTemplate>This Computer 16312.9 #_RESULT_# 29.1 329.8 12359.7 1740.1 691.8 7416.2 1735.5 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 9377.4 N/A N/A N/A N/A 2789.1</OutputTemplate>
    </ResultsParser>
    and so on.

    This works fine for HTML test output, when I run
    phoronix-test-suite.bat result-file-to-csv result > myoutfile.csv , the output I get is

    "local/performancetest-9 - ",Score,,6279.20
    " - ",Score,,2613.90
    " - ",Score,,5
    " - ",Score,,3.30
    and so on.

    The csv formatter seems to discard the Arguments description, making my csv unusable.
    Without supplying my results as arguments - since they are not actually user selectable, I would like to my csv output to be

    "local/performancetest-9 - Integer Math ",MOps./Sec.,,6279.20
    " - Floating Point Math",MOps./Sec.,,2613.90
    etc.

    I did not find any documentation or openbenchmarking tests using this approach, I am thankful for any help on this
    Tags: None
  • #2
    Hi garandor - so you are saying the result parser works when viewing the results as HTML but it doesn't work as CSV? That doesn't make a lot of sense as at that point it has nothing to do with the result parser. During the benchmark execution is when results are parsed and then stored in the composite.xml, after that point the result parser is never used again. So if it works viewing as a graph, it should work when exporting it as CSV, otherwise it's some unrelated CSV output bug.

    What version of PTS are you using? Are you able to upload the test profile? Or look inside the composite.xml test-result file to verify if the parsed values are in there?
    Michael Larabel
    https://www.michaellarabel.com/

    Comment

    • #3
      It might be a bug in result-file-to-csv, I'm using the v7.2.1 tagged version on git.
      Uploading the test profile is not possible because it uses licensed software

      The index.html contains the values correctly (have no permission to create attachment), the composite.xml contains them too

      <Result>
      <Identifier>local/performancetest-9</Identifier>
      <Title>PerformanceTest</Title>
      <AppVersion>9</AppVersion>
      <Arguments></Arguments>
      <Description> - Integer Math (MOps./Sec.)</Description>
      <Scale>points</Scale>
      <Proportion>Score</Proportion>
      <DisplayFormat>BAR_GRAPH</DisplayFormat>
      <Data>
      <Entry>
      <Identifier>2017-08-04 22:45</Identifier>
      <Value>6279.20</Value>
      <RawString></RawString>
      </Entry>
      </Data>
      </Result>
      <Result>
      <Identifier></Identifier>
      <Title>PerformanceTest</Title>
      <AppVersion>9</AppVersion>
      <Arguments></Arguments>
      <Description> - Floating Point Math (MOps./Sec.)</Description>
      <Scale>points</Scale>
      <Proportion>Score</Proportion>
      <DisplayFormat>BAR_GRAPH</DisplayFormat>
      <Data>
      <Entry>
      <Identifier>2017-08-04 22:45</Identifier>
      <Value>2613.90</Value>
      <RawString></RawString>
      </Entry>
      </Data>
      </Result>

      The generated result.csv only contains the following output though

      "local/performancetest-9 - ",Score,,6279.20
      " - ",Score,,2613.90
      " - ",Score,,5
      " - ",Score,,3.30
      " - ",Score,,2893.50
      " - ",Score,,619.30
      " - ",Score,,157.50
      " - ",Score,,2330.80
      " - ",Score,,706.40
      " - ",Score,,7.60
      " - ",Score,,45.10
      " - ",Score,,135
      " - ",Score,,44.70
      " - ",Score,,271.60
      " - ",Score,,254.10
      " - ",Score,,12.50
      " - ",Score,,15.30
      " - ",Score,,2.70
      " - ",Score,,8.20
      " - ",Score,,2.20
      " - ",Score,,324.80
      " - ",Score,,32.10
      " - ",Score,,5992.70
      " - ",Score,,4530.20
      " - ",Score,,3688.80
      " - ",Score,,2209
      " - ",Score,,37.30
      " - ",Score,,10838
      " - ",Score,,74.10
      " - ",Score,,48.40
      " - ",Score,,33.90
      " - ",Score,,1951.60
      " - ",Score,,274
      " - ",Score,,861.60
      " - ",Score,,565.70
      " - ",Score,,426.90
      " - ",Score,,982.70

      Comment

      • #4
        Originally posted by garandor View Post
        It might be a bug in result-file-to-csv, I'm using the v7.2.1 tagged version on git.
        Uploading the test profile is not possible because it uses licensed software

        The index.html contains the values correctly (have no permission to create attachment), the composite.xml contains them too

        <Result>
        <Identifier>local/performancetest-9</Identifier>
        <Title>PerformanceTest</Title>
        <AppVersion>9</AppVersion>
        <Arguments></Arguments>
        <Description> - Integer Math (MOps./Sec.)</Description>
        <Scale>points</Scale>
        <Proportion>Score</Proportion>
        <DisplayFormat>BAR_GRAPH</DisplayFormat>
        <Data>
        <Entry>
        <Identifier>2017-08-04 22:45</Identifier>
        <Value>6279.20</Value>
        <RawString></RawString>
        </Entry>
        </Data>
        </Result>
        <Result>
        <Identifier></Identifier>
        <Title>PerformanceTest</Title>
        <AppVersion>9</AppVersion>
        <Arguments></Arguments>
        <Description> - Floating Point Math (MOps./Sec.)</Description>
        <Scale>points</Scale>
        <Proportion>Score</Proportion>
        <DisplayFormat>BAR_GRAPH</DisplayFormat>
        <Data>
        <Entry>
        <Identifier>2017-08-04 22:45</Identifier>
        <Value>2613.90</Value>
        <RawString></RawString>
        </Entry>
        </Data>
        </Result>

        The generated result.csv only contains the following output though

        "local/performancetest-9 - ",Score,,6279.20
        " - ",Score,,2613.90
        " - ",Score,,5
        " - ",Score,,3.30
        " - ",Score,,2893.50
        " - ",Score,,619.30
        " - ",Score,,157.50
        " - ",Score,,2330.80
        " - ",Score,,706.40
        " - ",Score,,7.60
        " - ",Score,,45.10
        " - ",Score,,135
        " - ",Score,,44.70
        " - ",Score,,271.60
        " - ",Score,,254.10
        " - ",Score,,12.50
        " - ",Score,,15.30
        " - ",Score,,2.70
        " - ",Score,,8.20
        " - ",Score,,2.20
        " - ",Score,,324.80
        " - ",Score,,32.10
        " - ",Score,,5992.70
        " - ",Score,,4530.20
        " - ",Score,,3688.80
        " - ",Score,,2209
        " - ",Score,,37.30
        " - ",Score,,10838
        " - ",Score,,74.10
        " - ",Score,,48.40
        " - ",Score,,33.90
        " - ",Score,,1951.60
        " - ",Score,,274
        " - ",Score,,861.60
        " - ",Score,,565.70
        " - ",Score,,426.90
        " - ",Score,,982.70
        Are you able to send me the composite.xml file results then? Michael at phoronix.com if so. Though it looks like like the 'Proportion' tag for your test profile is invalud. It says 'Score' but it should either say 'HIB' (if higher results are better) or 'LIB' (lower score is better) where as the "Scale" is for specifying points/score. That might be why CVS output is giving problems.
        Michael Larabel
        https://www.michaellarabel.com/

        Comment

        • #5
          Thanks to Michael helping, I found the syntax my results-definition needs is

          <ResultsParser>
          <ArgumentsDescription>Integer Math</ArgumentsDescription>
          <ResultScale>MOps./Sec.</ResultScale>
          <ResultProportion>HIB</ResultProportion>
          <LineHint>This Computer</LineHint>
          <OutputTemplate>This Computer #_RESULT_# 7008.4 29.1 329.8 12359.7 1740.1 691.8 7416.2 1735.5 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 9377.4 N/A N/A N/A N/A 2789.1</OutputTemplate>
          </ResultsParser>
          <ResultsParser>
          <ArgumentsDescription>Floating Point Math</ArgumentsDescription>
          <ResultScale>MOps./Sec.</ResultScale>
          <ResultProportion>HIB</ResultProportion>
          <LineHint>This Computer</LineHint>
          <OutputTemplate>This Computer 16312.9 #_RESULT_# 29.1 329.8 12359.7 1740.1 691.8 7416.2 1735.5 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 9377.4 N/A N/A N/A N/A 2789.1</OutputTemplate>
          </ResultsParser>
          etc.

          with this result-definition.xml and a test-definition that doesnt set a proportion or scale, my result gets written to pts result correctly.
          Strangely CSV output still is broken though (bug?). Text output works. - i am running v7.2.1 on windows, the attachment I sent to Michael directly has outputs generated by
          phoronix-test-suite.bat result-file-to-text result > result.txt and
          phoronix-test-suite.bat result-file-to-csv result > result.csv
          as well as the results folder.

          pts-version:
          pts_git ((v7.2.1))
          $ git status
          HEAD detached at v7.2.1
          nothing to commit, working tree clean

          Comment

          Previous template Next
          Working...
          X