How to time limit / strict loop limit runs?

[Yves]

Hi there,

I am trying to benchmark a new software definied storage solution. So I have setup multiple vms on the compute cluster and I am able to run the same benchmark command to all the systems at the same time. I also set the environment variables for the TEST_RESULTS_IDENTIFIER, FORCE_TIMES_TO_RUN=12 but if I run for example phoronix-test-suite benchmark pts/aio-stress on all the vms simultaniously some still to 15 loops some even 40. which of course shifts the results since the load is less if one vm is already finished. Is there a workaround? For fio its even more important that all run for example 5mins rand read 4k then 5min rand write 4k etc. etc.

Thanks for your AMAZING benchmarking tool!

Regards,
Yves

[Michael]

FORCE_TIMES_TO_RUN should run all that X amount of times with no less, no more. Are you saying you are seeing different behavior? If so, what PTS version are you on?

[Yves]

Unfortunatly I am... I have set it to 10 which I can also see in the run it is writing

estimated trial run count: 10

but if I do a benchmark the sample are very variable sometimes 10 sometimes 12 or even a lot more. But the runs are always MORE and the additional runs are always with a * in the name for example

Running Interim Test Script @ 11:47:20 Started Run 4 @ 11:47:22[B] *[/B]

Version is Phoronix Test Suite v9.0.1

[Yves]

This is from right now:

Flexible IO Tester 3.16:
    pts/fio-1.12.0 [Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory]
    Test 1 of 1
    Estimated Trial Run Count:    10           
    Estimated Time To Completion: 6 Minutes [11:50 CEST]
        Running Pre-Test Script @ 11:45:31
        Started Run 1 @ 11:45:31
        Running Interim Test Script @ 11:46:05
        Started Run 2 @ 11:46:07
        Running Interim Test Script @ 11:46:40
        Started Run 3 @ 11:46:42
        Running Interim Test Script @ 11:47:15
        Started Run 4 @ 11:47:17
        Running Interim Test Script @ 11:47:50
        Started Run 5 @ 11:47:52
        Running Interim Test Script @ 11:48:25
        Started Run 6 @ 11:48:27
        Running Interim Test Script @ 11:49:00
        Started Run 7 @ 11:49:02
        Running Interim Test Script @ 11:49:35
        Started Run 8 @ 11:49:37
        Running Interim Test Script @ 11:50:10
        Started Run 9 @ 11:50:12
        Running Interim Test Script @ 11:50:45
        Started Run 10 @ 11:50:47
        Running Interim Test Script @ 11:51:19
        Started Run 11 @ 11:51:21 *
        Running Interim Test Script @ 11:51:54
        Started Run 12 @ 11:51:56 *
        Running Interim Test Script @ 11:52:29
        Started Run 13 @ 11:52:31 *

[Michael]

Hmmm, can you post the full command you are running? FORCE_TIMES_TO_RUN should never change, but guessing then that some other variable you might be setting could be overriding it.

[Yves]

I don't think I have more variable set than FORCE_TIMES_TO_RUN and TEST_RESULTS_IDENTIFIER. I just run phoronix-test-suite benchmark pts/fio (rand read -> libaio -> no -> yes -> 4k -> default test path -> yes)

[Michael]

Did you make any changes to your ~/.phoronix-test-suite/user-config.xml / (or if root, /etc/phoronix-test-suite.xml) files?

[Yves]

we are now on 23 runs... ;-)

phoronix-test-suite benchmark pts/fio



Phoronix Test Suite v9.0.1

    Installed:     pts/fio-1.12.0


Flexible IO Tester 3.16:
    pts/fio-1.12.0
    Disk Test Configuration
        1: Random Read
        2: Random Write
        3: Sequential Read
        4: Sequential Write
        5: Test All Options
        ** Multiple items can be selected, delimit by a comma. **
        Type: 1


        1: POSIX AIO
        2: Sync
        3: Linux AIO
        4: Windows AIO
        5: Test All Options
        ** Multiple items can be selected, delimit by a comma. **
        IO Engine: 3


        1: Yes
        2: No
        3: Test All Options
        ** Multiple items can be selected, delimit by a comma. **
        Buffered: 2


        1: No
        2: Yes
        3: Test All Options
        ** Multiple items can be selected, delimit by a comma. **
        Direct: 2


        1:  4KB
        2:  8KB
        3:  16KB
        4:  32KB
        5:  64KB
        6:  128KB
        7:  256KB
        8:  512KB
        9:  1MB
        10: 2MB
        11: 4MB
        12: 8MB
        13: Test All Options
        ** Multiple items can be selected, delimit by a comma. **
        Block Size: 1



System Information


  PROCESSOR:          4 x Intel Xeon E312xx
    Core Count:       4
    Extensions:       SSE 4.2 + AVX
    Cache Size:       4096 KB
    Microcode:        0x1

  GRAPHICS:           Red Hat QXL paravirtual graphic card
    Screen:           1024x768

  MOTHERBOARD:        oVirt Node
    BIOS Version:     1.11.0-2.el7
    Chipset:          Intel 440FX 82441FX PMC
    Network:          Red Hat Virtio device

  MEMORY:             4096MB

  DISK:               107GB QEMU HDD + 11GB QEMU HDD
    File-System:      ext4
    Mount Options:    relatime rw seclabel
    Disk Scheduler:   MQ-DEADLINE

  OPERATING SYSTEM:   Fedora 30
    Kernel:           5.0.9-301.fc30.x86_64 (x86_64)                                          
    Desktop:          GNOME Shell 3.32.1                                                      
    Display Server:   X Server                                                                
    Compiler:         GCC 9.0.1 20190312                                                      
    System Layer:     KVM                                                                     
    Security:         SELinux                                                                 
                      + l1tf: Mitigation of PTE Inversion                                     
                      + meltdown: Mitigation of PTI                                           
                      + spec_store_bypass: Mitigation of SSB disabled via prctl and seccomp   
                      + spectre_v1: Mitigation of __user pointer sanitization                 
                      + spectre_v2: Mitigation of Full generic retpoline IBPB: conditional IBRS_FW STIBP: disabled RSB filling

    Would you like to save these test results (Y/n): rand_read_4k_test
    Would you like to save these test results (Y/n): y

    Recently Saved Test Results:
       y                    [Today]
       aio-simultan         [Today]
       simultan-full-test   [Today]
       simulatv4            [Today]
       simultav3            [Today]

    Enter a name for the result file: rand_read_4k_test

Test Identifier: fedora1

If desired, enter a new description below to better describe this result set / system configuration under test.
Press ENTER to proceed without changes.

Current Description: KVM testing on Fedora 30 via the Phoronix Test Suite.

New Description: rand_read_4k_test

Flexible IO Tester 3.16:
    pts/fio-1.12.0 [Type: Random Read - IO Engine: Linux AIO - Buffered: No - Direct: Yes - Block Size: 4KB - Disk Target: Default Test Directory]
    Test 1 of 1
    Estimated Trial Run Count:    10           
    Estimated Time To Completion: 5 Minutes [11:50 CEST]
        Running Pre-Test Script @ 11:45:31
        Started Run 1 @ 11:45:31
        Running Interim Test Script @ 11:46:05
        Started Run 2 @ 11:46:07
        Running Interim Test Script @ 11:46:40
        Started Run 3 @ 11:46:42
        Running Interim Test Script @ 11:47:15
        Started Run 4 @ 11:47:17
        Running Interim Test Script @ 11:47:50
        Started Run 5 @ 11:47:52
        Running Interim Test Script @ 11:48:25
        Started Run 6 @ 11:48:27
        Running Interim Test Script @ 11:49:00
        Started Run 7 @ 11:49:02
        Running Interim Test Script @ 11:49:35
        Started Run 8 @ 11:49:37
        Running Interim Test Script @ 11:50:10
        Started Run 9 @ 11:50:12
        Running Interim Test Script @ 11:50:45
        Started Run 10 @ 11:50:47
        Running Interim Test Script @ 11:51:20
        Started Run 11 @ 11:51:22 *
        Running Interim Test Script @ 11:51:55
        Started Run 12 @ 11:51:57 *
        Running Interim Test Script @ 11:52:30
        Started Run 13 @ 11:52:32 *
        Running Interim Test Script @ 11:53:05
        Started Run 14 @ 11:53:07 *
        Running Interim Test Script @ 11:53:40
        Started Run 15 @ 11:53:42 *
        Running Interim Test Script @ 11:54:15
        Started Run 16 @ 11:54:17 *
        Running Interim Test Script @ 11:54:50
        Started Run 17 @ 11:54:52 *
        Running Interim Test Script @ 11:55:25
        Started Run 18 @ 11:55:27 *
        Running Interim Test Script @ 11:56:00
        Started Run 19 @ 11:56:02 *
        Running Interim Test Script @ 11:56:35
        Started Run 20 @ 11:56:37 *
        Running Interim Test Script @ 11:57:11
        Started Run 21 @ 11:57:13 *
        Running Interim Test Script @ 11:57:46
        Started Run 22 @ 11:57:48 *
        Running Interim Test Script @ 11:58:21
        Started Run 23 @ 11:58:23 *

[Yves]

Did you make any changes to your ~/.phoronix-test-suite/user-config.xml / (or if root, /etc/phoronix-test-suite.xml) files?

nope. nothing changed only ran these two commands

export TEST_RESULTS_IDENTIFIER=centos_vmware
export FORCE_TIMES_TO_RUN=10

[Michael]

Could you try a run without using TEST_RESULTS_IDENTIFIER to see if it behaves fine then? Otherwise will dive into the code this afternoon, but I use FORCE_TIMES_TO_RUN and have never hit that behavior, though I don't normally pair it with TEST_RESULTS_IDENTIFIER so guessing if that is causing some interference.