where do I find git sources for a test?

[Michael]

Michael how do you deal with a test that crashes or hardlocks the system?

I restart the system.

What about tests that take too long (because of a bug or performance issue) and won't complete in a practical amount of time - if ever at all?

I don't include tests that are known to be in bad shape. If it takes too long for user, they are welcome to kill it but I make no global policy for it as most tests don't take too long.

[nevion]

I restart the system.

I meant with an individual run option (Option-Menu-Entrys) - one run option that triggers a fault shouldn't sink the entire test. In the case of CUDA but especially OpenCL it is very easy to hardlock based on implementation bugs of the respective runtimes. For now I'm guessing you comment out offending tests on vulnerable configurations and then rerun tests...

I don't include tests that are known to be in bad shape. If it takes too long for user, they are welcome to kill it but I make no global policy for it as most tests don't take too long.

The tests I'm running - they're not in bad shape - it's almost certainly bugs in AMD's current OpenCL implementations - ROCm and AMDGPU-Pro (I'll report them upstream soon). But these tests (Cholesky factorization, sorts) are not going to go away simply because AMD is buggy right now. NVidia's runtime passes them just fine and I'd hazard a guess that fglrx would also run them fine. It's also only 6 tests (Option-Menu-Entrys) out of like 30 or so that fail. I'm experimenting with the timeout command in the launch script that will kill the program after 30 seconds if it hasn't completed, what do you think about this?

 timeout -s SIGKILL --preserve-status 30 ./\$@ -d \${COMPUTE_DEVICE} > \$LOG_FILE 2>&1
 echo \$? > ~/test-exit-status" > arrayfire

It seems you coded up things to simply die after the first failure... I don't think that makes a whole lot of sense (you make more work for yourself?). You should have a keep going flag/mode to test everything that can run. Make for instance has -k/keep-going.

[nevion]

Michael I've updated my test here:

https://github.com/nevion/arrayfire-pts.git

First my test runs (there's no problems base program the pts test is running).
But I am having some difficulty figuring out how to run this. I've used 2 option-menu-entry's and I"m not really sure how to specify them. This program can be run in opencl or cuda mode and then it's got a myriad of benchmarks it can run past that. I'd like to set the backend option (opencl) and run through all the tests automatically, or piecewise (due to the few crashing I mentioned previously).

I've launched like so, but I haven't been able to set the benchmark menu option, nor could I find how to cycle through all options this test presents.

LD_LIBRARY_PATH=/opt/rocm/opencl/lib/x86_64 PRESET_OPTIONS="local/arrayfire.Benchmark=Accumulate_2D_f64" phoronix-test-suite benchmark arrayfire
Phoronix Test Suite v6.8.0

    Installed: local/arrayfire


ArrayFire 1.0:
    local/arrayfire
    Processor Test Configuration


System Information

Hardware:
Processor: Intel Xeon E5-1650 v3 @ 3.80GHz (12 Cores), Motherboard: Supermicro X10SRA v1.01, Chipset: Intel Xeon E7 v3/Xeon, Memory: 129024MB, Disk: 480GB Micron_M500DC_MT, Graphics: AMD Radeon R9 FURY / NANO 4096MB, Audio: Intel C610/X99, Monitor: DELL U2417H, Network: Intel I210 Gigabit Connection

Software:
OS: Ubuntu 16.04, Kernel: 4.6.0-kfd-compute-rocm-rel-1.4-16 (x86_64), Display Driver: amdgpu 1.1.0, OpenCL: OpenCL 2.0 AMD-APP (2300.5), Compiler: GCC 5.4.0 20160609, File-System: ext4, Screen Resolution: 1920x1080

    Would you like to save these test results (Y/n): n


ArrayFire 1.0:
    local/arrayfire [Platform: OpenCL - Benchmark: Accumulate_1D_f32]
    Test 1 of 1
    Estimated Trial Run Count:    3
    Estimated Time To Completion: 1 Minute (18:24 EST)
        Started Run 1 @ 18:24:04
        Started Run 2 @ 18:24:07
        Started Run 3 @ 18:24:09  [Std. Dev: 0.76%]

    Test Results:
        1751.4
        1777.5
        1759.5

    Average: 1762.80 I¼s

Can you duplicate or figure what's going wrong in the pts invocation? FYI this also works on Nvidia too, by setting environment variable OpenCL_INCLUDE_DIR=/usr/local/cuda/include when installing the test.

update: nevermind, this was due to wrapping every entry in it's own Menu. I pushed the fix for this and now see many warnings as the tests are parsed but it seems to be executing correctly. It'd be great to be able to fix the parameter (say platform) to opencl or cuda while going through the combinatorics of batch runs...

[nevion]

Hey Michael do you mind giving it a try and commenting on results?

[Michael]

Hey Michael do you mind giving it a try and commenting on results?

Hi Nevion, giving what a try? And in regards to running through all options for a test, the easiest way is by selecting all options when building a suite via the build-suite option.

[nevion]

Michael A batch-run against the test suite I linked to/developed.

And for build-suite - you're saying that's a way to fix 1 argument variable (menu option) while performing the cross product against the other arguments - in a host specific friendly way?

[Michael]

Michael A batch-run against the test suite I linked to/developed.

Will try to find time in next day or two, ping me in forums later this week if I don't get back to you before then.

And for build-suite - you're saying that's a way to fix 1 argument variable (menu option) while performing the cross product against the other arguments - in a host specific friendly way?

FYI with the build-suite text prompts or whenever seeing any test run option prompts when being prompted for making a selection, you can always select multiples by comma, e.g. options 1,2,3 or by specifying like 1-3,5 etc.

[nevion]

Michael ping

[nevion]

Michael another bump - as you requested previously.

[Michael]

Michael another bump - as you requested previously.

Yep, been busy but hope to find the time in next day or so.