Benchmarking Radeon Open Compute ROCm 1.4 OpenCL

[boxie]

So... ROCm is definitely still in beta then - good to see progress though and I look forward to more performance and merges in the future

[orome]

hope this helps. I'm sure someone from AMD can correct/clarify stuff

Nice testing!

I really don't understand all the components though...
KFD

kernel fusion driver (amdkfd.ko). kernel driver handling the compute part. uses KGD (kernel graphics driver -- amdgpu.ko)

HSA
HSA+

Heterogeneous Software Architecture. the specs, and also the older name for the entire stack (now called ROCm)

Rock kernel driver,

ROCK - RadeonOpenCompute Kernel driver. Either patched KFD (amdkfd.ko) + KGD (amdgpu.ko). Or an entire linux kernel with AMD patches that haven't made it upstream (yet). last time I checked they did not really backport any fixes (security or otherwise), so I'd say it's not production ready.

the other KFD that "is NOT compatible with amdkfd that is distributed as part of the mainline Linux kernel from 3.19 and onward"

see above. ROCm stack includes patches that haven't made it upstream yet

kmt,

no idea what it stands for. it's a kernel interface library, that covers basic interaction with the KFD module (memory allocation and mapping, device queries, ...)

rocr runtime,
hsa runtime,

basically the same thing, renamed (old: hsaruntime, new rocr -radeon open compute runtime). includes finalizer and support routines to run hsa binaries

hcc,

Hetergeneous C Compiler. llvm/clang based compiler that implements C++AMP standard (previously called "clamp" and "kalmar"). original C++AMP can run on top of any OpenCL implementation. hcc includes CBackend to create CLC OpenCL code. hcc includes couple "heterogeneous extensions", although I'm not sure what exactly they include
it can either generate HSAIL/BRIG (intermediate language that needs to be finalized before running on a GPU), or AMDGCN binary code (differs between GCN generations)
hcc allows you to share (C++) source code that gets compiled for either cpu or gpu, or both. stuff that needs to be compiled for GPU still has some limitations (no recursion, ...)

thunk.

afaik, this refers to hsakmt/roct library

[mibo]

First of all thank you Michael for doing the testing.

I'm a bit disappointed I have to say.
I thought, after the AMD OpenCL is *converted* to open source, we immediately have the same performance and support of all the gpus.
After this benchmark it looks more like we have to wait again one more year to get the performance level that we could get with Catalyst two years ago. So, it seems there is a lot of work to be done.
I'm happy with the MESA+AMDGPU PRO OpenCL libs solution for now.

[pal666]

The Linux kernel should mainly contain abstractions for software components to efficiently communicate with each other

this linux kernel is called hurd

[pal666]

I thought, after the AMD OpenCL is *converted* to open source, we immediately have the same performance and support of all the gpus.

closed rocm and *converted* to open source rocm have the same performance and support of all the gpus. amdgpu-pro is different code so it has different performance and hardware support

[Niarbeht]

this linux kernel is called hurd

Does HURD actually... do anything yet?

[mibo]

closed rocm and *converted* to open source rocm have the same performance and support of all the gpus. amdgpu-pro is different code so it has different performance and hardware support

Unfortunately, *all the gpus* here means just Fiji and Polaris.
Let's see how long the GCN1.2, 1.1 and 1.0 owners have to wait to get support.

I was hoping that rocm and AMDGPU-PRO OpenCL are not so different.

[bridgman]

I really don't understand all the components though...

Mostly new names for the same components, associating them with ROC (Radeon Open Compute) branding:

ROCK = ROC Kernel = latest KFD (aka amdkfd) + associated amdgpu changes

ROCT = ROC Thunk = hsakmt=thunk (userspace wrapper for KFD, like libdrm-amdgpu is userspace wrapper for amdgpu)

ROCR = ROC Runtime = userspace driver that exposes ROC for use by various toolchains (eg HCC, OpenCL etc..)

We initially implemented the HSA stack for APUs, where the IOMMUv2 allows GPU access to unpinned memory, which was important for upstreaming since the alternative (allowing pinning from userspace) is generally not considered upstreamable. The initial dGPU support in ROC depended on pinning from userspace, which is common in proprietary drivers but verboten in upstream drivers. As a result, the upstream KFD includes support for Kaveri and Carrizo, but not for dGPUs.

We recently finished an initial implementation of "eviction" code, which provides the illusion of pinning to userspace drivers but actually allows buffers to be temporarily unpinned (suspending the compute process using them) when other processes (typically graphics) require memory.

This allows processes using ROC and user queues to behave similarly to processes using the graphics driver submit path, where userspace drivers are given the illusion of pinned buffers but the kernel drivers only actually guarantee pinning when commands from that process are running on the GPU. We needed a different implementation though, since processes running over amdgpu submit work via the kernel driver while processes running over ROC submit work directly to the HW via queues maintained in userspace.

Now that the eviction code is implemented and public (it's in 1.4) we can start working on upstreaming the latest KFD code, and are doing that now.

So... ROCm is definitely still in beta then - good to see progress though and I look forward to more performance and merges in the future

OpenCL for ROCM is definitely still in beta (we actually called it a developer preview, so more like an alpha), but not ROCM itself.

[boxie]

Mostly new names for the same components, associating them with ROC (Radeon Open Compute) branding:

<snip>

OpenCL for ROCM is definitely still in beta (we actually called it a developer preview, so more like an alpha), but not ROCM itself.

cool! shiny new things are fun and it seems like there is a whole bunch of developer effort involved in this. You are, as always, very informative and awesome bridgman - Thank you!

[orome]

Does HURD actually... do anything yet?

given the goals of the microkernel kernel, you can argue that it does everything it intends to ...
if the question is whether you can use it, debian/HURD runs ~80% of debian packages, including likes of Xorg and iceweasel.