AMD ROCm 1.9 Available With Vega 20 Support Plus Upstream Kernel Compatibility

[coder]

IIRC Tonga and Iceland were both designed and locked down before we wrote the first HSA stack to exercise the HW with. I believe there were some HW issues with both of them specific to the HSA/ROC stack, although I don't remember the details off the top of my head. Something related to AQL support is most likely.

Any chance of Polaris 12 getting supported? Not a big deal, but I have one and just curious if I'll ever be able to use it with the ROC stack.

I mostly got my RX 550 to replace an old HD 5450 as my "backup" card, but I had expected it to work with all the latest APIs. That said, the main thing I needed was OpenGL.

[coder]

Release the thing is a usable manner. As plugin, as lib, as whatever you want but make it work. There is no problem if people have to compile it, but give the package out with a documentation on how to inject ROCm between the CUDA part and the AMD GPU. Thats all.

There are people out there who want to try this stuff out. Don't let it die because of poor support.

It's one of those "look before you leap" situations. You need to rebuild whatever deep learning framework you're using, for one thing, assuming it even has a ROCm/OpenMI fork available. If you're not even prepared to do that much, then forget it.

If your friend expected it to just run CUDA code, then he clearly didn't RTFM.

BTW, I can save you a bit of trouble: a RX 580 will perform nothing like a recent Titan. So, if that was the goal, then you guys were being seriously naive. Try looking up the specs, for starters.

[TemplarGR]

IIRC Tonga and Iceland were both designed and locked down before we wrote the first HSA stack to exercise the HW with. I believe there were some HW issues with both of them specific to the HSA/ROC stack, although I don't remember the details off the top of my head. Something related to AQL support is most likely.

So that means that Tonga/Iceland ROCm will never happen?Or perhaps it could be possible in a future version? I am asking because experimental support for earlier architectures exists. Could we Tonga users attempt to run ROCm "unsupported", or it just does not run at all? I am not in a rush about using it, and by the time any proper support arrives, if it ever arrives, i will most likely have upgraded to Navi, but i am just curious.

[Shevchen]

And boom goes the dynamite.

Lets do it in order:

I never say should. But is reading documentation the only way to live in open source world? I bet he read plenty of them since he uses Arch.

He probably did, but as time is a constraint he can't fiddle around several days. And again: He already has a working system. I send him my RX580 so that he could try out an alternative, because he likes open source (who would have guessed). AMD is trying to break up a monopoly on GPU computing - CUDA, which means they need a working "plugin" that works on an already established infrastructure. At this point its not the (entire) job of the user to get it up and running, because there already *is* a working system.

Are you just talking about the fact that the source code has not been built and packaged for Arch yet (a day after release) or a larger issue ?

This is the first ROCm release that is a candidate for widespread packaging, since the required kernel code is now upstream and showing up in distro kernels, so it makes sense that your friend would not have found anything pre-packaged prior to the 1.9 release.

Okay, that makes sense. I guess we were put on the wrong track with the previous press releases then, where we had the impression that ROCm was working "somehow, somewhere". But in general, yes - that is/was the problem.

he selected arch, so he or some other arch user should

If I read the feedback correctly, there should be a working package for arch sometime in the future - lets see how it goes.

It's one of those "look before you leap" situations. You need to rebuild whatever deep learning framework you're using, for one thing, assuming it even has a ROCm/OpenMI fork available. If you're not even prepared to do that much, then forget it.

If your friend expected it to just run CUDA code, then he clearly didn't RTFM.

BTW, I can save you a bit of trouble: a RX 580 will perform nothing like a recent Titan. So, if that was the goal, then you guys were being seriously naive. Try looking up the specs, for starters.

From what I heard the framework is tensorflow (which we saw on the marketing slides) and aside from that it was only a test if it even works. If it works, the next step would probably be getting a Vega 20 in the near future to try out if it can push a Titan.

Also, point of RTFM is kinda unfair - if it would be his sole job to integrate and write frameworks, yeah - I can see your point. But he isn't and *nobody* beside him on the whole workgroup is even thinking of AMD (at least as far I can tell from the general feedback). Maybe it was naive, but getting mad over trying it *while* the pretty marketing charts say "We can do it" is also not the best way to handle it.

Anyway - I want to see ROCm succeed and the only thing I can contribute was my RX580 - cause I'm not doing any AI computing.

[Marc Driftmeyer]

Hooray! This is great news. Confirming the same on Ubuntu 18.10 cosmic and kernel 4.18.7

Here are some of my notes in case anyone needs help getting going. For those on a similar setup, I previously had AMD's 18.20-606296 installed for OpenCL support on my RX 480, so I did the following first to uninstall the AMD drivers that were installed alongside Mesa:

$ amdgpu-uninstall

The uninstall should remove the necessary packaegs but you can run the following to see if they got removed: $ sudo apt list --installed|grep 18.20-606296 (or whatever your version number is) to double check. On my system it was the following packages (might be slightly different if you have Vega): amdgpu-core amdgpu-pro-core clinfo-amdgpu-pro libopencl1-amdgpu-pro opencl-orca-amdgpu-pro-icd

Next, follow the instructions from ROCm's GitHub but don't install rocm-dkms or rock-dkms, as it doesn't build for a custom kernel (didn't for me at least on 4.18.7), you might have no problem if you use the distro kernel.

Note: If you already have the dkms packages installed and try to purge them, it'll likely try to remove all ROCm-related packages, which isn't what we what.

Instead, make sure to install ROCm without the two dkms packages. (so purge all if you need to, and reinstall with below command):

$ sudo apt install rocm-opencl rocm-clang-ocl rocm-dev rocminfo rocm-smi rocm-utils

The entire ROCm packages that I have installed on my system are:

hsa-ext-rocr-dev/Ubuntu 16.04,now 1.1.9-8-g51c00c2 amd64 [installed,automatic]
hsa-rocr-dev/Ubuntu 16.04,now 1.1.9-8-g51c00c2 amd64 [installed,automatic]
hsakmt-roct/Ubuntu 16.04,now 1.0.9-8-g238782c amd64 [installed,automatic]
hsakmt-roct-dev/Ubuntu 16.04,now 1.0.9-8-g238782c amd64 [installed,automatic]
rocm-clang-ocl/Ubuntu 16.04,now 0.3.0-7997136 amd64 [installed]
rocm-dev/Ubuntu 16.04,now 1.9.211 amd64 [installed]
rocm-device-libs/Ubuntu 16.04,now 0.0.1 amd64 [installed]
rocm-opencl/Ubuntu 16.04,now 1.2.0-2018090737 amd64 [installed]
rocm-opencl-dev/Ubuntu 16.04,now 1.2.0-2018090737 amd64 [installed]
rocm-smi/Ubuntu 16.04,now 1.0.0-72-gec1da05 amd64 [installed]
rocm-utils/Ubuntu 16.04,now 1.9.211 amd64 [installed]
rocminfo/Ubuntu 16.04,now 1.0.0 amd64 [installed]
rocr_debug_agent/Ubuntu 16.04,now 1.0.0 amd64 [installed]

Checking dkms, we should have nothing related to amd/rocm:

$ sudo dkms status
virtualbox, 5.2.18, 4.18.0-7-generic, x86_64: installed
virtualbox, 5.2.18, 4.18.7-041807+custom-generic, x86_64: installed
virtualbox-guest, 5.2.18, 4.18.0-7-generic, x86_64: installed
virtualbox-guest, 5.2.18, 4.18.7-041807+custom-generic, x86_64: installed

To see OpenCL info, use clinfo. Follow the instructions on their GitHub to add ROCm /bin to PATH variable if you get clinfo not found)

$ clinfo | grep 'Platform Version\|Device Version\|Device Board Name\|Max compute units\|Max clock frequency\|Global memory size'
Platform Version: OpenCL 2.1 AMD-APP (2679.0)
Max compute units: 36
Max clock frequency: 1303Mhz
Global memory size: 8589934592

rocm-smi goodness: $ watch -d -n 1 rocm-smi

==================== ROCm System Management Interface ====================
================================================================================
GPU Temp AvgPwr SCLK MCLK Fan Perf SCLK OD MCLK OD
0 61c 99.182W 1077Mhz 2000Mhz 40.0% manual 0% 0%
================================================================================
==================== End of ROCm SMI Log ====================

Thanks again to all that made this happen!

Does Blender recognize the OpenCL stack from ROCm?

FWIW: I bailed on installing rocm-dev which pulls in all sorts of junk I do not need and trashes Debian LLVM-5 and LLVM-6 installs.

Not sure how you're getting dkms to install as it bails on me with the Debian Sid 4.18 kernel.

[perpetually high]

Does Blender recognize the OpenCL stack from ROCm?

Yeah, but not when I had it installed through flatpak. Only worked when I downloaded the tar.bz2 file from Blender's site. Both versions were 2.79b.

The flatpak version:

OpenCL device capabilities:
No OpenCL platforms found

Running the binary from Blender's site:

OpenCL device capabilities:
Number of platforms: 1
Platform #0
    Platform Name: AMD Accelerated Parallel Processing
    Platform Vendor: Advanced Micro Devices, Inc.
    Platform Version: OpenCL 2.1 AMD-APP (2679.0)
    Platform Profile: FULL_PROFILE
    Platform Extensions: cl_khr_icd cl_amd_event_callback
    Number of devices: 1
        Device: #0
            Device Name: gfx803
            Device Board Name: Ellesmere [Radeon RX 470/480/570/570X/580/580X]
            Device Vendor: Advanced Micro Devices, Inc.
            Device OpenCL C Version: OpenCL C 2.0
            Device Profile: FULL_PROFILE
            Device Version: OpenCL 1.2
            Device Extensions: cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program

[perpetually high]

FWIW: I bailed on installing rocm-dev which pulls in all sorts of junk I do not need and trashes Debian LLVM-5 and LLVM-6 installs.

Not sure how you're getting dkms to install as it bails on me with the Debian Sid 4.18 kernel.

I believe the rocm package is optional, I'll update my post to exclude it. Thanks.

Also, I'm not installing rocm-dkms or rock-dkms if you look back at the post. I didn't need it for working OpenCL so I rather not have the modules installed.

[tuxd3v]

Does Blender recognize the OpenCL stack from ROCm?

FWIW: I bailed on installing rocm-dev which pulls in all sorts of junk I do not need and trashes Debian LLVM-5 and LLVM-6 installs.

Not sure how you're getting dkms to install as it bails on me with the Debian Sid 4.18 kernel.

I think you will need rocm-dev, because of:
/opt/rocm/opencl/include/CL/cl.h

Or I am confused now with AMDGPU-PRO...maybe that its the case..

I still have no Juice..: on rocm1.9
openat(AT_FDCWD, "/dev/kfd", O_RDWR|O_CLOEXEC) = -1 ENOMEM (Cannot allocate memory)

root@desktop:~# dkms status
amdgpu, 1.9-211, 4.15.0-34-generic, x86_64: installed
root@desktop:~# lsb_release -r
Release: 18.04

I still don´ t believe AMD is ignoring the APUs out there, as hosts..

[coder]

From what I heard the framework is tensorflow (which we saw on the marketing slides) and aside from that it was only a test if it even works. If it works, the next step would probably be getting a Vega 20 in the near future to try out if it can push a Titan.

Marketing slides are just that. The next step is to look at the framework you're using and see what the docs & release notes say about using AMD. Probably, you'll find that you need to use a fork on ROC's github - not even the mainline of that framework.

After you confirm the framework supports it, and understand what userspace components are needed and whether they're available via your distro or whether you have to build them, then you move on to the kernel/driver and again review what's available through your distro's official and unofficial repos.

Now, I can tell you this, because I've done it all. I'm using SuSE and had to build just about everything - I didn't even find any leap packages for CUDA-accelerated Caffe, and that's one of the earliest and most popular frameworks, on the most popular backend.

Also, point of RTFM is kinda unfair - if it would be his sole job to integrate and write frameworks, yeah - I can see your point. But he isn't and *nobody* beside him on the whole workgroup is even thinking of AMD (at least as far I can tell from the general feedback).

Call it what you want, but it's the reality, and it could've saved you both some time. I'm not advising you to do anything I didn't do myself. If you actually want stuff to work (not to mention minimizing wasted time and energy), then you can't afford to be naive about this stuff.

I had to evaluate deep learning hardware for my job, and I wanted to carefully consider all options.

[Happy Heyoka]

Correct - do not install rock-dkms but do perform the udev step. Do not move back to an older kernel... newer is generally better in this case.

Thankyou, that worked out fine in the end.

FYI

• github docs are currently a bit light on "let's not use rocm-dkms to pull everything in" install approach (just sayin' ) Synaptic to the rescue.
• Using 'xenial' as repo kind of grinds my OCD gears; when Ubuntu release 18.10 "Classy Chupacabra" (or whatever ), it might be nice to bump that.

In this case "interoperability" just means being able to have an application push data back and forth between (for example) OpenGL and ROC compute in real time

what I presume you mean by that is saving unnecessary transfers back and forth? I guess that'll get sorted eventually.

anyway, everything seems good!