Mining Ethereum With AMD Threadrippers Paired With Four RX Vega 64 GPUs
In this article I will write about my new protect to build 2 dual-use systems means Desktop+Ethereum-Mining.
If you do not have time to read the full article here the short version in one sentence: No, it is not practical to run Ethereum on AMD Vega 64 graphics cards on Linux because the ROCm OpenCL stack in the AMDGPU-PRO 17.30 is very slow in Ethereum so the whole system with four Vega64 cards make 16-23Mh/s and even a single old AMD-RX470 with the old Closed-source OpenCL AMDGPU-PRO stack run ~20+ Mh/s... This all could be the effect of the exponential function Ethereum-ICE-AGE Bomb...
But now the long version:
I ran into unexpected building problems in the time of build this 2 systems. The ASRock X399 Taichi has two short screws on the three-screw CPU socket this results in the problem that I was unable to screw in the screws by hand. So I needed a electric drill and relatively high pressure to get the first traction and then I used the hand-screw tool from the Threadripper 1900X/1920X CPU. The MSI X399 gaming pro carbon instead had a perfect mechanical socket and I was able to use the hand screw tool to mount the CPU perfectly. So the clear win in mechanical quality goes to MSI.
Both mainboards have WiFi included but only the ASRock X399 Taichi has onboard WLAN and the MSI does only have a WLAN PCIE card but if you use all four PCIE 16 slots with dual-high AMD Vega cards the result is that the MSI is without wireless because there is no space left to install the PCIe WiFi card. So ASRock is the clear WiFi/wireless/WLAN winner.
I ran into big problems because of the BIOS version of the boards: The 1900X was released one month after the 1920+1950 so you need a more expensive CPU to install the BIOS updates because 1900X is only supported in later revisions. Also I ran into problems because the AMD-VEGA is also not supported by earlier BIOS versions and also 2/3 of the RAM DIMMS I bought where also not supported and I even bought a more expensive 1920X cpu and three different kinds of RAM DIMMs to perform a BIOS update. Atfer the updates, all the RAM DIMMs worked fine as well as the VEGA GPU and 1900X CPU. I did not even try to make the same on the ASRock so O bought the ASROCK with Bios-update from the store.
This means if you are a beginner and you do not want to run into "BIOS" incompatibility problems it is highly recommended to buy the mainboard with pre-installed bios updates from the store.
The X399 mainboards all look like you can use four AMD Vega cards but they run into thermal overheating problems because the space between the cards in the middle is very tiny and even with mechanical tricks to expand the space between the cards I lost ~20-30% mining power of these expensive Radeon Vega cards. AMD really needs to bring mining edition cards to the market with passive copper cooling blocks without monitor HDMI plugs to make it possible to install external cooling fans to push the heat out of the PC-case.
I also ran into problems because the ASRock Taichi has no M.2 SSD slot screws and because of this I had to buy these screws for 7,99€ on eBay and the MSI X399 board had a perfect set of screws for every M.2 SSD slot on the mainboard. So the MSI beats the ASRock if you want to use the M.2 SSD slots.
The ROCm Stack in the AMDGPU-Pro 17.30 driver is very slow: a AMD VEGA 64 has results like 40Mh/s on the old Closed-source Catalyst/FGLRX/AMDGPU driver but now the ROCm stack is the new default for Linux installs on the LTE 16.04 ubuntu releases. I only needed like 2 minutes of clicking around with this (irony)quality(/irony) closed source driver then the KDE-info-center crashes with segmentation faults.
The 17.30 driver had no working default OpenCL because it was optional to install but the installer did not show any information about this optional openCL driver part. But AMD's John Bridgman wants to bring this info into the installer of the AMDGPU-pro driver installer or fix the default that the users get a working OpenCL after the install:
"I don't know why compute/OpenCL was left as optional but am asking around. Not sure we will be able to change that quickly but if not then at least I want to see us get a big message at the top of the install instructions noting that OpenCL/compute is not installed by default but that instructions are further down the page." Bridgman
But for sure this extra information and steps bring the OpenCL ROCm stack to work:
Installing the Optional ROCm Component This AMDGPU-Pro driver package incorporates the ROCm component that can be optionally installed for running Compute/OpenCL applications. You can install the component by issuing the following command:
sudo apt install -y rocm-amdgpu-pro
Configuring the Optional ROCm Component
The LLVM_BIN environment variable needs to be set prior to running ROCm applications.
To set it temporarily when running an individual ROCm command, such as clinfo, use:
env LLVM_BIN=/opt/amdgpu-pro/bin /opt/amdgpu-pro/bin/clinfo
To set it permanently for all bash and other sh-like shell users, you can use the following command:
echo 'export LLVM_BIN=/opt/amdgpu-pro/bin' | sudo tee /etc/profile.d/amdgpu-pro.sh
See the Ubuntu Environment Variables Community Help for more information.
More details via AMD Support.
Now comes the good part I asked for support here and the AMD-Linux support by Bridgman is legendary good.
And it really looks good for the future so we will get a closed-source OpenCL-only installer for the RadeonSI FOSS driver this should fix the segmentation fault of the KDE infocenter because with RadeonSI all desktop features work perfect:
"[...]but if you mean "install openCL and use it alongside radeonsi" then yes we are working on that.
It will be packaged and tested with the DKMS/KCL kernel driver (copied & pasted that last word) rather than upstream but we are also trying to support newer kernel/X releases so should work OK. That won't be in 17.40 though... ", wrote Bridgman.
According to Bridgman the performance problems should be fixed by the AMDGPU-PRO 17.40 driver at the end of this month:
"Initial feedback is that latest ROCm (1.6.3) has the performance fixes, but there will be a 17.40 release soon which also includes them. We are trying to arrange an early release of 17.40 which should be what you need. If you check airlied's drm-next branch you will see kfd commits starting to show up frequently; IIRC that goes into 4.15. We're going to try to also get that backported to upcoming LTS and enterprise distro releases."
As we can see AMD works on upsteaming the kernel features into the future 4.15 Linux kernel. The two performance relevant features will be 2MB pages and memory interleaving HBM banks
I do not know yet how much performance we will earn from these performance optimizations. But he also still wait for AMD staff to get to know what is the recommended driver stack for Ethereum:
"I'll find out what driver stack we are recommending for ethminer - IIRC we were seeing performance drops with latest workloads and made a couple of driver changes to address them. One was use of 2MB pages (which is not in 17.30 IIRC) and another was related to memory interleaving between HBM banks... each of them made a noticeable difference. The answer might be "use the ROCm stack in the short term" but will find out." Bridgman
A funny side note is that I used a 64-bit Windows 10 installation DVD to try Windows-mining too but my MSI X399 system was unable to start the windows 10 installer DVD. so if you think the Ethereum 6Mh/s from the ROCm Stack on linux is slow think about the 0 Mh/s on Windows! ;)
Also remember that Mega Hashrates on Ethereum is not always the same because of the Ethereum Ice Age mechanism according to this source:
"The Ethereum Ice Age is a difficulty adjustment scheme that was put in place to ensure that everyone has an incentive to move to the new blockchain once the hard-fork is implemented," from here.
This means on 18 October 2017 is the reset of the ICE-AGE bomb then we will see normal results again. Right now the difficulty adjustment goes very high because the exponential function ICE-AGE Bomb...
This mechanism is there to force the miners into using the newest software to make sure their development of Ethereum is unified.
Because of this I will write a second article after the 18.10-2017 Ethereum ICE-AGE-BOMB is disarmed then I will do some Energy bill Measurements and I will bring fresh numbers of the 17.30 vs 17.40 drivers.
Right now according to the Ethermine.org pool statistic my MSI mining PC show that my PC would make 40-50dollar a month if running 24 hours a day right now it only run at day time and I turn it off for sleeping. This means the ~4000€ PC would never pay out with 17.30 ROCm and ICE-AGE bomb...
Also the CPU usage is 0% so the 1920X Threadripper 12-core CPU brings no benefit over a 1900X 8core cpu. Right now I also use only a single 16GB RAM DIMM but the PC use less than 4GB RAM.
To be continued...
47 Comments