Llamafile 0.8.1 GPU LLM Offloading Works Now With More AMD GPUs

Written by Michael Larabel in Radeon on 27 April 2024 at 08:48 AM EDT. 8 Comments
It was just a few days ago that Llamafile 0.8 released with LLaMA 3 and Grok support along with faster F16 performance. Now this project out of Mozilla for self-contained, easily re-distributable large language model (LLM) deployments is out with a new release.

Most significant with Friday's Llamafile 0.8.1 release is getting GPU support working for more AMD graphics processors / accelerators. Due to some of the AMD offload code within Llamafile only assuming numeric "GFX" graphics IP version identifiers and not alpha-numeric, GPU offload was mistakenly broken for a number of AMD Instinct / Radeon parts. For hardware like the Instinct MI250 with the GFX90A IP, the "A" was not being correctly parsed and not passed to the HIP compiler. In turn this would error out and break Llamafile GPU acceleration on AMD GPUs having non-numeric characters as part of their GFX identifier. That's now fixed up with Llamafile 0.8.1 and thus AMD GPU acceleration working on more hardware for Llamafile-based large language model deployments.

Additionally, Llamafile 0.8.1 now ships pre-built NVIDIA and AMD ROCk modules for both Windows and Linux users for further easing the deployment of Llamafile single-file LLMs that support both CPU and GPU execution.

Llamafile 0.8.1 also adds support for the Phi-3 Mini 4k model, fixed a bug causing GPU model crashes. support for Command-R Plus has proper 64-bit indexing, and other fixes.

Downloads and more details on the new Llamafile 0.8.1 release via Mozilla-Ocho on GitHub.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week