Llamafile 0.8.5 Delivers Greater Performance: Tiny Models 2x Faster On Threadripper

Written by Michael Larabel in Programming on 25 May 2024 at 06:50 AM EDT. 7 Comments
PROGRAMMING
The Mozilla Ocho group has published their newest version of Llamafile, the open-source project that makes it very easy to distribute and run large language models (LLMs) as a single file. Llamafile is an excellent solution for easily sharing and running LLMs and supporting both speedy CPU-based execution as well as GPU acceleration where available.

Llamafile 0.8.5 is the newest version and delivers on yet more performance tuning... On top of the recent work around AVX2 optimizations, more AMD GPU offloading, and other work. Justine Tunney explained of the latest performance work in Llamafile 0.8.5:
"As of #435 the K quants now go consistently 2x faster than llama.cpp upstream. On big CPUs like Threadripper we've doubled the performance of tiny models, for both prompt processing and token generation for tiny models."

Doubling the performance for tiny models on AMD Ryzen Threadripper class hardware!

HP Z6 G5 A
HP Z6 G5 A with AMD Ryzen Threadripper PRO 7000 series


Llamafile 0.8.5 also delivers faster AVX2 matrix multiplication for MoE models and legacy quants. There are also some AMD Zen 4 performance optimizations, BF16 NVIDIA CUDA support, and other improvements.

Downloads and more details on the Llamafile 0.8.5 release via GitHub. I'll be working on new LLamafile benchmarks soon.
Related News
About The Author
Michael Larabel

Michael Larabel is the principal author of Phoronix.com and founded the site in 2004 with a focus on enriching the Linux hardware experience. Michael has written more than 20,000 articles covering the state of Linux hardware support, Linux performance, graphics drivers, and other topics. Michael is also the lead developer of the Phoronix Test Suite, Phoromatic, and OpenBenchmarking.org automated benchmarking software. He can be followed via Twitter, LinkedIn, or contacted via MichaelLarabel.com.

Popular News This Week