OpenVINO 2024.4 Prepares For Core Ultra Series 2, New Gen AI Models

[phoronix]

Phoronix: OpenVINO 2024.4 Prepares For Core Ultra Series 2, New Gen AI Models

Intel engineers today released OpenVINO 2024.4 as the newest version of their open-source AI toolkit. OpenVINO 2024.4 prepares for upcoming Intel Core Ultra Series 2 "Lunar Lake" processors, supports newer Gen AI models, now supports Python 3.12, and finally adds official support for Red Hat Enterprise Linux 9...

https://www.phoronix.com/news/OpenVINO-2024.4

[Jabberwocky]

Hardware specific attention optimizations are really fascinating to me. The trick of making everyone happy is extreme optimization in datacenter/supercomputer parts while keeping devs happy with decent speed and stable drivers on consumer hardware.

Nvidia and Intel seems to be doing a good job. Microsoft also has been doing some magic. Here's my very biased current experience:

CUDA > OpenVINO *1 > ZULDA *2 > DirectML*3 > ROCm

*1 I'm just speculating on the marketing material and Intel's decent software track-record
*2 Some iGPUs like 780M works with ZULDA but not yet on ROCm (BLAS limitations IIRC)
*3 Best general supported out there but not as fast as CUDA/ROCm and a lot faster than OpenCL

I still went with AMD even though their software support is the worst, their hardware is the best price to performance. The side effect of going with AMD is that I'm forced to learn how the entire software stack works and now that ROCm is open source it becomes easier to do that. ZULDA is still hit and miss, but when it hits... it hits hard!

Off-topic: I've gotten para-virtualization to work on my RDNA 3 GPU. Video encoding AV1,H265,H264 as well as Vulkan, OpenGL, D3D12, ROCm, ZULDA works. Sadly it only works with a Windows host and Windows guest. Since it also works under WSL2 it should in theory be possible for 1 engineer to get it working in a Linux guest. To get it working on a Linux host, now that would be a challenge that most likely would require many engineers. The advantage that this brings: I can play games and do compute work in a VM using a single GPU. I've got another system that uses VFIO, Linux host and Windows guest. I think this new Windows host is faster, allows me to run more games and uses less electricity. There are some drawbacks obviously. Like if you remove a 200gb+ file in your VM or in WSL2 then Windows will BSOD and it's a known problem for years.

PS: I noticed this was posted after I wrote everything https://www.phoronix.com/news/Mesa-Gfxstream-Merged It could make things easier in the future, but I think we are still years away from what Windows does today.