OpenVINO 2024.0 Brings More GenAI Features, New JavaScript API

Intel today released their open-source OpenVINO 2024.0 toolkit for optimizing and deploying AI inference across a range of hardware.

OpenVINO is Intel's wonderful open-source AI toolkit for speeding up AI inferencing across not only x86_64 CPUs but also ARM CPUs and other architectures, Intel integrated and discrete graphics, and more -- including their recent NPU plug-in for leveraging the Intel Neural Processing Unit found with new Core Ultra "Meteor Lake" SoCs.

With today's OpenVINO 2024.0 release there is more generative AI "GenAI" focus with a better out-of-the-box experience for TensorFlow sentence encoding models, support for Mixture of Experts (MoE), a JavaScript API for seamless access to the OpenVINO API, and validated models for Mistral and more.

OpenVINO 2024.0 also brings improved quality on INT4 weight compression for LLMs, enhanced LLM performance on Intel CPUs, easier optimization and conversion of Hugging Face models, and other Hugging Face integration improvements.

OpenVINO 2024.0 does discontinue Intel's prior Gaussian and Neural Accelerator (Intel GNA) with now focusing on the NPU found with Meteor Lake and newer. The Intel NPU plug-in for OpenVINO is now also distributed as part of the main OpenVINO package on PyPi. For non-Intel users, OpenVINO 2024.0 does also sport better ARM CPU performance and various platform improvements.

Downloads and more details on today's big OpenVINO 2024.0 feature release via GitHub. I'll have out some new OpenVINO benchmarks soon.