Intel Releases OpenVINO 2023 - Load TF Models Directly, Hybrid CPU Thread Scheduling

Intel has released a major update to its wonderful, open-source OpenVINO toolkit for optimizing and deploying AI inference. OpenVINO continues working out great for optimizing and running AI models on a variety of hardware and continues to introduce new features.

The OpenVINO 2023.0 release now allows for TensorFlow and TensorFlow Lite models to be loaded directly into the OpenVINO Runtime and OpenVINO Model Server. Loading the TF/TF-Lite models directly will cause the models to be converted automatically but the developers still recommend converting to OpenVINO IR in advance. There is also experimental support for loading PyTorch models directly too without first having to convert to ONNX.

OpenVINO 2023.0 is also significant in that it now supports the latest Python 3.11 series, ARM processor support in the OpenVINO CPU plug-in including official support for the Raspberry Pi 4 and Apple M1/M2, and broader model support and various new optimizations.

The OpenVINO 2023.0 CPU plug-in also adds thread scheduling support for Intel hybrid CPUs of Alder Lake 12th Gen Core and newer. With this thread scheduling you can choose to run inference on just E or P cores, or both a combination of the energy efficient and high performance cores. OpenVINO 2023.0 also has a new default inference precision where OpenVINO will default to the format that enables optimal performance, such as BF16 on the latest Intel Xeon Scalable CPus or FP16 when dealing with GPUs.

Downloads and more details on the OpenVINO 2023.0 release via GitHub.