An Open-Source Driver Effort Has Begun For Rockchip's NPU

Tomeu Vizoso who recently has been working on extending the Etnaviv open-source graphics driver to also support the Vivante NPU IP has made great progress on that with competitive performance to the proprietary NPU driver and upstreaming the Teflon framework into Mesa for handling the Neural Processing Unit. Tomeu Vizoso has now shifted his attention to working on an open-source, reverse-engineered NPU driver for the AI hardware found in various Rockchip SoCs.

With the Etnaviv/Vivante NPU open-source driver support in decent shape, Tomeu's next challenge has been to take on an open-source driver for the Rockchip NPU IP found in SoCs like the RK3588(S) and RK3568.

Currently, the official Rockchip driver consists of a GPL'ed out-of-tree kernel driver and a closed-source user-space. Vizoso summarized the current binary driver state as:

"The version of the NPU in the RK3588 claims a performance of 6 TOPS across its 3 cores, though from what I have read, people are having trouble making use of more than one core in parallel, with the closed source driver.
...
The kernel driver is pleasantly simple and relatively up-to-date in regards of its use of internal kernel APIs. The userspace stack though is notoriously buggy and difficult to use, with basic features still unimplemented and performance being quite below what the hardware should be able to achieve.

To be clear, this is on top of the usual problems related to closed-source drivers. I get the impression that Rockchip's NPU team is really understaffed."

His hope ultimately is to develop both a mainline-capable open-source kernel driver and relevant user-space for the Rockchip NPU. So far he's off to a good start:

"I started looking at a buffer that from the debug logs of the proprietary driver contained register writes, and when looking at the register descriptions in the TRM, I saw that it had to be closely based on NVIDIA's NVDLA open-source NPU IP.

With Rockchip's (terse) description of the registers and NVDLA's documentation and source code for both the hardware and the userspace driver, I have been able to make progress several faster than with VeriSilicon's driver (for which I had zero documentation).

Right now I am at the stage at which I am able to correctly execute TensorFLow Lite's Conv2D and DepthwiseConv2D operations with different combinations of input dimensions, weight dimensions, strides and padding. Next is to support multiple output channels."

More information on this new open-source NPU Linux driver adventure via Tomeu Vizoso's blog.