Linux's Hantro VPU Media Driver Looks To Be Promoted Out Of Staging

[phoronix]

Phoronix: Linux's Hantro VPU Media Driver Looks To Be Promoted Out Of Staging

The Hantro VPU driver that has been developed the past several years for supporting the Hantro video/image processing IP found on various Rockchip and NXP SoCs is set to be promoted out of the Linux kernel's staging area...

https://www.phoronix.com/scan.php?page=news_item&px=Hantro-VPU-Leaving-Staging

[johanb]

Since it mentions i.MX8M, does that mean that we're close to hardware accelerated decode (and encode?) on a mainline kernel for the Librem 5?
It seems like the "stateless api" mentioned in the mailinglist is a generic interface in v4l2, so I assume nothing else is needed in userspace?

This not my area of expertise, sorry if it's a stupid question

[LoveRPi]

I'm going to break this down for people who are not in the embedded system space.

There are various APIs accelerating the display of encoded videos. OpenMAX (OMX), VDPAU, and VAAPI. Vendors implement software to run their custom encoding and decoding hardware to these API standards so that other software can be hardware agnostic. The problem is that most of the implementations of the API interfaces use proprietary kernel interfaces. V4L2 M2M changes this.

On low power devices with limited CPU and memory performance, hardware acceleration is essential. The video encode/decode engine and the video output engine (VOP) are mostly UMA (uniform memory architecture). Where as on PCs, copying video buffers via DMA is a small proportion of the memory bandwidth budget, on low power devices, copying alone can use the entire memory bandwidth budget. V4L2 M2M allows the hardware video decode accelerator to decode the data to a buffer and pass the pointer of that buffer to the video engine to be scanned out to HDMI. It doesn't have to copy the buffer needlessly and waste limited memory bandwidth.

V4L2 M2M is broken down to two types, stateful and stateless. stateless is using the hardware features to accelerate components of the video stream. State machine management is not done in hardware/firmware. Stateful means that the hardware/firmware is also managing the state machine of the video stream.

[binarybanana]

Thanks for the clarificatiin, but you seem to suggest that APIs like VAAPI and VDPAU require copying buffers in and out of RAM, which isn't the case. mpv even lets you explicitly choose if you want to copy decoded frames back to RAM to for compatibility reasons or to apply some CPU based filters on top, but otherwise it all remains in VRAM.

Or did you mean to say that these blocks have their own memory (mapped into the memory space) into which they decode frames? Because otherwise I'm not sure why there would ve a need for a copy. These devices only have system RAM anyway (UMA as you say).

[LoveRPi]

Thanks for the clarificatiin, but you seem to suggest that APIs like VAAPI and VDPAU require copying buffers in and out of RAM, which isn't the case. mpv even lets you explicitly choose if you want to copy decoded frames back to RAM to for compatibility reasons or to apply some CPU based filters on top, but otherwise it all remains in VRAM.

Every paragraph section is independent and I didn't mean the convolute the ideas. The issue with VAAPI and VDPAU is that the underlying interface implementation is not fixed. It could be a kernel hardware interface with an userspace driver. V4L2 M2M implementation is strictly in kernel with the standard kernel interface.

There are upside and downsides to each way of implementing it. V4L2 M2M can be wrapped by an hardware agnostic VAAPI layer or VDPAU layer. V4L2 M2M do need to have more controls available though. Userspace applications can adopt it more.

[JuanSF]

Hello everyone. In the same way, at some point I hope the day that cedrus in cpu's allwinner is complete (although they can already be used). Plus the isp drivers in v4l2-m2m will also help with the processes related to the images in cameras

[JuanSF]

How wonderful!