Big Video Encoding Rework Lands For AMD Open-Source Mesa 24.3 Driver

A set of a dozen patches have reworked the video encode handling within Mesa 24.3 for the Video Acceleration (VA) front-end and the RadeonSI/VCN driver code. This rework aims to enable new features moving forward, enhance the overall driver, and bring "significant" memory savings for H.265/HEVC video encoding.

Open-source developer David Rosca has seen his dozen patches merged for reworking video encoding in Mesa 24.3 with a particular focus on the AMD Radeon driver support. In the merge request he explained of this work:

- Full support for features like long term references, ref pic reordering, ref pic invalidation and more
- Removes a lot of unnecessary driver logic
- Preparation for more improvements that wouldn't be possible to implement otherwise
- On radeonsi significant memory savings with HEVC encoding

Currently there are two main issues with encoding: how we handle ref pics in DPB and how we write headers.

Drivers now get only limited information about what references are used to encode given frame, and with this limited knowledge they implement their own logic to handle DPB (slot where to store current frame, which references to evict, ...). VAAPI requires applications to do their own DPB management, so this will easily break if application tries to do anything more advanced than the basic sliding window. Solution here is to simply switch to app DPB management, this completely fixes the issue and allows us to simplify the driver by removing the logic that shouldn't be there.

Due to above, drivers cannot implement correctly any advanced features like long term references, ref pic reordering and ref pic invalidation. VAAPI requires applications to write the headers and provide them to driver, so we need to ensure the parameters set there will also be present in the output bitstream. Currently we have limited support for parsing VPS/SPS/PPS/Slice headers, this MR significantly extends it make it possible to support above mentioned features. Drivers still need to write these headers, and possibly override some parameters if needed due to encoder constraints, so those can't be passed directly.

Plan is for all drivers to switch to the new interface, and then remove the old one (frame number references and other unused fields).

While this delivers on memory savings right now for GPU-accelerated HEVC encoding, improves driver logic, and other feature work, this code still needs to be adapted to benefit AV1 encoding. There are also plans to allow drivers to directly use the surfaces as DPB buffers, among other changes.

As of this morning this video encoding rework has landed in Mesa 24.3-devel for debut as part of the Q4-2024 driver feature release.