AMD Aims For AMF Decode In FFmpeg, Questioned Over Vulkan Video Commitment

AMD last week sent out a set of patches to enhance the open-source FFmpeg multimedia library with integration around the AMD Advanced Media Framework (AMF). The AMF SDK allows for "optimal" access to AMD GPUs for multimedia processing but this patch series questioned the need in an era of Vulkan Video APIs beginning to see adoption.

The newest AMD FFmpeg patch series for AMF is on adding hardware context "hwcontext_amf" support along with AMF-based H.264, HEVC, and AV1 decoders. The patches also enable AMD SmartAccess Video (SAV) functionality for AMF encoders. SAV enables the parallelization of encode/decode streams across multiple VCN hardware instances. There are also two AMF filters initially proposed for FFmpeg: "vpp_amf" for simple scaling and color conversion and then "sr_amf" for advanced scaling algorithms like FSR. There is already FFmpeg AMF support on the encode side.

Dmitrii Ovchinnikov explained with the patch series:

"Adds hwcontext_amf, which allows to use shared AMF context for the encoder, decoder and AMF-based filters, without copy to the host memory. It will also allow you to use some optimizations in the interaction of components (for example, SAV) and make a more
manageable and optimal setup for using GPU devices with AMF in the case of a fully AMF pipeline. It will be a significant performance uplift when full AMF pipeline with filters is used."

FFmpeg developer Lynne who has done a lot of the Vulkan Video enablement for this dominant open-source library questioned though why AMD is still working so much on AMF when they could leverage the more open Vulkan Video ecosystem. Lynne wrote in the messaging exchange:

"Vulkan encode is quite soon going to get all features needed to allow for all vendor-specific optimizations to be exposed, though I think you should already be in the loop. Is there a reason to add this code in now? AMF is still hardly that used, at least in the Linux world, as it isn't really packaged everywhere, and on Windows, D3D12 is gaining more traction.

AMD has also already released all FSR-based code for upscaling."

To which the response was sadly:

"DX12 and Vulkan native encoders will expose less features compare to AMF, at least in foreseeable feature. The missing features include low latency, PreAnalysis including look-ahead etc. AMF context on Windows allows fully enable SAV - ability to utilize VCNs in dGPU and APU in a single session. Eventually specialized multimedia AMD cards could be added seamlessly to FFmpeg with AMF integration. AMF FSR(VSR) includes YUV version with focus on videos which is not available in AMD FSR aimed for gaming."

FFmpeg developer Lynne still isn't convinced though whether this AMD AMF update should be merged into FFmpeg:

"Do you really not talk with each other over there? You should. This is a lot of vendor-specific code for which an overlap
with a standard API already exists, and I'd just prefer to know why this should be merged and maintained now, as Vulkan video adoption is finally starting."

The latest patches can be found here. So far the patch series hasn't been merged to upstream FFmpeg, so we'll see if it's ultimately accepted or if it's rejected in favor of encouraging more open / industry standard APIs in 2024.