Will H.264 VA-API / VDPAU Finally Come To Gallium3D?

Assuming the student developers participating in this year's Google Summer of Code achieve their work (after getting accepted of course), this year could be very interesting for Mesa / Gallium3D / X. While initially there was the very ambitious OpenGL 4.1 plans in a new Gallium3D state tracker that would be free of Mesa legacy code, that was changed to working on GLSL IR or something smaller (perhaps Clover, as in the long-awaited OpenCL state tracker for Gallium3D). There's also been a proposal for multi-GPU and hot-plugging support. Voiced just now by a French student is to create the -- also much-anticipated -- H.264 VA-API / VDPAU state tracker for Gallium3D drivers.

The proposal by Emeric Grange is laid out in this email. It entails creating an H.264 state tracker for Gallium3D where on the front-end it would interface with VDPAU (NVIDIA's Video Decode and Presentation API for Unix) or VA-API (the Video Acceleration API). Both APIs are great choices; more so than using AMD's XvBA or trying to wrestle XvMC into doing something greater. If VDPAU is targeted by the state tracker, it would also be possible for VA-API applications to leverage it by using Splitted Desktop System's VA-API to VDPAU library.

Generic video decoding is not new to Gallium3D or even to being worked on with Google's Summer of Code. Back in 2008 was the first attempts at Gallium3D video decoding when MPEG support in shaders and exposed by XvMC (X-Video Motion Compensation) was the target. It made progress with the Nouveau driver, but the work is not heavily used at this time.

AMD has also experimented with XvMC for their Gallium3D driver and last year it became possible to use XvMC with the R600g driver, complete with accelerating iDCT. X-Video has also been talked about and worked on within the Xorg state tracker, but that does even less work.

There's been some talk and early work towards implementing VDPAU on Gallium3D -- and a H.264 state tracker proposal for GSoC was proposed in past years -- but as of yet nothing has really materialized in this area. Emeric Grange hopes to change it this year.

He's looking to get H.264 video decoding using shaders working with VA-API / VDPAU for the R300g, R600g, Nouveau, and "basically all Gallium3D drivers." Additionally, "The project would be to write a state tracker which expose some of the most shaders-friendly decoding operations (like motion compensation, idct, intra-predictions, deblocking filter and maybe vlc decoding) through a common API like VDPAU or VA-API. These APIs can be used to decode mpeg2, mpeg 4 asp/avc, vc1 and others, but at first I intend to focus on the h264 decoding to save time, because I know it better and it is currently widely in use, but again the goal of the project is to be generic."

While in theory this would be generic across all Gallium3D drivers since it's using shaders, such H.264 video acceleration would not be using the AMD Unified Video Decoder 2 (UVD2) or NVIDIA PureVideo HD dedicated video ASICs found on modern Radeon and GeForce graphics cards, respectively. The only other hardware with H.264 video playback acceleration is Intel's Clarkdale/Arrandale/Sandy-Bridge graphics processors with the updated graphics stack that's exposed by VA-API. There's also Broadcom's Crystal HD dedicated card for accelerating similar work.

AMD isn't releasing code or specifications for their UVD/UVD2 engines over fear that it could compromise their Digital Rights Management abilities on other platforms. On the NVIDIA side, the Nouveau developers have only made early attempts at reverse-engineering NVIDIA's video engine. The Nouveau developers are also looking at exploiting video acceleration over OpenCL, but alas first there needs to be OpenCL support in the open-source Linux graphics drivers.

Let's hope though that this Gallium3D H.264 state tracker is accepted and that it actually materializes this summer.