Using Nouveau With Gallium3D Video Decoding
Gallium3D video decoding support was actually started with a focus on using the Nouveau driver for NVIDIA hardware around three years ago. It was started by Younes Manton, a student X.Org developer, as a Google Summer of code project. While XvMC was working okay back then with the hardware and open-source driver at that time, it's no longer in shape.
As said in this email message by Younes, there was a developer who has written basic NVIDIA NV50 support code for this video decoding work, but it's now outdated and hasn't been merged into the Mesa video branch. Another individual had also wrote NV40 support code.
One of the commonly talked about benefits of the Gallium3D driver architecture is that the GPU hardware drivers are simpler and that more code is shared between drivers. It's supposed to be to the point that one Gallium3D state tracker providing support for APIs such as OpenGL, OpenCL, OpenVG, OpenGL ES, XvMC, VDPAU, etc can in turn target any Gallium3D hardware driver in a near seamless manner. However, at this point, it doesn't work so well with all of the Gallium3D drivers still maturing and being in heavy development along with the core Gallium3D code itself.
The XvMC and VDPAU state trackers (and others) can't magically run on all drivers that don't provide the necessary interfaces to begin. For some of this video work, changes may be needed just not to the Gallium3D user-space driver, but to the kernel DRM driver as well.
The R300 Gallium3D driver for providing hardware support up through the Radeon X1000 series was also the first target for the ATI XvMC work. The support here should be a bit better, but at the moment the R600g driver is in the best shape.
Besides the differing states of driver support for XvMC / VDPAU, all of this work is still currently living in Mesa's pipe-video branch, which means regardless interested users will need to be building their own Mesa code if wishing to experiment with shader-based video acceleration under Gallium3D.