On the NV84 (GeForce 8600) through NV96 (GeForce 9400~9600~9700) GPUs and NVA0 (GT 200) graphics processors is a V2 engine that is now supported by the reverse-engineered open-source NVIDIA graphics driver. The NVIDIA VP2 engine is a bit-stream processor for decoding H.264 and a video processor to take care of certain video operations on MPEG/H.264/VC-1 streams.
A patch posted on Sunday loads a kernel for these engines, which are based on embedded Xtensa chips, and for loading the blob firmware from user-space. If you're using the NVIDIA binary Linux driver, this engine is under the "PureVideo HD" brand and the engine is exposed via VDPAU.
This VP2 engine work for select NVIDIA GeForce/Quadro GPUs was done by Ilia Mirkin and the first patch was published to the Nouveau mailing list as the support isn't yet part of the mainline Linux kernel.
Ilia Mirkin has now published the Gallium3D driver patches against the NV50 driver for exposing the VP2-accelerated H.264 and MPEG-2 codec support. While this is a step forward for hardware-accelerated video playback on Nouveau, H.264 interlacing isn't rendering right, there's some H.264 artifacts on occassion, and MPEG-2 with VDPAU shows artifacts. Via the Gallium3D driver, this support can be exposed via the VDPAU and XvMC video APIs.
In terms of performance, Mirkin notes with his current Gallium3D NV50 patch, "XvMC ends up ~20% faster than plain CPU decoding, and likely that percent improves on older CPUs that can't decode MPEG2 quite as quickly. VDPAU provides further improvements (likely because it is able to skip mb's while XvMC can't), but there are artifacts for reasons unknown."
The Gallium3D patch at just under two thousand lines of code hasn't yet been merged into Mesa but can be found on the mailing list. Meanwhile the VP2 Nouveau DRM patch will hopefully make it into the Linux 3.11 kernel.