H.264/MPEG2 video decode support was published last month for the Nouveau driver. The accelerated video playback works for some NVIDIA GPUs between the GeForce 8600 and GeForce 200 generations. The video decode is done using a "VP2 engine" found on these supported GPUs as part of NVIDIA's "PureVideo HD" technology.
The VP2 engine is a bitstream processor and by using it for decode, XvMC is about 20% faster than using CPU-based decoding for Nouveau and VDPAU will also benefit too via its Gallium3D state tracker.
The kernel-side support for exploiting this NVIDIA video engine was merged for the Linux 3.11 kernel. Merged now is the Nouveau NV50 Gallium3D driver change for supporting the NVIDIA VP2 engine from the open-source user-space.
The Gallium3D patch, which has been available on the mailing list since last month and is now only in Mesa master for Mesa 9.2, adds the H.264 and MPEG2 support. The VDPAU and XvMC state trackers as part of Gallium3D in Mesa can now make use of it.
Before getting too exciting, in order to use the VP2 engine for now you must be using the firmware/microcode from the proprietary NVIDIA Linux graphics driver. The Nouveau DRM isn't yet able to generate its own necessary firmware for controlling the engine (nor is there any freely-redistributable firmware), so you won't see this accelerated video support by default.
Beyond the big firmware shortcoming, H.264 interlaced videos aren't rendering properly, H.264 is showing artifacts on a small fraction of videos, and MPEG2 + VDPAU can also show some small artifacts.
The merged patch for VP2 in Gallium3D for the NVIDIA NV50 hardware is just under two thousand lines of code and can be found by this commit.