With the Linux 3.11 kernel there's Nouveau VP2 video engine support for accelerated H.264 and MPEG-2 playback. Merged last week was also the Nouveau Gallium3D support. This work builds upon already present Nouveau video decoding support and then the shared Gallium3D XvMC and VDPAU state trackers.
While the support is present in Mesa/Gallium3D and the Linux kernel, that's not all that's required for opening up video acceleration. Required still for this to work is closed-up NVIDIA firmware for being able to initialize the video engines. The Nouveau DRM driver isn't able to bypass the need for the firmware (or generate its own firmware) and there isn't any freely-redistributable firmware blobs.
Up to now for getting the VP1/VP2/VP3/VP4/V5 NVIDIA firmware has required loading the NVIDIA kernel module first on the system and then using MMIOtrace for dumping the firmware from memory. It's a bit of a headache, requires actually installing and temporarily using the closed-source driver, and just not end-user-friendly.
Fortunately, now a Python script has been authored for being able to analyze a NVIDIA Linux driver blob and extract the necessary firmware. The Python script will extract the firmware from a local copy of the NVIDIA Linux binary driver without actually needing to install or run the driver.
This script can be found on GitHub and makes it easier to setup the Nouveau video acceleration support for novice desktop Linux users. Complete steps along with other support/setup information for Nouveau hardware video acceleration can be found on the Nouveau Wiki. Of course, the other option is to just install and use the proprietary NVIDIA driver where you can have complete NVIDIA video decoding support over VDPAU in a relatively bug-free manner and experiencing great system performance.