VA-API, VDPAU, and XvBA are among the video decoding APIs out there already for Linux that currently work under X, but they must be modified to work with Wayland. Intel's started on the VA-API support for Wayland, since that's their video acceleration API of choice with it being used by their in-house Intel graphics driver. Intel Sandy Bridge hardware with VA-API is excellent and mighty fast for both video encoding and decoding.
Last week Zou Nanhai mentioned on the Wayland list that they (the China team) are working on implementing libva (the VA-API library) for Wayland. He proceeded to ask a few technical questions.
Kristian Høgsberg then mentioned that some libva work was already started in this Git repository. Benjamin Franzke started initial work on supporting VA-API under Wayland towards the middle of last year.
There's still no mainline support for Wayland VA-API yet, but it will likely be here by Wayland 1.0 later in the year. Meanwhile, there haven't been any visible efforts yet by other parties in enabling VDPAU to work within a Wayland compositor.
NVIDIA wouldn't be expected to support their Video Decode and API for Unix implementation in Wayland until their proprietary driver actually works with Wayland... When that will happen, we don't know yet due to the technical differences. It will likely be a while. NVIDIA's stance right now is they don't have any plans to support Wayland, but that is surely going to change. There is the VDPAU state tracker in Gallium3D that some drivers are supporting, but there hasn't been any pro-Wayland activity there quite yet.