AMD's X-Video Bitstream Acceleration video API is a mess and has been that way from the beginning. I was the first to publicly out this new AMD-created video API when it first appeared in their proprietary driver. The video API appeared in H2'2008 but it wasn't until months later that Phoronix exclusively detailed what is XvBA. AMD tossed in the XvBA libraries into their binary driver but never formally announced it nor did they provide any API documentation, program patches, or other support to actually make it work.
The XvBA bits were sitting in their Catalyst Linux driver for more than a year until they actually became useful. With AMD not officially acknowledging the X-Video Bitstream Acceleration interface or providing program support / API documentation, the libraries just sat there. In November of 2009 it finally became useful when a VA-API to XvBA wrapper was released. This was written by Gwenole Beauchesne of Splitted Desktop Systems. He obtained the XvBA documentation under NDA with AMD and so was able to write this closed-source wrapper library.
This library exposed VA-API to user-space programs and then translated it to the respective XvBA functionality. This library remains the main way to use video acceleration on the AMD Catalyst driver today, although it's not actively maintained any longer since Gwenole left Splitted Desktop Systems for Intel.
It was only in early 2011 that AMD opened up XvBA to make it more open, but still it's only their Catalyst driver that actually utilizes this application programming interface. Even the open-source AMD driver isn't supporting XvBA.
XvBA has been buggy from the start and -- as said in yesterday's article by the XBMC developers -- is still buggy and not actively progressing. With only the shoddy implementation provided by the notorious Catalyst Linux driver, and any public documentation / sample code being very late to leave AMD, XvBA really isn't targeted by the Linux multi-media applications but rather VA-API and VDPAU are the common support targets.
In a pristine state that AMD's XvBA will likely never reach, there is NVIDIA's VDPAU, a.k.a. the Video Decode and Presentation API for Unix. The NVIDIA-backed Linux video API has been around since 2008 as a way to expose PureVideo capabilities under Linux. While it's implemented in the proprietary NVIDIA driver, from the start this video API has been open and friendly towards multimedia program developers.
With an open and well supported video API from NVIDIA, VDPAU support is now widespread throughout the popular programs like MythTV, XBMC, VLC, MPlayer, etc. VDPAU is also supported by Adobe's Flash Player.
Besides the video API being supported across major software projects, the main implementation found in the NVIDIA binary driver is very well supported and continues to be advanced to this day. Across NVIDIA's range of graphics cards the VDPAU acceleration works great and rarely do I ever see any complaints or issues concerning its support. NVIDIA VDPAU is wonderful.
VDPAU has also begun to be supported via a Gallium3D state tracker for the open-source drivers, but there it's using GPU shaders rather than being accelerated on a dedicated video decoder (e.g. AMD UVD2 or NVIDIA PureVideo HD). The S3 Graphics driver even claims VDPAU support although I don't have any modern discrete S3 graphics card nor do I actually know anyone with such hardware to confirm this claim.
VA-API is the second dominant video API to VDPAU for suitable video acceleration under Linux. VA-API is the choice of Intel's open-source Linux graphics driver. Particularly if using Sandy Bridge or Ivy Bridge graphics hardware with their open-source driver, the Video Acceleration API is wonderful.
There's been some work on VA-API support via a Gallium3D state tracker -- also using GPU shaders, but reportedly the VDPAU support is in a better state for Gallium3D hardware drivers. There's also been some other driver implementations, but Intel's use of VA-API remains the main implementation. Besides the VA-API to XvBA wrapper there was also originally a VA-API to VDPAU wrapper, also developed by Splitted Desktop Systems.
Google's VAVDA for their web-browser / OS also is an implementation of VA-API. Currently lacking VA-API support is the mainline MPlayer (there's a separate branch / patches with the support) and Adobe's Flash Player, although the open-source "Gnash" player does support VA-API. Aside from that the VA-API support is just about as widespread as VDPAU. VA-API already works on Wayland too.
The main points for Linux video acceleration on the desktop come down to: if you're not scared of tainting your kernel with a binary blob, NVIDIA's VDPAU support offered by their proprietary driver is wonderful! If you want an open-source driver with video support, Intel Sandy/Ivy Bridge hardware paired with VA-API is also splendid.
Intel's solution is in fact the only real open-source GPU-based option (though while not on the GPU, Broadcom's Crystal HD solution is also reportedly good under Linux). AMD has yet to publish any public documentation or open-source code concerning their UVD video engine for any generation of hardware, due to Digital Rights Management concerns. The Nouveau developers have made some progress in reverse-engineering NVIDIA's PureVideo engine, but the code is not in a shape comparable to VDPAU with the binary driver.