Martin Peres and Timothée Ravier talked yesterday at XDC2012 about the graphics stack security and its current shortcomings.
There's some security safeguards within the Linux graphics stack currently over X.Org, but much more could be done. However, greater enhancements to secure the graphics stack from potential attackers will likely not come until Wayland is out there.
Potential outstanding attack vectors, ideas for improvements, and much more are shared in the video below.
On a similar security note, Alan Coopersmith also talked briefly about the X.Org Server privilege model that Oracle uses on Solaris and what parts of that could be upstreamed.