AMD drivers are getting into very good shape and are backed by AMD itself (maybe not as much as it should be, but they keep on improving the closed source ones in the meanwhile).
Only leaves the nvidia drivers. That is a problem. But i don't see any reason for nvidia to support an unused display server just yet. If it became widely used, i have no doubts that nvidia would be quick to support it.
Bottom line, i don't really see a big problem in the driver department.