Believe we are already doing this for GCN hardware. It's not actually a separate VM (didn't think Nouveau does that either), just a separate virtual address space**, but we do use the GPUVM page tables to control what the GPU can access. GPUVM is also implemented for Cayman but don't remember if it's enabled by default yet.The ideal way is isolating users in a separate VM by restricting a GPU user to its own data through abstracting the memory address space. This method is already used by the Nouveau driver for NVIDIA GeFore 8 hardware and newer while it's possible to be supported by the AMD Radeon HD 7000 series and newer along with Intel Sandy Bridge graphics and newer.
** Yay for confusing acronyms... VM can be Virtual Memory or Virtual Machine, but "separate VM" implies Virtual Machine...
AMD GPUs have hardware support to minimize context switching delays in Cayman, GCN and beyond... basically the ability to have multiple page tables, each associated with a "VM ID" or VMID, then have the hardware automatically switch between them as needed. The HW supports a finite number of VMIDs so they do need to be managed carefully but starting with Cayman there are enough VMIDs to mimimize the overhead.The problem with this separate VM approach though is that it does increase the context-switching delay, which could particularly cause problems when using DRI2 and Qt5 and other cases. Right now what the Radeon and Intel open-source drivers is command submission validation by making sure they aren't accessing bad areas of RAM. This method yields a lower context-switching delay and doesn't have any specific hardwae requirements, but does come at a cost of higher CPU usage with needing to check the CS packets for their validity.
As mentioned, it is possible to use GPUVM in earlier hardware but without VMID support the context switching delays make it impractical so we use continue to use command submission validation for older hardware.