Either you code a VM-aware version of the driver to work around this, and run an insecure VM that trusts the guest not to use the vid card to read memory outside the VM, or you need a hardware IOMMU. Some current hardware has IOMMUs, but I can't remember which. Except for server stuff, probably only AMD and maybe Nehalem, since it's a lot easier to implement when the PCIe <-> RAM path goes through the CPU anyway.
I've been wanting to do this for years, to run a multi-seat box (multiple users on one machine, each with their own kbd+mouse+vga). I was disappointed a couple years ago when I got a core2 to find out that even with HW virt support, giving the guest direct VGA access wasn't possible.
What current and future x86-64 hardware has an IOMMU suitable for guest video drivers? Anyone more up to date on this than I am?