Xen - VGA passthrough is the way to go!!!

[powerhouse]

Ok, so I've been giving this a shot from a different angle I think without a whole lot of success.

I installed XCP 1.6 to my home server:
xeon 3.3 ghz (with VT-d)
32gb ram
NVidia quadro 4000
2 tb hard disk and some other disks

I manage the server with XenCenter on a basic desktop. This part works beautifully. I can create vm's for days.

I've experimented with many combinations of os installs and drivers and have gotten mild gpu passthrough success. I installed Win 7 Pro SP1 to a vm and then installed the xs-tools that come with xcp. This gave me a pretty speedy and stable Win7 vm. I enabled gpu passthrough in xencenter and restart my vm and there it is in my device list. I install drivers and restart and everything seems configured properly, my display adapter says NVidia Quadro 4000 without any warnings. But I can't do anything with it. I don't see any improvements since enabling the gpu and I can't get any 3d applications to launch.

In case you are wondering: I decided to try this method because I'm pretty new to xen. I've been reading forums and the wiki for the last week or so trying to get a handle on it. I went this way because it seemed pretty simple, but obviously its not working the way I want it to. Any help would be appreciated.

Note: I tried to run 3dmark11 and this is the error I get -

Workload Single init returned error message: DXGI call IDXGIOutput::GetDisplayModeList failed:

The requested functionality is not supported by the device or the driver.

DXGI_ERROR_NOT_CURRENTLY_AVAILABLE

I'm not familiar with XCP, except that it's the cloud platform version. According to the Xen comparison matrix, XCP lags behind Xen and doesn't give you the latest features.

I use Xen 4.1.2 with Linux Mint and it works perfect with a Nvidia Quadro 2000.

Here is a How-to I wrote: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013

Hope it helps. If not, I'd be glad to help out if I can.

[dannysemi]

Thanks for the reply powerhouse. I've actually learned a lot since my post and what I mentioned in that post wasn't entirely true. I did succeed in achieving hardware acceleration unfortunately I don't think it works as I intended for it. I don't know if you are familiar with Microsoft Hyper-V at all, but I get a similar graphics acceleration to RemoteFX with XCP. I was wondering what your experience with Xen is like. Does the gpu acceleration have limitations, like does it work in fullscreen? Does it fully support your graphics card as if it were in the machine?

I looked and XCP 1.6 (the version I'm using) uses Xen hypervisor 4.1.3. I think it behaves very differently than what you are experiencing because you are using Xen from the same machine, where as I have XCP installed on a server and I am accessing it from a remote desktop on the LAN with a Gbit connection. I think the differences in behavior can be attributed to the functionality of XAPI but I'm not sure. As far as partial success, I am able to play Borderlands 2 off of Steam in fullscreen windowed mode at nearly max settings. The network lag does make the controls just slightly too slow to really be functional. Most other games give me errors, I think because the driver support isn't there yet.

I don't know if there is a better way to set this up, or if I can get Xen to function in the same way over the network. The weak link is probably a combination of XAPI and RDP but I'm not an expert.

As of right now, I'm rating my experience with XCP right behind Hyper-V because with Hyper-V I was at least able to get more games working. XCP is fantastic for whatever basic computing I might want to do, but the purpose of this experiment was to see if I could play games and I would say that, for the moment, has failed.

FWIW I also tried my Nvidia GeForce 660ti in the server and got identical results.

I really hope to see this technology advance in the future and give it another shot.

[powerhouse]

Thanks for the reply powerhouse. I've actually learned a lot since my post and what I mentioned in that post wasn't entirely true. I did succeed in achieving hardware acceleration unfortunately I don't think it works as I intended for it. I don't know if you are familiar with Microsoft Hyper-V at all, but I get a similar graphics acceleration to RemoteFX with XCP. I was wondering what your experience with Xen is like. Does the gpu acceleration have limitations, like does it work in fullscreen? Does it fully support your graphics card as if it were in the machine?

I looked and XCP 1.6 (the version I'm using) uses Xen hypervisor 4.1.3. I think it behaves very differently than what you are experiencing because you are using Xen from the same machine, where as I have XCP installed on a server and I am accessing it from a remote desktop on the LAN with a Gbit connection. I think the differences in behavior can be attributed to the functionality of XAPI but I'm not sure. As far as partial success, I am able to play Borderlands 2 off of Steam in fullscreen windowed mode at nearly max settings. The network lag does make the controls just slightly too slow to really be functional. Most other games give me errors, I think because the driver support isn't there yet.

I don't know if there is a better way to set this up, or if I can get Xen to function in the same way over the network. The weak link is probably a combination of XAPI and RDP but I'm not an expert.

As of right now, I'm rating my experience with XCP right behind Hyper-V because with Hyper-V I was at least able to get more games working. XCP is fantastic for whatever basic computing I might want to do, but the purpose of this experiment was to see if I could play games and I would say that, for the moment, has failed.

FWIW I also tried my Nvidia GeForce 660ti in the server and got identical results.

I really hope to see this technology advance in the future and give it another shot.

You are correct, we are trying to achieve different things. Here is what I'm doing:

a. Xen hypervisor runs on PC with 2 physical GPUs (graphics cards): AMD 6450 and Nvidia Quadro 2000, both connected to my NEC 26" screen (I switch between them with the "INPUT" button on my screen);
b. Linux Mint runs as dom0 (the admin VM) using the AMD 6450 graphics card;
c. Windows 7 64bit Pro runs as domU (VM) using the Nvidia Quadro 2000 card;
d. Keyboard/mouse are connected via KVM-USB switch to two different host adapters at the PC, one of which is passed through to the Windows 7 domU. There are other ways to get that working, like "synergy", but I found this to be the best way for me.

In the above setup, both Linux and Windows have direct access to the graphics card hardware. In both Linux and Windows I installed the latest (proprietary) graphics drivers for the respective card. Under Windows, I'm using screen calibration software and a color sensor to adjust my screen colors for accurate photo editing. The software writes the color correction curve directly to the NEC screen via the DVI cable. This can only work when the calibration software (running on Windows 7) has direct access to the graphics card - there is no way this would work with emulated cards.
In short, I am running two VMs on my PC with which I interact locally (i.e. not via remote desktop). I did this to avoid dual-booting Linux and Windows.

Of course I can access both the Linux dom0 and the Windows 7 domU via remote desktop, but to the best of my knowledge that doesn't give me any graphics acceleration support by any of the graphics cards.

If I understand you correctly, you are running XCP (or now Hyper-V) on a remote server, with VGA passthrough for your Windows 7 VM (under XCP)? I'm not sure if and how a remote desktop connection can make use of the graphics card acceleration provided by your Quadro 4000, but I would be very interested to hear from you about that.

To get hardware acceleration for graphics, the graphics card and software would need to render and forward the picture to a remote client, which would require the graphics card to be able to process DirectX or OpenGL instructions and return the output to the software to then be forwarded to a remote desktop (in very simple terms). Most normal graphics cards are made to output to the screen via their HDMI/DVI/VGA/etc. interfaces. I believe the Nvidia Quadro series is actually capable of computing pictures/frames, but it certainly requires some software to handle that.

The next problem is transporting the picture information to the remote client. There are a number of protocols available, from simple VNC via NX to PCoIP and SPICE, etc. (see http://spice-space.org/). Depending on your remote client (thick client, thin client, zero client) and the protocol used, results can vary greatly. Teradici has developed PCoIP chips that are used in a variety of thin and zero clients, as well as server boards. VMware is also pushing PCoIP. Other solutions often require a thick client (such as a regular PC, notebook or at least a netbook). So, if you ran XCP and tried VNC, you may want to use a different remote desktop protocol and see what that does. Try and compare NX, ssh with X window, and RDP. SPICE looks very interesting too, but I don't know how easy or difficult it would be to use.

The only company I can think of that has been doing it right is onlive by Steve Perlman. They've custom-built their server hardware to deliver a cloud-based gaming service. In the meantime a few companies or startups have picked up that lead and are developing hardware to better support remote desktop (VDI) applications.

Though Microsoft is a latecomer to the game, I can imagine that Hyper-V outperforms other solutions since they are the one holding the source code to Windows, which allows them better and easier integration as well as the development of PV drivers (para-virtualized). Xen has been doing that for ages, but only for Linux (Windows is closed-source, so there is little chance for outsiders to write PV drivers, and I doubt Microsoft will be cooperative).

Theory aside, here are some practical suggestions you may want to consider:
1. You can run 2 or even more "work stations" on one server, as long as there isn't much distance between them, and you don't mind running extra cables. With "work stations" I mean terminals, or a combination of screen and keyboard/mouse. In my setup, I could have as easily connected two sets of screen, keyboard and mouse to create two independent work places. (In fact, that would have been even easier.)
2. If one or more of your work places don't require high-performance video, then perhaps this http://www.mct.com.tw/index.php?_Pag...cid=21&_lang=E might be a solution (instead of installing another GPU). I have not tried it, and I have no experience with that vendor, it's just an idea!
3. With a graphics card and screen that has HDMI ports and audio support, you could run up to 25ft/8m of HDMI cable and up to 16ft/5m of USB 2.0 at "High-Speed", or more using HDMI and USB extenders (see http://www.usb.org/developers/usbfaq/#cab4 for more on USB).
4. All of the above do not use any client device (PC, etc.), just the screens and KB/mice directly plugged into your Xen server. Server is maybe the wrong word, because you actually use it as a workstation.
If you need only one high performance graphics workstation, you can do as follows:
- Use your server as the high-performance (gaming/whatever) workstation and pass-through the appropriate video card to your Windows VM (I assume you run Windows for gaming).
- Use remote desktop (RDP, X-screen, NX, SPICE, whatever) to connect to other VMs on your server, using what netbook/notebook/PC you got. A good option would be using a thin client, but you need to make absolutely sure the client supports the protocol you're going to use (or vice versa, you can setup the protocol that the thin client uses). I'm not sure if Xen supports PCoIP, as I believe it's proprietary.
- If you have a remote Linux machine, you can test some options with Remmina and it's plugins. It supports VNC, RDP, NX, SSH (with X window), and SFTP for file transfers. Of course you can connect to any VM on your server, including Windows. I don't know if SPICE (see above) is ready for use, but it might be worth considering.
- I sometimes use Remmina with RDP on my Linux dom0 to connect to my Windows 7 VM (though I can also just switch my KB/mouse/screen to it) and it works nicely. Actually, I'm hard pressed to see a difference between remote desktop and native display when doing standard stuff (editing documents, photos, etc.), EXCEPT when I run youtube or similar, but I attribute this to a Xen networking issue. I believe games too won't work in a remote screen (I could give Unigine Heaven a try?).

Bottom line: It's possible to connect a screen, mouse + keyboard over a relatively short distance (up to ~16ft/5m) to a "remote" server. The advantage is you don't need a remote client PC at all. With some effort and hardware you can extend that distance.
You can also connect to your VMs on the server using remote desktop / VNC / etc. but you won't get graphics hardware acceleration (at least not so easily), and the performance (lag) will also depend on the protocol and configuration you use and the network bandwidth and utilization. Xen networking can be a little tricky to setup, though, and has been a cause for throughput problems. Still, it should work just fine for normal office/editing/browsing tasks.

Last not least, I very much hope to be able to turn my Xen workstation into a server and replace all thick PCs/notebooks with zero clients (or thin clients) connected to the server.

[powerhouse]

@dannysemi: I forgot to mention that RemoteFX is the proprietary Microsoft "equivalent" to PCoIP and perhaps NX and SPICE. Your Quadro 4000 graphics card supports RemoteFX - see http://en.wikipedia.org/wiki/RemoteF...U_Requirements.

I believe the same graphics card could play nicely in a Xen/XCP based environment (the Quadro series is used by many professional studios and engineering companies who often run Linux), but I haven't checked it further. I wonder how SPICE would be able to utilize it?

About Linux and paravirtualization on Hyper-V: It's possible according to this http://en.wikipedia.org/wiki/Hyper-V#Linux_support, but perhaps for the wrong reason (see http://www.zdnet.com/blog/microsoft/...x-drivers/3433).

[qcjulle]

VGA passthrough with onboard AMD video?

Hi, I thought this might be a good place to ask. I have ordered an AMD A10-5700 APU with an SVM/IOMMU capable motherboard. The setup is going to go into a quite small passively cooled case so no room for a dedicated video card. Can I pass through the AMD IGP to a Win8 VM for light gaming and HD video processing with HDMI audio (XBMC, Netflix @1080p) or does it only work with Intel IGPs?

I have seen successes reported with dedicated AMD GPUs and integrated Intel GPUs but not with integrated AMD GPUs.

My goal is to get Linux Mint/Ubuntu and Windows 8 running without running the Linux inside Win8 (potential for too many disruptive reboots). I don't really need graphics for the hypervisor and the Linux side if I can use SSH/VNC to control them but I would like to have full GPU capabilities in Windows.

[powerhouse]

Hi, I thought this might be a good place to ask. I have ordered an AMD A10-5700 APU with an SVM/IOMMU capable motherboard. The setup is going to go into a quite small passively cooled case so no room for a dedicated video card. Can I pass through the AMD IGP to a Win8 VM for light gaming and HD video processing with HDMI audio (XBMC, Netflix @1080p) or does it only work with Intel IGPs?

I have seen successes reported with dedicated AMD GPUs and integrated Intel GPUs but not with integrated AMD GPUs.

My goal is to get Linux Mint/Ubuntu and Windows 8 running without running the Linux inside Win8 (potential for too many disruptive reboots). I don't really need graphics for the hypervisor and the Linux side if I can use SSH/VNC to control them but I would like to have full GPU capabilities in Windows.

If the CPU and motherboard support IOMMU or whatever it's called in AMD language, it should be possible. The graphics card or integrated GPU can be a tricky part, and I have no experience with the AMD ones.

If you have only one GPU you will need to pass it thru right at system boot. This will be more difficult with Linux Mint/Ubuntu than with Fedora, for the simple reason that Fedora 16/17 has pciback integrated into the kernel, whereas in LM/Ubuntu you load it as a module (the xen_pciback module).

I've written a how-to for LM 13 which should also work for Ubuntu - see here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013.

You install LM/Ubuntu and then the Xen hypervisor from packages (Ubuntu is easier to install on LVM, and I recommend LVM strongly). Then you need to install either a VNC server or ssh server so you can access your PC from another one via network (you need a second PC/laptop to be able to do the installation and configuration). Follow the instructions I've given in my how-to. Here is what's different:

In your case you need to pass-through and boot your Windows VM when you boot into Linux. So once you installed all the packages in LM/Ubuntu, blacklist the installed graphics driver. Use:

lspci -v | grep -A 11 -i vga

and check the Kernel driver in use, then try this http://community.linuxmint.com/tutorial/view/923 and run update-grub. An alternative method for blacklisting the graphics driver is this:

sudo echo blacklist nouveau > /etc/modprobe.d/blacklist-nouveau.conf

Replace the name of the driver with the one you found (most likely "radeon"). For example, /etc/modprobe.d/blacklist-radeon.conf with the following content:

blacklist radeon

Try the first method, it's better to disable the GPU driver as early as possible else you won't be able to detach the GPU.

When done, and before rebooting, make the network and other config file adjustments, create your win8.cfg file, and prepare an ISO file of your Windows 8 DVD that will be mounted via your win8.cfg file when you boot the guest. Also I recommend installing virt-manager and edit the /etc/xen/xend-config.sxp file to include the following:

(xend-unix-server yes)

Make sure you have ssh root access from your second PC. Now reboot. After the Xen hypervisor loads, and just when Linux Mint/Ubuntu take over, you should get a blank screen. Now connect via ssh -X ... or VNC and run:

sudo xm pci-list-assignable-devices #to see if the GPU is assignable - if it doesn't appear, something went wrong

sudo xm create /etc/xen/win8.cfg

Then you can run sudo virt-manager, select the win8 that is running and press the console button to get a VNC console to the Windows VM. Install Windows and I believe after the first reboot of the VM you should be able to select and install the video driver (if Windows hasn't done that already). Check under devices - the graphics card should be listed, perhaps in addition to the "virtual" VGA card Xen presents to Windows. By this time, or after another Windows reboot, Windows should show on your screen.

Once you succeeded to boot into Windows and have a working display, you can add a startup script to start win8 when you boot the PC.

Wish you good luck!

[Sythrar]

Look : http://www.youtube.com/watch?v=QZbFmDS_seM
SPGT Technology in QEMU-KVM Virtual Machine

someone could test?

http://sourceforge.net/projects/spgt/

[powerhouse]

Look : http://www.youtube.com/watch?v=QZbFmDS_seM
SPGT Technology in QEMU-KVM Virtual Machine

someone could test?

http://sourceforge.net/projects/spgt/

This looks interesting. I've downloaded SPGT but haven't tried it yet - not enough time right now.

[jockinator]

Anyone tried Linux Guest

one thing that I'm still not able to do, is to have also a Linux Guest, to test the latest steam linux game with my radeon HD7950.

each time I install catalyst in a Xen DomU with Ubuntu, it's just not working at all.
X start, I see the wallpaper, but unity/compiz are stuck, I can't click on anything.
I use some tricks to run fglrxinfo, and it segfault ...

looking at this thread :

http://www.gossamer-threads.com/lists/xen/users/282154

I also tested Ubuntu 12.04 as guest but still the same issue.

anyone tried PCI passthrough of a GPU with a linux guest ?

is there a way to use the secondary GPU on the Dom0 (maybe starting a separate X server ?)
maybe I will have to wait for wayland...

[powerhouse]

one thing that I'm still not able to do, is to have also a Linux Guest, to test the latest steam linux game with my radeon HD7950.

each time I install catalyst in a Xen DomU with Ubuntu, it's just not working at all.
X start, I see the wallpaper, but unity/compiz are stuck, I can't click on anything.
I use some tricks to run fglrxinfo, and it segfault ...

looking at this thread :

http://www.gossamer-threads.com/lists/xen/users/282154

I also tested Ubuntu 12.04 as guest but still the same issue.

anyone tried PCI passthrough of a GPU with a linux guest ?

is there a way to use the secondary GPU on the Dom0 (maybe starting a separate X server ?)
maybe I will have to wait for wayland...

Here is another one reporting the same issue: http://ubuntuforums.org/showthread.p...1#post12647351. One of the Ubuntu releases worked for him.

EDIT: Just noticed this is the same poster.

I'm using an AMD card with proprietary fglrx driver under dom0 (my first/primary VGA card) and my second VGA card passed through to a Windows 7 HVM guest. AFAIK, passthrough can only work with HVM guests, this is also true for Linux. I will have to try my second graphics card with a Linux guest.