Announcement

Collapse
No announcement yet.

Upgrades for vfio gaming PC

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Upgrades for vfio gaming PC

    I've made a few attempts at vfio/iommu pci-e pass-through setup over the years. I failed 3 times using various versions of Ubuntu, but finally had success using Archlinux in 2016 and also on Fedora 25. In both cases I used it only for a few weeks. I experienced various issues which brought me to the conclusion that it was too much hassle and I just ended up going back to dual boot.

    My current gaming/dev PC:
    cpu: Intel Core i7 3770 Ivy Bridge (Non-K with vt-d)
    m/b: Gigabyte GA-Z77X-UD3H (official BIOS fails in many aspects*)
    ram: 32GB F3-1866C9D-16GSR Sniper 1.5V (x4 8GB)
    gfx: 8GB Sapphire Nitro+ RX 480
    sound: Asus Xonar U7 7.1 Usb Soundcard And Amplifier
    ssd: x1 256GB Crucial M550 (Windows 10 Pro)
    ssd: x1 500GB Samsung 960 EVO NVME M.2 (Fedora 27)
    hdd: x2 4TB WD RED
    hdd: x1 8TB WD RED
    *I made my own BIOS based on a Gigabyte Z97 BIOS in order to fix various BIOS bugs and to obtain NVME boot functionality.

    I am planning to upgrade the following parts. I'm looking for advice on the motherboard, hoping for good iommu groups...

    cpu: AMD Ryzen R7 1800X
    m/b: ASRock FATAL1TY X370 Professional Gaming
    ram: G.SKill F4-3200C14D-32GTZ TridentZ Series 32GB (2x16GB) 3200MHz DDR4 CL14 1.35V
    extra gfx; Sapphire 11267-00 Radeon RX 560 Pulse 4GB GDDR5 128-bit PCI-E 3.0

    I would like stop dual boot all together. Hopefully with this upgrade to will motivate me to work on the issues I've experienced to address it properly or to even attempt to fix/workaround it. Lag/small spikes in games was the biggest issue, but I also experienced game and OS cashes form time to time. I play games competitively so just a single crash will normally take me back to dual booting for a few months. Are there any people out there that do competitive gaming on pci-e pass-through setups? Do you have any advice for me?

    PS:
    My steamID: 0_x
    Has anyone tried gaming on QubesOS?

  • #2
    (Reserved for updates)

    Update 9 Feb 2018 - Gaming Phase

    Great Success


    Let start with the results. I'm going to compare the following 3 "systems". Systems A and B are the same physically, but due to the VM config the guest OS refers to it as "AMD Opteron 63xx class CPU" instead of "AMD Ryzen 7 1800X". System C is my old gaming PC which has now been replaced with System A. The systems are running at stock speed, non have been overclocked.

    A. AMD Ryzen 7 1800X (Windows 10 VM optimized, 4.14 mainline kernel. Edited QEMU config)
    B. AMD Ryzen 7 1800X (Windows 10 VM, stock Ubuntu 17.10 kernel. Config by virt-manager GUI)
    C. Intel Core i7-3770 Processor (Windows 10 Native)

    Code:
    3D Mark Fire Strike:
    Points: 11 446 System A
    Points: 10 849 System C
    Points:  9 853 System B
    
    3D Mark API Overhead Test:
    DirectX 11 Multi-threaded draw calls per second
    1 471 454 System A
    1 171 727 System C
      201 170 System B
    
    DirectX 11 Single-threaded draw calls per second
    1 316 305 System A
    1 262 224 System C
      202 168 System B
    
    DirectX 12 draw calls per second
    18 853 976 System A
    17 008 706 System C
     3 837 357 System B
    It's interesting to see that Fire Strike (the actual featured benchmark) does not show a big difference between System A and B (basically NPT on/off respectively), however the API Overhead Test shows a massive difference in both DirectX 11 and 12 tests. I do not have Time Spy test at the moment, perhaps it also tests similar aspects. This is the cause of the stuttering I was trying to describe in my 9 Feb 2018 update below.

    References:
    Fire Strike: https://www.3dmark.com/compare/fs/14...19/fs/14895427
    API Overhead Test: https://www.3dmark.com/compare/aot/2...361/aot/248289

    I am happy with my current setup! I still think there is room for improvement. I feel like an idiot for assuming Ryzen NPT fixes would be back-ported/not checking what version the patch was accepted into mainline kernel !! I finally have one desktop that can both game and use for software development and Linux is running at the core, no Windows auto update firmware crap. I also managed to workaround my sound issues, I am now passing one of my USB controllers to the VM instead of passing USB devices individually. I'm guessing there a latency or address issue with the Windows sound driver which is related specifically to the Asus Xonar U7 7.1 Usb Soundcard, however my knowledge of what goes on in the hardware/driver level is nonexistent. I might buy an extra PCI-E USB card just for isolated mapping sake, I'm sure BIOS updates can improve mapping too, but I have no idea if/when that will happen.

    From here onward, once Linux 4.16 (or better Fedora 28) is released I would like to try https://github.com/gnif/LookingGlass maybe some stability tests in the meanwhile. Ansible script to setup my Host OS could save me time/frustration in the future. I want to take a break from updating pci bus ids in various files and rebuilding grub/initramfs manually. I have not tried overclocking, I'll probably leave that up to some of my friends that I will be assisting with the same setup as this.

    A Sad Story,
    I found out that I have the compilation segfault issue with All my Ryzen CPUs (I/my colleague bought: x3 1800X ~end of Nov 2017). I was unable to RMA any of those since AMD does not ship to/from South Africa. I know about forward address like Stackry, but based on passed experiences I can't take chances with customs, the risk ridiculous tax is just too damn high. I'll have to find a workaround to the problem in software/config, CPU pinning perhaps? If all else fails I'll have to disable cores until I save up for a replacement or zen+ upgrade. I'm quite baffled that I received chips that were produced in week 05 when I purchased it at the end of 2017, anyway it is what it is.

    Update 9 Feb 2018 - Testing Phase Success/Failure

    Fail

    I made a few attempts at getting Qubes OS working, but I could not install in UEFI mode. Qubes currently uses Fedora 25 for their install. I downloaded F25 and tried UEFI boot, it worked. It was not a upstream issue so I reported it and left it at that. I spoke to the guys on IRC, I was surprised to hear that they are actually looking into gpu passthrough even as one of thier Google Code of Summer projects albeit targeting intel igpu first.

    Success

    I had success using vfio/iommu pci-e pass-through using plain old Ubuntu. I preferred using Fedora in the past, but I'm using geforce 1030 on my host, proper nouveau support is only landing in ~4.16 and nomodeset's llvmpipe messes up the refresh rate of my screen, depending on the background color I can actually see the screen flickering which hurts my eyes. Nvidia's proprietary driver did not work on Fedora 27's latest kernel. I used the following documentation https://forum.level1techs.com/t/ubun...-14-rc1/119639 there is a lot more people trying it out today and level1techs / reddit has a lot of information and support. I found it easier to setup pci-e passthough than getting geforce 1030 working on my host OS.

    This is the hardware I ended up with:
    cpu: AMD Ryzen R7 1800X
    m/b: ASRock FATAL1TY X370 Professional Gaming
    ram: G.SKill F4-3200C14D-32GTZ TridentZ Series 32GB (2x16GB) 3200MHz DDR4 CL14 1.35V
    gfx: 8GB Sapphire Nitro+ RX 480
    gfx host: 2GB Zotac GeForce GT 1030 ZT-P10300A-10L
    ssd: x1 500GB Samsung 960 EVO NVME M.2
    hdd: 8TB WD RED

    I configured my VM using virt-manager, had to play around with CPU setting as Ryzen is not supported yet with qemu/kvm. I installed windows 10 just had to mount the virtio drivers iso during install. I'm really glad Redhat signed those, it made the process very easy. I installed steam and played a few games.

    Play for a hour or so, input lag was worse than native and stuttering was very bad at times. I can still play competitive games like pubg so I'm happy for now. It's a lot better than anything else (waiting for native pupg support or waiting for wine). There are still optimizations that I need to do. I'm going to try out Windows 7 and although most of my iommu groups are really good on the m/b. I don't have an isolated USB controller and for some reason my Asus Xonar U7 7.1 Usb Soundcard does not work when I pass it to Windows 10 using USB passthrough. I know a lot of people use USB headsets that they can pass though without issues. For now I'm stuck with pulseaudio which I really want to avoid. Probably just going to end up putting in a pci-e usb card and pass that through.

    I'm still looking forward to test Virgl support for Windows, just because I'm curious not expecting that much of it. Looking Glass also seems like a project with good potential.

    6 Dec 2017 - Research Phase

    AMD/Ryzen NPT Fix https://www.phoronix.com/scan.php?pa...U-Pass-Through
    Headless PCIe Passthrough (25 Nov 2017): https://www.youtube.com/watch?v=1MI1s4hZ_yE

    Other (future) solutions:
    PreAlpha Virgl Windows Driver: https://gist.github.com/Keenuts/1991...62cf0011147c0b
    Last edited by Jabberwocky; 02-12-2018, 11:36 AM.

    Comment


    • #3
      There are many native Linux games that are played at the professional level. Focus to them, with win virus hoover you waste your time to virus protection, installing removing software one at the time and problem solving takes many times longer, because of of poor logging system and binary only code. I hope you get good money for your used Rx480, otherwise there is no much sense to buy a slower card. RX400 and RX500 series cards use the same Linux driver, see pci ids from amdgpu_drv.c. I do get that KVM and other low level virtualisation solutions need two gpus. However, virtualisation in the Linux kernel is made for the networked servers, not for gaming. Mostly Linux is running in those virtual machines.

      Use wine-staging csmt enabled for the rest of the games. See winehq appdb what windows games works and what not.
      Last edited by debianxfce; 12-03-2017, 07:38 AM.

      Comment


      • #4
        Thanks for the reply debianxfce .

        TL;DR My problem is with dual-booting, I consider wine as a step backwards from dual-booting.

        I'm kind of happy with gaming on Windows, sure I would prefer it if all my games worked on Linux just as well as it did on Windows. I actively used/modified wine between 2004 and 2011. It took a lot of time to maintain all the different versions with the patches for each game that I used to play. I actually spent more time on wine research/patching/configuration than the amount of reboots and formatting Windows XP needed. I still considered it worth it because I'm very stubborn and everything worked (sometimes faster than on Windows). These days popular competitive games does not work on wine anymore, so it's not really an option in my books even if I had the time.

        Logically it makes sense (to me) to focus on pass-through/para-virtualization rather than translating Windows system calls into POSIX-compliant system calls, given that developer time is a finite resource and Window's system calls are still growing . I still believe Windows' place is in a virtual machine (which is why I posted under software category). For gaming it's not easy at the moment which is why I am trying to find effective solutions in terms of performance and reliability. I'm not so much phased about hardware cost. I'm hopeful that if vfio/iommu pci-e pass-through gaming becomes more popular there will be optimizations in hardware and software to reduce cost.

        PS: My Rx 480's silicon is amazing! I'm actually curious if the Nitro+ OC cards are binned. Anyways in Windows it's kicking the 1060's ass, even passing some 1070s on games like rise of the tomb raider, hitman and battlefield 1. Not going to talk about power draw LOL.

        Comment

        Working...
        X