VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson /...
Transcript of VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson /...
![Page 2: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/2.jpg)
The current state ofVGA assignment
![Page 3: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/3.jpg)
VGA assignment defined:Graphics card assigned as primary graphics for the VMUses VGA BIOS for initialization and runtime services
![Page 4: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/4.jpg)
Status:Works for discrete graphics cards
AMD Radeon (HD5xxx+)NVIDIA GeForce (8-series+)
Requires experimental vfio-pci optionx-vga=on
Requires working VGA arbitrationStill some device specific glitches
BSODs with AMD Catalyst + HDMI audioReset issues on some Radeon cardsCode 43 on GeForce
![Page 5: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/5.jpg)
Issues:Requires experimental vfio-pci option, x-vga=on
Not supported by libvirt because experimentalExperimental because...
Requires working VGA arbitrationDrivers don't participate (vgacon)Drivers lie (i915)Xorg DRI can't handle multiple arbitration participants
![Page 6: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/6.jpg)
Can't we fix these?Drivers don't participate
Drivers lie
Xorg DRI can't handle multiple arbitration participants
fixable
Once upon a time...i915 hardware allowed control of VGA resources
nobody noticed?!i915 continues to opt-out anyway
But if we fix that...
designers dropped this feature
![Page 7: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/7.jpg)
Start with fixing XorgDRI wants to mmap VGA MMIO spacePossible options...
Provide a new VGA arbiter interface allowing mmapsRemove the mmap requirement
But what about compatibility?
“WE DO NOT BREAK USERSPACE!”-Linus Torvalds
![Page 8: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/8.jpg)
VGA assignment status:
Stuck
![Page 9: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/9.jpg)
What else could we do?
![Page 10: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/10.jpg)
Do we really need VGA?
![Page 11: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/11.jpg)
Secondary graphicsEmulated VGA for VM primary displayAssigned graphics card for secondarySupported by NVIDIA
Requires K-series Quadro/GRID/TeslaSupported in RHEL7.0
Some reports of success with Radeon
Depends on guest driver support
![Page 12: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/12.jpg)
Legacy-free UEFI VMNo VGA required
No VGA arbitration required!
![Page 13: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/13.jpg)
Problem solved?VGA Issues:
Requires "experimental" vfio-pci optionNot supported by libvirt because experimentalExperimental because...
Requires working VGA arbitrationDrivers don't participate (vgacon)Drivers lie (i915)Xorg DRI cannot handle multiple arbitrationparticipants
![Page 14: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/14.jpg)
No Blockers
![Page 15: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/15.jpg)
How do UEFI?
![Page 16: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/16.jpg)
Step 1: OVMFOpen Virtual Machine Firmware
An implementation of UEFI built on the EDK II code baseReplaces Seabios
![Page 17: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/17.jpg)
OVMF Setupvia command line:
via libvirt:
/usr/libexec/qemu-kvm ... \ -drive if=pflash,format=raw,readonly,file=/path/to/OVMF_CODE.fd \ -drive if=pflash,format=raw,file=/copy/of/OVMF_VARS.fd
<domain type='kvm'> ... <os> <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/> ... </os> </domain>
![Page 18: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/18.jpg)
Step 2: EFI graphics card ROMPCI ROMs can support muliple imagesMost newer cards already support EFI
![Page 19: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/19.jpg)
Step 3: UEFI compatible guestNewer versions of LinuxWindows 8, Server 2012
![Page 20: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/20.jpg)
That's it!
![Page 21: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/21.jpg)
Other GPU Topics
![Page 22: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/22.jpg)
440FX vs Q35Q35 originally recommended for VFIO-VGA
Largely FUD
Q35 looks more like the host platformSome Linux drivers assume an upstream PCIe port
In reality
Windows doesn't careLinux is fixable
440FX is likely the easier path for Windows guests
![Page 23: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/23.jpg)
NVIDIA: Code 43Driver detects KVM hypervisor, fails to initialize*
Nvidia - "Accidental" breakage, won't fix, unsupportedWe can't solve it, but we can work around it
Hide the hypervisorvia command line: via libvirt:
*NVIDIA driver version 338.77+
-cpu [type],kvm=off
<domain type='kvm'> ... <features> <kvm> <hidden state='on'/> </kvm> ... </features> </domain>
![Page 24: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/24.jpg)
NVIDIA: Code 43 ReduxDriver detects Hyper-V extensions, fails to initialize*
Workaround: Disable use of Hyper-V extensionsPerformance hit for Windows VMs
Have we entered an arms race?
*NVIDIA driver version 344.11
![Page 25: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/25.jpg)
QuirksGeForce: Radeon:
One still required*
Not needed!
*QEMU now enables this independent of x-vga (2.2)
![Page 26: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/26.jpg)
Unresolved IssuesRadeon device resets
Sea Island GPUs don't reset correctlySMC firmware issue?
Southern Island GPUs indicate PM reset capabilitydoesn't work
Radeon BSODTypically resolved by avoiding audio function
Intel Graphics supportIn progress? KvmGT?
Host/Guest suspendAnd ponies!
![Page 27: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/27.jpg)
ResourcesVFIO tips and tricks blog:
This presentation:http://vfio.blogspot.com
http://awilliam.github.io/presentations/KVM-Forum-2014
![Page 28: VFIO, OVMF, GPU, and You · The state of GPU assignment in QEMU/KVM Alex Williamson / alex.williamson@redhat.com. The current state of VGA assignment. VGA assignment defined: Graphics](https://reader034.fdocuments.in/reader034/viewer/2022051605/6011bef15f496d65b636156a/html5/thumbnails/28.jpg)
Thank you