Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH technical deep-dive

George Dunlap

Edinburg – 21-23 October, 2013

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Intro: PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 2 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 3 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 3 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 3 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 4 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PV

I PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)

I Qemu processI Legacy bootI Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PVI PVMMU in Linux

I 64-bit hypercalls

I Issues with HVM (PVHVM)

I Qemu processI Legacy bootI Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)

I Qemu processI Legacy bootI Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)

I Qemu processI Legacy bootI Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)I Qemu process

I Legacy bootI Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)I Qemu processI Legacy boot

I Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PV and HVM

I Issues with PVI PVMMU in LinuxI 64-bit hypercalls

I Issues with HVM (PVHVM)I Qemu processI Legacy bootI Devices emulated in Xen

Edinburg – 21-23 October, 2013 PVH technical deep-dive 5 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH

I “PV in an HVM container”

I Goal: take the best aspects of PV and HVM

I Written by Mukesh Rathor @ Oracle

I Significant revisions by George Dunlap @ Citrix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH

I “PV in an HVM container”

I Goal: take the best aspects of PV and HVM

I Written by Mukesh Rathor @ Oracle

I Significant revisions by George Dunlap @ Citrix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH

I “PV in an HVM container”

I Goal: take the best aspects of PV and HVM

I Written by Mukesh Rathor @ Oracle

I Significant revisions by George Dunlap @ Citrix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH

I “PV in an HVM container”

I Goal: take the best aspects of PV and HVM

I Written by Mukesh Rathor @ Oracle

I Significant revisions by George Dunlap @ Citrix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 6 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Xen’s perspective

I Begin with HVM guest

I Disable HVM-specific things not needed

I Start & pin in 64-bit paged mode

I Enable PV path for a handful of things

Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Xen’s perspective

I Begin with HVM guest

I Disable HVM-specific things not needed

I Start & pin in 64-bit paged mode

I Enable PV path for a handful of things

Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Xen’s perspective

I Begin with HVM guest

I Disable HVM-specific things not needed

I Start & pin in 64-bit paged mode

I Enable PV path for a handful of things

Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Xen’s perspective

I Begin with HVM guest

I Disable HVM-specific things not needed

I Start & pin in 64-bit paged mode

I Enable PV path for a handful of things

Edinburg – 21-23 October, 2013 PVH technical deep-dive 7 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Things disabled

I Device model (qemu)

I Emulated hardware (APIC, PIT, etc)

I Nested HVM

I MSIX

Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Things disabled

I Device model (qemu)

I Emulated hardware (APIC, PIT, etc)

I Nested HVM

I MSIX

Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Things disabled

I Device model (qemu)

I Emulated hardware (APIC, PIT, etc)

I Nested HVM

I MSIX

Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Things disabled

I Device model (qemu)

I Emulated hardware (APIC, PIT, etc)

I Nested HVM

I MSIX

Edinburg – 21-23 October, 2013 PVH technical deep-dive 8 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Pinning in 64-bit

I Set initial values ofr cr0, cr4, EFER

I Certain paging things have to happen at start of day

I Disable writes to EFER

I Don’t allow guest to change paging-related cr0 bits

Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Pinning in 64-bit

I Set initial values ofr cr0, cr4, EFER

I Certain paging things have to happen at start of day

I Disable writes to EFER

I Don’t allow guest to change paging-related cr0 bits

Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Pinning in 64-bit

I Set initial values ofr cr0, cr4, EFER

I Certain paging things have to happen at start of day

I Disable writes to EFER

I Don’t allow guest to change paging-related cr0 bits

Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: Pinning in 64-bit

I Set initial values ofr cr0, cr4, EFER

I Certain paging things have to happen at start of day

I Disable writes to EFER

I Don’t allow guest to change paging-related cr0 bits

Edinburg – 21-23 October, 2013 PVH technical deep-dive 9 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PV paths

I Enable PV hypercalls

I PV e820 map

I PVH VCPU boot

I PV CPUID

I PV PIO

Edinburg – 21-23 October, 2013 PVH technical deep-dive 10 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up

Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up

Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up

Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up

Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH from Linux’s perspective

I xen hvm domain() false, xen pv domain() true

I Act natural (get rid of PV special cases)

I Autotranslate side effects

I Use PVHVM callback vector setup

I VCPU bring-up

Edinburg – 21-23 October, 2013 PVH technical deep-dive 11 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

I No PV IDT

I No PV irq ops

I No PV CPUID

I Native Syscall/sysenter

I No VM assists

I No event / failsafe callbacks

I No need to set IOPL

Edinburg – 21-23 October, 2013 PVH technical deep-dive 12 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

MMUOPS:

I No need to pin pagetables

I No pfn/mfn conversion

I No need to special-case PT protections

I Only pvmmu op: flush tlb others

Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

MMUOPS:

I No need to pin pagetables

I No pfn/mfn conversion

I No need to special-case PT protections

I Only pvmmu op: flush tlb others

Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

MMUOPS:

I No need to pin pagetables

I No pfn/mfn conversion

I No need to special-case PT protections

I Only pvmmu op: flush tlb others

Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Things disabled

MMUOPS:

I No need to pin pagetables

I No pfn/mfn conversion

I No need to special-case PT protections

I Only pvmmu op: flush tlb others

Edinburg – 21-23 October, 2013 PVH technical deep-dive 13 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PV

I Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVH

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by Xen

I Real MFN in pagetablesI “Map this page here” hypercall

I PVH

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by XenI Real MFN in pagetables

I “Map this page here” hypercall

I PVH

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVH

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVH

I Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVHI Pagetables controlled by guest

I gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVHI Pagetables controlled by guestI gpfn in pagetables

I Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I PVI Pagetables controlled by XenI Real MFN in pagetablesI “Map this page here” hypercall

I PVHI Pagetables controlled by guestI gpfn in pagetablesI Every mapped page must be in the p2m

Edinburg – 21-23 October, 2013 PVH technical deep-dive 14 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I “Special pages”: grant frame, make a hole in the p2m

I Need to add p2m entries to map foreign pages

I Grant table: similar to HVM (but not the same)

Edinburg – 21-23 October, 2013 PVH technical deep-dive 15 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I “Special pages”: grant frame, make a hole in the p2m

I Need to add p2m entries to map foreign pages

I Grant table: similar to HVM (but not the same)

Edinburg – 21-23 October, 2013 PVH technical deep-dive 15 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: Autotranslate

I “Special pages”: grant frame, make a hole in the p2m

I Need to add p2m entries to map foreign pages

I Grant table: similar to HVM (but not the same)

Edinburg – 21-23 October, 2013 PVH technical deep-dive 15 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: VCPU bring-up

I Brought up via hypercalls (PV)

I Guest in control of IDT, Xen can’t guarantee loading properly

I Only set GS by default

I All other state must be set in on-cpu boot code

Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: VCPU bring-up

I Brought up via hypercalls (PV)

I Guest in control of IDT, Xen can’t guarantee loading properly

I Only set GS by default

I All other state must be set in on-cpu boot code

Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: VCPU bring-up

I Brought up via hypercalls (PV)

I Guest in control of IDT, Xen can’t guarantee loading properly

I Only set GS by default

I All other state must be set in on-cpu boot code

Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Linux: VCPU bring-up

I Brought up via hypercalls (PV)

I Guest in control of IDT, Xen can’t guarantee loading properly

I Only set GS by default

I All other state must be set in on-cpu boot code

Edinburg – 21-23 October, 2013 PVH technical deep-dive 16 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH: Not yet working

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH: Not yet working

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH: Not yet working

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

PVH: Not yet working

I 32-bit

I vtsc

I shadow mode

I vcpu hotplug

Edinburg – 21-23 October, 2013 PVH technical deep-dive 17 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: HVM paths

I Original idea: “Lightweight” container

I Reality: Large amount of code necessary for minimumfunctionality

I Very large amount of code duplication

I Current patch: use HVM paths, with special cases for PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: HVM paths

I Original idea: “Lightweight” container

I Reality: Large amount of code necessary for minimumfunctionality

I Very large amount of code duplication

I Current patch: use HVM paths, with special cases for PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: HVM paths

I Original idea: “Lightweight” container

I Reality: Large amount of code necessary for minimumfunctionality

I Very large amount of code duplication

I Current patch: use HVM paths, with special cases for PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: HVM paths

I Original idea: “Lightweight” container

I Reality: Large amount of code necessary for minimumfunctionality

I Very large amount of code duplication

I Current patch: use HVM paths, with special cases for PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 18 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config space

I “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Direct access permitted

I pv PIT

I cmos idx

I pci config spaceI “Abusing” IO with SMM

I Execute exact instruction with guest GPRs

Edinburg – 21-23 October, 2013 PVH technical deep-dive 19 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I Problems

I Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities

I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guests

I Race condition because of double checking

I Possibilities

I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities

I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I Possibilities

I Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I PossibilitiesI Need 2 sets of access controls: For user procs, and for guest

I PV only has oneI PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has one

I PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Issues: PIO instructions

I ProblemsI Ugly changes to allow PV paths to emulate for PVH guestsI Race condition because of double checking

I PossibilitiesI Need 2 sets of access controls: For user procs, and for guestI PV only has oneI PVH: Have two, so may not need to emulate anymore

Edinburg – 21-23 October, 2013 PVH technical deep-dive 20 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table update

I HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness

I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory write

I Shadow: Trip through Xen

I TLB effectiveness

I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness

I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectiveness

I HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entries

I Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement cost

I Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement costI Shadow: Worst case, 4 memory reads

I HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?

I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?I Kernel build: Shadow 30% slower than HAP

I SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

HAP vs Shadow

I Page table updateI HAP: Memory writeI Shadow: Trip through Xen

I TLB effectivenessI HAP: Superpages allow 2MiB / 1GiB TLB entriesI Shadow: Still only 4k TLB entries

I TLB replacement costI Shadow: Worst case, 4 memory readsI HAP: 9-16 memory reads

I Does this matter?I Kernel build: Shadow 30% slower than HAPI SpecJBB: Shadow 30% faster than HAP

Edinburg – 21-23 October, 2013 PVH technical deep-dive 21 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Overview

I PV, HVM, and PVH

I PVH from Xen’s perspective

I PVH from Linux’s perspective

I Issues in PVH

Edinburg – 21-23 October, 2013 PVH technical deep-dive 22 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 23 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 23 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Goal

I Give you a technical overview of PVH, so that...

I you can understand the characteristics, advantages, anddisadvantages...

I and approach the code to improve and fix

Edinburg – 21-23 October, 2013 PVH technical deep-dive 23 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Questions

Questions?

Feedback? george.dunlap@citrix.com

Check out our blog: http://blog.xen.org/

Edinburg – 21-23 October, 2013 PVH technical deep-dive 24 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25

Intro PV and HVM PVH and Xen PVH and Linux PVH Issues Spare slides

Xen: PVH hypercalls

I memory op

I console io

I vcpu op

I mmuext op

Edinburg – 21-23 October, 2013 PVH technical deep-dive 25 / 25