Xen in Ubuntu Raring

PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux

Agenda

Brief overview of Xen

What's new in 4.2

What's coming in 4.3

What is a great Ubuntu Xen experience?

Integration issuesQemu

Libvirt

Other improvements

Dom 0:In a typical Xen set-up Dom0 contains a smorgasboard of functionality:System boot

Device emulation & multiplexing

Administrative toolstack

Drivers (e.g. Storage & Network)

Etc.

LARGE TCB BUT, Smaller as in a Type 2 hypervisor

Driver/Stub/Service Domains: also known as Disaggregation

Xen Overview

PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux

Basic Xen Concepts

Xen HypervisorControl domain
(dom0)Host HWVMnVM1VM0Guest OSand Apps

XL, XM (deprecated)

MemoryCPUsI/O

Scheduler, MMU

One or more
driver, stub or
service domainsControl Domain aka Dom0Dom0 kernel with drivers

Xen Management Toolstack

Trusted Computing Base

Guest DomainsYour apps

E.g. your cloud management stack

Driver/Stub/Service Domain(s)A driver, device model or control service in a box

De-privileged and isolated

Lifetime: start, stop, kill

Dom0 KernelDom 0:In a typical Xen set-up Dom0 contains a smorgasboard of functionality:System boot

Device emulation & multiplexing

Administrative toolstack

Drivers (e.g. Storage & Network)

Etc.

LARGE TCB BUT, Smaller as in a Type 2 hypervisor

Driver/Stub/Service Domains: also known as Disaggregation

Xen HypervisorControl domain
(dom0)Host HWGuest VMnAppsMemoryCPUsI/O

Linux PV guests have limitations:limited set of virtual hardware

AdvantagesFast

Works on any system
(even without virt extensions)

Driver DomainsSecurity

Isolation

Reliability and Robustness

HW DriversPV Back EndsPV Front Ends

Driver Domain
e.g. Disk

Network

HW DriverPV Back EndDom0 Kernel**) Can be MiniOS

PV Domains & Driver DomainsGuest OS

HVM

Xen HypervisorDom0Host HWGuest VMnDisadvantagesSlower than PV due to Emulation
(mainly I/O devices)

AdvantagesInstall the same way as native Linux

Stub DomainsSecurity

Isolation

Reliability and Robustness

Device ModelHVM & Stub Domains

IO Emulation IO EventVMEXITStubdomnDevice ModelMini OSGuest VMn

IO Emulation IO EventVMEXITDevice Model emulated in QEMUModels for newer devices are much faster, but for now PV is even faster

Xen 4.2 and 4.3

PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux

Xen 4.2 Release

xl default toostacklibxl stable interface

xend deprecated

Scalability4095 host CPUs, 5TiB RAM

cpupools for more flexible partitioning

Scheduler, NUMA improvements

Security improvements

http://wiki.xen.org/wiki/Xen_4.2_Feature_List

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

Who writes Xen?

Xen 4.3 Release

Slated for 17 June 2013

ARM server port

PVH mode: PV with some HVM extensions

NUMA

Numerous I/O path improvements

Qemu upsteam

Roadmap: http://wiki.xen.org/wiki/Xen_Roadmap/4.3

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

Xen and Ubuntu

PVOPS is the Kernel Infrastructure to run a PV Hypervisor on top of Linux

Integration: qemu

Debian developmentsQemu-dm: Xen qemu fork

4.2: qemu-upstream missing pass-through, migration

4.3: qemu-upstream will be default

Debian has already removed qemu-dm

SolutionsQemu-xen-dm package

Backport features

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

Integration: libvirt

Libvirt 0.10.2 has bindings for xend, 4.1 libxl

4.2 libxl bindings incompatible, xend deprecated

OptionsStick with 4.1 (bad)

Xen 4.2, but only xend bindings

Back-port 4.2 libxl support

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

What does a great Xen on Ubuntu look like?

As a Xen hostEasy to install, set up

Reliable, good performance

Switching between Xen and non-Xen modes

Good integration with libvirt, &c

As a Xen guestInstallation

Reliable, good performance

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

Potential improvements

Xen Host option in installer

Make configuring Grub2 easier

Switching between Xen / non-Xen

Getty for PV console (hvc0)

Xen-tools

Guest installation?

Keeping an eye on linux-xen perf tweaks to backport

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

Backup slides

Architecture Considerations

Type 1: Bare metal HypervisorA pure Hypervisor that runs directly on the hardware and hosts Guest OSs.

Type 2: OS HostedA Hypervisor that runs within a Host OS and hosts Guest OSs inside of it, using the host OS services to provide the virtual environment.Provides partition isolation + reliability,
higher securityLow cost, no additional drivers
Ease of use & installationHost HWMemoryCPUsI/O

Host HWMemoryCPUsI/O

HypervisorSchedulerMMU

Device Drivers/ModelsVMnVM1VM0Guest OSand AppsHost OS

Device Drivers

Ring-0 VM Monitor
Kernel VMnVM1VM0Guest OSand Apps

User
AppsUser-level VMM

Device Models

At this point I want to make a quick detour into the different hypervisor architectures from a viewpoint of security.

Lets look at type 1 hypervisor:Basically a very simple architecture, where the Hypervisor replaces the kernel

The architecture is significantly simpler that a Type 2 hypervisor, because it does not need to provide rich process semantics, like user, filesystems, etc.

BUT: the trade-off is that all the device drivers need to be rewritten for each hardware platform

Type 2 is hosted- The hypervisor is just a driver that typically works with user-level monitor.HW access is intercepted by the ring 0- VM monitor passed to the User level Virtual Monitor, which passes requests to the kernel

Re-use of device drivers is traded off against security and a large trusted computing base (green)

Xen: Type 1 with a Twist

Control domain
(dom0)Host HWVMnVM1VM0Guest OSand AppsMemoryCPUsI/O

Thin hypervisor

Functionality moved to Dom0

Using Linux PVOPSTake full advantage of PV

PV on HVM

No additional device drivers (Linux 3.x dom0)

In other wordslow cost (drivers)

Ease of use & Installation

Isolation & Security

HypervisorSchedulerMMU

Drivers

Device Models

Linux, BSD, etc.

XSM

Dom 0:In a typical Xen set-up Dom0 contains a smorgasboard of functionality:System boot

Device emulation & multiplexing

Administrative toolstack

Drivers (e.g. Storage & Network)

Etc.

LARGE TCB BUT, Smaller as in a Type 2 hypervisor

10 Freescale i.MX53 Loco Quickstart boards

Running Debian "armhf" with a mainline 3.2.0 kernel

Speed up development of Xen for Cortex A15
(avoid cross compilation)

A bit of fun: our ARM Build Farm

Xen 4.2 Release

Security: Intel Supervisor Mode Execution Protection, XSM / Flask improvements

Scalability: increased VM density for VDI use-cases, up to 256 Host CPUs for 64 bit HV , Multiple PCI segment support, prefer oxenstored

Performance: PCI pass-through for Linux Guests, AMD SVM DecodeAssist support, Remus memory image compression

EFI support

Libvchan cross domain comms in Xen mainline

XL improvements, XEND is formally deprecated

Documentation improvements (e.g. man pages)

Detailed ListGeneralDocumentation improvements (e.g. man pages)Lots of bug fixing of course.[edit]Toolsxl is now default toolstack and xend is formally deprecatedlots of xl improvements.we should highlight xend deprecation (not effectively maintained since 2008)Remus compression (compression of memory image improves performance)Prefer oxenstored when available (improves scalability!)Support for upstream qemu; nearing feature parity (non default still, but we want people to be testing it)Added libvchan to xen mainline(cross domain comms)[edit]XenImprovements to paging and sharing, enabling higher VM density for VDI use-casesEFI (extensible Firmware Interface) support for HV (i.e. if I have a machine that has EFI, I can use Xen on it)Support up to 256 Host CPUs for 64 bit h/v (from 128)Support dom0 kernels compressed with xzPer-device interrupt remapping (increases scalability)Support for pvhvm guest direct pirq injection (Performance improvement for PCI passthrough for Linux Guests)Intel SMEP (Supervisor Mode Execution Protection) supportMem event stuff? (Allows to externally observe what guests are up to and can be used for external virus checking - not sure what the right terminology is)Multiple PCI segment supportAdded xsave support(floating point)Lots of XSM / Flask fixes (security)AMD SVM "DecodeAssist" support (AMD CPU feature that avoids emulation and increases performance)[edit]Removed FunctionalityACM (alternative XSM to Flask) was removed (unmaintained)Removed vnet (unmaintained)[edit]Xen Development SupportCan build with clangAdded "make deb" targetLots of xentrace improvementsupdate ocaml bindings and make them usable by xapi (which previously had it's own fork of the same codebase)

Click to edit the title text format

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the title text formatClick to edit Master title style

Click to edit the title text formatClick to edit Master title style

Click to edit the title text formatClick to edit Master title style

Click to edit the title text formatClick to edit Master title style

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the title text formatClick to edit Master title style

Click to edit the title text formatClick to edit Master title style

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline Level

Seventh Outline LevelClick to edit Master text styles

Second level

Third level

Fourth level

Fifth level