The Paravirtualization Spectrum, Part 2 - From poles to a spectrum

7/28/2019 The Paravirtualization Spectrum, Part 2 - From poles to a spectrum

1/8

31/10/12 The Paravirtualization Spectrum, Part 2: From poles to a spectrum blog.xen.org

1/8blog.xen.org/index.php/2012/10/31/the-paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/

Skip to content

Xen.org Community Blog

AboutLog inRegister

Categories:

AnnouncementsCloud XenCommunity

EventsLinuxPartner Announcements

Project KronosUncategorizedXCPXedn Hypervisor

Xen BooksXen Case StudyXen DayXen Development

Xen HistoryXen HypervisorXen MigrationXen Security

Xen Static CheckingXen SummitXen Support

Xen WebXen-APIXen.org Promotion

The Paravirtualization Spectrum, Part 2: From

poles to a spectrum

In part 1 of this series, I introduced the concepts offull virtualization and paravirtualization (PV), aswell as the hardware virtualization (HVM) feature used by Xen (among other things) to implementfull virtualization. I also introduced the concept of installing paravirtualized drivers on a fully virtualized

system.

This small step, from full virtualization towards paravirtualization, begins to hint at the idea of aspectrum of paravirtualization. In this article, I will cover the historical reasons for the development of

PVHVM, and finally of the newest mode, PVH.


2/8



Problems with paravirtualization: AMD and x86-64

It comes as a surprise to many people that while 32-bit paravirtualized guests in Xen are faster than 32-bit fully virtualized guests, when running in 64-bit mode, paravirtualized guests can sometimes beslower than fully virtualized guests. This is due to some changes AMD made when designing thearchitecture which simplified things for them, but made things more difficult for Xen.

Most modern operating systems need just two levels of protection: user mode and kernel mode. Kernelmode memory is protected from user mode memory via the pagetable supervisor mode bit.

When running a virtual machine, you need at least three levels of protection: user mode, guest kernel,

and hypervisor. The hypervisor memory needs to be protected from the guest kernel, and the guestkernel memory needs to be protected from the user. The pagetable protections only provide two levelsof protection, so Xen uses another processor feature, called a segmentation limit, to provide the third

level of protection. Segmentation limits are processor feature that was in common use before paging wasavailable. But since paging has been available, segmentation limits have basically not been used; so Xenwas able to commandeer them to provide the extra level of necessary protection. The pagetableprotections protect both the guest kernel and Xen from userspace; the segmentation limits protect Xen

from the guest kernel.

Unfortunately, at the time that Xen team was developing clever new uses for this little-used feature,AMD was designing their 64-bit extensions to the x86 architecture. Any unused processor feature

makes hardware much more complicated to design, reason about, and verify. Since basically nooperating systems use segmentation limits, AMD decided to get rid of them.

This may have greatly simplified the architecture for AMD, but it made it impossible for Xen to squeezein 3 levels of protection into the same address space. Instead, for 64-bit PV guests, both guest kernel and

guest user-space need to run in ring 3, each with their own address space. Every time a guest process

needs to make a system call, it has to bounce up into Xen, which will context-switch to the guest kernel.This not only takes more time for each system call, but requires flushing one of the key CPU caches,

called a TLB. Frequent flushing of the TLB causes the all of the execution to run more slowly for sometime afterwards, as the TLB is filled up again.

In 64-bit HVM mode, the problem doesnt occur. The HVM extensions make it easy to have three

different protection levels without needing to play clever tricks with little-used processor features. Somaking system calls in 64-bit HVM mode is just as fast as on real hardware. For this reason, a lot ofpeople began running 64-bit Linux in fully virtualized mode.

Paravirtualizing little by little: PVHVM modeBut fully virtualized mode, even with PV drivers, has a number of things that are unnecessarilyinefficient. One example is the interrupt controllers: fully virtualized mode provides the guest kernel

with emulated interrupt controllers (APICs and IOAPICs). Each instruction that interacts with the APICrequires a trip up into Xen and a software instruction decode; and each interrupt delivered requiresseveral of these emulations.

As it turns out, many of the the paravirtualized interfaces for interrupts, timers, and so on are actuallyavailable for guests running in HVM mode; they just need to be turned on and used. The paravirtualized

interfaces use memory pages shared with Xen, and are streamlined to minimize traps into the hypervisor.

So Stefano Stabellini wrote some patches for the Linux kernel that allowed Linux, when it detects thatits running in HVM mode under Xen, to switch from using the emulated interrupt controllers and timersto the paravirtualized interrupts and timers. This mode he called PVHVM mode, because although it


3/8



runs in HVM mode, it uses the PV interfaces extensively.

(PVHVM mode should not be confused with PV-on-HVM mode, which is a term sometimes usedin the past for fully virtualized with PV drivers.)

With the introduction of PVHVM mode, we can start to see paravirtualization not as binary on or off,but as a spectrum. In PVHVM mode, the disk and network are paravirtualized, as are interrupts and

timers. But the guest still boots with an emulated motherboard, PCI bus, and so on. It also goes through

a legacy boot, starting with a BIOS and then booting into 16-bit mode. Privileged instructions arevirtualized using the HVM extensions, and pagetables are fully virtualized, using either shadow

pagetables, or the hardware assisted paging (HAP) available on more recent AMD and Intel processors.

Problems with paravirtualization: Linux and the PV MMU

PVHVM mode allows 64-bit guests to run at near native speed, taking advantage of both the hardwarevirtualization extensions and the paravirtualized interfaces of Xen. But it still leaves something to bedesired. For one, it still requires the overhead of an emulated BIOS and legacy boot. Secondly, itrequires the extra memory overhead of a qemu instance to emulate the motherboard and PCI devices.

For this reason, memory-conscious or security-conscious users may opt to use 64-bit PV anyway, evenif it is somewhat slower.

But there is one PV guest that can never be run in PVHVM mode, and that is domain 0. Because

having a domain 0 with the current Linux drivers will always be necessary, it will always be necessaryto have a PV mode in the Linux kernel.

But whats the problem, you ask? Werent all of the features necessary to run Linux as a dom0

upstreamed in Linux 3.0?

Yes, they were; but they are still occasionally the source of some irritation. The core changes required toparavirtualize the page tables (also known as the PV MMU) are straightforward and work well once

the system is up and running. However, while the kernel is booting, before the normal MMU is up andrunning, the story is a bit different. The changes required for the early MMU are fragile, and are ofteninadvertently broken when making seemingly innocent changes. This makes both the x86 maintainersand the pvops maintainers unhappy, consuming time and emotional energy that could be used for other

purposes.

Almost fully PV: PVH mode

A lot of the choices Xen made when designing a PV interface were made before HVM extensions wereavailable. Nearly all hardware now has HVM extensions available, and nearly all also includehardware-assisted pagetable virtualization. What if we could run a fully PV guest one that had no

emulated motherboard, BIOS, or anything like that but used the HVM extensions to make the PVMMU unnecessary, as well as to speed up system calls in 64-bit mode?

This is exactly what Mukeshs PVH mode is. Its a fully PV kernel mode, running with paravirtualized

disk and network, paravirtualized interrupts and timers, no emulated devices of any kind (and thus noqemu), no BIOS or legacy boot but instead of requiring PV MMU, it uses the HVM hardwareextensions to virtualize the pagetables, as well as system calls and other privileged operations.

We fully expect PVH to have the best characteristics of all the modes a simple, fast, secure interface,

low memory overhead, while taking full advantage of the hardware. If HVM had been available at thetime the Xen hypervisor was designed, PVH is probably the mode we would have chosen to use. Infact, in the new ARM Xen port, it is the primary mode that guests will operate in.


4/8



Once PVH is well-established (perhaps five years or so after its introduced), we will probably considerremoving non-PVH support from the Linux kernel, making maintenance of Xen support for Linux

much simpler. The Xen kernel will probably support older kernels for some time after that. However,rest assured that none of this will be done without consideration of the community.

Given the number of other things in the fully virtualized paravirtualized spectrum, finding a descriptive

name has been difficult. The developers have more or less settled on PVH (mainly PV, but with alittle bit of HVM), but it has in the past been called other things, including PV in an HVM container

(or just HVM containers), and Hybrid mode.

What about KVM?

At this point, some people may be wondering, how would KVMs virtualization fit into this spectrum?

Strictly speaking, KVM is just a set of kernel extensions designed to help processes implementvirtualization. When most people speak of using KVM, they mean qemu-kvm, which means qemurunning configured to use the KVM extensions. (There are other projects, such as the Native LinuxKVM tool, which also use the KVM extensions.) When I say KVM here, I mean qemu-kvm.

KVM supports both legacy boot, starting in 16-bit mode with a BIOS (or EFI) to load the kernelbootloader, and booting directly into a kernel passed on the qemu command-line. It also provides anemulated motherboard, PCI bus, and so on. It can provide both emulated disk and network cards; and

thus it is capable of supporting guests running in fully virtualized mode.

KVM also provides virtio devices, which can be considered paravirtualized, as well as a PV clock, foroperating systems that can be modified to support them. KVMs typical method of paravirtualization is

somewhat different than Xens. Virtio devices expose a normal device interface, with MMIO controlpaths and so on, and could in theory be implemented by real hardware. Xens PV interfaces are based

on shared memory and lockless synchronization. The kinds of actions that need an MMIO contextswitch for virtio devices probably correspond pretty closely to actions that need hypercalls for Xen PV

devices; but in Xen no instruction emulation needs to be done.

KVM does not have a paravirtualized interface for timers or interrupts; instead (if I understand correctly)it uses an emulated local APIC. Handling a full interrupt cycle for an emulated local APIC typically

requires several MMIO accesses, each of which requires a context switch and an instruction emulation.The Xen PV interrupt interface is based on memory shared with the hypervisor, supplemented byhypercalls when necessary; so most operations can be done without context switches, and those that do

require only a single context switch (and no instruction emulation). This was one of the major reasonsfor introducing PVHVM mode for Xen guests.

So KVM has paravirtualized devices and a paravirtualized clock, but not paravirtualized interrupts;placing KVM on the spectrum, it would be one step more paravirtualized than FV with PV drivers,

but not as paravirtualized as PVHVM.

The paravirtualization spectrum

So to summarize: There are a number of things that can be either virtualized or paravirtualized whencreating a VM; these include:

Disk and network devices

Interrupts and timersEmulated platform: motherboard, device buses, BIOS, legacy bootPrivileged instructions and pagetables (memory access)


5/8



Each of these can be fully virtualized or paravirtualized independently. This leads to a spectrum ofvirtualization modes, summarized in the table below:

The first three of these will all be classified as HVM mode, and the last two as PV mode forhistorical reasons. PVH is the new mode, which we expect to be a sweet spot between full virtualization

and paravirtualization: it combines the best advantages of Xens PV mode with full utilization ofhardware support.

Hopefully this has given you an insight into what the various modes are, how they came about, and

what are the advantages and disadvantages of each.

Share and Enjoy:

Posted in Uncategorized.

No comments

By dunlapg October 31, 2012

0 Responses

Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.

You must be logged in to post a comment.

Fedora 18 Virtualization Test Day

Subscribe


6/8



About blog.xen.org

Welcome to the Xen.org open source community blog site. All members of the Xen.org community arefree to post blogs on any topic related to the Xen hypervisor solution. If you want to run a story, pleaseget in touch with the XEN community manager Lars Kurth. Share and Enjoy:more

Search for: Search

Archives

October 2012September 2012

August 2012July 2012June 2012

May 2012April 2012March 2012February 2012

January 2012December 2011November 2011October 2011

September 2011August 2011

July 2011June 2011

May 2011April 2011March 2011

February 2011January 2011December 2010November 2010

October 2010

September 2010August 2010July 2010

June 2010May 2010April 2010

March 2010February 2010January 2010December 2009

November 2009

October 2009September 2009August 2009

July 2009June 2009


7/8



May 2009April 2009

March 2009February 2009January 2009December 2008

November 2008October 2008

September 2008August 2008July 2008June 2008May 2008

April 2008March 2008February 2008

Meta

Register

Log inEntries RSSComments RSSWordPress.org

Blogroll

Ascii for BreakfastGrant McWilliams XCP HowTo'sGrant McWilliams Xen HowTosOracle Virtualization

Xen SupportXen Virtualization on Linux and SolarisXenServer Blog

Xen Tags

CitrixClarkson UniversityCommunityFujitsuGoogle Summer of CodeIan PrattLinuxNewsletterOpen SourceOracle

RegistrationRelease CandidateRequest for CommentSurveyUSENIXWikiXCPxenXen 3.3Xen HypervisorXen

SummitXen Summit TokyoXen WikiXen-APIXen.org

#XEN Twitter Feed

Intro to the Paravirtualization Spectrum With Xen (Part 2) ... https://t.co/tvm91yuF #xen

Intro to the Paravirtualization Spectrum With Xen (Part 2) ... https://t.co/tvm91yuF #xen []

#xen #fedora Re: Xen install test: On Fedora 17, install Fedora 17. Mouse problem?http://t.co/XER19GZ7

#xen #fedora Re: Xen install test: On Fedora 17, install Fedora 17. Mouse problem?http://t.co/XER19GZ7 []#xen #fedora Xen install test: On Fedora 17, install Fedora 17. Mouse problem?


8/8



http://t.co/M409YPq6

#xen #fedora Xen install test: On Fedora 17, install Fedora 17. Mouse problem?

http://t.co/M409YPq6 []XEN: Como que eu saio de uma mquina virtual #linux #infra #xen #cloud http://t.co/uxtvLioFXEN: Como que eu saio de uma mquina virtual #linux #infra #xen #cloud http://t.co/uxtvLioF[]

#xen #fedora Fwd: [fedora-virt] Fedora 18 Virt Test Day is Thurs Nov 1, we need you!http://t.co/43155VAG

#xen #fedora Fwd: [fedora-virt] Fedora 18 Virt Test Day is Thurs Nov 1, we need you!http://t.co/43155VAG []

Once again amazed how "normal" virt is comparted to #xen pv Virtualbox on sys full of SSDsand things just don't come to an end.Once again amazed how "normal" virt is comparted to #xen pv Virtualbox on sys full of SSDs

and things just don't come to an end. []@megazone would you rather virtualize a server environment in #vmware or #xen@megazone would you rather virtualize a server environment in #vmware or #xen []#xen #fedora [Fwd: Re: [fedora-virt] xen 4.2 and libvirt] http://t.co/GhWgoLJG

#xen #fedora [Fwd: Re: [fedora-virt] xen 4.2 and libvirt] http://t.co/GhWgoLJG []

#xen #fedora Call for Participation to the Fedora Virt Test Day http://t.co/RqbujPM3#xen #fedora Call for Participation to the Fedora Virt Test Day http://t.co/RqbujPM3 []Virtualizao com XEN - Aprenda na prtica #linux #infra #xen #cloud http://t.co/FXbZkIVf

Virtualizao com XEN - Aprenda na prtica #linux #infra #xen #cloud http://t.co/FXbZkIVf[]

Proudly powered by WordPress running on a Slicehost Xen VPS.

Carrington Theme by Crowd Favorite

The Paravirtualization Spectrum, Part 2 - From poles to a spectrum

Documents

Transcript of The Paravirtualization Spectrum, Part 2 - From poles to a spectrum