1

HARDWARE SUPPORT FOR

EFFICIENT VIRTUALIZATIONLennox Wu

2

Outline•Classifications

•Processor virtualization• Two main Software-based solutions• Challenges to virtualize Intel x86(software-only)• Hardware-based Virtualization

• Intel VT-x : x86• Intel VT-I :Itanium (X)• Intel EPT/AMD NPT • AMD-V• Sun SPARC (X)• ARM Virtualization Extensions(X)• IBM Power(X)

• I/O virtualization• Intel VT-d• AMD IOMMU(AMD-V)• Intel VT-c

•Dose these techniques work?

3

Classifications•VMM(virtual machine monitor) = hypervisor

•By techniques•Full Virtualization•Paravirtualization•Hardware Assisted Virtualization

•Robert P. Goldberg(Harvard University,1973)•Type 1/native/bare metal hypervisors

• Hypervisors run directly on the host's hardware to control the hardware and to manage guest operating systems.

• Oracle VM Server for SPARC, the Citrix XenServer, KVM, VMware ESX/ESXi, and Microsoft Hyper-V hypervisor.

•Type 2/hosted hypervisors• Hypervisors run within a conventional operating system environment.• VMware Workstation/player, Microsoft Virtual PC and VirtualBox

4

TYPE1/TYPE2 hypervisors

5

PROCESSOR VIRTUALIZATION

6

Two main Software-based solutions(1)•Full virtualization using binary translation

•Transforming guest OS binaries on-the-fly• Guest applications don’t use privilege instructions

•Pros• Support unmodified OSs (The only way of pure-software

solutions)• Offer best isolation and security• Simplify migration and portability of guest OS

•Cons: low performance•Examples: VMware, MS Virtual PC, Virtual box

• Disable HW virtualization

7

Full virtualization using binary translation

8

Two main Software-based solutions(2)• OS assisted virtualization or paravirtualization

•OS assisted virtualization guest OSs help the VMM• paravirtualization refers to communication between the guest OS and

the VMM to improve performance and efficiency•Modify the guest OS to cooperate with the VMM

• Modify the OS kernel to replace non-virtualizable instructions with hypercalls(the functions provided by the VMM)

•Pros: Offer higher performance•Cons: Need the source code of an OS•Example: Xen, KVM(*), VMware(*) • (*) Vmware tool uses some paravirtualization techniques

• optimize virtual device drivers• time synchronization • logging and guest shutdown. • Vmxnet is a paravirtualized I/O device driver

9

OS assisted virtualization or paravirtualization

10

KVMKVM

11

Challenges to virtualize Intel x86(software-only)(1/3)•Ring Aliasing

•Guest-OSes run at the Ring3• Original: OS:Ring 0, APP:Ring 3 (Ring0>ring3)

•A guest OS can know its run level

•Address-Space Compression •VMM must use some of the guest’s virtual-address space to manage transition between guest OS and VMM

•VMM’s address spaces must be protected• Guest could detect that it is running in a VM

12

Challenges to virtualize Intel x86(software-only)(2/3)•Non-Faulting Access to Privileged State

•Some instructions should be intercepted by VMM do not involve faults

•Adverse Impact on Guest System Calls•VMM must emulate every system calls

•Interrupt Virtualization•A VMM may manage external interrupts and deny guest to control interrupt masking

•Some OS frequently mask and unmask• VMM must process these requests.

13

Challenges to virtualize Intel x86(software-only) (3/3)•Ring Compression

•Guest OS runs at the same privilege level as applications• The guest OS can’t protect guest applications

•Frequent Access to Privileged Resources•VMM should deny the accesses

•Address translation•Guest OS doesn’t know the physical address, so the VMM must intercepted guest page table updates

14

Intel VT-x overview(1/4)

•VT=virtualization technology

• Two new form of CPU operation•VMX root operation : for VMM•VMX non-root operation: for guest-software•Both forms of operation support all four privilege levels(Ring0~Ring3)• Guest OS can run at its intended privilege level

15

Two new form of CPU operation

16

Intel VT-x overview(2/4)

•Two new transitions•VM entry

•VMX root operation (VMM) non-root operation(VM)•VM exit

•VMX non-root operation (VM) root operation (VMM)

•Under VMX non-root operation, many instructions/events cause VM exits•configurable

17

Intel VT-x overview(3/4)

•VMCS (Virtual Machine Control Structure)•A new data structure includes guest-state area and host-state area

•VM entry: load the guest-state area and save the host-state area

•VM exit : load the host-state area and save the guest-state area

•The exiting conditions controlled by the VM-execution fields•Switch the structure will switch the address space

18

Intel VT-x overview(4/4)

•VMCS supports interrupt virtualization•Determine the conditions of VM to cause VM exit

• All interrupt• Whenever guest OS is ready to receive interrupts• Which exception?• Which port access attempts?• Which Model Specific Register access attempts?

19

Intel EPT / AMD NPT(1)

•EPT (Extended Page Tables)•“EPT provides performance gains of up to 48% for MMU-intensive benchmarks and up to 600% for MMU-intensive microbenchmarks.” –VMware

• AMD’s nested page table (NPT) is similar to EPT

•A.k.a Rapid Virtualization Indexing (RVI)

•“RVI provides performance gains of up to 42% for MMU-intensive benchmarks and up to 500% for MMU-intensive microbenchmarks.” --VMware

20

Intel EPT / AMD NPT(2)

•Software MMU (software-only)•Hardware uses the shadow page table•VMM must maintain the shadow page table

21

Intel EPT / AMD NPT(3)•Hardware MMU•Guest-OS maintains guest page tables•VMM maintains PPN->MPN mappings in an additional level of page tables

•The hardware will find the LPN->MPN with the two pages

22

AMD-V(1/2)

•Tagged TLB • Add the ASID• Hardware features that facilitate efficient switching between

virtual machines for better application responsiveness

• Two new form of CPU operation• Host mode : for VMM (similar to Intel’s VMX root operation) • Guest mode : for guest software (similar to Intel’s VMX non-root

operation)• new instructions

• vmrun : host mode guest mode• exit : guest mode host mode• vmcall: it lets the operating system and VMM communicate directly

• A new structure• Virtual Machine Control Block (VMCB)

• Similar to Intel’s VMCS

23

AMD-V(2/2)

•Nested page table (NPT)/ Rapid Virtualization Indexing (RVI)

•VMM migration•Use the CPUID to identify the ability of the processor where the VMM runs, and the VMM use the supported functions.

24

Hardware-base solution with VT-x(1/2)•Address-Space Compression

•VM Exits / VM Entries change the linear address space

•Ring Aliasing & Ring Compression•VT-x allows guest OS to run at its intended privilege level

•Nonfaulting Access to Privileged State•Either causes transition to VMM•Or becomes unimportant to VMM

25

Hardware-base solution with VT-x(2/2)•Guest System Calls

•a guest OS can run at privilege level 0

•Frequent Access to Privileged Resources•VT-x provides TPR shadow. VMM is only involved when the value drops below the threshold VMM only processes the situation it cares.

26

Hardware Assisted Virtualization of x86

27

I/O VIRTUALIZATION

28

Current I/O virtualization techniques•Emulation

• The VMM supports virtual devices that guest OS can recognize• The virtual device models are responsible to translate commands

and data. • Pros. No requirement to modify guest-OSs• Cons. Low performance

•Paravirtualization• Modify the guest software (driver)to enhance the performance• Pros. better performance• Cons. Limited applicability. (modify need the source code)

•Direct assignment• Bind a specify device to a VM• VMM allow the owning VM to connect directly• Issue command (go) low overhead• DMA? (back)

29

DMA on a virtualizing system •DMA

•Driver issue a packet consists of command, physical address, etc.

•DMA controller read/write data from/to the physical address•Challenge?

• A physical address that a Guest-OS knows is not really physical !• The really physical address space is managed by the VMM• The DMA controller will incorrectly write data to an address.

30

Intel VT-d(1/2)

•Need the support of the North bridge

•Two functions•Bind devices to a specify VM

• DMA remapping • Interrupt virtualization

• Interrupt remapping

•DMA remapping •DVA (DMA Virtual Address), GPA(Guest Physical Address), HPA(Host Physical Address)

•A guest-OS issue a DMA request with DVA(=GPA)•The VT-d hardware will translate the DVA to HPA

• The concept: lookup tables

31

DMA remapping

32

Intel VT-d (2/2)

• Interrupt Remapping•Assign an interrupt attribute

• Destination processor, vector, etc.•A VMM enables the interrupt requests from the I/O device to target the physical CPUs running the appropriate virtual CPUs of the legacy VM

•AMD IOMMU is similar to Intel VT-d

33

Intel VT-c

•Virtualization Technology for Connectivity•Virtualization on devices

•A collection of technologies that improve the performance of network I/O on a virtualized system

•VT-c is comprised of two components •VMDq (Virtual Machine Device Queues)

• A hardware-base enhancement• Target: throughput

•VMDc (Virtual Machine Direct Connect)• Virtualizing physical I/O ports of a network controller into

multiple virtual I/O ports, and then to map the virtual ports to individual VMs

• Target :VT-x + VT-d + VT-c nearly native performance

34

Why VMDq?

35

36

VMDc

37

DO THESE TECHNIQUES WORK?

38

Ubuntu 11.10: Xen vs. KVM vs. VirtualBox(1)

39

Ubuntu 11.10: Xen vs. KVM vs. VirtualBox(2)

40

Ubuntu 11.10: Xen vs. KVM vs. VirtualBox(3)

41

Ubuntu 12.10: KVM vs. Xen (1)

42

Ubuntu 12.10: KVM vs. Xen (2)

43

Ubuntu 12.10: KVM vs. Xen (3)

44

Ubuntu 12.10: KVM vs. Xen (4)

45

Ubuntu 12.04 KVM/Xen Virtualization: Intel vs. AMD(1)

•Ubuntu 12.04 LTS, an Intel Core i7 3960X "Sandy Bridge" Extreme Edition and AMD FX-8150 "Bulldozer" systems were used.

46

Ubuntu 12.04 KVM/Xen Virtualization: Intel vs. AMD(2)

47

Intel Ivy Bridge Linux Virtualization Performance(1)

48

Intel Ivy Bridge Linux Virtualization Performance(2)

49

Summarization of Hardware Assisted Virtualization •Hardware provides some mechanisms to reduce overheads of virtualization to improve performance

•Pros.•The highest performance in theory (a counter example, 2006 VMware)

•Support unmodified Oss•Simplify the development of VMM

•Cons.•Need newer processors

•Example•KVM(basic requirements)

50

References

• Performance Evaluation of Intel EPT Hardware Assist, VMware

• I/O Virtualization and AMD's IOMMU• http://developer.amd.com/documentation/articles/pages/892006101.aspx

• Processor-Based Virtualization, AMD64 Style, Part I• http://developer.amd.com/documentation/articles/pages/630200614.aspx

• Processor-Based Virtualization, AMD64 Style, Part II • http://developer.amd.com/documentation/articles/pages/630200615.aspx

• Intel technology Journal, vol 10, issue 3, 2006• Intel virtualization technology: Hardware Support for Efficient processor virtualization• Intel virtualization technology for Directed I/O

• ARM virtualization Extension Architecture Specification

• A Comparison of software and hardware techniques for x86 virtualization,Vmware

• http://www.intel.com/network/connectivity/solutions/vmdc.htm

• http://www.intel.com/network/connectivity/solutions/vmdq.htm

• http://software.intel.com/en-us/blogs/2009/09/30/understanding-vt-c-virtualization-technology-for-connectivity/

51

References

•Ubuntu 11.10: Xen vs. KVM vs. VirtualBox http://www.phoronix.com/scan.php?page=article&item=ubuntu_1110_xenkvm&num=1

•Ubuntu 12.04 KVM/Xen Virtualization: Intel vs. AMD http://www.phoronix.com/scan.php?page=article&item=ubuntu_1204_virt&num=1

•Intel Ivy Bridge Linux Virtualization Performance http://www.phoronix.com/scan.php?page=article&item=intel_ivy_virtualization&num=5

•http://en.wikipedia.org/wiki/Hypervisor

52

Q & A

53

THANK YOU