Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM
-
Upload
the-linux-foundation -
Category
Technology
-
view
48 -
download
7
Transcript of Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM
![Page 1: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/1.jpg)
Hypervisors on ARMOverview and Design choices
Julien Grall <[email protected]>
Root Linux Conference 2017
© ARM 2017
![Page 2: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/2.jpg)
About me
Working on ARM virtualization for the past 4 years
With ARM since 2016
Co-maintaining Xen on ARM - with Stefano Stabellini [Aporeto]
2 © ARM 2017
![Page 3: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/3.jpg)
Virtualization, what is it?
3 © ARM 2017
![Page 4: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/4.jpg)
Virtualization refers to the act ofcreating a virtual version of something
Wikipedia
4 © ARM 2017
![Page 5: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/5.jpg)
5 © ARM 2017
![Page 6: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/6.jpg)
Use cases
6 © ARM 2017
![Page 7: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/7.jpg)
Type of hypervisors
Figure: From wikipedia
7 © ARM 2017
![Page 8: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/8.jpg)
Kind of virtualization
Full hardware virtualizationOS is running unmodifiedGuest I/O are either
emulatedhandled by virtualization-aware hardware
Para-virtualizationOS is aware of the hypervisorPrivilege instruction are replaced by hooksDevices (network, block...) are para-virtualized
The trend is a mix of bothUse as much as possible hardware-assisted virtualizationDevices (network, block...) para-virtualized or passthrough-edEmulation very limited
8 © ARM 2017
![Page 9: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/9.jpg)
ARM virtualization
9 © ARM 2017
![Page 10: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/10.jpg)
ARMv8-A Privilege Model
Support both AArch32 and AArch64 execution modes
32-64bit inter-working limited to exception boundaries
AArch64 always has a higher privilege than AArch32
AArch64 state is a superset of AArch32 state
10 © ARM 2017
![Page 11: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/11.jpg)
ARM virtualization
Introduced with the latest version of ARMv7 architecture
New hypervisor execution state
Non-Secure world, higher privilege than EL1
11 © ARM 2017
![Page 12: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/12.jpg)
Virtualization in a nutshell
Second stage of memory translationAdds an extra level of indirection between guests and physical memoryTLBs are tagged by Virtual Machine ID (VMID)
Ability to trap access of most system registersThe hypervisor decides what it wants to trap
Can handle IRQs, FIQs and asynchronous abortsThe guest doesn’t see physical interrupts firing, for example
Guests can call into EL2 mode (HVC instruction)Allows para-virtualizated services
Standard architecture peripherals are virtualization-awareGIC and timer have specific features to help virtualization
12 © ARM 2017
![Page 13: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/13.jpg)
EL2: Not EL1++ (ARMv8.0-A)
EL2 is not a superset of NS-EL1Orthogonal mode to EL1Allows multiplexing of NS-EL1 guests on the hardware
Own translation regimeSeparate Stage-1 translation, no Stage-2 translation
It would be difficult to run Linux in EL2Requires too many changes to be practical
EL2 could be used as a ”world switch”Between guests (barametal hypervisor/Type I)Between host and guest (hosted hypervisor/Type II)This makes the host a form of specialized guest.
13 © ARM 2017
![Page 14: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/14.jpg)
Hypervisor architecture - Type I
EL1
EL0
EL2
Guest Kernel
Hypervisor
Guest Kernel Guest Kernel
Guest
Userspace Userspace UserspaceGuest Guest
14 © ARM 2017
![Page 15: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/15.jpg)
Hypervisor architecture - Type II
EL1
EL0
EL2
HYP
Host Kernel
Switching Code
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
15 © ARM 2017
![Page 16: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/16.jpg)
EL2 enhancement (ARMv8.1-A)
The Virtualization Host Extension (VHE) expands the capability of EL2:
Designed to improve the support of the Type-2 hypervisors
Allows the host OS to be run at EL2
The host OS requires minimal changes to run at EL2
User-space still runs at EL0
Host has no software running at EL1
AArch64 specific
EL2 becomes a strict superset of EL1
16 © ARM 2017
![Page 17: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/17.jpg)
Hosted hypervisor architecture on platform without VHE
EL1
EL0
EL2
HYP
Host Kernel
Switching Code
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
17 © ARM 2017
![Page 18: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/18.jpg)
Hosted hypervisor architecture on platform with VHE
EL1
EL0
EL2
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
Host Kernel + HYP
18 © ARM 2017
![Page 19: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/19.jpg)
Nested Virtualization (ARMv8.3-A)
The Nested Virtualization extension allows an hypervisor in a VM.
Unmodified guest hypervisor running in NS EL1Implementation of a host hypervisor required
Running at EL2
AArch64 specific
19 © ARM 2017
![Page 20: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/20.jpg)
Why using ARM virtualization
Robust set of virtualization featuresNot just about CPU virtualizationCovers the whole systems architecture
Scalable architecturePower to IoT-like devices ...... all the way to server-grade systems
An architecture in motion:ARMv8.1-A: https://goo.gl/Ox4thVARMv8.2-A: https://goo.gl/0Ns37UARMv8.3-A: https://goo.gl/CJv1n0
20 © ARM 2017
![Page 21: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/21.jpg)
OpenSource Hypervisors
21 © ARM 2017
![Page 22: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/22.jpg)
22 © ARM 2017
![Page 23: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/23.jpg)
KVM
23 © ARM 2017
![Page 24: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/24.jpg)
Kernel-based Virtual Machine
First version of KVM was merged in Linux 2.6.20AArch32 support merged in Linux 3.9AArch64 support merged in Linux 3.11
Source code available as GLP v2
Hosted hypervisor
24 © ARM 2017
![Page 25: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/25.jpg)
KVM virtual machine
Use of assisted hardware virtualizationDevices are
emulated (QEMU)para-virtualized (VIRTIO)
25 © ARM 2017
![Page 26: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/26.jpg)
KVM architecture
26 © ARM 2017
![Page 27: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/27.jpg)
KVM architecture with ARMv8.0-A
EL1
EL0
EL2
KVM
Host Kernel
Switching Code
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
27 © ARM 2017
![Page 28: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/28.jpg)
KVM architecture with ARMv8.1-A
EL1
EL0
EL2
Guest Kernel Guest Kernel
Host
Userspace Userspace UserspaceGuest Guest
Host Kernel + KVM
28 © ARM 2017
![Page 29: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/29.jpg)
Resource management
All CPUs are using the same scheduler
guest vCPU is a task for the host OSResource management can be done using cgroup
Standard way in Linux to control resources
29 © ARM 2017
![Page 30: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/30.jpg)
Jailhouse
30 © ARM 2017
![Page 31: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/31.jpg)
Jailhouse
Created at Siemens in 2013Partioning hypervisor
Type-I hypervisorLinux will load Jailhouse
Source code available as GPL v2
Small code base: <10K lines
31 © ARM 2017
![Page 32: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/32.jpg)
Jailhouse architecture
32 © ARM 2017
![Page 33: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/33.jpg)
Xen
33 © ARM 2017
![Page 34: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/34.jpg)
Xen
First released in 2003ARM officially supported since Xen 4.4
Source code available as GPL v2
Small code base: 30K
Bare-metal hypervisor
34 © ARM 2017
![Page 35: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/35.jpg)
Xen architecture
35 © ARM 2017
![Page 36: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/36.jpg)
Xen architecture - 2
36 © ARM 2017
![Page 37: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/37.jpg)
Xen schedulers
37 © ARM 2017
![Page 38: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/38.jpg)
Xen schedulers
38 © ARM 2017
![Page 39: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/39.jpg)
Summary
https://www.linux-kvm.org/
https://github.com/siemens/jailhouse
https://xenproject.org/
39 © ARM 2017
![Page 40: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/40.jpg)
Questions?
40 © ARM 2017
![Page 41: Rootlinux17: Hypervisors on ARM - Overview and Design Choices by Julien Grall, ARM](https://reader034.fdocuments.in/reader034/viewer/2022042511/58ee4ea01a28abcd668b4657/html5/thumbnails/41.jpg)
The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM limited
(or its subsidiaries) in the EU and/or elsewhere. All rights reserved. All other marks featured may be
trademarks of their respective owners.
Copyright © 2017 ARM Limited
© ARM 2017