KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf ·...
Transcript of KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf ·...
![Page 1: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/1.jpg)
KVM/ARM
Linux Symposium 2010
Christoffer Dall and Jason Nieh{cdall,nieh}@cs.columbia.edu
Slides: http://www.cs.columbia.edu/~cdall/ols2010-presentation.pdf
Friday, July 16, 2010
![Page 2: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/2.jpg)
We like KVM
• It’s Fast, Free, Open, and Simple!
• Integrates well with Linux
• Always maintained
• Supports x86, ia64, PowerPC, and s390
Friday, July 16, 2010
![Page 3: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/3.jpg)
ARM devices are everywhere
Friday, July 16, 2010
![Page 4: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/4.jpg)
Google Nexus One SpecificationsProcessor Qualcomm Snapdragon QSD8250
CPU Core Qualcomm Scorpion
Architecture ARM v7
Clock speed 1000 MHz
Technology 65 nm
Memory 512 MB
...and they are getting really powerful
Friday, July 16, 2010
![Page 5: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/5.jpg)
KVM relies on hardware support
• x86 and ia64 (Itanium)
• PowerPC, and s390
Friday, July 16, 2010
![Page 6: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/6.jpg)
KVM relies on hardware support
• x86 and ia64 (Itanium)
• PowerPC, and s390
VirtualizationExtensions
Friday, July 16, 2010
![Page 7: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/7.jpg)
KVM relies on hardware support
• x86 and ia64 (Itanium)
• PowerPC, and s390
Virtualizable
VirtualizationExtensions
Friday, July 16, 2010
![Page 8: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/8.jpg)
Hardware Support for Virtualization
• Guest kernel runs in user mode
• Sensitive instructions are instructions that depend on CPU mode
• Virtualizable if all sensitive instructions trap
• Trap-and-emulate
• Hardware virtualization features provide extra mode where all sensitive instructions trap
Friday, July 16, 2010
![Page 9: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/9.jpg)
Problem
• ARM is not virtualizable
• ARM has no hardware virtualization extensions
Friday, July 16, 2010
![Page 10: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/10.jpg)
31 Sensitive instructionsCPS LDRT STC RSBS
MRS STRBT ADCS RSCS
MSR STRT ADDS SBCS
RFE CDP ANDS SUBS
SRS LDC BICS
LDM (2) MCR EORS
LDM (3) MCRR MOVS
STM (2) MRC MVNS
LDRBT MRRC ORRS
Friday, July 16, 2010
![Page 11: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/11.jpg)
31 Sensitive instructionsCPS LDRT STC RSBS
MRS STRBT ADCS RSCS
MSR STRT ADDS SBCS
RFE CDP ANDS SUBS
SRS LDC BICS
LDM (2) MCR EORS
LDM (3) MCRR MOVS
STM (2) MRC MVNS
LDRBT MRRC ORRS
and 25 of them are non-privileged
Friday, July 16, 2010
![Page 12: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/12.jpg)
Solution
• We use lightweight paravirtualization
• Retains simplicity of KVM architecture
• Minimally intrusive to KVM and the Kernel
• Uses on QEMU for device emulation
Friday, July 16, 2010
![Page 13: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/13.jpg)
• KVM
• CPU virtualization on ARM
• Memory virtualization on ARM
• World Switch details
• Implementation status
Friday, July 16, 2010
![Page 14: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/14.jpg)
KVM Architecture
Hardware
Linux KernelKVM
Processes
VM
Guest kernel
QEM
U
Friday, July 16, 2010
![Page 15: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/15.jpg)
KVM execution flow
Friday, July 16, 2010
![Page 16: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/16.jpg)
Start QEMU
Friday, July 16, 2010
![Page 17: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/17.jpg)
Start QEMU
Alloc memory
Friday, July 16, 2010
![Page 18: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/18.jpg)
Start QEMU
Alloc memory
Friday, July 16, 2010
![Page 19: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/19.jpg)
Start QEMU
Alloc memory
Create VM
Friday, July 16, 2010
![Page 20: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/20.jpg)
Start QEMU
Alloc memory
Create VM
Friday, July 16, 2010
![Page 21: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/21.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Friday, July 16, 2010
![Page 22: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/22.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Friday, July 16, 2010
![Page 23: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/23.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
Friday, July 16, 2010
![Page 24: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/24.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
Friday, July 16, 2010
![Page 25: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/25.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
Friday, July 16, 2010
![Page 26: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/26.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
Friday, July 16, 2010
![Page 27: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/27.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
World switch
Kernel
Guest
Friday, July 16, 2010
![Page 28: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/28.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Friday, July 16, 2010
![Page 29: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/29.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
Friday, July 16, 2010
![Page 30: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/30.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
World switch
Friday, July 16, 2010
![Page 31: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/31.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
World switch
Handleexit
Friday, July 16, 2010
![Page 32: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/32.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
World switch
Handleexit
Handle I/O?
Friday, July 16, 2010
![Page 33: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/33.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
World switch
Handleexit
Handle I/O?
Emulation
Friday, July 16, 2010
![Page 34: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/34.jpg)
Start QEMU
Alloc memory
Create VM
Register memory
Create VCPU
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
World switch
Handleexit
Handle I/O?
Emulation
Friday, July 16, 2010
![Page 35: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/35.jpg)
New KVM architecture
• Logical separation of architecture dependent and independent code
•kvm_arch_XXX
•kvm_XXX
Friday, July 16, 2010
![Page 36: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/36.jpg)
• KVM
• CPU virtualization on ARM
• Memory virtualization on ARM
• World Switch details
• Implementation status
Friday, July 16, 2010
![Page 37: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/37.jpg)
ARM virtualization
• ARM is not virtualizable - nor does it have hardware virtualization support
• Possible solutions:
• binary translation
• or paravirtualization
Friday, July 16, 2010
![Page 38: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/38.jpg)
Binary Translation
• Traditionally done out-of-place with a translation cache
• Difficult to make it fast
• Contradicts idea of KVM
Friday, July 16, 2010
![Page 39: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/39.jpg)
Paravirtualization
• Changes the guest kernel to replace code with sensitive instructions with hypercalls
• Guest kernel is modified by hand
• Hard to merge changes with upstream Kernel versions
Friday, July 16, 2010
![Page 40: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/40.jpg)
Original code:
mrs r2, cpsr @ get current modetst r2, #3 @ not user?bne not_angel
Lightweight-paravirtualization(LPV)
Friday, July 16, 2010
![Page 41: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/41.jpg)
Original code:
mrs r2, cpsr @ get current modetst r2, #3 @ not user?bne not_angel
Lightweight-paravirtualization(LPV)
Friday, July 16, 2010
![Page 42: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/42.jpg)
Original code:
swi 0x022000 @ get current modetst r2, #3 @ not user?bne not_angel
Lightweight-paravirtualization(LPV)
Friday, July 16, 2010
![Page 43: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/43.jpg)
Lightweight-paravirtualization(LPV)
• Replace sensitive instructions with traps
• Traps encode original instruction and operands
• Emulate replaced instructions in KVM
• Script-based solution applicable to any vanilla kernel tree
Friday, July 16, 2010
![Page 44: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/44.jpg)
LPV encoding example mrs r2, cpsr
swi 0x022000
Status register access function
23 20 19 16 15 14 12 0 +--------------+-------------------+--+-------------+-----------------------------+ | 0 | Rd | R| 2 | OIF | +--------------+-------------------+--+-------------+-----------------------------+
MRS encoding
Friday, July 16, 2010
![Page 45: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/45.jpg)
LPV implementation
• Uses regular expressions to search for sensitive assembly instructions
• ~150 lines (written in Python)
• Supports inline assembler, preprocessor macros and assembler files.
Friday, July 16, 2010
![Page 46: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/46.jpg)
LPV requirements
• Assumes guest kernel does not make system calls to itself
• Module source code must also be handled
• GCC does not generate sensitive instructions from C-code
Friday, July 16, 2010
![Page 47: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/47.jpg)
LPV key points
• Encodes each sensitive instructions to a single trap
• As efficient as trap-and-emulate
• Fully automated
• Doesn’t affect kernel code size
Friday, July 16, 2010
![Page 48: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/48.jpg)
• KVM
• CPU virtualization on ARM
• Memory virtualization on ARM
• World Switch details
• Implementation status
Friday, July 16, 2010
![Page 49: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/49.jpg)
Virtual memory
0 4 GB
DevicesPhysical
Addresses RAM
Kernel
4 GB0
User space applicationVirtual
Addresses
MMU
Page Tables
Friday, July 16, 2010
![Page 50: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/50.jpg)
New address space
0 4 GB
DevicesHost physical
(Machine) Addresses RAM
Guest Kernel
4 GB0
Guest user space applicationGuest virtual Addresses
MMU
0 4 GB
DevicesGuest physical
AddressesRAM
Friday, July 16, 2010
![Page 51: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/51.jpg)
New address space
0 4 GB
DevicesHost physical
(Machine) Addresses RAM
Guest Kernel
4 GB0
Guest user space applicationGuest virtual Addresses
MMU
Shadow page tables
0 4 GB
DevicesGuest physical
AddressesRAM
Friday, July 16, 2010
![Page 52: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/52.jpg)
Shadow page tables
• Map
• Guest Virtual Addresses to
• Host Physical Addresses
• One per guest page table (process)
• Start out empty and add entries on page faults (on demand)
Friday, July 16, 2010
![Page 53: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/53.jpg)
Address translation
KVM processVirtual Memory
Guest physical
Guest virtual
Host kernel Guest memory
Machine memory
Friday, July 16, 2010
![Page 54: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/54.jpg)
Address translation
KVM processVirtual Memory
Guest physical
Guest virtual
Host kernel Guest memory
Machine memory
Walk guest page tables in software:gva_to_gfn(...);
Friday, July 16, 2010
![Page 55: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/55.jpg)
Address translation
KVM processVirtual Memory
Guest physical
Guest virtual
Host kernel Guest memory
Machine memory
Built-in KVM functionality:gfn_to_hva(...);
Walk guest page tables in software:gva_to_gfn(...);
Friday, July 16, 2010
![Page 56: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/56.jpg)
Address translation
KVM processVirtual Memory
Guest physical
Guest virtual
Host kernel Guest memory
Machine memory
Kernel functionality:page = virt_to_page(...);pfn = page_to_pfn(page);
Built-in KVM functionality:gfn_to_hva(...);
Walk guest page tables in software:gva_to_gfn(...);
Friday, July 16, 2010
![Page 57: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/57.jpg)
Shadow page table consistency
• Caching shadow page tables is an optimization
• Keep cached page tables in sync by protecting guest page tables and tracking updates
Friday, July 16, 2010
![Page 58: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/58.jpg)
• Goal
• Protect host from guest
• Honor intended guest protection
• ARM provides flexible protection methods
• Access is specified per CPU privilege level
Memory Protection
Friday, July 16, 2010
![Page 59: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/59.jpg)
Access Protection Bits
AP Privileged User
00 None None
01 R/W None
10 R/W R/O
11 R/W R/W
Friday, July 16, 2010
![Page 60: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/60.jpg)
Access mapping example
• Guest page table specifies:
• Privileged: R/W
• User: No Access
• Shadow page table bits in guest user mode:
• User: No Access
• Shadow page table bits in guest priv. mode:
• User: R/W
Friday, July 16, 2010
![Page 61: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/61.jpg)
Access mapping example
• Guest page table specifies:
• Privileged: R/W
• User: No Access
• Shadow page table bits in guest user mode:
• User: No Access
• Shadow page table bits in guest priv. mode:
• User: R/W
Friday, July 16, 2010
![Page 62: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/62.jpg)
Access mapping example
• Guest page table specifies:
• Privileged: R/W
• User: No Access
• Shadow page table bits in guest user mode:
• User: No Access
• Shadow page table bits in guest priv. mode:
• User: R/W
Friday, July 16, 2010
![Page 63: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/63.jpg)
• KVM
• CPU virtualization on ARM
• Memory virtualization on ARM
• World Switch details
• Implementation status
Friday, July 16, 2010
![Page 64: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/64.jpg)
KVMRUN
User space
Kernel
World switch
Native guest execution
Guest
Interrupt
World switch
Handleexit
Handle I/O?
Emulation
World Switches
Friday, July 16, 2010
![Page 65: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/65.jpg)
World switch
• Disable interrupts
• Store host state
• Switch page tables
• Load guest state
• Enable interrupts
• Jump to guest code
• Store exit state
• Switch page tables
• Restore host state
• (Host kernel IRQ handler)
• Enable interrupts
• Return to ioctl call
To guest From guest
Friday, July 16, 2010
![Page 66: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/66.jpg)
World switch
• Disable interrupts
• Store host state
• Switch page tables
• Load guest state
• Enable interrupts
• Jump to guest code
• Store exit state
• Switch page tables
• Restore host state
• (Host kernel IRQ handler)
• Enable interrupts
• Return to ioctl call
To guest From guest
Friday, July 16, 2010
![Page 67: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/67.jpg)
World switch
• Disable interrupts
• Store host state
• Switch page tables
• Load guest state
• Enable interrupts
• Jump to guest code
• Store exit state
• Switch page tables
• Restore host state
• (Host kernel IRQ handler)
• Enable interrupts
• Return to ioctl call
To guest From guest
Friday, July 16, 2010
![Page 68: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/68.jpg)
World switch
• Disable interrupts
• Store host state
• Switch page tables
• Load guest state
• Enable interrupts
• Jump to guest code
• Store exit state
• Switch page tables
• Restore host state
• (Host kernel IRQ handler)
• Enable interrupts
• Return to ioctl call
To guest From guest
Friday, July 16, 2010
![Page 69: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/69.jpg)
Switch page tables
PC
Friday, July 16, 2010
![Page 70: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/70.jpg)
Shared Page
Machine memory
Guest Kernel
4 GB0
User space applicationGuest Virtual
Addresses
Host Kernel
4 GB0
QEMU virtual memoryHost VirtualAddresses
0xFFFF1000
0xFFFF1000
Friday, July 16, 2010
![Page 71: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/71.jpg)
Shared Page
Machine memory
Guest Kernel
4 GB0
User space applicationGuest Virtual
Addresses
Host Kernel
4 GB0
QEMU virtual memoryHost VirtualAddresses
0xFFFF1000
0xFFFF1000
Friday, July 16, 2010
![Page 72: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/72.jpg)
Shared Page Internals
Temporary Data
Code
TemporaryStack
0xffff 1000
0xffff 1fff
Friday, July 16, 2010
![Page 73: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/73.jpg)
• KVM
• CPU virtualization on ARM
• Memory virtualization on ARM
• World Switch details
• Implementation status
Friday, July 16, 2010
![Page 74: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/74.jpg)
Status
• Successfully boots Linux VMs
• Host built on Android Kernel 2.6.27
• Tested guest kernels from 2.6.17 to 2.6.33
Friday, July 16, 2010
![Page 75: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/75.jpg)
• Improve performance
• Cache shadow page tables
• Avoid unnecessary world-switches
• Binary patching
• Test device support
• Upstream!
Future work
Friday, July 16, 2010
![Page 76: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/76.jpg)
ARMv6
• Physically tagged caches
• TLB “Application Space Identifiers” (ASID’s)
• New instructions
Friday, July 16, 2010
![Page 77: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/77.jpg)
Related Work
• Commercial solutions:
• VMWare MVP, OK Labs, VirtualLogix, ...
• Open-source:
• QEMU
• XenARM
Friday, July 16, 2010
![Page 78: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/78.jpg)
Conclusions
• ARM virtualization is important
• With LPV we now have KVM/ARM
• LPV is simple, fully automated, and efficient
• Minimally intrusive
• It works!
Friday, July 16, 2010
![Page 79: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/79.jpg)
Tasks• Caching of shadow page tables
• Moving things to shared page
• Coalesced MMIO
• GDB support
• Testing devices (on BeagleBoards, IGEPv2 boards etc.)
• ...
Friday, July 16, 2010
![Page 80: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/80.jpg)
Want to contribute?
• Mailing list:[email protected]
• WIKI:http://wiki.ncl.cs.columbia.edu
• Source code:http://git.ncl.cs.columbia.edu/git
Friday, July 16, 2010
![Page 81: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/81.jpg)
Extra Material
Friday, July 16, 2010
![Page 82: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/82.jpg)
• Same as on x86:
• Test and Development
• OS freedom
• Multiple Personas
• Virtualization features
Use cases
Friday, July 16, 2010
![Page 83: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/83.jpg)
Exceptions
• Traps & Interrupts
• CPU changes mode and execution starts from “vectors” at either:
• 0x00000000 + offset
• or 0xFFFF0000 + offset
Friday, July 16, 2010
![Page 84: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/84.jpg)
Exceptions and KVM/ARM
• KVM/ARM uses custom handlers to handle exceptions while executing guest
• Exceptions are the only way to:“exit from the guest”
• IRQ’s are forwarded to the host kernel handlers
• Traps are handled by KVM/ARM
Friday, July 16, 2010
![Page 85: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/85.jpg)
Guest Kernel
4 GB
User space applicationGuest Virtual
Addresses
0xFFFF0000
Hardware exceptionvector page
0x0
Guest exceptionvector page
Guest exceptions
Guest uses “low” vectors
Friday, July 16, 2010
![Page 86: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/86.jpg)
What happens at a conflict?
• KVM/ARM’s vectors are mapped with no-access for user mode code at 0xffff0000
• The guest tries to access 0xffff0000 page
• KVM/ARM handles the permission fault
Friday, July 16, 2010
![Page 87: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/87.jpg)
Guest Kernel
4 GB
User space applicationGuest Virtual
Addresses
0xFFFF0000
Hardware exceptionvector page
0x0
Guest exceptionvector page
Exception page conflict
Friday, July 16, 2010
![Page 88: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/88.jpg)
Guest Kernel
4 GB
User space applicationGuest Virtual
Addresses
0x0
Hardware exceptionvector page
0xffff0000
Guest exceptionvector page
Exception page conflict
Guest uses “high” vectors
Friday, July 16, 2010
![Page 89: KVM/ARM - Department of Computer Science, Columbia Universitycdall/pubs/ols2010-presentation.pdf · World switch • Disable interrupts • Store host state • Switch page tables](https://reader033.fdocuments.in/reader033/viewer/2022053019/5f22aa69f6017c4649243da5/html5/thumbnails/89.jpg)
Guest Kernel
4 GB
User space applicationGuest Virtual
Addresses
0xFFFF0000
Hardware exceptionvector page
Exception page conflict
Guest uses “high” vectors,but needs access to page 0
Friday, July 16, 2010