Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for...
Transcript of Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for...
![Page 1: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/1.jpg)
Yabusame: Postcopy Live Migration for Qemu/KVM
Takahiro Hirofuchi, AIST
Isaku Yamahata, VALinux Systems Japan K.K.
KVM Forum 2011
This project is partially funded by Ministry of Economy, Trade and Industry.
![Page 2: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/2.jpg)
Outline
• What is Postcoy Live Migration?
– Comparison with Precopy
– Experience from an early, ad-hoc prototype
• Postcopy for Qemu/KVM
– Production-level design
– Qemu/KVM upstream merge
Takahiro
Isaku
![Page 3: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/3.jpg)
Precopy v.s. Postcopy
• Precopy live migration
– Copy VM memory before switching the execution host
– Widely used in VMMs
• Postcopy live migration
– Copy VM memory after switching the execution host
– Yabusame
3
![Page 4: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/4.jpg)
Demo http://grivon.apgrid.org/quick-kvm-migration
![Page 5: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/5.jpg)
5
1. Copy all memory pages to destination
2. Copy memory pages updated during the previous copy again
3. Repeat the 2nd step until the rest of memory pages are enough small
4. Stop VM
5. Copy CPU registers, device states, and the rest of memory pages.
6. Resume VM at destination
VM
Machine A Machine B
Precopy Live Migration (1) Copy VM memory before relocation
![Page 6: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/6.jpg)
6
Memory update speed (pages/s)
Mig
ration t
ime (
s)
Migration time depends on memory update speeds.
Precopy Live Migration (2) Copy VM memory before relocation
![Page 7: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/7.jpg)
Postcopy Live Migration (1) Copy VM memory after relocation
7
VM
Machine A Machine B
Stop 1.Stop VM
2.Copy CPU and device states to destination
3.Resume VM at destination
4.Copy memory pages
RAM
![Page 8: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/8.jpg)
Postcopy Live Migration (2) Copy VM memory after relocation
8
VM
Machine A Machine B
1.Stop VM
2.Copy CPU and device states to destination
3.Resume VM at destination
4.Copy memory pages
Copy CPU and device states 512KB or so (w/o VGA)
=> Less than 1 sec for relocation
RAM
![Page 9: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/9.jpg)
Postcopy Live Migration (3) Copy VM memory after relocation
9
VM VM
Machine A Machine B
1.Stop VM
2.Copy CPU and device states to destination
3.Resume VM at destination
4.Copy memory pages
Resume
RAM
![Page 10: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/10.jpg)
Postcopy Live Migration (4) Copy VM memory after relocation
10
VM VM
Machine A Machine B
1.Stop VM
2.Copy CPU and device states to destination
3.Resume VM at destination
4.Copy memory pages
Copy memory pages • On-demand • Background (Precache)
RAM
![Page 11: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/11.jpg)
Precopy v.s. Postcopy
Precopy Postcopy
Before the execution host is switched, memory pages are transferred to the destination.
After the execution host is switched, memory pages are transferred to the destination.
The time until the execution
host is switched
The time until all states are
removed
RAM size
Network speed + alpha
RAM size
Network speed + alpha
RAM size
Network speed
200-300ms
• alpha: depends on memory update speed (non deterministic!) • Note the above values are the worst case.
• Qemu skips zero-filled page.
![Page 12: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/12.jpg)
Question (1)
• Is there performance loss after relocation?
– Yes, (hopefully?) slightly.
– The working set of memory pages is limited.
– These pages are precached as soon as possible.
![Page 13: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/13.jpg)
13
Migrate an heavily-loaded Web Server VM
The execution host is instantaneously switched.
Avera
ge R
esp
onse
Tim
e (
s)
Using the SPECweb 2005 Banking benchmark.
Performance loss is limited.
![Page 14: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/14.jpg)
14
Transferred Page Offsets (Precache Disabled)
Most page offsets are limited in this area. => Precache this area for better performance.
![Page 15: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/15.jpg)
Question (2)
• Useful for dynamic server consolidation?
– Hopefully, yes.
– Enables quick, deterministic load balancing.
![Page 16: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/16.jpg)
Dynamic Consolidation
16
VM
Assure VM Performance
Distribute running VMs into other hosts
Power up new physical hosts
The CPU usage of physical machines
Remove overload
When datacenter load becomes high,
![Page 17: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/17.jpg)
Postcopy v.s. Precopy for dynamic consolidation
17
The consolidation system using precopy cannot quickly migrate VMs, resulting serious performance loss on sudden load changes.
Postcopy contributes alleviating performance loss.
Performance Degradation(%)
Detection overhead
Memory Update Intensity of Workloads(Gbyte/s@CPU100%)
Using
Using
![Page 18: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/18.jpg)
Summary of the first half
• Yabusame
– Postcopy Live Migration for Qemu/KVM
– The execution host is switched in 200-300ms.
– The total migration time is shorter than precopy. It is deterministic!
– An early-stage, ad-hoc, ad-hoc, concept-proof prototype is here.
• http://grivon.apgrid.org/quick-kvm-migration
• Do not use it!
• Next, discuss a upstream merge-able design…
![Page 19: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/19.jpg)
Why re-design/implementation● Next step of YABUSAME project
● We'd like to merge post copy livemigartion into the upstream
● The existing patch was implemented for ● academic research● proof-of-concept
● So, its design/implementation is● Ad-hoc, quick-hack● Not suitable for the upstream merge
● => re-design/implement it for the upstream merge
![Page 20: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/20.jpg)
Prerequisite● New implementation should satisfy
● Allow qemu/kvm features● target/host agnostic
● accelerator(tcg, kvm)
● Devices
– emulated devices, virtio, vhost
– assigned-device won't be supported
● Allow kvm host features● Swap, KSM, THP/hugetlb, async page fault, coredump...
● Zero overhead after migration completes● CPU, memory
● Minimal administrative operation● Don't require special preparation on migration source
![Page 21: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/21.jpg)
qemu-kvm
characterdevice
qemu-kvm
guest RAMvma::fault2. page fault is hooked byCharacter device.
destination
daemon
3. Request for page1. access toguest RAM
4. page contents is sent backConnection for live-migrationis reused
5. page contentsis passed down tothe driver
Host kernel
6. resolve page fault
0. mmap()
source
Implementation proposal
guest RAM
Host kernel
![Page 22: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/22.jpg)
Design points● Who on the destination handles page requests
● An independent daemon or a thread in qemu-kvm
● connection between source/destination● Re-use the connection of live-migration or● Create new connection
● Page transfer protocol. Based on● Qemu live migration or● other protocol: nbd, iSCSI, AOE
● How to hook guest RAM access
![Page 23: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/23.jpg)
Hooking guest RAM access
● Insert hooks all accesses to guest RAM● Character device driver● Backing store(block device or file)● Swap device
![Page 24: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/24.jpg)
Insert hooks
● Insert hooks all accesses to guest RAM● Carefull code inspection is required● Pros
● Portable. May work with qemu tcg without any kernel drivers
● Cons● Impractical
![Page 25: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/25.jpg)
Backing device/file approach● Use block device or file as backing store for guest RAM
● Pros● New device driver isn't needed
● Cons● Future improvement would be difficult
● Some KVM host features wouldn't work.(KSM, THP)
Regular file orBlock device
qemu-kvm
guest RAM
destination
Host kernel
mmap()
IO
FUSE or nbd orUser block device
![Page 26: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/26.jpg)
Character device approach● Character device to handle page fault on guest RAM area
● Pros● Straight forward● Future improvement would be easy
● Cons● New driver is necessary● Some KVM host features wouldn't work.(KSM, THP)– VMA isn't anonymous
– Can be fixed characterdevice
qemu-kvm
guest RAM
vma::fault
destination
Host kernel
mmap()
pagerequest
![Page 27: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/27.jpg)
Swap device approach● On destination, set up such that all guest RAM are swapped out
● Pros● Every thing would be normal after migration completes
● New device driver isn't needed
● Cons● Future improvement would be difficult
● Administration– setting up swap device
Block device
qemu-kvm
guest RAM
destination
Host kernel
mmap()
IO
nbd oruser block device
swap
![Page 28: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/28.jpg)
ComparisonPros Cons
Modify VMM portability impractical
Backing store No new device driver Difficult future improvementSome kvm host features wouldn't work
Character Device Straight forwardFuture improvement
Need to fix kvm host features
Swap device Everything is normal after migration
AdministrationDifficult future improvement
![Page 29: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/29.jpg)
Future work after the merge● Evaluation/benchmark
● Optimization● Another connection for background page transfer
– Bandwidth control
● Reduce unnecessary page fault● Mix precopy/postcopy ● Avoid memory copy● Hint not to send page contents● Not to fetch pages when writing/clearing whole page
– cleancache/frontswap might be good candidate
● Libvirt support is necessary?
● Cooperate with Kemari
![Page 30: Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum](https://reader030.fdocuments.in/reader030/viewer/2022040121/5eae22a6ea2a600e295738d5/html5/thumbnails/30.jpg)
Questions/discussions● We want the community feedback
● Project page● http://sites.google.com/site/grivonhome/quick-kvm-migration
● Enabling Instantaneous Relocation of Virtual Machines with a Lightweight VMM Extension: proof-of-concept, ad-hoc prototype. not a new design● http://grivon.googlecode.com/svn/pub/docs/ccgrid2010-hirofuchi-paper.pdf
● http://grivon.googlecode.com/svn/pub/docs/ccgrid2010-hirofuchi-talk.pdf
● Reactive consolidation of virtual machines enabled by postcopy live migration: advantage for VM consolidation● http://portal.acm.org/citation.cfm?id=1996125
● http://www.emn.fr/x-info/ascola/lib/exe/fetch.php?media=internet:vtdc-postcopy.pdf
● Qemu Wiki● http://wiki.qemu.org/Features/PostCopyLiveMigration