Sogang University Distributed Computing & Communication Lab.
Concurrent Direct Network Access for Virtual Machine MonitorsPaul Willmann, Jeffrey Shafer, David Carr, Aravind Menon,
Scott Rixner, Alan L. Cox, Willy Zwaenepoel
Kwon-yong LeeKwon-yong Lee
Distributed Computing & Communication Lab.Distributed Computing & Communication Lab.(URL: http://dcclab.sogang.ac.kr)(URL: http://dcclab.sogang.ac.kr)
Dept. of Computer Science Dept. of Computer Science Sogang UniversitySogang University
Seoul, KoreaSeoul, Korea
Tel : +82-2-3273-8783Tel : +82-2-3273-8783 Email : [email protected] : [email protected]
2
Necessity of new I/O Virtualization Tech.
Improving the efficiency of VMMs Hardware support for virtualization Reducing the software overhead of virtualization
Networking by VMM Virtual Network Interface
• Multiplexed in software onto a physical network interface card (NIC)
Overhead of the software-based network virtualization• Limiting network performance
No clear solution to improve the efficiency of I/O virtualization
3
Xen
Key functions in order to provide VM environments Allocating the physical resources of the machine to the
guest OSes and isolating them from each other
Receiving all interrupts in the system and passing them on to the guest OSes, as appropriate
All I/O operations go through Xen.• In order to ensure fair and non-overlapping access to I/O
devices by the guests
PerformanceSystem Transmit (Mb/s) Receive (Mb/s)
Native Linux 5126 3629
Xen 3.0 Guest 1602 1112
Organization of the Xen VMM
4
Networking in Xen
HypervisorControl
Hypervisor
PageFlippin
gPacketData
Interrupt Dispatch
NIC
Driver DomainBack-End
DriverEtherne
tBridge
NIC Driver
Back-End
Driver
GuestDomain 1
Front-End
Driver
GuestDomain 2
Front-End
Driver
CPU / Memory / Disk / Other Devices
DriverContro
lPacketData Control +
DataInterrupts
5
Hypervisor and Driver Domain Operation
Hypervisor
PageFlippin
gPacketData
Interrupt Dispatch
NIC
Driver DomainBack-End
DriverEtherne
tBridge
NIC Driver
Back-End
Driver
GuestDomain 1
Front-End
Driver
GuestDomain 2
Front-End
Driver
CPU / Memory / Disk / Other Devices
DriverContro
lPacketData
Packet Transmit
6
Hypervisor and Driver Domain Operation
Hypervisor
PageFlippin
gPacketData
Interrupt Dispatch
NIC
Driver DomainBack-End
DriverEtherne
tBridge
NIC Driver
Back-End
Driver
GuestDomain 1
Front-End
Driver
GuestDomain 2
Front-End
Driver
CPU / Memory / Disk / Other Devices
DriverContro
lPacketData Interrupts
Packet Receive
Virtual Interrupts
Virtual Interrupts
7
Device Driver Operation
Interactions between the device driver and the NIC Programmed I/O (PIO) operations from the driver to the
NIC Direct memory access (DMA) transfers by the NIC to
read or write host memory Physical interrupts from the NIC to invoke the device
driver
The device driver directs the NIC To send packets from buffers in host memory To place received packets into pre-allocated buffers in
host memory
8
Device Driver Operation
Host OS
NIC
mailbox
DeviceDriver Buffer
PIO
Transmit Receive
DMA descriptor ring
Producer Index
DMA descriptor
(Buffer’s lengthand
physical address)
Newpacke
t
NewDMA
descriptor
Packet Transmit
A new descriptor is available.(ring index)
DMA read
Consumer Index
With CDNA, each guest operating system is connected directly to its own network interface.→ Elimination many of the overheads of network virtualization Communication overheads between the guest and driver domains Software multiplexing overheads within the driver domain
9
Concurrent Direct Network Access
10
Concurrent Direct Network Access
CDNA Network Interface Supporting multiple contexts in hardware
• Acts as if it is an independent physical network interface• Controlled by a separate device driver instance
Each guest can transmit and receive network traffic using its own private context directly.
Network interface Multiplexing the traffic across all of its active contexts
Hypervisor Providing protection across the contexts
11
Concurrent Direct Network Access
Multiplexing network traffic on the NIC Network interface
• Identifying the source/target OS for all network traffic• Providing independent hardware contexts and associating a
unique Ethernet MAC address with each context• Each context must include a unique set of mailboxes.
– This isolates the activity of each guest OS, so that the NIC can distinguish between the different guests.
Hypervisor• Assigning a unique hardware context on the NIC to each guest
OS– By simply mapping I/O locations for that context’s mailboxes into
the guest’s address space
Device driver within the guest OS• Interact with its context exactly
– Creating DMA descriptors and Updating a mailbox on the NIC via PIO
12
Concurrent Direct Network Access
Interrupt Delivery The hardware contexts on the NIC must be able to
interrupt their respective guests.
NIC• Encoding the interrupt set of contexts in an interrupt bit
vector• Transferring into the hypervisor’s memory space using
DMA• Raising a physical interrupt, which invokes the hypervisor’s
interrupt service routine
Hypervisor• Decoding all of the pending interrupt bit vectors• Scheduling virtual interrupts to each of the guest OSes
13
Concurrent Direct Network Access
DMA Memory Protection Key I/O protection violation
• Transmit packets : Security hole• Receive packets : Corrupting memory that is in use
Additional security features are needed. The CDNA architecture for the isolation between guests
• Validating and protecting all DMA descriptors • Ensuring sources or targets of DMA accesses
Protection provided by CDNA• The device driver in each guest must call into the
hypervisor to perform the en-queue operation.• Hypervisor’s exclusive write access to the host memory
region– Preventing from independently en-queuing unauthorized DMA
descriptors
14
CDNA NIC Implementation
RiceNIC Programmable and reconfigurable FPGA-based Gigabit
Ethernet network interface
Additional mailbox storage and handling logic
Modification• To provide multiple protected contexts• To multiplex network traffic• To interact with the hypervisor through a dedicated context
to allow privileged management operations
15
Evaluation
Xen 3.0 Unstable
NIC A single dual-port Intel Pro/1000 MT NIC Two RiceNICs configured to support CDNA
16
Evaluation – Single Guest Performance
17
Evaluation – Single Guest Performance
CDNA I/O virtualization architecture provides significant performance improvements over Xen for both transmit and receive. Transmit side
• CDNA requires half the processor resource to deliver about 200 Mb/s higher throughput.
Receive side• CDNA requires 60% of the processor resources to deliver
about 750 Mb/s higher throughput
18
Evaluation – Memory Protection
The software-based protection mechanisms in CDNA can potentially be replaced by a hardware IOMMU.
19
Evaluation – Scalability
Summary
To support CDNA, A VMM would only need to add mechanisms To deliver interrupts as directed by the network
interface To perform DMA memory protection
Required to enable a commodity NIC to support CDNA The NIC must provide multiple contexts that can be
accessed by programmed I/O. The NIC must support several mailboxes within each
context.
20
Top Related