andrew.warfield@cl.cam.ac.uk

Virtual Devices for Virtual Machines

Andrew Warfield

or

Building Extensible Deviceswith a Chainsaw and some Duct Tape

Despite being a coarse-grained tool, VMMs are a great substrate for

low-level extensibility.

Take-home message:

“VMMs: They’re boxy, but they’re good!”

Soft Device

Why do we want to extend devices?

1. Add new device-level systems features.• Versioning storage, encryption, checksum offload.

2. Manage I/O for VMs in a cluster.• They move around!• I/O looks a lot like a “service”.

VM(OS + Apps)

Physical Device(e.g. disk)

Extension(e.g. encryption)

What are the challenges:

1. Performance.• “Fast enough to be useful.”

2. Safety. • Extensions shouldn’t cause system to melt.

3. “Software Engineering”.• Extension writers shouldn’t have to piece together the

internals of an OS, learn undocumented interfaces and locking disciplines, or spend all their time tracking versions. Extensions should be portable across OSes.

The rest of this talk:

Why VMMs are a good platform for device extensions.a.k.a. how devices work in Xen.

An extension interface for Xen.Case: Packet symmetry enforcement.Case: The Parallax storage service.

New and somewhat crazy stuff. Conclusion.

Devices in Xen.

OS

Web Mail

HW

VMM

OS

DD VD.D VD.D

Switch from single OS to VMM: Web and mail servers arenow isolated (or consolidated), but how to share devices?Option 1: VMM runs physical device driver. VM drivers for“virtual” device. Either real (emulated) HW, or idealized.

Problem 1: Maintenance.VMM is now in the “driver trap”… has to support new hardware, is constantly chasing driver evolution.

Problem 2: Stability.Drivers are a huge amount of (often buggy)code. We don’t want to count on them for reliability.

Sometimes they crash.

A digression…

Duct Tape:(Simple yet effective connective mechanism.)

No generalized IPC in the VMM Event notification (virtual irqs) Memory sharing primitives

Chainsaw:(Somewhat imprecise surgical instrument)

The OS is a component. OSes work, have lots of tools, and

know how to talk to other OSes. Maintained by other people.

Devices in Xen.

OS

Web Mail

HW

VMM

OSOS(Device VM)

DD

VD.D VD.DVD.S

Option 2: VMM exports physical hardware to a device VM.Use OS driver, OS mechanisms (e.g. packet forwarding)

Devices in Xen.

Mail

HW

VMM

OSOS(Device VM)

DD VD.DVD.S

Option 2: VMM exports physical hardware to a device VM.Use OS driver, OS mechanisms (e.g. packet forwarding)

1. Coarse-grained sharing

R R R

1. Coarse-grained sharing 2. Simple one-bit notification

Devices in Xen.

Mail

HW

VMM

OSOS(Device VM)

DD VD.DVD.S

Option 2: VMM exports physical hardware to a device VM.Use OS driver, OS mechanisms (e.g. packet forwarding)

1. Coarse-grained sharing

R RR

1. Coarse-grained sharing 2. Simple one-bit notification

RRR

Devices in Xen.

Mail

HW

VMM

OSOS(Device VM)

DD VD.DVD.S

Option 2: VMM exports physical hardware to a device VM.Use OS driver, OS mechanisms (e.g. packet forwarding)

1. Coarse-grained sharing

R

1. Coarse-grained sharing 2. Simple one-bit notification

RR

Point of discussing that in (some) detail:

• There is a fast, general mechanism for moving requests around.

•Classes of drivers use simple protocols for requests.

•VMM is largely uninvolved in the details.

•Device driver is an extension.

Extending Devices

Mail

HW

VMM

OSOS(Device VM)

DD VD.DVD.S

OS(Extention VM)

Dev.Tap

Extension App.

Logging. e.g. debug/analysisRedirecting. e.g. add versioningRewriting. e.g. encryption

The device tap acts as a switch. In addition to interposingAll messages, requests may be terminated or passed through.

Devices in Xen.

Mail

HW

VMM

OSOS(Device VM)

DD VD.DVD.S

Extending Devices

HW

VMM

OS(Device VM)

DD

Mail

OS

VD.DVD.S

OS(Extention VM)

Dev.Tap

Extension App.

Logging. e.g. debug/analysisRedirecting. e.g. add versioningRewriting. e.g. encryption

The device tap acts as a switch. In addition to interposingAll messages, requests may be terminated or passed through.Trading isolation for performance. We may move the tapAnd extension code onto the device VM.

Extension Goals:

• Safety. •Software Engineering. •Performance.

Extension Case Study: Packet Symmetry

DoS is a big concern in virtual hosting facilities. Well-connected servers, paying for bandwidth.

exploited into a botnet == not good!

But provider can offer protection:Enforce a symmetry on RX:TX packet counts.

See our HotNets paper (Kreibich et al) Interesting extension bits:

Extension written in python.Never drops packets.

Extension Case Study: Parallax

Managing storage in VM clusters. Storage service “owns” local disks, and the OSes

that manage them. Parallax virtualizes storage, fast snapshots, etc.

Initially proposed as HotOS’05 paper. Ongoing, with NetApp and Google.

Local disks as persistent caches.Candidate selection for migration.Support explicit snapshot request from OSs.Not using a big filer.*

Take-home message:

Pre-conclusion:

…now the fun new part...

Despite being a coarse-grained tool, VMMs are a great substrate for

low-level extensibility.

Full-system extensions.

VMM

Control VM

DD

DiskNet

ND

Protected VM

VN VD

I/O Taint

Taint Pagemap

1. Inbound pages are marked as tainted. Fine-grained taintDetails in extension, page-granularity bitmap in VMM.2. VM traps on access to a tainted page. Tainted pagesMarked not-present. Throw VM to emulation.

Qemu*

Protected VM

VN VD

3. VM runs in emulation, tracking tainted data. Qemumicrocode modified to reflect tainting across data movement.4. Taint markings are propagated to disk. Disk extensionmarks tainted data, and re-taints memory on read.

Full-system extensions.

VMM

Control VM

DD

DiskNet

ND

I/O Taint

Taint Pagemap

1. Inbound pages are marked as tainted. Fine-grained taintDetails in extension, page-granularity bitmap in VMM.2. VM traps on access to a tainted page. Tainted pagesMarked not-present. Throw VM to emulation.

Qemu*

3. VM runs in emulation, tracking tainted data. Qemumicrocode modified to reflect tainting across data movement.4. Taint markings are propagated to disk. Disk extensionmarks tainted data, and re-taints memory on read.

Protected VM

VN VD

Protected VM

VN VD

Despite being a coarse-grained tool, VMMs are a great substrate for

low-level extensibility.

Conclusion:

VMM is a good base for device extensions. With device tap, can write extensions in user-

space, in any OS, without messing with individual kernels.

Treating devices as services seems to be a good approach to managing I/O in clusters.

V2E/E2V is very cool.

Fin.