Adding Multitenant Storage

Capability to the illumos Kernel

Richard Elling and Garrett D’Amore

Presented at the SCALE 11x ConferenceFebruary 24 2012

• Why multitenancy?

• Survey of available technologies

• OS virtualization

• Multitenant storage services in illumos

Agenda

Presented at the SCALE 11x ConferenceFebruary 24 2012

Why Multitenancy?

Presented at the SCALE 11x ConferenceFebruary 24 2012

Kernel

Userland Process

Device Driver

Userland Process

Simple model: multitasking OS

Presented at the SCALE 11x ConferenceFebruary 24 2012

Strike up the band!

Presented at the SCALE 11x ConferenceFebruary 24 2012

Guest

Native Hypervisor Model

Kernel

Userland Process

Hypervisor Device Driver

Examples: VMware ESX, XenServer, KVM

Device Driver

Presented at the SCALE 11x ConferenceFebruary 24 2012

Java

Java Virtual Machines are Virtual Machines, too

JVM

Java Application

OS Device Driver

APIs

Presented at the SCALE 11x ConferenceFebruary 24 2012

Hosted Hypervisor Model

UserlandProcess

Examples: VMware Workstation, VirtualBox

Guest

Kernel

Userland Process

Hypervisor Device Driver

Device Driver

Kernel Device Driver

Presented at the SCALE 11x ConferenceFebruary 24 2012

HVMGuestOS

HostOS

QEMU

Device Driver

Userland Process

Device Driver

KVM

I/O Proxy

Presented at the SCALE 11x ConferenceFebruary 24 2012

DomU

Front-end driver

Device Driver

Userland Process

XenServer

Dom0

Back-end driver

Device Driver

Userland I/O Proxy

Presented at the SCALE 11x ConferenceFebruary 24 2012

Kernel

Userland Process

Device Driver

Userland Process

Return to simple model: multitasking OS

Presented at the SCALE 11x ConferenceFebruary 24 2012

Zone

KernelDevice Driver

Userland Process

Zone

Userland Process

Presented at the SCALE 11x ConferenceFebruary 24 2012

BSD Kernel

Jail

BSD Jails

Storage Device Drivers

IP AddressDirectorystructure

File system

Userland Process

NIC Driver

Presented at the SCALE 11x ConferenceFebruary 24 2012

Kernel

Zone

illumos OS

Storage Device Drivers NIC Driver

Classification Engine

TCP/IP

Virtual NICFile system

Userland Process

Presented at the SCALE 11x ConferenceFebruary 24 2012

The first thing we do, let’s kill all the lawyers

King Henry VIAct IV, Scene II

Presented at the SCALE 11x ConferenceFebruary 24 2012

The first thing we do, let’s kill all the lawyers

King Henry VIAct IV, Scene II

The first thing we do, let’s kill all the layers

UNIX Performance Gurus

Presented at the SCALE 11x ConferenceFebruary 24 2012

Zone

KernelDevice Driver

Userland Process

Zone

Userland Process

Presented at the SCALE 11x ConferenceFebruary 24 2012

Kernel

Storage Device Drivers NIC Driver

VirtualSwitch

Zone A

Zone B

TCP/IP TCP/IP

File systemBlock device Virtual NIC Virtual NIC

Transactional Object Store and Data Management Unit

NFS

File system

SMB

File system

NFSiSCSI

Presented at the SCALE 11x ConferenceFebruary 24 2012

Global state Per-zone state

NFS - rpc now zoned

SMB - in-kernel, multithreaded, zoned

iSCSI - per-zone block devices

➾

Presented at the SCALE 11x ConferenceFebruary 24 2012

Big shared caches Inefficient small caches

~ 100 MB/tenant > 1000 MB/tenant

Kernel

Zone

Adaptive Replacement Cache

NFS

Zone

ZoneiSCSI

NFS

SMBTraditionalHypervisor

VMNFS

kernel

VM

kernel

Samba

FS Cache

FS Cache

Presented at the SCALE 11x ConferenceFebruary 24 2012

Kernel

Deep access to devices

Storage Device Drivers NIC Driver

VirtualSwitch

Zone A

Zone B

TCP/IP TCP/IP

Awesome New

DeviceBlock device

Virtual NIC Virtual NIC

Transactional Object Store and Data Management Unit

New interface?

ANP

Block device

iSCSIiSCSI

Presented at the SCALE 11x ConferenceFebruary 24 2012

Zone

IP Multipath Group

Virtual Networking

TCP

NFS iSCSI

Zone

IP Multipath Group

TCP

NFS SMB

Physical NIC

Virtual Switch

Physical NIC

Virtual Switch

Virtual NICVirtual NIC Virtual NICVirtual NIC

Presented at the SCALE 11x ConferenceFebruary 24 2012

Kernel statistics kstats✦

Zone A

Resource capsNetwork usageI/O usage

Zone B

Resource capsNetwork usageI/O usage

All kstats

Presented at the SCALE 11x ConferenceFebruary 24 2012

All in due time?

Presented at the SCALE 11x ConferenceFebruary 24 2012

Tree AB

Domain B

Domain A

Domain C

Tenant A

Tenant B

Tenant C

SMB

SMB

NFS SMB✦

A

B

C

Presented at the SCALE 11x ConferenceFebruary 24 2012

• Efficient, multitenant storage in illumos

• Sit out of the traditional VM dance

• OS virtualization rulez!

Summary

Presented at the SCALE 11x ConferenceFebruary 24 2012

www.illumos.org

www.dey-sys.com

www.RichardElling.com

Richard.Elling@dey-sys.comGarrett.D’Amore@dey-sys.com

Adding Multitenant Storage

Capability to the illumos Kernel

Richard.Elling@dey-sys.comGarrett.D’Amore@dey-sys.com