Efficient Image Management for OpenStack using Block ...for OpenStack using Block Storage Features...

© Hitachi Data Systems Corporation 2015. All rights reserved.

Efficient Image Management

for OpenStack using

Block Storage Features

Tomoki Sekiyama

1


Background

New Features for Efficient Image Handling

Use-case of Volume-backed Images

Usage of Volume-backed Images

Future Work

Wrap-up

2

Contents


Background

3


OpenStack Components and Features

OpenStack cloud consists of many components:

Cinder Glance

Nova Ironic

4

Storage Array

VM VM

VM VM

VM VM

VM VM

Compute Nodes

•Compute: Manage VMs

•Volume: Provides block storages

service

•Manage images of OS & data for

VMs and baremetal nodes

•Support various image stores:

Local filesystem, Ceph, Swift, volumes, …

•Manage baremetal nodes

Baremetal nodes

Volume

Volume

Volume

Image Image Image


Example Target Environment

Each services are connected via network (LAN, SAN)

• Cinder + Storage array

Cinder

Glance

Nova-api

Ironic

VM

* Currently baremetal does not

support Cinder volumes

5

Baremetal nodes

VM

VM VM

VM VM

VM VM

Storage Array

Compute Nodes

Control Node

SAN (FC/iSCSI)

etc.

Nova-compute

Nova-compute

Nova-compute

LAN


HDD

Problems (1/2)

Agility is an important factor for OpenStack clouds

• Required to rapidly boot instances

When the image size is large, “nova boot” takes a long time

to download the image from Glance

• Nova caches images (per host), but not effective to first boot

• Cinder-volume-boot always downloads the image by default

6

Download Image Cached

Image

Guest

Image

VM

Volume VM

Image

Compute

Node

Storage

Array

(Image cache is optional) Control Node


Image copy may affect disk I/O performance of instances

• Example: sysbench OLTP benchmark during volume creation

– Measured on a KVM instance with a volume on LVM-iSCSI backend

– During the image copy to a volume, I/O performance degrades

– The interference can be mitigated by image copy I/O bandwidth limit, but

image copy takes longer time

Volume creation from a snapshot is rapid, but snapshots cannot be

shared among projects (tenants)

• Not suitable for public images (e.g. operating systems)

Problems (2/2)

7


(3) Storage

virtualization

and automatic data

optimization using

thin provision and

tiering

Volume

Features of Block Storage Arrays

8

VM

(4) Taking backup and

snapshot instantly via

storage feature

(2) Provide stable

IOPS and low

latency using boot

from volume

Volume

(1) Create a boot volume

using COW snapshot

SATA

SAS

Flash

Stable Performance

Virtualization/ Optimization

Business Continuity

Agility of booting an instance


New Features for Efficient Image Handling

9


Cinder New features for Image Handling In Liberty release, some features are added to Cinder for efficient

image management

• Image-Volume Cache

– Cache recently used images as “image volumes”

• Volume-backed Image

– Store an Glance image data in a Cinder volume

In both features, a new bootable volume can be created rapidly by

cloning the image volume

Both features are disabled by default

10


Internal project

Overview of Image-Volume Cache

Images recently used by Cinder are cached as Cinder volumes

• Each image is stored in a volume (Image-Volume)

• Image volumes are placed in the internal project

Automatic management

• If cache volumes exceed the specific amount size, recently

unused cache volumes are deleted

Support various disk formats (converted to raw before cached)

Can coexist with the volume-backed Image feature

11

Download

(First time only)

Image Vol.

(Cache)

Image

Volume VM

Clone Boot Storage

Array


Overview of Volume-backed Image

Register an image volume as a Glance image

• Utilize Glance’s Cinder store

– The image data is stored in a Cinder volume (Image-Volume)

• No image data transfer between Cinder and Glance

New volume can be created rapidly by cloning the image volume

12

Image

Volume

Volume-

backed

Image

Volume VM

Clone Boot

location = cinder://1234-abc..

Storage

Array


Use-case of Volume-backed Images

13


Rapid Boot for Virtual Machine Instances

Rapidly boot an instance with a new volume from an image

• Reduce time to launch instances

• Reduce I/O workload

Significant on booting multiple instances

Example:

• Booting an instance from a new Cinder volume

created from 20 GB operating system image

– Measured with thin-provisioning LVM backend

14

0 50 100 150 200 250

default

Volume- backed

Setup instance Volume creation Volume attach Spawning

[s]

246s

11s


Copy-offload of Image Data

Leverage the storage array’s copy-offloading features

• Some storages support copy-on-write based cloning

– No data transfer

– No interference to instances’ performance

Example: sysbench OLTP benchmark during volume creation

• When thin-provisioning is supported, this feature also improve

storage capacity

15


Project C

(app dev)

Sharing volume data among projects

Sharing volume data among projects

• The visibility of the Volume-backed image is managed by

Glance’s ACL feature

– Public image volume

– Sharing among specific members

• Useful to share base images such as operating system, common

application images, master datasets, etc.

16

Master

image

Project A Project B

Guest 2 Guest 4

Guest 1 Guest 3

Public

Volume-

backed

Image

*The image must be

registered by volume owner


Usage of Volume-backed Images

17


Cinder-store for Glance Enables Glance to store the image as a Cinder volume

• Until Kilo release, it can only “point” the volume, but read/write

access to the image data was not provided.

– Image volumes were only to create new volumes inside Cinder

• In Mitaka release, it supports read/write access experimentally.

– The image volume is attached to the Glance node using “os-brick” library

– Glance transfers the data to its clients (e.g. Nova, OpenStack CLI)

18

Image

Volume

Volume

backed

Image Attach

Volume


Enable Volume-backed Image Feature (1/2)

Glance settings (/etc/glance/glance-api.conf)

• Enable Cinder store

– [glance_store] :: stores = file,http,swift,cinder

• Expose image locations (URL)

– [DEFAULT] :: show_multiple_locations = True

Cinder settings (/etc/cinder/cinder.conf)

• Enable Glance API version 2

– [DEFAULT] :: glance_api_version = 2

• Enable volume creation by cloning image volumes

– [DEFAULT] :: allowed_direct_url_schemes = cinder

• (optional) To use “cinder upload-to-image” to create volume-backed image

– Backend section :: image_upload_use_cinder_backend = True

19


Enable Volume-backed Image Feature (2/2)

By default, image volumes are stored in current user’s project

• Only accessible from the user → cannot be shared among projects

• Visible to users → users may delete them by mistake

– All image volumes should be stored in the internal service project to avoid

these issues

Glance settings (/etc/glance/glance-api.conf)

• Specify the internal project to store image volumes

– [glance_store] :: cinder_store_project_name = service

– [glance_store] :: cinder_store_user_name = glance URL of “keystone” (auth service)

– [glance_store] :: cinder_store_password = ********

– [glance_store] :: cinder_store_auth_address = http://1.2.3.4/identity/v2.0

Cinder settings (/etc/cinder/cinder.conf)

• To use “cinder upload-to-image” to store the image volume to internal project

– Backend section :: image_upload_use_internal_tenant = True

– [DEFAULT] :: cinder_internal_tenant_project_id = 0123456789abcdedf

– [DEFAULT] :: cinder_internal_tenant_user_id = abcdedf01234567890

20


Registration of Volume-backed Image (1/3) When “image_upload_use_cinder_backend = True”:

• openstack image create --volume <volume> <image-name> OR

• cinder upload-to-image <volume> <new-image-name>

• The specified volume is cloned to create a new image volume

• A new Glance image is created

• The cloned volume’s URL is registered to the image

• Only “raw” format is supported

– If other format (qcow2 etc.) is specified, image is converted and

transferred to Glance.

21

Image

Volume

New

Image

Volume

Clone

Register

location

Storage

Array


Registration of Volume-backed Image (2/3) Using Glance CLI (with Image API v1)

• glance image-create --disk-format raw ＼ --container-format bare --name <image-name> ＼ --file <path-to-image-file> --store cinder

• --store option is only available in Image API v1, which might be deprecated in

future releases

• In Image API v2, need to set cinder to default store …

• NOTE: This is only available in Mitaka release

22


Registration of Volume-backed Image (3/3) Manual registration of volumes (using Image API v2)

• Create the image volume(s) manually first – NOTE: the registered volume data must not be modified.

• glance image-create --disk-format raw ＼ --container-format bare --name <image-name>

• glance location-add <image-uuid> ＼ --url cinder://<volume-uuid>

• If Cinder has multiple backends, location-add should be repeated for each.

+------------------+-------------------------------------------------------------+ | Property | Value | +------------------+-------------------------------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2015-09-22T21:31:34Z | | disk_format | raw | | id | f698173d-b96f-43be-aae9-fa0d13751c09 | | locations | [{"url": "cinder://95e571f9-5ccd-45eb-b282-441c3ce9a5db", | | | "metadata": {}}] | … | size | 1073741824 | | visibility | private | +------------------+-------------------------------------------------------------+ 23


New Volume from Volume-backed Image

To create a new volume from a volume-backed Image:

• cinder create --image <image-uuid> <size>

• The new volume is cloned from the image volume

• The volume is extended when the volume size is larger than the

image volume

In Horizon, a new instance

can be launched from a new

volume cloned from the

volume-backed image

24


Current Limitation

Cannot clone volumes between multiple hosts / backends

• Need to create an image volume on each host / backend

• and register their locations to the volume-backed image

Volume-backed images must be in raw format

• Image-Volume Cache supports various disk formats

– Automatically converts to raw type

25


Future Work

26


Booting baremetal nodes requires to copy the image to local disk

• Download & copy takes a long time

• Causes high network traffic

• Ironic caches the downloaded images

same as Nova but image copy to

attached disk is always required.

Problems in Ironic boot time

27

HDD

1. Boot with

deploy image

2. Export the disk

as iSCSI target

Image

3. Download & Copy Image

4. Reboot

Attach

Baremetal

Node


Boot Volume Boot Volume

Ironic: Baremetal Volume-Boot

Specs for supporting volume-boot of baremetal nodes is proposed:

• Ironic-spec: https://review.openstack.org/#/c/200496/

• Nova-spec: https://review.openstack.org/#/c/211101/

Combining with volume-boot and cinder-backed images, the image

can be rapidly deployed to the baremetal node.

28

Image Volume

Boot Volume

iSCSI / FC

1. Clone

2. Attach & Boot

Baremetal Nodes

https://review.openstack.org/







Wrap-up

29


Wrap-up

Cinder volume-backed images are useful to:

• Rapid boot of volume-boot VM instances

• Rapid boot of baremetal instances in the future

– Ironic work for volume-boot is ongoing

• Share volume data (e.g. base OS image) among tenants

• Leverage the storage features for image management

30


Wrap-up

31

# Action Source Kilo Liberty Mitaka

1 Boot from volume Glance Image

2 Cinder snapshot

3 Image volume

4 Boot from image Glance Image

5 Image volume

6 Create image volume Cinder volume

7 Local file (upload) *1

8 Nova instance

image

*2

*1: It requires cinder-store to be set to default store, or usage of Image API v1 (may be deprecated in the future).

*2: It requires cinder-store to be set to default store.


Disclaimer

The OpenStack® Word Mark and OpenStack Logo are either registered

trademarks/service marks or trademarks/service marks of the OpenStack

Foundation in the United States and other countries and are used with the

OpenStack Foundation's permission. We are not affiliated with, endorsed or

sponsored by the OpenStack Foundation, or the OpenStack community.

Other company, product or service names may be trademarks or service mark

of others.

32


Backup slides

34


Enable Image-Volume Cache

Image-Volume Cache and Volume-backed image feature can

coexist

Cinder settings (/etc/cinder/cinder.conf) • Enable internal tenant

– [DEFAULT] :: cinder_internal_tenant_project_id = ...

– [DEFAULT] :: cinder_internal_tenant_user_id = ...

• Enable Image-Volume Cache

– Backend section :: image_volume_cache_enabled = True

• Limit max capacity / number of cache volumes

– Backend section :: image_volume_cache_max_size_gb = ...

and/or

– Backend section :: image_volume_cache_max_count = ...

35

Efficient Image Management for OpenStack using Block ...for OpenStack using Block Storage Features...

Documents

Transcript of Efficient Image Management for OpenStack using Block ...for OpenStack using Block Storage Features...