[Cinder] Support LVM on a shared LU · /dev/sdx (LU1) VG1LV1 1-3. Work flow of volume creation and...

0

[Cinder] Support LVM on a shared LU

Mitsuhiro Tanino <[email protected]>

July/2/2014

1-1. Big Picture

1

FC Storage LU1

Storage Controller

qemu-kvm

Nova Compute1 Nova Compute2

VM1 VM2 VM3 VM5 VM6 API

Scheduler

Cinder node

VM4

Proposed LVM volume Driver Attach/Detach via qemu-kvm

qemu-kvm

Attach/Detach via qemu-kvm

Control Node Compute Nodes

VG1 LV1 LV2 LV3 LV4

/dev/sdx (LU1)

VG1 LV1 LV2 LV3 LV4

/dev/sdx (LU1)

VG1 LV1 LV2 LV3 LV4

/dev/sdx (LU1)

VG1 LV1 LV2 LV3 LV4

Volume operations

I/Os from VMs on compute nodes FC Storage

• Volume operations are implemented

by using LVM, which is light weight.

•I/Os are issued to a storage controller,

which has enough scalability.

Proposed fix Proposed fix

1-2. Preparation by administrator

2

FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4

Proposed LVM volume Driver qemu-kvm


/dev/sdx (LU1)

/dev/sdx (LU1)

/dev/sdx (LU1)

(1) Set up FC connectivity between storage,

control and compute nodes.

(2) Create LU1 on FC storage.

(3) Attach the LU1 to control and compute

nodes as a shared LU outside of

OpenStack. After (3), LU1 appears in all

nodes as a /dev/sdx.


1-2. Preparation by administrator

3

FC Storage

FC Storage

LU1

Storage Controller

qemu-kvm



Scheduler

Cinder node

VM4

Proposed LVM volume Driver qemu-kvm


VG1

/dev/sdx (LU1)

VG1

/dev/sdx (LU1)

VG1

/dev/sdx (LU1)

VG1

(4) Create VG1 on top of LU1 at control node.

(5) Scan LVM metadata on LU1 on each node.

As a result, VG1 appears in all nodes.

(6) Edit cinder.conf and use VG1 as a

“volume_group” for proposed LVM driver.


4

qemu-kvm



Scheduler

Cinder node

VM4

Proposed LVM volume Driver

Create/Delete Extend/etc

qemu-kvm


/dev/sdx (LU1)

VG1

/dev/sdx (LU1)

VG1

/dev/sdx (LU1)

VG1

1-3. Work flow of volume creation and attachment

FC Storage

FC Storage

LU1

Storage Controller

VG1

(1) Create LV1 on the top of VG1 at control

node.

LV1


5

qemu-kvm



Scheduler

Cinder node

VM4



qemu-kvm


/dev/sdx (LU1)

VG1 LV1

/dev/sdx (LU1)

VG1 LV1

/dev/sdx (LU1)

VG1

FC Storage

FC Storage

LU1

Storage Controller

VG1

(2) LV1 is written to the shared LU.

(3) Before volume attachment, scan LVM

metadata on LU1 at compute1.

As a result, LV1 appears in compute1.


LV1


6

qemu-kvm



Scheduler

Cinder node

VM4




qemu-kvm



/dev/sdx (LU1)

VG1 LV1

/dev/sdx (LU1)

VG1

/dev/sdx (LU1)

VG1

FC Storage

FC Storage

LU1

Storage Controller

VG1 LV1

(4) Activate LV1 at compute1.

=> After activation, access device path

/dev/VG1/LV1 is created at compute1.

(5) Attache LV1 to VM1.

=> Libvirt and qemu can handle device path

and attach a volume using this.


LV1


7

Cinder Plugins

Software-based

File system based

DFS

NFS

Block based

Hardware-based

Fibre Channel

iSCSI

NFS

GlusterFS, GPFS,

RBD(ceph)

There are many kinds of Cinder plugins available.

LVMiSCSI

Vendor specific

drivers, such as,

Dell, EMC, HP,

Hitachi and etc …


2. Target of Proposed LVM volume driver

3. Benefit of Shared LVM driver

8

Reduce hardware based storage workload by offloading the

workload to software based volume operation.

In OpenStack cloud environment, workloads of storages have been

increasing than usual and it is difficult to manage the workloads

because every user have a permission to execute storage

operations via cinder.

In order to use expensive storage more efficiently, I think it is better

to reduce hardware based storage workload by offloading the

workload to software based volume operation on a case by case

basis.

Using these two drivers to one enterprise storage, cloud provider

can provide volumes both way as the situation demands. For

example,

As for "Standard" type storage, use proposed software based

SharedLVM driver.

As for "High performance" type storage, use hardware based

cinder driver. (Ex. Higher charge than "Standard" volume)

3. Benefit of Shared LVM driver

9

Enable cinder to any kinds of shared storage volumes without

specific cinder storage vendor driver. User can apply this driver to a storage which does not have specific

cinder driver or does not have features such as efficient snapshot,

backup.

Provide quicker volume creation and snapshot creation without

storage workloads. LVM has lightweight volume creation. And also lightweight snapshot

using copy on write.

Better I/O performance using direct volume access via Fibre

channel. This driver can issue I/O without scsi target, therefore performance

is better compared to LVMiSCSI driver.

I/O bandwidth and latency are almost same as raw FC volume.(See

test result at P13 and P14)

4. Comparison of Proposed LVM volume driver

10

LVMiSCSI Proposed

LVM

FC

(Vendor Driver)

Implementation

of volume LV (managed by LVM)

LV (managed by LVM)

LU (managed by storage)

Volume

Operation By software (LVM)

By software (LVM)

By hardware (Storage)

Work Load Server side Server side Storage side

Supported

Storage Any storage (Storage independent)

Any storage (Storage independent)

Specific storage (Requires specific

plugin)

Volume Access

Path

Via software

iSCSI target

Direct from

fibre channel

Direct from

fibre channel

Performance Better I/O

performance Better I/O

performance

HA Active/Passive：〇

Active/Active ：ー Active/Passive：〇

Active/Active ：ー Active/Passive：〇

Active/Active ：〇

Less volume

operations

to storage.

Better

support

coverage.

Better I/O

performance.


A/A is not

supported.


11

LVMiSCSI Proposed

LVM

FC

(Vendor Driver)

Admin

work

- Initial

setup

5 steps

(a) Setup network

connectivity for all nodes.

(b) OS and OpenStack

setup

(c) Create a storage

volume and attach it to

cinder node.

(d) Create volume group

on top of storage volume

(e) Configure cinder.conf

5 steps

(a) Setup network

connectivity for all nodes.


setup

(c) Create a storage

volume and attach it to

cinder node.

(d) Create volume group

on top of storage volume

(e) Configure cinder.conf

4 steps

(a) Set up FC connectivity

between storage, cinder

node and compute nodes.


setup

(c) Create storage pool in

the storage.

(d) Configure cinder.conf


Volume group creation

step is required.


12

LVMiSCSI Proposed

LVM

FC

(Vendor Driver)

Admin

work -

add new

server

2 steps

(a) Setup network

connectivity for new

node.


setup

3 steps


between storage and new

compute node

(b) Set up an access

control to a Shared LU

from new node.

(c) OS and OpenStack

setup

2 steps


between storage and new

compute node.


setup


Access control(FCZone or

Shared LU) is required

13

Bandwidth of I/O [O_DIRECT] Unit: [KB/s]

I/O size LVMiSCSI SharedLVM Raw FC volume

Read 1M 49788.0 85146.0 88203.0

512K 62382.0 160517.0 157810.0

4K 4026.4 8630.8 8865.2

Write 1M 76691.0 141315.0 144173.0

512K 59200.0 142408.0 144006.0

4K 3870.1 7738.9 7867.1

Randread 1M 16152.0 17665.0 17105.0

512K 13677.0 20397.0 19971.0

4K 417.3 480.6 476.8

Randwrite 1M 15606.0 17067.0 16526.0

512K 13666.0 20381.0 19955.0

4K 417.1 4803.3 476.5

I/O performance

of SharedLVM

and Raw FC

volume are

almost same.

*Larger is good.

5. Comparison of I/O performance

- Following results are I/O performance of issuing direct I/O from single

instance to single volume.

14

Average latency

of SharedLVM

and Raw FC

volume are

almost same.

Average latency of I/O [O_DIRECT] Unit: [msec]

I/O size LVMiSCSI SharedLVM Raw FC volume

Read 1M 20.56 12.02 11.60

512K 8.20 3.18 3.24

4K 0.99 0.46 0.45

Write 1M 13.35 7.24 7.10

512K 8.64 3.59 3.55

4K 1.03 0.51 0.50

Randread 1M 50.47 52.31 54.22

512K 28.75 21.58 22.13

4K 8.38 7.76 7.84

Randwrite 1M 13.37 5.84 5.83

512K 8.68 3.51 3.50

4K 1.42 0.75 0.73 *Smaller is good.

5. Comparison of I/O performance

[Cinder] Support LVM on a shared LU · /dev/sdx (LU1) VG1LV1 1-3. Work flow of volume creation and...

Documents

Transcript of [Cinder] Support LVM on a shared LU · /dev/sdx (LU1) VG1LV1 1-3. Work flow of volume creation and...