High Performance Pooled Block Storage with Cinder...High Performance Pooled Block Storage with...
Transcript of High Performance Pooled Block Storage with Cinder...High Performance Pooled Block Storage with...
High Performance Pooled Block Storage with Cinder
Maciej Szwed
Tushar Gohad
SPDK, PMDK & Vtune™ Amplifier Summit
Agenda
• SPDK orchestration?
• OpenStack
• OpenStack NVMe-oF initiative
• Cinder and SPDK
• Usage
• Future plans
2
SPDK, PMDK & Vtune™ Amplifier Summit 3
Configuration file, JSON RPC calls, SPDK CLI
SPDK orchestration?Focused on building from blocks rather than complete solutions
Orchestration, Cloud Infrastructure?
SPDK, PMDK & Vtune™ Amplifier Summit 4
• OpenStack and K8S the top contenders
• Start with OpenStack -
• API driven access to compute, network and storage resources
• Support for largest number of storage and networking systems
• Flexibility to deploy bare metal, VMs and containers
• Continually evolving
open Cloud infra and spdk
SPDK, PMDK & Vtune™ Amplifier Summit 5
openstack
Open source software for creating private and public clouds
OpenStackClient HorizonCommand-line client web-based client
CinderSoftware Defined
Block Storage Service
NovaCompute Instances
Provisioner
NeutronNetwork
Connectivity Provisioner
SPDK, PMDK & Vtune™ Amplifier Summit 7
• 2016: SPDK Team demo’ed Cinder/Nova NVMe-oF Volume Driver POC
• 2017: Joint project started with Mirantis, Mellanox, OpenStack community to add NVMe-oF base infrastructure to Cinder
• 2017-18: SPDK Logical Volume support (Cinder baseline requirement)
• 2018: OpenStack Rocky – Kernel NVMe-oF Target available
• 2019: Openstack Stein – SPDK volume and target drivers available
OpenStack NVMe-oF initiative
2016 2017 2018 2019
Barcelona Summit
NVMeT driverimplementation
started
Vancouver Summit
Cinder NVMeT driver merged
SPDK Cinder drivers merged
SPDK Logical Volumes
SPDK, PMDK & Vtune™ Amplifier Summit 6
Cinder and SPDKHorizon
CinderSPDKSPDK bdev SPDK Volume
Driver
SPDK NVMe-oF Driver
Logical Volumes
NVMe-oF Target
Nova
VM
Cinder/Nova Control PathSPDK JSON RPC PathData Path
SPDK, PMDK & Vtune™ Amplifier Summit 7
• Create/extend/copy/delete volume
• Create/delete snapshot
• Create volume from snapshot
• Attach/detach volume
• Create image from volume
• Create volume from image
Cinder and SPDKSupport for all common operations:
SPDK Cinder volume driver
SPDK Logical Volumes
SPDK NVMe-oF Target
SPDK Cinder target driver
SPDK, PMDK & Vtune™ Amplifier Summit 9
openstack and SPDK NVMe-of
• OpenStack with fast storage
• Easily deployable infrastructure with SPDK
• Automatic resource tracking
• SPDK as a complete cloud solution
SPDK, PMDK & Vtune™ Amplifier Summit 10
Usage – Cinder configuration file
[DEFAULT]
...
enabled_backends = spdk
...
[spdk]
spdk_rpc_ip = 192.168.0.3
spdk_rpc_port = 8000
spdk_rpc_username = user
spdk_rpc_password = password
target_ip_address = 192.168.0.2
target_port = 4260
target_protocol = nvmet_rdma
target_helper = spdk-nvmeof
target_prefix = nqn.2014-08.org.spdk
volume_driver = cinder.volume.drivers.spdk.SPDKDriver
volume_backend_name = SPDK
SPDK, PMDK & Vtune™ Amplifier Summit 11
Usage – Openstack cli
DEMO
SPDK, PMDK & Vtune™ Amplifier Summit 12
Usage – openstack horizon
SPDK, PMDK & Vtune™ Amplifier Summit 13
Usage – openstack horizon
SPDK, PMDK & Vtune™ Amplifier Summit 14
Usage – openstack horizon
SPDK, PMDK & Vtune™ Amplifier Summit 15
Usage – openstack horizon
SPDK, PMDK & Vtune™ Amplifier Summit 16
• Cinder support for SPDK Logical Volumes on local NVMe and NVMe-over-TCP
• Kubernetes CSI plugin for NVMe-oF with SPDK support
• Nova support for SPDK vhost accelerated storage backends
Future plans
SPDK, PMDK & Vtune™ Amplifier Summit 18
Backup
SPDK, PMDK & Vtune™ Amplifier Summit 19
Usage – JSON RPC
# construct_lvol_bdev -l lvs lvol0 1024
# nvmf_create_transport -t rdma
nvmf_create_transport -t rdma# nvmf_subsystem_create nqn.2016-06.io.spdk:cnode1 -a
# nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 lvs/lvol0
# nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t rdma -a 192.168.0.2 -s 4260
SPDK, PMDK & Vtune™ Amplifier Summit 20
Usage – Openstack cli
# openstack volume create --size 1 volume1
# openstack server create --flavor m1.small --image cirros-x86_64-disk
# nova volume-attach <server-ID> <volume-ID>