Post on 21-Jan-2018
Storage as microservice (a.k.a OpenEBS)
Built using Docker
Docker Chennai Meetup
@openebs
#containerizedstorageforcontainers
3 June 2017
Let us revisit containers
Image source : https://newsroom.netapp.com/blogs/containers-vs-vms/
VM
Container
Docker -> Portability
Docker registry → DevOps bridging
Storage controllers software version control
Monolithic Storage Kernel
iSCSI Software
A
ReplicationSnapshot
EncryptionB
QoS&
MgmtC
volume1
volume2
Storage software is same for all
volumes
Should we containerize storage ?
Version control of application software is made easy
through Docker images
How about the storage software ?
When storage is containerized
Docker Hub
Storage can also be a micro service
Containerized Storage Kernel
(OpenEBS)
volume1
Storage ContainerVSM 1
SoftwareA, B, C
volume2
Storage ContainerVSM 2
SoftwareA’, B’, C
Dockerized storage
https://hub.docker.com/r/openebs/jiva/
What is OpenEBS ?
➔ OpenEBS is containerized storage for containers
➔ OpenEBS solves the persistent storage issue for containers
Legacy
storage
X
Application
Pod
★ Application pod management
automatically takes care of storage
volume management
Dedicated mode of deployment
Separate hosts
Network
Data
OpenEBS Maya
Hyperconverged mode of deployment
K8s master
K8s minions
VSM or Storage PodApplication Pod
TCMU/iSCSI
OpenEBS Maya Storage
Orchestrator
Application ( yaml )
What is Maya ?
Maya
Manifests express intent
K8S used with special care for apps requiring
persistence because they require brittle tight coupling.
Container 1 Container 2 Container 2
Hard wired connections via plug-in
stateless stateful
NAS SAN S3 NAS SAN S3
Without OpenEBS DevOps workflow is broken
Manifests express intent
stateless stateful
No changes to DevOps workflow even for containers requiring
persistence. Users manifest their intent and the storage and
storage controllers adjust automatically as needed.
OpenEBSPolicy engine
Remote Storage
Local Storage
OpenEBS Storage Hosts
OpenEBS VSMs
OpenEBS Storage ClusterEBS
APIs
Containers and underlying storage,
local on host or dedicated storage
pods OR remote S3 or EBS storage
all grouped into a storage cloud that
just works. Billing by AWS.
Manifests express intent
With OpenEBS the workflow works
DevOps operations with OpenEBS
OpenEBS got the storage ops off my back!
Because of OpenEBS we are rolling into
production.
How is OpenEBS built ?Storage containerization
Storage orchestration
Deep storage operations
Core Storage features
is built using
is integrated with
is built into (OpenEBS Maya)
is built into (OpenEBS)
Smart Capacity Management Volume Latency Monitoring and Auto-healingAdapt to varying workloads - Application Aware Storage (Zones)AWS EBS Like APIsStorage Analytics!
Enterprise HA Intelligent QoSDistributed Scaleout Smart Tiering for higher performance
OpenEBS - Word of mouth
Containerized storage for containers
Calico / Flannel owns networking for K8S. OpenEBS owns
storage.
The only open source solution built for this. CEPH is not built
to be containerized.
I run (use) OpenEBS just like I run (use) K8S (EBS).
OpenEBS got the storage ops off my back! Because of OpenEBS
we are rolling into production.
Persistent storage is still an issue today
https://clusterhq.com/2016/07/17/dockercon-ecosystem-clusterhq/
"Containers have moved
beyond ephemeral storage
to persistent storage to
hold data and protect your
applications."- 451
Research
Mesosphere Declares
‘Container 2.0,’ the Stateful
Era
Kubernetes Version 1.3
include additional
resources for supporting
stateful containers
Persistence Storage Types
https://portworx.com/stateful-enterprise-applications-docker/
Flocker
Convoy
Rex-ray
Torus
Infinit
PortWorx
OpenEBS*
Docker Plugin
K8s Plugin
Typical Use cases
Where does OpenEBS fit?
DevOps and containers are the futureWhich is already here, but unevenly distributed
Real computing needs persistenceBut - persistence for DevOps & containers is broken
Usecase : Reuse existing SAN in
your cloud to build container friendly storage
platform
Build container friendly storage platform
AWS
VMs
EBSDocker
Application Pod (Application has
container-native storage)
● The existing SAN storage (EBS or GCP PD or
any other block storage) in your cloud can be
integrated into K8s application pods by
OpenEBS
● Persistent storage need not be managed
separately in your cloud
● Your persistent storage is managed from within
your application yaml files
Usecase : Build AWS EBS or GCP PD type platform On-Premise
Deploy EBS style platform On-Premise (Dedicated)
Your data center
Bare metal servers
Docker engines (Compute)
Bare metal storage
servers
Docker engines (OpenEBS)
Application
pod
yaml
A single application yaml
file controls both the
compute and storage
needs
Deploy EBS style platform On-Premise (Hyperconverged)
Your data center
Bare metal servers
Docker engines (Compute+OpenEBS)
Application
pod
yaml
A single application yaml
file controls both the
compute and storage
needs
Comparing AWS EBS & OpenEBS
Feature AWS EBS OpenEBS
Network Block Storage as a
service
Yes No. This is a technology
that can be used to build a
service or platform like AWS
EBS
Technology form factor Unknown Open Source
High Availability grade Five nines of availability Five nines of availability
Availability zones Yes Yes
Point in time Snapshots to
S3
Yes Yes, to S3 or any S3
equivalent (locally hosted
Minio for example)
Connectivity in AWS EBS (Optimized for Cloud service)
Login into AWS Console
Create an EBS volume
Attach to an instance
Format and put it to use
by application
Connectivity in OpenEBS (Optimized for Container service)
Login into Kubernetes
Set up an application
yaml with OpenEBS as
the storage provider
Volume is created on
OpenEBS and mounted
on K8S minion
Format and put it to use
by application
kubectl create -f demo-mysql-iscsi.yaml
Snapshots on AWS EBS vs OpenEBS
Incremental
Snapshot
Restore
Amazon
Cloud
Enterprise
data center
Incremental
Snapshot
Restore
Amazon
CloudAny other S3 Cloud
storage provider
S3 storage
in the data center
Scality
Minio
OpenEBS
volumes
Apps in
Containers
Incremental
Snapshot
Restore
Incremental
Snapshot
Restore
Incremental
Snapshot
Restore
Other features comparison of AWS EBS & OpenEBS
Feature AWS EBS OpenEBS
Volume migration across tiers
(Magnetic, General Purpose
SSD, Provisioned IOPS)
Yes Yes, VSMs can be migrated
seamlessly across tiers.
QoS control Yes (Provisioned IOPS) Yes, granular control of IOPS
on a per volume basis
Burst IOPS, credit bucket for
unused IOPS
Yes Yes. Min/Max IOPS, burst
IOPS, credit bucket
functionality
Auto scaling of the storage
nodes
Yes (Not sure. Internal to EBS
architecture)
Yes. OpenEBS storage nodes
are added to clusters to allow
auto scaling of either capacity
or performance
OpenEBS Roadmap
0.1
0.4
1.0
0.3
0.2
● Soft launch / Basic version
● Containerized controller
● Longhorn integration basics
0.2 - Basic k8s integration
0.3 - Full k8s integration(Hyper-Converged)
0.4 - Tiering and distributed storage
1.0 - First enterprise edition(Full functionality for basic production usage)
K8s Provisioning via EBS-like driver
S3 building blocks
Complete longhorn integration
First usable release for community
consumption
Complete K8s integration
Fully hyper-converged
Support for local backing store
Building blocks of Q0S
Building blocks of Tiering
Tiering and QoS demonstrated
Building blocks of ML for storage analytics
DevOps operator use cases demonstrated
Tiering and QoS demonstrated
Building blocks of ML for storage analytics
DevOps operator use cases demonstrated
Nov, 2017
Sep, 2017
Jun, 2017
Mar, 2017
Jan, 2017
Take a peek at our github project and star us if you like
https://github.com/login/oauth/autho
rize?client_id=768d7617e686c9c2b
d81&scope=user%20public_repo&s
tate=openebs1234cloudbyte1234&
allow_signup=true
github.com/openebs
What do you think?
https://github.com/openebs
Join the community
#slackSlack.openebs.io
blog.openebs.io
@openebs