Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data...
Transcript of Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data...
![Page 1: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/1.jpg)
Persistent storage for ContainersAnil Degwekar
![Page 2: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/2.jpg)
2 of 17
What are we talking about?
Containers have become popular – replacing many Physical / Virtual Machine use cases
But: The persistent storage problem for containers is still not fully solved
– Considered by many to be the #1 challenge for containers adoption in the Enterprise
![Page 3: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/3.jpg)
3 of 17
The problem
• As Containers move from one server to another, the associated storage also needs to move
• Easy to accomplish, if– Using cloud storage– Networked storage (NFS)
• Not so easy for block storage
![Page 4: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/4.jpg)
4 of 17
The Container mobility problem
Node 1 Node 2
Container
Storage Array
Container
Volume export
Docker volume
volume
Docker volume
![Page 5: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/5.jpg)
5 of 17
Docker volume plugins
• Docker has a large collection of volume plugins (drivers)
• Limitations– Plugin required on every node– Plugin options can vary quite a lot– Container movement across node is not seamless– Volume spec is somewhat preliminary– Volumes can get orphaned– No data management features (snapshots, etc.)
![Page 6: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/6.jpg)
6 of 17
Kubernetes persistent volumes
• Kubernetes also has a large collection of volume plugins (drivers)
• Persistent Volume plugins have limitations similar to Docker– But the volume spec is somewhat more advanced compared to Docker
• Spec differs considerably from Docker volume plugin spec
![Page 7: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/7.jpg)
7 of 17
Past attempts to solve these problems
• ClusterHQ (Flocker)
• Portworx
• Rex-Ray
• CSI
![Page 8: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/8.jpg)
8 of 17
DockerFlocker-Plugin
DockerFlocker-Plugin
Flocker-Control-Service
user
Container 1 Container 2
Storage Array
Flocker volume
Flocker volume
Flocker-Agent Flocker-AgentArray - plugin Array - plugin
Compute node1 Compute node2
ClusterHQ (Flocker) Architecture
![Page 9: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/9.jpg)
9 of 17
Portworx architecture
Portworx “Software Defined Storage” Layer
Off-premStorage (Cloud)
Driver for Cloud Driver for Storage Array
On-premStorage (Array)
Docker Kubernetes
• Allows Container volumes to span arrays
• A single array volume can be split into multiple container volumes
• Supports additional services– HA– Snapshots– Encryption– Etc.
Mesosphere
![Page 10: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/10.jpg)
10 of 17
Rex-Ray overview
Rex-RAY framework
Storage driver for Array 1
Storage driver for Array 2
Storage driver for Array 3
Docker Kubernetes
Volume plugin for Docker Volume plugin for Kubernetes
• Common framework for all Container orchestrators
• Runs as a container in Docker
• Open source
• Multiple deployment modes– Standalone– Agent and Controller
![Page 11: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/11.jpg)
11 of 17
Container Storage Interface (CSI)
• Interface between Orchestrators and Storage Plugins
• Promise– Write a plugin once, and use it with any Container
Orchestrator
• Managed by CNCF
• But: spec is at a preliminary stage
![Page 12: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/12.jpg)
12 of 17
Timeline
2013 2014 2015 2016 2017 2018
First release of Flocker
ClusterHQclosed
First release of
CSI
First release of
Docker
First release of
Kubernetes
First release of Portworx
First release of Rex-Ray
![Page 13: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/13.jpg)
13 of 17
What more is needed
• Advanced data management features (snapshots, clones)
• Data reduction features (de-duplication, compression)
• Encryption
Many stateful applications need these services to migrate to Containers
![Page 14: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/14.jpg)
14 of 17
What are we doing in this space
• EMC had a partnership with ClusterHQ
• Rex-Ray project was open source - part of {code} sponsored by Dell EMC
• Dell EMC is a major contributor to CSI
• Volume plugin available for ScaleIO
• CoprHD has a container solution – open source
![Page 15: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/15.jpg)
15 of 17
Call for action
• Storage vendors– Keep the container story in mind when developing your solutions– Participate in CNCF and CSI
• Standards bodies– Need to come up with some common standards in this space
• Application developers– Be aware of this issue– If your application uses Block storage– And you want to migrate it to Containers
![Page 16: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/16.jpg)
Q & A
![Page 17: Persistent storage for Containers - SNIA · –Plugin options can vary quite a lot ... –No data management features (snapshots, etc.) 6 of 17 Kubernetes persistent volumes •Kubernetes](https://reader036.fdocuments.in/reader036/viewer/2022062916/5ec989ddb83f5f77ec2d4582/html5/thumbnails/17.jpg)