STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data...
Transcript of STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data...
![Page 1: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/1.jpg)
STORAGE FOR DATA PLATFORMS
Federico Lucifredi & Kyle Bader
OpenStack Summit, Vancouver 2018
![Page 2: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/2.jpg)
INSERT DESIGNATOR, IF NEEDED2
COMMON ARCHITECTURAL MODEL -PUBLIC OR PRIVATE CLOUD
PUBLIC CLOUD (AWS) PRIVATE CLOUD
AWS EC2 PROVISIONING
OPENSTACKPROVISIONING
AWS S3SHARED DATASETS
CEPH S3/SWIFTSHARED DATASETS
Hadoop
Presto
Spark Hadoop
Presto
Spark
![Page 3: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/3.jpg)
THREE PILLARS FOR DATA PLATFORMS
Object Persistent Block
Low latency local block
![Page 4: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/4.jpg)
OBJECT
● Shared persistence layer across compute clusters
● Extreme scalability○ Capacity○ Throughput
● Economical
![Page 5: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/5.jpg)
OBJECT WORKLOADS
![Page 6: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/6.jpg)
PERSISTENT BLOCK
● Arbitrary block capacity
● Capacity based IOPS
● Persistence helps with management of database lifecycles
○ Detach and reattach on larger instance○ Snapshots for backups and copy-on-write for secondaries
![Page 7: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/7.jpg)
PERSISTENT BLOCK WORKLOADS
![Page 8: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/8.jpg)
LOW LATENCY LOCAL BLOCK
● For distributed applications that tolerate instance failures
● Big data workloads○ Scratch data○ Intermediate spill data○ Buffers
![Page 9: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/9.jpg)
LOW LATENCY LOCAL BLOCK
![Page 10: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/10.jpg)
INSERT DESIGNATOR, IF NEEDED10
COMMON ARCHITECTURAL MODEL -PUBLIC OR PRIVATE CLOUD
PUBLIC CLOUD (AWS) PRIVATE CLOUD
AWS EC2 PROVISIONING
OPENSTACKPROVISIONING
AWS S3SHARED DATASETS
CEPH S3/SWIFTSHARED DATASETS
Hadoop
Presto
Spark Hadoop
Presto
Spark
![Page 11: STORAGE FOR DATA PLATFORMS - Red Hatpeople.redhat.com/~flucifre/talks/Storage for Data Platforms.pdf · STORAGE FOR DATA PLATFORMS Federico Lucifredi & Kyle Bader OpenStack Summit,](https://reader033.fdocuments.in/reader033/viewer/2022050406/5f832dbbc4e5564af852c27d/html5/thumbnails/11.jpg)
THANK YOU!
COME TO SEE US AT THE RED HAT BOOTH