7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack...
42
7 STEPS to Roll-out a Private OpenStack Swift Cluster Joe Arnold April 19, 2012 OpenStack Conference
-
Upload
joe-arnold -
Category
Technology
-
view
5.879 -
download
0
description
The OpenStack Object Storage system, aka "Swift", was built to power Rackspace's public cloud storage service, Cloud Files. Subsequently, Swift was released into open source as part of the OpenStack project. While Swift is increasingly being utilized to run many large scale public storage clouds outside of Rackspace, Swift is also showing great promise for use in private storage clouds - which look and feel like the public object storage systems from Amazon/Rackspace, yet meeting the needs of the individual company. In this talk, I will present a step-by-step plan on getting OpenStack Swift up and running in your organization. I will cover: - when to use Swift (and when not to) - convincing your boss (the economics) - selecting appropriate hardware (benchmarking) - achieving high durability and availability (cluster design) - knowing how it's working (management & monitoring) - what interfaces exist (compatible clients) - how to build killer apps (features for developers) About Joe Arnold: Joe Arnold is the CEO of SwiftStack. Joe managed the first public OpenStack launch of Swift independent of Rackspace, and has subsequently deployed multiple large-scale cloud storage systems. He is currently building Swift deployment and management tools with his firm, SwiftStack. Prior to SwiftStack, Joe held leadership positions at Cloudscaling, Engine Yard and Yahoo!.
Transcript of 7 steps to roll out a private open stack swift cluster - joe arnold - swiftstack - OpenStack...
7 STEPS to Roll-out a Private OpenStack Swift Cluster
Joe Arnold
April 19, 2012OpenStack Conference
OpenStack SwiftPowers Rackspace Cloud Files
http://www.flickr.com/photos/josefgrunig/
Data is Sticky
App
Storage
Compute
App
Storage
Compute
App
Storage
Compute
Storage
Compute
’09 ’10
Rackspace Cloud Files
Openstack Launch
59 developers contributing to Swift
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
Swi ft is good for
- Application data
- Filesharing
- Backups
- Unstructured data
Swift is laser focusedSwift Attributes
Scalable
Swift Attributes
Durable
Swift Attributes
Concurrency
Not good at . . .Databases - transactional workloads
No File Locking
Not good at . . .Single Stream
I/O
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
Data Durability - Zones
Data Durability - Replication & Consistency
High Availability
Proxy
+
Zones
Example: 2 Nodes
- Whole Files- 3 Replicas- Withstand multiple disk failures
Example: 5 Nodes
- Withstand multiple Node Failures
- Consider Separate Proxy Tier
Example: 50 Nodes
- Withstand Rack Failures- Requires Separate Proxy Tier
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
Example: $42,520 for 105TB usable1 x Dell r815 ($4,500)2x Supermicro JBODs ($5,400)144 x 3TB hard drives ($33,120)Thailand flooding drive prices
Swi ft is economical
$0.40/GB Capex
Swi ft is economical
Size Configuration Cost $ / GB
145 TB
2 ToR Switch2 Proxy/Auth Servers5 Object Stores...and cables, rack, etc
$95k $0.65
1.3 PB
2 Agg Switches6 Proxy/Auth Servers5 ToR Switches50 Object Stores...and cables, racks, etc
$750k $0.52
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
High volume web /mobile application
Various
WorkloadsEnterprise Filesharing
Backup /Active Archive
Measure and Test
File Sizes ConcurrencyFrequency
of upload vs download
PROXY NODES
STORAGE NODES
RAM - 12GBNetworking - 2 x 10GbE
CPU - quad-core
RAM - 1GB of RAM / 2TB
Drives - 2-3 TB Drives
Controller Cards - No RAID
Using SSDs - User Accounts & Container Listings
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
People are Expensive
Time-Series BackendGanglia / Graphite
StatsD
LightweightUDP StatsD
Swift ProcessSwift ProcessSwift ProcessSwift ProcessSwift ProcessSwift Process
Added StatsD to Watch EVERYTHING
Operating System + Kernel
OS, Node & Drive Monitoring
Swift Runtime Monitoring
OpenStack Object Storage
SSL Termination Authentication
Load Balancing
Integrating Components around Swift
Controller
Managing a distributed system
Capa
city
time
Capacity adjustments
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
S3Cloud Files
’06 ’08’10
When to use Swift (and when not to)
Achieving High Durability and Availability (cluster design)
Convincing Your Boss (the economics)
Selecting Appropriate Hardware (benchmarking)
Knowing How It’s Working (management & monitoring)
What Interfaces Exist (compatible clients)
How to Build Killer Apps (features for developers)
Data Storage
HTTP API
Media Delivery
Direct to Device
Integrates to CDN
Streaming
Static Documents
Direct from Device
TRY http://swiftstack.com
@joearnold - @swiftstack
Swi ft workshop
tomorrow at
1 : 30 - 4 : 00
๏Install๏Build an Application
