C* Summit 2013: Dude, Where's My Tweet? Taming the Twitter Firehose by Andrew Noonan
-
Upload
planet-cassandra -
Category
Technology
-
view
972 -
download
0
description
Transcript of C* Summit 2013: Dude, Where's My Tweet? Taming the Twitter Firehose by Andrew Noonan
#Cassandra2013
Dude, Where’s My Tweet? Taming the Twitter Firehose
Andrew Noonan Software Engineer at Gnip
@noonanisms
#Cassandra2013
Gnip
Cassandra
Rainbows
Unicorns
???
#Cassandra2013
#Cassandra2013
Social Data
#Cassandra2013
90% of Fortune 500
120 Billion Activities Delivered Per Month
#Cassandra2013
Lots-O-Data
Redundancy & Reliability
Availability
#Cassandra2013
#Cassandra2013
High Write Throughput ✔
Scalable ✔
Highly Available ✔
Persistent ✔
#Cassandra2013
Right?
#Cassandra2013
Not Exactly…
#Cassandra2013
No Maintenance? Bad Idea
Begin Maintenance -> 2X Data Growth
Scalable, Right?
Bootstrap Failures Due To Cluster Load
#Cassandra2013
Reconsider (Life) Choices?
#Cassandra2013
Size Tiered Compaction vs Leveled Compaction
How Much Data To Store Per Node
Your Write Pattern Matters Too
#Cassandra2013
compaction_throughput_mb_per_sec
16-32X write rate?
Lots-o-options – explore them
#Cassandra2013
Lookup by Tweet ID
Read Rate < Write Rate
Dynamic ColumnFamilies
#Cassandra2013
For realz this time!?
#Cassandra2013
#Cassandra2013
Bloom Filter False Positive Rate
Index Intervals
Only Change Schema On One Node! (For Now)
#Cassandra2013
You Won’t Always Fit The Mold and That’s Okay
Explore Your Options No Matter What
Understand The Consequences Of Your Choices
Staging Environment Identical To Production
#Cassandra2013
www.gnip.com
@noonanisms