Stepping Through the Lifecycle of a Service Offering with Cassandra
Chris McEniry, Igor von Nyssen | Sr. Systems Architect, Sr. Software Architect | Sony Network Entertainment
* aka Ops guy * Bs EECS MIT
* 15 years ops experience * Python, Perl, C, LISP,
Ruby, JS, some English * C* 0.6 * [email protected]
Chris “Mac” McEniry Igor von Nyssen * aka Dev guy * MS CS TU Chemnitz * MBA USC * 15 years dev experience * Java, Ruby, English,
German * C* 1.0 * [email protected]
* Preconception – Solution looking for a problem * In utero – Connect and talk about it * Infant – Initial Development * Kid – Production rollout for an insignificant item * Teenager – Start real maturing * Adult – Take over the world
What we’re talking about: Lifecycle
* Large entertainment ecommerce application * 100s of Millions of customers * Billions of transactions per year * Worldwide operations: 59 countries, 21 languages and dialects * Peaks above 10,000 tps
What does SNEI look like?
* Global start-up that has to comply with any three letter requirement know to man.
* Pressure to innovate
* Pressure to make money
What does SNEI feel like?
Pre-Conception
Recognize Concerns/Motivations
Concerns
In-Utero
What’s the Real Problem?
Our Problem
Speed of Light Problem
Need to provide a service that
− Provides user authen0ca0on − Is Always-‐On − Has low latency for every customer
Our Problem: Latency and Availability
Can I beat the speed of Light?
Infant
Find Allies
Be willing to be uncomfortable
Strategically decide who you’re not going to involve
The New RACI Chart
Role Responsible Avoided Circumvented Ignored You X Friend X Frenemy X X X
Child
Learn on a Small Scale
What’s your Minimally Viable Product?
Teenager
Spread the Word
Embody the Mantra
Adult
Adult
Where are we now?
THANK YOU
Slideshare
Stepping Through the Lifecycle of a Service Offering with Cassandra
Chris McEniry, Igor von Nyssen | Sr. Systems Architect, Sr. Software Architect | Sony Network Entertainment
* aka Ops guy * Bs EECS MIT
* 15 years ops experience * Python, Perl, C, LISP,
Ruby, JS, some English * C* 0.6 * [email protected]
Chris “Mac” McEniry Igor von Nyssen * aka Dev guy * MS CS TU Chemnitz * MBA USC * 15 years dev experience * Java, Ruby, English,
German * C* 1.0 * [email protected]
* Preconception – Solution looking for a problem * In utero – Connect and talk about it * Infant – Initial Development * Kid – Production rollout for an insignificant item * Teenager – Start real maturing * Adult – Take over the world
What we’re talking about now: Lifecycle
* Large entertainment ecommerce application * 100s of Millions of customers * 9 figure revenue per year * Billions of transactions per year * Worldwide operations: 59 countries, 21 languages and dialects * Peaks above 10,000 tps
What does SNEI look like?
* Global start-up that has to comply with any three letter requirement know to man.
* Pressure to innovate
* Pressure to make money
What does SNEI feel like?
Pre-Conception
Recognize Concerns/Motivations
* Curiosity * Personal Growth * Less pain * The Hype – Exploit it
Personal
Ops Concerns
* How do we keep it from blowing up? * How do we keep it cared for for its lifetime? * How do we keep from being a bottleneck? * How do we keep from being woken up at 4AM? * How do we keep up with “gimme!gimme!gimme?"
Ops Concerns
* Marketing features vs. real features − What is on the outside of the box? − What does actually work?
* C* is young − our understanding of it is young
* Does it solve our problem adequately? − POCPOCPOC − We analyzed lots of in-memory data grids
-‐ Some had marketing feature problems -‐ Some were too young -‐ Some were too expensive -‐ None of them addressed partitioning very well -‐ If for nothing else, the cold startup time is a killer
* Adoption − Can we implement C* in our organization?
Dev Concerns
* Dev – you're not running production. * Ops – you’re not writing the code. * Work with someone
− Who sees the same problem − Is excited about the technology − Agrees on the direc0on − Does not have an inherent conflict of interest
Can’t do it by yourself
In-Utero
What’s the Real Problem?
* Be ready to move once you find a suitable problem − anytime you have a window, you have to move
* Create the window
* But, be sure that you want it * Be sure it’s the right problem
You Have a Solution Looking for a Problem
Our Problem
SF viewed from the World
Need to provide a service that
− Provides user authen0ca0on − Is Always-‐On − Has low latency for every customer
Our Problem: Latency and Availability
Can I beat the speed of Light?
* Not a problem that can be solved by slapping a cache in front of existing technology
* Need distributed storage * Global distribution helps with both latency and disaster recovery
− Data replica0on and synchroniza0on are built-‐in − Par00on events are handled out of the box
* Continuous delivery is part of the C* philosophy
Why Does C* Fit Here?
Infant
Find Allies
* Good Ally – Someone who − Sees the same problem − Is excited about the same technology − Agrees on the direction
* Bad Ally – Someone who − Will have an inherent conflict of interest
Good Allies, Bad Allies
* Develop your elevator pitch for the problem * Proof it with your allies * Keep it simple
Latency, Latency, Latency
Coin the Mantra
Be willing to be uncomfortable
Strategically decide who you’re not going to involve
The New RACI Chart
Role Responsible Avoided Circumvented Ignored You X Friend X Frenemy X X X
* It's new technology, so you're outside your comfort zone * You're evangelizing, so the proof is on you * Account for different adoption speeds by different people * Stay positive
Remember: Shiny, But Unknown and Unproven
* Ops − Service
-‐ Central -‐ Embedded
− Support -‐ Formal -‐ Community -‐ On-your-own
− Schema definition ownership
Figure Out Your Models * Dev
− How many nodes? − How many copies? − How many data centers?
* No budget approvals * Easy to get started
Once you have an idea of what you're talking about, then start to ask
the better questions.
Cassandra Is Open Source – Use That Fact
Child
Learn on a Small Scale
What’s your Minimally Viable Product?
* operational tasks − Setup
-‐ roll everything in rpms -‐ there's a bit of a manual setup - cluster join/remove
− Backup − infrastructure changes (node additions, etc.)
-‐ there's a bit of a manual setup - cluster join/remove − Monitoring
-‐ jolokia to tie into existing systems, but have to re-invent all of the metric metadata -‐ Availability
− incident response -‐ training and see monitoring/visibility
− handling regular tasks -‐ nodetool repair -‐ add keyspaces -‐ updating keyspaces for infrastructure changes
* have a clear service description * end user support
− self-help community − email list (subscription), chat room
Minimum Viable (Go Live) Product
* Store some data − start with transient data − TTLs help here − If it’s lost, it’s an acceptable loss
* Select which client you will use − Hector, Astyanax, or Datastax
* Figure out how clients connect to the server * Understand how C* stores data
− Data model − Serializers − Comparators
* Free your mind − Learn to love primary-key-only indexing − Start de-normalizing − Give up on transactions – they are evil to begin with
Minimum Viable (Go Live) Product
Teenager
* Take daddy’s credit card * Get serious about other data classes * Start putting something that you need to preserve in there * Data life cycle management * Security concerns * Acknowledge gaps – where either the technology or you or your
implementation of the technology are just not ready * Revisit your assumptions
Teenager
Spread the Word
* Allies − Security − RDMS zeal^Wusers
* Problems − Adjacent problems – things that are close to what you’ve already
solved, but bigger and more complicated * Key to success
− It is still early enough to be their idea − But it is established enough that people take it seriously
Gather More
* More visibility à More battles * Continue to choose wisely
− Decide where not to focus? − What not to try? − What to be compliant with?
* Remember: Proof is on you − Other groups will closely watch your success or failure − Auditors will ask you the questions. There is no vendor certification (yet).
Teenage Angst
* If you really must do large multipart transactions, choose a different product.
* Or just don’t. Change your mental model − From: transactions and conflict prevention − To: conflict detection and remediation
* Read this: http://www.eaipatterns.com/ramblings/18_starbucks.html * Avoid sequences that attempt to centralize counting
Teenage Angst
* Search (DSE) * Solidify your backup * Solidify your deployment model * Solidify your support * Solidify your security
− Audit − Authentication and Authorization − Encryption
Growth Spurts
Embody the Mantra
Latency, Latency, Latency
* You can ask better questions. * You have to ask the better questions.
Now you have experience
Adult
Adult
Where are we now?
* In terms of − Data − Applicable context − Ability to adopt and adapt
* Less inertia * Moving faster * Less energy for the same distance : “More efficient”
Underlying Problem: Critical Mass
Summarize * Preconception – Explore it * In utero – Start small * Infant – Be comfortably uncomfortable * Kid – Grow * Teenager – Get help * Adult – Profit
THANK YOU
Top Related