Desert Code Camp 2016.1 - Stateful Distributed Systems
-
Upload
joe-rawlings -
Category
Engineering
-
view
85 -
download
1
Transcript of Desert Code Camp 2016.1 - Stateful Distributed Systems
Statefulness in Distributed
Systems
Amazon is hiring!
To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: [email protected]
Our mission is to provide the most innovative, scalable, and reliable systems in the
world.
@JoeRawlings
https://goo.gl/O9PAUK
Outline
OutlineWhat’s that smell?
OutlineWhat’s that smell?
What’s the meaning of this?
OutlineWhat’s that smell?
What’s the meaning of this?
Pics or it didn’t happen.
OutlineWhat’s that smell?
What’s the meaning of this?
Pics or it didn’t happen.
Swallowing the red pill.
OutlineWhat’s that smell?
What’s the meaning of this?
Pics or it didn’t happen.
Swallowing the red pill.
Challenge accepted!
What’s that smell?
Stateful Services
DESIGN SMELL
Stateful Services
ANTI-PATTERNDESIGN SMELL
Stateful Services
Stateful Services
Stateful Services
WEB SERVER
TIER
CLIENT TIER
DATABASE TIER
WEB SERVER
TIER
CLIENT TIER
DATABASE TIER
WEB SERVER
TIER
CLIENT TIER
DATABASE TIER
WEB SERVER
TIER
CLIENT TIER
DATABASE TIER
What’s the meaning of this?
Stateful Services
Stateful ServicesYou’re already doing it!
Required to do something useful
Stateful ServicesYou’re already doing it!
Required to do something useful
Stateful Services
(most of the time)
You’re already doing it!
Required to do something useful
Stateful Services
(most of the time)
You’re already doing it!
stateful processing systems share similar concerns
Required to do something useful
Stateful Services
(most of the time)
You’re already doing it!
stateful processing systems share similar concerns
(This is where the fun is)
Stateful Processing
Stateful ProcessingData locality
data intensive systems
Stateful ProcessingData locality
data intensive systems
Stateful Processing
Strong Consistency
Data locality
data intensive systems
Stateful Processing
Strong Consistency
Data locality
High performance
Stateless Services
Stateless Services
does not care what has happened
Stateless Services
does not care what has happened
does not care what has changed
Stateful Processing
Needs to care what has happened
Stateful Processing
Needs to care what has happenedNeeds To care what has changed
Stateful Processing
Needs to care what has happenedNeeds To care what has changed
Divorce Process lifecycle from data lifecycle
Stateful Processing
CLIENT TIER
WEB SERVER
TIER
DATABASE TIER
Stateful Processing
service
CLIENT TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Pics Or It Didn’t Happen
https://github.com/graphite-project/whisper https://github.com/graphite-project/carbon
https://www.elastic.co/products/elasticsearch http://lucene.apache.org/
Swallowing The Red Pill
Stateful Processing
Gossip protocols
Stateful Processing
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Gossip Protocols
Gossip ProtocolsUsage
Gossip ProtocolsUsagedissemination - event / background
Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repair
Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network
Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network
SWIM
Gossip ProtocolsUsagedissemination - event / backgroundanti-entropy - data repairaggregation - compute across the network
SWIMscalable weakly-consistent infection-style process group membership protocol - membership list (non-faulty processes)
SWIM Protocol
SWIM Protocolfailure detection
SWIM Protocolfailure detection1. ping random process from member list
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! end
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
2. send ping request to k processes from member list
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
2. send ping request to k processes from member list3. k processes try to ping “failed” process
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
2. send ping request to k processes from member list3. k processes try to ping “failed” process
A. K processes respond back to original process about status
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
2. send ping request to k processes from member list3. k processes try to ping “failed” process
A. K processes respond back to original process about status
dissemination
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
2. send ping request to k processes from member list3. k processes try to ping “failed” process
A. K processes respond back to original process about status
disseminationmulticast failure (vol. leave) updates
SWIM Protocolfailure detection1. ping random process from member list
A. Receives Ack, Great! endB. does not receive ack, goto 2
2. send ping request to k processes from member list3. k processes try to ping “failed” process
A. K processes respond back to original process about status
disseminationmulticast failure (vol. leave) updatesmulticast new members
Gossip protocols
Stateful Processing
consensusGossip protocols
Stateful Processing
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
http://zookeeper.apache.org/
Paxos
PaxosProposers, Acceptors, Learners
PaxosProposers, Acceptors, Learners
Proposers:
PaxosProposers, Acceptors, Learners
Proposers: Ask Acceptors to Approve Proposals
PaxosProposers, Acceptors, Learners
Proposers: Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version
PaxosProposers, Acceptors, Learners
Proposers:
Acceptors:
Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version
PaxosProposers, Acceptors, Learners
Proposers:
Acceptors:
Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version
Do not have to approve or accept
PaxosProposers, Acceptors, Learners
Proposers:
Acceptors:
Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version
Do not have to approve or acceptOnly Approve/Accept what has been proposed
PaxosProposers, Acceptors, Learners
Proposers:
Acceptors:
Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version
Do not have to approve or acceptOnly Approve/Accept what has been proposed
LEARNERS:
PaxosProposers, Acceptors, Learners
Proposers:
Acceptors:
Ask Acceptors to Approve ProposalsAsk Acceptors to accept a proposal & Version
Do not have to approve or acceptOnly Approve/Accept what has been proposed
LEARNERS: Proposed Value is chosen when a majority of Acceptors have accepted the value
PaxosProposers, Acceptors, Learners
PaxosProposers, Acceptors, Learners
Phase 1:
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values2) send back highest (accepted) version
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values2) send back highest (accepted) version
Phase 2:
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values2) send back highest (accepted) version
Phase 2:1) proposer Receives approval from majority
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values2) send back highest (accepted) version
Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest
version
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values2) send back highest (accepted) version
Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest
version
Phase 3:
PaxosProposers, Acceptors, Learners
Phase 1:1) Proposers propose a value2) Acceptors approve value
1) not approve or accept smaller values2) send back highest (accepted) version
Phase 2:1) proposer Receives approval from majority2) sends accept! to acceptors with highest
version
Phase 3: 1) acceptors notify all learners
Gossip protocolsconsensus
Stateful Processing
consistent hashing
Gossip protocolsconsensus
Stateful Processing
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Stateful Processing
service
CLIENT TIER
Storage TIER
Consistent Hashing
Consistent HashingDistributed Hash Tables
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key
Issue
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key
IssueHorizontal scaling
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key
IssueHorizontal scalingFault Tolerance
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key
IssueHorizontal scalingFault Tolerancehash(object) (mod n)
n = number of slots
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for lookups given a key
IssueHorizontal scalingFault Tolerancehash(object) (mod n)Remapping keys to more slots
n = number of slots
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups
Solution
K = keys, n = number of slots
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups
Solutionconsistent hashing
K = keys, n = number of slots
Consistent HashingDistributed Hash TablesConvenient, Fast O(1)Great for Key / Value lookups
Solutionconsistent hashingDuring resize, k / n keys are remapped
K = keys, n = number of slots
Consistent Hashing
Challenge Accepted!
Challenges
Stateful Processing
Challengesresult composition
Stateful Processing
Challengesresult compositionwork distribution
Stateful Processing
Challengesresult compositionwork distributioncode deployments
Stateful Processing
Challengesresult compositionwork distributioncode deploymentsunbounded data structures
Stateful Processing
Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory management
Stateful Processing
Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory managementpersistence strategies
Stateful Processing
Challengesresult compositionwork distributioncode deploymentsunbounded data structuresmemory managementpersistence strategiesconcurrency
Stateful Processing
Key Takeaways
Key Takeaways
Stateful and Stateless systems co-exist
Key Takeaways
Stateful and Stateless systems co-exist
process lifecycle vs data lifecyCle
Key Takeaways
Stateful and Stateless systems co-exist
process lifecycle vs data lifecyCle
Stateful services have their place
Key Takeaways
Stateful and Stateless systems co-exist
process lifecycle vs data lifecyCle
Stateful services have their place
Interesting opportunities
Key Takeaways
Stateful and Stateless systems co-exist
process lifecycle vs data lifecyCle
Stateful services have their place
Interesting opportunities
But not all is rosy
Dynamo: Amazon’s Highly Available Key-value Store http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
Papers We Love
The Chubby lock service for loosely-coupled distributed systemsresearch.google.com/archive/chubby-osdi06.pdf
Paxos made simplehttp://research.microsoft.com/en-us/um/people/lamport/pubs/paxos-simple.pdf
Time, Clocks, and the Ordering of Events in a Distributed Systemhttp://research.microsoft.com/en-us/um/people/lamport/pubs/time-clocks.pdf
The Google File Systemhttp://research.google.com/archive/gfs-sosp2003.pdf
Resources
Caitie McCaffrey’s talks - https://goo.gl/8MSdRz
Apache Foundation - http://apache.org/
graphite - https://graphiteapp.org/
elastic - https://www.elastic.co/
Thanks! Q&A
Amazon is hiring!To learn more about our Dev Centers: http://bit.ly/phxdevcenters To learn more about current opportunities, email: [email protected]