Amazon`s Dynamo
-
Upload
sarang33 -
Category
Technology
-
view
216 -
download
4
description
Transcript of Amazon`s Dynamo
![Page 1: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/1.jpg)
Amazon`s
Presented By
Sarang Metkar
Dynamo: Highly Available Key-value Store
Giuseppe DeCandiaDeniz HastorunMadan JampaniGunavardhan Kakulapatiand others
![Page 2: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/2.jpg)
Introduction
• Highly available and scalable distributed data store
• Flexible key – value data model
![Page 3: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/3.jpg)
Key-Value Data Model
• Simple Key-Value pairs
• Table - collection of items
• Item - collections of attributes
![Page 4: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/4.jpg)
Introduction
• Highly available and scalable distributed data store
• Flexible key – value data model
• Fast performance with seamless scalability
• Eventually consistent
• Decentralized system
![Page 5: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/5.jpg)
Motivation
• ‘Always On’ experience to large customer base
• Reduce impact of failure without compromising performance
• Diverse applications with different storage and data access requirement
• Configurable to achieve stringent SLAs
![Page 6: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/6.jpg)
SLA requirements
• Decentralized service
oriented architecture
• Multiple dependencies
hence tight constraints
• 99.9th percentile SLA
measurement
Reference : Dynamo: Amazon’s Highly Available Key-value Store : Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Alex Pilchin, Peter Vosshalland Werner Vogels
![Page 7: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/7.jpg)
Design Considerations - Consistency
• Availability using optimistic replication - eventual consistency
• Challenges in conflict resolution
• When to resolve ?
- Always writable requirement
• Who resolves?
- Application assisted
- Data store`s “last write wins” policy
![Page 8: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/8.jpg)
Other Design Considerations
• Incremental Scalability
• Symmetry
• Decentralization
• Heterogeneity
![Page 9: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/9.jpg)
System Interface• Object storage and access
• get(key)
- Locate object replicas
- Return single or list of objects
• put(key, context, object)
- Determine location of replica
- Context for conflict resolution
![Page 10: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/10.jpg)
Partition and Replication
• Consistent hashing for load and data
distribution
• Less impact of addition or deletion of
nodes
• Virtual nodes account for heterogeneity
• Coordinator node stores preference list
![Page 11: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/11.jpg)
Eventual Consistency
• Asynchronous updates of replicas
• Versioning, based on vector clocks
• Reconciliation
• Syntactic reconciliation
• Semantic reconciliation
• Sloppy quorum like consistency
protocol
• Configurable R , W and N
[ R + W > N ]
Reference : Dynamo: Amazon’s Highly Available Key-value Store : Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Alex Pilchin, Peter Vosshalland Werner Vogels
![Page 12: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/12.jpg)
Handling failures
• High Availability and Durability requirement
• Hinted handoff – temporary failures
• Replica synchronization – permanent failures
• Merkle Trees
- Less data transfer and faster replication
- One for each key range on node
- Recalculation of tree on key range changes
![Page 13: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/13.jpg)
Membership and Failure Detection
• Manual addition or removal of nodes
• Gossip based protocol to reconcile membership changes
• Partitioning and node-to-token sets mapping information propagation
• Seeds avoids logical partitioning
• Decentralized failure detection
![Page 14: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/14.jpg)
Implementation
• Local persistence engine
• Application specific
• Pluggable
• Request coordination
• Read/write request execution
• Read repair
• ‘Read-your-writes’ consistency
• Java NIO channel
• Membership and failure detection
![Page 15: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/15.jpg)
Key Learnings - 1• Common (N,R,W) configuration – (3, 2, 2)
• Balancing Performance and Durability• Buffering of write operations
• Durable write
![Page 16: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/16.jpg)
Key Learnings [cont`d]
• Uniform load distribution• More load imbalance for low load
• Q/S tokens per node, equal-sized partition• Faster bootstrapping and recovery
• Ease of archival
![Page 17: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/17.jpg)
Key Learnings [cont`d]
• Divergent Versions• Failures in system
• Concurrent writes to single object by multiple nodes
• Client driven coordination• Request coordination at client
• Pull membership information
• Reduces latency
• Admission Control mechanism for background tasks
![Page 18: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/18.jpg)
Related Work
• Peer to Peer systems
• Unstructured peer-to-peer network
• Gnutella [1]
• Freenet [2]
• Structure peer-to-peer network
• Oceanstore [3]
• Beehive [4]
• Distributed File Systems and Databases
• Google File System [5]
• Bayou [6]
![Page 19: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/19.jpg)
Conclusion
• Application specific configuration for availability, durability, performance and consistency
• Evaluation of different techniques to build highly available system
• Use of eventually consistent storage system in production
• Tuning of various techniques to meet strict production performance requirements
![Page 20: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/20.jpg)
References
• [1] http://www.gnutella.org/
• [2] http://freenetproject.org/
• [3] Kubiatowicz, J., Bindel, D., Chen, Y., Czerwinski, S., Eaton, P., Geels, D.,
Gummadi, R., Rhea, S., Weatherspoon, H., Wells, C., and Zhao, B. 2000.
OceanStore: an architecture for global-scale persistent storage.
• [4] Ghemawat, S., Gobioff, H., and Leung, S. 2003. The Google file system. In
Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles
• [5] Ramasubramanian, V., and Sirer, E. G. Beehive: O(1)lookup performance for
power-law query distributions in peer-to-peer overlays.
• [6] Terry, D. B., Theimer, M. M., Petersen, K., Demers, A. J., Spreitzer, M. J., and
Hauser, C. H. 1995. Managing update conflicts in Bayou, a weakly connected
replicated storage system.
![Page 21: Amazon`s Dynamo](https://reader034.fdocuments.in/reader034/viewer/2022052311/559458881a28ab642f8b478c/html5/thumbnails/21.jpg)
Thank You