Db presentation google_megastore
-
Upload
alanoudsalqoufi -
Category
Data & Analytics
-
view
164 -
download
0
Transcript of Db presentation google_megastore
![Page 1: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/1.jpg)
Department of information SystemCourse:IS533-Advanced topics in database
Done by :Alanoud Saad AlqoufiID :435920068
Supervised by : Prof.Almetwally Mohamed MostafaDate :5/3/2015
Google Megastore
![Page 2: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/2.jpg)
Outline• Introduction• Architecture and data model• Transactions and concurrency control• Replications• Data structurs and algorithms• Failure Detection• Throughput • Limitation • Related work• Experience• Questions
![Page 3: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/3.jpg)
What is Google Megastore
A database over Bigtable with high availability• Widely deployed in Google• Used on more than 100 application• Handle more than 3 billion write and 20 billion read• Store nearly a petabyte of primary data• Available on GAE since Jan 2011
1
![Page 4: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/4.jpg)
Motivation• Scalability• Availability • Consistency• Responsive• Rapid development
2
![Page 5: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/5.jpg)
Megastore=RDBMS+NOSQLRDBMS NOSQL
Slow performance High performanceNot scalable Scalable
Fixed data model No schemaEasier to code Complicated
Consistent Less consistent
3
![Page 6: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/6.jpg)
Megastore
4
![Page 7: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/7.jpg)
Toward Availability and Scalability
1. Data Replication Paxos Algorithm
2. Data Partitioning Entity Groups
5
![Page 8: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/8.jpg)
Entity Group Operations
6
![Page 9: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/9.jpg)
Design of Megastore• De-normalized dataData Model• The data model declared in schema• Each schema has a set of tables • Table could be Root or Child table• The root along with all child entities called Entity
Group
7
![Page 10: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/10.jpg)
Sample schema for Photo Sharing ServiceCREATE SCHEMA PhotoApp;
CREATE TABLE User { required int64 user_id;required string name;} PRIMARY KEY(user_id), ENTITY GROUP ROOT;
CREATE TABLE Photo {required int64 user_id;required int32 photo_id; required int64 time;required string full_url;optional string thumbnail_url;repeated string tag;} PRIMARY KEY(user_id, photo_id),IN TABLE User,ENTITY GROUP KEY(user_id) REFERENCES User;
CREATE LOCAL INDEX PhotosByTimeON Photo(user_id, time);CREATE GLOBAL INDEX PhotosByTagON Photo(tag) STORING (thumbnail_url); 8
![Page 11: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/11.jpg)
IndexesSecondary indexes are supported
• Local index• Global index• Storing clause• Repeated index• Inline index
9
![Page 12: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/12.jpg)
Mapping to Big tableBigtable column name=Megastore table name+Property name
10
![Page 13: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/13.jpg)
Transactions and concurrency controlConcurrency Control
MVCC• Read consistency• Current• Snapshot• Inconsistent reads
• Write consistency
11
![Page 14: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/14.jpg)
Transactions and concurrency control
Complete transaction lifecycle in Megastore1. Read2. Application logic3. Commit4. Apply5. Clean up
12
![Page 15: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/15.jpg)
Transactions and concurrency control
13
Queues• Example: Calendar application
![Page 16: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/16.jpg)
2 Phase commit
Transactions and concurrency control
14
![Page 17: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/17.jpg)
Paxos
15
Replications
![Page 18: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/18.jpg)
Modified Paxos
16
Fast reads Fast writes
Replications
![Page 19: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/19.jpg)
New Replica Types• Full Replicas• Witness Replicas• Read-only Replicas
17
Replications
![Page 20: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/20.jpg)
Replicated Logs
18
Data structurs and algorithms
![Page 21: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/21.jpg)
Reads
19
Data structurs and algorithms
![Page 22: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/22.jpg)
Writes
20
Data structurs and algorithms
![Page 23: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/23.jpg)
Failure Detection
• Chubby lock service
21
![Page 24: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/24.jpg)
Write Throughput• Sharding entity groups• Place replicas in same region• Bulk processing
22
![Page 25: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/25.jpg)
Limitations• Latency• Chain gang throttling• Not enforce policies on physical layout
23
![Page 26: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/26.jpg)
Experience
24
![Page 27: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/27.jpg)
Related Work• NoSQL Bigtable, Cassandra, Yahoo PNUTS, Amazon SimpleDB
• Data replication processHbase, CouchDB, Dynamo
• Paxos algorithmSCALARIS, Keyspace
25
![Page 28: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/28.jpg)
Questions1. Megastore is built upon2. Synchronous replication based upon3. Partitioned into a vast space of small databases each with its own
replicated4. The data model is declared in a strongly typed5. Megastore tables are either or tables6. 3 levels of read consistency: 7. Cross entity group updates are supported by:
26
BigTable
Paxos
log
SchemaRoot
Child
CurrentSnapshotInconsistent
2 Phase commit
![Page 29: Db presentation google_megastore](https://reader036.fdocuments.in/reader036/viewer/2022070516/5871e6dd1a28ab6a7b8b728d/html5/thumbnails/29.jpg)
Any Questions?