MongoDB: Awesomely Dangerous
-
Upload
ethan-gunderson -
Category
Technology
-
view
2.788 -
download
1
description
Transcript of MongoDB: Awesomely Dangerous
![Page 1: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/1.jpg)
MongoDB:Awesomely Dangerous
Twin Cities Code CampOctober 2010
Sunday, October 10, 2010
![Page 2: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/2.jpg)
Ethan Gundersonhttp://ethangunderson.com
Twitter & Github: ethangunderson
Sunday, October 10, 2010
![Page 3: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/3.jpg)
Sunday, October 10, 2010
![Page 4: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/4.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 5: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/5.jpg)
30,000 foot overview
• Schema-less
• Scalable
• High-Performance
• Open Source
• Document Database
Sunday, October 10, 2010
![Page 6: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/6.jpg)
Driving Principles
1) Performance
2) Performance
3) Scalability
Sunday, October 10, 2010
![Page 7: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/7.jpg)
Documents
• Similar to traditional rows
• First attribute is _id, which is an ObjectId
Sunday, October 10, 2010
![Page 8: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/8.jpg)
ObjectID
1) Timestamp
2) Machine Id
3) Process Id
4) Counter
4b6857a07613c367094426b2
Sunday, October 10, 2010
![Page 9: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/9.jpg)
Embedded Documents• Documents in Documents
• Indexable
• Queryable
Sunday, October 10, 2010
![Page 10: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/10.jpg)
Max Document Size
• 4MB limit on individual documents
• Realistically ~250kb
• Used to force better data modeling
Sunday, October 10, 2010
![Page 11: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/11.jpg)
Collections• Similar to traditional tables
• Collections of like documents
• Schema-less
Sunday, October 10, 2010
![Page 12: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/12.jpg)
Capped Collections
• Fixed size collections
• Must be explicitly created
• Limited functionality (no deletes, limited updates)
Sunday, October 10, 2010
![Page 13: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/13.jpg)
Sunday, October 10, 2010
![Page 14: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/14.jpg)
BSONA language independent data interchange format
• The language of Mongo
• Similar to JSON, but BETTER
• Fast
• 10gen driver support for: C, C++, Java, Javascript, Perl, PHP, Python, Ruby
• Community support for:REST, C#, Clojure, Coldfusion, Scala, and a lot more
Sunday, October 10, 2010
![Page 15: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/15.jpg)
B-Tree Indexes• Similar to a traditional RDBMS
• Can index any field, including arrays
• Missing keys in a unique will be given a value of null
• Blocking by default
Sunday, October 10, 2010
![Page 16: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/16.jpg)
Sunday, October 10, 2010
![Page 17: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/17.jpg)
Inserting
Sunday, October 10, 2010
![Page 18: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/18.jpg)
Updating
Sunday, October 10, 2010
![Page 19: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/19.jpg)
Modifier Operators
$inc$set$push$pushAll$pop$pull$pullAll$addToSet
Conventional updates do work, they’re just not as fast.
Sunday, October 10, 2010
![Page 20: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/20.jpg)
Querying
Sunday, October 10, 2010
![Page 21: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/21.jpg)
Query Operators$in$nin $all $ne $gt$gte$lt
$lte$size$where$limit $offset$sort$slice
Sunday, October 10, 2010
![Page 22: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/22.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 23: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/23.jpg)
Map / Reduce• Replaces GROUP BY in SQL
• Similar in spirit to Hadoop with all info coming from a collection and going to a collection
• Runs in parallel on all shards, but only one thread per node
• map and reduce functions written in Javascript
Sunday, October 10, 2010
![Page 24: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/24.jpg)
GIS
• Built with location based queries in mind
• Assumes a flat map model of the Earth(!)
Sunday, October 10, 2010
![Page 25: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/25.jpg)
Near Queries
Sunday, October 10, 2010
![Page 26: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/26.jpg)
Bounded Queries
Sunday, October 10, 2010
![Page 27: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/27.jpg)
GridFS• How you store large files in Mongo
• Spreads data to multiple 256kb documents in a ‘chunks’ collection
• Meta data about the file is stored in the files collection
• Permits range operations (x bytes from file)
Sunday, October 10, 2010
![Page 28: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/28.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 29: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/29.jpg)
The journey of a write
Sunday, October 10, 2010
![Page 30: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/30.jpg)
Memory Mapped File
Save!Success!
Sunday, October 10, 2010
![Page 31: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/31.jpg)
Sunday, October 10, 2010
![Page 32: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/32.jpg)
Safe Mode
• Allows you to determine the durability of a write per query
• Sacrifice performance for safety
• Options for each stage of the write
Sunday, October 10, 2010
![Page 33: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/33.jpg)
Safe mode query
Sunday, October 10, 2010
![Page 34: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/34.jpg)
Memory Mapped File
Save!
Success!
Sunday, October 10, 2010
![Page 35: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/35.jpg)
FsyncEvery 60 seconds, or when the kernel forces it
Sunday, October 10, 2010
![Page 36: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/36.jpg)
Save mode with fsync
Sunday, October 10, 2010
![Page 37: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/37.jpg)
Memory Mapped File
Save!
Success!
Sunday, October 10, 2010
![Page 38: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/38.jpg)
Safe mode with replication flag
Sunday, October 10, 2010
![Page 39: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/39.jpg)
Memory Mapped File
Save!
Success!
Sunday, October 10, 2010
![Page 40: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/40.jpg)
Save Flag with fsync and replication
Sunday, October 10, 2010
![Page 41: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/41.jpg)
Memory Mapped File
Save!
Success!
Sunday, October 10, 2010
![Page 42: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/42.jpg)
Low and High Value
• Useful to determine when building queries
• Allows you to be more careful(and slow), with data that is more important
Sunday, October 10, 2010
![Page 43: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/43.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 44: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/44.jpg)
Sunday, October 10, 2010
![Page 45: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/45.jpg)
Single Server DurabilityIt’s not
Sunday, October 10, 2010
![Page 46: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/46.jpg)
Sunday, October 10, 2010
![Page 47: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/47.jpg)
What 10gen has to say...
http://blog.mongodb.org/post/381927266/what-about-durability
True single server durability is almost never done correctly.First, there are many scenarios in which that server loses all its data no matter what. If there is water damage, fire, some hardware problems, etc… no matter how durable the software is, data can be lost.
The path to true durability is replication.
Sunday, October 10, 2010
![Page 48: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/48.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 49: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/49.jpg)
ScalingSince we’re forced to think about it
Sunday, October 10, 2010
![Page 50: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/50.jpg)
Determining how to scale
Reads
Writes
Sunday, October 10, 2010
![Page 51: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/51.jpg)
Replica Sets
• Distribute reads across the cluster
• Replaces the traditional Master/Slave setup
• Replication is done via an ops log
• Auto failover
• Rack and Datacenter aware
• Smart, very smart
Sunday, October 10, 2010
![Page 52: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/52.jpg)
Master
SlaveSlave Slave
Sunday, October 10, 2010
![Page 53: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/53.jpg)
Master
SlaveSlave Slave
X
Sunday, October 10, 2010
![Page 54: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/54.jpg)
Now a slave
New Master
Slave Slave
X
Sunday, October 10, 2010
![Page 55: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/55.jpg)
Slave
New Master
Slave Slave
Sunday, October 10, 2010
![Page 56: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/56.jpg)
Slave
Master
SlaveSlave
Sunday, October 10, 2010
![Page 57: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/57.jpg)
Slave
Master
SlaveSlave
X
Sunday, October 10, 2010
![Page 58: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/58.jpg)
Master
Slave Slave
X
Me! Me!
Sunday, October 10, 2010
![Page 59: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/59.jpg)
Master
Slave Slave
X
Me! Me!
Web Slice(Arbiter)
You
Sunday, October 10, 2010
![Page 60: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/60.jpg)
Determining how to scale
Reads
Writes
Sunday, October 10, 2010
![Page 61: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/61.jpg)
Being write heavyCurrently, Mongo can only process one concurrent write.
Usually not a problem, as writes are wicked fast
Sunday, October 10, 2010
![Page 62: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/62.jpg)
Auto-Sharding• Partitions data across the cluster in an
order preserving manner
• No support for load based partitioning
• Automatic failover and balancing of nodes
• Distributes writes across the cluster
• Based very heavily off of Yahoo!’s PNUTS and Google’s BigTable
Sunday, October 10, 2010
![Page 63: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/63.jpg)
Sunday, October 10, 2010
![Page 64: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/64.jpg)
Sunday, October 10, 2010
![Page 65: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/65.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 66: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/66.jpg)
Takeaways• Mongo is fast, but it does interesting things to
be that fast
• Mongo is not SQL. You will need to learn new things
Sunday, October 10, 2010
![Page 67: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/67.jpg)
Our agenda1) Intro to MongoDB
2) Advanced Features
3) Write Path
4) Durability
5) Scaling
6) Takeaways
7) Qs and As
Sunday, October 10, 2010
![Page 68: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/68.jpg)
Qs and As
http://spkr8.com/t/4756
Sunday, October 10, 2010
![Page 69: MongoDB: Awesomely Dangerous](https://reader033.fdocuments.in/reader033/viewer/2022051611/54b74b064a7959ef448b45ed/html5/thumbnails/69.jpg)
ResourcesOfficial MongoDB sitehttp://mongodb.org
BSON sitehttp://bsonspec.org
Comprehensive writeup of mongo featureshttp://www.markus-gattol.name/ws/mongodb.html
Sunday, October 10, 2010