Ops Jumpstart: MongoDB Administration 101
-
Upload
mongodb -
Category
Technology
-
view
233 -
download
5
description
Transcript of Ops Jumpstart: MongoDB Administration 101
![Page 1: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/1.jpg)
Senior Director of Performance Engineering, MongoDB
Alvin Richards
#MongoDBDays
5 Things you need to know about Administering MongoDB
![Page 2: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/2.jpg)
Senior Director of Performance Engineering, MongoDB
Alvin Richards
#MongoDBDays
Or…A Dev and Ops "Love Story"
![Page 3: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/3.jpg)
Before we start…
• I'm assuming you have at least – installed MongoDB– Have done some basic Admin
• I'm not assuming you have– Been running in production for any length of time
![Page 4: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/4.jpg)
Dev are from Mars and Ops are from Venus…
DEV OPS
![Page 5: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/5.jpg)
5 Things you need to know
1. Know Thy Beast
2. Protecting data
3. Scaling Writes & Reads
4. Monitoring
5. Backing up & Restoring
![Page 6: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/6.jpg)
#1Know thy beast
![Page 7: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/7.jpg)
Architecture
• Shipped as Binaries
• tarball, zip file• Packages(rpm, yum)• MSI (for Windows)
• Runs a background process mongod (or mogod.exe)
• Uses Memory Mapped Files
![Page 8: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/8.jpg)
Memory Mapped files – dude whoa
• To MongoDB memory is an infinite-ish pool
• 64 TB Linux / Windows 2012 R2 / Windows 8.1• 4 TB earlier Windows versions
• MongoDB accesses memory locations in virtual memory
• O/S may have to page in that data from storage
• O/S may have to page out data from RAM
• Physical RAM acts as a LRU cache
![Page 9: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/9.jpg)
Collection 1
Index 1
![Page 10: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/10.jpg)
Virtual Address Space 1
Collection 1
Index 1
This is your virtual memory size (mapped)
![Page 11: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/11.jpg)
Physical RAM
Collection 1
Index 1 This is your
resident memory size
Virtual Address Space 1
![Page 12: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/12.jpg)
Physical RAM
DiskCollection 1
Index 1
Virtual Address Space 1
![Page 13: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/13.jpg)
Physical RAM
DiskCollection 1
Index 1
100 ns
10,000,000 ns
=
=
Virtual Address Space 1
![Page 14: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/14.jpg)
• Working Set– Active Data & Indexes in Memory
• Sorting
• Aggregation & Map/Reduce
• Connections
What Memory is used for
![Page 15: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/15.jpg)
Working Set Exceeds Physical Memory
![Page 16: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/16.jpg)
> db.serverStatus( { workingSet: 1 } )
…
"workingSet" : {
"note" : "thisIsAnEstimate",
"pagesInMemory" : 378,
"computationTimeMicros" : 4104,
"overSeconds" : 4646
},
Where can I find this info?
![Page 17: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/17.jpg)
> mongostat
…
Insert query update delete getmore command flushes mapped vsize res
*0 *0 *0 *0 0 1|0 0 160m 2.75g 5m
*0 *0 *0 *0 0 1|0 0 160m 2.75g 5m
> iostat
> sar
> vmstat
Windows: Perfmon
Where can I find this info?
![Page 18: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/18.jpg)
#2Protecting data
![Page 19: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/19.jpg)
Deployment - Single Node
Good
• Write ahead journaling for crash protection
Bad
• Cannot survive loss of one node
Primary – A
![Page 20: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/20.jpg)
Oppps
http://i.ytimg.com/vi/PzWK9QCYgH8/hqdefault.jpg
![Page 21: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/21.jpg)
Replica Set – Initialize
![Page 22: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/22.jpg)
Replica Set – Failure
![Page 23: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/23.jpg)
Replica Set – Failover
![Page 24: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/24.jpg)
Replica Set – Recovery
![Page 25: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/25.jpg)
Replica Set – Recovered
![Page 26: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/26.jpg)
Oops… Multiple Failures
http://f1-grandprix.com/wp-content/uploads/wppa/565.jpg
![Page 27: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/27.jpg)
Deployment–Single Data Center
Good
• Automated failover
• Tolerates server failures
• Tolerates rack failures
• Number of replicas defines failure tolerance
Bad
• Cannot survive loss of Data Center
Primary – A
Secondary – B
Secondary – C
![Page 28: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/28.jpg)
Deployment - Active/Active Data Center
Good
• Tolerates server, rack, data center failures, network partitions
Bad
• Complexity but "MMS Automation to the rescue" ™
Data Center - US
US-A
Data Center - AsiaData Center - Europe
US-B
EU-C
EC-D
ASIA-E
![Page 29: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/29.jpg)
> rs.status(){
"set" : "foo","date" : ISODate("2014-09-08T01:29:58Z"),"myState" : 1,"members" : [
{"_id" : 0,"name" : "vero.local:31000","health" : 1,"state" : 1,"stateStr" : "PRIMARY","uptime" : 28,
Useful commands
![Page 30: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/30.jpg)
> rs.printReplicationInfo()
configured oplog size: 192MBlog length start to end: 0secs (0hrs)oplog first event time: Sun Sep 07 2014 18:17:52 GMT-0700 (PDT)oplog last event time: Sun Sep 07 2014 18:29:52 GMT-0700 (PDT)now: Sun Sep 07 2014 18:30:49 GMT-0700 (PDT)
Useful commands
![Page 31: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/31.jpg)
Application Considerations
Dev Ops
![Page 32: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/32.jpg)
Tunable Data Durability
Memory
Journal Secondary
Other Data Center
RDBMS
w=1
j=true
w="majority"w=nw="myTag"
Latency
Durability
async
sync
![Page 33: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/33.jpg)
#3Scaling Writes & Reads
![Page 34: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/34.jpg)
#3.1Scaling Writes
![Page 35: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/35.jpg)
Sharding - Range distribution
shard01 shard02 shard03
sh.shardCollection("test.tweets", {user: 1} , false)
![Page 36: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/36.jpg)
Sharding - Range distribution
shard01 shard02 shard03
a-i j-r s-z
![Page 37: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/37.jpg)
Shard Key Options
• Single or Compound Key
• Regular or Hashed key
• Tagged
![Page 38: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/38.jpg)
Sharding – Auto Splits
shard01 shard02 shard03
a-i ja-jz s-z
k-r
![Page 39: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/39.jpg)
Sharding – Auto Splits
shard01 shard02 shard03
a-i ja-ji s-z
ji-js
js-jw
jz-r
![Page 40: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/40.jpg)
Sharding - Auto Balancing
shard01 shard02 shard03
a-i ja-ji s-z
ji-js
js-jw
jz-r
js-jw
jz-r
![Page 41: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/41.jpg)
Sharding - Goal Equilibrium
shard01 shard02 shard03
a-i ja-ji s-z
ji-js
js-jw
jz-r
![Page 42: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/42.jpg)
Sharding - Find by Shard Key
shard01 shard02 shard03
a-i ja-ji s-z
ji-js
js-jw
jz-r
find({user: "alvin"})
![Page 43: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/43.jpg)
Sharding - Find by Attribute
shard01 shard02 shard03
a-i ja-ji s-z
ji-js
js-jw
jz-r
find({email: "[email protected]"})
![Page 44: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/44.jpg)
Application Considerations
Dev Ops
![Page 45: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/45.jpg)
Shard Key
• What key to use?
• Plain or Hashed?
• What's the distribution & usage?
• Predicate & Sorts– Use the Shard key?– Scatter Gather + Sort?
![Page 46: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/46.jpg)
> sh.status()... shards:
{ "_id" : "shard0000", "host" : "localhost:29001", "tags" : [ "dc-emea" ] }
{ "_id" : "shard0001", "host" : "localhost:29002", "tags" : [ "dc-asia" ] }
{ "_id" : "shard0002", "host" : "localhost:29003", "tags" : [ "dc-na" ] } databases:
{ "_id" : "scaleout", "partitioned" : true, "primary" : "shard0000" }
scaleout.blogsshard key: { "loc" : 1 }chunks: shard0000 1{ "loc" : { "$minKey" : 1 } } -->>
{ "loc" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)
Useful commands
![Page 47: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/47.jpg)
// Commands to start/Stop the balancer
> sh.startBalancer()
> sh.stopBalancer()
// Disable balancing for a specific collection
> sh.disableBalancing("students.grades")
// Runt the balancer during a specific time window
> db.settings.update({ _id : "balancer" },
{ $set : { activeWindow : { start : "<start-time>", stop : "<stop-time>" } } }, true )
Useful commands
![Page 48: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/48.jpg)
#3.2Scaling Reads
![Page 49: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/49.jpg)
Immediate vs. Eventual Consistency
• Do you want to read your own writes?– Use Sharding to Scale Reads
• Are you tolerant to read older versions of data?
– May read from Secondary's
![Page 50: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/50.jpg)
Application Considerations
Dev Ops
![Page 51: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/51.jpg)
PrimaryThread #1
Insert
Update
Read
Read
v1
✔
✔
v2
Immediate Consistency
![Page 52: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/52.jpg)
Eventual Consistency
Primary Secondary
Thread #1
Insert
Read
v1
Thread #2
v1
✖
v1 does not exist
reads v1Update
Read
v2
v2
✖
![Page 53: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/53.jpg)
Eventual Consistency with a Write Concern
Primary Secondary
Thread #1
Insert (w=2)
Read
v1
Thread #2
v1 reads v1
✔
✖
v1 does not exist
![Page 54: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/54.jpg)
#4Monitoring
![Page 55: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/55.jpg)
Pick your tool dejour
• Plugins for– Cacti, Zabix, Munin, NewRelic….
• MongoDB Monitoring (MMS)– mms.mongodb.com– Focused on MongoDB cluster well being
![Page 56: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/56.jpg)
MMS
![Page 57: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/57.jpg)
KPIs you care about
• MongoDB– MongoDB ops/sec– Read/Write queues– Index Misses– Locked %– Flush times– Replication Lag
![Page 58: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/58.jpg)
KPIs you care about
• Storage– IOPS– Size– Data & Loading
Patterns
• Memory– Working Set
• CPU– Speed– Cores
• Network– Latency– Throughput
![Page 59: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/59.jpg)
Example
MongoDB ops/sec
Page Faults / Sec
![Page 60: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/60.jpg)
#5Backup & Restore
![Page 61: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/61.jpg)
Options
• mongodump / mongorestore
• File system snaphot / rsync
• MMS Backup– Automated Cluster Backup / Restore– Snapshots– Point in Time recovery– OpLog based
![Page 62: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/62.jpg)
MMS Backup & Restore
![Page 63: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/63.jpg)
Other talks today
• 10.25 – Running MongoDB with confidence with MMS
• 12.45 – MongoDB Sharding
• 1.05 – Securing Your MongoDB Deployment
• 4.10 – Diagnostics and Debugging
![Page 64: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/64.jpg)
Get MongoDB Certified
• 20% off our October exam with the code
• MongoDBSeattle20
• TODAY ONLY
• university.mongodb.com
![Page 65: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/65.jpg)
A happy ever after?
Dev Ops
![Page 66: Ops Jumpstart: MongoDB Administration 101](https://reader031.fdocuments.in/reader031/viewer/2022020306/5453d0fcaf795959498b618a/html5/thumbnails/66.jpg)
[email protected] / @jonnyeight
Senior Director of Performance Engineering, MongoDB
Alvin Richards
#MongoDBDays
Thank You