Transcript of Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go
- 1. Putting the Go in MongoDB Wisdom Omuya Software Engineer
wisdom@mongodb.com
- 2. MongoDB v3.0.0
- 3. WHAT TOOLS? tools
- 4. downloads.mongodb.org
- 5. Import mongofiles
- 6. Export
- 7. Monitoring mongostat mongotop
- 8. Others bsondump mongooplog
- 9. ROUGH EDGES tools
- 10. Mongorestore Concurrency in 2.6
- 11. Invalid Documents Nested CSV, TSV imports { "_id":
ObjectId("5542593837096bdf8bbb6d91"), name.first: Wisdom,
name.last: Omuya" } name.first,name.last Wisdom,Omuya
- 12. mongo> db.test.find({name.first":Wisdom"}) Fetched 0
record(s) in 0ms Unretrievable Data { "_id":
ObjectId("5542593837096bdf8bbb6d91"), name.first: Wisdom,
name.last: Omuya" }
- 13. REQUIREMENTS tools
- 14. Loose Coupling
- 15. Shorter Release Cycle
- 16. Portability
- 17. Concurrent Processing CPU
- 18. General Storage engine support Wiredtiger, RocksDB, etc
Backwards compatibility Dump and restore BSON Import and export
JSON
- 19. Excellent support for concurrency Runs on all supported
platforms Easier to write and debug Fun! Why Go?
- 20. MONGOIMPORT tools
- 21. mongoimport
- 22. Mongoimport Architecture
- 23. New Mongoimport Flags --numDecodingWorkers (autodetect to
max) --numInsertionWorkers (default 1) --batchSize (default 10k)
--numThreads (autodetect to max) --maintainInsertionOrder (default
false)
- 24. Input Validation No more broken imports, illegal field
names a, b.
- 25. Dot-nesting Nested CSV, TSV imports name.first,name.last
Wisdom,Omuya { "_id": ObjectId("55425a3c37096bdf8bbb6d93"), "name":
{ "first": Wisdom", "last": Omuya" } }
- 26. Import/Export Type Fidelity
- 27. Write Concern Specificity New default w=majority on
import/restore/files o safer, matches what our users assume is
happening --writeConcern flag e.g. {w: 3, j: true, fsync: false,
wtimeout: 400}
- 28. Mongoimport Concurrency in 3.0
- 29. MONGORESTORE tools
- 30. Mongorestore Architecture
- 31. New Mongorestore Flags --numParallelCollections (aka -j
default=4) --numInsertionWorkersPerCollection (default=1)
--batchSize(default 10k) --numThreads (autodetect to max)
--maintainInsertionOrder (default false)
- 32. Mongorestore Concurrency in 3.0 2X speedup 4X speedup
- 33. Whats next? mongorestore and mongodump now support
compression and archiving No need for directories containing BSON
files mongodump -d weather -h localhost --archive --gzip |
mongorestore -h remotehost --archive --gzip
- 34. Resources Repository:
https://github.com/mongodb/mongo-tools Usage Guide:
https://docs.mongodb.org/master/reference/program/ Issue Tracking:
https://jira.mongodb.org/browse/tools/
- 35. THANKS! tools