Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go

1. Putting the Go in MongoDB Wisdom Omuya Software Engineer [email protected]

2. MongoDB v3.0.0

3. WHAT TOOLS? tools

4. downloads.mongodb.org

5. Import mongofiles

6. Export

7. Monitoring mongostat mongotop

8. Others bsondump mongooplog

9. ROUGH EDGES tools

10. Mongorestore Concurrency in 2.6

11. Invalid Documents Nested CSV, TSV imports { "_id": ObjectId("5542593837096bdf8bbb6d91"), name.first: Wisdom, name.last: Omuya" } name.first,name.last Wisdom,Omuya

12. mongo> db.test.find({name.first":Wisdom"}) Fetched 0 record(s) in 0ms Unretrievable Data { "_id": ObjectId("5542593837096bdf8bbb6d91"), name.first: Wisdom, name.last: Omuya" }

13. REQUIREMENTS tools

14. Loose Coupling

15. Shorter Release Cycle

16. Portability

17. Concurrent Processing CPU

18. General Storage engine support Wiredtiger, RocksDB, etc Backwards compatibility Dump and restore BSON Import and export JSON

19. Excellent support for concurrency Runs on all supported platforms Easier to write and debug Fun! Why Go?

20. MONGOIMPORT tools

21. mongoimport

22. Mongoimport Architecture

23. New Mongoimport Flags --numDecodingWorkers (autodetect to max) --numInsertionWorkers (default 1) --batchSize (default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)

24. Input Validation No more broken imports, illegal field names a, b.

25. Dot-nesting Nested CSV, TSV imports name.first,name.last Wisdom,Omuya { "_id": ObjectId("55425a3c37096bdf8bbb6d93"), "name": { "first": Wisdom", "last": Omuya" } }

26. Import/Export Type Fidelity

27. Write Concern Specificity New default w=majority on import/restore/files o safer, matches what our users assume is happening --writeConcern flag e.g. {w: 3, j: true, fsync: false, wtimeout: 400}

28. Mongoimport Concurrency in 3.0

29. MONGORESTORE tools

30. Mongorestore Architecture

31. New Mongorestore Flags --numParallelCollections (aka -j default=4) --numInsertionWorkersPerCollection (default=1) --batchSize(default 10k) --numThreads (autodetect to max) --maintainInsertionOrder (default false)

32. Mongorestore Concurrency in 3.0 2X speedup 4X speedup

33. Whats next? mongorestore and mongodump now support compression and archiving No need for directories containing BSON files mongodump -d weather -h localhost --archive --gzip | mongorestore -h remotehost --archive --gzip

34. Resources Repository: https://github.com/mongodb/mongo-tools Usage Guide: https://docs.mongodb.org/master/reference/program/ Issue Tracking: https://jira.mongodb.org/browse/tools/

35. THANKS! tools

Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go

Technology

Transcript of Putting the Go in MongoDB: How We Rebuilt The MongoDB Tools in Go