1. Open source, high performance databaseAnti-social Databases: NoSQL and MongoDBWill LaForestSenior Director of 10gen Federalwill@10gen.com@WLaForest 1

2. SQLDynamic inventedWeb Content released10gen Web applicationsfoundedOracleClientIBMs IMSfoundedServer SOABigTable19651970 1975 1980 1985 1990 1995 2000 20052010 Codd publishesPCs gain3 tierCloud relational model paper traction architecture Computing in 1970Brewers CapNoSQLWWW born Movementborn2 3. AttributeTupleRelation 3 4. Category BlogCategory N 1 N 1 N 1N 1 Author BlogBlogTagNN Content Tag4 5. Data stored in a RDBMS is very compact (disk wasmore expensive) SQL and RDBMS made queries flexible with rigidschemas Rigid schemas helps optimize joins and storage Massive ecosystem of tools, libraries and integrations Been around 40 years! 5 6. Gartner uses the 3Vs to define Volume - Big/difficult/extreme volume is relative Variety Changing or evolving data Uncontrolled formats Does not easily adhere to a single schema Unknown at design time Velocity High or volatile inbound data High query and read operations Low latency6 7. VOLUME/VELOCITY &NEW ARCHITECTURES Systems need to scale horizontalnot vertically Commodity servers Cloud ComputingDATA VARIETY &VOLATILITY Extremely difficult tofind a single fixedschema Dont know dataschema a-prioriAGILE DEVELOPMENT Iterative & continuous New and emerging Apps 7 8. Non-relational been hanging around (heard ofMUMPS?) Modern NoSQL theory and offerings started in early2000s NoSQL = Not Only SQL A collection of very different products Alternatives to relational databases when a bad fit Common motivations Horizontally scalable (commodity server/cloud computing) Schema Flexibility8 9. I group by data model Some people use or include data arangement Data arrangement is important and cuts across datamodels (see my talk at BigData DC next week) Column/Document Consistent hashing partitioning Range based partitioning Key/Value Big Table Descendents Document Oriented Graph 9 10. Value (Data) mapped to a key (think primary) Some typed, some just BLOBs Fast hash based queries No range queries, no ordering, simple data model Will will@10gen.comChris chris@10gen.com Will-obj [4e61 6d65 3a57 10 11. Data stored on disk in a column oriented fashion Predominantly hash based indexing Rudimentary or no secondary indexes Range queries and ordering on one dimension (row key) Some consistent hashing some range basedrow keys column family column family contact personal twitter: wlaforestbio: Will attended row Will email: will@10gen.compicture: phone: 555-555-5555bio: email: chris@10gen.comrowChrispicture: phone: 555-555-5555hobby: golf11 12. Data modeled as documents or objects (XML and JSON) No fixed schema Richest data model Consistent hashing and range based partitioning {name: will, name: jeff, {name: brendan,eyes: blue, eyes: blue,aliases: [el diablo]}birthplace: NY, height: 72,aliases: [bill, la boss: ben} ciacco],{name: matt,gender: ???, pizza: DiGiorno,boss: ben}name: ben, height: 72,hat: yes}boss: 555.555.1212} 12 13. Not a database! Map Reduce on HDFS or other data source Great for grinding through data When you want to use it Cant use a index Distributing custom algorithms ETL Many NoSQL offerings have native map reducefunctionality13 14. 14 15. 2007 founded 2009 first release of MongoDB MongoDB is open source 10gen has a Redhat-like business model Subscriptions (subscriber build, support) Training Consulting ~80M in funding Sequoia, NEA, In-Q-Tel, Union Square Ventures, FlybridgeCapital, 15 16. #2 on Indeeds Fastest Growing JobsJaspersoft BigData IndexDemand forMongoDB, thedocument-orientedNoSQL database, sawthe biggest spikewith over 200%growth in 2011. 451 Group Google SearchesMongoDB increasing its dominance16 17. #2 ON INDEEDS FASTEST GROWING JOBS17 18. MongoDB INCREASING ITS DOMINANCE 18 19. Scale horizontally over commodity hardware Agility essential (schema free/heterogeneousinterface) RDBMSs great so keep what works Rich data models Adhoc queries Fully featured indexes What doesnt distribute well? Long running multi-row transactions Join Both artifacts of the relational data model 19 20. Data stored as documents (JSON) Schema free CRUD operations (Create Read Update Delete) Atomic document operations Consistent (but is tunable advanced topic) Rich indexing (secondary, geospatial, covered) Ad hoc Queries like SQL Equality Ranges Regular expression searches Geospatial Replication HA, read scalability, geo centric reads Sharding (sometimes called partitioning) for scalability 20 21. RDBMSMongoDBDatabase DatabaseTableCollectionRowDocument21 22. 22 23. var p = { author: roger, date: new Date(), text: Spirited Away, tags: *Tezuka, Manga+-> db.posts.save(p)23 24. >db.posts.find() { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "roger", date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Spirited Away", tags : [ "Tezuka", "Manga" ] }Notes: - _id is unique, but can be anything youd like 24 25. Create index on any Field in Document // 1 means ascending, -1 means descending >db.posts.ensureIndex({author: 1}) >db.posts.find({author: roger}) { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"), author : "roger", ... } 25 26. Conditional Operators $all, $exists, $mod, $ne, $in, $nin, $nor, $or, $size, $type $lt, $lte, $gt, $gte// find posts with any tags> db.posts.find( {tags: {$exists: true }} )// find posts matching a regular expression> db.posts.find( {author: /^rog*/i } )// count posts by an author before a certain date> db.posts.find( {author: roger, date:, $lt: Sat -- ).count() 26 27. $set, $unset, $inc, $push, $pushAll, $pull, $pullAll, $bit> comment = { author: fred, date: new Date(), text: Best Movie Ever-> db.posts.update( { _id: ... -,$push: {comments: comment} ); 27 28. { _id : ObjectId("4c4ba5c0672c685e5e8aabf3"),author : "roger",date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)",text : "Spirited Away",tags : [ "Tezuka", "Manga" ],comments : [ { author : "Fred", date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)", text : "Best Movie Ever" }]} 28 29. // Index nested documents> db.posts.ensureIndex( comments.author:1 ) db.posts.find(,comments.author:Fred-)// Index on tags> db.posts.ensureIndex( tags: 1)> db.posts.find( { tags: Manga - )// geospatial index> db.posts.ensureIndex( author.location: 2d )> db.posts.find( author.location : , $near : [22,42] } ) 29 30. 30 31. Do them client side (Python or R) Native Map/Reduce in JS in MongoDB Distributes across the cluster with good data locality New aggregation framework Declarative (no JS required) Pipeline approach (like Unix ps -ax | tee processes.txt | more) Hadoop Intersect the indexing of MongoDB with the brute force parallelizationof hadoop Hadoop MongoDB connector 31 32. 32 33. Prod clusters on order of 100B objects 50k qps per server ~1400 nodesBetter data locality In-Memory Auto-ShardingCaching Replication /HAHorizontal Scaling 33 34. Sharding Details Replica Set Details Consistency Details Common Deployment Scenarios Citations34 35. 35 36. Write Primary Secondary Read SecondaryDriver Read 36 37. Write PrimaryDriver Read Secondary Secondary 37 38. 1. WritePrimarySecondary 1. Replicate 2. ReadSecondaryDriver 2. Read 38 39. Fire and forget Wait for error Wait for fsync Wait for journal sync Wait for replication39 40. Driver Primary write apply in memory 40 41. Driver Primarywrite getLastError apply in memory 41 42. DriverPrimarywrite getLastErrorapply in memory j:trueWrite to journal 42 43. Driver Primarywrite getLastError apply in memoryfsync:true fsync 43 44. Driver Primary Secondarywrite getLastError apply in memory w:2 replicate 44 45. ValueMeaningReplicate to N members of replica setmajority Replicate to a majority of replica set members Use cutom error mode name45 46. 46 47. > db.runCommand( { shardcollection: test.users,key: { email: 1 }} ){name: Jared,email: jsr@10gen.com,}{name: Scott,email: scott@10gen.com,}{name: Dan,email: dan@10gen.com,}47 48. - +48 49. -+ dan@10gen.comscott@10gen.comjsr@10gen.com 49 50. Split!-+ dan@10gen.comscott@10gen.comjsr@10gen.com 50 51. This is a Split!This is a chunk chunk-+ dan@10gen.comscott@10gen.comjsr@10gen.com 51 52. -+ dan@10gen.comscott@10gen.comjsr@10gen.com 52 53. Split!-+ dan@10gen.comscott@10gen.comjsr@10gen.com 53 54. -adam@10gen.com1adam@10gen.comjared@10gen.com 1jared@10gen.com scott@10gen.com 1scott@10gen.com +1 Stored in the config serers Cached in mongos Used to route requests and keep cluster balanced 54 55. mongosconfig balancerconfigChunks!config1 234 13 14 15 16 25 26 27 2837 38 39 405 678 17 18 19 20 29 30 31 3241 42 43 449 10 11 1221 22 23 24 33 34 35 3645 46 47 48 Shard 1 Shard 2Shard 3 Shard 4 55 56. mongosconfig balancerconfigImbalance config 1 234 5 678 9 10 11 12 21 22 23 24 33 34 35 3645 46 47 48Shard 1 Shard 2 Shard 3 Shard 4 56 57. mongosconfig balancerconfig Move chunk 1 toconfig Shard 2 1 234 5 678 9 10 11 1221 2223 24 33 34 35 3645 46 47 48Shard 1 Shard 2 Shard 3 Shard 4 57 58. mongos configbalancer config config 1 234 5 678 9 10 11 1221 22 23 24 33 34 35 3645 46 47 48Shard 1 Shard 2Shard 3 Shard 458 59. mongos configbalancer config Chunks 1,2, and 3 have migrated config 4 5 678 1 23 9 10 11 1221 22 23 24 33 34 35 3645 46 47 48Shard 1 Shard 2Shard 3 Shard 459 60. 60 61. 1 1.Query arrives at mongos4mongos 2.mongos routes query to a single shard 3.Shard returns results 2 of query 3 4.Results returned to clientShard 1 Shard 2Shard 3 61 62. 1 1.Query arrives at mongos 4mongos 2.mongos broadcasts query to all shards 3.Each shard returns 2 results for query 2 2 3 3 3 4.Results combined and returned to clientShard 1 Shard 2Shard 362 63. 1 1.Query arrives at mongos 6 2.mongos broadcasts querymongos to all shards 5 3.Each shard locally sorts results 2 4.Results returned to 2 mongos 2 4 4 4 5.mongos merge sorts individual results 333 6.Combined sorted resultShard 1 Shard 2Shard 3 returned to client63 64. Inserts Requires shard db.users.insert({keyname: Jared, email: jsr@10gen.com})Removes Routed db.users.delete({ email: jsr@10gen.com})Scattereddb.users.delete({name: Jared})Updates Routed db.users.update( {email: jsr@10gen.com}, {$set: { state: CA}})Scattereddb.users.update( {state: FZ}, {$set:{ state: CA}} )64 65. By ShardRouteddb.users.find({email: jsr@10gen.com})KeySorted by Routed in order db.users.find().sort({email:-1})shard keyFind by non Scatter Gatherdb.users.find({state:CA})shard keySorted by Distributed merge db.users.find().sort({state:1})sortnon shardkey 65 66. 66 67. Data Center Primary Secondary Secondary 67 68. Data Center Primary Secondary Secondary hidden=truebackups 68 69. Active Data CenterStandby Data CenterPrimarySecondarySecondarypriority = 1 priority = 169 70. West Coast DC Central DCEast Coast DC Secondary Primary Secondary priority = 170 71. 71 72. History of Database Management (http://bit.ly/w3r0dv) EMC IDC Study (http://bit.ly/y1mJgJ) Gartner & Big Data (http://bit.ly/xvRP3a) SQL (http://en.wikipedia.org/wiki/SQL) Database Management Systemshttp://en.wikipedia.org/wiki/Dbms) Dynamo: Amazons Highly Available Key-value Store(http://bit.ly/A8F8oy) CAP Theorem (http://bit.ly/zvA6O6) NoSQL Google File System and BigTable(http://oreil.ly/wOXliP) NoSQL Movement whitepaper (http://bit.ly/A8RBuJ) Sample ERD diagram (http://bit.ly/xV30v)72 73. 73 74. 74 75. $project $match$limit$skip $unwind$group$sort{db.article.aggregate( title : this is my title , { $project : { author : bob , author : 1, posted : new Date () , tags : 1, pageViews : 5 ,}}, tags : [ fun , good , fun ] ,{ $unwind : "$tags" }, comments : [ { $group : {{ author :joe , text : this is cool } , _id : $tags,{ author :sam , text : this is bad }authors : { $addToSet : "$author" } ], }} other : { foo : 5 } );}75 76. 76 77. Input data MAP Intermediate data REDUCE Output data11223377 78. Input data MAP Intermediate data REDUCE Output data11223378 79. Impossible for a distributed computer system tosimultaneously provide all three of the followingguarantees Consistency - All nodes see the same data at the same time. Availability - A guarantee that every request receives a response about whether it was successful or failed. Partition tolerance - No set of failures less than total network failure is allowed to cause the system to respond incorrectly 79