Use Your MySQL Knowledge to Become a MongoDB Guru
-
Upload
tim-callaghan -
Category
Technology
-
view
2.592 -
download
1
description
Transcript of Use Your MySQL Knowledge to Become a MongoDB Guru
®
Use Your MySQL Knowledge to Become a MongoDB Guru
Percona Live London 2013
Robert HodgesCEO
Continuent
Tim CallaghanVP/Engineering
Tokutek
Tuesday, November 12, 13
®
Our Companies
Robert Hodges• CEO at Continuent• Database nerd since 1982 starting with M204, RDBMS since
1990, NoSQL since 2012; designed Continuent Tungsten• Continuent offers clustering and replication for MySQL and
other fine DBMS types
Tim Callaghan• VP/Engineering at Tokutek• Long time database consumer (Oracle) and producer (VoltDB,
Tokutek)• Tokutek offers Fractal Tree indexes in MySQL (TokuDB) and
MongoDB (TokuMX)
Tuesday, November 12, 13
®
MongoDB -- The New MySQL
One Bad Thing about MongoDB
One Good Thing about MongoDB
Tuesday, November 12, 13
®
One Bad Thing about MongoDB
MySQL> select * from table1 where column1 > column2;> ... 5 row(s) returned
MongoDB> db.collection1.find({$field1: {gt: $field2}});> ReferenceError: $field2 is not defined
[current] MongoDB query language is <field> <operator> <literal>
Tuesday, November 12, 13
®
One Good Thing about MongoDB
Robert’s “ease of use” demo
Tuesday, November 12, 13
®
Today’s Question
How can you use your MySQL knowledge to get up to speed on MongoDB?
Tuesday, November 12, 13
®
Topic:
Schema Design
Tuesday, November 12, 13
®
How Do I Find Things in MongoDB?
8
mongod server
database
collectionBSON document
key/value pair
key/value pair
key/value pair
BSON document...
== mysqld
== MySQL schema
== MySQL table~ Sort of like a MySQL row
!= MySQL column
Tuesday, November 12, 13
®
How Do I Create a Table and Insert Data?
# Ruby CodeMongoClient.new("localhost"). db("mydb"). collection("sample"). insert({"data" => "hello world"})
9
Connect
Primary key generated
automatically
Use database
Choose collection
Insert data to materialize database and collection
Tuesday, November 12, 13
®
How Do I Change the Schema?
# Ruby CodeMongoClient.new("localhost"). db("mydb"). collection("sample"). insert({"data" => "hello again!",
"author" => “robert”})
10
Just add more data
Tuesday, November 12, 13
®
How Do I Validate Schema?
rs0:PRIMARY> db.samples.find(){ "_id" : 1, "data" : "hello world" }{ "_id" : 2, "daata" : "bye world” }{ "_id" : 3, "data" : 26.44 }
rs0:PRIMARY> show databaseslocal !2.0771484375GBmydb! 7.9501953125GBmydb1! 0.203125GB
11
Typo from an early run
Software bugs?
Tuesday, November 12, 13
®
How Do I Remove Data? (Part 1)
12
Drop a databasers0:PRIMARY> db.dropDatabase(){ "dropped" : "mydb", "ok" : 1 }
Drop a collectionrs0:PRIMARY> db.samples.drop()true
Drop a column?rs0:PRIMARY> db.foo.update( { author: { $exists: true }}, { $unset: { author: 1 } }, false, true )
Tuesday, November 12, 13
®
How Do I Remove Data? (Part 2)
13
(Remove documents based on TTL index)> db.samples.ensureIndex( {"inserted": 1}, {"expireAfterSeconds": 60})> db.samples.insert( {"data": "hello world", inserted: new Date()})> db.table.count()1...> db.table.count()0
(Capped collections do same with space)
Tuesday, November 12, 13
®
How Does MongoDB Do Joins?
14
It Doesn’t!(It is your job to denormalize or do
application level joins. This includes thinking about storage.)
Tuesday, November 12, 13
®
Topic:
Data Storageand Organization
Tuesday, November 12, 13
®
How is My Data Stored, Logically?
MongoDB storage is very similar to MyISAM
16
_id indexsecondary index(es)
collection data (documents)
etc.
Tuesday, November 12, 13
®
How is My Data Stored, Physically?
But it does look different in the file system.
17
• start MongoDB with “--directoryperdb” to put files in database folders• pro-tip : do this to gain IOPs by database
MyISAM<db>/<table>.frm<db>/<table>.myd <db>/<table>.myi
MongoDB<db1>.ns<db1>.1 .. <db1>.n<db2>.ns<db2>.1 .. <db2>.n
Tuesday, November 12, 13
®
How Much Memory Does It Use?
All of it!
18
Tuesday, November 12, 13
®
How does MongoDB Manage Memory?
• MyISAM–key_cache_size determines index caching–data is cached in Operating System buffers
• InnoDB– innodb_buffer_pool_size determines index/data
caching• MongoDB–memory mapped files–mongod grows to consume available RAM–good : no knob–bad : operating system is in charge of cache–bad : available RAM may change over time
19
Tuesday, November 12, 13
®
How Will It Perform for My Workload?
• It depends...–Determine your “working set”
oThe portion of your data that clients access most oftenodb.runCommand( { serverStatus: 1, workingSet: 1 } )
– If working set <= RAMoPerformance generally very goodoBe careful in high-concurrent-write use cases
– If working set >= RAMoLikely IO boundoSharding to the rescue!
20
Tuesday, November 12, 13
®
How Can Schema Affect Working Set?
• Field names are stored with the document– On disk and in memory
• Plan ahead, specially for large collections
21
BAD!
{ first_name: “Timothy”, middle_initial: “M”, last_name: “Callaghan”, address_line_1: “555 Main Street”, address_line_2: “Apt. 9” }
GOOD!
{ fn: “Timothy”, mi: “M”, ln: “Callaghan”, al1: “555 Main Street”, al2: “Apt. 9” }
Tuesday, November 12, 13
®
Topic:
Query Optimization
Tuesday, November 12, 13
®
How Does the Query Optimizer Work?
• MySQL–Optimizer find useable indexes for the query–For each index, optimizer asks the storage engine
oWhat is the cardinality for the given keys?oWhat is the estimated cost?
–The “best” plan is chosen and used for the query
• This occurs for every single query
23
Tuesday, November 12, 13
®
How Does the Query Optimizer Work?
• MongoDB–All candidate indexes run the query in parallel
o“candidate” meaning it contains useful keys
–As matching results are found they are placed in a shared buffer–When one of the parallel runs completes, all
others are stopped–This “plan” is used for future executions of the
same queryoUntil the collection has 1,000 writes, mongod restarts, or
there is an index change to the collection
24
Tuesday, November 12, 13
®
A Simple Yet Elegant Solution?
• No more wrestling with the optimizer• Hints are supported ($hint)–Force a particular index–http://docs.mongodb.org/manual/reference/
operator/meta/hint/• Easier since MongoDB does not support joins
25
Tuesday, November 12, 13
®
Topic:
Transactions
Tuesday, November 12, 13
®
MySQL Transactions and Isolation
mysql> BEGIN;...mysql> INSERT INTO sample(data) VALUES (“Hello world!”);mysql> INSERT INTO sample(data) VALUES (“Goodbye world!”);...mysql> COMMIT;
27
MyISAM locks table and commits each row immediately
InnoDB creates MVCC view of data; locks updated rows, commits atomically
Tuesday, November 12, 13
®
How Does MongoDB Implement Locking?
# Update data ranges of documents to # show effects of database lock. @col.update( {key => {"$gte" => first.to_s, "$lt" => last.to_s} }, { "$set" => { "data.x" => rand(@rows)}})
28
Test Total Requests/SecSingle thread updating single collection 197Two threads updating two collections, same DB 80 + 80 = 160Four threads updating two collections, same DB 29+29+30+30 = 118Two threads updating two collections, different DBs 190 + 179 = 369
Locks database
Tuesday, November 12, 13
®
How Does MongoDB Implement Isolation?
29
• MongoDB does not prevent threads from seeing partially committed data
• Example: Index changes can result in “double read” of data if query uses index while index is changing
• Experiment: Construct a test to:• Select from numeric index and count rows• Simultaneously update index to shift lower
values past end of previous high value
Tuesday, November 12, 13
®
How Does MongoDB Implement Isolation?
# Select values.count = [email protected](“k1” => {"$gte" => 120000}). each do |doc| count += 1endputs "Count=#{count}"
Count=50000Count=50000Count=100000 <--Index shifts over tailCount=50000Count=50000
30
# Run update to [email protected]( {"_id" => {"$exists" => true}}, {"$inc" => {“k1” => increment}}, {:multi => true})
Tuesday, November 12, 13
®
Topic:
Replication and HA
Tuesday, November 12, 13
®
Review of MySQL Replication
32
Master Slaveset globalread_only=1;
BinlogRelay Log
BinlogRelay Log
Master-master configuration for fast failover
Tuesday, November 12, 13
SECOND-ARY
®
How Does MongoDB Set Up Replication?
33
SECOND-ARY
PRIMARY
ReplicationReplication
Heartbeat
Heartbeat
Tuesday, November 12, 13
®
Where Is The Replica Set Defined?
$ mongo localhost...# rs0:PRIMARY> rs.config(){! "_id" : "rs0",! "version" : 8,! "members" : [! ! {! ! ! "_id" : 0,! ! ! "host" : "mongodb1:27017"! ! },! ! {! ! ! "_id" : 1,! ! ! "host" : "mongodb2:27017"! ! },! ! {! ! ! "_id" : 2,! ! ! "host" : "mongodb3:27017”! ! }! ]}
34
Tuesday, November 12, 13
®
How Do Applications Connect?
# Connect to MongoDB replica set. client = MongoReplicaSetClient.new( ['mongodb1', 'mongodb2', 'mongodb3'])
# Access a collection and add datadb = client.db("xacts")col = db.collection("data")col.insert({"data" => "hello world"})
35
Tuesday, November 12, 13
®
How Do You Read From a Slave?
# Connect to MongoDB replica set. client = MongoReplicaSetClient.new( ['mongodb1', 'mongodb2', 'mongodb3'], :slave_ok => true)
# Access a collection and select documents.db = client.db("xacts")col = db.collection("data")col.find()
36
Tuesday, November 12, 13
®
Where’s the Binlog?
rs0:PRIMARY> use localrs0:PRIMARY> db.oplog.rs.find().... sort({ts:-1}).limit(1)
{ "ts" : Timestamp(1383980308, 1), "h" : NumberLong("9112507265624716453"), "v" : 2, "op" : "i", "ns" : "xacts.data", "o" : { "_id" : ObjectId("527ddd116244f28f4592f6a8"), "data" : "hello world!" } }
37
Find last document in the OpLog
Tuesday, November 12, 13
®
How Do You Lock the DB to Back Up?
38
(= FLUSH TABLES WITH READ LOCK)rs0:SECONDARY> db.fsyncLock(){! "info" : "now locked against writes, use db.fsyncUnlock() to unlock",! "seeAlso" : "http://dochub.mongodb.org/core/fsynccommand",! "ok" : 1}...(tar or rsync data)...(= UNLOCK TABLES)rs0:SECONDARY> db.fsyncUnlock(){ "ok" : 1, "info" : "unlock completed" }
Tuesday, November 12, 13
®
How Do You Fail Over?
39
• Planned failover: update rs.config and save:rs0:SECONDARY> cfg = rs.conf()
rs0:SECONDARY> cfg.members[0].priority = 1rs0:SECONDARY> cfg.members[1].priority = 1
rs0:SECONDARY> cfg.members[2].priority = 2
rs0:SECONDARY> rs.reconfig(cfg)
• Unplanned failover: kill or stop mongod
Tuesday, November 12, 13
®
Topic:
Sharding
Tuesday, November 12, 13
®
How is Partitioning Like Sharding?
• MySQL partitioning breaks a table into <n> tables– “PARTITION” is actually a storage engine
• Tables can be partitioned by hash or range–Hash = random distribution–Range = user controlled distribution (date range)
• Helpful in “big data” use-cases• Partitions can usually be dropped efficiently–Unlike “delete from table1 where timeField < ’12/31/2012’;”
41
Tuesday, November 12, 13
®
How Does Partitioning Help Queries?
select * from big_table where column1 = 5;
42
select * from big_table where dateCol = ’10/12/2013’;
Aug-2013 Sep-2013 Oct-2013 Nov-2013
Partitioned big_table on dateCol by month.
Tuesday, November 12, 13
®
Can I Finally Scale My Workload Horizontally?
• MySQL partitioning is helpful, but is still constrained to a single machine
• MongoDB supports cross-server sharding–huge plus: it’s “in the box”–MySQL fabric is bringing something, we’ll see–Many other 3rd Party MySQL options exist
• Only shard the collections that require it• Each MongoDB shard is a replica set (1
primary and 1+ secondaries)
43
Tuesday, November 12, 13
®
What Does MongoDB Sharding Look Like?
44
Master Slave shard1
Master Slave shard2
Master Slave shardn
...
mongos1
mongosn
...
client app1
client app2
...
Tuesday, November 12, 13
®
How Does Sharding Help Queries?
select * from big_table where column1 = 5;
45
select * from big_table where dateCol = ’10/12/2013’;
Aug-2013 Sep-2013 Oct-2013 Nov-2013
Sharded big_table on dateCol by month.
shard1 shard2 shard3 shard4 shard...shard...
Tuesday, November 12, 13
®
How Do I Pick a Shard Key?
• MongoDB shards on one or more fields• Simple example– “orders” collection (customerId and productId)–1: shard on customerId
oeach order writes to a single shardo reads by customer on single shardo reads by product on entire cluster
–2: shard on productIdoeach order writes to several shardso reads by customer on entire clustero reads by product on single shard
–3: store everything twice and shard both waysoworst case for writesobest cast for reads (either is shingle shard)
46
Tuesday, November 12, 13
®
Topic:
Security
Tuesday, November 12, 13
®
How Secure is It?
• basic username/password• by database• roles– read = read any collection– readWrite = read/write any collection–dbAdmin = create index, create collection, rename
collection, etc.
48
Tuesday, November 12, 13
®
What About Advanced Security?
• Kerberos support in MongoDB Enterprise Edition
• SSL is supported, but–Note: The default distribution of MongoDB does
not contain support for SSL. To use SSL, you must either build MongoDB locally passing the “--ssl” option to scons or use MongoDB Enterprise.
49
Tuesday, November 12, 13
®
What Else is There to Learn?
• Tools - mongostat, mongo[export/import], mongo[dump/restore]• Aggregation Framework• Think SQL aggregate functionality• Map/Reduce
Tuesday, November 12, 13
®
What Should You Do?
Tuesday, November 12, 13
®
Summary
We liked...• Ease of install• Ability to just “jump in”
Look [out] for...• Query language (Tim says hang in there!)• You have to think about storage and queries in advance
Highly Recommended Reading• Karl Seguin’s “The Little MongoDB Book”
• http://openmymind.net/mongodb.pdf
• MongoDB’s “SQL to MongoDB Mapping Chart”• http://docs.mongodb.org/manual/reference/sql-comparison/
Tuesday, November 12, 13
®
Questions?
Tim CallaghanVP/Engineering, Tokutek
[email protected]@tmcallaghan
Robert HodgesCEO, Continuent
[email protected]@continuent
Tuesday, November 12, 13