What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability...

52
What’s new in Mongo 4.0 Vinicius Grippa Percona

Transcript of What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability...

Page 1: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

What’s new in Mongo 4.0

Vinicius GrippaPercona

Page 2: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

2

About me

• Support Engineer at Percona since 2017

Working with MySQL for over 5 years- Started with SQL Server

• Working with databases for 7 years

Page 3: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

3

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 4: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

4

A brief history

• MongoDB was developed in 2007 by organization called 10gen

• MongoDB has grown from being a niche database solution to the Swiss army knife of the NoSQL technologies

• It was developed as a PAAS (Platform as a service)

• Currently there is MongoDB Atlas Database as a Service (DBaaS)

Page 5: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

5

MongoDB Roadmap

Page 6: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

6

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 7: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Transactions Support

Page 8: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

8

Transactions Support

• Transactions feature has been in development since MongoDB version 3.0 (Two Phase Commit)

• Logical sessions added on Mongo 3.6

• MongoDB 4.0 adds support for multi-document transactions

Page 9: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

9

Transactions - ACID

• Atomicity

• Consistency

• Isolation

• Durability

Page 10: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

10

Transactions - ACID

• Atomicity

• "all or nothing" guarantee for multi-document transaction operations

• Data changes are only made visible outside the transaction if it is successful.

• If a transaction fails, all of the data changes from the transaction is discarded.

Page 11: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

11

Transactions - ACID

• Consistency

• Handled by MongoDB

• example: trying to change a value that fails schema validation, will cause inconsistent data

• (permitted transactions should not corrupt data)

Page 12: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

12

Transactions - ACID

• Isolation

Snapshot isolation level

• … creates a WiredTiger snapshot at the beginning of the transaction

• … uses this snapshot to provide transactional reads throughout the transaction.

Page 13: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

13

Transactions - ACID

• Durability

• When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee that it is returned after the transaction log is committed.

• Even if a crash occurs, MongoDB can recover according to the transaction log.

• If the {j: true} level is not specified, even after the transaction is successfully committed, the transaction may be rolled back (in case of crash recovery) – Needs journal enabled

Page 14: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

14

Transactions Support

• Start Transaction

1234

session = db.getMongo().startSession()session.startTransaction()session.getDatabase("percona").test.insert({today : new Date()})session.getDatabase("percona").test.insert({some_value : "abc"})

Page 15: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

15

Transactions Support

• Commit/Rollback Transaction

12

session.commitTransaction()session.abortTransaction()

Page 16: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

16

Transactions and Replica set

In the replica set configuration, an oplog will be recorded on commit

• … including all the operations in the transaction.

The slave node pulls the oplog and replays the transaction operations locally

(the document size limit is 16 MB, so whole transaction can not exceed 16MB)

Page 17: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

17

Transactions and Wired Tiger

Unified Transaction TimingWiredTiger has supported transactions for a long time

• (guarantee the modification atomicity of data, index, and oplog)

The problem was with timing:

• MongoDB used the oplog timestamps to identify the global order

• WiredTiger used the internal transaction IDs to identify the global order

MongoDB version 4.0 / WiredTiger 3.0 introduced transaction timestamps

• MongoDB can now explicitly assign a commit timestamp to the WiredTiger transaction (read "as of" a timestamp)

• When the oplog is replayed, the read on the slave node will no longer conflict with the replayed oplog, and the read request will not be blocked by replaying the oplog.

Page 18: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

18

Transactions - Conflict

foo:PRIMARY> session = db.getMongo().startSession()

session { "id" : UUID("b312c662-247c-47c5-b0c9-23d77f4e9f6d") }

foo:PRIMARY> session.startTransaction()

foo:PRIMARY> session.getDatabase("percona").test.update({trx : 0}, {trx: 2})

WriteCommandError({

"errorLabels" : [

"TransientTransactionError"

],

"operationTime" : Timestamp(1529675754, 1),

"ok" : 0,

"errmsg" : "WriteConflict",

"code" : 112,

"codeName" : "WriteConflict",

"$clusterTime" : {

"clusterTime" : Timestamp(1529675754, 1),

"signature" : {

"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),

"keyId" : NumberLong(0)

Page 19: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

19

Transactions – Write Concern

MongoDB is a distributed database- be aware of the different options for consistency.

Write concern describes the level of acknowledgement requested from MongoDB for write operations

For multi-document transactions, you set the write concern at the transaction level, not at the individual operation level.

Session.startTransaction({ writeConcern: { w: <level>} })

// w: 1, majority

If you commit using "w: 1" write concern, your transaction can be rolled back during the failover process.

Page 20: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

20

Transactions – Read Concern

The readConcern option allows you to control the consistency and isolation properties of the data read from replica sets and replica set shards.

For multi-document transactions, you set the read concern at the transaction level, not at the individual operation level.

If unspecified at the transaction start, transactions use the session-level read concern or, if that is unset, the client-level read concern.

Session.startTransaction({ readConcern: { level: <level>} })

Page 21: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

21

Transactions – Read Concern

Read concern "snapshot" is only available for multi-document transactions.

Multi-document transactions support read concern "snapshot" as well as "local", and "majority".

Session.startTransaction({ readConcern: { level: "snapshot"} })

Page 22: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

22

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 23: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Snapshot Read Concern

Page 24: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

24

Snapshot Read Concern

• readConcern option helps in achieving consistency, and isolation properties of the data. Snapshot Read Concern ensures that a consistent view of the data is returned to the client, irrespective of whether that data is being simultaneously modified by concurrent operations.

Page 25: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

25

Snapshot Read Concern

• If the transaction is not part of a causally consistent session, upon transaction commit with write concern "majority", the transaction operations are guaranteed to have read from a snapshot of majority-committed data.

• If the transaction is part of a causally consistent session, upon transaction commit with write concern "majority", the transaction operations are guaranteed to have read from a snapshot of majority-committed data that provides causal consistency with the operation immediately preceding the transaction start.

Page 26: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

26

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 27: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Security Enhancements

Page 28: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

28

Security Enhancements

SHA-2 Authentication

With MongoDB 4.0, authentication has been updated to the latest SHA-2 family (SHA-256), providing a stronger alternative to SHA-1

Page 29: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

29

Security Enhancements – Mongo 3.6

authenticationRestrictions: [ { clientSource: ["<IP>" | "<CIDR range>", ...] serverAddress: ["<IP>" | "<CIDR range>", ...] },

Page 30: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

30

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 31: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Non-Blocking Secondary Reads

Page 32: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

32

Non-Blocking Secondary Reads

Non-Blocking Secondary Reads

MongoDB previously blocked secondary reads while oplog entries were applied.

• when the replication threads were writing to the database the readers must wait

MongoDB 4.0 adds the ability to read from secondaries while replication is simultaneously processing writes.

Page 33: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

33

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 34: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Extension to Change Streams

Page 35: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

35

Extension to Change Streams

• Change streams introduced in version 3.6 helps applications to access real-time data changes (similar to tail the collection but better as it is replication aware).

• In 4.0 Change Streams can be configured to track changes across an entire database or whole cluster. Also, it will return a cluster time associated with an event (to provide an associated wall clock time for the event)

Page 36: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

36

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 37: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Data Type conversions

Page 38: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

38

Data Type Conversions

A new expression $convert has been added to the aggregation framework

https://docs.mongodb.com/manual/reference/operator/aggregation/convert/

• Helps for ETL workloads

• Also allow the MongoDB BI Connector to push down work to MongoDB Server (and avoid sending data over the wire)

Page 39: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

39

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 40: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Improved Sharding Operations

Page 41: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

41

Improved Sharding Operations

Operators can now list and kill queries running in a sharded cluster directly on a mongos node. Query terminated in any one of the shards will be reflected in all other shards too.

Page 42: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

42

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 43: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Hybrid Cursor Caching

Page 44: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

44

Hybrid Cursor Caching

• Improvements in cursor caching unlock up to 2x performance gains on deployments running with a large number of collections and enable operations like dropping a collection to execute without causing any performance degradation on other active collections.

Page 45: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

45

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 46: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

Slow Query logging on MongoS

Page 47: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

47

Slow Query logging on MongoS

In previous versions of MongoDB profiling can be enabled on the mongod. Now in 4.0 its easy to track the slow queries from mongos by enabling the profiler.

Page 48: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

48

Agenda

• Transactions Support

• Snapshot Read Concern

• Security enhancements

• Non-Blocking Secondary Reads

• Extension to Change Streams

• Data type conversions

• Improved Sharding Operations

• Hybrid Cursor Caching

• Slow Query Logging on MongoS

• The Future Features

Page 49: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

The Future

Page 50: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

50

Future Features

• Transaction compatible chunk migrations

• More extensive WiredTiger

• Transaction manager

• Global point in time reads

• Oplog applier prepare supports for transactions

Page 51: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

51

Rate My Session

Page 52: What’s new in Mongo 4.0 Percona Vinicius Grippas Ne… · 13 Transactions - ACID • Durability • When a transactions use WriteConcern {j: true} (default), MongoDB will guarantee

52

Thank You Sponsors!!