1 / 191 - DataOps Barcelona | Databases · GR is a plugin for MySQL, made by MySQL and packaged...

1 / 191

2 / 1912 / 191

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purpose only, and may not beincorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up inmaking purchasing decisions. The development, release and timing of any features or functionality described for Oracle´s productremains at the sole discretion of Oracle.

3 / 191

about.me/lefred

Who am I ?

4 / 191

Frédéric Descamps@lefredMySQL EvangelistManaging MySQL since 3.23devops believerliving in Belgium 🇧🇪https://lefred.be

5 / 191

Group Replication: heart of MySQL InnoDB Cluster

6 / 191

Group Replication: heart of MySQL InnoDB Cluster

7 / 191

MySQL Group Replication

but what is it ?!?

8 / 191

GR is a plugin for MySQL, made by MySQL and packaged withMySQLGR is an implementation of Replicated Database StateMachine theoryPaxos based protocolGR allows to write on all Group Members (cluster nodes)simultaneously while retaining consistencyGR implements conflict detection and resolutionGR allows automatic distributed recovery

Automatic failoverAutomatic membership configuration

Adding/removing membersNetwork partitions, failures

Prevents data lossSupported on all MySQL platforms !!

Linux, Windows, Solaris, OSX, FreeBSDCompletely Open Source - GPL ! No license required to haveHigh Availability

but what is it ?!?

9 / 191

Group Replication is nice, but how does it work ?

10 / 191

it´s just ...

11 / 191

it´s just ...

12 / 191

it´s just ...

... no, in fact the writesets replication is synchronous and then certification and apply of the changes are local to each nodes andasynchronous.

13 / 191

it´s just ...

... no, in fact the writesets replication is synchronous and then certification and apply of the changes are local to each nodes andasynchronous.

not that easy to understand... right ? As a picture is worth a 1000 words, let´s illustrate this...

14 / 191

15 / 191

16 / 191

17 / 191

18 / 191

19 / 191

20 / 191

21 / 191

22 / 191

23 / 191

24 / 191

25 / 191

26 / 191

27 / 191

MySQL Group Replication (full transaction)

28 / 191

29 / 191

30 / 191

31 / 191

32 / 191

33 / 191

34 / 191

35 / 191

36 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocol

37 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State Machine

38 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocol (a variant of Mencius)

39 / 191

MySQL Group Communication System (GCS)MySQL Xcom protocolReplicated Database State MachinePaxos based protocol (a variant of Mencius)its task: deliver messages across the distributed system:

40 / 191

atomically

41 / 191

atomicallyin Total Order (Writes)

42 / 191

atomicallyin Total Order (Writes)

MySQL Group Replication receives the Ordered 'tickets' from this GCS subsystem.

43 / 191

Architecture, Stack, Core, ...

44 / 191

MySQL Group Replication - Architecture

Node Types:

R: Traffic routers/proxies: MySQL Router, HAProxy, ProxySQL...M: mysqld nodes participating in MySQL Group Replication

45 / 191

MySQL Group Replication - Stack

46 / 191

Server calls into the plugin through a generic interface

Most server internals are hidden from the plugin

Plugin interacts with the server through a generic interface

Replication plugin determines the fate of the commitoperation through a well defined server interfaceThe plugin makes use of the relay log infrastructure toinject changes in the receiving server

MySQL Group Replication - Core

47 / 191

Maintaining distributed execution contextDetecting and Resolving conflictsHandling distributed recovery

Detect membership changesDonate state if neededCollect state if needed

Proposing transactions to other membersReceiving and handling transactions from other membersDeciding the ultimate fate of transactions

commit or rollback

MySQL Group Replication - GR Plugin

48 / 191

The communication API (and binding) is responsible for:

Abstracting the underlying group communication system implementation fromthe plugin itselfMapping the interface to a specific group communication system implementation

The Group Communication System engine:

Paxos implementation (Similar to Paxos Mencius)Building block to provide distributed agreement between servers

MySQL Group Replication - GCS

49 / 191

Total Order

GTID generation

50 / 191

MySQL Group Replication uses MySQL replication framework bydesign:

binary logs

51 / 191

binary logs

relay logs

52 / 191

binary logs

relay logs

GTIDs: Global Transaction IDs

53 / 191

How does Group Replication handle GTIDs ?There are two ways of generating GTIDs:

54 / 191

AUTOMATIC: the transaction is assigned with an automatically generated id during commit. Where regular replication uses thesource server UUID, on Group Replication, the group name is used.

55 / 191

AUTOMATIC: the transaction is assigned with an automatically generated id during commit. Where regular replication uses thesource server UUID, on Group Replication, the group name is used.

ASSIGNED: the user assigns manually a GTID through SET GTID_NEXT to the transaction. This is common to any replicationformat and the id is assigned before the transaction starts.

56 / 191

57 / 191

Group Replication : Total Order Delivery - GTID

58 / 191

59 / 191

60 / 191

61 / 191

62 / 191

63 / 191

64 / 191

65 / 191

66 / 191

67 / 191

68 / 191

69 / 191

70 / 191

71 / 191

72 / 191

73 / 191

74 / 191

75 / 191

76 / 191

Group Replication : GTIDThe previous example was not totally in sync with reality. In fact, a writer allocates a block of GTID and when we have multiplewrites (multi-primary mode) all writers will use GTID sequence numbers in their allocated block.

The size of the block is defined by group_replication_gtid_assignment_block_size (default to 1M)

77 / 191

Group Replication : GTIDExample:

Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355

78 / 191

New write on an other node:

Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355,1000354

79 / 191

Let's write on the third node:

Executed_Gtid_Set: 0b5c746d-d552-11e8-bef0-08002718d305:1-355:1000354:2000354

80 / 191

Let's write on the third node:

And writing back on the first one:

81 / 191

82 / 191

done !

Return from Commit

83 / 191

Group Replication: return from commitAsynchronous Replication:

84 / 191

Group Replication: return from commit (2)Semi-Sync Replication:

85 / 191

(*): eventual

Group Replication: return from commit (3)Group Replication (*):

86 / 191

(*): before

87 / 191

(*): after

88 / 191

Does this mean we can have a distant node and always let it ack later?

89 / 191

90 / 191

Because the system has to wait for the noop (single skip message) from the “distant” node where latency is higher

The size of the GCS consensus messages window can be get and set from UDF functions:group_replication_get_write_concurrency(), group_replication_set_write_concurrency()

mysql> select group_replication_get_write_concurrency();+-------------------------------------------+| group_replication_get_write_concurrency() |+-------------------------------------------+| 10 |+-------------------------------------------+

91 / 191

Event HorizonGCS Write Consensus Concurrency

92 / 191

group replication write concurrency

93 / 191

94 / 191

95 / 191

96 / 191

97 / 191

98 / 191

99 / 191

100 / 191

conflict

Optimistic Locking

101 / 191

Group Replication : Optimistic LockingGroup Replication uses optimistic locking

102 / 191

during a transaction, local (InnoDB) locking happens

103 / 191

during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)

104 / 191

during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification

105 / 191

during a transaction, local (InnoDB) locking happensoptimistically assumes there will be no conflicts across nodes (no communication between nodes necessary)cluster-wide conflict resolution happens only at COMMIT, during certification

Let´s first have a look at the traditional locking to compare.

106 / 191

Traditional locking

107 / 191

Traditional locking

108 / 191

Traditional locking

109 / 191

Traditional locking

110 / 191

Traditional locking

111 / 191

Traditional locking

112 / 191

Optimistic Locking

113 / 191

Optimistic Locking

114 / 191

Optimistic Locking

115 / 191

Optimistic Locking

116 / 191

Optimistic Locking

117 / 191

Optimistic Locking

118 / 191

Optimistic Locking

The system returns error 149 as certification failed:

ERROR 1180 (HY000): Got error 149 during COMMIT

119 / 191

Optimistic Locking (2)The conflict detection can happen at two different stage:

Only one of the two transactions was sent to the Group and the other one is still running (local).

If both transaction were sent to the Group at almost the same time and both reached GCS/XCOM.

120 / 191

Optimistic Locking (3)

Only one transaction was sent to GCS and the other conflicting one is still local:

In this case, it's the high priority transaction mechanism of InnoDB that kills the local one:

If you try any statement in the transaction's session you will see:

ERROR: 1213: Deadlock found when trying to get lock; try restarting transaction

If you try to commit the transaction you will see:

ERROR: 1180: Got error 149 - 'Lock deadlock; Retry transaction' during COMMIT

121 / 191

Optimistic Locking (4)

Both transactions were sent to GCS:

The second transaction (the conflicting one) will return:

ERROR 3101 (40000): Plugin instructed the server to rollback the current transaction.

122 / 191

Such conflicts happen only when using multi-primary group !

not totally true in MySQL < 8.0.13 when failover happens

123 / 191

Drawbacks of optimistic lockinghaving a first-committer-wins system means conflicts will more likely happen when writing on multiple members with:

124 / 191

large transactions

125 / 191

large transactions

long running transactions

126 / 191

large transactions

long running transactions

hotspot records

127 / 191

128 / 191

Configurable Consistency Guarantees

Consistency Levels

129 / 191

Consistency: EVENTUAL (default)By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read thesame data on another node, it is eventually there.

130 / 191

Consistency: EVENTUAL (default)By default, there is no synchronization point for the transactions, when you perform a write on a node, if you immediately read thesame data on another node, it is eventually there.

Since MySQL 8.0.16, we have to possibility to set thesynchronization point at read or at write or both (globally or for asession).

131 / 191

Consistency: BEFORE (READ)

132 / 191

Consistency: AFTER (WRITE)

133 / 191

Consistency: BEFORE_AND_AFTER

134 / 191

135 / 191

can the transaction be committed ?

Certification

136 / 191

CertificationCertification is the process that only needs to answer the following unique question:

137 / 191

can the write (transaction) be committed ?

138 / 191

can the write (transaction) be committed ?based on yet to be applied transactions

139 / 191

can the write (transaction) be committed ?based on yet to be applied transactionssuch conflicts must come for other members/nodes

140 / 191

happens on every member/node and is deterministic

141 / 191

happens on every member/node and is deterministicresults are not reported to the group (does not require a new communication step)

142 / 191

pass: commit/queue to appy

143 / 191

pass: commit/queue to appyfail: rollback/drop the transaction

144 / 191

serialized by the total order in GCS/XCOM + GTID

145 / 191

serialized by the total order in GCS/XCOM + GTIDcost is based on trx size (# rows & # keys)

146 / 191

terminology

Write vs Writeset

147 / 191

Let's illustrate a table:

148 / 191

Now let's make a change

149 / 191

and at commit time:

150 / 191

WritesetsContain the hash for the rows PKs that are changed and in some cases the hashes of foreign keys or others dependencies thatneed to be captured (e.g. non NULL UKs). Writesets are gathered during transaction execution.

151 / 191

WritesetsContain the hash for the rows PKs that are changed and in some cases the hashes of foreign keys or others dependencies thatneed to be captured (e.g. non NULL UKs). Writesets are gathered during transaction execution.

WritesCalled also write values, refers to the actual changes. Write values are also gathered during transaction execution.

152 / 191

Writeset - examples+-------+-----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-------+-----------+------+-----+---------+-------+| id | binary(1) | NO | PRI | NULL | || name | binary(2) | YES | | NULL | |+-------+-----------+------+-----+---------+-------+

153 / 191

mysql> insert into t2 values (1,2);

154 / 191

pke: PRIMARY | test | t2 | 1 | 1 hash: 11853456929268668462

155 / 191

mysql> update t2 set name=3 where id=1;

156 / 191

pke: PRIMARY | test | t2 | 1 | 1 hash: 10002085147685770725pke: PRIMARY | test | t2 | 1 | 1 hash: 10002085147685770725

157 / 191

Writeset - examples (2)+-------+-----------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+-------+-----------+------+-----+---------+-------+| id | binary(1) | NO | PRI | NULL | || name | binary(2) | YES | UNI | NULL | || name2 | binary(1) | YES | | NULL | |+-------+-----------+------+-----+---------+-------+

158 / 191

mysql> insert into t3 values (1,2,3);

159 / 191

160 / 191

161 / 191

162 / 191

[after image][before image]

163 / 191

Certification

164 / 191

165 / 191

Enhanced support for large transactions

Message Fragmentation

166 / 191

Fragmenting an outgoing message

(1). If the message size exceeds the maximum size that the user allows(group_replication_communication_max_message_size), the member fragments the message into chunksthat do not exceed the maximum size.

(2). The member broadcasts each chunk to the group, i.e. forwards each chunk individually to XCom.

167 / 191

Reassembling an incoming message - first fragment

(2). The members conclude that the incoming message is actually a fragment of a bigger message.

(3). The members buffer the incoming fragment because they conclude the fragment is a chunk of a still-incomplete message.(Fragments contain the necessary metadata to reach this conclusion.)

168 / 191

Reassembling an incoming message - second fragment

(5). Same as step 3.

(6). Same as step 4.

169 / 191

Reassembling an incoming message - last fragment

1. The members conclude that the incoming message is actually a fragment of a bigger message.

2. The members conclude that the incoming fragment is the last chunk missing, reassemble the original, whole message, andprocess it.

170 / 191

Houston we have a problem !

Flow Control

171 / 191

Flow ControlIn Group Replication, every member send statistics about its queues (applier queue and certification queue) to the other members.Then every node decide to slow down or not if they realize that one node reached the threshold for one of the queue.

172 / 191

So when group_replication_�ow_control_mode is set to QUOTA on the node seeing that one of the othermembers of the cluster is lagging behind (threshold reached), it will throttle the write operations to the a quota that is calculatedbased on the number of transactions applied in the last second, and then it is reduced below that by subtracting the “over thequota” messages from the last period.

173 / 191

So when group_replication_�ow_control_mode is set to QUOTA on the node seeing that one of the othermembers of the cluster is lagging behind (threshold reached), it will throttle the write operations to the a quota that is calculatedbased on the number of transactions applied in the last second, and then it is reduced below that by subtracting the “over thequota” messages from the last period.

This mean that the threshold is NOT decided on the node being slow, but the node writing a transaction checks its threshold flowcontrol values and compare them to the statistics from the other nodes to decide to throttle or not.

174 / 191

Flow Control - on writer

>quota

175 / 191

Flow Control - on all members

176 / 191

Flow Control - configuration variablesAs in MySQL 8.0.16:

+-----------------------------------------------------+-------+| Variable_name | Value |+-----------------------------------------------------+-------+| group_replication_�ow_control_applier_threshold | 25000 || group_replication_�ow_control_certi�er_threshold | 25000 || group_replication_�ow_control_hold_percent | 10 || group_replication_�ow_control_max_quota | 0 || group_replication_�ow_control_member_quota_percent | 0 || group_replication_�ow_control_min_quota | 0 || group_replication_�ow_control_min_recovery_quota | 0 || group_replication_�ow_control_mode | QUOTA || group_replication_�ow_control_period | 1 || group_replication_�ow_control_release_percent | 50 |+-----------------------------------------------------+-------+

177 / 191

transaction's lifecycle in Group Replication

Summary

178 / 191

begin;

179 / 191

begin;update table1 set c = 999 where id =2;

180 / 191

begin;update table1 set c = 999 where id =2;update table1set b = "eee"where id = 3;

181 / 191

commit;

182 / 191

commit;

writesets+ gtid_event+ write values

183 / 191

commit;

certify

184 / 191

commit;

certifycertify

185 / 191

commit;

certifycertify

certify

186 / 191

commit;

commit finalized

certifycertify

certify

+ GTID

binlog

187 / 191

commit;

commit finalized

certifycertify

certify

+ GTID

binlog

+ GTID

188 / 191

commit;

commit finalized

certifycertify

certify

+ GTID

binlog

+ GTID

+ GTIDrelaylog

relaylog

189 / 191

commit;

commit finalized

certifycertify

certify

+ GTID

binlog

+ GTID

+ GTIDrelaylog

relaylog

binlog

190 / 191

Thank you !

Any Questions ?

191 / 191

1 / 191 - DataOps Barcelona | Databases · GR is a plugin for MySQL, made by MySQL and packaged...

Documents

Transcript of 1 / 191 - DataOps Barcelona | Databases · GR is a plugin for MySQL, made by MySQL and packaged...

MySQL Replication Update MySQL 5.5 (GA) & MySQL …assets.en.oreilly.com/1/event/56/MySQL Replication Update... · MySQL Replication Update MySQL 5.5 ... Be sure that slave has received

MySQL Introduction to the MySQL products. Agenda Company Overview Open Source & MySQL Momentum Why MySQL? MySQL OEM, Community & Enterprise offerings.

Using MySQL for Big Data Advantage - · PDF fileUsing MySQL for Big Data Advantage Integrate for Insight ... MySQL Cluster 7.2 MySQL Cluster Manager 1.4 MySQL Utilities 1.0.6 MySQL

MySQL Enterprise Monitor€¦ · mysql-monitor.log MySQL log file. Replication 1.dot The calculated MySQL server replication structure preferences.properties MySQL Enterprise Monitor

PHP and MySQL php-mysql PHP and MySQL

Why MySQL Enterprise Edition? MySQL Enterprise Edition ...€¦ · MySQL Enterprise Audit Adds regulatory compliance to MySQL applications (HIPAA, Sarbanes-Oxley, PCI, etc.) MySQL

Three DataOps strategies to monetize your data

DataOps: Let's learn how to work with physical objects in IRIS

Strata+hadoop data kitchen-seven-steps-to-high-velocity-data-analytics-with dataops

Upstream Downstream - FOSDEM · HA Audit MySQL 5.6 MySQL Workbench 6.1 MySQL Utilities MySQL Applier for Hadoop MySQL Workbench 5.2 & 6.0 MySQL Enterprise Oracle Certifications Windows

Screw DevOps, Let's Talk DataOps

MySQL Whitepaper MySQL HA Solutions

DataOps with Project Amaterasu

Getting DataOps Right

MySQL Strategy - itreegroup.eu · MySQL Strategy Morten Andersen, MySQL Enterprise Sales . ... MySQL Workbench 6.2 MySQL Database 5.7 DMRs * MySQL Cluster 7.4 DMR *Development Milestone

MySQL GUI Tools MySQL & PHP - University of …kopmanis/present/Mysql-PHP-pres.pdf · MySQL GUI Tools MySQL & PHP Jeff Kopmanis, Mathematics kopmanis@umich.edu

E-Book 4 Intro to DataOps - Morris & Opazo - Business Solutions in … · 2020-04-21 · DataOps is the alignment of People, Process, and Technologyto enable the rapid, automated,

Best Practices: Implementing DataOps with a Data Science Platform

Trilha Big Data - Amazon S3€¦ · Passo a Passo para implementar DataOps em projetos de Big Data Eduardo Hahn . Globalcode –Open4education + BuzzWord “DataOps is an automated,

1 / 65 - MySQL · MySQL High Availability made easy Frédéric Descamps - MySQL Community Manager - Oracle 4 / 65. ... GR provides fault tolerance GR enables update-everywhere setups