卢钧轶 MMM & MHA in DP
Transcript of 卢钧轶 MMM & MHA in DP
![Page 1: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/1.jpg)
MySQL High AvailabilityMMM & MHA in DP
卢钧轶
![Page 2: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/2.jpg)
Agenda
● Why we need a HA solution● MMM in DP● MHA in DP
![Page 3: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/3.jpg)
When We Need to Switch Server
● Master Server down● Master Server maintenance
○ Logical : DDL on big table ○ Physical: memory replacement
![Page 4: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/4.jpg)
Why We Need A Failover Solution
Failover without tool:● Notify DBA for MySQL Failure● Kill old Master● Find the latest Slave as Master Candidate● Execute change Master to on each Slave● Direct APP to new Master IP & restart
![Page 5: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/5.jpg)
Difficulties For MySQL Failover
Slave2file2:3000
Masterfile2:3000
Slave3file1:3000
Slave1file1:1000
![Page 6: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/6.jpg)
MySQL HA options
Stand-by● MMM● MHA + VIP● DRDB + (heartbeat,keepalive)
Proxy● MHA + HAProxy
Cluster● MySQL Cluster● Galera
![Page 7: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/7.jpg)
Why We Chose MMM in 2010
MMM MHA
Entire HA solution Yes only Master failover
VIP Built-in support Needs user defined script
CLI interface All included in mmm_control
Several Perl script
Deployment CPAN+Monitor+Agent CPAN+Manager+Agent+ SSH authorization
![Page 8: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/8.jpg)
MMM in DP
Used in Production since 201015+ M-M / M-M-S ClustersModifications in Source Code
https://github.com/cenalulu/mysql-mmm
![Page 9: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/9.jpg)
MMM
Is● VIP based HA solution● Message between Monitor & Agent● Auto Failover for both M/S
Is Not:● SQL router● Load Balancer● Active ( latest code release in 2010-05-07 )
![Page 10: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/10.jpg)
Simple Usage
Mastervip1
StandbyMaster
Web Server
![Page 11: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/11.jpg)
Basic Usage
Slavevip2
Mastervip1
StandbyMaster
Slavevip3
Web Server
Read
Write
![Page 12: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/12.jpg)
Role Based Machine Group
Frontend Groupvip1 & vip2
Backend Groupvip3 & vip4
Job Groupvip5
Slavevip3 / vip5
Mastervip1
Slavevip4
Mastervip2
![Page 13: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/13.jpg)
MMM architecture
Monitor
Slavevip3
Mastervip1
Slavevip4
Mastervip2
DB1(online):role1DB2(online):role2DB3(online):role3DB4(online):role4
checker
replication
role status
![Page 14: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/14.jpg)
MMM Monitor Logic
![Page 15: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/15.jpg)
MMM Slave Switch
process_command
Monitor
Slavevip3
Mastervip1
Slave
Mastervip2
DB1(online):role1DB2(online):role2DB3(online):role3DB4(offline)
mmm_control set_offline DB4
OFFLINE
![Page 16: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/16.jpg)
MMM Slave Switch
distributed_roles
Monitor
Slavevip3
Mastervip1
Slave
Mastervip2
DB1(online):role1DB2(online):role2DB3(online):role3,role4DB4(offline)
![Page 17: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/17.jpg)
MMM Slave Switch
send_status_to_agents
Monitor
Slavevip3
Mastervip1
Slave
Mastervip2
DB1(online):role1DB2(online):role2DB3(online):role3,role4DB4(offline)
role4 added
![Page 18: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/18.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Mastervip1
Slavevip4
Mastervip2
![Page 19: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/19.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Mastervip1
Slavevip4
Mastervip2set global read_only=1
![Page 20: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/20.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Master
Slavevip4
Mastervip2remove VIP
![Page 21: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/21.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Master
Slavevip4
Mastervip2
select MASTER_POS_WAIT()
![Page 22: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/22.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Master
Slavevip4
Mastervip2
show master status
![Page 23: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/23.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Master
Slavevip4
Mastervip2
change master to
Data Consistency ???
![Page 24: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/24.jpg)
How MMM Do Master Mark-Down
Monitor
Slavevip3
Master
Slavevip4
Mastervip1&vip2
vip1 online
![Page 25: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/25.jpg)
Cons of MMM
● Global read_only is hard to get on busy servers
● Slow slave bring more downtime● Change master on slaves is sequential● Communication between agent & monitor is
based on message
MMM is fundamentally broken -- Baron Schwartz: http://www.xaprb.com/blog/2011/05/04/whats-wrong-with-mmm/
![Page 26: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/26.jpg)
Con #1
Slow slave brings more downtimeVIP1 cannot be accessed during this period
Monitor
Slavevip3
Master
Job Slavevip4
Mastervip2
select MASTER_POS_WAIT()
![Page 27: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/27.jpg)
Eliminate Replication Lag
Logical Methods:● Chop● oak-chunk-update● Avoid peak hour jobs
Physical Methods:● Prefetch ● Disk upgrade
![Page 28: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/28.jpg)
兼具可靠性和安全性
Smart FTL – Flash Translation Layer- 热、冷数据动态跟踪- GC/WL 综合动态平衡,写放大因子最小化 - 双重数据保护机制- 最大化NAND Flash 寿命
完善容错数据保护机制- 高达 40bit/1KB ECC或更高- 读写,擦除出错处理及数据保护- 页面,坏块出错处理及数据保护
端到端数据保护- 企业级端到端数据链路保护 (Data path protection)- 多重数据完整性及正确性校验
掉电数据保护- 完善的突发掉电数据保护机制- 防止系统不正常关机的数据完整性和安全性
过热保护机制- 防止系统过热对系统造成不可恢复损伤
Shannon Systems Direct-IO™ PCIe SSD
6.4TB 全球单卡最大容量
67/9µs 超低随机读/写延迟
25W 更小峰值功耗 节省运维成本
![Page 29: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/29.jpg)
Shannon Direct-IOTM PCIe SSD G2T 测试数据
Capacity 800GB 1.2TB 1.6TB 3.2TB/6.4TB
Flash
闪存类型MLC MLC MLC MLC
Read Bandwidth
读宽带1.4GB/s 2.0GB/s 2.6GB/s 2.6GB/s
Write Bandwidth
写宽带1.2GB/s 1.8GB/s 1.8GB/s 1.9GB/s
Random Read Latency(4KB)
随机读延迟67us 67us 67us 67us
Random Write Latency (4KB)
随机写延迟9us 9us 9us 9us
Random Read IOPS(4KB)
随机读IOPS300,000 450,000 590,000 590,000
Random Write IOPS(4KB)
随机写IOPS310,000 460,000 480,000 480,000
Endurance
寿命10PB 15PB 20PB 40PB
![Page 30: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/30.jpg)
What Have We Done
However replication lag can still be exist Improvement:
Only wait for new master to catch up
![Page 31: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/31.jpg)
Con #2
Global read_only is hard to get on busy servers
1. Invalidates the Query Cache.2. Waits for all in-flight updates to complete and at the
same time it blocks all incoming updates. 3. Closes all open tables and expels them from the table
cache. Wait for all SELECT queries to complete. Incoming SELECT queries will get blocked.
4. Blocks COMMITs.
![Page 32: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/32.jpg)
What Have We Done
Manually check long sql before switchImprovement:● remove master vip before read_only● to kill sql before read_only
![Page 33: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/33.jpg)
What Have We Done
![Page 34: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/34.jpg)
How DP-MMM Do Failover
Monitor
Slavevip3
Master
Slavevip4
Mastervip2remove VIP
Clients
![Page 35: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/35.jpg)
How DP-MMM Do Failover
Monitor
Slavevip3
Master
Slavevip4
Mastervip2Kill Active Trx
Clients
![Page 36: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/36.jpg)
How DP-MMM Do Failover
Monitor
Slavevip3
Master
Slavevip4
Mastervip2
Clients
set global read_only=1
![Page 37: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/37.jpg)
How DP-MMM Do Failover
Monitor
Slavevip3
Master
Slavevip4
Mastervip2
MASTER_POS_WAIT()&
SHOW MASTER STATUS
Clients
![Page 38: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/38.jpg)
How DP-MMM Do Failover
Monitor
Slavevip3
Master
Slavevip4
Mastervip2&vip1
VIP1 added
Clients
![Page 39: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/39.jpg)
How DP-MMM Do Failover
Monitor
Slavevip3
Master
Slavevip4
Mastervip2
CHANGE MASTER TO
Clients
![Page 40: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/40.jpg)
Con #3
Why MMM is fundamentally broken:● Communication based on message between
monitor & agent● Agent manage its own state● Data Loss
![Page 41: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/41.jpg)
Example of Broken
mmm_control @cluster set_offline db1success!
![Page 42: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/42.jpg)
Cause of Broken
Monitor Instance 1State:
DB1: ONLINE (role: write)DB2: ONLINE (role: reader)
DB1
Monitor Instance 2State:
DB1: ONLINE (role: write)DB2: ONLINE (role: reader) DB2
You are ONLINE
You are ONLINEVIP2
![Page 43: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/43.jpg)
Cause of Broken
Monitor Instance 1State:
DB1: ONLINE (role: write,reader)DB2: OFFLINE
DB1
Monitor Instance 2State:
DB1: ONLINE (role: write)DB2: ONLINE (role: reader) DB2
You are OFFLINE
You are ONLINE
VIP2
mmm_control set_offline db2
![Page 44: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/44.jpg)
What Have We Done
Fix:multiple monitor instance bug.
But we cannot fix its architecture:mmm_control set_online db1 mean NOTHING but confirmation of command
![Page 45: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/45.jpg)
Cons of MMM
● Global read_only is hard to get on busy servers
● Slow slave bring more downtime● Change master on slaves is sequential● Communication between agent & monitor is
based on message
![Page 46: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/46.jpg)
Why We Chose MHA in 2014
MMM MHA
Data Loss Every Time after failover Almost none
Communicate with DAL
N/A User defined script
HA Solution without VIP
N/A Supported
![Page 47: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/47.jpg)
How MHA Do Failover
Monitor
Slave2
Master
Slave3Slave1
![Page 48: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/48.jpg)
How MHA Do Failover
Monitor
Slave2
Master
Slave3Slave1
Connect to all nodes and check: ● master is really dead● all slaves is the slave of dead master● last failover is ealier than $last_failover_minute
![Page 49: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/49.jpg)
How MHA Do Failover
Monitor
Slave2
Master
Slave3Slave1
![Page 50: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/50.jpg)
Call master_ip_failover_script
Monitor
Slave2file2:3000
Master
Slave3file1:3000
Slave1file1:1000
master_ip_failover_script Force Shutdown
![Page 51: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/51.jpg)
Find Latest Slave Relay Log Position
Monitor
Slave2file2:3000
Master
Slave3file1:3000
Slave1file1:1000
![Page 52: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/52.jpg)
Save Master Binlog if Alive
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:3000
Slave1file1:1000
file2:3000-4000
![Page 53: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/53.jpg)
Find Relay Log Base Slave
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:3000
Slave1file1:1000
file2:3000-4000
![Page 54: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/54.jpg)
Find Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:3000
Slave1file1:1000
file2:3000-4000
![Page 55: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/55.jpg)
Candidate Master Configuration
candidate_master: this server is prioritized in election
no_master: this server will not be new master in election
latest_priority:use order by block_name instead relay_log position in
election
![Page 56: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/56.jpg)
Candidate Master Election Rule
IF nothing special configuredelect relay log base slave
IF $latest_priority=1elect latest slave with candidate_master=1
ELSE elect cacndidate_master by name
IF $latest_priority=1elect latest slave
ELSE elect remaining slave
![Page 57: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/57.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:2000
Slave1file1:1000
file2:3000-4000
file1:2000-end
file2:0-3000
![Page 58: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/58.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:2000
Slave1file1:1000
file2:3000-4000
file1:2000-end
file2:0-3000
file2:3000-4000
![Page 59: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/59.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:2000
Slave1file1:1000
file2:3000-4000
file1:2000-end
file2:0-3000
file2:3000-4000
![Page 60: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/60.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file1:end
Slave1file1:1000
file2:3000-4000
file1:2000-end
file2:0-3000
file2:3000-4000
![Page 61: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/61.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file2:3000
Slave1file1:1000
file2:3000-4000
file1:2000-end
file2:0-3000
file2:3000-4000
![Page 62: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/62.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file2:4000
Slave1file1:1000
file2:3000-4000
file1:2000-end
file2:0-3000
file2:3000-4000
![Page 63: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/63.jpg)
Recover Candidate Master
Monitor
Slave2file2:3000
Masterfile2:4000
Slave3file2:4000
Slave1file1:1000
file2:3000-4000
file1:3000-end
file2:0-3000
file2:3000-4000
show master status
![Page 64: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/64.jpg)
Parallel Recover Slaves
Monitor
Slave2file2:3000
Slave3file2:4000
Slave1file1:1000
file2:3000-4000
file2:3000-4000
file1:1000-endfile2:0-3000file2:3000-4000
![Page 65: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/65.jpg)
Cons of MHA
No Slave ManagementCLI Not So Friendly compared to MMMSSH login without password within all nodes
![Page 66: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/66.jpg)
What Have We Done
MHA::SlaveFailoverMHA::NodeOnlinemasterha_wrapperUser Defined Script interact with DAL
![Page 67: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/67.jpg)
Interact With DAL
Monitor
Slave2
Master
Slave3Slave1
Web
Mark Down DB3
![Page 68: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/68.jpg)
Interact With DAL
Monitor
Slave2
Master
Slave3Slave1
Web
Mark Down DB3
![Page 69: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/69.jpg)
Interact With DAL
Monitor
Slave2
Master
Slave3Slave1
Web
Kill Connection
![Page 70: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/70.jpg)
Interact With DAL
Monitor
Slave2
Master
Slave3Slave1
Web
Stop Slave
![Page 71: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/71.jpg)
Interact With DAL
Monitor
Slave2 Slave3Slave1
WebMark Down Master
Master
read_only=ONread_only=ONread_only=ON
read_only=OFF
![Page 72: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/72.jpg)
Interact With DAL
Monitor
Slave2 Slave3Slave1
Web
Master
read_only=ONread_only=ONread_only=ON
Searching For Writable Server
![Page 73: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/73.jpg)
Interact With DAL
Monitor
Slave2 Slave3Slave1
Web
Master
read_only=ONread_only=ONread_only=ON
Searching For Writable Server
set read_only=ON
![Page 74: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/74.jpg)
Interact With DAL
Monitor
Slave2 Slave3
Slave1
Web
read_only=ONread_only=ON
read_only=ON
Searching For Writable Server
![Page 75: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/75.jpg)
Interact With DAL
Monitor
Slave2 Slave3
Slave1
Web
read_only=ONread_only=ON
read_only=OFF
Searching For Writable Server
set read_only=OFF
![Page 76: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/76.jpg)
Future
● GTID + mysqlfailover● Percona lock_binlog_for_backup
![Page 77: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/77.jpg)
GTID
● Possible to identify a transaction uniquely across the replication servers.
● Make the automation of failover process much easier. No calculations!
● At application level it is easier to do WRITE/READ split. ● Development of new automation tools isn’t a pain now.
![Page 78: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/78.jpg)
GTID
![Page 79: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/79.jpg)
Flush Tables With Read Lock
1. Invalidates the Query Cache.2. Waits for all in-flight updates to complete and at the
same time it blocks all incoming updates. 3. Closes all open tables and expels them from the table
cache. Wait for all SELECT queries to complete. Incoming SELECT queries will get blocked.
4. Blocks COMMITs.
Reference: http://www.mysqlperformanceblog.com/2014/03/11/introducing-backup-locks-percona-server-2/
![Page 80: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/80.jpg)
Lock Binlog For Backup
Unlike FTWRL, the LOCK TABLES FOR BACKUP statement:
* does not invalidate the Query Cache;* never waits for SELECT queries to complete
regardless of the storage engines involved;* never blocks SELECTs, or updates to InnoDB,
Blackhole and Federated tables.
http://www.mysqlperformanceblog.com/2014/03/11/introducing-backup-locks-percona-server-2/
![Page 81: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/81.jpg)
ConclusionTips for takeaway
![Page 82: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/82.jpg)
When to Use MMM
● Simple Master -- Master MySQL deployment environment
● Can afford several seconds data loss● VIP based HA solution already exist
![Page 83: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/83.jpg)
When to Use MMM - DP version
● Already using MMM● Slow Slaves included in cluster● Massive load on master
![Page 84: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/84.jpg)
When to Use MHA
● Cannot afford data loss● No requirement for Slave High Availability● Proxy already exists in current deployment● Interact with client side needed when switch/failover
![Page 85: 卢钧轶 MMM & MHA in DP](https://reader030.fdocuments.in/reader030/viewer/2022020622/61ec4b9c1e8b733b054ff92b/html5/thumbnails/85.jpg)
Q&AThanks!
https://github.com/cenalulu/mysql-mmm
卢钧轶/cenalulu @ dianping.com