High Availability MySQL with DRBD and Heartbeat: MTV Japan ...
-
Upload
webhostingguy -
Category
Documents
-
view
1.415 -
download
5
Transcript of High Availability MySQL with DRBD and Heartbeat: MTV Japan ...
High-Availability MySQL with DR:BD and Heartbeat: MTV Japan mobile services
©2008 MTV Networks Japan K.K.
Let’s get started
©2008 MTV Networks Japan K.K.
About MTVNJ
Company
Profile
Company Name MTV Networks Japan K.K.
President Peter Robert Bullard
Address 2-8-2 Jingumae, Shibuya-ku, Tokyo 150-0001
TEL: 03-6434-3111 FAX: 03-6434-3150
Business
Description
MTV Networks Japan K.K. (MTVNJ) operates the mobile and PC
digital media services for all MTV Networks brands including MTV and
Nickelodeon in Japan. In addition to digital media services, MTVNJ
also operates a 24-hour Nickelodeon channel on CS/CATV, and the
consumer products business for Nickelodeon's character franchises in
Japan.
©2008 MTV Networks Japan K.K.
MTVNJ Mobile Brands
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project intro
©2008 MTV Networks Japan K.K.
{database}
DB Selection
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
Database requirements:• flexibility
• performance
• scalability
• redundancy
• support
• cost
Project summary:Merge all the database services for the MTV Flux and
MTV Mobile services on to a single database platform.
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
DB Selection
reqs/DB Oracle MS SQLMySQL
Community
MySQL
Enterprise
flexibility △/× × ○ ○
performance ○ ○ ○ ○
scalability ○/△ ○ ○ ○
redundancy ○/△ ○ ○ ○
support ○ ○ × ○
cost × × ◎ △
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
DB Selection
MTV Flux/MTV Mobile DB project
We chose MySQL Enterprise
©2008 MTV Networks Japan K.K.
DB Selection
MTV Flux/MTV Mobile DB project
What we looked for:• flexibility
• performance
• scalability
• redundancy
• support
• cost
MySQL Architectures:• Single/multi Master
• Master/Slave
• Cluster
• DR:BD/Heartbeat HA Master
©2008 MTV Networks Japan K.K.
MySQL Architectures
www www www
Single/multi Master
• cheap(!)
• easy configuration
• load ↑ ⇒ performance ↓
• high risk (if master dies, no more DB)
flexibility performance scalability redundancy support cost
○ ○/△ × × ○ ◎
Master
×
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
MySQL Architectures
www www www
Master/Slave
• some redundancy (read/writes separate)
• commonly used architecture
• still some risk (operator needed for outage recovery)
• manual failover for outages
Master Slave
×
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
flexibility performance scalability redundancy support cost
○ △ ○ △ ○ ◎
MySQL Architectures
www www www
Cluster
• redundant!
• can scale
• complicated configuration
• limited support in Japan (2007)
Cluster
△
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
flexibility performance scalability redundancy support cost
○ ○ ◎ ◎ ○/△ △
MySQL Architectures
www www www
DR:BD/Heartbeat HA Master
• redundant (master fully redundant)
• can scale (add salves)
• reduced outage risk
• can use failover for maintenance tasks
• complicated configurationPrimary Secondary
◎
MTV Flux/MTV Mobile DB project
©2008 MTV Networks Japan K.K.
flexibility performance scalability redundancy support cost
◎ ○/△ ○/△ ◎ ○ ○
MySQL Architectures
MTV Flux/MTV Mobile DB project
MySQL HA Master
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
HA Master
VIP
/etc/my.cnf
local
/var/lib/mysql
DRBD
/var/lib/mysql
DRBD
/etc/my.cnf
local
heartbeat
drbd
Application Application
Primary Secondary©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
VIP
DRBD
heartbeat
Application Web site or whatever
Manages/monitors DB process and mounts:
• provides application connection point (Virtual IP)
• monitors network status• watches the mysqld process
• manages the /var/lib/mysql mount point
mysqld (the database)
Synchronous device layer data replication
©2008 MTV Networks Japan K.K.
HA Master
MTV Flux/MTV Mobile DB project
VIP
/etc/my.cnf
local
/var/lib/mysql
DRBD
/var/lib/mysql
DRBD
/etc/my.cnf
local
heartbeat
drbd
Application Application
Primary Secondary
×Failover
processnetwork or mysqld
failure
1 – Application loosesconnection to mysqld
2 – heartbeat
detects the problem
©2008 MTV Networks Japan K.K.
HA Master
4 – Failover complete!3 – heartbeat does:
• pDB mysql dir unmount
• DRBD switch
• sDB mysql dir mount• sDB mysqld start
drbd
2 – drbd
fails over
1 – primary server fails!3 – heartbeat
detects outage
MTV Flux/MTV Mobile DB project
VIP
/etc/my.cnf
local
/var/lib/mysql
DRBD
/var/lib/mysql
DRBD
/etc/my.cnf
local
heartbeat
drbd
Application Application
Primary Secondary©2008 MTV Networks Japan K.K.
HA Master
Failover
processserver outage
4 – Failover complete!3 – heartbeat does:
• sDB mysql dir mount• sDB mysqld start
MTV Flux/MTV Mobile DB project
HA MySQL install
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
HA Master Install
local disk local disk
A Master
local disk
B Master Master
Dual Master Single Master
Old DB
architecture
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
VIP
/etc/my.cnf
local
/var/lib/mysql
DRBD
/var/lib/mysql
DRBD
/etc/my.cnf
local
heartbeat
drbd
New DB
architecture
HA Master Install
Primary Secondary©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
HA Master Install
Database migration work plan:• stop production services• perform full data backup (mysqldump)
• Replace MySQL architecture
• restore data from backup
• test database failover
• test application
• re-open production services
©2008 MTV Networks Japan K.K.
5 – test application
connectivity
local disk×local disk local disk
MTV Flux/MTV Mobile DB project
A Master B Master Master
Migration
processVIP
/etc/my.cnf
local
/var/lib/mysql
DRBD
/var/lib/mysql
DRBD
/etc/my.cnf
local
drbd
heartbeat
1 – remove old
database components
2 – install/configure
drbd and mysql
3 – install
heartbeat4 – configure VIP
HA Master Install
Primary Secondary
ש2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
HA Master Install
Actual migration work times:•production services stop – 00:10• full data backup (mysqldump) – 00:05
• MySQL architecture replacement – 01:30
• data restore – 00:10
• database failover test – 00:15
• application test – 03:00
• production services fully restored!
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
HA MySQL lessons learned
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
Lessons learned
Heartbeat:• no room for configuration mistakes
• pay special attention to peer connectivity
• run lots of tests before going to production!!!
Impact on our application:• essentially the same as a single master
• can use same HA for:
• single master - master/slave - multi-master
• failover fast enough that DB outages don’t affect application
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
Lessons learned
DR:BD:• requires separate partition (can be LVM)
• like heartbeat – test your configurations!!!
• always check your DRBD status after a failover
• peers sometime require a manual re-sync
mysqld:• can take advantage of redundancy to swap out nodes
• never try to run both primary and secondary at once!!
• leave all the redundancy to heartbeat
©2008 MTV Networks Japan K.K.
MTV Flux/MTV Mobile DB project
Lessons learned
Overall – good points:• we were able to deploy HA in a short maintenance window
• commercial support available (MySQL – LINBIT)
• MySQL consultative services were a big help
• also usable with MySQL Community
• can be part of other architectures
• multi-master - master/slave
Overall – weak points:• lots to learn, lots to test (heartbeat and DRBD)
• configuration and failover is complex• mysqld restart requires some extra steps
• all admins/vendors need to understand failover basics
©2008 MTV Networks Japan K.K.
HA MySQL in other MTVNJ environments
©2008 MTV Networks Japan K.K.
Questions?
©2008 MTV Networks Japan K.K.
DRBD related links
http://www.drbd.org/users-guide/s-heartbeat-r1.html
http://www.drbd.org/users-guide/s-heartbeat-crm.html
MTVNJ links
http://www.mtvn.jp/en/index.html
http://www.mtvn.jp/en/service.html
Patrick Bolduan
MTV Networks Japan KK
2-8-2 Jingu-mae, Shibuya-ku, Tokyo 150-0001
http://www.mtvjapan.com