PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub...
Transcript of PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub...
![Page 1: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/1.jpg)
How people build software
!
"
Automating Schema Changes using
Percona Live Open Source Database Conference 2017-04-27 | 12:50 PM - 1:40 PM | Ballroom D
![Page 2: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/2.jpg)
How people build software!
!
2
Tom Krouper
• @CaptainEyesight • ! @tomkrouper
• Database Infrastructure Engineer
![Page 3: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/3.jpg)
How people build software!
MySQL at GitHub• GitHub stores repositories in git, and uses MySQL
as the backend database for all related metadata: • Repository metadata, users, issues, pull
requests, comments etc. • Website/API/Auth/more all use MySQL.
• We run a few (growing number of) clusters, totaling
around 100 MySQL servers. • The setup isn’t very large but very busy. • Our MySQL service must be highly available.
3
!
![Page 4: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/4.jpg)
How people build software!
!
4
GitHub Online Schema
Transmogrifier
![Page 5: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/5.jpg)
How people build software!
gh-ost --host=replica.with.rbr.com --database="my_schema" --table="my_table" --alter="engine=innodb" --max-load=Threads_running=25 --critical-load=Threads_running=1000 --critical-load-interval-millis=3000 --throttle-http="http://freno/${cluster}"
—throttle-control-replicas=“replica1,replica2" --switch-to-rbr --exact-rowcount --concurrent-rowcount --panic-flag-file=/tmp/ghost.panic.flag --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --hooks-path=/path/to/hooks/ --hooks-hint=“@$(whoami)" ... [--execute]
!gh-ost
5
!"!
"!
"
https://github.com/github/gh-ost
![Page 6: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/6.jpg)
How people build software!
Migrationswithout the locking ALTER
6
!
![Page 7: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/7.jpg)
How people build software!
!Rails
7
$ cat db/migrate/20170427125019_add_some_id_to_example_table.rb # frozen_string_literal: true
class AddSomeIdToExampleTable < GitHub::Migration def up add_column :example_table, :some_id, :integer end
def down remove_column :example_table, :some_id end end
gh-ost ...
—table="example_table" —alter=“ADD COLUMN some_id int(11) DEFAULT NULL” ...
![Page 8: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/8.jpg)
How people build software!
!
8
Hubot Chatops automation
![Page 9: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/9.jpg)
How people build software!
!Chatops commands
9
.migration
.mysql
.qmtd
.queue for github
.deploy
.nagios
.xtrabackup
.mysqlproxy
.truck me
.where is
![Page 10: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/10.jpg)
How people build software!
Putting it all together
automate all the things… well, at least some of the things
10
!
credit: https://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html
![Page 11: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/11.jpg)
How people build software!
!Schema Migrations
11
$
![Page 12: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/12.jpg)
How people build software!
!Schema Migrations
12
$$ cat db/migrate/20170427125019_add_some_id_to_example_table.rb # frozen_string_literal: true
class AddSomeIdToExampleTable < GitHub::Migration def up add_column :example_table, :some_id, :integer end
def down remove_column :example_table, :some_id end end
![Page 13: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/13.jpg)
How people build software!
%!Schema Migration: Automation
13
Usage: .migration-queue <command> [<args>...] Workflow chatops for database migrations
show [needs_review|reviewed|scheduled] - Show all migrations in queue. add <pr-number> - Add a migration to the queue. schedule <pr-number> - Schedule a migration to be run. completed <pr-number> <migration-version> - Mark a migration as completed.
.migration-queue
![Page 14: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/14.jpg)
How people build software!
!Schema Migrations
14
$
![Page 15: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/15.jpg)
How people build software!
!Schema Migrations
15
$
![Page 16: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/16.jpg)
How people build software!
%!.migration queue show
16
![Page 17: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/17.jpg)
How people build software!
%!.migration queue add
17
![Page 18: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/18.jpg)
How people build software!
%!.migration queue schedule
18
![Page 19: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/19.jpg)
How people build software!
%!Schema Migrations
19
•CREATE
•DROP
![Page 20: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/20.jpg)
How people build software!
%!Schema Migrations
20
•CREATE
•DROP
![Page 21: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/21.jpg)
How people build software!
!Schema Migrations
21
%•CREATE
• Scheduling migrations outputs the CREATE TABLE statement
•DROP • Outputs chatops command to “RENAME TABLE“
![Page 22: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/22.jpg)
How people build software!
%!Schema Migrations
22
•ALTER
![Page 23: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/23.jpg)
How people build software!
%!Schema Migrations
23
•ALTER • Runs a script that calls gh-ost
![Page 24: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/24.jpg)
How people build software!
!gh-migrate-ghost
24
gh-ost \ --conf=/etc/mysql/gh-ost.cnf \ --host=$ghost_replica \ --database="$database_name" --table="$table_name" --alter="$ddl" \ --max-load=Threads_running=25 --critical-load=Threads_running=1000 \ --critical-load-interval-millis=3000 --chunk-size=$chunk_size \ --throttle-http="http://${freno}:8111/check/gh-ost/mysql/${cluster}" \ --max-lag-millis=500 --heartbeat-interval-millis=100 \ --switch-to-rbr --allow-master-master --cut-over=default \ --timestamp-old-table \ --exact-rowcount --concurrent-rowcount --default-retries=120 \ --panic-flag-file=/tmp/ghost.panic.flag \ --postpone-cut-over-flag-file=/etc/github/ghost.postpone.flag \ --serve-socket-file="$socket_file" \ --hooks-path=${hooks_directory} --hooks-hint="${hooks_hint_user}" \ --verbose \ $execute $extra_args
![Page 25: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/25.jpg)
How people build software!
%!Running Migration Automation
25
Manage running gh-ost migrations
Usage: .migration sup show brief status of running migrations .migration status show full status of running migrations .migration local show status of running migrations on localhost (useful from SSH logins)
.migration nice-ratio <ratio> set new nice-ratio for active migration .migration max-lag-millis <maxlag> set new max-lag milliseconds for active migration .migration max-load <status=val> set new status check for gh-ost to pause on, e.g., threads_running=30 .migration critical-load <status=val> set new status check for gh-ost to stop on, e.g., threads_running=1000 .migration throttle-control-replicas <hosts> set the replicas using commma delimited list of hosts. .migration throttle-http <url> change freno URL for running migrations. Empty string to disable throttling via freno.
.migration throttle|pause|suspend force throttling of active migrations .migration no-throttle|continue|resume terminate forced throttling (other throttling reasons may still apply)
.migration unpostpone|cut-over <table_name> cease to actively postpone; proceed to cut-over and completion .migration panic kill the running migration (requires magic_word)
.migration
![Page 26: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/26.jpg)
How people build software!
%!Running Migration Automation
26
.migration sup
![Page 27: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/27.jpg)
How people build software!
%!Running Migration Automation
27
.migration status
![Page 28: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/28.jpg)
How people build software!
%!Running Migration Automation
28
.migration nice-ratio <ratio>
.migration max-lag-millis <maxlag>
.migration max-load <status=val>
.migration critical-load <status=val>
.migration throttle-control-replicas <hosts>
.migration throttle-http <url>
![Page 29: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/29.jpg)
How people build software!
%!Running Migration Automation
29
.migration throttle
.migration resume
.migration cut-over <table_name>
.migration panic
![Page 30: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/30.jpg)
How people build software!
%!Running Migration Automation
30
.migration cut-over <table_name>
![Page 31: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/31.jpg)
How people build software!
%!Finishing the Migration
31
.migration-queue completed <pr>
![Page 32: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/32.jpg)
How people build software!
&!Merging the PR
32
•qmtd <url for pr> •… wait •update branch •wait for ci tests •merge (deploy)
![Page 33: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/33.jpg)
How people build software!
$ … &!Next Pull Request
33
![Page 34: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/34.jpg)
How people build software!
#!Automation Issues
34
•Migration Scheduler •Automatic gh-ost runs •Automatic pull request merge
![Page 35: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/35.jpg)
How people build software!
$!Lessons Learned
35
•Smaller changes •Ask for help •Dig in as deep as you can first •You can’t automate everything
![Page 36: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/36.jpg)
How people build software!
!Opportunity for improvements
36
•Everything is iterative •More automation ideas
• PR approval could include `.migration-queue add`. • merge `.migration-queue add` & `schedule` • rails level changes to avoid merge conflicts
![Page 37: PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub stores repositories in git, and uses MySQL as the backend database for all related](https://reader036.fdocuments.in/reader036/viewer/2022062602/5ec955b0cc38ac5a2a5ee477/html5/thumbnails/37.jpg)
How people build software
!
"
@CaptainEyesight [email protected]
Questions / Thanks
We’re Hiring:
Platform Data Engineer
https://bit.ly/platform-data