Migrating from PostgreSQL to MySQL Without Downtime
-
Upload
matt-graham -
Category
Technology
-
view
8.063 -
download
3
description
Transcript of Migrating from PostgreSQL to MySQL Without Downtime
![Page 1: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/1.jpg)
Migrating from PostgreSQL to MySQL... without downtime
Matthew GrahamPercona Live NYC 2011
![Page 2: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/2.jpg)
• 8 million members
• 800k active shops
• 1 billion page views / month
• $38.7 million goods sold
• 2 million items sold
• 1.8 million new items listed
• 390k new members joined
April 2011Total
Global marketplace forbuying and selling handmade goods
![Page 3: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/3.jpg)
• 8 million members
• 800k active shops
• 1 billion page views / month
• $38.7 million = $900 / min
• 2 million items sold
• 1.8 million new items listed
• 390k new members joined
April 2011Total
Global marketplace forbuying and selling handmade goods
![Page 4: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/4.jpg)
Downtime = -$
![Page 5: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/5.jpg)
![Page 6: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/6.jpg)
$0m
$80m
$160m
$240m
$320m
2006 2007 2008 2009 2010
Gross Merchandise Sales
![Page 7: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/7.jpg)
![Page 8: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/8.jpg)
Reasons To Migrate
• Horizontal Scaling
• Reduce Types of Databases
• Licensing Costs
• Functional Partitioning
• Schema Refactor
![Page 9: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/9.jpg)
Deciding to Switchexpected quality
- transition-----------------
net quality > actual quality
![Page 10: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/10.jpg)
![Page 11: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/11.jpg)
![Page 12: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/12.jpg)
Source TargetMigration
![Page 13: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/13.jpg)
Why ?
![Page 14: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/14.jpg)
![Page 15: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/15.jpg)
• Migrate One Table at a Time
• Progressive Ramp Up
• Data Duplication During Transition
Foundational Principles
![Page 16: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/16.jpg)
1. Create Target Tables
2. Tee Writes
3. Backfill
4. Read from Target Tables
5. Wrap Up
5 Steps Per Table
![Page 17: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/17.jpg)
Create the Target
![Page 18: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/18.jpg)
Merge into the Target
![Page 19: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/19.jpg)
Separate the Target
![Page 20: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/20.jpg)
Source Only
Source
ApplicationWrites
Target
![Page 21: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/21.jpg)
Teed Writes
Source
ApplicationWrites
Target
![Page 22: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/22.jpg)
Generating IDsSourceSource TargetTarget
ID Name ID Name
1 Adam 1 Adam
2 Bill 2 Bill
3 Charlie 3 David
4 David 4 Charlie
Ticket Server: http://bit.ly/dbtickets
![Page 23: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/23.jpg)
Backfill
![Page 24: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/24.jpg)
Application Code Backfill
function backfill_user(user_id) { old_row = find_old_user(user_id); new_row = create_new_user(); new_row.id = old_row.id; ... store_new_user(new_row);}
![Page 25: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/25.jpg)
ETL Backfill
Extract
Transform
Load
Source Target
![Page 26: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/26.jpg)
Don’t Overload
• Script bulk inserts 100-2000 row batches
INSERT INTO account (id, firstname, lastname)VALUES(1, ‘Alan’, ‘Alda’),(2, ‘Barry’, ‘Bonds’),(3, ‘Charlie’, ‘Chaplin’);
![Page 27: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/27.jpg)
Backfill Speed
• Easier to Write • Faster Run Time
Application Code ETL
![Page 28: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/28.jpg)
• Easier to Write
• Less Likely to Get Out Of Sync
• Faster Run Time
• Needs an Extract Source
Application Code ETL
Backfill Extract
![Page 29: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/29.jpg)
• Easier to Write
• Less Likely to Get Out Of Sync
• Handles Duplicates from Multiple Executions
• Faster Run Time
• Needs an Extract Source
• REPLACE and INSERT ON DUPLICATE KEY UPDATE
Application Code ETL
Backfill Reruns
![Page 30: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/30.jpg)
Teed Writes then Backfillor
Backfill then Teed Writes
![Page 31: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/31.jpg)
Verification by Diff
$ diff_user 111222333target user row is missing
$ diff_user 123456789- source.user.address_state = ‘CA’+ target.user.address_state = ‘NY’
![Page 32: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/32.jpg)
Verification by DiffCOMPARING 200 ROWS From: 111222197
User Id: 111222333target user row is missing
User Id: 111222345- source.user.address_state = ‘CA’+ target.user.address_state = ‘NY’
SUMMARY: total rows with errors: 2/200
![Page 33: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/33.jpg)
Read from Target
![Page 34: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/34.jpg)
Progressive Ramp Up
Application Reads
100% 0%Source Target
![Page 35: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/35.jpg)
Progressive Ramp Up
Application Reads
99% 1%Source Target
![Page 36: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/36.jpg)
Progressive Ramp Up
Application Reads
95% 5%Source Target
![Page 37: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/37.jpg)
Progressive Ramp Up
Application Reads
75% 25%Source Target
![Page 38: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/38.jpg)
Progressive Ramp Up
Application Reads
0% 100%Source Target
![Page 39: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/39.jpg)
Ramp Up Example# global configurationuse_new_tables: employees: true percent: 1
# application codeif (enabled(‘use_new_tables’)) { $result = read_from_target();} else { $result = read_from_source();}
![Page 40: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/40.jpg)
Enabled or Disabled?
• Check cookies if user already assigned
• $enabled = configured threshold > random %
• Store $enabled in a cookie for future requests
![Page 41: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/41.jpg)
Continuous Deployment
• Ramp Up / Ramp Down
• Backfill Fixes
• Need Code Running on Prod to Proceed
• Makes it Easier
![Page 42: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/42.jpg)
No Foreign Keys
![Page 43: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/43.jpg)
Wrapping Up
![Page 44: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/44.jpg)
Analytical Data
![Page 45: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/45.jpg)
Teed Writes
Source
ApplicationWrites
Target
![Page 46: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/46.jpg)
Target Only
Source
ApplicationWrites
Target
![Page 47: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/47.jpg)
Things to Remove
• Code to use old tables
• Code to switch on configuration
• Configuration
• Drop the old tables... eventually
![Page 48: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/48.jpg)
1. Create Target Tables
2. Tee Writes
3. Backfill
4. Read from Target Tables
5. Wrap Up
5 Steps Per Table
![Page 49: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/49.jpg)
• Migrate One Table at a Time
• Progressive Ramp Up
• Data Duplication During Transition
Foundational Principles
![Page 50: Migrating from PostgreSQL to MySQL Without Downtime](https://reader033.fdocuments.in/reader033/viewer/2022052315/556253ffd8b42a1b4b8b4fc3/html5/thumbnails/50.jpg)
• Yes, we’re hiring! Specifically MySQL Opshttp://bit.ly/etsywantsawesome
• http://codeascraft.etsy.com
• http://twitter.com/lapsu
Questions?
Matthew GrahamPercona Live NYC 2011