Self-healing databases: managing schema updates in the field
-
Upload
jonathan-oxer -
Category
Technology
-
view
4.434 -
download
1
description
Transcript of Self-healing databases: managing schema updates in the field
![Page 2: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/2.jpg)
The problem
Self-Healing Databases Jonathan Oxer <[email protected]>
Applications are not static.
New versions mean schema changes.
App / schema mismatches are bad.
Schema changes mean pain.
![Page 8: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/8.jpg)
Reasons for change
Self-Healing Databases Jonathan Oxer <[email protected]>
New tables required.
New columns required.
Alterations to columns.
Alterations to contents of tables.
![Page 9: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/9.jpg)
Failure modes
Self-Healing Databases Jonathan Oxer <[email protected]>
New tables required.
New columns required.
Alterations to columns.
Alterations to contents of tables.
“Unknown table”
“ Unknown column”
?
?
![Page 11: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/11.jpg)
Smart error trapping
Self-Healing Databases Jonathan Oxer <[email protected]>
1. Run queries blindly.
2. Detect failure conditions.
3. Fix them.
4. Profit!
![Page 12: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/12.jpg)
But...
Self-Healing Databases Jonathan Oxer <[email protected]>
...if you don't have a dbabstraction layer you're
stuffed!
![Page 13: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/13.jpg)
Build, Borrow or Steal
Self-Healing Databases Jonathan Oxer <[email protected]>
One centralquery executor
![Page 14: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/14.jpg)
MySQL errors
Self-Healing Databases Jonathan Oxer <[email protected]>
MySQL has built-in error reporting: use it!
In PHP:$errno = mysql_errno($link);$error = mysql_error($link);
Specify the link or you'll get the valuefrom the last opened connection, notthe last error from your connection.
![Page 15: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/15.jpg)
MySQL errors
Self-Healing Databases Jonathan Oxer <[email protected]>
Check for specific errors, such as:
1146: Table doesn't exist 1054: Unknown column
dev.mysql.com/doc/refman/5.0/en/error-handling.html
![Page 16: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/16.jpg)
Missing table
Self-Healing Databases Jonathan Oxer <[email protected]>
● Store reference schemas in app● Trap “1146” errors● Examine error to determine table name● Load reference schema● Create table● Rerun original query● Return result
The user never even notices a glitch :-)
![Page 17: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/17.jpg)
Missing table
Self-Healing Databases Jonathan Oxer <[email protected]>
Embed reference schemas into your app.
[modulename]/sql/articles.sql:
CREATE TABLE `articles` (`Serial` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,`Title` VARCHAR( 255 ) NOT NULL ,`Article` TEXT NOT NULL) ENGINE = MYISAM ;
![Page 18: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/18.jpg)
Missing column
Self-Healing Databases Jonathan Oxer <[email protected]>
● Record schema changes in “alter” file● Trap “unknown column” errors● Load and execute alter file● Rerun original query● Return result
No harm, no foul.
![Page 19: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/19.jpg)
Missing column
Self-Healing Databases Jonathan Oxer <[email protected]>
Make execution of “alter” file idempotent.
[modulename]/sql/alter.php:
if (!$dbase->field_exists(“news”, “Modified”)){
$s = “ALTER TABLE news ADD `Modified` TIMESTAMP NOT NULL”;
$dbase->query($s);}
![Page 20: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/20.jpg)
Missing trigger
Self-Healing Databases Jonathan Oxer <[email protected]>
Missing triggersare a problem:silent death
![Page 21: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/21.jpg)
Missing trigger
Self-Healing Databases Jonathan Oxer <[email protected]>
[modulename]/sql/alter.php:
if (!$dbase->trigger_exists(“UPDATESTOCK”)){
$s = “CREATE TRIGGER UPDATESTOCK ...”;$dbase->query($s);
}
![Page 22: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/22.jpg)
Missing procedure
Self-Healing Databases Jonathan Oxer <[email protected]>
● Put procedure additions in “alter” file● Trap “1106”, errors● Load and execute alter file● Rerun original query● Return result
(Note: trap “1107” and “1108” errors too, for handling altered procedures)
![Page 23: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/23.jpg)
Missing procedure
Self-Healing Databases Jonathan Oxer <[email protected]>
[modulename]/sql/alter.php:
if (!$dbase->procedure_exists(“DISCOUNTCALC”)){
$s = “CREATE PROCEDURE DISCOUNTCALC ...”;$dbase->query($s);
}
![Page 25: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/25.jpg)
Problem:
Self-Healing Databases Jonathan Oxer <[email protected]>
Multiple moduleinstances requiredata partitioning
![Page 26: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/26.jpg)
Solution:
Self-Healing Databases Jonathan Oxer <[email protected]>
Three-tierdynamic table
naming scheme
![Page 27: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/27.jpg)
Dynamic table names
Self-Healing Databases Jonathan Oxer <[email protected]>
1: Module instance2: Module name3: Specific table
hotstuff_news_articleshotstuff_news_comments
![Page 28: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/28.jpg)
Benefits
Self-Healing Databases Jonathan Oxer <[email protected]>
Storage of schema withmodule: error handler candeduce path from table.
Upgrade of tables when youdon't know their name.
![Page 29: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/29.jpg)
Schema templates
Self-Healing Databases Jonathan Oxer <[email protected]>
Placeholders in reference schemas
[modulename]/sql/articles.sql:
CREATE TABLE <articles> (`Serial` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,`Title` VARCHAR( 255 ) NOT NULL ,`Article` TEXT NOT NULL) ENGINE = MYISAM ;
![Page 30: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/30.jpg)
“Alter” templates
Self-Healing Databases Jonathan Oxer <[email protected]>
Make table names in “alter” file dynamic[modulename]/sql/alter.php:$articles = $instance.'_news_articles';if (!$dbase->field_exists($articles, “Modified”)){
$s = “ALTER TABLE $articles ADD `Modified` TIMESTAMP NOT NULL”;
$dbase->query($s);}
![Page 31: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/31.jpg)
Benefits
Self-Healing Databases Jonathan Oxer <[email protected]>
Stop caring about:
● App / schema mismatches● Knowing what tables are called● Telling users to run upgrade scripts
![Page 32: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/32.jpg)
Caveat:
Self-Healing Databases Jonathan Oxer <[email protected]>
this messes with
triggers and
procedures
![Page 33: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/33.jpg)
Is This A Fairy Tale?Is This A Fairy Tale?Is This A Fairy Tale?
Self-Healing Databases Jonathan Oxer <[email protected]>
Is This A Fairy Tale?Is This A Fairy Tale?Is This A Fairy Tale?
![Page 34: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/34.jpg)
Is this a fairy tale?
Self-Healing Databases Jonathan Oxer <[email protected]>
Technique in production use in theSiteBuilder web application frameworkand modules for more than 7 years:
● 1.2 million lines of PHP● 149 modules● 11,779 SQL statements● 1,247 embedded table schemas
![Page 35: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/35.jpg)
Is this a fairy tale?
Self-Healing Databases Jonathan Oxer <[email protected]>
Deployments include:
Siemens national intranet with over 5,000dynamically managed tables
Shaver Shop e-commerce system with tensof thousands of transactions / year
Gift Store Online with over 800k users
Brisbane Airport security credential systemwith 10k users / 30k cards
![Page 36: Self-healing databases: managing schema updates in the field](https://reader034.fdocuments.in/reader034/viewer/2022051412/54be00484a7959cb1f8b4572/html5/thumbnails/36.jpg)
Self-healing databases
Thankyou :-)These slides: jon.oxer.com.au/talks
We're hiring: www.ivt.com.au/jobs
Flames: Jonathan Oxer ([email protected])