Patterns and Tools for Database Versioning, Migration, Data Loading and Test Data.
-
Upload
alan-pinstein -
Category
Technology
-
view
5.898 -
download
0
description
Transcript of Patterns and Tools for Database Versioning, Migration, Data Loading and Test Data.
DB BEST PRACTICES FOR WEB APPS
Patterns and Tools for Database Versioning, Migration, Data Loading and Data for Tests
CodeWorks - Atlanta - 9/29/2009
WHO AM I?
Alan Pinstein@[email protected] Entrepreneur, current project is Neybor.comPHP Since 3.0 (~1997), C/C++/Obj-C, Cocoa, etcPHOCOA PHP Framework http://phocoa.comOn GitHub: http://github.com/apinsteinAtlanta Co-working:IgnitionAlley.com 151Locust.com
AGENDA
Common DB management problems that cause frustration and stress
Benefits of formalized solutions
Case Studies
Tools: MP, fixtures, rake, capistrano
Q & A
LIVE Q&A & POLLS
http://codeatl09.eventhive.com/
• Live, crowd-moderated questions & feedback• Live polls
[production-db] update user set password_md5 = md5(password);[production-db] alter table user drop column password;
COMMON DB FRUSTRATIONS
Where should I keep track of schema changes?
What changed since last production push?
Did I apply this patch already to this server?
Did I run the migrations on this copy of the app?
Did I run the update script after updating the schema?
Someone help the new developer set up the project on his machine...
BENEFITS OF BEST PRACTICES
Reproducibility
Reduction of complexity and risk
Reduced implementation time for DB changes
Increase in development throughput
Leverage from improvements in third-party tools
Community familiarity with workflows
CASE STUDY: BEFORE
$ vi updates.sql>> alter table add column foo boolean;$ git commit updates.sql -m “DB change”
$ ssh production@www0$ git pull && psql -U production mydb> \i updates.sql
$ echo “” > updates.sql
CASE STUDY: BEFORE
FRAGILE process.
Quickly falls apart with 2+ devs or 2+ servers.
Branches? Fugghetaboutit.
CASE STUDY: AFTER
$ mp -n # create new migration$ vi migrations/20090723_163051.php$ mp -m head$ git commit -a -m “Added new column.”
$ rake db:clone_production$ rake db:migrate # test on copy of real data
$ cap deploy # push to staging$ cap production deploy -s tag=staging-2009-09-01.1
CASE STUDY: AFTER
Each DB change encapsulated in a single migration scriptOnly code what’s unique to your appDeployment script automatically runs all necessary migration scripts on production database immediately after code deploysNothing to remember / eliminates human factorEasy to test, easy to reproduce
A version is the structure of a database at a point in timeA migration is code that changes the database structure between versions
VERSIONS VS MIGRATIONS
schema
version
2migration
schema
version
3migration
schema
version
1
SCHEMA VERSIONING
Tracking Blocks of Changes
Sharing Changes With Others
Deploying Changes to Staging & Production
TRACKINGSCHEMA CHANGES
Explicit
Expressing changes as SQL or meta DDLs
Implicit
Schema Diffs
MP: MIGRATIONS FOR PHP
Generic migrations infrastrructure for any project
Open Source (MIT)http://github.com/apinstein/mp/tree/master
Within 3 weeks, had:28 public clones1 fork8 watchers
GitHub rocks
MP ARCHITECTURE
DB Serveruser: me
db: myproj(no schema)
clean()
schema version
1M1->up()
schema version
2M2->up()
clean()
M2->down()
MP SETUP
$ mp -fMP - The PHP Migrator.No migrations dir found; initializing migrations directory at ./migrations.Current version: 0
$ mp -f -nCreated migration 20090805_201842 at ./migrations/20090805_201842.php.
$ ls -1 ./migrations20090801_175831.phpclean.phpversion.txt
BASIC USAGE $ mp -x “pgsql:dbname=myproject;user=me” -m=> migrates to latest version
TIP: alias myproj-mp=”mp -x ‘pgsql:dbname=myproject;user=me’”
$ myproj-mp -r -m=> resets to clean, then migrates schema to latest
$ myproj-mp -m redo=> run down() then up() on current migration=> useful when developing migrations
$ mp -f -l20090717_130521: Initial sql code for project20090717_145424: Add e-commerce subsystem20090804_145315: Add payment_method columns.
class Migration20090804_145315 extends Migration{ public function up() { $sql = <<<SQL alter table payment_method add column account_name text, add column nickname text, add column address_id integer, ADD CONSTRAINT fk_payment_method_address FOREIGN KEY (address_id) REFERENCES address(address_id) ON DELETE RESTRICT;SQL; $this->migrator->getDbCon()->exec($sql); } public function down() { $sql = <<<SQL alter table payment_method drop column account_name, drop column nickname, drop column address_id;SQL; $this->migrator->getDbCon()->exec($sql); } public function description() { return "Add payment_method columns account_name, nickname, and address_id."; }}
MP IN PRACTICE
NEVER edit a migration once it’s been relied upon by anyone else.
Ideally never edit once pushed to Version Control.
ALWAYS update clean.php to deal with each migration.
ALWAYS test migrations with redo and clean.
ALWAYS test migrations against a production clone before deployment.
MP CAVEATS
Post-schema change update scripts that rely on model code at a certain version can break
Migration A Adds blog.someFeatureFieldRuns model code to copy data from blog.oldFeatureField to blog.someFeatureField
Migration BRemoves blog.someFeatureField
Migration A will now fail since latest model code no longer knows about blog.someFeatureField
Branches with migrations that can run out-of-order could be problematic.
MP CAVEATS
Possible solutions:
Remove migration once it’s run everywhere
Catch errors and fail gracefully
Write upgrade scripts without using model code.
MP CLI$ php externals/mp/mp -hmigrate usage: VERSION STORAGE (must use one or the other) -x --dsn <PDO DSN> NOTE: if you supply a DSN then mp will automatically store the version number in your database in a table named public.mp_version. If you do not supply a dsn then you will need to use the -f flag to tell mp to use the filesystem for storage of the version. -f --version-in-file Store the version on the filesystem.
COMMANDS -r --clean Run clean before doing anything else. -n --create Create a new migration. -s --show-version Shows the current version. -V --set-version <version> Set the current version number maintained by mp to be <version>. Does not run migrations; used to initialize mp for an existing system. -m --migrate <TargetVersion> or 0|up|down|head|redo * redo is an alias for down + up -l --list List all migrations in VERSION - DESCRIPTION format
OPTIONS -v --verbose Verbose messages -q --quiet Supress all message -h --help Show this help -d --dir <MigrationDir>
Default command is "show status".
MP API$m = new Migrator(array( Migrator::OPT_PDO_DSN => $dsn, Migrator::OPT_VERSION_PROVIDER => new MigratorVersionProviderDB() ));$m->clean();$m->setVersion(‘xxx’);$m->migrateToVersion(Migrator::VERSION_HEAD);
Easily Integrate with your own framework, app, build system, etc.
When called programmatically your infrastructure is bootstrapped, so you have access to all of your codebase for grabbing DB connections, etc.
Optionally supply a delegate, available to clean(), up(), down()
DB BEST PRACTICES
DB Naming Conventions:Name: project, project_dev, project_stagingUser: userPassword: different for each machine
Script to clone production database to local/staging
ssh -C user@host \ 'pg_dump \ --clean --schema-only -U appUser -h 10.0.10.48 \ myProject' \| psql -U appUser myProject_dev
DATA LOADING - FIXTURES
Example - using WFFixture from phocoa php framework
Declarative, model-aware infrastructure makes setup of test rigs fast and easy
YAML for readability
Useful for loading real baseline data as well as test data
BLOG EXAMPLE
BlognamehomePageUrl
Postnamepoststatus
Multi-Tenant Blog App
Commentemailcomment
$fixture = <<<ENDBlog: reasonablyOpinionatedBlog: name: Reasonably Opinionated homePageUrl: 'http://reasonablyopinionated.com'Post: rant1: Blog: reasonablyOpinionatedBlog name: Database Migration is Hard post: Database Migration is Hard... status: structureTypeId: <?php BlogStatus::NOT_YET_PUBLISHED ?> Comment: # to-many - email: [email protected] comment: this post sucks - email: [email protected] comment: you suck! rant2: Blog: # to-one name: Some Other Blog homePageUrl: http://someotherblog.com name: Cantankerous rant on php 3 post: PHP3 is so old... status: structureTypeId: <?php BlogStatus::PUBLISHED ?>END; $results = WFFixture::WFFixture()->loadFromYaml($fixture);$this->rant1 = $results['rant1'];
WORKFLOW INTEGRATION
AUTOMATE!!!
Use build tools: phing, rake, capistrano etc
RAKE FORDB MANAGEMENT
rake db:backup_production # Backup production DB - for safety before deployment onlyrake db:clone_production # Restore Production DB to Local DBrake db:create # Create Databaserake db:drop # Drop Databaserake db:export:features # Export all feature data for later reloading.rake db:export:geographies # Export all geography data for later reloading.rake db:import:development_data # Load useful data for doing development.rake db:import:features # Load feature data.rake db:import:geographies # Load geography data.rake db:migrate # Migrate to latest db versionrake db:migrate:check # Verify project is already set up for MP.rake db:migrate:clean # Clean database.rake db:migrate:head # Update to latest version.rake db:migrate:rebuild # Rebuild db from clean state and update to latest version.rake db:model:rebuild # Rebuild model (reverse om convert-conf) from existing database state.rake db:rebuild # Rebuild the entire database, reload baseline data, and rebuild the propel ORM model.
MP ROADMAP
Support for pinning migrations to specific VCS tags?
Support for manifesting irreversible migrations?
Questions?