Continuous DB migration

Boris Trofimov

@b0ris_1

www.btrofimoff.com

Agenda

Dealing with changes

Perfect solution or Continuous DB migration

Carbon5 Introduction

Maven-driven mode

Embedded mode

Pros/Cos and best practices

Dealing with Changes

Applications evolve. God, bless bugs, refactoring and

business changes.

We love applications and configure CI, VCS in order to

track source code/application version.

But what’s about DB?

Dealing with Schema Change

FACT: Some modifications might require changes in DB

schema.

Dealing with Schema Change

FACT: Some modifications might require changes in DB

schema.

Changes are simple and risky.

Dealing with Schema Change

FACT: Some modifications might require changes in DB

schema.

Changes are simple and risky.

Simple changes:

– add one more table

– add index

Dealing with Schema Change

FACT: Some application modifications might require

changes in DB schema.

Changes are simple and risky.

Simple changes:

– add one more table

– add index

Risky changes:

– rename/add column

– change foreign key

– migrate data from one table/column to another

– denormalization on a fly

Dealing with Schema Change

What’s about old apps upgrade and multiple schema changes?

App v1

App v10

The Impact

Risks to lose data

Painful downtime

Risks to break application (see #1 and #2)

Resources, efforts and budget

Friday’s deployment in some teams

How people solve it

Deny Friday’s deployments

Manual deployment

Hibernate-like schema update + SQLexec

Self-developed tools based on version number approach

Perfect solution

Dedicated reusable framework

Every Feature request leads to separated SQL migration script.

Framework tracks which changes were not applied and applies

them.

Configurable time to launch apply procedure

Prevent Double changes

Have Change Log

Simple integration to existent project

Continuous DB Migration approach

min + automatic migration = Continuous DB Migration

Carbon V

Why Carbon V

DB-migration framework

Open Source project

Lightweight framework

Simple usage

How it works

Dedicated project folder for SQL migration scripts. This folder

should be available when framework takes control.

Each SQL migration script is pure SQL/DDL file with migration

SQL code for specific feature

Each Delta file should have name

YYYYMMDDHHMMSS_<FEATURE>.sql

When framework take control it checks and applies only new

changes.

C5 uses own JDBC connection.

Way to re-usue existent DataSources with connection params

No magic, just…

C5 creates own table “schema_version” inside your database

C5 controls this table by itself (no needs to create or update).

sql_file_name date duration

201405170900_user_auth.sql 2014-05-17 13:00:00

8 sec

201404130400_award_4565.sql 2014-04-14 14:00:00

6 sec

When to initiate migration procedure?

Maven-driven approach

Embedded mode

Maven-driven mode

Maven-driven mode

Configure Maven project

Update POM definition

Configure Explicit DB connection

Can be configured depending on specific maven profiles

(staging, production)

Add SQL migration files

Use Project/src/main/db/migrations directory

Launch it

performs migration

$ mvn db-migration:migrate

resets migration

$ mvn db-migration:reset

check if DB is up to date

$ mvn db-migration:validate

create new feature

$ mvn db-migration:new -Dname=new_feature

Use cases

Manual runs from developer machine

Part of Continuous Deivery process – Add just one more Maven action inside specific CI configuration before

deployment action

Embedded mode

Embedded mode

Schema Check is constituent part of application.

SQL changes are built-in into the application

Any checks and possible DB migration is performed every time

when application launches.

Dedicated bean to carry out this responsibility

Create application bean import com.carbonfive.db.migration.DataSourceMigrationManager;

import com.carbonfive.db.migration.DriverManagerMigrationManager;

import com.carbonfive.db.migration.MigrationManager;

import com.carbonfive.db.migration.ResourceMigrationResolver;

public class DBMigratorImpl {

DataSource dataSource;

...

public void init(){

DataSourceMigrationManager migrationManager = new

DataSourceMigrationManager(dataSource);

migrationManager.setMigrationResolver(new

ResourceMigrationResolver("classpath:/db/migrations"));

migrationManager.migrate();

}

}

Inject the bean into application context

Seamless integrated with Spring DI It is possible to initialize ResourceMigrationResolver

DataSourceMigrationManager directly though Spring DI without any line of code

Two places where bean might be integrated inside Spring context Before Persistent beans

After Persistent beans

Embedded vs maven driven

E: Consistent and grace application upgrade

E: No delays between deployment and schema upgrade

M: Simple integration with CI

M: External commands like validate & reset

M: Quick manual migrations (hotfixes for instance)

Carbon5 Benefits

Lightweight framework

Brilliant extensibility (thanks to reasonable design)

Support many DBMS in unofficial mode

Simple integration

Real continuous DB migration

Need more features?

Flyway framework

Rich support for different build tools

The same integration approach on Carbon V

Pluggable architecture

Java-based migrations (two options: through JDBC Connection &

Spring JDBCTemplate)

Extended service table

Best practices

Feature driven development (each SQL feature change in own

file)

Make sql files safe in order to prevent it from double execution

Keep database connection settings in single place (use shared

DataSource)

Do not make structure changes to the DB outside of your

framework

What’s about schemaless

?

NoSQL migration approach

No dedicated explicit migration procedure (solved on business

level in case of needs)

No shutdown or downtime

Rolling update – Every domain entity has version number

– your v2 application should handle the v1 db format

– Write code to convert entity to v2 on a fly (repository level)

– Write modified entities to v2 on demand

Some DBs like RavenDB are able to perform auto-migration

References

https://code.google.com/p/c5-db-migration/

http://flywaydb.org/

Martin Fowler Evolutionary Database Design

http://www.martinfowler.com/articles/evodb.html

Refactoring Databases

Presentation in a nutshell

Keep in mind this problem

Do not invent wheels, use external frameworks/techniques

And do not be afraid of Friday’s deployment then.

Thank you!

Q&A