Automated testing with

MongoDB and OffScale

Get your database under control

Omer GertelCTO @ OffScale

Managing databases is complicated

Many people are involved

Developers QA DBAs IT

Production

Dev

QA

Integration

Multiple environments

Multiple copies and

versions

We Help Companies…

Iterate Faster &Release Better Products

By…

Shortening product cycles with DB management

automation

Production

Dev

QA

Integration

Copy dataWrite codeWrite unit tests

Clone SchemaCopy dataTest the application

Clone schemaCopy dataRun integration

Copy dataWrite codeWrite unit tests

Clone SchemaCopy dataTest the application

Clone schemaCopy dataRun integration

DataGroveGit for databases

Production

Dev

QA

Staging

For developers, by developers

$ datagrove tag -n=new_tag

$ datagrove list

$ datagrove load -n=old_tag

$ datagrove unload

• Works automagically• Easy to integrate with existing tools• Your personal database time machine for FREE

Automated Testing

• Integration tests

• Build servers and continuous integration

• Migrations

–Some MongoDB specific issues

Automated Tests

• Isolated

• Repeatable

• Researchable

• Fast

Automated Tests

• For each test:

– Populate database

– Run test

– Drop database

What happens if a test breaks?

• Does it affect other tests?

• Does it stop the build cycle?

• How do you discover what went wrong?

• Is the bug related to the data?

• How can you reproduce the bug?

• What happens when you push a fix?

Automated Tests

• For each test:

– Load dataset

–Run test

– If test fails:

• Tag dataset at end of run

Schema Migrations

Schema Migrations

• Migrate ALL the objects

– Test before you run

• Support older structures in code

– Test code against old objects

– Have datasets ready tagged by version

Schema Migrations

Object Structure & Index Changes

Make MongoDB a first class citizen of the development cycle

Version management that doesn’t suck

Existing MongoDB SupportFor single node environments

$ datagrove load -n=tag_name

Create virtual mount point: /media/offscale(1-10 sec)

mongod --dpath= /media/offscale/media/offscale

Mongod

path-to-old-datapath-to-store-

changes

1

2

(optional)

Existing MongoDB SupportFor single node environments

$ datagrove tag -n=tag_name

Tag point in time in virtual drive(1-10 sec)

{fsync:1,lock:1} (using Java driver)

/media/offscale

Mongod

path-to-old-datapath-to-store-

changes

2

1

(optional)

db.$cmd.sys.unlock.findOne()3

Replica Set

Primary

Slave

Slave

Replica Set

Primary

Slave

Slave

Upcoming MongoDB Cluster SupportConceptual architecture for taking a snapshot

Replica Set

Primary

Slave

Slave

One node managed under OffScale, per replica set, allows for backup, recovery and creating new nodes

Configuration cluster

Cfg 1

Cfg 2

Cfg 3

One or more nodes managed by OffScale for backup and recovery of configuration servers

$ datagrove tag-n=tag_name

2Issue datagrove tag for configuration server

1Pause shard balancing

4Resume shard balancing

3 Issue datagrove tag, one shard at a time

• Data is a first class citizen in your environment

• Testing is important to keep your code working

• Iterate faster & release better products

Automated testing with OffScale & MongoDB

Now available at

http://off-scale.com/

Get your database under control