© 2013 IBM Corporation
A Continuous Delivery Safety Net for Databases
2 © 2015 IBM Corporation
Speaking Today
• Eric is a DevOps Evangelist with IBM where he helps customers get the most out of their build, deploy and release processes.
• Today he works with customers and industry leaders to find the best ways of adopting continuous delivery and DevOps.
• Eric is the author of Application Deploy and Release for Dummies.
• @EricMinick
Eric Minick
• Yaniv is the Co-Founder and CTO of DBmaestro, the leading provider of DevOps for Database solutions which enable control of database development and deployment.
• Yaniv has spent the last years raising awareness about the challenges around database development and deployment, and how to support database Continuous Delivery.
• @DBmaestro
Yaniv
Yehuda
3 © 2015 IBM Corporation
Why Waterfall Fails
4 © 2015 IBM Corporation
Continuous Delivery helps us learn faster
Automated flow from Build to “ready for prod”
Push button release to production
Lots of feedback Emphasis on always
shippable
Themes
5 © 2015 IBM Corporation
“Classic” CD
build dev
test
system
test UAT sign-off staging prod
6 © 2015 IBM Corporation
The Build Pipeline
build dev
test
system
test UAT sign-off staging prod
for (env in testEnvironments) {
deploy( build, env );
runTests (env.testType, env);
}
7 © 2015 IBM Corporation
Composite apps: many tiers & components
8 © 2015 IBM Corporation
Composite apps: many tiers & components
An Application might have
dozens of components
9 © 2015 IBM Corporation
Composite apps: many tiers & components
An Application might have
dozens of components
Delivered by Different Teams
10 © 2015 IBM Corporation
Composite apps: many tiers & components
Which build does “Login” test?
11 © 2015 IBM Corporation
Build pipelines in composite applications
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
build dev
test
system
test UAT sign-off staging prod
12 © 2015 IBM Corporation
Combine Versioned Stuff into Versioned App
• Manage a set of tested component versions as a single unit
• Guarantee the validated versions of related components are
deployed together
• Ease of promoting applications between environments
1
1
2
3
UAT Stage PROD
Promote Promote
Web
Mid. Code
Mid. Config
DB
Snapshot
13 © 2015 IBM Corporation
Expanding “All the Stuff”
Systems of Record
– z/OS
– iSeries
Mobile
– iOS / Android
– Mobile Testing
Infrastructure
– Treating environment as part of Application
14 © 2015 IBM Corporation
Combine Versioned Stuff into Versioned App
• Manage a set of tested component versions as a single unit
• Guarantee the validated versions of related components are
deployed together
• Ease of promoting applications between environments
1
1
2
3
UAT Stage PROD
Promote Promote
Web
Mid. Code
Mid. Config
DB
Snapshot
15 © 2015 IBM Corporation
• Old adage but true
• The database is often neglected and therefore can become the weakest link
• Manual processes
• Database/Code Silos exist…
• Don’t always communicate effectively
• Need to follow same procedures & best practices
• Essential from a compliance and business point of view
• Should be the strongest link
The Weakest Link In a Chain ???
16 © 2015 IBM Corporation
Recently Conducted DBmaestro Survey – Over 150 Companies
– Over 200 participants
Why is that happening?
Continuous Delivery Moving Ahead!
17 © 2015 IBM Corporation
Why Continuous Delivery?
18 © 2015 IBM Corporation
Why?
What about the database?
19 © 2015 IBM Corporation
Mistrust…
So why not move forward?
20 © 2015 IBM Corporation
Mistrust…
21 © 2015 IBM Corporation
What is the problem?
Root Causes for issues: – Challenging manual version control process
– Static deployments code overrides
– Dynamic deployments unaware of version control
– No release automation red-flags – don’t know when to stop the line…
22 © 2015 IBM Corporation
Mistrust in version control…
Version Control Process
(file based)
Development Process
Check-Out Script
Modify Script
Get updated Script from
DB
Check-In Script
Compile Script in DB
Debug Script in DB
23 © 2015 IBM Corporation
Version Control Process
(file based)
Development Process
Check-Out Script
Modify Script
Get updated Script from
DB
Check-In Script
Compile Script in DB
Debug Script in DB
A
A’
Mistrust in version control…
24 © 2015 IBM Corporation
Version Control Process
(file based)
Development Process
Check-Out Script
Modify Script
Get updated Script from
DB
Check-In Script
Compile Script in DB
Debug Script in DB
?
? ?
?
A
A’
Mistrust in version control…
No single source of truth…
25 © 2015 IBM Corporation
Scripts & Version Control
Challenges…
– Code-overrides
– Working on the wrong revisions
– Scripts do not always find their way to the version control solution
– Out of process updates go unnoticed
– Hard to locate outdated update scripts
Playing safe? what we really need:
– The actual code of the object
– The upgrade script
– A roll-back script
Scripts
– Hard to test in their entirely (holistically)
– Hard to test due to colliding dependencies
– Need to run in a specific order…
– Much harder to deal with project scope changes
26 © 2015 IBM Corporation
1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Mistrust in static scripting automation…
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.1 1.1 1.4 1.7 ? 1.1.1
27 © 2015 IBM Corporation
1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.1 1.1 1.4 1.7
1.1 1.2
1.2 1.3
1.3 1.4
? 1.1.1
Mistrust in static scripting automation…
28 © 2015 IBM Corporation
1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.1 1.4 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
? 1.1.1
Mistrust in static scripting automation…
29 © 2015 IBM Corporation
1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.1 1.4 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of
Process
Change
? 1.1.1
Mistrust in static scripting automation…
30 © 2015 IBM Corporation
X 1.1 1.1.1 1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.1 1.4 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of
Process
Change
X
X
X
X
X
? 1.1.1
X
Mistrust in static scripting automation…
31 © 2015 IBM Corporation
Scripts are static…
Scripts, unless super sophisticated:
– Unaware of changes made in the target environment
– Time passed from their coding to the time they are run
– Potentially overriding production hot-fixes or work done in parallel by another team
Content changes are very hard to manage
– Metadata & lookup content does not practically fit into the VC
– In most cases they are simply not managed
32 © 2015 IBM Corporation
Gaining Trust!
Coordinated Process Traceability
Start in the Beginning
No Out-of-Process Changes
Impact Analysis
Automation
Task Based Development
Well Defined Processes
33 © 2015 IBM Corporation
Version Control - One Enforced Process
34 © 2015 IBM Corporation
Dealing with challenges…
Integrated Database Version Control process – Leverage proven version control best practices
• Forcing check in & out for changes
• Labels
• etc..
– No code-overrides
– Always working with the correct revision
– All changes are documented
– Always know who did what, when, why and from where
– No out-of-process changes
– Supporting structure, code and content
No time spent on manual coding of the change scripts
35 © 2015 IBM Corporation
Bonus points – dealing with scope changes
Task based development… – Correlate each database change with a change request
• Task ID
• Work Item
• Trouble Ticket
• CR
• etc…
…enables task based deployments – Partial deployments (a feature, a collection of bugs, etc…)
– Scope changes easily synced between code and database
36 © 2015 IBM Corporation
For deployment automation we need:
Repeatability & Safety
Using tools make sense …
37 © 2015 IBM Corporation
Build & Deploy On Demand
1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.1 1.4
1.1 1.1 1.1 1.4 1.7 1.1.1 1.7 1.1.1 1.7
38 © 2015 IBM Corporation
Build & Deploy On Demand
1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1 1.1 1.1 1.4 1.7 1.1.1 1.7 1.1.1 1.7
39 © 2015 IBM Corporation
Build & Deploy On Demand
1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1 1.1 1.1 1.4 1.7
Out of
Process
Change
1.1.1 1.7 1.1.1 1.7
40 © 2015 IBM Corporation
Build & Deploy On Demand
1.1 1.2 1.3 1.4 1.5 1.6 1.7
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
1.1 1.1 1.1 1.4 1.7
File Based
Version
Control
Out of
Process
Change
1.1.1 1.7 1.1.1 1.7
41 © 2015 IBM Corporation
Build & Deploy On Demand
1.1 1.2 1.3 1.4 1.5 1.6 1.7
*
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to
defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
1.1 1.1 1.1 1.4 1.7
File Based
Version
Control
Out of
Process
Change
1.1.1 1.7 1.1.1 1.7
42 © 2015 IBM Corporation
Using tools
Test cases using compare & sync tools:
An index exists in source (QA) but not in target (Production)
What should we do? Add the index or not?
43 © 2015 IBM Corporation
Compare & Sync tools
Safe to automate?
Sure… (?)
44 © 2015 IBM Corporation
Compare & Sync tools
Safe to automate?
No. Requires manual inspection…
45 © 2015 IBM Corporation
Safe?
Simple, right? NO! we are going to BREAK production without even
knowing…
46 © 2015 IBM Corporation
Why break production???
A compare & sync tool: Is unaware of any changes that occurred before the time it ran
Has no knowledge of changes that took place at the target environment
Does not leverage version control for more information
Unable to deal with conflicts & merges between different teams
Requires manual inspection
Requires detailed knowledge regarding each change as part of the process
Mistrust AGAIN…
So…no automation… as we fear automating problems into production and a major risk!!!
47 © 2015 IBM Corporation
We need to leverage version control into deployment decisions…
48 © 2015 IBM Corporation
Safety Net For Deployment Automation
Static script based deployment
49 © 2015 IBM Corporation
Deploying Changes if Needed
Development Baseline
Previous Label /
Production Golden Copy
Production
If we had the index in the baseline =>
we should take it down from production…
(Deploy Change)
50 © 2015 IBM Corporation
Or Protecting Target Environment…
Development Baseline
Previous Label /
Production Golden Copy
Production
BUT… If no index in baseline =>
we should protect the NEW index on production!!!
(Protect Target)
51 © 2015 IBM Corporation
Deployment Automation
And Merge!!!
52 © 2015 IBM Corporation
Conflict Resolving – Database Code
53 © 2015 IBM Corporation
Impact Analysis – Code & Content
54 © 2015 IBM Corporation
Impact Analysis! not Damage Control…
55 © 2015 IBM Corporation
Safety Net For Deployment Automation
Database Safe Deployment Automation: Leverages version control (baselines & previous revisions)
Has a flexible scope (deploy multi schema to single task or work item)
Can be run as a batch process (repeatable & consistent)
Integrates to ALM (labels, CRs, Continuous Integration & Delivery)
Deals with conflicts & merges to match code agility
Can raise red flags to stop the line…
if requires human intervention
56 © 2015 IBM Corporation
What is DBmaestro TeamWork?
Database Enforced Change Management solution + Database version control + Enforce best practices + Plugs into the ALM (change request, tickets & work items)
+ Database merge & change impact analysis + Know who can do what, where, when & why
DevOps Solution for databases + Baseline aware deployment automation, rollback &
recovery + Reduce database deployment issues + Plugs into release management & Continuous Delivery
57 © 2015 IBM Corporation
IBM UrbanCode Deploy Visibility and automated control of your application deployment process
Mobile App
Team
Middleware
team Backend Systems of
Record
• Manage application
components and versions
• Manage configurations across
all environments
• Offer secure ‘self-service’ capabilities
• Increase transparency
• Ensure governance and
compliancy
59 © 2015 IBM Corporation
© Copyright IBM Corporation 2014. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. IBM, the IBM logo, Rational, the Rational logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.
www.ibm.com/devops
Top Related