Why retail companies can't afford database downtime
-
Upload
dbmaestro-database-devops -
Category
Technology
-
view
141 -
download
2
Transcript of Why retail companies can't afford database downtime
Uri Margalit
Why Retail Companies Can’t Afford Database Downtime
2
• You will be on mute for the duration of the event
• We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)
• If you have questions during the session, please submit them on the Q&A bar on your gotowebinar dashboard
• A recording of the full webinar will be put up online
Before We Begin
3
Presenter
Uri Margalit
Director of Products at DBmaestro
About DBmaestro
• Founded in 2008, product launched in 2010• Founded by Yariv Tabac and Yaniv Yehuda• Headquartered in Israel, Global Operations
5
• Retail challenges• The database uniqueness• Real life tales• Common approaches analysis• The solution from DBmaestro TeamWork
Agenda
6
Retailers are smack in the middle of three to five years of major changes that will redefine the industry.
I would say the last time there was this big [of] a change in the industry was in 1960, when many of the big boxes opened up and really opened the huge discount department store. The biggest change before that - the invention of modern retailing in postwar America.Alison Paul, vice chairman and U.S. retail and distribution leader at consultancy Deloitte
Retail’s Unique Challenges
“
7
• Business• Downtime = Loss revenue• Heavily regulated – large amount of customer data
stored across multiple locations• Migration from physical stores to web-based online
stores using Augmented Reality experiences• Freeze toward year-end changes for fear of
download and delaying agility because of lack of confidence in process
• Rapidly change & Agile market needs in a highly seasonal environment – downtime is critical
• Streamline development & operations team
Retail’s Unique Challenges
8
• Technical• Complex architecture and internal dependencies• Responding quickly to business changes• 3rd party integrations• Same metadata across many environments• Heterogenic environments• Changing applications structures from packaged
applications to manage physical spaces and stock to more customized/home grown web-facing applications
• Streamline geographically diverse development & operational teams
Online Gaming Unique Challenges
9
Why Manage Change?
Copyright@2008, Juniper Networks, Inc.
50% 40% of changes fail
of unplanned downtime is due to Change
80%
of this, half is due human errors50%
More than
10
• DECM - Database Enforced Change Management• Part of ALM (Application Lifecycle Management)
solutions
DECM & ALM
11
• The database holds your essential information
• Changes can impact the entire system• Need to be synchronized with other changes• Often overlooked
Database is a Key Component
12
• There is more to a database than SQL scripts• Schema structure• Code• Content and meta-content• Internal dependencies• …
• Ensure that changes are not made without authorization
• Ensure no out-of-process changes
Reaching Inside the Database
13
• Old adage but true• The database is often “forgotten” and
therefore can become the weakest link• Essential from a compliance point of view• Should be the strongest link
The Weakest Link In a Chain
“
“
It was difficult to track who made a change to a database object and what change they made.(working-around and failing to enforce usage of file based version control)
Sr. DBA @ Large USA Bank
It took hours to get releases working. some changes were not documented and left out. we actually preferred crashes in integration. It is much worse when something works, but works wrong. in production…(manual and error prone releases. Change control & change scope issues)
Sr. R&D Manager @ Credit Card company
The Database is a constraint
We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $(an out-of-process update to QA that was not properly tracked. Same for integ/stage/prod…)
DBA @ Algo Trading company
We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes(code overrides, partial versions, wrong versions – all pushed to production)
CTO @ Credit Card company
“
“
The Database is a constraint
We had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release.
we realized that we needed to bring more discipline and rigor to our database changes.(manual process are hard to repeat over and over without errors)
Sr. DBA @ Large USA Bank
“The Database is a constraint
17
Common DB Development Methods
• Treating database scripts as native code• Small scripts• One large script
• Automatically generated scripts (Compare & Sync)
18
Two isolated Processes
Version Control Process (file based)
Development Process
(real database)
Check-Out Script
Modify Script
Get updated Script from DB
Check-In Script
Compile Scriptin DB
Debug Scriptin DB
?
??
?
A
A’
Version control repository is out of sync from the tested database and cannot acts as Single Source of Truth
19
Scripts & Version Control
• Challenges… • Working on the wrong revisions • Code-overrides• Scripts do not always find their way to the version control
solution• Out of process updates go unnoticed• Hard to locate outdated update scripts
• Playing safe? what we really need (manual steps):• The actual code of the object• The upgrade script• A roll-back script
20
Scripts are static…
• Scripts, unless super sophisticated:• Unaware of changes made in the target environment• Time passed from their coding to the time they are run• Potentially overriding production hot-fixes or work done
in parallel by another team
• Content changes are very hard to manage• Metadata & lookup content does not practically fit into
the VC• In most cases they are simply not managed
21
Testing Scripts
• Single object based scripts• Hard to test in their entirely (holistically)• Hard to test due to colliding dependencies• Need to run in a specific order…
• Large multi object based script• Represents the entire update - can deal with
dependencies• Much harder to deal with project scope changes• Hard to mange – a big list of commands
22
X1.11.1.11.11.21.31.41.51.61.7
Build Once Deploy Many (Scripts)
Int QA Pre Prod Prod
Database Deploy Script
Environment
Re-Base (due to defects)
DevDev
DevModel
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.11.11.41.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process Change
X
X
X
X
X
? 1.1.1
X
23
Scripts – Bottom Line
• Overhead time• No trust in the version control repository• Difficult and very complex to maintain• No agility to business changes
24
Compare & Sync
• Challenges… • Handling out of process changes• Handling parallel development• Handling conflicts• Not an integral part in development phase - version
control repository is NOT the single source of truth
Source vs. Target
Action
= No Action
≠ ? We do not have all of the information
27
1.11.21.31.41.51.61.7
Build & Deploy On Demand(Compare & Sync)
*
Int QA Pre Prod Prod
Database Deploy Script
Environment* Execute the same script being executed at the Stage environment
Re-Base (due to defects)
DevDev
DevModel
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
1.1 1.1 1.11.41.7
File Based Version Control
Out of Process Change
1.1.11.7 1.1.11.7
28
Deployment Automation Safety Net
An index exists in Target (Production) but not in Source (QA). What should we do? Drop the index or not?
Baseline Aware AnalysisSource vs. Target
Action
= No Action
≠ ?
Source vs. Baseline
Target vs. Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have all of the information
With Baselines and 3 way analysis the unknown is now known
Simple Comp & Sync
29
So why not move forward?
30
• Database Enforced Change Management• Database version control• Plugs into the ALM (change request, tickets & work items)• Database change impact analysis• Database deployment automation
• DevOps Solution for databases • Deployment, rollback & recovery• Plugs into release management
What is DBmaestro TeamWork
31
With DECM - One Enforced Process
32
• Raise red flags on conflicts • Support out-of-process changes• Utilize baseline aware analysis• Understand the nature of the changes
Safety Net Deployment Automation
Source vs. Baseline
Target vs. Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
33
• Development• Database change repository• Follow SCM best practices (Check-Out/Check-In) • Documentation of who did what, where, when & why
• Operation• Integrated deployment engine• Business level audit• Roles & responsibilities enforcement
• Management• Complete visibility into changes in progress• Management reports• No silos
Benefits – By Role
34
• Productivity improvements• No manual work and less reworking• No time spent in testing deployment scripts• Supports parallel development
• Quality improvements• No code overrides• No promoting wrong code• Supports partial releases
• Less deployment failures• Accurate & complete deploy script• Supports task based development
• Auditing & Regulations• A complete documentation of who did what,
where, when & why
Other Benefits
35
Focusing on changes rather than managing changes and dealing with re-work, boosted overall productivity of 200-300 developers. We estimate we were able to do 15% more with the same resources.
We went from several fix-centric deployments a day, to one feature-centric deployment a week.
The amount of incidents in production has declined as well. We had 20% less incidents. CIO @ Credit Card company
Testimonials - Efficiency
“
36
Regulation requirement - SOX & derived 357
Auditing - change management approach with change management auditing. You have to do it anyway, do it automatically and efficiently.
Change management in production itself is regulatory required (ITIL). But you cannot ensure it without managing the whole process starting at Dev.
We have to comply to regulation - but the business benefits from it. CIO @ Credit Card company
“Testimonials - Regulation
37
Selected Customers
And more…
38
Customer Testimonials
“Massachusetts Department of Education uses DBmaestro TeamWork to reduce database change deployment time by up to 95%”Mr. Dan Boyle, Supervisor of Database Administration & Development
“Isracard has increased the throughput of our development team immensely and along with that our team is more motivated and satisfied on the job. With the TeamWork™ proprietary Automation Module we are able to deploy changes between environments automatically and this reduces the number of human errors and increases the production system's stability”Ofer Krichman, Head of Infrastructure Software Department , Isracard LTD
“DBmaestro TeamWork facilitated the database development process in the Ministry of Education in the current IT development projects. The ROI of DBmaestro TeamWork is doubtless...”Mrs. Sofia Mintz, CIO
“By implementing the DBmaestro solution, We can finally take control of our databases. By Integrating the application alongside our source control and continuous integration paradigm, We are able to deploy our application within minutes.”Mr. Jonathan Bar-Sela, Development Manager, Clalbit systems
“DBmaestro TeamWork, provides us full control on our databases environments. Having our schemas sync between all environments made a big change in the DBA’s and developers efficiency. TeamWork integration to MS-TFS, provides us End-to-End products development life cycle”Mr. Shahaf Azriely, Director of IT, IncrediMail Ltd.
“By implementing TeamWork™, Sura finally has the ability to apply the SCM best practices on database development…Finally, Sura's database deployment will be quick and efficient with an ideal rollback option.”Juan David Perez Chief Architect, Sura Insurance
39
Summary
• Enforce best practices for database development• Closely manage your database development• Know who can do what, where, when & why• Build a safety net to catch problems early• Reduce deployment issues• Enable automated deployments
40
Demo
DeploymentScreen shots
41
Integrations
• ALM Solutions• TFS• RTC
• Task Management Systems• Clear Quest• Perforce• JiRA• TeamForge• Any tool, using open API
• Continuous Integration• Jenkins• Bamboo• Any tool, using web-services
• Release Automation• IBM UrbanCode Deploy• CA Release Automation• Serena• BMC DevOps• Any tool, using web-services
• IDEs• Oracle SQL Developer• SQL Server Management Studio• Visual Studio• JDeveloper
• Other Database Tools• Delphix (Virtual Branches)• DM Suite (Content Masking)• Oracle SQL Developer (Unit Test
& Code Coverage)
42
Merge Dev1 to ForkMerge to dev2
Dev2
Dev1Merge to dev1
Merge Dev2 to Fork
Trunk
Merge Dev1 to Fork
Merge Dev2 to Fork
DBVC
Fork
Fork
Fork
Fork
Merge, Rebase, Fork, …
43
Use of Baseline
Int / QA
Dev1
Label/Tag/BaselineBackup Label/TagMerge
1
Ver 1.0
Ver 1.0
2 3
MS 1.1Merged - B
Before MS 1.1
MS 1.1Merged - M
Branch Merging MS 1.1
MS 1.1
7 8
MS 1.2 Merged - B
Before MS 1.2
Branch Merging MS 1.2 MS 1.2
Merged - MMS 1.2
Dev2 Ver 2.0 MS 2.1Merged - B
Before MS 2.1 MS 2.2 Merged - B
Before MS 2.2
4 5 6 9 10
MS 2.1Merged - MBranch
Merging MS 2.1
Branch Merging MS 2.2
MS 2.2 Merged - M
MS 2.1 MS 2.2
Deploy Source Target Baseline
1 Int / QA (Live) Dev1 …
2 Int / QA (MS 1.1) Dev1 (Live) Dev1 (Ver 1.0)
3 Dev1 (MS 1.1 Merged - B) Int / QA (Line) Int / QA (MS 1.1)
…