DEVOPS+DATA:WORKING WITH SOURCE CONTROL

Ed Leighton-Dick

ABOUT MEEmail: ed@leightondick.comBlog: edleightondick.comTwitter: @eleightondick

GOALS• Understand DevOps• Understand source control• How to apply source control to data

DEVOPS

WHAT IS DEVOPS?• “DevOps is the practice of operations and development

engineers participating together in the entire service lifecycle, from design through the development process to production support.”• Corollary: “DevOps is also characterized by operations staff

making use [of] many of the same techniques as developers for their systems work.”

Source: https://theagileadmin.com/what-is-devops/

WHAT IS DEVOPS?• Incorporates• Structured processes favored by operations• Flexibility favored by developers

• Built around consistent processes and practices• Evolved from need to better support Agile• Often includes Continuous Integration/Continuous Delivery

WHAT’S THE BIG DEAL?• Improved communication and collaboration• Improved productivity and efficiency•More regular, more consistent software updates• Improved software quality• Reduced time to recover from failed releases• Lower risk

STAGES• Code• Build• Verify• Package• Release• Configure•Monitor

Image Credit: By Kharnagy - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=51215412

DEVOPS+DATA• Databases are harder to automate• Shared resource• Code mixed in with data• Mix of static and dynamic data• Objects must be modified, not simply replaced

• Databases are integral components of most applications

SOURCE CONTROL

SOURCE CONTROL• Practices and supporting systems that support the

management of changes to application code• Stores history of changes• Facilitates merging of conflicting changes

• AKA: version control, source management, revision control

TYPES OF SOURCE CONTROL• Centralized• PVCS• Microsoft Visual Source Safe• Microsoft Team Foundation Server• Subversion (SVN)

• Distributed• Git• Mercurial

COMPARISON

Centralized

• One snapshot + deltas•Working copy =

Most recent version•Work solely on

central repository• One source of truth

Distributed

• Git: Snapshots•Working copy =

Full copy of repository•Work primarily with

local repository• ??

TRADITIONAL• Repository•Working folder• Check-out and check-in• Commits• Branching and merging

DISTRIBUTED• Repository• Local repository• Cloning• Pull• Push• Branching and merging

VCS+DATA

VCS+DATA• Databases should be included in source control• Database code = Application code• Static data = Application code• Most applications cannot run without their database(s)

VCS+DATA• Database professionals have been slow to adopt source

control• Many DBAs have no development experience• Lack of communication about why• Lack of communication/inclusion with development teams

• Migration vs deployment• Lack of pre-built tools to facilitate complex database changes

VCS+DATA: WHY NOW?• The situation has changed• Modern tools now exist to facilitate technical side• DevOps changes the conversation

VCS+DATA MODELS• Schema-only• Changes-only• Hybrid

THE SCHEMA-ONLY MODEL• Code for all database objects stored•Mirrors model used by most application code• Can produce a clean copy of a specific version of a database• Cannot migrate to a new version of a database without help• Used by…• Roll-your-own• Most third-party tools* - Microsoft, Redgate, ApexSQL, DBMaestro

THE CHANGES-ONLY MODEL• Scripts to upgrade between versions are stored• Scripts to create a clean copy may not be stored• Used by…• Roll-your-own (very common)

THE HYBRID MODEL• Stores scripts used to create objects• Stores scripts used to migrate between versions• Used by…• Redgate SQL Source Control (with migration scripts)• Redgate ReadyRoll• Others?

FIRST THINGS FIRST…• All changes must be saved to source control for source

control to be useful• Database cannot enforce this – Unaware of source control• Source control cannot enforce this – Unaware of changes• External tools may be able to monitor but cannot yet enforce –

No control over database or source control• Cannot move further in DevOps without good source control

FIRST THINGS FIRST…• Decide on your process before implementing tools• What objects are saved to source control?• What changes are saved?• Who saves changes?• How often?• Who/what will enforce process? How?

SHARED VS DEDICATED• Two models for working with development databases• Shared• Dedicated

SHARED DEVELOPMENT MODEL• Data kept in central location•More efficient from technical perspective• Less disk space• Less management overhead

• Developers must be concerned with conflicts with each others’ work• Integration done at time of development• Refreshing can be difficult

DEDICATED DEVELOPMENT MODEL• Each developer has their own copy of the database• Developer is able to work more efficiently• No conflicts with other developers’ work• Copy of database is stored locally in many cases• Data is more easily refreshed

• Integration occurs between development and testing• Concerns• Transporting and storage of large databases• Security of data• Management overhead

WHICH IS BETTER?• No right answer• Often dictated by circumstances

•My preference: As close to dedicated as possible• Redgate SQL Clone can help facilitate refreshing,

management• Solves some problems• Still requires connection to central image

RELEASING CHANGES• The problem:

How do we generate the deltas that matter to us?• Source control deltas = Differences between stored versions

(Not useful for migrations)• Databases require change scripts of some sort

TOOLS•Microsoft Visual Studio Database Projects• Feature of Visual Studio• Developers works with creation scripts, rather than directly with

database• “Declarative database development” model• Schema Comparison used to generate change scripts

TOOLS• Redgate approaches• SQL Source Control• More “behind the scenes” than database projects• Integrated into SQL Server Management Studio• Uses SQL Compare to generate change scripts at time of release• Manually-created migration scripts override generated change scripts

• ReadyRoll• Integrated into Visual Studio• Uses SQL Compare to generate migration scripts at time of commit• Edit generated migration scripts to override

DEMO

BEST PRACTICES•Make database source control part of your team’s standard

development process• Don’t make it only the DBA’s responsibility

• One object/change per file• Check-in changes on a regular basis• Find a solution that integrates with what your developers already

use• Don’t allow code changes outside of the process• Store the database build number in the database

WRAP-UP

GOALS• Understand DevOps• Understand source control• How to apply source control to data

QUESTIONS?Email: ed@leightondick.comBlog: edleightondick.comTwitter: @eleightondick

THANK YOU FOR ATTENDING!~ Please remember to fill out your comment cards ~