SUNZ 2010

Version Control and Automated Deployment of Source Code in a Data Warehouse Environment

Mike O’NeilMSD Data Warehouse Team

michael.oneil011@msd.govt.nz

Content

• Background and Context• Motivation• Change Management• Automation of Deployment• Subversion and TortoiseSVN• Standards and Practices• SAS Features and Functionality• Where to from here?

Background and Context

• IAP Data Warehouse (SAS based)• Established 1994• 30+ Source Systems• 10,000 Information Consumers• UNIX and Windows SAS platforms• 600+ Control-M scheduled jobs• Huge SAS Code Base• Frequency of Change

Motivation

• Quality improvement• Reduction in Failure• Productivity and Efficiency• Cost• Physical Environment Separation• Maturing process• Removal of emotion• Increasing accountability and ownership

Change Management

• Change Management– Policy and process

• Change Control– Recording and approval

• Version Control– Asset Management

Change Management

Survey Results• Not all changes logged 95%• Changes not thoroughly tested 90%• Lack of process enforcement 85%• Poor communication and dissemination 65%• Lack of centralized process ownership 60%• Lack of change approval policy 50%• Frequent change notification after the fact 40%

Reference: Edward Stickel quoting Harris Kern in article athttp://www.itsmwatch.com/itil/article.php/11700_3367151_1

Automation of Deployment

• Common feature of commercial software development

• Continuous integration• Regression testing• Configuration driven (what goes where)• Version Control

IAP Automation of Deployment

• Subversion for Version Control• TortoiseSVN for Developer Interface• Configuration driven• Deployment Application written in SAS

(and UNIX shell script)

Subversion & TortoiseSVN

• Subversion – Database, Server, CLI• TortoiseSVN – Windows Explorer plug-in• Basic developer workflow:

– Check Out to create a Working Copy– Branch to manage your own stuff– Commit changes– Update when new release hits trunk– Tag when ready for release

Subversion

IAP Automation of Deployment

• Configuration “database”– Configuration_Objects – names all objects – Configuration_Paths – says where they go

• Owned and maintained by developers

IAP Automation of Deployment

• Reconciliation• Export from repository• Pre-deploy steps• Create target directory structure• Deploy software objects to target• Post-deploy steps

Standards and Practice

Typical IAP approach• Driver Shell

– Invoked by Control-M– Creates environment variables– Calls Driver SAS program

• Driver SAS– Retrieves environment variables (%sysget)– %includes SAS code to do work

Standards and Practice

• Driver Shell ScriptExport ENVAR=valueexport SASLIB=/lev1_3/app/sasdataexport SASCODE=/r02/app/sasprogs

• Driver SAS Program%let saslib = %sysget(SASLIB);libname applib “&saslib”;%let sascode = %sysget(SASCODE);filename appsrc “&sascode”;%include appsrc(worker1.sas);

SAS Features and Functionality

• Configuration “database” in CSV files– SAS Proc Import

• SVN List command –xml– SAS xml libname engine and xml maps

• Programs that write programsPut “cp a.sh /r02/test/ControlM/shells”; Put “chgrp develop /r02/test/ControlM/shells/a.sh”; Put “chmod 2771 /r02/test/ControlM/shells/a.sh”;

• Standards and practice (using %sysget)

Where to from here?

• Acceptance• Training• Complete repository set up• Complete required refactoring• Integrate with JIRA and Remedy• Run under Control-M

Questions?

Thank you

michael.oneil011@msd.govt.nz