© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 1

RelayHealth Build System

Ron CordellRelayHealth Clinicals

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 2

RelayHealth Build SystemAgenda

Development/Deployment Life Cycle Local Developer Builds Automated Build System Deployment System Planned Changes

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 3

Development/Deploy Life Cycle

• 7 Million Lines of Code, 20+ services, 20+ IIS sites, 20+ DBs• Ancilliary, 3rd Party processes• SQL Server 2008R2, MongoDB, RabbitMQ …• Developers run full system stack on their workstations• 12 Dev Teams working on 1 Product/Code Base• Production topology is somewhat different• All development teams work on TRUNK (typically no

branching/merging)• Code changes are deployed behind soft switches• Deploy weekly (currently)

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 4

Development/Deployment Life Cycle

Code/Unit Test/Commit (Everyday)

Automated Build/Deploy

(TEST)

Deployment Branch Cut

(Friday)

Automated Build/Deploy

(STAGE)

Deploy to Production (Thursday)

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 5

RelayHealth Build SystemAgenda

Development/Deployment Life Cycle Local Developer Builds Automated Build System Deployment System Planned Changes

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 6

Local Developer Builds - MorningCoffee

• Clean local output and cache folders• Turn off Windows Services and stop IIS• SVN update• Update SQL Server database (run delta scripts)• Create new Mongo collections (if any)• MSBUILD.EXE against targets file, DEBUG Profile, unsigned• Run Ndepends Analysis, Sync switches to database• Run Unit Tests (Nunit)• Create all IIS site definitions• Create all Windows service definitions• Start all services & IIS• WatiN Automation Tests run manually as needed

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 7

RelayHealth Build SystemAgenda

Development/Deployment Life Cycle Local Developer Builds Automated Build System Deployment System Planned Changes

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 8

Automated Build System – CC.Net

• CruiseControl .NET runs continuously on the build servers• Performs similar steps the MorningCoffee build, broken out into

separate CruiseControl steps.• MSBUILD -> Release Profile, Signed & Versioned Assemblies• CC.Net adds a deployment step and a test automation step before

a build is called “green”• Only green builds are promoted to Test System, usually twice daily.• Build failures result in automated rollback of check-ins since the

last “green” build, including the databases.

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 9

CC.Net Tray Status App

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 10

Build Step Status

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 11

RelayHealth Build SystemAgenda

Development/Deployment Life Cycle Local Developer Builds Automated Build System Deployment System Planned Changes

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 12

Deployment System

• Code deployed to Stage once a week from latest TRUNK “green” build.

• Code is branched for deployment in SVN• Branch is built similarly in CC.Net for the Deployment• Code is deployed to STAGE environment for final testing• Automated Tests are run - BVT• Switches as set to proper state to enable new code behavior• QA Tests as needed

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 13

Deployment to Production

• STAGE testing by QA from Monday – Wednesday• PROD deployment on Thursday• BVT / Automated Tests run on deployment in PROD• Switches set to correct state• Site is never “down” for deployment

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 14

RelayHealth Build SystemAgenda

Development/Deployment Life Cycle Local Developer Builds Automated Build System Deployment System Planned Changes

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 15

Planned Changes

• Move to a complete, enterprise-hosted TFS system?• TFS source control• TFS Build Farms, Test Farms• Take advantage of impact analysis, gated check-ins, traceability

• Move to a more “best of breed” model?• Enterprise GitHub for source control• Team City for build system (gated check-ins)• Other tools for impact analysis, test traceability, etc.

• True Continuous Build and Deployment• Better Test Automation• Correct issues preventing continuous deployment

• Package Management, a la NuGet

© 2011 RelayHealth and/or its affiliates. All Rights Reserved. Proprietary and Confidential. For Internal Use Only. 16

Q&A

We’re Hiring!Finance Director Product ManagerQuality Assurance Analyst Sr. Software EngineerSr. Software Engineer Client Services DirectorEntry Level Software Engineer DevOps DeveloperSoftware Engineer Scrum Master Agile Project ManagerSoftware Engineer Software QA ManagerStorage Systems Engineer Director of Product DevelopmentKnowledge Manager Sr Software Eng (.Net Asp.Net Sql Svr Visi

o)Applications Security Architect Web Applications

RH InternSr. Software Engineer Product Manager Agile SaaSSoftware Engineer in Test Scrum Master Agile Project ManagerSoftware Quality Analyst Sr. Systems AdministratorQuality Engineer 4 Manager User ExperienceSoftware Automation Engineer Quality Assurance AnalystSr. Software Quality Engineer SW QA Automation EngineerSoftware Architect (.Net Sql Svr Asp.net) Project Manager EMR ImplementationsInteraction Designer Software Quality ArchitectSr. Software Engineer Interface Implementation Services Manag

erProgram Director Security Analyst CISSP