Obsidian Agile DevOps

CSC AND MERCK

PROTECT WHAT MATTERS — ENABLING SUPERIOR SECURITY

Cloud Integration IT Services Solutions Architecture

Agile DevOpsProviding a comprehensive solution for the government for agile

software development, continuous integration, continuous testing and

continuous delivery.

www.obsidiang.com

3 Metro Center, Suite 700 Bethesda, MD 20814

Agile DevOps Obsidian Proprietary Page i

Cloud Integration • IT Services • Solutions Architecture

Table of Contents Executive Summary .................................................................................................................... 1

Agile DevOps Approach ............................................................................................................. 2

Agile Development in Federal Contracting ..................................................................... 3

Continuous Integration (CI), and Continuous Delivery (CD) ......................................... 4

Continuous Monitoring and Continuous Feedback ....................................................... 6

Measuring Agile DevOps Success ............................................................................................. 7

Agile DevOps Metrics for Baselining and Measuring Success ..................................... 7

Process Metrics ................................................................................................................ 7

Why Ask for an External Agile DevOps Maturity Assessment ...................................... 8

Next Steps ................................................................................................................................... 9

Conclusion .................................................................................................................................. 9

Agile DevOps Obsidian Proprietary Page 1


Executive Summary

Today we live in a culture where we must have everything instantly from social media, news,

instant messaging, and the web. Customers, whether they are commercial customers or Federal IT

customers, all live in the same era of astonishing technology, instant information and rampant

social networking. IT organizations have been traditionally setup to deal with a long tedious

Software Development Lifecycle (SDLC) that has customers waiting for releases every quarter, bi-

annually or annual updates to software that doesn’t meet the end-user needs, and still has defects

requiring addional updates.

In 2001, The Manifesto for Agile Software Development (Agile) was first introduced to address

changing requirements, incremental delivery, self-organizing development teams, and quality

software. Agile development, particularly SCRUM, started becoming prevalent in the Federal

market around 2010, and yet many Federal IT organizations are still using traditional waterfall

methodology to deliver capability to agency customers. Why? The reasons are simple. 1.) Many

agile development efforts have been seen as failures in the federal government, 2.) adoption has

been slow because of FAR regulations, and 3.) educating both the contractor and federal IT

community didn’t start making traction until after 2010. Some Federal CIO’s might have lobbied

earlier, but in general RFI’s and RFP’s didn’t start including agile as a preferred methodology for

software development until after 2010 and still today we see many RFI’s and RFP’s that specify

waterfall as the preferred methodology for delivering software.

Is Agile the answer to delivering software faster, better, and cheaper to customers? Yes and No.

Agile only addresses the requirements through software development and doesn’t address rapid

delivery of software to production systems. To address the rapid delivery to production and

disconnect between development and operation teams, a new trend has immerged - software

developmentand operations (DevOps). DevOps addresses the collaboration, and automation

between software development and operation teams. Agile DevOps is Obsidian’s approach to

agile development, continuous integration, continuous testing, and continuous deliverythrough the

use of automated tools, and streamlined processes. We deliver incremental development

continuously to production, which reduces defects, eliminates excess cycle time, provides

continuous feedback and eliminates outage windows when deploying to production.

In this white paper, we will take a deeper dive into Obsidian’s Agile DevOps approach and how

leaders in the federal government can leverage this solution for faster adoption of DevOps into

existing and new federal IT programs.



Agile DevOps Approach

While most IT Executives, Program Managers, and technical staff are well versed in both Agile and

DevOps methodologies, most implementations are partial forms of either Agile or DevOps because

of a number of extenuating circumstances. For example, many programs implementing Agile suffer

from a lack of understanding as to how to manage requirements, deal with change, and still deliver

quality code while staying “agile”. Other programs implementing DevOps do not have a complete

understanding of the processes/toolsets required, and how to manage diverse teams who may be

working under different contracts, using different toolsets and may not be incentivized to work

collaboratively to achieve a continuous stream of production-ready software.While there is not a

single approach to address all scenarios, Obsidian has developed a methodology for Agile DevOps

and an approach for leaders in the federal government to adopt these methodologies within their IT

programs.

Figure 1. Obsidian’s Agile DevOp Approach. Our approach enables rapid implementation with

continuous feedback throughout software development, testing, and deployment to deliver quality services.

The Obsidian Agile DevOps approach encompasses the entire SDLC. As seen in Figure 1, our

approach includes agile development, continuous integration, continuous testing, and continuous

delivery while providing continuous monitoring and feedback throughout every phase of the

lifecycle. Automation is a critical component of a successful DevOps approach, and the tools in this

space continue to rapidly advance.Our approach is agnostic to a particular toolset, and is

001 Obsidian, 02/24/2015

Agile DevOps

Continuous Feedback

Continuous Feedback

Continuous Feedback

Continuous Feedback

Continuous Testing

Continuous Integration

Continuous Delivery

Agile Development

Product

UAT

QA

Repository

Manager

Provisioning Tools

CI Server

Collaboration

Test Scripts

Test Suite

CI Server

Testing Metrics

INT QA UAT

Issue Tracking

Dev Code Repository

CI Server

Build + Unit Test + Code Quality

2 Weeks

Daily Standup

SprintBacklog

Product Backlog

BurndownChart

PotentiallyShippable Product

User Stories

Commit

Code Quality Metrics

Artifact

Test Environment

AutoTicket Creation

Infrastructure as code

Repository

Manager



customizable based on customer preference. The key steps for automation that enable our Agile

DevOps approach include:

1. Daily Code Commit. Developers on a daily basis check-in code into a central source code repository.

2. Automated Builds. A Continuous Integration (CI) server is continually polling the source repository for changes, and when a change occurs the code is checked out of the repository and built. The built software is stored in a repository manager by the CI server.

3. Automated Testing. The code is automatically unit tested, code quality tested, smoke and UI tested, and performance tested.

4. Automated Delivery. The built version is deployed using provisioning tools that treat infrastructure as code.

The entire pipeline, described in the next sections, is tailorable to add in manual validation at any

phase. We increase efficiencies by automating the promotion of software code from development

to testing and into production. Our Agile DevOps approach forms collaborative teams, and reduces

the number of issues occurring during development, deployment, and operations, resulting in faster

delivery of applications/features, and reduces Operations & Maintenance (O&M) costs.

Agile Development in Federal Contracting

Our agile development approach is flexible

enough to use Scrum, Kanban, Test Driven

Development (TDD), Feature Driven

Development (FDD), SAFe, and others. Our

experience is that many federal contracts are

using Scrum, shown in Figure 2, and are

becoming quite successful at using it. The trend

as of late has been for the federal government to

act as the Product Owner and ScrumMaster. In

our experience it is more beneficial to have the

ScrumMaster be part of the contractor

development team to identify impediments and remove them without buderning our government

customer. In either case, there are going to be pro’s and con’s to both models, and both can be

successful. It’s up to the federal government which model works best for them. In cases where the

government might not have the best certified ScrumMasters it might work best for the contractor to

fill that role.

The most important challenge to overcome is the management of the product backlog with the

requirements. There are two common scenarios we’ve encountered:

1. Thousands of requirements are provided at the time of RFP.

2. No detailed requirements exists for the product backlog.

Figure 2. Obsidian’s Agile Development.

Our agile development teams produce quality software quickly, and manage

changing requirements efficently providing continuous feedback to stakeholders.

Agile Development

2 Weeks

Daily Standup

SprintBacklog

Product Backlog

BurndownChart

PotentiallyShippable Product

User Stories



With programs where thousands of the requirements are created and included in a RFP, we have

found that the requirements are usually created without the domain knowledge or understanding of

the system needed to develop systems that accurately represents the desired outcome. In this

scenario, we overcome this challenge by providing a Sprint 0 where the entire team does a product

backlog grooming exercise. We work as a partner with the government to identify requirements

that are valid,requirements that should be discarded, and requirements that need further

clarification so they are reflective of what the government wants.

In the scenario where no requirements exist, we work with the government in a Sprint 0 to develop

a product backlog at the Epic level and develop a product roadmap. Taking this approach provides

the government with a high level product roadmap which gives them the ability to forecast for the

option years of the contract and is usually done under a single award BPA. Using the product

backlog that is developed during Sprint 0, the Product Owner can prioritize the Epics, and then

break the epics into user stories and detailed tasks for each sprint.

We find that taking the time up-front to develop a good product backlog that is reflective of the

government’s desires, and setting up our Agile DevOps environments is key to program success.

With a good product backlog, and the proper environments we can develop a good sprint backlog

for Sprint 1+, and then conduct development using agile Scrum. We understand that requirements

and priorities are going to change, but by establishing a good foundation we can alleviate the

majority of the complexity that comes with implementing an agile development contract.

Continuous Integration (CI), and Continuous Delivery (CD)

Our Continuous Integration (CI) and Continuous Delivery (CD) approach is designed to create an

automation environment for the entire end-to-end release process so that every change to the

application results in a releasable version that is built automatically. With our approach applications

are built from a very early stage in the development process at specific frequent intervals or on

every change checked in by the developers. This effectively eliminates the need for integration

testing because the code is incrementally being integrated on a daily basis. This removes the cost

associated with developers spending time on this phase. The enablement of frequent incremental

builds and mandating a comprehensive automated testing process also allows developers to

detect problems early and as a result, ensure higher application quality.

A typical CI/CD workflow is shown in Figure 3.



Figure 3. CI/CD Workflow. Our approach eliminates the extra workload from merges and multiple

baselines.

To support rapid deployment and release we use tools that allow us to automate provisioning of

infrastructure resources and platforms. The server configuration, packages installed, relationships

with other servers are modeled with code, and is automated and has predictable outcomes,

removing error-prone manual steps.

Our approach uses software development best practices for the infrastructure code and stores the

code in a Code Repository with tags and branches, and releases the code just as if it were

applications software. Our infrastructure code is continuously integrated, tested and deployed right

along the application software and is treated no differently.

We configure our CI server with build steps to check for coding style, coding standards, static

analysis, and other features using tools such as SonarQube. We run unit tests from the CI server

and deploy the code to the development integration environment and execute additional functional

test scripts. We use tools like Selenium for smoke and UI testing. Performance testing is part of our

automated testing using tools like JMeter for load testing. Using CI server, we produce artifacts

that documents the results of the build, unit testing, and deployment and functional testing along

with the source version used for the build.

Our automated deployment provides a continuous delivery pipeline that automates deployments to

test and production environments. Our approach significantly reduces the manual intensive tasks,

resource lag time and errors prone from manual repetition. We provide automated deployment

002 Obsidian, 03/09/2015

• The Continuous

Integration server

monitors the version

control system for

changes and launches the build

process.

• The Continuous

Integration server

runs unit tests to

validate the quality.

• The code is then run through the code

quality tools like

SonarQube.

• The Continuous

Integration server

deploys the same

package on the

testing environments (created and

configured through

infrastructure as

code tools like

Puppet or Chef) for thorough user

acceptance testing.

• Once the

acceptance tests

are successful the

same package is

deployed on the production servers

(created and

configured through

infrastructure as

code tools like Puppet or Chef).

Implementing

infrastructure and

release automation

enables end to end automation and

allows provisioning

and deploying the

application

packages on all servers with a click

of a button.

Commit BuildUnit Test and

Quality Check

Deploy Testing

Environments

Deploy

Production

• During the

development of the

application, the

developers are

working on their local machines and

once they are ready

to submit their new

code changes they

commit them to the central source code

repository.



tools and processes reducing deployment risk, and giving the option of deploying code multiple

times per day without any degradation in service. The releases are small which reduces the risk for

system instabilities and customer user experience issues. Every change is easier to roll back and

easier to test because the number of changes per release is very small.

Continuous Monitoring and Continuous Feedback

Continuous monitoring across all phases of the application development, testing and deployment is

crucial for a successful Agile DevOps implemenation. Our approach lowers the costs of errors and

changes by providing continuous feedback throughout each phase of the lifecycle.

Obsidian offers a unique approach to monitoring using multiple tools to monitor the applications,

environments, and systems. Using tools like Evolven and LMC for environment management

provides real-time user tracking and work flows to manage the technical users that develop and

operate the application on a daily basis. We use tools like Splunk for log analysis for developers

and tools like New Relic to monitor the performance of the applications from the user’s perspective

such as page-load times, database-transactions, and systems monitoring to focus on CPU load,

memory utilization, and disk space. These tools allow our teams to better understand issues and

metrics, and ensures that we are optimizing resources to reduce operational expenditures.



Measuring Agile DevOps Success

A survey from the Vanson Bourne market research agency (with CA) late in 2013 indicated that

39% of those surveyed had adopted some form of DevOps and 27% were planning to do so in the

near future. Despite this being such a hot topic in the IT sector with a high level of adoption, the

question we are still most commonly asked is: “Where do we start?”. Organization’s must first

baseline their current position. Having a baseline means you can build a business case, apply

targets and goals to your projects and measure your success as you progress through your

project.

Agile DevOps Metrics for Baselining and Measuring Success

There are hard, quantifiable technical and financial metrics we can measure, such as:

Number and frequency of software releases

Volume of defects

Time/cost per release

Mean Time To Repair (MTTR*)

Number and frequency of outages / performance issues

Revenue/profit impact of outages / performance issues

Number and cost of resources

Process Metrics

Agile DevOps encompases the entire SDLC that affects both development and operations staff to

greater or lesser degrees that need to be taken into consideration. All of these process

components can be improved and then optimized as the appropriate automation tools are

integrated into the environment. An ultimate goal of an Agile DevOps project is often to attain true

continuous delivery (CD) by linking these processes and tools together to allow fully tested,

production ready, committed code to proceed to productionwithout impediment. When baselining

the current state, it’s useful to measure these component processes and their relative maturity

(taking into account use of existing tools and success of implementation). We look at:

Requirements management

Agile development

Build

Release and deployment

Unit testing

User Acceptance Testing

Quality Assurance

Application Performance Monitoring

* The MTTR is the Mean Time to Repair, Resolve or Resolution - each of the definitions means the same and can be used interchangeably. This term is more commonly used when talking about Application Performance Management and the speed at which an outage or performance issue can be fixed, but equally can be used when talking about testing and eliminating defects



Why Ask for an External Agile DevOps Maturity Assessment

While no one is going to understand your business better than you do, we often meet organizations

who are struggling to find the time to focus on becoming more effecient. They know there are

improvements to be made but they are so busy with firefighting they can’t conceive of stopping to

baseline and evaluate their current position. It’s easy to stick with “the devil you know”, but that

won’t help you with your competitive posture or allow you to use newer technologies to cut costs

and improve quality. That’s why it’s important to have an external assessment of your approach to

Development and Operations.



Next Steps

Understanding Agile DevOps and its benefits are just the beginning. Obsidian can help facilitate

the adoption of Agile DevOps in existing and new programs. Obsidian offers expertise to help cli-

ents align business process and technology to their business needs. We help clients develop a

strategic roadmap for how Agile DevOps technology may be used to meet mission and business

requirements. Our approach helps clients understand the goals and IT strategies within the

roadmap, and we execute projects to implement these Agile DevOps strategies. Our Agile DevOps

experts assist clients in identifying the drivers, key considerations, and other important factors for

migrating and modernizing IT Services. Our experts develop comprehensive IT Transformation

plans that provide clear steps to increase the value of technology.

Conclusion

Our Agile DevOps approach addressess the entire SDLC. With the implemenation of automation

tools and repeatable processess, we continuously deliver to production systems without outage

windows, and unnecessary manual processes. Our approach reduces costs by providing

environments that are automated thus removing the need for staff to spend time with manual

processes. We make processes repeatable which allows for more frequent and less error-prone

releases. Our vendor-agnostic approach drives increased efficiencies through improved

development and operational processes, transparency via continuous monitoring and feedback,

improved quality from continuous integration and testing, and less risk due to an automated

enviornment. Obsidian has the vision, experience, and technical know how to assist Federal

agencies to take full advantage of Agile DevOps.

For more information on Obsidian’s Agile DevOps solution, contact David Callner,

[email protected], 571.425.3031.

mailto:[email protected]



About the Author

Mr. Callner is currently the Chief Technology Officer for Obsidian Global, LLC

and is responsible for the establishment and oversight of the technical direction

of the company. He is charged with ensuring that Obsidian takes a proactive

approach to identifying and deploying new technologies and ensuring alignment

with our client’s technology and strategies. Mr. Callner received a M.S. in

Computer Science from Johns Hopkins University, and his B.S. in Computer

Science from University of Texas.

About Obsidian Global, LLC

Obsidian Global, LLC is a certified Small Business provider of Agile Development, DevOps, Cloud

Integration Services, Solution Architecture Services, and IT Services. For more information, visit

www.obsidiang.com.

http://www.obsidiang.com/

Obsidian Agile DevOps

Technology

Transcript of Obsidian Agile DevOps