DevOps for the Enterprise - MyrtecEnabling and facilitating teams working within a DevOps process...

DEVOPS FOR THE ENTERPRISE | Strategy and positioning for enterprise technical teams

Published by AWS Professional Services 12/11/13

DevOps for the Enterprise

STRATEGY AND POSITIONING FOR ENTERPRISE TECHNICAL TEAMS



Table of Contents Introduction .................................................................................................................................................. 2

DevOps as Agile Evolved ............................................................................................................................... 3

Team Structure and Positioning ................................................................................................................... 4

Aligning and Structuring Projects .................................................................................................................. 5

Automation Orchestration System ........................................................................................................... 8

Functional Testing Tools ........................................................................................................................... 8

Deployment Management Process ........................................................................................................... 9

Defining a Feedback Loop ......................................................................................................................... 9

IT Team Convergence via Source Control Management and Automation ................................................. 10

DevOps, Better with Cloud ......................................................................................................................... 10

Implementing with AWS ............................................................................................................................. 11



Introduction The latest trends in software development and IT operational management speak directly to the need

that businesses continue to require faster, more nimble and higher quality technical work at a reduced

operational cost. This is not a new set of requirements, per se, though teams in medium to large

enterprises have had significant difficulty reconciling policies with new technologies. This is largely due

to the evolving nature of newer tools and products which accelerate the process and enable teams to do

what, until very recently, was exceedingly difficult to do. Implementing a DevOps strategy is a shift in

mindset for most enterprise teams, but leveraging the flexibility of this workflow benefits not only the

health of the system but also the productivity of the team. Implementing with AWS acts as a force

multiplier for people, budget and technology, adding enormous potential for massive benefit at both the

bottom and top lines. As IT organizations make the shift to service management and seeing business

units within their own companies as clients, the entire technical landscape is shifting which presents a

unique opportunity to modernize systems and join the scores of teams that are implementing DevOps

practices. The benefit is simple: implementing a DevOps-based process within the enterprise IT

organization has the potential improve the speed and quality of deliverables, allowing you to be more

responsive to customers both internal and external.

At the heart of the matter is how people interact and work within the gated and hand-off based

processes for handling software releases and the operational management of those systems. These

processes are inherently flawed, cleanly bisecting the teams with direct and material knowledge of the

software from those who are experts in systems administration. Process misalignment due to this

common pattern can easily be the source of frustration and resistance between teams, necessitating

better and more integrated ways of accomplishing the same end result faster, with fewer defects and

far less heartache.

To be clear, DevOps is a paradigm and set of concepts rather than a concrete means of running a

software development lifecycle (SDLC) or application management lifecycle (ALM) process. While

certainly a flashy moniker within the popular press, the concepts found within the DevOps community

offer a unique perspective and, with thoughtful consideration and planning, are uniquely able to meet

the needs of enterprises looking to improve the ability of their technical teams to quickly meet the

needs of the business. The DevOps concept defines, at its core, the ways in which the various roles in a

technical organization interface and behave in order to achieve a full lifecycle management process that

engenders a collective sense of ownership from all team members. Additionally, these interfaces

certainly have a variety of means of implementation since this framework is not specific to any one

technology, toolset or software package. While current methodologies in software development

attempt to define frameworks to embrace change and manage complex and chaotic systems, it is all too

common that even well-oiled development processes are stopped cold when they transition into the

production system. At the same time, while software teams move toward more collaborative and

iterative processes, the disciplines needed to ensure proper test process, audit trails, reporting and

visibility tend to diminish as the pace of development increases. This discord between speed and



communication is the crux of the issue that SDLC-only approaches (such as agile and waterfall

development processes) tend not to address in the larger enterprise and rarely become standard

operating procedure within the operations and production support teams.

In broad terms, this document will outline a few high-level concepts related to organizing and running a

DevOps team within an enterprise environment. By implementing a DevOps-based process, you can

achieve faster time to live, fewer defects per release and faster system recovery time via repeatable,

consistent, collaborative, and automated workflows. This pattern will also allow teams to deliver more

value to the business by allowing them to make changes rapidly and more reliably with a higher degree

of confidence.

DevOps as Agile Evolved In looking at a broad survey of the history of modern project management styles within the IT and

software development disciplines, there are often seen variations of two basic concepts: upfront

requirements gathering (waterfall) and iterative or continual requirements gathering (Agile, et. al.)

methodologies. While this evolution of processes has been beneficial, both waterfall and agile

methodologies lack a critical perspective—an operational understanding of the environment where

applications actually live.

When waterfall methodologies were formalized, they refined the process of developing software and

added required nuances that differentiated the process of managing these projects from other non-

technical work. These methodologies have the benefit of managing risk and ensuring that requirements

are well composed, though teams working under this methodology are less able to be responsive to the

changing needs of the business while taking a significant amount of time from the inception of a project

to delivery of a working system due to the need to gather a full set of requirements before starting

development work.

The next major evolution in technical project management came in the form of Agile frameworks. These

methodologies (XP, pair programming, Scrum, etc.) aim to lower the barrier to entry for projects by

focusing on iterative design and requirements development along with iterative delivery of smaller,

more focused pieces of software that, in the end, add up the sum total of the intended system1. This

shift also aims to be more responsive to the changing needs of the business2 and overtly recognizes that

some requirements tend not to be articulated or understood until the system is already under

construction3. In practice, Agile has been well accepted within software development teams, but the

process of releasing applications, changes or updates to production environments still followed a gated,

waterfall-like process in transitioning to a more operations-focused IT team who would then handle the

release process. This misalignment in process and management style often leads to further

1 Beck, Kent (1999). "Embracing Change with Extreme Programming". Computer 32 (10): 70–77. doi:10.1109/2.796139 2 "Agile Methodologies Survey Results" Shine Technologies 3 Iterative and Incremental Development: A Brief History, Craig Larman and Victor Basili, IEEE Computer, June 2003

http://dx.doi.org/10.1109%2F2.796139

http://www.shinetech.com/attachments/104_ShineTechAgileSurvey2003-01-17.pdf

http://doi.ieeecomputersociety.org/10.1109/MC.2003.1204375



organizational confusion and tension between teams operating under different project management

styles4.

DevOps, in turn, builds on Agile as a means of managing change in an iterative, feedback-focused and

responsive process by automating the vast majority of managing the process pipeline between the

source code repository and the production environment. As a critical component to enabling this, a

cloud-first strategy can assist in addressing objections to infrastructure availability issues (procurement,

CAPEx expenditures, approvals, etc.) while also reinforcing the IT operation teams’ need to embrace

automated configuration management over manual setup and management of systems. DevOps aims

to unify the team around both the development and the operational effort, often resulting in developers

taking on-call time while operations staff works to more deeply understand the application stack. This

commonly leads to insightful performance improvements and new non-functional requirements that

improve the overall operational health of the application and drives a better end user and operational

experience overall

Team Structure and Positioning Historically, the teams designing software for release into the production environment have had limited

(at best) access to the production environment where real-world issues occur. While this model works,

there are two critical problems with it: disparate teams are unable to effectively coordinate solutions to

regulatory or compliance requirements that span the development and release environments, and

operations policies end up restricting optimal application design.

The shift in moving to a DevOps-oriented view of systems development and management requires that

teams align and manage themselves according to the quality of the product they are responsible for.

While software design and production environment operations were historically owned by two

disparate teams, the concept of DevOps is (as simple as the portmanteau suggests) to remove the

separation and both give developers responsibility for the operational system and give operations teams

the ability to influence and work within the development lifecycle. Often this has occurred due to the

interpretation of regulatory or compliance requirements that were otherwise difficult to achieve

through implementing different processes such as utilizing credential tokenization techniques and

targeted, detailed access logs for root cause analysis purposes. With the commoditization of resources

across the IT industry, this is no longer the case and leveraging solutions like AWS’ CloudHSM offering

for hardware-based tokenization management and security isolation capabilities at the network (VPC)

and instance (EC2 Security Group) levels. Technology, in this case, has matured to serve the use case far

more directly and is far easier to manage, configure and design than ever before.

Along with the divide in the management of teams, the decision making process related to how best

manage a given application becomes the responsibility of the team who owns the production

environment, not the team(s) who are responsible for the design and development of it. This is not to

4 http://blog.utest.com/why-agile-development-fails-sometimes/2012/11/

http://blog.utest.com/why-agile-development-fails-sometimes/2012/11/



say that an enterprise-wide standard should not exist, but since the success and quality of the

production environment also falls to the same team, they need to be able to adopt processes and

workflows that both leverage the advantages of and mask the potential deficiencies of the application’s

architecture. DevOps teams working on customer-facing (either internal or external) must also take

ownership of support issues and consider the in the overall context of the workflow. Ideally, by

streamlining the process of releasing updates to the production environment, fixes to end user-

impacting problems would take less time and effort.

Managing the shift in process might seem complicated, but when implementing this type of strategy,

there is a strong opportunity to consolidate a standard set of tools (such as task/issue management,

automation toolsets and test frameworks) that can serve as the base process that teams start with when

designing their specific processes. As with other teams, the teams that manage this core architecture

and toolset must take ownership of the tools and support DevOps teams who consume their tools and

services. Enabling and facilitating teams working within a DevOps process requires a minimal set of

tools that will likely sound familiar, though their usage within an enterprise might expand or change to

meet the needs of the individual teams. More information on specific toolsets can be found in the

Collaborative Automation section below.

Aligning and Structuring Projects Organizing the work to be done around the teams such that they are manageable and are sufficiently

discreet to enable end-to-end ownership by the teams working on it becomes the next concern in

enabling the DevOps workflow for enterprises. As a rule of thumb, modules that are partitioned should

encapsulate a fully functional process or set of processes. By doing this, the question of ownership over

any given module or set of modules won’t be ambiguous as teams will be assigned at a module level and

managing the process of implementing good development practices related to testing and deployment

management can become easier.

Application design best practices extend through to deployment management as well in including

strategies such as:

Automating everything from continuous integration to deployment and fail-back processes

Loosely coupling disparate systems

Utilizing rate-limiting processes within service registration and management workflows

Managing infrastructure tasks in parallel processes as much as is possible to speed deployment and

recovery processes. Let us discuss a hypothetical customer relationship management (CRM) package as

a way of looking at restructuring an application for this type of team workflow. For simplicity's sake, let

us assume a simple system that tracks customers, captures contact history data, a generic set of

marketing tasks along with some low-level logging and application instrumentation tasks.



Figure 1 - Monolithic application structure

An easy first decision in this case would be to break the system up by the three layers--user interface,

business objects or to call it a middle tier and a common set of low-level tools common to all projects.

While there is benefit to decomposing the layers of an application in this way, aligning teams to this

structure is still difficult in larger applications and changes in one area, at a minimum, could require a

disruption in other areas of the system for redeployments and other updates to the infrastructure.

Further decomposing the system into smaller components can afford greater flexibility in team

alignment and ownership as well as greater deployment stability. Rather than being tightly coupled and

bound within a monolithic architecture, modular components offer a potential for becoming more fault

tolerant by designing components to handle failure gracefully. Further to that, cascading failures within

a modular design, so long as the dependency tree of the components is well designed, do not

necessarily have to result in a full disruption of service.



Figure 2 - Potential modular redesign structure

While the example above describes a simplistic, legacy application, using the overall concept of modular

design allows for deployment and infrastructure management concepts to influence the design and

development of systems, but also offers the opportunity to inject infrastructure-as-code into the

workflow within the development effort. While not a requirement or the only pattern implemented

with DevOps teams, SOA tends to be a popular concept, consideration or current strategy for teams that

are working toward building a cloud-focused architecture strategy, this encapsulation method also

serves well as the basis for such a design paradigm. Pairing deployment and infrastructure code with

each service neatly encapsulates the entirety of the lifecycle of that service in one place Collaborative

Automation

The intersection of software development and IT operations exemplifies the opportunity that exists for

cross-pollination and the types of creative solutions that the separate IT disciplines can identify and

build upon when they share ownership of a product from end-to-end. With the technical work

organized to fit with the structure of the team(s) working on it, automating the process of managing the

end-to-end process from check in through to environment deployment and testing is critical. While

automation in the IT space is not a new concept at all, using consistent techniques from development

through to the production environment is certainly a rarity and an area where universal needs of the

DevOps community have yielded a number of new automation tools in the commercial and open source

markets that operate at scales previously deemed unmanageable. The catalyst in the change of mindset



and approach was the concept of looking at infrastructure automation process as code and using well-

known processes that have been finely tuned by the software development community such as source

control management and continuous integration.

Automating processes across multiple IT disciplines is key and it reduces the amount of overhead and

human interaction required to manage the application lifecycle. While it's certainly helpful to automate

a process to build and package an application on a regular or continuous basis, a build that has not been

tested (unit, functional and deployment testing) has little value by itself. To accomplish this though, it is

necessary to include a few specific types of tools to manage and orchestrate the process. From a

strategic perspective though, many teams opt to repurpose tools they already have or look to open

source toolsets as a means of saving cost and providing a flexible set of tools that can be extended or

customized on demand. Certainly teams that are not currently implementing this level of automated

testing can layer it on if and when it becomes appropriate.

Automation Orchestration System

Building automation around your application management lifecycle starts with building your automation

toolset around your application(s). Many solutions on the market today offer the ability to build for

multiple platforms, languages and types of applications. This level of flexibility is key for large-scale

codebases that might incorporate multiple technologies within the architecture. There are a number of

both open source and commercial products that are designed to manage the process of building and

packaging applications of a variety of types. A specific variety of these, which are positioned as

'Continuous Integration' tools, have built-in features to monitor a source control repository and trigger

events from check-ins or other actions being performed by the team. In looking for a tool that will

facilitate the rest of the automation discussed, it is important to ensure that the automation tool is

extensible and can execute tasks written in your language of choice.

Functional Testing Tools

While there is definite value to human interaction with a system to ensure it is operating properly, there

is certainly a high percentage of tasks that can be automated with certain tools and possibly a suite of

tools that are specifically designed for the type of application (web, desktop, service) being tested.

Regardless, the critical point of choosing a functional testing tool is that it integrates with the

automation orchestration system that's been implemented as the corporate standard. Ideally, it would

do so with a familiar or reasonably similar set of technologies (language, database type, etc.) that

individual teams would be familiar with. Along with the software engineers, software engineers in test

(or QA engineers) should also be able to leverage the same source control management tools that the

software development team uses in order to locate all of the project's assets in one place. Executing

functional tests as a part of the build, package and deploy process could then rely on one set of

versioned tests. A simple and well-managed feedback cycle allows the team to move quickly and

increase project velocity and cross-functional collaboration throughout the application lifecycle.



Deployment Management Process

Managing deployments for integration, test and staging environments should leverage the same

framework and be integrated tightly into the overall development lifecycle. While deploying a specific

individual or set of applications to a specific target environment is important, handling the deployment

rollback process is just as critical. While refining existing strategies and techniques will lead to a decent

approach to this problem, resource management in the cloud-focused infrastructure paradigm leads to

an inflection point of innovation which leverages the programmability of cloud resources along with the

economics of pay-for-use resource allocation. Resource allocation in the cloud is no longer a function of

your physical datacenter footprint or your capacity to buy physical hardware fast enough. Concepts

such as managing database backups via volume snapshots rather than database-specific extracts and file

artifacts leads to the possibility to version not only your application code and your server configuration,

but also the database (and its data). With a strong configuration management toolset and a commodity

perspective on compute resources, spinning up new pools of servers for a new version of an application

and using auto-scaling policies to increase and decrease capacity on demand allows for true A-B

production rollout configurations where failing back the application tier requires that you simply

reinstate the old pool of servers.

By treating infrastructure configuration documents as code by versioning the configuration itself in a

source control repository, re-launching an environment for defect debug and discovery or even system-

wide restore and rollback are far more achievable workflows than ever before.

Defining a Feedback Loop

The final component in the suite of DevOps tools is to implement a ticketing or issue management

system as a means of providing a feedback loop--for feature requests, bugs, environmental,

performance and technical-level items to be addressed by the team overall. Issue management is not

anything new—the key differentiator in this workflow is that environment ownership is played out by

the team’s collaborative effort to resolve issues across multiple disciplines. The mark of a well-run

DevOps team is the collaborative effort to resolve problems as opposed to a continual handoff between

disparate resources.



While an enterprise system might already contain an issue tracking system, commonly they are over-

engineered and in terms of implementing an issue tracking system as a feedback mechanism to the

DevOps team, creating a simple to use, easily understood and open system is critical. Simplicity in

managing feedback is critical as allowing for greater transparency and for better management of

requirements, issue definition and enhancement requests. True to the DevOps form, this too should fall

under a process of ‘Automate and Simplify’, wherein as much as possible is automated and the rest is

simple as possible. Automating this process could include features within an issue tracking system

including auto-populating fields such as reporting user, system affected and even setting an issue

template based on the type of issue reported for a specific application or system. You might also choose

an issue tracking system that includes features such as a built-in screen capture utility or browser

integration (for web-based applications).

IT Team Convergence via Source Control Management and Automation Through the process of organizing teams, application artifacts and technology processes under a DevOps

strategy, all teams contributing technical work to the macro-level system should be designing and

developing their software, tests and automation processes and using a source control management

system to retain the history of updates over the course of time. While this has been common practice

for most software development teams for years, this might be a new practice for operations, test and

release management teams.

By placing all assets related to an application in a single source control repository, teams can easily re-

create the state of the application for issue resolution debug and testing. Paired with a consistent, well-

architected database snapshot strategy, even the database (with its data intact) can be restored to

provide a closely representative copy of the production environment that can be used to fully

investigate the root cause of any issue from deployment process failure through to end-user bugs

introduced by a given version of code.

DevOps, Better with Cloud The paradigm of cloud computing enables a number of novel strategies in managing resources that were

previously cost prohibitive, difficult to orchestrate or manage or simply had no feasible technical

implementation. The seemingly endless storage capacity in the cloud allows for strategies that leverage

both short and long-term usage of massive amounts of disk capacity and server resources that, in a

standard datacenter, would require ownership of the entirety of the capacity needed—a number that is

difficult to plan for and inherently self-limiting. Lifting these hard capacity limits by working within the

AWS platform, enabling the ability to rapidly restore and debug issues with actual data quickly and

efficiently is only possible by leveraging automation, api-based access to resources and infrastructure

code that is versioned alongside the application code.

Let us consider the example of database backup management within the software release cycle. Ta

backup of an RDBMS system before applying updates in a production system is good practice to ensure



that there is a viable rollback plan in the event that an update to the database or application fails. In

legacy architectures, the amount of time required to generate a standard backup, even for modest

databases, adds meaningful time to the release process along with a disruption to the performance of

the system for the entirety of the time the backup is executing. In the event that the release does fail,

restoring that same backup might also take significant time, wherein the system will be fully unavailable

to users.

In contrast, using EBS volume-level snapshots for the data and log volumes for the RDBMS, the amount

of time required to create the backup artifact is only as long as it takes for the system to pause writes

and flush the write buffers. This, in contrast to RDBMS-based backup processes, consumes far less time

by orders of magnitude and introduces only a slight impact to the systems’ performance for a much

shorter period of time. Additionally, and more importantly, the number of snapshots that have been

created has a novel cost as the cost of a given snapshot is related only to the delta between the previous

snapshot and the current one being created. In terms of the ease of use in restoring the snapshot, the

amount of time required to reconstitute snapshots into volumes is significantly less than that of having

to restore or rebuild an RDBMS server from a backup file, often only requiring a reattachment of the

restored volumes and a restart of the database server itself. Additionally, leveraging more managed

services such as Amazon RDS can simplify the operational management of the database servers

themselves and provide a far simpler interface for managing database snapshots in one place.

Knowing how snapshots for volumes operate along with the nominal additional operational cost

compared to other backup methodologies, the concept of versioning production database snapshots is

far less of a daunting and costly endeavor. Using a simple tag-base identifier to associate a pre-release

database snapshot to the given source control ID of the change being released, the full context of the

system prior to applying the change is organized in such a way to enable the team to re-create the

environment to debug issues from release failures, system performance problems to end-user reported

bugs.

Implementing with AWS The strength of a DevOps strategy and an enterprise’s ability to fully leverage the benefits of realigning

its IT organization to it is heavily dependent on the organization’s cloud strategy. Utilizing features such

as volume snapshots and removing the hard-limit ceiling on storage and server provisioning is only

possible by breaking out of the physical data center into the public cloud. The more features,

configuration options and architectural strategies and implementation choices that are available to

DevOps teams, the more creative they can be with their technical solutions while still being cost-

conscious. AWS provides most capacity with the richest Infrastructure as a Service portfolio available on

the market today and has been noted as the clear leader in Gartner’s Magic Quadrant for Cloud

Infrastructure as a Service5.

5 http://www.gartner.com/technology/reprints.do?id=1-1IMDMZ5&ct=130819&st=sb

http://www.gartner.com/technology/reprints.do?id=1-1IMDMZ5&ct=130819&st=sb



The result: combining existing, familiar toolsets, ownership-focused and technology-aligned teams with

the flexibility and on-demand nature of the Cloud nets the perfect catalyst for enabling DevOps in the

enterprise.

DevOps for the Enterprise - MyrtecEnabling and facilitating teams working within a DevOps process...

Documents

Transcript of DevOps for the Enterprise - MyrtecEnabling and facilitating teams working within a DevOps process...