1 University of Oulu, Tampere University of Technology ... · University of Oulu, Tampere...

DevOps in Practice

University of Oulu, Tampere University of Technology, Aalto University DIMECC - N4S - 13.12.2016

1

Background and ObjectiveMeasurable DevOps: Best Practices and Anti-Patterns in 5 selected companies

● Several research topics○ Practices applied in DevOps projects○ Elements of DevOps in company practices○ DevOps Mindset and Culture○ The impact of DevOps○ Customers’ needs and benefits of DevOps practices○ How DevOps builds quality


2

Background

3

1. Selecting companies and setting

objectives

N4S Sync and Follow-up teleconferences *Cases: Internet based services with small co-located teams

2. Company visits

Interviews and Observations: 5 persons per

company

3. Analyzing

Qualitative Analysis:

Thematic coding and discussions

4. Reporting and publications

Results:• Publications• Workshops in N4S Review

e.g., initial results in N4S review 26.-27.April, DevOps impacts workshop

DevOps Understanding


4

".It is as a culture, as a way of

doing development.This team does pretty much

everything, from beginning to the end. But, related to that are all the, tools in my opinion, you need to have the tools in order to make

the process work. " (Gofore)

"..the team is able to do all the development and the

operations, instead of only development and then handing over the software to others to

operate" (F-Secure)

"doing DevOps is when you have full responsibility..you tend to make

the culture of quality..it makes it easy to reason yourself that you

should run or build or code this kind of way so that you can be sure that it will run in production smoothly"

(Yle-Reaktor)

" the most important thing is that, we same persons who are developing the applications are also persons

who are maintaining those in production. If something goes wrong

then these same persons are investigating the problems. And fixing

those. (Fonecta)

"..the team has the complete control over the

infrastructure, and the deployment and, every

possible thing that can be automated is automated"

(Solita)

DevOps UnderstandingDevOps is about

• Sharing common goals, incentives and agile culture amongst development team, operations and business

that is achieved by • having a single cross-functional development team taking the

responsibility for the whole service from requirements to running it in production and,

• Automation of manual activities as far as possible

in order to • Reduce workload, increase transparency and reliability as tacit

knowledge becomes explicit.• Delivering customer value fast, experimenting and driving the

development with fast feedback.


5

Development Context


6

Case A Case B Case C Case D Case E

System Web-based road maintenance reporting tool

Websites for safety at work, occupational health and quality portal

Web-based application to access and manage acquired services by customer

REST API for television, radio and other media content

Security cloud services

Customer National Transport Agency An independent research organization in the field of occupational health and safety

Other product teams Other product teams Other product teams

End users National Transport Agency, Regional transport and environment centres and road maintenance contractors

Occupational health providers, researchers,employers and all people interested in occupational health topics

Companies and private people.

Product team, external developers

Corporate customers and external developers

Deployment Environments

CI, test, staging, production CI, Dev, acceptance and production

CI, QA/Test, Production CI, Test, Production CI, Alpha, Beta and Production

Deployment artifacts JAR file Docker image AWS machine image Debian package AWS machine image, Docker container

Deployment frequency Daily to staging Several times during the two weeks sprint cycle

Daily Daily 1-2 weeks from Alpha to Beta to Production

Tools Deveo; Jenkins; Selenium; Ansible

GitHub; Jenkins, Docker, Ansible Git; Jenkins; Chef; New Relic; Git;Jenkins, Puppet, New Relic

Bitbucket; Jenkins;Amazon CloudFormation; Kinesis

Programming languages Clojure Java, JavaScript, Node.js, Python Java, Scala and node.js Clojure, Scala Python, Java and C

Deployment pipelines


7

Practices Overview

ETUNIMI SUKUNIMI

DIMECC 30.6.2016

LOREM IPSUM8

Practices Summary description Cases

A B C D E

Automated deployment mechanism

Automated deployment mechanism. Deployment to production by development team.Deployments were triggered from Jenkins (Case A, B, C), Amazon CloudFormation (Case E) and using company tailored shell script (Case D).

x x x x x

Infrastructure as Code Version controlled deployment and configurations scripts. Environment provisioning and management using Ansible (Case A). Chef ( Case C), Puppet (Case D), and Amazon CloudFormation (Case B and E)

x x x x x

Continuous Monitoring Monitoring of system health checks using New Relic by developers using radiators x x x x

Trunk-based development Frequent merges of branches to the mainline in all Cases in order to minimize effort in resolving conflicts resulting from code merges. In Case D, feature branches were released and thus maintained for some APIs.

x x x x x

Change-based code review Code reviews done prior to merging code to mainline in Case A, D and E and after merging to mainline in Case B. Main purpose: to ensure quality and an opportunity to knowledge about coding style and standards. Factors in code review considered important: knowledge of the reviewer, size of code changes to be reviewed and review environment support. .

x x x x

Continuous Integration CI to execute unit tests and build deployable packages for all Case using Jenkins. Additional automated integration tests for some specific software components for case E Radiators and dashboards used to communicate the status of builds.

x x x x x

Agile and Lean practices (and their adaptation)

No explicit agile method in Cases, except for Case B that was using Scrum. However, implicit adaptation to agile Scrum and lean Kanban. From Scrum practices e.g., daily/weekly stand-ups meetings, retrospectives, task estimation. From lean, task tracking and a Kanban board were used in all cases. Rationale for not using Scrum in its entirety was: to be more efficient and less constrained to the procedures.

x x x x x

DevOps Practices - Infrastructure as Code• Rationale

• Automates the provisioning of infrastructure so that it is easy and repeatable

• Allow development team to provision environments by themselves

• Minimize tacit knowledge; speeds up application deployments and make deployment environments similar infrastructure-wise

• Lessons Learned• Similar coding practices to the scripts e.g., review, version control

• Choice of tool influenced by several factors e.g., developer preferences, budget constraints, ease of integration

• Shared access, version controlled scripts help to motivate their use and modification in Dev team

• Some things are difficult to automate e.g., network; may serves as documentation


9

“There's some saying about DevOps that, you do not treat servers as pets. You are treating them as cattle” (Yle-Reaktor)

Foundational Practices - Agile and Lean

• Rationale• Incremental development and delivery

• Smaller tasks can effectively be solved by developers and are deployable even when they don’t offer functionality

• Kanban board for visibility and ensuring non-accumulation of tasks

• Lessons learned• The need to educate customer about Agile and Lean (Fixed price project are difficult with Agile)

•

ETUNIMI SUKUNIMI

DIMECC 30.6.2016

LOREM IPSUM10

Foundational Practices - Code Reviews

• Rationale• Improves quality of code

• Good for sharing knowledge, educating and enhancing skills e.g., about coding style, standards

• Lessons Learned• Code review done before/after merges

• Review is effective when considering: knowledge of reviewer, size of code changes to be reviewed, review tool support

• Review procedure is affected by criticality of code changes e.g., bug fixes

ETUNIMI SUKUNIMI

DIMECC 30.6.2016

LOREM IPSUM11

DevOps Practices - Automated Deployment Mechanism • Rationale

• Eliminate handovers and allow developers to deploy own application

• Make deployment easier and faster

• Lessons Learned• Immutable infrastructure for deployment mechanisms

• Container based deployments are advantageous in terms of time, reliability but require changes to architecture and application monitoring

• Testing deployments facilitated by monitoring e.g., canary release

• Roll-back e.g., deploy previous build packages


12

DevOp Practices - Continuous Monitoring of Production by Development team

• Rationale• Important for ensuring quality of application, deployments and reliability of

infrastructure, i.e., a form of quality control • Crucial to see how things behave in production - feedback

• To observe the effects of small changes with the possibility to revert

• Lessons Learned• Modern monitoring tools, e.g., New Relic facilitate agents that help monitoring

at different layers• Effective monitoring is particularly challenging and important in micro-services,

e.g., the use of diverse monitoring tools and metrics• Setting up new values to monitor is fairly simple.

• Developers create metrics that they find useful and add them to the dashboards


13

DevOp Practices - Continuous Monitoring Examples


14

Company D Company E

What is monitored by Dev team

- CPU utilization- Disk space usage- Number of requests per minute to API- API average response time- Response times - Error rate

- CPU utilization (in some services automatic scaling)- Service load and availability- Number of requests to the API- Latency of the API requests - Errors (Front-end and server-side)- Request execution times at back-end- Service quality metrics, identifying anomalies in the service- Check for certificate expiration

How is it communicated

- Physical monitors in team premises

- E-mail and Chat alerts- PagerDuty-alert notification

system sending SMS or emails

- Physical monitors in team premises (visual monitoring)

- Email alerts

DevOp Practices - Mechanisms for communicating Monitored Data

Radiators

ETUNIMI SUKUNIMI

DIMECC 30.6.2016

LOREM IPSUM15

Chat Applications

E-Mail Alerts

PagerDuty

Amazon Automatic Alarms

3. Work Morale

4. Quality

5. Change in Business Thinking

1. Development Speed

2. Productivity of Operations

Impacts of DevOps: Five Themes 1. Development Speed and

Continuous Deployment• Fast lead times into production• Continuous deployment and

feedback

2. Productivity of Operations Work• Removing the Dev <-> Ops barrier

and handoffs• Increased level of automation • Less operations workload• Direct cost savings

4. Quality• Increased developer responsibility for

the production quality• Increased confidence to the quality of

production deployments and the team’s capability to handle any issue

3. Work Morale• Team autonomy and ownership &

responsibility of their services• Developers’ job satisfaction and

well-being

5. Change in Business Thinking• Mindset change from project towards

service thinking


16

Speed of Development

Ops Workload Decrease &

Redistribution

Removing Barriers and

Handoffs

Team’s Autonomy, Ownership and

Happiness

Improved Quality

Cost Savings

Reduced Deployment Risk

and Stress

Automation as DocumentationDiverse

Developer Skills and Experience

Service Thinking & Lean Flow

Improved Feedback and Transparency

BIZ

DEV

Small Deployments

Continuous Deployment &

Experimentation


17

18

Cultural Pyramid of DevOps

Challenges hindering the impact and cultural change

• Sacrificing quality • for the speed of the development

• Resources, budgets and bottlenecks• restricting the development pace

• Required variety of developer skills & expertise• challenging to achieve, not all are willing to learn

• Multitude of the responsibilities for developers to handle• the team becomes overwhelmed with tasks

• Making the cultural change• in management and throughout the organization

• Fast feedback cycle breaks• lead time of user feedback• Availability of the monitoring data

• Technical challenges• automation, infrastructure and monitoring


19

20

Impacts ChallengesSpeed of development and continuous deployment

Sacrificing quality for the speed of the developmentResources, budgets and bottlenecks

More diverse developer skills and responsibilities

Required variety of developer skills & expertiseMultitude of the responsibilities for developers to handle

Removing barriers and hand-offsLess maintenance workload and sharing the Ops responsibilitiesDevelopers' motivation and happiness, autonomy and ownership

Making the cultural change in management and through the organizationService thinking

Improved feedback from the market or users Immediate monitoring data

Lead time of user feedbackAvailability of the monitoring data

Increased confidence in production quality and reduction of deployment risk & stress

Technical challenges of automation, infrastructure and monitoring

Automation as documentationDirect cost savings

1 University of Oulu, Tampere University of Technology ... · University of Oulu, Tampere...

Documents

Transcript of 1 University of Oulu, Tampere University of Technology ... · University of Oulu, Tampere...