Intelligent CloudMake your CI/CD pipeline a tool for driving your DevOps live site

culture

Jørn Floor Andersen

ABOUT CLOUDEON

Specialized - Denmark's largest Cloud and Digital Pure Play provider

Visionary - always challenging status quo and forward-thinking

Passionate - we believe in change, and are driven by making a difference for

cl ients.

Trustworthy - serving major cl ients in Finance, Energy, Retail, Logist ics and

Public

Growing - started operations in February 2016 , today 75 Digital & Cloud

specialists

International – Offices in Copenhagen, Aarhus and London

T E C H N O L O G Y C H A N G E S

B U S I N E S S

x

Agenda

• Introduction• What is ”Live site culture”• 10 Steps

• Idea/Requirement Handling• Version Control• Code Review• Test• Automated Build (CI)• Automated Deploy (CD)• Monitoring• Alerts & Actions• Security• Release Gates

Introduction

Like ten years ago, when teams were doing daily stand ups and thinking that was making them practice Scrum, today everyone is making a CI/CD pipeline and thinking now they are doing DevOps. In this session we look at how you can take your CI/CD pipeline one step further and integrate all the other stuff that makes up your application and be a tool for driving a DevOps culture. We will look at integrating Infrastructure as Code (IaC), application monitoring, platform monitoring, alerts handling and quality gates.

Introduction

CLOUD CHANGES EVERYTHING

• New possibilities• No more infrastructure• Self provisioning of platform services• Huge service catalog• Access to production• Easy Automation• Offloading work streams (patching, DB optimization, …)

• New roles and responsibilities• Developer must embrace production and operations

• Developers now have oncall duties

• Product must be build Ops’able• Data Management and GDRP

• Security

• Monitoring/Alerts

• Taking action on live site issues Is part of app design/architecture

• Must be reviewed as part of functionality/features

• New process of involving business• Must continuously be involved to define, prioritize, give feedback

Introduction - Definition of Done Vision

• PBI (p)reviewed and fulfilled

• UX requirements reviewed and fulfilled

• Security requirements reviewed and fulfilled

• Data handling and GDRP requirements reviewed and fulfilled

• Code reviewed

• Code put under version control

• Verified OSS frameworks and components used

• Unit test (L0, L1) written and green

• Change + L0 tests included in automatic build (CI)

• L2-L3 test written and green

• Code Instrumented and metrics defined

• “Monitorable” metrics and Alerts added to Azure Monitor and Application Insights

• Self-healing (Ops) with watcher tasks and actions added to Azure Automation

• Change + L1,L2,L3 tests Included in deployment pipeline (CD)

• Documentation generated

What is ”Live site culture”

• Getting everyone to feel and have ownership of the full product, from first idea/requirement to running and supporting the software in production.

• A full CI/CD pipeline must embrace“Live site culture” supporting it from idea to working software

Picture copyright, The Devops blog, Sam Guckenheimer

10 Steps - Idea/Requirement Handling

• Idea/requirement handling• Common shared backlog• PBI preview• Keep notes and conversation

on PBI• Think Months/Weeks/Days/Hours

• Recommendations• Do use an agile planning tool, that is integrated with the other DevOps tools• Do comment directly on the requirements (work items) themselves• Do breakdown task in sizes that can be handled in less than a day• Do set aside a band for technical debt (service health and optimization)• Do use VSTS Business/Architecture field to isolate discussing technical enhancements• Consider breaking down requirements in three layers• Consider having sprints (iterations) defined across teams

10 Steps – Version Control

• Version Control• If it is not under version control it doesn’t exist• Must be able to build from assets under version control• Combine entry with Code Review - Pull Request (PR)

• Recommendations• Do have a branching strategy in place and enforce via

policies• Do only accept Pull Request (PR) into master• Do require associating work item with PR for traceability• Do not keep secrets in source code• Do not place binaries into source control• Do not squash branch upon completing the pull request• Do delete branch after merge • Consider using feature flags when using trunk-based development• Consider using branch name as PR title• Consider using Git

10 Steps – Code Review

• Code Review• Have code review as integral part of development

practice, so:• Knowledge is shared about code base• Learnings are shared• Individuals makes the extra effort up front

• Recommendations• Do include the purpose of the Pull Request• Do remember that anyone in the company could be reading this Pull Request, so the content

and tone may inform people other than those taking part, now or later.• Do be explicit about what feedback you want, if any• Do @mention individuals or teams that you specifically want to involve in the discussion and

mention why• Consider providing an overview of why the work is taking place• Consider (p)reviewing the PBI as part of the exercise

10 Steps - Test

• Test• L0-L3 tests

• L0: ”Real” Unit test, no dependencies except for assembly or js file• L1: Unit test with single dependency• L2: Near fully functional testing, but with parts mocked out for convienience• L3: Fully functional testing end-to-end (requires working environment)

• Continue manual testing for:• Hard to configure tests• UAT• Bug bashes• Hotfix verification

• Recommendations• Do exercise L0 tests as part of the automated build• Do exercise L1-L3 tests as part of the Release Definitions• Do keep focus on test automation• Do combine with services for Load, ping and security testing and other functional testing• Consider keeping test code close to code being tested

10 Steps – Automated Build (CI)

• Automated Build (CI)• Needs to be automated and can build

from version control on a machine separate from the dev machine

• Eliminate ”Builds on my machine”

• Enforces first set of policies• ”Ensure good stomach feeling”

• Recommendations• Do have an automatic build on a dedicated

machine separate from the development PC• Do have a CI build• Do include L0 tests (and not L1-L3)• Do enable code coverage• Do publish source symbols• Do not publish any packages or deploy binaries to any environment• Consider having Build definitions as Code (yaml)• Consider using automatic checks for code quality and conformity, like:

• WhiteSource for open source checking (security, bugs and license issues)• SonarCube for code quality• VeraCode or OWAS Proxy Scan for security scanning

• Consider static analysis and stylecop analyzers

10 Steps – Automatic Deployment (CD)

• Automatic Deloyment (CD)• Automated build is for first verification and unit tests and must be fast

as it is part of ”inner-dev” cycle• A deployment pipeline breaks it up in stages enabling more

and more extensive verification• Dev, Test, Load, QA, Production environments

• Recommendations• Do have full automatic deployment ready from first build to ”Dev”• Do generate full environment with App AND Platform• Do parameterize your deloyments• Do deploy the same way for every environment• Do use IaC (e.g. ARM templates)• Do keep ARM templates under version control• Do continuously delete deployments from the Resource Group• Do incremental provisioning of environment on every deployment as part of IaC• Do include database changes as part of your pipeline• Do not share environment specific resources across environments• Consider running smoke/integration/performance tests in parallel on separate environments• Consider approval gates between environments to protect integrity and consistency• Consider provisioning a unique environment for performance tests and deleting after completion

10 Steps – Automated Deployment

• Automated Deloyment (CD) –continued• An important aspect of building cloud based systems is management; among others

policies for naming, tagging, resource types and locations, encryption, locking and cost

• Enforced via ”Azure Policy Service” or Lock resource +JSON + PowerShell/CLI

• Recommendations• Do have a naming standard (e.g. <company><system><service>[<instance>]<environment>)• Do enforce Tags• Do tag with CostCenter• Do have a CanNotDelete lock in place to protect

any critical resource• Consider tag with environment and system• Consider white list resource locations

10 Steps - Monitoring, Alerts and Actions

Monitoring is the act of collecting and analyzing data to determine the performance, health, and availability of your business application and the resources that it depends on.

• An effective monitoring strategy helps understand the detailed operation of the components of the application.

• It also helps increase uptime by proactively notifying of critical issues so that you can resolve them before they become problems

10 Steps - Monitoring, Alerts and Actions

• As the platform is now the responsibility of the dev team, including monitoring, discovering issues and hot fixing the following monitoring resources become part of the DevOps toolbox:

• App: Application Insights• Platform: Azure Monitor, Activity Log, (Service Health, Advisor)• App + Platform: Log Analytics (from other sources)• Platform: Management Solutions (e.g. Azure SQL Analytics,

Container)

• Alerts and Actions• ”Kill the big screens”

• Take action automatically (e.g. Restart App Service when load is more than 95% for more than 3 minutes)

• Define standard Alerts and Actions for platform services• Standard platform services (best practices for PaaS services)• According to application architecture (App specific)

Azure Monitor

ApplicationInsights

Log Analytics

Management Solutions

10 Steps - Monitoring, Alerts and Actions

• Application Insights• Application health

• Performance, page loads, AJAX calls, browser exceptions, users and sessions count

• Application Usage• Custom events and metrics from code

• Application Log consolidation• NLog, Log4Net

• Recommendation• Do use Application Insights• Do consolidate App log to Application Insights from logging frameworks (if used)• Do provide availability health check endpoints (AAI can probe)• Consider using Application Insights TrackTrace directly• Consider using Application Insights for Usage tracking• Consider configuring Availability tests• Consider providing compensating events for failures

10 Steps - Monitoring, Alerts and Actions

• Application Insights (continued)• How to enable

• Add Application Insight to platform services via ARM

• Add JavaScript to pages that should be tracked

• For ASP.NET Core based apps:

10 Steps - Monitoring, Alerts and Actions

• Azure Monitoring Services• Monitors

• Metrics• Activity Logs (performed by ARM)• Diagnostics logs (performed by resource)

• Logs data across entire stack• Infrastructure, Platform, Application

• Alerting on• Metrics and logs• Log search• Activity log

• Action Groups• Named groups of notifications and actions

that can be reused in multiple alerts

10 Steps - Monitoring, Alerts and Actions

• Automate• Define: Create Alert via ARM template

• Act: Create Action Group via ARM template • Call Azure Function, Azure Automation, Runbooks, Web Hooks via Action Groups

• Automate above functions via ARM templates

• Recommendations• Do define and follow a monitoring strategy

• Do include Alerts and Actions for App and Platform services monitoring in your IaC

10 Steps - Monitoring, Alerts and Actions• Metric Alert via ARM template

• resourceId: the resource it isalerting on, e.g. the App Service

• criteria: Rule

• actionGroupId: What should happen?

{"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#","contentVersion": "1.0.0.0", "parameters": {

....

"resources": [{

"name": "[parameters('alertName')]","type": "Microsoft.Insights/metricAlerts","location": "global","apiVersion": "2018-03-01","tags": {},"properties": {

"description": "[parameters('alertDescription')]","severity": "[parameters('alertSeverity')]","enabled": "[parameters('isEnabled')]","scopes": ["[parameters('resourceId')]"],"evaluationFrequency":"[parameters('evaluationFrequency')]","windowSize": "[parameters('windowSize')]","criteria": {

"odata.type": "Microsoft.Azure.Monitor.SingleResourceMultipleMetricCriteria", "allOf": [{

"name" : "1st criterion","metricName": "[parameters('metricName')]","dimensions":[], "operator": "[parameters('operator')]","threshold" : "[parameters('threshold')]","timeAggregation": "[parameters('timeAggregation')]"

}]},"actions": [

{"actionGroupId": "[parameters('actionGroupId')]"

}]……

{...

"resourceId": {"value": "/subscriptions/subscription-id/resourceGroups/resourceGroup-

name/providers/Microsoft.Compute/virtualMachines/resource-name"},"metricName": {

"value": "Percentage CPU"},"operator": {

"value": "GreaterThan"},"threshold": {

"value": "80"},"timeAggregation": {

"value": "Average"},"actionGroupId": {

"value": "/subscriptions/replace-with-subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Insights/actionGroups/replace-with-action-group"}

}}

10 Steps - Security

• Security and privacy

• Recommendations• Do use Azure KeyVault• Do not include secrects in code, config files or automation code• Do limit the deployment Application Identity access to a subscription or resource

group• Do encrypt at rest• Do not roll your own custom security implementation, but use built-in standards• Do protect you API transport with SSL (require HTTPS/TLS 1.2 as a minimum)• Do protect and restrict API access with OAuth2 through Azure AD• Consider using role-based access control• Consider having Azure KeyVault as shared resource

10 Steps – Release Gates

VSTS Release Gates• Automatic collection of health signals from

external services, and promotion of the release when all signals are successful at the same time or stop the deployment on timeout

Scenarios• System Health• Approvals/Regulatory• Change/incident/issue management

Gates• Azure Monitor Alerts• Azure Functions/REST API• Query Work Items

10 Steps – Release Gates

• VSTS Release Gates (continued)• Configure Azure Monitor Alert via Release Definition in VSTS UI or YAML• Note: VSTS still only see

Classic Alerts

• Recommendations• Do use Release Gates when an environment is regulated (e.g. UAT env)• Consider using continuous monitoring to automate release gates• Consider Release Gates, when checks can’t be resolved automatically

Take Aways

• The Dev Team is responsible for the App AND the Platform

• New tasks for the developer, but also old ones going away

• Writing and running ops’able products requires knowledge sharing

• Strategies, taxonomy and automation enables room for creativity

• Live site culture comes from being involved

• Do chose PaaS over IaaS

Expo Sponsors

Event Sponsors

Expo Light Sponsors