ABN AMRO Transforms with CICD to Accelerate Software Delivery and Improve Security

• DevOn Summit

• Utrecht 14th Mar 2018

• Stefan Simenon

• Stefan Simenon

• Head of IT Tooling & Software Development

• Email: stefan.simenon@nl.abnamro.com

• Tel: + 31 6 51478665

• Studied Physics & Information Technology

• 20+ years IT experience in various roles

• Currently responsible for Tooling, Software Quality & CICD in

ABN AMRO IT department

• Speaker at several software conferences like Jenkins World,

XebiaLabs DevOps Leadership Summit, AllDayDevOps,

Software Quality conferences

Introduction

ABN AMRO is a leading bankwith an operating income of EUR 8588 million

22,000 employees servicing retail, private and corporate finances worldwide

Headquartered in Amsterdam

5,000 associates working in IT

300+ agile teams

ABN AMRO

Many manual handovers and approvals

Long lead time for software delivery

Software quality issues found at a late stage

Code merging happening at a late stage

Inefficient cooperation between DEV and OPS

Big non-frequent releases to Production

Challenges Faced at ABN AMRO

Financial services market is growing fast, on multiple fronts

WaterfallFull Agile enterpriseTraditional Enterprise,

Agile teams

CICD / DevOps Full DevOps enterprise

Agile & DevOps transition

Produce automated builds and detect errors as soon as possible, by integrating and testing all changes on a regular (daily) basis.

High frequency delivery of a tested functional piece of software that can be deployed to production rapidly.

Fully automated process including deployment to production without human interaction.

Continuous Integration Continuous DeploymentContinuous Delivery

The case for faster response to client needs is clear

CICD program: Set-up

Extend techno-logies

Move to ET

Automated production

release

Mature in UT/ST

Start CICD in UT/ST

PAVE THE WAY

ToolingInfra prerequisitesIntegrationPipelines

MAKE IT HAPPEN

Change ManagementMindset & BehaviourSimplify processesCoaching for agile teams

Front end, Java

CICD program: Approach

• CICD is not only about tooling but mainly mindset & behaviour, a changed Way of Working andprocess improvements.

• The project organisation is set up into a cluster with a central and a decentralized orientation.

1. Pave the way: set up the conditions for the teams to get working.

2. Make it happen: the actual ‘decentral’ CI/CD implementation within the teams.

• Agile teams will be supported once the right tools are available, so start with Java/Front End/BPM TIBCO.

• Strong alignment across DEV, OPS and SECURITY departments

• We know other large companies which need 3 - 8 years, and changed their approach along theway.

• Therefore we keep the overall stages in mind, but plan for the coming three months. Focus on learning and improving instead of long term planning.

Pave the Way – Results so far (1)

• All tools required for Continuous Integration implemented and rolled out• Various Continuous Integration pipelines defines and implemented• Pipelines and their integrations are continuously improved and extended

• JIRA agile toolset defined and implemented• Standard Way of Working defined and roll out in progress• From 2000 to 10000+ users in 2,5 years

• Tooling for release and deployment management selected: XL Release and XL Deploy• Release & Deployment management WoW defined and roll out in progress• Test & Production environments for XL Release and XL Deploy delivered, installation process

has been fully automated• Standard CD pipeline for Java/WebSphere, Open Banking and IIB delivered and connected to

standard CI pipeline. • VSTS selected and implemented for applications based on MicroSoft technology• > 100 applications onboarded for automated deployments• > 500 XL Release users

Pave the Way – Results so far (2)

• SonarQube for code quality, HPE Fortify for secure coding, Nexus Life Cycle for OSS library management

• Governance to manage software quality setup and roll out in progress• Build breakers defined and roll out in progress

• Tools implemented to enable automated testing• Test Service Virtualization rolled out• Automated test data management and governance implemented and roll out in progress• Automated Test framework defined and implemented

• Mainframe tools upgraded to latest versions• Identified strategy to clean unused components and activities to recompile programs

based upon latest Cobol compiler 6.1. This will lead to improved memory usage and less MSU usage.

• Mainframe pipeline based on Compuware TOPAZ, ISPW, Jenkins and SonarQube in progress.

Midrange Build & Delivery pipeline: orchestration

Acceptance environment (ET) Production

environment (PRD)

Test environment(ST)

Zero touch platforms

Deployment

Build

Static secure code

Package

Develop

Source code

Build &Unit

Tests

Code quality scans

ContinuousIntegration

Build artefacts

Continuous Delivery

Test data mgmt

ATAF Test suites

Release management

Tooling

Java

Front End

BPM/TIBCO

MicroSoft

Siebel

PowerCentre/

ETLIIB

Mainframe

CoTS

Mobile

Pipelines within ABN AMRO

Dependency scan

Standard CI pipelines within ABN AMRO and build breakers

Check out project from SCM

Developer triggers build

Build project and execute unit tests

Code quality scan

Secure coding scan

PublishDeployable

artifact

N

Y

Build breaker criteria and governance

• Software quality governance in place.

• If software quality criteria are not met, build will fail and software developer needs to fix/improvethe software before being able to publish a deployable artifact.

• Software quality criteria and roll out of build breakers are defined by a development community consisting of central quality teams, representatives in agile teams, our application development partners and security department.

• Initial build breakers in place for software quality, secure coding and dependency management, build breakers criteria will be strengthened in the future.

• Build breakers lead to improved software and less exception discussions in agile teams.

• Senior management commitment in place.

An IT4IT organisation has been set up to enable the CICD implementation

JIRA dedication team

Software Logistics team

Application Deployment

support team

Test tooling team

Application monitoring team

Change & configuration mgmt team

Portfolio mgmtteam

Application logging team

Implement tooling upgrades

Implement new tools

Enhance and improve CICD pipelines

Implement new CICD pipelines

Handle user management

Support Agile teams

Conduct incident & problem management

Mainframe modernization

1.

Automate all repetitive tasks

2.

Integrate quickly and often

3.

Everyone is equally responsible

4.

Keep changes small

5.

Get continuous feedback

ABN AMRO CICD Key Principles

Make It Happen – Results so far

• CICD summer event held incl. CICD leadership program, demo’s, best practice sharing, trainings • Change management program set up with lots of focus upon Mindset & Behaviour

• Various communities set up• Internal meet ups and hackathons regularly held• Platform set up in which teams can present their successes, failures and how they learn• Internal meetups held with external speakers and tooling suppliers (eg. Jez Humble, Josh

Long, Cloudbees, Sonatype, XebiaLabs, SonarSource)

• CICD coaching framework defined and rollout in progress• 100+ boot camps organised and teams coached• Framework based upon certain set of deliverables and team needs• CICD E-Learning module delivered and rolled out

Test environment uptime improved

Improved code quality & secure coding

Improved cooperation across stakeholders

Improved time to market

Improved development processes

Realised benefits within ABN AMRO

Source code mgt

Build & Unit test

Code quality review

PackageDevelop Compo-nent mgt

Deploy Release tests (ET) Deploy

Continuous integration

Continuous delivery

Continuous deployment

Prod checksDeploy Test (ST)

Zero touch platforms

Code push flow Deployment flowBuild, QA and package flow

x3 deployments to UT x2,5 deployments to ET+20% successful Builds -100% Package creation time -75% Testing time

We never thought it would be possible to develop, test

and deploy something completely in one sprint

I-Markets doubled velocity after 1 sprint containing

CICD improvements only

From 4 Internet Banking releases to 18

releases per year

Core review times have been shortened and

violations when merging are being

prevented

Changes are being rolled out as soon as

they are available

Increased velocity

Private Banking Interlnational team reduced

build from 5 hours to 5 minutes

First continuous deployment realised by identity access

mgmt team

Release times halved for teams using XL Release

Take aways

Senior management commitment & involvement

Invest in reducing technical debt

Create a safe environment (failing is ok)

Do not focus on tooling only

Do not underestimate the journey and complexity

Do not focus on long term but small improvements

Database automation

Automate and improve tooling pipelines

Hybrid cloud strategy

Further transform to DevOps

Improve WoW and Mindset & Behaviour

Facilitate increased team autonomy

Way forward

CICD metrics

22

Questions