DevOps at Capital One Focusing on Pipeline and Measurement

@TopoPal

Tapabrata “Topo” Pal

tapabrata.pal@capitalone.com @TopoPal

@TopoPal

Capital One

! Millions of accounts! One of the largest Digital Banks! #1 Information Week’s Elite 100! ~ 20 years old

@TopoPal

Different DNA

! Build our own software

! Build on public cloud

! MicroServices

! Open Source

! DevOpsSec and Continuous Delivery

@TopoPal

• Enterprise Architecture• DevOpsSec Strategy Owner• DevOps Evangelist

• Shared Technology Group• Product Manager of Continuous Delivery

Tools Platform• DevOps Evangelist• Core Contributor and Community

Manager of Hygieia

Personal Journey

@TopoPal

@TopoPal

• Waterfall• Manual Build• Manual Deployment• Manual Test• Data Center• Closed Source First

• Agile• Automated Build• Automated Deployment• Automated Test• Public Cloud• Open Source First

Agile & DevOps Transformation Journey

@TopoPal

Mostly Out-Sourced Mostly In-Sourced

Agile & DevOps Transformation Journey

Vertical Silos Product Team

Dev, Ops, QA, RM Engineers

@TopoPal

! DOES 2014 Building out Automation steps

! DOES 2015 Scaling DevOps, Open Source, Cloud, Innovation

! DOES 2016 Measure, Improve, Mature

@TopoPal

Typical DevOps Success Story

Code Commit Random 100s /day

Deployment

Prod

Manual Automated

Integration Monthly 15 mins

QA, Perf Monthly 4 / day

Monthly/Quarterly

Once / sprint

Testing Manual Automated

@TopoPal

2016What’s in your pipeline?

@TopoPal

http://www.devopsdays.org

@TopoPal

Deliver High Quality Working Software Faster

@TopoPal

Deliver High Quality Working Software Faster

• No security flaws

• No legal flaws

• Minimum defects

• All levels of testing done

• Code reviewed and source controlled

• Across LOBs, Shared Services and 3rd Parties

• Tested end-to-end

• All dependencies are satisfied

• How fast? ASAP?

@TopoPalhttps://upload.wikimedia.org/wikipedia/commons/c/c8/Can_We_Do_it_Better_or_Faster...We_Want_Your_Ideas_-_NARA_-_534240.jpg

@TopoPal

@TopoPal

Feb 8, 1700 — March 17, 1782

Daniel J. Bernoulli

@TopoPal

Constrict flow, Increase Speed, Lessen Pressure

https://www.khanacademy.org/science/physics/fluids/fluid-dynamics/a/what-is-volume-flow-rate

@TopoPal

Commit

Deploy

@TopoPal

http://www.netuba.org/

@TopoPal

https://en.wikipedia.org/wiki/Oil_refinery

@TopoPal

https://commons.wikimedia.org/wiki/File:US_Navy_060906-N-8257O-026_Damage_Controlman_1st_Class_Petty_Officer_Derrick_Harney_assists_his_students_in_repairing_a_broken_pipeline_during_the_hands_on_patch_training_portion_of_the_Damage_Control_Wet_Trainer.jpg

@TopoPal

• Design• Measure• Improve

Pipeline

@TopoPal

Pipeline Design

@TopoPal

Pipeline must have 16 gates Source code version controlOptimum branching strategy

Static analysis> 80% Code coverage

Vulnerability scanOpen source scan

Artifact version controlAuto provision

Immutable serversIntegration testing

Performance testingBuild, Deploy, Testing automated for every commit

Automated Change OrderZero downtime release

Automated rollbackFeature Toggle

@TopoPal

Pipeline Measurement

@TopoPal

https://devops-research.com/

@TopoPal

https://devops-research.com/ https://github.com/capitalone/Hygieia

@TopoPal

Increase Speed = Reduce Wait Time

@TopoPal

Opportunities

• Branching Strategy• Process

@TopoPal

Pipeline Improvement

Improve Branching

@TopoPal

Branching

• We recommend “Trunk based” development.• Other option:

@TopoPal

Pipeline Improvement

Improve Process

• Automate Release Process• Revisit Audit & Compliance

@TopoPal

Risks are real

• Intentional damage• Unintentional damage• Untested code in production

But….There is a better way

@TopoPal

Hypothesis

• DevOpsSec & CI/CD provide better controls• A model with ~30 practices can satisfy audit

and compliance• If everything is source code, no one needs

access to production• For emergency, “Break Glass”

@TopoPal

Result

Production Release 1+ / dayOnce / sprint

# of Applications with Release Automation: 20+

Max. # of Releases in 1 day for 1 Application: 34

With “Segregation of Duties”

@TopoPal

Goal

Release Automation

without

classic “Segregation of Duties”

@TopoPal

Coming Soon to Open Source

• A secure & compliant pipeline model• A forked and enhanced version of “LGTM”

@TopoPal

@TopoPal

Thank You!