Part of the pipeline: Why continuous testing is essential

Adam Auerbach @Bugman31 Tapabrata Pal @TopoPal

@Bugman31 @TopoPal

Capital One

! 70 million accounts! One of the largest Digital Banks! ~ 20 years old

@Bugman31 @TopoPal

“…the winners in banking will have the capabilities of a world-class software company.”

Rich Fairbank Founder & CEO

Capital One

@Bugman31 @TopoPal

Different DNA

! Build our own software

! Build on public cloud

! MicroServices

! Open Source

! DevOpsSec and Continuous Delivery

@Bugman31 @TopoPal

@Bugman31 @TopoPal

Deliver High Quality Working Software Faster

@Bugman31 @TopoPal

Deliver High Quality Working Software Faster

• No security flaws

• No legal flaws

• Minimum defects

• All levels of testing done

• Code reviewed and source controlled

• Testing of application, configuration, scripts etc.

• Across LOBs, Shared Services and 3rd Parties

• Tested end-to-end

• All dependencies are satisfied

• How fast? ASAP?

@Bugman31 @TopoPal

http://www.netuba.org/

@Bugman31 @TopoPal

https://en.wikipedia.org/wiki/Oil_refinery

@Bugman31 @TopoPal

https://commons.wikimedia.org/wiki/File:US_Navy_060906-N-8257O-026_Damage_Controlman_1st_Class_Petty_Officer_Derrick_Harney_assists_his_students_in_repairing_a_broken_pipeline_during_the_hands_on_patch_training_portion_of_the_Damage_Control_Wet_Trainer.jpg

@Bugman31 @TopoPal

Code Build Release MonitorDeploy + Test Execution

App

Test

Infra

DEV INT QA PERF PROD

DEV INT

SEC

QA SEC PERF PROD

DEV INT QA SEC PERF PROD Infra

App

Flow Feedback

Automated Audit and Security Controls at every step

@Bugman31 @TopoPal

Backlog

Design

Sprints

Hardening

@Bugman31 @TopoPal

Continuous Testing

High Quality Working Software

@Bugman31 @TopoPal

Continuous Testing

! Static Scan

! Unit Testing

! Acceptance Test

! Performance Test

! Security Tests

@Bugman31 @TopoPal

Challenges

! Devices and Browsers

! Test Environments

! Test Data

! Dependencies

! What & When

! Visibility and Transparency

! People

@Bugman31 @TopoPal

Solution

! Devices and Browsers - Use SaaS

- Needs a bit of engineering

@Bugman31 @TopoPal

Solution

! Test Environments - On Demand, On the fly - Cloud

@Bugman31 @TopoPal

Solution

! Test Data - On demand

- Freshly loaded

- Production Like

@Bugman31 @TopoPal

Solution

! Dependencies - Service Virtualization

@Bugman31 @TopoPal

Solution

! What & When - No good answer

- Fail fast

- Keep pipeline speed intact

@Bugman31 @TopoPal

Visibility: Dashboard Everything

@Bugman31 @TopoPal

@Bugman31 @TopoPal

@Bugman31 @TopoPal

People

@Bugman31 @TopoPal

Scaling Continuous Testing: DIY

@Bugman31 @TopoPal

Scaling Continuous Testing: Community

@Bugman31 @TopoPal

Scaling Engineering: Internal Conference

1 location2 days1,200+ attendees 28 Tech Expo Booths52 Sessions0 Vendor

2 locations1 day

2,000+ attendees 54 Tech Expo Booths

146 Sessions0 Vendor

2015 2016

@Bugman31 @TopoPal

Any Question?