Using Process Maturity and Agile to Strengthen Cyber Security

"The views expressed in this presentation are those of the author(s) and do not necessarily reflect the official policy or position of the Air Force, the Department of Defense, or the U.S. Government."

Programs can meld Process Maturity, Agile Development, and DevSecOps to produce more resilient systems with reduced vulnerabilities

Bottom Line Up Front

But it takes commitment, data, tools and process discipline

Presentation Agenda

§ About OST§ Problem Statement§ “Agile5” Framework § DevSecOps§ Case Study: Increasing Velocity and Quality

with Agile5§ Application to Cyber Security§ Questions

Who is OST

We Support the Air Force

51 Task Orders Awarded180 FTEs 16 Air Force Bases$80M Contract Values Awarded

Problem Statement

§ High maturity process discipline (CMMI 5), responsive delivery (Agile/DevSecOps) and high quality (low defect rate) are like cost, schedule and performance on a program. One constraint always binds the others…

§ … Or does it?

§ This presentation offers an approach that can overcome limitations of the triple constraint and avoid compromise

Process Maturity

High Quality Responsive Delivery

You Probably Know Agile

Release

Preparation

SprintReview

SprintRetrospective

SCRUMPROCESS

SprintPlanningMeeting Daily

Scrum call

Potentially Shippable Product Increment

SCRUMARTIFACTS

Sprint Backlog

Product Backlog

Sprint Burn Down Chart

Product Increment

Story

SCRUMROLES

Product owner ScrumMaster

UpdateProduct Backlog

Grooming User StoriesEpics

Development Team

Applying CMMI Level 3 to Agile

Releases

Preparation SprintPlanningMeeting

SprintReview

SprintRetrospective

Daily Scrum

callCMMI L3 Process

Execution Potentially Shippable Product Increment

PP – Project PlanningRM – Req. Mgt.

RD – Req. DevTS – Tech. Sol.PI – Prod. IntegrationVR – VerificationCM – Configuration Mgt.DAR – Decision Analysis

PMC – Monitor & ControlRSK – Risk Mgt.IPM – Integrated Project Mgt.MA – Measurements

Sprint Backlog

Close the Gaps &Recommendation

PPQA– Quality Assurance

OT – Org. TrainingOPD – Process DefinitionOPF – Process Focus

Institutionalize

UpdateProduct Backlog

Grooming User StoriesEpics

Level 5 Tool Kit

§ Use data to create simulation to predict outcomes for velocity, defect density, utilization§ Adjust controllable factors to optimize performance§ Apply to “real world”

Applying CMMI Level 5 to Agile

1. Simulate2. Predict Outcomes3. Adjust Levers4. Repeat ….

Common goal for Organization’s success§ Dev team knows what Ops team looks for

§ Ops Team knows what Dev is working on§ They work hand-in-hand

You May Know DevOps

Automation of Continuous Delivery Pipeline

Recovery Enables Low Risk Releases

Measurement of Everything

Lean Flow Accelerates Delivery

Culture of Shared Responsibility

DevSecOps

DevSecOps

Continuous Security

FAA Spectrum Engineering and Assignment Group

§ Provide the safest, most efficient aviation system in the world

§ Key role in the protection of the National Airspace System (NAS)§ Secures, manages, and protects all civil aviation radio frequency spectrum resources

§ Sustain and Enhance the Spectrum Engineering and Automation Support

§ Provide system and software engineering services to protect interference-free communications, navigation, and surveillance systems operations for all U.S. civil aviation and CONUS military aviation in the NAS

§ Deliver new capabilities through Agile and CMMI Practices

Case Study

Customer Mission and Focus:

OST’s Mission and Focus:

§ Customer Objectives

§ OST Goals

§ Budget – Sequestration; Reductions

§ Team Satisfaction

§ Provides FAA Spectrum Engineering optimal cost efficient services

§ Improves performance of key modules

Case Study – Goal Formulation

Project Goal Based On: Goal That:

GoalImprove the velocity from the current baseline of Mean = 0.77; Std Dev = 0.22 to Mean =2.0 and Std Dev = 0.20 without sacrificing quality* and team satisfaction**

*Quality is defined as production defect density – less than or equal to 10%** Team satisfaction is measured by the employee survey – Avg. score 22

Case StudyProcess Performance Model – High Level

Face of model showing “Agile Stations” including design; development; test; rework.

Agile 5 –Release

Planning and Monitoring

• Number of story points that will be completed in a given release or sprint

• Velocity

• Before sprint starts• Twice a week for

forecast and sprint resource adjustments

SEAS Team /Corporate Process Group

SEAS PM /Scrum Master

Model Outcome Predicted Frequency of Use Created By Used By

Case Study – Model Details

§ Output = # of story points that will be completed for a given release/sprint length

§ Inputs = Development times, testing times, test case development times, defect densities, no. of resources, story points & no. of user stories

§ Each user story is rated by complexity:

§ Low : 02 Story Points

§ Medium : 08 Story Points

§ High: 24 Story Points

§ Developer Velocity is determined using story points and hours taken to complete

Case Study – The Journey

Sprint Monitoring§ Burn down§ What-if’s§ Control Charts

Data AnalysisBaselines (PPBs)§ Stratification§ Hypo Tests

Model (PPMs)

Sprint Planning

Goal

Iterative development(added “stations over time”)

• CAR results in changed baselines

• Model refinement

Hypo TestingRecalibration

• Partitioned dataset to build and test the model

• Intuition

Case Study - Results

Case Study – Outputs and Outcomes

OUTPUTS§ Velocity improved from (0.77, .23) to (1.38, 0.28)

with a production defect density of 3%§ Team felt a sense of accomplishment§ Made our “successes” more repeatable because of

the CAR§ Opportunity for professional growth: Impacts of

actions on efficiency and quality

OUTCOMES§ Productivity gain of $600K in 3 releases§ Customer received 20% more requirements than expected§ Didn't have to sacrifice stakeholder satisfaction for regulatory

compliance§ Received kudos from their management§ Available for use in other business areas§ Applied the case study to:

§ Improve the Testing Process§ Keep building upon the model§ New team members

§ Predicts interim and final release outcomes, number of stories that will be delivered, time to complete product

§ Uses controllable factors – Stories, Resources assigned

§ Models factor variation, allowing us to understand prediction and confidence intervals

§ Connects upstream development/test activities with defect density

Application to Cyber Security

§ Analysis of historical data to identify “vulnerability characteristics”

§ Define response techniques for these characteristics

§ Use of data and predictive models to identify where to expect the “soft spots”

§ Apply response techniques as part of the sprint§ Include security enablers in product backlog§ Integrate vulnerability testing into the sprint

Summary

The combination of:§ high maturity process discipline (Data collection, analytics, predictive modeling, etc.),§ agile (Development team and business working together, quick delivery of functional code, etc.),§ and DevSecOps (Continuous delivery, heavy use of automation, etc.)

… is not only possible, but enabling

Benefits:§ Higher productivity, faster product delivery

§ Lower production defect density for higher product quality§ Responsiveness to evolving/changing requirements

§ And when applied to cyber security – Reduced vulnerabilities and the ability to more quickly adjust/respond to evolving threats

Questions?