@RealGeneKim@jdeluccia

Session ID:Gene KimJames DeLuccia

Keeping The Auditor Away:

DevOps Audit Compliance Case Studies

@RealGeneKim@jdeluccia

OMG. Developers Deploying Code?!?

@RealGeneKim@jdeluccia

Introductions

Gene Kim▪ Co-author of "The Phoenix Project”▪ Founder and CTO of Tripwire, Inc. for

13 years▪ Worked with Jez Humble (co-author

of “Continuous Delivery book) to benchmark 14K technology organizations

▪ Co-chaired SOX-404 Scoping Committee at the Institute of Internal Auditors (2005)

James DeLuccia▪ Author, “IT Compliance & Controls”▪ Ernst & Young, leader for Americas

Certification & Compliance Services▪ Focus: startups, technology,

governance, security▪ Patent holder - crypto privacy

comparison system

@RealGeneKim@jdeluccia

Golly, Why Are You Attending This Talk?

▪ How many people have to deal with compliance?

▪ On a scale of 1-10, how painful are your interactions with auditors? (1=delightful, 10=awful beyond words?)

@RealGeneKim@jdeluccia

Problem StatementGene ● DevOps and continuous delivery introduce problems with audit,

because the work patterns are so different than traditional SDLC● Agile also had issues (e.g., testing at end of project, requirements

phase at the beginning), but is not as radical as DevOps○ tens/hundreds of deploys/day (change is risk; can’t rely on

change approvals, separation of duty)

● No widespread agreement on what DevOps control requirements should look like

James ● Auditors must work off a mature and testable environment● They must stake their livelihood that what you say is correct,

completely● A partnership is needed between you and them to ensure such an

environment exists (of course, it also needs to operate and be amazing .. but that is another talk)

@RealGeneKim@jdeluccia

Agenda

▪ The Top-Down, Risk Based Audit Process

▪ What Goes Wrong

▪Scoping

▪Control Testing

▪ Scenarios From The DevOps Audit Defense Toolkit

▪ Ask An Auditor Anything!

@RealGeneKim@jdeluccia

The DevOps Audit Defense Toolkithttp://bit.ly/DevOpsAudit

James DeLuccia IVJeff Gallimore

Gene KimByron Miller

@RealGeneKim@jdeluccia

What Is Audit

▪ Management is defined as those who are there to achieve the goals of the organizations, which includes the officers of the company (e.g., CEO, CFO, etc.), executives and managers, as well as everyone who reports to them.

▪ Includes some board of directors, GRC departments

▪ Audit is defined to be the function inside the organization that resides outside of management to serve as an independent, objective source of assurance that the organization can achieve its goals.

▪ Includes internal auditors, external auditors (regulators, assessors, etc.)

@RealGeneKim@jdeluccia

Internal Controls“a process, effected by an organization’s board of directors, management, and other personnel, designed to provide reasonable assurance regarding the achievement of objectives related to operations, reporting, and compliance.”

- Operations (effectiveness, efficiency)

- Financial Reporting (accuracy of account balances and values)

- Compliance (with relevant laws and regulations, contractual obligations: PCI DSS, US Export Law, FEDRAMP, SOC-2)

Source: http://coso.org (Committee of Sponsoring Organizations for the Commission on Fraudulent Financial Reporting)

@RealGeneKim@jdeluccia

How Audit Plans Are Built And Run

▪ Business objectives

▪ Risks

▪ Control objectives

▪ Control procedures

Unfortunately, most contact with auditors start with control procedures…It’s totally appropriate to ask to show work and start from beginning...

@RealGeneKim@jdeluccia

The Audit Cycle

▪ Planning▪ Gaining an understanding of the organization▪ Scoping▪ Sampling, reporting period, types of evidence needed, recipient of

report▪ Schedule

▪ Fieldwork▪ Controls testing▪ Substantive testing

▪ Reporting▪ Management responses▪ Attestation by auditor and delivered to regulator/clients

@RealGeneKim@jdeluccia

When Scoping Goes Wrong

@RealGeneKim@jdeluccia

When Scoping Goes Wrong

▪ 2001: Enron fails ($63B market

cap), Arthur Andersen dissolution

▪ 2002: WorldCom (peak $117B

market cap)

▪ Leads to Sarbanes-Oxley Act of

2002

@RealGeneKim@jdeluccia

When Scoping Goes Wrong

Source: KPMG

@RealGeneKim@jdeluccia

Problem: Bottom Up Auditing

Source: ISACA

@RealGeneKim@jdeluccia

Analysis: Audit control testing work was scoped properly,

linking controls to compliance objectives and risk.

Control failures must result potentially undetected

material financial reporting errors

The Problem: Improperly Scoped Audits

@RealGeneKim@jdeluccia

Financial Reporting Material Weakness

What happens when an audit generates a material weakness?

@RealGeneKim@jdeluccia

Under-Scoping Operating Risk

@RealGeneKim@jdeluccia

▪ When we don’t understand why we are being audited

▪ “Why are we doing this audit?” (customers, SOX, regulatory; who is it

for?)

▪ When we are asked for something we don’t have (e.g., “evidence of SoD or

change approvals)

▪ “What is the control objective? Can we rewrite the control procedure

for this asset?”

▪ Do this before the auditor shows up

When Auditors Attack Unexpectedly

These are delicate conversations, with potentially large impacts on scope, cost, risk...

@RealGeneKim@jdeluccia

▪ If we are reacting to these conversations before we’ve done any of our

homework, we may be trouble

▪ Extra work (average time to respond to audit is 40 hours; that’s one

Dev sprint)

▪ Audit cost and schedule overages: a 3 hour audit test just turned into a

16 hour audit project

▪ Reduced confidence from auditors, increased visibility from audit and

management

When Auditors Attack Unexpectedly

The DevOps Audit Defense Toolkit

@RealGeneKim@jdeluccia

The DevOps Audit Defense Toolkithttp://bit.ly/DevOpsAudit

James DeLuccia IVJeff Gallimore

Gene KimByron Miller

@RealGeneKim@jdeluccia

Practice: Enabling A Shared Understanding

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

Practice: Enabling A Shared Understanding

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

Practice: Enabling A Shared Understanding

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

Walk Through Of DevOps Risk And Control Strategies

What does an effective DevOps

control environment look like?

@RealGeneKim@jdeluccia

DevOps Orgs Actually Love Process

“Facebook values people, tools, and way, way

down the list is process.”

Jay ParikhVP Infrastructure Engineering, Facebook

Not true! They are conflating “process” and “approvals!”

@RealGeneKim@jdeluccia

High Performing DevOps Orgs

Source: 2014 Puppet Labs State Of DevOps

30xmore frequent deployments

8,000xfaster lead times than their peers

@RealGeneKim@jdeluccia

High Performing DevOps Orgs

Source: 2014 Puppet Labs State Of DevOps

2xhigher change success rates

12xfaster mean time to recover (MTTR)

@RealGeneKim@jdeluccia

High Performing DevOps Orgs

Source: 2014 Puppet Labs State Of DevOps

more likely to exceed profitability, market share & productivity goals

2xhigher market capitalization growth over 3 years*

50%

@RealGeneKim@jdeluccia

Top Predictors Of Performance

▪ Version control of all production artifacts

▪ Continuous integration and deployment

▪ Automated acceptance testing

▪ Peer-review of production changes (vs. external change approval)

▪ High trust culture

▪ Proactive monitoring of the production environment

▪ Win-win relationship between Dev and Ops

@RealGeneKim@jdeluccia

DevOps Orgs Need Hardcopy

DevOps has higher automation and closer monitoring controls than

traditional deployment environments and therefore reduced points

for human failure

The documentation of ephemeral systems, tools, and deployment

processes into a hardcopy breakdown will communicate and

simplify this management long term.

@RealGeneKim@jdeluccia

Practice: Document Risks & ControlStrategy

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

Practice: Document Control Strategy

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

Practice: Document Control Strategy

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

Practice: Document Control Strategy

Source: DevOps Audit Defense Toolkit http://bit.ly/DevOpsAudit

@RealGeneKim@jdeluccia

▪ Gained an understanding of the organization and its

objectives

▪ Understood how our service fits in and where we jeopardize

those objectives

▪ Designed and documented our control environment so that

auditors can share our understanding

▪ Enable auditors to do their work effectively

What We Have Done

@RealGeneKim@jdeluccia

▪ Save the date: October 21-23, 2014

▪ DevOps Enterprise is a conference for horses, by horses

▪ Macy’s, Disney, GE Capital, Blackboard, Telstra, US Citizen and Immigration Services, CSG,

Raytheon, Ticketmaster/LiveNation, Capital One, Nordstrom, Union Bank of California

▪ Leaders driving DevOps transformations will talk about

▪ The business problem they set out to solve

▪ The obstacles they had to overcome

▪ The business value they created

▪ Submit talks at: http://devopsenterprisesummit.com/

DevOps Enterprise Summit

@RealGeneKim@jdeluccia

▪ We don’t need to wait for auditors to learn about DevOps -- by learning about audit,

we can successfully bridge the gap

▪ DevOps control environments can be even more secure than traditional control

environments

▪ The DevOps Audit Defense Toolkit might be able to help you! http://bit.ly/DevOpsAudit

▪ We’d love your scrutiny and case studies!

▪ DevOps Enterprise Summit: http://devopsenterprise.io

▪ Emailing us: genek@realgenekim.me, jdeluccia@gmail.com

Conclusion

@RealGeneKim@jdeluccia

Ask An Auditor Anything!

▪ Ask the Auditor and the audience anything:

▪ Separation of Duties?

▪ Security beyond checkboxes and non-contextual requirements?

▪ Governance effects of DevOps and/or Agile?

▪ Integration and dialogues and timing with Management, Auditors, and the effect?

▪ Ask Gene on practical examples

▪ Questions for the audience:

▪ Are you using ISO 27034 as a reference architecture?

@RealGeneKim@jdeluccia

Results Of Halving Deployment Interval

@RealGeneKim@jdeluccia

Results Of Halving Deployment Interval

And customers got the feature in half the time!

Source: Scott Prugh, CSG

@RealGeneKim@jdeluccia

Results Of Halving Deployment Interval

Source: Scott Prugh, CSG

@RealGeneKim@jdeluccia

Call to Action

● We're looking for case studies○ Rough life lessons and smooth successes○ Submit to:

■ DevOps Audit Defense Toolkit: Google+ Community: http://bit.ly/DevOpsAudit

● Look at the DevOps Audit Defense Toolkit● DevOps Enterprise Summit

○ http://devopsenterprise.io/