Brian HenereyDirector of Technology & Operations, OpinionLab

bhenerey@gmail.com | @bhenerey | October 2014

Controlling Devops

Me

• Working in tech since 1998• Discovered Puppet – 2006• Discovered Devops – 2010• First Devopsdays – TODAY!

Trust us, we’re Engineers!

• Did you delete those users accounts?• Do those automated backups work?• Do you know what code was released to Prod?• Who authorized those changes?• Who reviewed them?• Are people ssh-ing into Prod? Did they make

any changes?• How many people have access to customer

data?

Trust, Actually

• Trust is a core element of Devops• We want to trust and empower our people, that’s

why we hired them

• Yet, we need some controls in place…

This Talk...

• I’m going to briefly describe what a SOC2 is and how it works

• I’m going to tell how to rub some Devops on one

SOC(2) it to me

A Report on the Controls at a Service Organization Relevant to:

– Security– Availability– Confidentiality– Processing Integrity– Privacy

These are called ‘Trust Services Principles’. Defined here: http://en.wikipedia.org/wiki/Service_Organization_Controls

American Institute of CPAs (AICPA)

Why would you want one?

• To assert that your company meets all the criteria of the Trust Services Principles

• You want an external auditor to make a statement that you’ve done this

Who has SOC2 reports?

• SaaS’s• Datacenters• AWS (http://blogs.aws.amazon.com/security/blog/tag/SOC+2)

How does a SOC2 work?

You/Your Org• Write System Description• Write Controls• Write Policies based on Controls• Write Assertion about your System

Auditor• Helps you prepare the above• Performs audit for 3-12 months• Produces Auditor’s report

What does the have to do with Devops?

1. Accountants are more familiar with traditional IT practices– Devops practices require different controls

2. A SOC2 is driven by mitigating risk. – So is DevOPs

What about ITIL?

• ITIL is the baby that got thrown out with the bathwater.

• This is just my experience.

Over-optimized for speed

• There’s a ton more emphasis on speed of feature delivery than creating operable systems

• You can not wish the operational pain away

• So let’s put some controls in place at the start

Getting started with Controls

TSP Criteria Risk Example

ControlYour

Control Evidence

What is a Control?

• Your organization’s practice for mitigating specific risks

• Tip: You’re better off with multiple controls to address each Risk

What kind of Controls are there?

• AICPA introduced the ‘common criteria’ which cover:– Organization and management– Communications– Risk management and design and implementation of

controls– Monitoring of controls– Logical and physical access of controls– System Operations– Change Management

Example Control

• Risk: “Breaches and incidents recur because preventive measures are not implemented after a previous event. “

• Control: “At least monthly, the Technology Management Team meet to review the Operations Review Register and discuss plan for resolution of issues, or recap of resolved issues.”

Evidence of Control Performing Well

• “Monthly meeting minutes OR checklist of categories of issues/projects discussed at meetings, attendees, date, and indication if any projects/issues need to be communicated to other employees.”

• Tip: this is what you want automated or baked into your processes as much as possible. No one likes busy work

Is a Control a Policy?

No.

• Controls are used only by the auditor• Policies and Procedures are written for

employees with clear instructions on how to perform their jobs.

How do I write a control?

Auditor interviews me

Auditor writes

Review together

How many controls will I need?

• Around 200– Maybe 20% of these are duplicates. The means the

same control addresses multiple risks.

• I spent 50-60 hours with the auditor writing controls over 2-3 months

These are your Controls!

• Start off by writing what your company actually does

• The most valuable part of writing these controls is discovering areas to improve

What do Accountants like?

• Clarity• Hierarchy-based Approval• Consistency• Ownership• Accountability

How is Devops different?

• More collaboration• More autonomy• Less segregation of responsibilities• Broader range of impact

– Systems, code, security, networks, databases

• More bottom-up leadership

Devops Criteria for writing Controls

• Lightweight• Minimal impact to workflow• No bureaucracy• No busy work• Automate/bake-in all the evidence• Team members empowered to do their jobs

Up-front versus Back-end Controls

• Up-front control is more powerful, but more stifling– People cheat the system in favor of ‘getting things

done’

• Devops controls are on the back-end: The team is trusted to make changes. We’ll review what changed after the fact. – Need multiple controls to increase the effectiveness

Don’t fear the policy

• Okay, maybe fear it a bit.

Leverage your ticketing system

• Already has history• Already can create reports• Easy to add approval to it• Can automate ticket creation for

weekly/monthly/quarterly tasks• Easy to add screenshots or attachments as

evidence

Leverage your version control

• Name your branches/commits based on Ticket-ID

• Can review list of commits after deployment to make sure changes were approved/intended

Leverage your tools + logging

• Have your tools create log events– Who’s using the tool– Datestamp– What change was made

– i.e. Chef-handler for logstash• https://github.com/lusis/logstash_handler

Side benefit of doing this

• The more you think about your organization, you’ll probably find things you should really be doing but aren’t.

• You’ll also find you aren’t consistent in your behavior, which is why you need layers of Control.

Example: Alert fatigue

• You probably already have a policy on responding to alerts

• If you’re not following this process, perhaps it’s because you have too many alerts, or they’re un-actionable

• If it’s hard to follow your process, it’s because you have debt in your system which should be paid down

Detailed example of a Control

Common Criteria 5.6

“Logical access security measures have been implemented to protect against security, availability, or confidentiality threats from sources outside the boundaries of the system.”

CC5.6 continued

Risk

“Threats to the system are obtained through external points of connectivity”

CC5.6 Control examples 1

“For VOC system infrastructure components, logical access is controlled through each component's native security using group/role based permissions when possible. The Windows network is further protected by a 3rd party multi-factor authentication system.”

Evidence: Network diagram that includes VOC infrastructure components

CC5.6 Control example 2

“For VOC custom developed applications, logical access is controlled through role-based security with configurations stored in the applications' underlying database. For the 3rd party reporting tool that is integrated with the VOC custom developed applications, logical access is controlled through that tool's native security.”

Evidence: Diagram or documentation that identifies the software/applications that comprise the VOC system

CC5.6 Control example 3

“Firewalls are initially setup as deny all and then configured to allow access for approved services. At least quarterly, the configuration of each firewall is reviewed by the Technology Operations Team and updated as deemed necessary according to industry best practices.”

Evidence: Quarterly review of firewall configuration indicating who reviewed, date reviewed, and steps taken (e.g. firewall rules updated to…).

CC5.7 Control example 4

“Remote access to VOCF systems and infrastructure components requires single factor VPN authentication followed by a second factor authentication at the server or network level. Certain VOCF system component require two factor authentication.”

Evidence: VPN configuration screen prints

Thanks!!

Questions?

Special thanks to Mike Becker

Partner-in-charge, Risks and Controls

@ FGMK

Judge a man by his questions

rather than his answers

-Voltaire