DevOps DevSecOpsBuilding Security in to your DevOps Pipeline

Eric Johnson | Эрик Джонсон

• AWS Senior Developer Advocate - Serverless

• AWS Certified• Solutions Architect

• SysOps Administrator

• Developer

• Father of Five

• Musician | Drummer

• Avid Diet Dr. Pepper Drinker

• Lover of Pizza

@edjgeek

DefinitionsGetting Everyone on the Same Page

DevOps

Development Operations

Refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

De

vO

ps

DevSecOps

Development Operations

De

vO

ps

Wrapping the DevOps process with security best practices

DevOps ExamplesThings We See In The Wild

Deploying to Instances

• Deploying code to EC2 Instances

• Deploying to on premise instances

Deploying to Containers

• Amazon ECS

• Amazon EKS

Deploying to Serverless

• API Gateway

• Lambda

• DynamoDB

• SQS

• SNS

The ”DevOps” Part

• CodePipeline

• CodeBuild

• CodeCommit

• CloudFormation

• ECR

• OpsWorks

The Security RealityNot “If” but “When”

Everything fails all of the timeWerner Vogels – VP & CTO, AWS

“What If” Guy

Solid Security is Like Onions

It Has Layers

DevOps Layers of SecurityBuilding the Walls

The Account

Beta Staging Production

AWS

CloudFormation

AWS

CodeCommitAWS

CodeBuild

AWS

CodeDeploy

The Account

Beta Staging Production

The Account

Beta Staging Production

Master

The User

Root User

• Supports MFA

• Dashboard Access

• Programmatic Access

• Full Account Access

• CANNOT be locked out

IAM User

• Supports MFA

• Dashboard Access

• Programmatic Access

• Fine Grained Access

• CAN be locked out

The Root User

• Use a recoverable email address

• Remove programmable access

• Enable MFA

• Store in a VERY secure place

• Do not use

• LOCK IT UP!!

Best Practices

The IAM User

• Enable MFA

• Grant Least Privileges

• Disable Unused Users

• Employ user Groups

• Employ Managed Policies

• Use Groups to Assign Permissions

• Configure A Strong Password Policy

• Rotate Credentials Often

Best Practices

The Role & Policies

Roles

• An IAM entity that defines a set of permissions

• Can contain one or more policies

• Is assumable by trusted entities such as IAM Users or AWS Services

Policies

• Policies define permissions for an action

• Multiple types of policies• Identity

• Resource

• Access Control List (ACL)

The Role & Policies

• Break up policies by resource

• Use AWS Defined Policies When Possible

• Grant Least Privilege

• Use Policy Conditions for Extra Security

Best Practices

Role

Policy

Policy

SecretsKeeping Secrets… Secret

Common Patterns for Secrets

• Stored in code repo

• Stored on server

• Stored on developers machines

• Shared with other developers

• Maximum “blast radius”

• Available at rest and in transit

Hard Coded Values

Common Patterns for Secrets

• Where to store?

• Management nightmare

• Smaller blast radius

• Becomes problematic in managed services

• Passing in as parameters

• Exposed on dashboard

Environmental Variables

Common Patterns for Secrets

• Where to store?

• Smaller blast radius

• Hand rolled fetch and implementation

• Can add latency depending on location

External Central Config Files

Best Practices for Secrets

• Rotates Secrets Safely on Supported Services

• RDS

• Other Select Databases*

• Secure and Audit Secrets Centrally

• Manage Access with Fine-Grained Permissions

• Provides Code Examples!!

• Can store non-rotated strings as well

• Encryption and Decryption managed for you

• Minimal blast radius

• Accessible through CloudFormation*

AWS Secrets Manager

Best Practices for Secrets

1. User inserts secrets to AWS Secret Manager

2. AWS Secret Manager creates an AWS Lambda to manage rotation

3. Lambda rotates password in AWS Secrets Manager and the Amazon RDS Instance

4. User adds code to get secrets

5. Code calls AWS Secrets Manager to get secret DB User and Password

6. Code uses obtained credentials to query RDS database.

AWS Secrets Manager

AWS

Secrets Manager

AWS

Lambda

Amazon

RDS

Amazon

EC2

1

2

3

3

4 5

6

Best Practices for Secrets

1. User inserts secrets to Amazon EC2 Systems Manager Parameter Store

2. User adds code to get secrets

3. Code calls AWS Secrets Manager to get secret DB User and Password

4. Code uses obtained credentials to query RDS database.

Amazon SSM Parameter Store

Amazon

RDS

Amazon

EC2

1

2 3

4

Parameter

Store

Best Practices for Secrets

• Allow users to connect to RDS databases with an IAM Role instead of database credentials

• Apply IAM Roles to EC2 instances to allow permission to other services

• Allow authenticated users to assume roles using Cognito Federated Identities

• Provide fine-grained, row level access to DynamoDB tables through assumed roles

Don’t Use Them – Use IAM Access When Possible

AutomationAutomate all the things

“The first rule of any technology used in a

business is that automation applied to an efficient

operation will magnify the efficiency. The second

is that automation applied to an inefficient

operation will magnify the inefficiency.”

- Bill Gates

Why Automation

• Reproducible Processes

• Predictable Outcomes

• Minimize the Blast Radius

• Removing Human Error

• Reduce Wasted Man Hours

Humans = Security Risk

Not Automated

GIT

Bob the Builder

developers

Tammy the Tester

Debbie the Deployer

Automating Integration & DeploymentContinuous Integration and Continuous Delivery

AWS

CodeCommit

GitHub

Bitbucket

AWS

CodePipeline

AWS

CodeBuild

AWS

CodeDeploy

AWS

CloudFormation

Amazon ECR Amazon

ECS,

EKS, Fargate

EC2

Instance

Automating Configuration

• AWS CloudFormation

• AWS Systems Manager

• AWS OpsWorks

• AWS Service Catalog

Building and Maintaining Consistency

Automating Monitoring

• Amazon CloudWatch

• AWS CloudTrail

• AWS Config

Watching & Reacting

Ask MeI will be answering questions in the Ask Me Section

Thank You!Благодарю вас!

Eric Johnson @edjgeek