Securing Application Deployments in CI/CD Environments

Hello!I am Binu RamakrishnanPrincipal Security Engineer at Yahoo

You can find me at:@securitysauce

Overview◎CI/CD platform overview.

◎High level objectives.

◎Threat modeling.

◎Security patterns and best practices.

CI/CD

CI/CD Deployments◎Single-tenant◎Multi-tenant

Single-tenantDedicated build environments.

Multi-tenantShared build environment.

Driven by economies of scale.

CI/CD in Nutshell

Enterprise CI/CD

High Level Security Objectives

Verifiable Chain of Trust

No Elevated Privileges for Build Jobs

Threat Modeling

CI/CD Admin Exclusive access to CI/CD

platform

Actors

Developer User/customer of the

platform.

CI/CD Platform

DeveloperLimited admin access to

platform.

Trust Boundaries.

https://www.flickr.com/photos/nasamarshall/14596371842

Trust Boundaries

Web interface◎ UI

◎ REST APIs

Application

Trust Boundaries

Isolation based on Docker containers or VMs.

Host

Trust Boundaries

Segmentation

Network

Attack SurfaceEntry Points

Exit Points

Entry Points

◎ Build web interface.

◎ Commit and build notification handlers.

◎ Source and package dependencies.

Internal Entry Points

◎ Build slave endpoint (to launch builds; SSH).

◎ Other control plane endpoints/REST APIs.

Exit Points

◎ Deployable artifacts.

◎ Build notifications - eg emails, IRC and Hipchat

messages.

◎ Build console logs.

◎ Git repo update with build status and badges.

Threats

Common Threats

◎ User account compromise & insider threats.

○User and platform admins.

◎ Network Intrusion.

○CI/CD internal and external endpoints.

Build Slave CompromiseLarge attack surface, spread across multiple networks (iphone, Android, Server app etc.)

How?◎ A network level compromise, exploiting a vulnerability in build slave.

◎ Jobs break out of build container.Impact◎ Access to production servers.

◎ Listen to the network, spoof identity and access unauthorized data.

Shared/Non-ephemeral Keys◎ Locally stored long-lived keys to access protected

services.Two forms:

◉ Shared key

◉ Per-job keys (eg. OAuth tokens)

SSH over Unrestricted ShellAllows arbitrary commands to get executed on a remote host.

Building External CodeAn attacker can take this path to get into internal networks, either by adding backdoors or exploiting known vulnerabilities with open source software.

Baseline security controls that are expected to be there in 2016

Existing Security Controls◎ Authenticated endpoints.

◎ Web application security practices - CSRF, HTTPS,

HTTPOnly cookies, XSS protection etc.

◎ Source control: granular authorization for repos.

◎ Use KMS to manage secrets; no hardcoded secrets.

Security Patterns for Risk Reduction

Ephemeral Keys

https://www.flickr.com/photos/articnomad/241620406

Stateless Auth Architecture

https://www.flickr.com/photos/18946008@N06/14551311971/

◎ Events:

○ Commit trigger.

○ Manual trigger from build UI.

○ Automated/cron job.

○ Trigger a downstream job.

◎ Upstream service stores downstream service credentials (OAuth, Shared Keys etc.).

◎ Equal trust on all components in the pipeline

Audit Trails

https://www.flickr.com/photos/adriensifre/8403355648

Container, ToolChainHardening

http://www.publicdomainfiles.com/show_file.php?id=13493588417214

Network Segmentation

https://www.flickr.com/photos/bretagne-balades/15355029654

Do not pull PII or other sensitive info to build machine.

Minimal Builds

Few more...SSH: Use Restricted ShellsHeadless SSH access for automated deployment should use a restricted shell.

Roll Keys PeriodicallyEstablish a process to periodically roll trust anchor keys (and do it periodically).

Restrict Job Console LogsRestrict build job console logs only to authorized users

Enable 2FAAdmins must follow good security hygiene and use 2FA to access platform application and hosts

Prune Admin Access ListKeep admin list small for build systems and Git repo access

Vulnerability Patch MgmtMaintain an inventory of all packages in use and have a mechanism to patch the system in response to a disclosure

Acknowledgements◎ Christopher Harrell

◎ St John Johnson

◎ Mike Shema

◎ Jeremiah Wuenschel

Let’s recap major conceptsEphemeral KeysEphemeral keys are the future. Service providers should start supporting ephemeral keys for authorization

Stateless Auth ArchitectureAugment the trust dependency of 1:1 relationship between the pipeline components with workflow job tokens.

Audit LogsA verifiable chain of trust based on traceable audit logs is a foundational requirement for CI/CD.

Minimal BuildsAvoid pulling PII or other production sensitive data to build environments. Keep the builds to minimum

Network SegmentationNetwork level Isolation of CI/CD machines from other machines

ToolChain HardeningBuild tools and Docker containers must be adequately hardened.

Thanks!Any questions?You can find me at:@securitysauce

Appendix-A

Use◎ Source code◎ Pre-built packages

Risks◎ Targeted back doors◎ Vulnerabilities

Hard Problem: Securing Supply Chain

This threat is more to do with applications than the CI/CD platform itself.

Open source components constitute a large part of modern Internet based applications.

Appendix-B

Credits

Special thanks to all the people who made and released these awesome resources for free:◎ Presentation template by SlidesCarnival◎ Photographs by Unsplash & Death to the Stock Photo (

license)