Containers and Kubernetes Put a Lid on It: Securing for ...€¦ · Containers and Kubernetes on...

#vmworld

Put a Lid on It: Securing Containers and Kubernetes

on vSphere and in CloudNolan Karpinski, VMware, Inc.Steve Hoenisch, VMware, Inc.

Wei Fu, VMware, Inc.

CNA1656BU

#CNA1656BU

VMworld 2018 Content: Not for publication or distribution

Disclaimer

2©2018 VMware, Inc.

This presentation may contain product features orfunctionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.


Agenda


The Cloud-Native Stack

Container Security Risks and Threats

Security in Pivotal Container Service on vSphere

Security in VMware Kubernetes Engine on AWS and Demo

Security with AppDefense and Demo

Summary



The Cloud-Native StackLayers and Processes Requiring Security

Applications

Infrastructure

Orchestration Platform

Container Registry

Container Runtime

Container Host

Container Images

Container Management



Container LifecycleSecurity for Developers Takes Place in this Workflow

DeployBuild Run



Risks and ThreatsAttack Vectors throughout the Stack Identified by NIST



• Just like traditional apps, containerized apps are vulnerable to typical flaws

• Vulnerability can be exploited

• Containers give you a prescribed way to apply security principles with depth and breadth

• Core protection strategies help secure cloud workloads

The Application

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




• Images with vulnerabilities, malware, embedded secrets, etc.

• Images that are outdated or unpatched or of unknown origin

• Insecure connectivity

• Inadequate access control

• Insecure configuration (running as privileged user or with SSH)

• The rogue containers of hasty developers

• Operating systems with a large attack surface, shared kernel, or package vulnerabilities

Container ManagementPortability and Reuse Heighten Risks

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




• Unsecured components, like the Kubernetes Dashboard, on the Internet

• Lack of standard directory service and RBAC

• Unencrypted data

• Apps sharing the same virtual networks

• Mixing workloads with different sensitivity levels and threat postures

Risks to the Orchestration SystemUnsecure Components, Lack of Access Control, and Mixed Workloads Pose Threats

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




• Unprotected data in transit and in storage heightens risks of a breach

• Lack of logging, monitoring, and visibility can make it hard to identify intrusions and respond quickly

Infrastructure

Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images




Security in VMwarePivotal Container ServiceContainer Images

Access Control

Micro-Segmentation

Logging and Monitoring



Architecture



Harbor: Image Registry Imposes CountermeasuresScans, Secures, and Signs Images and then Prescribes Use with Policies

DevelopmentTeam

RBAC

UAA AUTH REPLImage Pull

K8s Cluster deployed by PKS

Build Image Push Image Scan ImageFor CVEs

Sign Trusted Image

kubectl run

ImageRegistry

Image Scanning

Image Signing

HarborProjects

AUDIT LOGGING



Identity and Access ManagementControlling Access to Kubernetes Clusters

Operator admin

Developers Developers

Kubernetes

Namespace Namespace

Namespace NamespaceKubernetes

Namespace Namespace

Namespace Namespace

PKS Broker

UAA PKS API

BOSH CredHub

• Authentication and RBAC for PKS CLI and Kubernetes API

• Centralized credential generation and management with CredHub

• PKS secures Kubernetes Dashboard by default with user authentication



PKS with NSX-T Provides Strong Isolation for Kubernetes ClustersEach Cluster Isolated on its own Network Segment

Virtual Switch

K8 Worker

K8 Worker

K8 Worker

Kubernetes Cluster

T1 Router

Virtual Switch

K8 Worker

K8 Worker

K8 Worker

Kubernetes Cluster

T1 Router

T0 RouterLB LB

Cloud Provider Uplinks

• Gives tenants separate clusters

• Isolates orchestrator traffic from workload traffic

Cluster Networking



NSX-T Isolates Namespaces with Logical Switches and RoutersProvides Stronger Isolation than Default K8s Namespaces

admin@k8s-master:~$ kubectl create namespace foonamespace ”foo" created

admin@k8s-master:~$ kubectl create namespace barnamespace ”bar" created

admin@k8s-master:~$ kubectl run nginx-foo --image=nginx -n foodeployment "nginx-foo" created

admin@k8s-master:~$ kubectl run nginx-bar --image=nginx -n bardeployment "nginx-bar" created

Namespace: foo Namespace: bar

K8s Topology with NSX

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

NAT boundary

K8s nodesK8s MastersVMworld 2018 Content: Not for publication or distribution


NSX-T Can Use K8s Network Policies for Micro-SegmentationSegments Pods by Applying Dynamic Security Groups and Policies

admin@k8s-master:~$ vim nsx-demo-policy.yamlapiVersion: extensions/v1beta1kind: NetworkPolicymetadata:name: nsx-demo-policy

spec:podSelector:matchLabels:app: web

ingress:- from:

- namespaceSelector:matchLabels:ncp/project: db

ports:- port: 80protocol: TCP

admin@k8s-master:~$ kubectl create -f nsx-demo-policy.yaml

Namespace: foo Namespace: bar

NSX / K8s topology

10.24.0.0/24 10.24.1.0/24 10.24.2.0/24

NAT boundary

DBLabel: app=db

WebLabel: app=web



Pod Micro-SegmentationNSX Uses Network Policy to Dynamically Create Security Groups and Policy

$ kubectl create -f nsx-demo-policy.yaml

Dynamic Creation of Security Groups

Dynamic Creation of Security Policy based on K8s Network PolicyVMworld 2018 Content: Not for publication or distribution


Getting Visibility across the StackTraceflow with NSX-T

Pods

Pods

Containers Pods Cluster Host Datacenter

Traceflow • Monitor network traffic

• Trace packets from containers to physical networks

• Visualize traffic flows across the stack



vRealize Log Insight and vRealize Operations

• Gain comprehensive visibility across apps and infrastructure

• Analyze logs for suspicious activity

Monitoring, Logging, and AnalyticsOperational Visibility and Auditing

vRLI vROPS

Logs Metrics



Lifecycle Management Maintains SecurityBOSH Patches and Repairs Nodes

VMware GCP

AvailabilityZone

master

etcd

worker

etcd etcd

master

worker worker

master

etcd

worker

etcd etcd

master

worker worker

AvailabilityZone

AvailabilityZone

BOSH

Health Monitor

Health Monitor

worker workerPatch

K8sNewVer

Release

Repair

Deploy

ScaleUpgradePatchRepair

Day 1

Day 2

workerworker

PKS

Con

trol

Pla

ne

CVE



Operator admin

PKS Security Overview

InfrastructureStorageCompute Networking

Container Mgmt.

Namespace

vSphere Google Cloud Platform

Hybrid

Namespace

Namespace Namespace

Container Mgmt.

Namespace Namespace

Namespace

Developers

IAM

Events & Monitoring

Build Pipelines

Namespace

Platform LCM

Kubernetes Cluster Kubernetes Cluster

Apps Apps

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Micro-service

Platform LCM


Infrastructure

LDAP or AD

IAM

VNFM

vRealizeOperations

Monitoring

Logging

vRealize Log Insight



Security in VMware Kubernetes EngineOverview

Access Control Policies (Lightwave)

Linux Container Host (Photon OS)



US West

US East

Europe West

Fully managed Kubernetes service on AWS

Smart Clusters• Elastic cluster size• HA configuration• Automatic recovery from failure

Granular multi-tenant access policies

Multi-cloud ready

VMware Kubernetes Engine OverviewHighly Secure and Available Kubernetes Service



• Manage user identities centrally in VMware Cloud Services

• Access VMware Cloud Services, VKE, and all your Kubernetes clusters with your identity

Single Sign-On with Unified Identity at cloud.vmware.com

25

Part of VMware Cloud Services



Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images


Access Control Policies



Multi-Tenant Access PoliciesAccess Control Encompasses Entire Service

ENGINEERING FOLDER QUOTAPOLICY

DEV/TEST PROJECT

SMART CLUSTER

NAME SPACE

SMART CLUSTER

PRODUCTION PROJECT

SMART CLUSTER SMART CLUSTER

AI PROJECT

SMART CLUSTER

FINANCE FOLDER

EU-West-1US-West-2 US-East-1 US-West-2

SMART CLUSTER

EU-West-1

ACCESS POLICIES

ACCESS POLICIES

QUOTAPOLICY

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE

NAME SPACE



Tenants can organize clusters with folders and projects

Roles grant a configurable set of permissions, such as administering or viewing a cluster or a namespace

Access polices are sets of roles bound to users or groups

Roles are inherited down the tree

VKE pushes policies to Kubernetes and translates them into Kubernetes RBAC

Role-Based Access Control (RBAC)

Organization

Folder 1 Folder 2

Project 1 Project 2

Cluster 1 Cluster 2

Namespace 1 Namespace 2

SmartCluster.Admin: ClusterAdmin1


SmartCluster.Admin: ClusterAdmin2SmartCluster.Admin: ClusterAdmin1



Project 1

Cluster 1



Demo 1: VKEAccess Controls Policies in Action



Grant a user ”smartcluster.admin” role at folder level• The user becomes smart cluster admin on all three clusters in that folder• The user can only access the Kubernetes dashboard using her VMware cloud identity

Revoke a user’s “smartcluster.admin” role at folder level• The user cannot administer any smart clusters in that folder

Grant permission through group membership• The user becomes “smartcluster.admin” on all three clusters in that folder

Demo 1: VKE Access Policies



Lightwave in VKEDirectory Service, Certificate Authority, Secure Token Service

31

LIGHTWAVEDIRECTORY SERVICE

LW D

C1

LW D

C2 LW

DC

1

LW D

C2

ESXiHOST

ESXiHOST

ESXiHOST

ESXiHOST

ESXiHOST

C

K8SK8S

NNN

NNN

Cluster

NN

NN

Cluster

NNN

NNN

Cluster

T1T4 T2N

NN

Cluster

T3

K8SK8S

AWS US-Region

N

K8S K8S

N N N

N N

Cluster

N N

N N

Cluster

N N N

N N N

Cluster

T1 T4T2 N

N N

Cluster

T3

K8S K8S

CC

1 NSXManager

CC

2

CC

3

CC

3NSXManager

CC

2

CC

1

AWS EU-RegionLW

DC

3

LW D

C3

N

Photon OS

https://github.com/vmware/lightwaveVMworld 2018 Content: Not for publication or distribution


Minimalist: The number of packages is limited to the minimum necessary for hosting containers

Security-hardened Linux: The kernel is configured according to the recommendations of the Kernel Self-Protection Project (KSPP)

Curated packages and repositories: Packages are built with hardened security flags

Advanced lifecycle management: There are timely security patches and updates to container packages, such as Docker and Kubernetes

Project Lightwave integration: Lightwave clients are installed on Photon OS by default, which let it join to Lightwave domain and be managed by Lightwave consistently

Photon OSA Linux Container-Optimized Operating System

https://github.com/vmware/photonVMworld 2018 Content: Not for publication or distribution


Security Managed for You

Data encryption at rest

Data encryption in motion with TLS

Encrypted K8s secrets in etcd

Data Encryption

Kubernetes

Automatic secure OS updates on Kubernetes nodes

Patches and Upgrades



Each Organization is mapped to a AWS account managed by VKE

Each production Smart Cluster is deployed on a separate Network segment

Infrastructure Isolation on AWS

VMware Kubernetes Engine

Acme Globex

user@Acme user@Globex

VPC VPC VPC

AWS cloud

AWS Account(vke.tenant.acme) AWS Account(vke.tenant.globex)

Smart Cluster Smart Cluster Smart Cluster



Container Security at RuntimeAppDefense and Aqua Security



Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images


Application Layer



Opportunity to Focus on Core Protection StrategiesGartner Market Guide for Cloud Workload Protection Framework

AV

Deception

HIPS withVulnerability Shielding

Server Workload EDRBehavioral Monitoring

IaaS Data at Rest Encryption

Exploit Prevention / Memory Protection

Application Control / Whitelisting

System Integrity Monitoring / Management

Network Firewalling, Segmentation and Visibility

Hardening, Configuration and Vulnerability ManagementFoundational

Less Critical

Optional Protection Strategies

Core Protection Strategies

Important, but often provided outside of CWPP

Figure 1. Cloud Workload Protection Controls Hierarchy, © 2018 Gartner, Inc.

Source: Gartner, Market Guide for Cloud Workload Protection Platforms, Neil MacDonald, March 26th 2018. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document.VMworld 2018 Content: Not for publication or distribution


Gartner Market Guide for Cloud Workload Protection Framework

AV

Deception

HIPS withVulnerability Shielding

Server Workload EDRBehavioral Monitoring

IaaS Data at Rest Encryption

Exploit Prevention / Memory Protection

Application Control / Whitelisting

System Integrity Monitoring / Management

Network Firewalling, Segmentation and Visibility

Hardening, Configuration and Vulnerability ManagementFoundational

Less Critical

Optional Protection Strategies

Core Protection Strategies

Important, but often provided outside of CWPP

Figure 1. Cloud Workload Protection Controls Hierarchy, © 2018 Gartner, Inc.

Source: Gartner, Market Guide for Cloud Workload Protection Platforms, Neil MacDonald, March 26th 2018. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document.

Post-deployment Controls

Opportunity to Focus on Core Protection Strategies



Cyber ThreatsResidual Risk

Apps Data

Cyber HygieneAttack Surface

EncryptionMicro-Segmentation

RepavingLeastPrivilege

Multi-FactorAuthenticationVMworld 2018 Content: Not for publication or distribution


Architecture

40

vSphere

AppDefense Manager

AppDefenseAppliance

vCenter

SaaS

Guest OS

Guest Module

Host Module

Customer Data Center

Guest OS

Guest Module

Guest OS

Guest Module

1. Application Context2. Central Intelligence

ContainerContainerContainerContainerContainerContainer

ContainerContainerContainer



1001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010010101011010010101010011001010010101010101101001010101001100101001010101010110100101010100110010100101010101011010010101010011001010010101010101010010101010011001010010101010101101

Changing the Application Security ModelFrom chasing bad to ensuring good

Pro

cess

es

Pro

cess

es

Pro

cess

es

OSXX,000,000 XX

Chasing Bad Ensuring Good



Ensuring Known Good

Detect Deviations RespondCapture & Analyze

Learn Intended State Protect Intended State

ManifestApp ManifestApp



LearnCapture the purpose and intended state of a container

Intended State Engine

vCenter IaaS/PaaS

AppScope

Off-the-shelf images Custom imagesCI/CD pipeline

Machine Learning!

Learn Protect

Detect RespondCapture & Analyze

ManifestApp

ManifestApp

ManifestApp



DetectRuntime application attestation and secure manifest store

Co

ntai

ner

Co

ntai

ner

Co

ntai

ner

OS

AppDefenseMonitor

Co

ntai

ner

Co

ntai

ner

Co

ntai

ner

OS

AppDefenseMonitor

Co

ntai

ner

Co

ntai

ner

Co

ntai

ner

OS

AppDefenseMonitor

Protected zone

Learn Protect

Detect RespondCapture & Analyze

ManifestApp ManifestAppManifestApp



RespondOrchestrated incident response routines for the SOC

Secureinfrastructure

IntegratedEcosystem

Learn Protect

DetectRespond

Capture & Analyze

Add Behavior Block/Alarm

Quarantine Repave!



Integrated Solution Architecture

46

ContainerContainerContainer

Hypervisor

AppDefense Manager

Aqua Manager

Aqua Agent

SaaS

Guest OS

Container Host

Integration Container



Demo 2: AppDefensewith Aqua



Applications

Infrastructure


Container Registry

Container Runtime

Container Host

Container Images


Summary



NIST Application Container Security Guide

NIST Security Assurance Requirements for Linux Application Container Deployments

Cloud-Native Stack Security: How VMware Pivotal Container Service Secures Containers and Kubernetes

Control Access with VMware Kubernetes Engine

Containers on Virtual Machines or Bare Metal? Deploying and Securely Managing Containerized Applications at Scale

Glossary of Cloud-Native Terms

Securing Cloud Platforms with Project Lightwave

Photon OS: A Linux Container-Optimized Operating System

A Dash of Security: Locking Down Kubernetes Admin Access

References and Other ResourcesGuides and White Papers


https://doi.org/10.6028/NIST.SP.800-190

https://doi.org/10.6028/NIST.IR.8176

https://assets.cloud.vmware.com/v3/assets/blt719094f4883f620b/blt0e1847c6dbf03990/5b30443d073818f857728872/download

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vmw-wp-conatiner-on-vms-a4-final-web.pdf

https://assets.cloud.vmware.com/v3/assets/blt719094f4883f620b/bltcee5618b0aca3766/5a8da0f511ef86064e4106d9/download?disposition=inline

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/cloudnative/vmware-securing-cloud-platforms-with-project-lightwave.pdf

https://vmware.github.io/photon/assets/files/photon-os-datasheet.pdf

https://blogs.vmware.com/cloudnative/2018/07/20/a-dash-of-security-locking-down-kubernetes-admin-access-to-thwart-crytojacking/


Summary: Best PracticesImplement Container-Specific Countermeasures• Integrate countermeasures into life cycle and pipeline

• Monitor containers thru life cycle & stack for full visibility

• Enforce security with policies for RBAC and image use

• Use only the latest patched, scanned, and signed images

• Run images as non-privileged, immutable containers without SSH and manage through the orchestrator

• Securely store secrets, encrypted, in orchestrator, not image

• Connect to registries, dashboards over secure channels

• Control access to registries, orchestrators, and dashboards with RBAC using principles of least privilege and separation of duties

• Provide single sign-on with a single, standard directory

• Log, monitor, and audit access to registry and orchestrator

• Encrypt data at rest using container-specific methods; see NIST 800-111

• Segment orchestrator traffic into discrete virtual networks by sensitivity level

• Only mix workloads of the same sensitivity level and threat posture on the same host

• Use a patched, up-to-date, CVE-monitored runtime

• Constrain network access from containers

• Profile and protect apps at runtime to ensure known good

• Use an up-to-date, minimalist container OS to narrow the attack surface; see NIST SP 800-123

• Set the root file system to read-only

• Limit, log, and audit host OS access to detect anomalies and privileged operations


PLEASE FILL OUTYOUR SURVEY.Take a survey and enter a drawingfor a VMware company store gift card.

#vmworld #CNA1656BU


THANK YOU!

#vmworld #CNA1656BU


Containers and Kubernetes Put a Lid on It: Securing for ...€¦ · Containers and Kubernetes on...

Documents

Transcript of Containers and Kubernetes Put a Lid on It: Securing for ...€¦ · Containers and Kubernetes on...