CNA1142BE Developer-Ready Infrastructure from or …...Pivotal Cloud Foundry 101 6 war PCF Elastic...

Merlin Glynn (Vmware)Ramiro Salas (Pivotal)

CNA1142BE

#VMworld #CNA1142BE

Developer-Ready Infrastructure from VMware and Pivotal

VMworld 2017 Content: Not fo

r publication or distri

bution

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

2



bution

Agenda

1 Pivotal Cloud Foundry 101 Why do my Developers want it?

2 Site Reliability Engineering (SRE) The role of the Platform Operator

3 Ops: PCF & vSphere

4 Ops: PCF Network & Security

5 Ops: PCF Monitoring & Logging

6 Ops: PKS & Developer Ready Infrastructure

3



bution

Pivotal Cloud Foundry 101Why do my Developers want It?



bution

What Pivotal Cloud Foundry solves for …

5

Ag

ilit

y &

Co

st

Savin

gs

Cloud Native Maturity

NetworkingStorage

ServersVirtualization

O/SMiddleware

RuntimeDataApps

Apps T

eam

Manage A

ll

Old School

Apps M

anages

NetworkingStorage


O/S

MiddlewareRuntime

DataApps

IaaS + Containers

Ops

Apps M

anages

NetworkingStorage


O/S

MiddlewareRuntime

DataApps

Cloud Native Platform

Ops

PC

F

Pla

tform

_O

psVMworld 2017 Content: N

ot for publicatio

n or distribution

Pivotal Cloud Foundry 101

6

war

PCF Elastic Runtime

Availability Zone 1 Availability Zone 2 Availability Zone 3

Staging

Root

FS

Build

Pack

war

`cf push`

Drop

let

A

I

A

Imyapp.foo.com

PCF Routing PCF Routing PCF Routing

“Here is my source code

Run it on the cloud for me

I do not care how”

Developer

URL Request:

myapp.foo.com



bution

PCF Elastic Runtime


7

Availability Zone 1

myapp.foo.com

PCF Diego

VM=Cell

VM=Cell

PCF Routing

VM=

GoRouter

A

I

A

I

• Running Containers are called ‘Application Instances’ (AIs)

• PCF Diego schedules & maintains the health of containers on special VMs called Diego Cells.

• GoRouters are another type of VM that balances & forwards requests to the correct Cell AIs

How do the Containers run?URL Request:

myapp.foo.com



bution


8

PCF Elastic Runtime


A

I

A

I


Developer

• Self Service & Automated Scaling

• Diego Dynamically keeps apps Healthy in case of IaaS faults

• No Intervention from the Developer and no Tickets ☺

• PCF also provides auto recovery when IaaS is repaired

CF API

`cf scale myapp –i 3`

A

Imyapp.foo.comA

I

Cluster Cluster ClustervSphere



bution


9

PCF Elastic Runtime


A

I

A

Imyapp.foo.com

• PCF Service Broker

– Self Service to app services

– PCF Managed & External

• Binds creds to app (12 Factor)

Developer PCF Mysql PCF RabbitMQ PCF Spring Svcs

DATA MSG BUSNetflix

OSS

User Provided

CF API

`cf create-service mysql`

jdbc url & creds

vcap env {}

`cf bind-service mysql`



bution

BOSH


10

PCF Elastic Runtime


A

I

A

I


• Pivotal Cloud Foundry is a distributed system made up of many VMs

• Automated deployment and health management is provided by Pivotal Operations Manager & BOSH

• Enables API driven & repeatable deployment

CF API

`bosh deploy mypcf`

A

Imyapp.foo.comA

I

Cluster Cluster ClustervSphere

Platform

Operator

Developer

Ops Manager

(OVA)



bution

Site Reliability Engineering (SRE)The role of the Platform Operator



bution

Key Developer Ready Infrastructure Personas

12

Platform

Operator

Developer

IT

Operator

– SRE (Site Reliability Engineering)

– Deploy, Scale, Operate Platform

– Innovation of Business Capability as Cloud native Apps

– Develop, Deploy, Scale, Monitor Apps

– Physical Infrastructure is Operated

– Network & Security Control Policy is defined

Cloud-Native App Services

Automation

Monitoring, Security & Logging

Cloud-NativePlatform

PhysicalInfrastructure

App Lifecycle



bution

Key Developer Ready Infrastructure Personas

13

Developer

IT

Operator



– Innovation of Business Capability as Cloud native Apps

– Develop, Deploy, Scale, Monitor Apps

– Physical Infrastructure is Operated

– Network & Security Control Policy is defined

• Platform Operators

– Site Reliability Engineers (SRE)’s

• Role Shift

– In most case the VI Admins (IT Ops), are becoming the Platform Operators

• Cloud Native Applications at scale can & should be kept running by a 2 Pizza Team mentality (DevOps in Action)

Platform

Operator



bution

14

Site Reliability Engineers

Platform

Operator

• Platform is Reliable

• Capacity Is planned for

• Platform is Secured & Controlled

• Platform is Auditable

• Developers are Agile

• Platform as Code{}



What do they Do?

% of Time Spent

Traditional Ops Coding Ops

They combine the knowledge &

skills of the IT Ops guys with the

mandate of providing ops

processes as code {}



bution

Ops: PCF & vSphere



bution

vSphere Fundamentals for PCF

16

BOSH

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2

go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr

ESX

Cluster

Developer

PCF OrgPCF Space

App App

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

ESX

Cluster

ESX

Cluster

• PCF is many services deployed as many VMs, called instances

• PCF distributes instances across availability zones (AZ)

• vSphere Clusters & Resource Pools map to PCF AZs



bution


17

BOSH

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)



ESX

Cluster

Developer

PCF OrgPCF Space

App App

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

ESX

Cluster

ESX

Cluster

• VMotion Helps keep load balanced within clusters

• vSphere HA helps recover instances



bution


18

BOSH

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)



ESX

Cluster

Developer

PCF OrgPCF Space

App App

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

ESX

Cluster

ESX

Cluster

• vSAN, VMFS, NAS Datastores are supported

• Storage SVMotion is NOT recommended

Data

store

Data

store

Data

store



bution


19

BOSH

AZ1

Platform

Operator

Ops Manager

(OVA)

Cell_0 Cell_1 Cell_2

go_rtr go_rtr

Developer

PCF OrgPCF Space

App App

OTHER

INSTANCES

OTHER

INSTANCES

OTHER

INSTANCES

ESX

Cluster

• Single PCF Availability Zone deployments are supported

• Limited Platform availability

• Usually smaller scale deployments



bution


20

BOSH

AZ1 AZ2 AZ3

Platform

Operator

Ops Manager

(OVA)

ESX

Cluster

ESX

Cluster

ESX

Cluster

• Most instances do not deploy dynamically

• Platform Operator needs to plan for scale based on Application Instances (AIs)

• https://pcfsizer.cfapps.io

100 AIs



bution

Ops: PCF Network & Security



bution

Network Fundamentals for PCF

22

VTEP VLAN VTEP VLAN

ESX

Cluster

ESX

Cluster

ESX

Cluster

VTEP VLAN

• NSX Logical Switches provide a single Layer 2 over Layer 3 Network.

• Operator can have a VLAN per Physical Fault Domain (Cluster)

• Logical Switch presents as a simple single network to PCF

NSX Logical Switch 192.168.10.0/24

NSX Logical Switch Primer

172.16.101.1/24 172.16.102.1/24 172.16.103.1/24



bution


23

ERT

AZ1 AZ2 AZ3

SVC1 Infra

• NSX Logical Switches give PCF a common Layer 2 Network across AZs

• Recommend 1 Switch per deployment, sometimes called a PCF ‘tile’

VTEP VLAN VTEP VLAN

ESX

Cluster

ESX

Cluster

ESX

Cluster

VTEP VLAN

Elastic RunTime Deployment

MySQL DeploymentBOSHOperations

Manager

192.168.0.0/28 192.168.8.0/22 192.168.12.0/24



bution


24

ERT

AZ1 AZ2 AZ3

SVC1 Infra

• NSX Distributed Logical Router (DLR) Allows for hundreds of connected Logical Switches

– NSX Edge has a 10 Interface limit

– Optimized East & West traffic

VTEP VLAN VTEP VLAN

ESX

Cluster

ESX

Cluster

ESX

Cluster

VTEP VLAN



Manager

192.168.0.0/28 192.168.8.0/22 192.168.12.0/24

NSX Distributed Logical Router



bution


25

ERT

AZ1 AZ2 AZ3

SVC1 Infra

• PCF requires a production grade external load balancer

• NSX Edge provides:

– SNAT & DNAT

– SSL Term & Acceleration

– HTTP & TCP Load Balancing

• RCF 1918 on the Logical Switches = Repeatability

VTEP VLAN VTEP VLAN

ESX

Cluster

ESX

Cluster

ESX

Cluster

VTEP VLAN



Manager

192.168.0.0/28 192.168.8.0/22 192.168.12.0/24


Routing Logical Switch

NSX EdgePUBLIC VIPS

INTERNAL NAT



bution

26

ERT

AZ1 AZ2 AZ3

SVC1 Infra

VTEP VLAN VTEP VLAN

ESX

Cluster

ESX

Cluster

ESX

Cluster

VTEP VLAN


MySQL DeploymentBOSH

NSX

Distributed Firewall

(DFW)192.168.0.0/28 192.168.8.0/22 192.168.12.0/24


Routing Logical Switch

NSX EdgePUBLIC VIPS

INTERNAL NAT

• Use DFW

– Single Policy Engine across all Logical Switches

– Use for Internal East/West and Egress Control

Network Security & Controls

• Use Edge for Perimeter Ingress Control



bution


PCF OrgPCF Space

AppA AppB AppC

cf create-security-group SECURITY-GROUP PATH-TO-RULES-FILE cf create-security-group dev-mssql mssql.json

PCF Application Security Groups (ASG):

– Uses iptables in the Diego Cell Server

– Controls Egress only at the container source level

– Can control any IP address as the target

• Operator Declares in the Platform

[ {

"protocol": "tcp",

"destination": "10.0.11.0/24",

"ports": "1-65535"

},

{

"protocol": "udp",

"destination": "10.0.11.0/24",

"ports": "1-65535"

} ]

Platform

Operator

Prod Mssql

192.168.11.10

Prod Mssql

10.0.11.10



bution


PCF OrgPCF Space

AppA AppB AppC

cf allow-access SOURCE-APP DESTINATION-APP --protocol PROTOCOL --port PORT• cf allow-access “AppA” “Appc” --protocol TCP --port 443

Developer

PCF Container to Container Networking:

– Creates and Overlay (VXLAN)

– Controls ingress & egress between Ais(containers)

– Uses CNI

• NSX-T in development

• “batteries included” protocol today

– Developer can Declare in CI/CD



bution

Ops: PCF Monitoring & Logging



bution

Monitoring & Logging

30

METRICS

LOGS

Metrics & Logs will emit from many sources:

• PCF Platform

• PCF Applications

• vSphere

• NSX

• Physical & Logical

Platform Operator MUST leverage ALL of them



bution


31

Developer

Virtual Data Center

– I need to keep my apps healthy

– I need self service to my Apps Log’s

– I need to instrument my Apps (APM)

Platform

Operator

– I need to keep the Platform healthy

– I need to plan capacity

– I need to watch & Alert on KPIs

– I need to audit & Report

vRops

vRealize Operations (vRops) KPI Visualization & Alerting for:

PCF vSphere NSX



bution


32

Developer

Virtual Data Center




Platform

Operator





vRLI

vRops

vRNI

vRealize Log Insight (vRLI) Log Aggregation & Alerting for:

PCF vSphere NSX

vRealize Network Insight (vRLI) Network & Security Reporting for:

Physical & Logical Networks



bution


33

Developer

Virtual Data Center




Platform

Operator





vRLI

vRops

vRNI

vRealize Log Insight (vRLI) Long term log Aggregation & Alerting for:

Applications Running in PCF



bution


34

Developer

Virtual Data Center




Platform

Operator





vRLI

vRops

Wavefront

vRNI

Wavefront by VMware, Application Performance Monitoring (APM) for:

Applications Running in PCF



bution

Wrapping It up w/ PKSand DRI …



bution

Kubernetes 101

36

K8s Cluster

Worker

`kubectl apply –f myapp.yml`

Developer

Worker

kube-proxyMaster

etcd

kube-proxy

Service: nodeport

POD POD

Load Balancer

URL Request:

myapp.foo.com/k8siscool

Docker

Registry



bution

K8s Cluster

WorkerWorker

Kubo

37CONFIDENTIAL

WorkerWorker

kube-proxyMaster

etcd

kube-proxy

service

POD POD

Load Balancer

Platform

Operator

BOSH

Day 1

Day 2

Master

etcdetcd

What is KUBO?

• Kubernetes Powered By BOSH

• Can Be deployed independent of PCF

• Can Deploy & Manage Multiple K8S Clusters



bution

38

VMware PKS

Kubernetes on BOSH (Kubo)

BOSH

NSX

Analytics Automation

SecurityOperations

Monitoring

GCP

Service Brokermasteretcd worker

Logging

vSANvSphere

masteretcd workerContainer

Registry

(PKS)



bution

PCF + PKS

Developer

BOSH

PCF + PKS

• Integrated PCF Service Broker “I need a K8S cluster”

`cf create service mykubo`

Why Kubernetes + PCF…

– App Packaging Need Only

– Need Specific hardware and image stack (example GPU)

– Complex multi VM Data services with perisistent diskj … like ELK … not a good fit for PCF Elastic Runtime staging



bution

PCF + PKS

Developer

BOSH

PCF + PKS

• Integrated PCF Service Broker

• Integrated PCF Routing

“I need to route to my K8S Service”

`http://myk8sapp.io`



bution

Developer Ready Infrastructure

Compute Network Storage

Application

Services

Container

ServicesA

uto

ma

tio

n

Platform

Operator

Developer

App Monitoring

• Automation

• Day 2 Operations

• Control

• Application Services or Container Services

• Application Logging & Monitoring

Solves for DevOps Reqs …

App Logging

Platform

Monitoring

Platform

Logging

Self Service



bution


vSphere NSX vSAN

Pivotal Cloud Foundry

PCF

PKS

BOSH powered KubernetesB

OS

H

Platform

Operator

Developer

Wavefront

Self Service

• Automation

• Day 2 Operations

• Control

• Application Services or Container Services

• Application Logging & Monitoring

Solves for DevOps Reqs …

vRLI (Dev)

vRops

vRLI (Ops)

vRNI



bution

43

VMworld US Key Focus Description

CNA1509BU DRI Developer-Ready Infrastructure from VMware & Pivotal

CNA1612BU PCF & PKSUse Cases: Deploying real-world workloads on Kubernetes and Pivotal Cloud

Foundry

CNA2006BU DRIDeep Dive: Architecting Container Services with VMware and Pivotal


CNA2080BU PKS Deep Dive: How to Deploy and Operationalize Kubernetes

CNA3429BU PKSBasics of Kubernetes on BOSH: Run Production-grade Kubernetes on the

SDDC

CNA3430BU PCFYour Enterprise Cloud-Native App Platform: An Introduction to Pivotal Cloud

Foundry

MGT2871BUPCF & vRops,

vRLI

Bridging the Operations Gap Between the Software-Defined Data Center

and Pivotal CF for VMware Deployments

NET1523BU PCF & NSX Integrating NSX and Cloud Foundry

PAR4411PU DRIEmerging Technologies with VMware and Pivotal - presented jointly by

VMware, Pivotal and Special Guest Speakers from Cognizant and WWT

Developer Ready Infrastructure @ VMworld



bution



bution

Thank You &Any Questions …



bution



bution

CNA1142BE Developer-Ready Infrastructure from or …...Pivotal Cloud Foundry 101 6 war PCF Elastic...

Documents

Transcript of CNA1142BE Developer-Ready Infrastructure from or …...Pivotal Cloud Foundry 101 6 war PCF Elastic...