Advanced Architecture OpenShift Container Platform (OCP)

CONFIDENTIAL Designator

OpenShift 4.x Architecture Workshop

OpenShift Container Platform (OCP) Advanced Architecture

July 2019

2

Self-Service

Multi-language

Automation

Collaboration

Multi-tenant

Standards-based

Web-scale

Open Source

Enterprise Grade

Secure

3

ANYCONTAINER

Amazon Web Services Microsoft Azure Google CloudOpenStackDatacenterLaptop

ANYINFRASTRUCTURE

APPLICATION LIFECYCLE MANAGEMENT

ENTERPRISE CONTAINER HOST

CONTAINER ORCHESTRATION AND MANAGEMENT(KUBERNETES)

OPENSHIFT CONTAINER PLATFORM

4

OPENSHIFT CONTAINER PLATFORM

Automated Operations

Kubernetes

Red Hat Enterprise Linux or Red Hat CoreOS

Application Services

CaaS PaaSBest IT Ops Experience Best Developer Experience

Cluster Services

Developer Services

Middleware, Service Mesh, Functions, ISV Metrics, Chargeback, Registry, Logging Dev Tools, Automated Builds, CI/CD, IDE

5

OPENSHIFT ARCHITECTURE

EXISTING AUTOMATION

TOOLSETS

SCM(GIT)

CI/CD

SERVICE LAYER

ROUTING LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

C C

C

C

C CC C

RED HATENTERPRISE LINUX

MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

PHYSICAL VIRTUAL PRIVATE PUBLIC HYBRID


Cotainer Concepts Overview

7

A container is the smallest compute unit

CONTAINER

8

containers are created from container images

CONTAINERCONTAINERIMAGE

BINARY RUNTIME

9

IMAGE REGISTRY

container images are stored in an image registry

CONTAINER

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

CONTAINERIMAGE

10

an image repository contains all versions of an image in the image registry

IMAGE REGISTRY

frontend:latestfrontend:2.0frontend:1.1frontend:1.0

CONTAINERIMAGE

mongo:latestmongo:3.7mongo:3.6mongo:3.4

CONTAINERIMAGE

myregistry/frontend myregistry/mongo

11

PODPOD

containers are wrapped in pods which are units of deployment and management

CONTAINER CONTAINERCONTAINER

IP: 10.1.0.11 IP: 10.1.0.55

12

pods configuration is defined in a deployment

image namereplicaslabelscpumemorystorage

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

DEPLOYMENT


OpenShift Architecture

14

YOUR CHOICE OF INFRASTRUCTURE


NODES RHEL INSTANCES WHERE APPS RUN

15

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE


RHEL

NODE

c

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

C C

C

C

C CC C

APPS RUN IN CONTAINERS

16

Container Image

Container

Pod

17

PODS ARE THE UNIT OF ORCHESTRATION

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

C C

C

C

C CC C

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

18

MASTERS ARE THE CONTROL PLANE


MASTER


RHEL

NODE

RHEL

NODE

RHEL

NODE

19

API AND AUTHENTICATION

RHEL

NODE

RHEL

NODE

RHEL

NODE


MASTER

API/AUTHENTICATION


RHEL

NODE

RHEL

NODE

RHEL

NODE

20

DESIRED AND CURRENT STATE

RHEL

NODE

RHEL

NODE

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE


21

INTEGRATED CONTAINER REGISTRY

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE

NODE

REGISTRY

RHEL

22

ORCHESTRATION AND SCHEDULING

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE

RHEL

NODE

RHEL

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

NODE

REGISTRY

RHEL

23

PLACEMENT BY POLICY

RHEL

NODE

RHEL

NODE

RHEL

NODE



MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULERREGISTRY

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

c

C

C

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

c

C

C

24

AUTOSCALING PODS



MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULERREGISTRY

HEALTH/SCALING

25

SERVICE DISCOVERY

SERVICE LAYER

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE

CRED HATENTERPRISE LINUX

MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING


26

PERSISTENT DATA IN CONTAINERS

SERVICE LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING


27

ROUTING AND LOAD-BALANCING

SERVICE LAYER

ROUTING LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING


28

ACCESS VIA WEB, CLI, IDE AND API

EXISTING AUTOMATION

TOOLSETS

SCM(GIT)

CI/CD

SERVICE LAYER

ROUTING LAYER

PERSISTENTSTORAGE

REGISTRY

RHEL

NODE

C

C

RHEL

NODE

C C

RHEL

NODE

c

C

C

RHEL

NODE

C C

RHEL

NODE

C

RHEL

NODE


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING



Networking

30

● Built-in internal DNS to reach services by name

● Split DNS is supported via DNSmasq

○ Master answers DNS queries for internal services

○ Other name servers serve the rest of the queries

● Software Defined Networking (SDN) for a unified cluster network to enable pod-to-pod communication

● OpenShift follows the Kubernetes Container Networking Interface (CNI) plug-in model

OPENSHIFT NETWORKING

NODE172.16.1.10

31

OPENSHIFT NETWORKING

POD10.1.2.2

POD10.1.2.4

NODE172.16.1.20

POD10.1.4.2

POD10.1.4.4

IP Network

VxLAN Overlay Network

32

OPENSHIFT NETWORK PLUGINS

OpenShift SDN

(OVS)

OPENSHIFT

KUBERNETES CNI

Flannel** NuageTigera

Calico & CNX

JuniperContrail

CiscoContiv &

Contiv-ACIBig Switch

Fully Supported Validated

VMwareNSX-T

In-Progress

kuryr-k8s

OpenShift SDN

(OVN*)

OpenDaylight(CNI & Kuryr)

RH-OSPNeutronPlugin

Default in OCP 4.1

With OSP 14

FLAT NETWORK

● All pods can communicate with each other across projects

MULTI-TENANT NETWORK

● Project-level network isolation

● Multicast support

● Egress network policies

NETWORK POLICY (Default)

● Granular policy-based isolation33

OPENSHIFT SDN

NODE

POD POD

PODPOD

NODE

POD POD

PODPOD

PROJECT A PROJECT B

DEFAULT NAMESPACE

✓

PROJECT C

Multi-Tenant Network

PROJECT A

34

OPENSHIFT SDN - NETWORK POLICY

POD

POD

POD

POD

PROJECT B

POD

POD

POD

POD

Example Policies● Allow all traffic inside the project● Allow traffic from green to gray● Allow traffic to purple on 8080

✓

✓

8080

5432

✓

apiVersion: extensions/v1beta1kind: NetworkPolicymetadata: name: allow-to-purple-on-8080spec: podSelector: matchLabels: color: purple ingress: - ports: - protocol: tcp port: 8080

✓

35

services provide internal load-balancing and service discovery across pods

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

POD

CONTAINER

role: backend

role: backendrole: backendrole: backendrole: frontend10.110.1.11 10.120.2.22 10.130.3.3310.140.4.44

172.30.170.110

36

apps can talk to each other via services

InvokeBackend API

POD

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

POD

CONTAINER

role: backend

role: backendrole: backendrole: backendrole: frontend10.110.1.11 10.120.2.22 10.130.3.3310.140.4.44

172.30.170.110

37

BUILT-IN SERVICE DISCOVERYINTERNAL LOAD-BALANCING

SERVICE

app=payroll role=frontend

POD

app=payroll

role=frontend

POD

app=payroll

role=frontend

Name: payroll-frontendIP: 172.10.1.23Port: 8080

POD

app=payroll

role=backendversion=1.0 version=1.0

38

BUILT-IN SERVICE DISCOVERYINTERNAL LOAD-BALANCING

SERVICE

app=payroll role=frontend

POD

app=payroll

role=frontend

POD

app=payroll

role=frontend

POD

app=payroll

role=frontend

Name: payroll-frontendIP: 172.10.1.23Port: 8080

POD

app=payroll

role=backendversion=2.0 version=1.0 version=1.0

39

ROUTE SPLIT TRAFFIC

SERVICE A

App A App A

SERVICE B

App B App B

ROUTE

10% traffic90% traffic

Split Traffic Between Multiple Services For A/B Testing, Blue/Green and Canary Deployments

● NodePort binds a service to a unique port on all the nodes

● Traffic received on any node redirects to a node with the running service

● Ports in 30K-60K range which usually differs from the service

● Firewall rules must allow traffic to all nodes on the specific port

40

EXTERNAL TRAFFIC TO A SERVICE ON A RANDOM PORT WITH NODEPORT

NODE192.10.0.12

NODE192.10.0.11

NODE192.10.0.10

SERVICE

INT IP: 172.1.0.20:90

POD

10.1.0.1:90

POD

10.1.0.2:90

POD

10.1.0.3:90

connect 192.10.0.10:31421192.10.0.11:31421192.10.0.12:31421

CLIENT

NODE192.10.0.12

NODE192.10.0.11

NODE192.10.0.10

41

EXTERNAL TRAFFIC TO A SERVICE ON ANY PORT WITH INGRESS

SERVICE

EXT IP: 200.1.0.10:90INT IP: 172.1.0.20:90

POD

10.1.0.1:90

POD

10.1.0.2:90

POD

10.1.0.3:90

connect 200.1.0.10:90

CLIENT

● Access a service with an external IP on any TCP/UDP port, such as

○ Databases

○ Message Brokers

● Automatic IP allocation from a predefined pool using Ingress IP Self-Service

● IP failover pods provide high availability for the IP pool

42

CONTROL OUTGOING TRAFFIC SOURCE IP WITH EGRESS ROUTER

NODEIP1

EGRESSROUTER

PODIP1

EGRESS SERVICEINTERNAL-IP:8080

EXTERNAL SERVICE

Whitelist: IP1

POD

POD

POD

43

● Pluggable routing architecture○ HAProxy Router○ F5 Router

● Multiple-routers with traffic sharding

● Router supported protocols○ HTTP/HTTPS○ WebSockets○ TLS with SNI

● Non-standard ports via cloud load-balancers, external IP, and NodePort

ROUTING AND EXTERNAL LOAD-BALANCING

44

POD

routes add services to the external load-balancer and provide readable urls for the app

CONTAINER

POD

CONTAINER

POD

CONTAINER

BACKEND SERVICE

ROUTEapp-prod.mycompany.com

> curl http://app-prod.mycompany.com

45

SERVICE

POD POD

ROUTER

POD

EXTERNAL TRAFFIC

INTERNAL TRAFFIC

ROUTE EXPOSES SERVICES EXTERNALLY

Container to Container on the Same Host

46

OPENSHIFT SDN - OVS PACKET FLOW

NODE

POD 1veth0

10.1.15.2/24

br010.1.15.1/24

192.168.0.100

eth0

POD 2veth1

10.1.15.3/24

vxlan0

NODE 2

NODE 1

47


POD 1veth0

10.1.15.2/24br0

10.1.15.1/24vxlan0

POD 2veth0

10.1.20.2/24br0

10.1.20.1/24vxlan0

192.168.0.100

eth0

192.168.0.200

eth0

Container to Container on the Different Hosts

Container Connects to External Host

Container to Container on Different Hosts

48


NODE 1

POD 1veth0

10.1.15.2/24br0

10.1.15.1/24tun0

192.168.0.100

ExternalHost

eth0


OpenShift Monitoring / Clustering

50

AUTO-HEALING FAILED PODS

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

51

AUTO-HEALING FAILED CONTAINERS

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

52


RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

53


RHEL

NODE

RHEL

NODE

c

RHEL

NODE

RHEL

NODE

c

RHEL

NODE

C

C

RHEL

NODE

C

C


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

54


RHEL

NODE

RHEL

NODE

RHEL

NODE

RHEL

NODE

C

C

RHEL

NODE

C

C

c


MASTER

API/AUTHENTICATION

DATA STORE

SCHEDULER

HEALTH/SCALING

C

c


OpenShift persistant Storage

56

● Persistent Volume (PV) is tied to a piece of network storage● Provisioned by an administrator (static or dynamically)● Allows admins to describe storage and users to request storage● Assigned to pods based on the requested size, access mode, labels

and type

PERSISTENT STORAGE

NFS

GlusterFS

OpenStack Cinder

Ceph RBD

AWS EBS

GCE Persistent Disk

iSCSI

Fiber Channel

Azure Disk

Azure File

FlexVolume

VMWare vSphere VMDK

Container Storage Interface (CSI)**

* Shipped and supported by NetApp via TSANet** Tech Preview

NetApp Trident*

PROJECT

POOL OF PERSISTENT VOLUMES

57

PERSISTENT STORAGE

NFSPV

iSCSIPV

NFSPV

Admin

User

register PV

create claim

NFSPV

GlusterFSPV

Pod

claim

Pod

claim

Pod

claim

CephRBDPV

58

DYNAMIC VOLUME PROVISIONING

Admin

User

define StorageClass

create claim: Fastest

SlowAzure-Disk

FastAWS-SSD

FastestNetApp-Flash

NetAppProvisioner

AWSProvisioner

Pod

claim

PV

OpenShiftPV Controller

provision

AzureProvisioner

bound


Thank you !

Advanced Architecture OpenShift Container Platform (OCP)

Documents

Transcript of Advanced Architecture OpenShift Container Platform (OCP)