Put your Kubernetes into Jail - inovex · Put your Kubernetes into Jail Deployment and Operations...
Transcript of Put your Kubernetes into Jail - inovex · Put your Kubernetes into Jail Deployment and Operations...
Put your Kubernetes into Jail
Deployment and Operations on AWS China
Timo Heinrichs, Arnold Bechtoldt 17.04.2018
2
Arnold Bechtoldt
Systems Engineer
Timo HeinrichsCloud Platform Engineer
3https://github.com/kubernetes/kops/blob/master/docs/aws.md
Rahmenbedingungen des Projekts
› „Wir möchten Docker Container orchestrieren“
› AWS ist gesetzt
› Kubernetes ist gesetzt
› Vendor Lock-In soll vermeiden
4
Kubernetes Deployment mit KopsKubernetes Operations
“The easiest way to get a production grade Kubernetes cluster up and running.”
Wir brauchen:
› AWS Account
› S3 Bucket
› kubectl
5
Kubernetes Deployment mit KopsCluster Setup
kops create cluster \
--name=kubernetes-cluster.example.com \
--state=s3://kops-state-1234 \
--zones=eu-west-1a \
--node-count=2 \
--yes
6
Dealbreaker: IAM Policies
› „Der Kunde“ verwaltet IAM (User, Roles, Policies)
› Kops braucht› AmazonEC2FullAccess
› AmazonRoute53FullAccess
› AmazonS3FullAccess
› IAMFullAccess
› AmazonVPCFullAccess
7
Die Lösung: Terraform und Bash-Magic
8
Easy…
sed s/eu-central-1/cn-north-1/g kops.yaml
China?
9
AWS ChinaAWS Region ist gleich AWS Region?
› Treuhänder betreiben AWS CN
› Kein Route53 (Cloud ohne DNS?!)
› EC2 Images (AMI) teilweise nicht verfügbar und/oder veraltet (CoreOS ist verfügbar)
› IAM Policy Prefix "arn:aws-cn:s3:::mybucket"
› Great Firewall of China = Great 56k Modem of China
› Keine Google Services (z.B. GCR) erreichbar
10
11https://www.weave.works/blog/weave-gossip-dns/
Gossip DNS
12http://suraj.pro/post/blog11/
Die Kraft der zwei Registries„Offline Deployment“
13
Kops Assets
› Kops benötigt zum installieren weitere Tools› protokube
› nodeup
› CNI Plugin
› kubectl
› Lösung: Dateien nach S3 hochladenEC2 VM mit „signing proxy“ erlaubt einen HTTP Download
14
One more thing…
15https://www.zenlayer.com/internet-content-provider-license/
ICP License"Internet Content Provider" License
› Port 80 und 443 standardmäßig gesperrt
› Kein Webdienst in China ohne ICP License
› Domain gebunden
› Explizite Angabe der IP Adressen
16
Ingress ELB
› AWS Support vergibt statische IP-Adressen
› Kubernetes-managed ELBs sind flüchtig
› Service Type „NodePort“ anstatt „LoadBalancer“
› ELB wird über Terraform gemanaged
› nginx als Ingress
17
Kubernetes Deployment mit KopsKubernetes Operations
“The easiest way to get a production grade Kubernetes cluster up and running.”
Wrap-up
18
Fragen?
Vielen Dank
Arnold [email protected]
Timo [email protected]
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
20
Kops Setup Demo:
https://asciinema.org/a/97298
21
kops.yaml apiVersion: kops/v1alpha2kind: Clustermetadata: creationTimestamp: 2017-05-04T23:21:47Z name: k8s.example.comspec: api: loadBalancer: type: Public authorization: alwaysAllow: {} channel: stable cloudProvider: aws configBase: s3://example-state-store/k8s.example.com etcdClusters: - etcdMembers: - instanceGroup: master-us-east-2d name: a - instanceGroup: master-us-east-2b name: b - instanceGroup: master-us-east-2c name: c name: main - etcdMembers: - instanceGroup: master-us-east-2d name: a - instanceGroup: master-us-east-2b name: b - instanceGroup: master-us-east-2c name: c name: events kubernetesApiAccess: - 0.0.0.0/0 kubernetesVersion: 1.6.6 masterPublicName: api.k8s.example.com networkCIDR: 172.20.0.0/16 networkID: vpc-6335dd1a
networking: weave: {} nonMasqueradeCIDR: 100.64.0.0/10 sshAccess: - 0.0.0.0/0 subnets: - cidr: 172.20.32.0/19 name: us-east-2d type: Private zone: us-east-2d - cidr: 172.20.64.0/19 name: us-east-2b type: Private zone: us-east-2b - cidr: 172.20.96.0/19 name: us-east-2c type: Private zone: us-east-2c - cidr: 172.20.0.0/22 name: utility-us-east-2d type: Utility zone: us-east-2d - cidr: 172.20.4.0/22 name: utility-us-east-2b type: Utility zone: us-east-2b - cidr: 172.20.8.0/22 name: utility-us-east-2c type: Utility zone: us-east-2c topology: bastion: bastionPublicName: bastion.k8s.example.com dns: type: Public masters: private nodes: private
22
Cluster Specs
› eu-central-1 (FRA)
› Container Linux (CoreOS)
› „Private Topology“ -> Nodes haben keine public IPs
› 3 HA Kubernetes Master
› Nodes und Master in Autoscaling Groups
› Weave als Overlay Netzwerk
› Automatisches Route53 Setup