OpenShift Dedicated 3 Developer Guide - Red Hat...OpenShift Dedicated 3 Developer Guide
UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT...
Transcript of UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT...
![Page 1: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/1.jpg)
![Page 2: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/2.jpg)
UNDER THE HOOD OF OPENSHIFT:TURBOCHARED BY RED HATENTERPRISE LINUX
Ian PilcherSr. Solution Architect, Red HatDaniel WalshSr. Principal Software Engineer, Red HatJune 13, 2013
![Page 3: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/3.jpg)
Contents
● OpenShift Overview
● Control Groups
● SELinux
● Namespaces
● Demo
● Q&A
![Page 4: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/4.jpg)
OPENSHIFT OVERVIEW
![Page 5: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/5.jpg)
Brokers and Nodes
AWS / CloudForms / IaaS (OpenStack) / Virtual (RHEV) / Bare Metal
Nodes are where User Applications live.Brokers keep OpenShift running.
Brokers Node Node Node
RHEL RHEL RHELRHEL
![Page 6: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/6.jpg)
Gears
RHEL RHEL
OpenShift GEARS represent secure containers in RHEL
Broker Node Node Node
RHEL
JBoss
My Gear
AWS / CloudForms / IaaS (OpenStack) / Virtual (RHEV) / Bare Metal
![Page 7: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/7.jpg)
Lots of Gears!
Broker
RHEL RHEL RHEL
Node Node
AWS / CloudForms / IaaS (OpenStack) / Virtual (RHEV) / Bare Metal
![Page 8: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/8.jpg)
What Is a Gear?
● User and group● Name == gear UUID
● UID and GID 1000+
● SELinux category● c${UID}
● Control Group● /openshift/${UUID}
● Home directory● /var/lib/openshift/${UUID}
● Running processes
![Page 9: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/9.jpg)
OpenShift Operating System Requirements
Resource ManagementEnsures that a gear can consume only its allocated portion of a shared resource.
Control groups (cgroups)Filesystem quotas
Access ControlPrevents a gear from inappropriately reading or modifying system resources.
SELinuxFilesystem permissionsNamespaces
PolyinstantiationProvides the appearance of access to a system-wide resource.
Namespaces
Provided By ...
CPU
Provided By ...
Provided By ...
Memory Disk Network
![Page 10: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/10.jpg)
LINUX CONTROL GROUPS(cgroups)
![Page 11: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/11.jpg)
OpenShift Control Groups
● One cgroup per gear
● /openshift/${UUID}
● Created by openshift-cgroups service
● cgrulesengd places processes in correct group (based on process EUID)
● Parameters in /etc/openshift/resource_limits.conf
CG
roup
s CG
roups
MyApp
![Page 12: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/12.jpg)
Resource Controller Limits
cpu● cpu.cfs_period_us = 100000● cpu.cfs_quota_us = 30000● cpu.rt_period_us = 100000● cpu.rt_runtime_us = 0● cpu.shares = 128
memory● memory.limit_in_bytes = 536870912● memory.memsw.limit_in_bytes = 641728512● memory.soft_limit_in_bytes = 9223372036854775807● memory.swappiness = 60
SmallGear
(default)
![Page 13: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/13.jpg)
Additional Resource Controllers
cpuacct● Gathers CPU usage statistics for all
group (gear) processes.
● Statistics not currently used.net_cls
● Tags all network packets generated by gear with class identifier (generated from gear's SELinux category).
● Class identifier can be used for traffic shaping.
● Traffic shaping not currently used.
freezer● Stops all processes in group (gear) from
executing.
● Used by OpenShift Online to achieve massive scale.
● Not used in OpenShift Enterprise.
![Page 14: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/14.jpg)
SECURITY-ENHANCED LINUX(SELinux)
![Page 15: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/15.jpg)
OpenShift and SELinux
Broker
RHEL RHEL
SE
Linu
x
SELinux
SE
Linux
MyApp
MyApp
Node
AWS / CloudForms / IaaS (OpenStack) / Virtual (RHEV) / Bare Metal
![Page 16: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/16.jpg)
SELinux is a LABELING System
● Everything has a label● Process,file,dir, chr_file, blk_file, port, node.
● SELinux Policy defines that access between process labels and all other labels.
● The Kernel controls the access.
![Page 17: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/17.jpg)
Containers != Security
● Running root in a container, machine pwned
● Local Privilege Escalation, machine pwned
● Much of the system is not containerized.● Audit● /sys
● selinuxfs, cgroupfs, sysfs
● Need to block mount● Need to block mknod
![Page 18: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/18.jpg)
Security Goals
http://en.wikipedia.org/wiki/Maginot_line
![Page 19: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/19.jpg)
![Page 20: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/20.jpg)
SELinux is Type Enforcement
system_u:system_r:openshift_t:s0:c1,c2
system_u:system_r:openshift_var_lib_t:s0:c1,c2
seinfo -t | grep openshift
openshift_mail_tmp_t, httpd_openshift_content_t, openshift_cgroup_read_tmp_t, openshift_initrc_tmp_t, openshift_var_lib_t, openshift_var_run_t, openshift_app_t, openshift_min_t, openshift_net_t, openshift_tmp_t, openshift_min_app_t, openshift_net_app_t, openshift_cgroup_read_t, httpd_openshift_script_exec_t, openshift_cron_tmp_t, openshift_initrc_t, httpd_openshift_script_t, openshift_cron_exec_t, openshift_initrc_exec_t, openshift_rw_file_t, openshift_log_t, openshift_cron_t, openshift_mail_t, openshift_port_t, httpd_openshift_ra_content_t, httpd_openshift_rw_content_t, httpd_openshift_htaccess_t, openshift_cgroup_read_exec_t, openshift_t, openshift_tmpfs_t
![Page 21: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/21.jpg)
SELinux is Type Enforcement
● Process Labels can be on Files● File Labels != Process Labels
● openshift_t -> Process● openshift_var_lib_t -> File
![Page 22: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/22.jpg)
SELinux is MCS – Multi Category System
system_u:system_r:openshift_t:s0:c1,c2
system_u:system_r:openshift_var_lib_t:s0:c1,c2
● MCS Enforcement separates “same types”● openshift_t:s0:c1,c2 -> openshift_var_lib_t:s0:c1,c2● openshift_t:s0:c3,c4 -> openshift_var_lib_t:s0:c3,c4● openshift_t:s0:c1,c2 openshift_var_lib_t:s0:c3,c4
![Page 23: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/23.jpg)
Kernel
Host Hardwarememory, storage, etc.
openshift_t:s0:c3,c4openshift_t:s0:c1,c2
MCS In action
openshift_var_lib_t:s0:c1,c2
SELinux
openshift_var_lib_t:s0:c3,c4
![Page 24: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/24.jpg)
MCS Labeling based on UID
def gen_level(uid): SETSIZE=1023 TIER=SETSIZE ORD=uid; while ORD > TIER: ORD = ORD - TIER; TIER= TIER - 1; TIER = SETSIZE - TIER; ORD = ORD + TIER; return "s0:c%d,c%d" % (TIER, ORD)
![Page 25: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/25.jpg)
How do the labels get on gears
● Host receives packet for a gear● OpenShift server
● launches application with correct SELinux label.● Sends packet to application
● If connection comes in via git or ssh● Ssh uses pam_openshift
● Launch sh with correct context● Launch git with correct context
![Page 26: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/26.jpg)
LINUX NAMESPACES
![Page 27: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/27.jpg)
OpenShift & Linux Namespaces
RHEL6 Openshift
● Mount : mounting/unmounting filesystems
● /tmp, /var/tmp and /dev/shm
RHEL7 Openshift
● IPC : SysV message queues, semaphore/shared memory segments
● Network: IPv4/IPv6 stacks, routing, firewall, proc/net /sys/class/net directory trees, sock
● Critical to fix localhost problem● Pid: Private /proc, multiple pid 1's
![Page 28: UNDER THE HOOD OF OPENSHIFT - Red Hat · UNDER THE HOOD OF OPENSHIFT: TURBOCHARED BY RED HAT ENTERPRISE LINUX Ian Pilcher Sr. Solution Architect, Red Hat Daniel Walsh Sr. Principal](https://reader033.fdocuments.in/reader033/viewer/2022053009/5f0c991b7e708231d43632fa/html5/thumbnails/28.jpg)
DEMO