Dockerized - Amazon Web Services · 21 • Two Candidates – Using Tizen subset (Tizen minimal)...
Transcript of Dockerized - Amazon Web Services · 21 • Two Candidates – Using Tizen subset (Tizen minimal)...
2
Abstract
update
Cloud
Tizen Pla.orm ECO System
Cloud
Management (Update)
Build CI
Monitoring
Store Data (DB)
Docker
Container ECO System
Cloud Infrastructure
Dockeriza:on
3
Agenda
• Platform Management • Docker Introduction • Embedded Container • Dockerization • Demo • Challenges
5
• Development, Deployment & Operations – Tizen Platform have life-cycles ? – Platform builder – F/W upgrade – Remote control
Platform Mgt. | Situation
6
• In IoT devices, The platform should be – Simple as a single application – Faster to create application – Easy to distribute – Support remote control (update, monitoring, …) – Safe for system failure
• Docker can be a solution ?
Platform Mgt. | Challenges
9
Container - Similar to VM
- but, based on Linux system call (no Virtual OS)
- OCI (Open Container Initiative)
- Isolated name space with executable packages
Docker (Container platform) - Build container image, Run container
- ECO system for container image
- Services (deploy, management)
Docker | Basic Concept
h1ps://www.docker.com/what-‐container
11
Orchestration Management - Connection to cloud server
- Device Clustering
Docker | Extended Workflow
container container container A
container B
replica9on
container C
container D
POD-‐A
Docker
NODE-‐A
container E
Docker
NODE-‐B
container E
Docker
NODE-‐C
container E
Docker
NODE-‐D
HW-‐A HW-‐B HW-‐C
SERVICE Node Cluster
SERVICE Cloud Server
12
Docker | Services
Monitoring - Host : CPU load, Memory, Disk Space, Running containers / Host UP time
- Containers : CPU load, Memory, Disk I/O, Network I/O
Container Deploying - Rolling update, Rollback
Logging - System log, Containers log
Container Mgt. - Scaling, load balancing
14
Embedded Container | Concept
• Docker in embedded device • Container has a initializer (/sbin/init instead of /bin/bash) • Running container with privileged permission
– Full HW resources
Lightweight Host OS
Linux Kernel
Embedded Pla.orm Container
Docker
15
Embedded Container | Usage
container A
container A
container container container A
container B
Docker
Server Infra structure
container A
container A
container container container A
container B
Docker Lightweight Host OS
PlaHorm + App Container A
Docker
Lightweight Host OS
PlaHorm + App Container A
Docker
Cloud Service
Cloud Service U:lize
• Service oriented (regardless of physical device) • Homogeneous app containers in server infra
• Device oriented • Homogeneous app containers in different device • Proper to IoT system
Exis:ng Usage for Server New Usage for Embedded Device
16
• Platform Managements with Docker
• Tizen Platform as a Embedded Container
Embedded Container | Tizen Platform
Docker service features Pla.orm management tools
Build CreaMon/modificaMon
Deployment DistribuMon
Update Upgrade
Docker-‐registry PlaHorm store
Lightweight Host OS
Linux Kernel
Tizen PlaHorm
Dockeriza:on
Linux Kernel
Container
Docker
18
Cloud Server
Overall Architecture
Docker Registry
[Host OS] Mzen-‐minimal
/ bare-‐os
[Container] Mzen-‐headless
dockzen-‐ launcher
Linux kernel
docker-‐client
docker-‐daemon
containerd
OCI::runc
container-‐shim
container-‐ctr
docker-‐engine swarm
ca-‐cerMficate
dockzen -‐agent
update
security
monitor
Network (Wi-‐Fi)
kernel + Host
Mzen-‐headless
kernel + Host
Mzen-‐headless
Mul:media fw
kernel + Host
Mzen-‐headless
Voice App
Create images (+ fw)
Create images (+App)
[Tizen Pla.orm Containers]
19
Dockerization | Kernel Patches
Enable cgroup “FATA[0001] Error starMng daemon: Devices cgroup isn't mounted” Fix : { CONFIG_CGROUP_DEVICE=y, CONFIG_CPUSETS=y, CONFIG_BLK_CGROUP=y}
iptables error “FATA[0002] Error starMng daemon: Error iniMalizing network controller: Error creaMng default "bridge" network: Failed to program NAT chain: Failed to inject docker in PREROUTING chain: iptables failed: iptables -‐-‐wait -‐t nat -‐A PREROUTING -‐m addrtype -‐-‐dst-‐type LOCAL -‐j DOCKER: iptables: No chain/target/match by that name.” Fix : {CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y }
rooHs mount error “error=oci runMme error: container_linux.go:247: starMng container process caused "process_linux.go:359: container init caused \"rooHs_linux.go:54: mounMng \\\" to rooHs \\\" at \\\" caused \\\ such device\\\"\"“” Fix : {CONFIG_POSIX_MQUEUE=y}
cgroup memory path error “ERRO[0187] containerd: noMfy OOM events error=cgroup path for memory not found panic: standard_init_linux.go:175: exec user process caused "exec format error“” Fix : {CONFIG_MEMCG=y, CONFIG_MEMCG_SWAP=y, CONFIG_MEMCG_KMEM=y}
Enable Overlayfs Fix : {CONFIG_OVERLAY_FS=y}
docker-‐runc keyring failure “error=oci runMme error: container_linux.go:247: starMng container process caused "process_linux.go:359: container init caused \"could not create session key: funcMon not implement“ Fix : enable keyctl syscall compaMbility for 32bit with 64bit kernel {CONFIG_KEYS_COMPAT}
Host OS
Kernel Docker & FW
Container
• Kernel Has Docker Dependencies
20
• Required Packages in Host-OS – rootfs – cgroup – Network – Certification – Docker & Frameworks
Dockerization | Host OS
[Host OS] Mzen-‐minimal
/ bare-‐os dockzen-‐launcher
Linux kernel
docker-‐client
docker-‐daemon
containerd
OCI::runc
container-‐shim
container-‐ctr
docker-‐engine
swarm
ca-‐cerMficate
dockzen-‐agent update security monitor
Network (Wi-‐Fi)
Host OS
Kernel Docker & FW
Container
21
• Two Candidates – Using Tizen subset (Tizen minimal) – Create New for docker (BareOS)
Dockerization | Host OS
cerMficate : 1MB
docker : 52MB
wifi netconfig base
: about 60MB
Tizen minimal BareOS Arch type arm arm Size (ROM) 123MB 66M Size (RAM) 250MB (run dockerd : 311MB) 53M (run dockerd : 113M) Kernel version 4.4.19 4.4.19 Docker version v1.13.1 v1.13.1 Init system systemd sysVinit Package manager tpk None Filesystem ext4 ext4
cerMficate : 1MB
docker : 52MB
wifi / base : about 13MB
Tizen minimal RAM Size (113 MB)
BareOS RAM Size (66 MB)
Host OS
Kernel Docker & FW
Container
22
• Manage docker life-cycle • Manage Container life-cycle • Monitoring APIs
Dockerization | Dockzen-launcher
command
API
docker engine
dockzen-‐launcher
device dockerd connect config file
systemd
content
state
Service MainLoop
test
json parser API parser
dockzen-‐agent
Host OS
Kernel Docker & FW
Container
23
• Binding as a Container • Connection to Cloud
– Manage Device uuid – Authentication
• Configure Update Policy
Dockerization | Dockzen-agent
web connecMon <<back-‐end>>
API
agent
converter connect
dockzen-‐agent
Server
dockzen-‐launcher
Host OS
Kernel Docker & FW
Container
24
Containerization | Initial Creation Host OS
Kernel Docker & FW
Container
• Platform Binaries to Tizen Container Image In Host PC 1. Download platform binaries (https://download.tizen.org/) 2. Loopback mount using mnt-img.sh
• $ ./mnt-img.sh mount tizen-common_xxx_common-wayland-3parts-armv7l-artik.tar.gz
3. Compress tarball • $ sudo tar --xattrs -cvf ../[tar-name] .
In Target 4. Docker-import
• $ cat [tar-name] | docker import – [local-container-name]
5. Push into Docker-Hub • $ docker tag [local-container-name] [dockerhub-id]/[image-name] • $ docker push [dockerhub-id]/[image-name]
25
• Docker-Build with Dockerfile 1. Install yum pkg-mgr
• Add yum into base container image
2. Case Study • Add curl application à New Image
Containerization | Re-Creation
### base_packages.repo [base_packages] name=base_packages type=rpm-‐md baseurl=h1ps://download.Mzen.org/snapshots/Mzen/base/latest/repos/arm/packages enabled=1 gpgcheck=0 sslVerify=false
### common_packages.repo [common_packages] name=common_packages type=rpm-‐md baseurl=h1ps://download.Mzen.org/snapshots/Mzen/common/latest/repos/arm-‐wayland/packages enabled=1 gpgcheck=0 sslVerify=false
v yum package files ### dockerfile for added yum_pkg and exampleApp ### FROM base-‐image # install yum # ADD yum/yum_pkg /usr/tmp/yum_pkg/ RUN rpm -‐Uvh -‐-‐nodeps -‐-‐force /usr/tmp/yum_pkg/*.rpm ADD yum/*.repo /etc/yum.repos.d/
### install rpm pkg and exampleApp ### FROM base-‐image-‐yum # install packages # RUN yum install curl
Host OS
Kernel Docker & FW
Container
26
• Tizen uses Smack Security – Extended attributes : security.SMACK64, security.capability – Need to check xattr operations in docker
• patch#1 : Capability error – Failure in Tizen Container running – Occurred permission error checking “CAP_MAC_ADMIN” – In OverlayFS, upper layer can’t sync into lower layer as permission – http://www.spinics.net/lists/linux-unionfs/msg00593.html
• patch#2 : xattr copy error – Failure in docker commands (commit, push, …) – Extended attribute lists doesn’t be copied (in case of overlay, not overlay2)
Issues | Smack Security
27
• /sbin/init (systemd) vs. /bin/bash – Much discussions about “systemd in docker”
• systemd requires privileged permission – Initialize overall services regarding HW devices – Necessary in Tizen container
• Patches adding “-- privileged” – Docker-build – Docker-service
Issues | Privileged Container
28
• Union file system – Handled by layer architecture – Avoid duplication and isolation
• History – Early 2013 : AUFS – Late 2013 : Device Mapper – Early 2017 : Overlay
• Apply for Tizen – OverlayFS – Stability / mainline support – Performance
Issues | Union File System
29
• Security – Need to minimize privileged permission
• Fail safe – Robust Host-os – Container can be recovered(reboot)
• Resource management – Violation occurred in network resource – CPU and memory is separated – Disk is controlled by same journaling thread
Quality Inspection
31
Scenario Structure
Developers
Docker Registry (official / public)
Docker Registry (public / private)
Release
Service Server
Docker Registry
Build Tizen Container Image
Register Devices
Push New Image
Update Images
Dash-board • Update • Monitoring
<3rd Party Develop> Product Container Image
Embedded Device
(ARTIK710)
Web UI • Image Repository
33
Demo Structures
Container / Mzen-‐headless
dockzen-‐launcher
Linux kernel 4.4
docker-‐engine
dockzen-‐agent
Container / others
ARTIK7
[dockzen-‐OS] base on
Mzen-‐minimal
docker api
IPC
PoC Server
H1p Server
server
dockzen-‐backend agent backend websocket
registry
registry-‐web
H1p Server
Docker Registry
container mgt.
rest
Docker-‐registry Web Dash-‐board Web
websocket
Target Device
34
• Packages – Artik7 boot&kernel – Host os
• docker-engine • docker framework
– Tizen container image
• Instructions – Download boot&kernel – Download host os – Execute Tizen container image (only first time)
Development
36
• Improvement – Extend target device (raspi-3) – Create Tizen 4.0 reference container images – Optimize host-os embedded on Docker
• Serviceability – Service to support Tizen docker is in development – 3rd Party can deploy Tizen docker in the future
Next
37
• Github organization : https://github.com/dockzen – Docker source-code (patched for tizen)
• https://github.com/dockzen/docker • https://github.com/dockzen/containerd • https://github.com/dockzen/runc
– Docker framework • https://github.com/dockzen/dockzen-launcher • https://github.com/dockzen/dockzen-agent
– Host-os : https://github.com/dockzen/dockzen-os – Artik7 kernel : https://github.com/dockzen/linux-artik7-docker
• Docker-hub containers • https://hub.docker.com/u/dockzen/
Contributing…