Docker feedbackFeedback about Docker first experience at Tribal Nova2015/01/19

Nicolas DEGARDIN <ndegardin @tribalnova.com>

Overview

• Docker intro• Docker• Architecture: nginx multi-projects• Architecture: nginx single project• Architecture: nginx single-project test procedure• Conventions• Vagrant• Docker + Vagrant• Docker + Vagrant provisioner• Docker + Vagrant provider• Docker + Vagrant + PHPStorm• Docker + other tools• Hints• Conclusions so far

Docker intro

What is Docker?

A container management tool:Allows to embed an application and its dependencies in a virtual and portable container.

Technologies involved:•cgroups•LXC (linux container)•AUFS (Another Unix File System)

At the time of writing this document: Docker 1.4.

Docker intro

What is not Docker?

A virtual machine management tool.A provisioning tool.

Docker is voluntarily restrained to disallow build parametrizing.Technics involving generating/altering the build file make no sense: this is not the point of Docker, and there are better suited tools to serve this purpose.

Docker intro

My first objectives:

•Full development environment quickly set up•Get rid of the dependencies problems•Unified configuration between developers/projects•Reusable containers for other purposes:

–Deployable in other environments than DEV–Tests/continuous integration

•Configurable containers•But keeping configuration simple (avoiding too heavy command lines/deep understanding of these containers)

•Integrate the docker image inside the project•If possible, no adaptation to our projects configuration

Docker

Installation:

•As a Linux package (lxc-docker)

•With « boot2docker » under Windows and Mac OS– may not run directly under these environments as it is built upon Linux 2.6 kernel features

– boot2docker is supposed to turn transparent the OS gap– boot2docker is really lightweight: no package manager. Not meant to be used as a host VM

•Via a vagrant provisioner…

Docker

Images based upon Dockerfile, aimed at being hierarchical.No parametrization, no factorization (no includes, no reusable files)

FROM …

ADD…RUN …RUN ...

ENTRYPOINT …CMD …

FROM A

ADD…RUN ...

ENTRYPOINT …CMD …

FROM …

RUN ...

FROM A

ADD…RUN …RUN ...

CMD …FROM B

RUN…RUN ...

A

B

C

D

Docker

Containers can communicate via sockets, file volumes and environment variables. These communication modes are not bidirectionnal.

first_container:A second_container:A third_container:B

Volumes from second_container:the folders from the second container are visible in the first container

Linked to third container:network features and env variables that allow socket communication

Docker

Note about volumes:•When a volume from a container is mounted into another one, both can read and write inside at runtime

•A mounted volume can erase another one (or cover it – AUFS)•If the volume is defined during the build, files inside its directory may disappear (before or after the volume is defined, different behaviours have been observed for similar cases)

•Volume files cannot be commited and are not inherited by child images

So the developer should be cautious and avoid too audacious structures. I observed surprising/strange behaviours when experimenting for some edge cases. It also seems more secure to declare volumes when running the container.

Architecture

Here comes my feedback from different container architectures experimentations.

I represent:•containers by ugly grey rectangles

•Image inheritance by black arrows

•Mounted volumes by blue lines, with a dot to show who’s the boss

•Linked containers by red lines (implying shared environment variables and certainly a bit of scripting)

Baseimage

Architecture

The involved images are:•Phusion, a lightweight, Docker oriented image based upon Ubuntu•Baseimage: our base image adding a few common features•Common web image (composer, git, PHP and so on…)•Web: nginx•Web dev: web developer tools (xdebug, XHProf, PHPUnit, PHPCS…)•MySQL•Let’s add PHPMyAdmin and XHProf separated containers•Our project source code images:

–GMA (Go Math Academy) and CW (Curious World), relying upon PHP5.5 (ubuntu 14.04 images)

–ILW (iLearnWith) and C&M (Charly & Max) relying upon PHP5.3 (ubuntu 12.04 images)

–Each with their database images (embedding SQL dumps)

Architecture : a try with an nginx multi-projects architecture

Baseimage1404

Baseweb1404

WebSymfony1.5

Web devSymfony1.5

ProjectGMA

ProjectILW

XHProf

Phusion/ubuntu1404

Phusion/ubuntu1204

Baseimage1204

Baseweb1204

WebSymfony1.0

Web devSymfony1.0

ProjectCW

ProjectC&M

MySQL

GMA db CW db

C&M db ILW db

PHPMyAdmin

Architecture : nginx multi-projects

•Classical unique nginx server holding manywebsites•The projects sources are builts as containers,and shared as volumes

•The project sources also embed nginxconfiguration files

•The DB dump is embedded in avolume container

•A web server with dev/debug toolsinherits from the minimalist one

Baseimage1404

Baseweb1404

WebSymfony1.5

Web devSymfony1.5

ProjectGMA

XHProf

Phusion/ubuntu1404

MySQL

GMA db

PHPMyAdmin

Architecture : nginx multi-projects

Improvements:•Put together the database build scripts and theproject source

•Remove baseweb (was supposed to holdonly base build softwares likePHP-cli and composer for a futureunit tests/builder container)

WebSymfony1.5

Web devSymfony1.5

Project/DBGMA

XHProf

Phusion/ubuntu1404

MySQLPHPMyAdmin

Architecture : nginx multi-projects

In nginx.conf:include /etc/nginx/sites-enabled/*/*.conf;

Allows to mount website configurationsas volumes from the projectcontainer.

Project deployment at runtime, toallow parametrizing according toenvironment variables (composer options for instance).(suits our project configuration)

WebSymfony1.5

Web devSymfony1.5

Project/DBGMA

XHProf

Phusion/ubuntu1404

MySQLPHPMyAdmin

Architecture : nginx multi-projects

Weaknesses:•volume modifications may not be commited, badif we want to push tagged versions of the project

•The project cannot change anything in the webfile system (the fact of transmittingproject specific scripts wasconsidered but seemed twisty…)

•The web server must hold a large,compatible configuration fittingevery project

•Web server/projects dependenciesthat’s what Docker is designed toprevent

WebSymfony1.5

Web devSymfony1.5

Project/DBGMA

XHProf

Phusion/ubuntu1404

MySQLPHPMyAdmin

Architecture : nginx multi-projects

The counter-example of XHProf

Web Symfony1.5

php5-xhprofphp5-mcrypt

Other php5 libsXhgui.conf (nginx)

Project/DB GMA

XHGUI

php5-xhprofphp5-mcrypt

Xhgui.conf (nginx)

Only the sources and the nginx scripts are needed, but composer must be run to load it, and expects

php5-mcrypt and php5-xhprof

php5-mcrypt and php5-xhprof must be the version expected by XHGui. Dependency concerns?Must embed the requirements of every project

XHGui must set “auto_prepend_file=…” with the path to its sources for every project that use it (in nginx fastcgi_params or PHP-FPM conf). Hard to do at startup (and in a parametrable way) if it’s an isolated container.

Architecture : nginx single-project

•nginx server running one project•A project may inherit from the minimalist serveror from the dev server

•web tools, including XHGui are on the devversion

•it is easy for a project to adapt configurationfiles or extensions

•more Docker-friendly

Drawback: if a dev version of an image isdone, the dev version of the project doesn’tinherit of the release one…

Baseimage

Web-php55

Symfony1.5dev

ProjectGMA dev

Phusion/ubuntu1404

MySQL

PHPMyAdmin

ProjectGMA

Symfony1.5

Architecture : nginx single-project

To address this issue, dev scripts may beadded to each image.

•any inheriting image may use it if needed•a developer opening a bash session ona container may launch one of these scriptsto deploy tools if needed

Drawback: changing a script affects thechild images, that need to be rebuilt (whichmay take a long time).

Baseimage

Web-php55

ProjectGMA dev

Phusion/ubuntu1404

ProjectGMA

Symfony1.5

install-dev-tools.shInstalls zsh, wget, vim…

install-web-dev-tools.shInstalls xdebug, xhprof…install-web-php-tools.shInstalls phpunit, phpmd,

phpcs, phpcsfixer…

runs install-dev-tools.sh, install-web-dev-tools on build

Architecture : nginx single-project

Another point of this architecture:how to add inheritance to the ENTRYPOINTand RUN commands, and manageenvironment variables at each level?

FROM tribalnova/baseimageADD files/startup.sh /var/docker/startup.shENTRYPOINT ["/var/docker/startup.sh"]CMD nginx

Web-php55 Dockerfile

FROM tribalnova/web-php55ADD files/project-startup.sh /var/docker/web-startup.shENTRYPOINT ["/var/docker/startup.sh"]

symfony15 Dockerfile

Baseimage

Web-php55

ProjectGMA dev

Phusion/ubuntu1404

ProjectGMA

Symfony1.5

Architecture : nginx single-project

I combine the startup.sh scripts on build, using the shebang as a place holder:

#!/bin/sh

if [ $# -gt 0 ]; then echo "Executing: $*" $*fi

Web-php55 Startup.sh#!/bin/sh

some commands

if [ $# -gt 0 ]; then echo "Executing: $*" $*fi

Web-php55 Startup.sh

#!/bin/sh

some commands

Symfony15 Startup.sh

RUN (perl -i -p -000 -e s@'#\!/bin/sh'@"$(cat /var/docker/startup-web-dev.sh|sed -e 's/\\/\\\\/g'|sed 's/\$/\\\$/g')"@ /var/docker/startup.sh && rm /var/docker/startup-web-dev.sh)Perhaps is there a simpler approach… Replacement in a script is a bit tricky(escaped characters). Would be nice to have a PRERUN Docker instruction

Architecture : nginx single-project

It was also considered to use a init.d likedirectory, or even the init system itself.

The Phusion base image embeds runit,which allows to define simply startup scripts.

But concatenation allows to generate asingle startup.sh script, easy to read, launchAnd debug when wandering inside thecontainer. And limiting the use of specificfeatures (runit) allows to switch to anotherbase image if needed.

Baseimage

Web-php55

ProjectGMA dev

Phusion/ubuntu1404

ProjectGMA

Symfony1.5

Architecture : nginx single-project

Baseimage1404

Web-php55

Symfony1.5

ProjectILW

Phusion/ubuntu1404

Phusion/ubuntu1204

Baseimage1204

Web-php53

Symfony1.0

ProjectC&M

MySQL

PHPMyAdmin

ProjectILW-dev

ProjectC&M-dev

ProjectCW

ProjectGMA

ProjectCW-dev

ProjectGMA-dev

MongoDB(for XHGui)

Full stack

Architecture : nginx single-project

Baseimage1404

Web-php55

Symfony1.5

Phusion/ubuntu1404

Phusion/ubuntu1204

Baseimage1204

Web-php53

Symfony1.0

Ressources sharing between close images

•Docker builds from a file called « Dockerfile »,the name of the file cannot be anything else.

•The Dockerfile can only use ressource fromits directory or below, never upper.

•When building, Docker rejects links

Project Project2

Architecture : nginx single-project

Baseimage1404

Web-php55

Symfony1.5

Phusion/ubuntu1404

Phusion/ubuntu1204

Baseimage1204

Web-php53

Symfony1.0

Ressources sharing between close images

The only way I found to share ressources (factorize) was to build « variants » of theDockerfile in a subdirectory and temporarilyplace them in the main folder (would be niceto be able to specify the Dockerfile name).

#!/bin/shSCRIPT_DIR=`readlink -f $(dirname $0)`DOCKERFILE="$SCRIPT_DIR/../../Dockerfile"

if [ -f $DOCKERFILE ]; then mv $DOCKERFILE "$DOCKERFILE.bak" cp "$SCRIPT_DIR/Dockerfile" $DOCKERFILE docker build -t tribalnova/baseimage-ubuntu1204 "$SCRIPT_DIR/../.." mv "$DOCKERFILE.bak" $DOCKERFILEfi

baseimage build.sh

Project Project2

Architecture : nginx single-project test procedure

Project

Run with env var UPDATE=install

Commit Project-stable (project with sources)

Build

Run test with env var UPDATE=install-dev

Karma Run karma tests

Run jMeter consistency tests

Tests OK?

End

Push project-stableyep

nope

Conventions

Our needs have led to a few conventions:

Scripts naming convention:install-xxx.shScript meant to be executed on build

startup-xxx.shScript meant to be executed at runtime

Currently, every added ressource is put into /var/docker

For ressources other than scripts, if possible, use directories to sort them by feature/role.

Directory convention:

Image-name• .dockerignore• build.sh• test.sh• DockerfileFiles

• Install-xxx.sh• Startup-xxx.sh• Other ressources (in subdirs)

VariantsImage-variant-name

• .dockerignore• build.sh• test.sh• Dockerfile

Won’t be necessary if the Dockerfile may be specified

Vagrant

Providers/provisioners to build a virtual machine

AWS

Vagrant

Virtual Box VMWare

Chef + recipes

Puppet + modules

Shell scripts

Providers

Provisioners

Docker + Vagrant

Docker can be used as a vagrant provider and as a vagrant provisioner.

Even if seemingly opposed, somewhat outcomes to the same…

AWS

Vagrant

Virtual Box VMWare

Chef + recipes

Puppet + modules

Shell scripts

Providers

Provisioners

Docker

Docker

Docker + Vagrant provisioner

Deploys a virtual machine with a pre-installed Docker

See https://docs.vagrantup.com/v2/provisioning/index.html

Advantages:•One command line to deploy a VM holding a full web development environment: vagrant up

•Shared configuration (Vagrantfile)•Vagrant configuration features: SSH, port forwarding, ressources allocation…

•Can be completed with other vagrant providersVagrant.configure("2") do |config| config.vm.box = "ubuntu/trusty64" config.vm.provision "docker"end Minimum Vagrantfile

Docker + Vagrant provisioner

Vagrant.configure("2") do |config| config.vm.host_name = "docker" config.vm.box = "ubuntu/trusty64"

config.vm.provider "virtualbox" do |v| v.memory = 2048 v.cpus = 2 end

config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests" puppet.module_path = "puppet/modules" puppet.manifest_file = "site.pp" puppet.options = "--verbose" end

config.vm.synced_folder "./docker", "/docker" config.vm.synced_folder "../..", "/var/www/gma"

config.vm.provision "docker" do |d| d.build_image "-t tribalnova/web-php55 /docker/web" end

config.vm.network :forwarded_port, guest:80, host:80end

Vagrantfile sample

Docker + Vagrant provider

Launches containers directly under Linux, or runs lightweight boot2docker VMs to emulate them under Mac OS or Windows.

See http://docs.vagrantup.com/v2/docker/basics.html

Usage:•To start a vagrant provider script: vagrant up –provider=docker•To see logs: vagrant docker-logs•To run a command: vagrant docker-run

Docker + Vagrant provider

Vagrant.configure("2") do |config| config.vm.define "mysql" do |a| a.vm.provider "docker" do |d| d.name = "mysql" d.image = "mysql:5.7.5" d.ports = ["3306:3306"] d.env = {"MYSQL_ROOT_PASSWORD" => "root"} end end config.vm.define "jenkins" do |a| a.vm.provider "docker" do |d| d.name = "jenkins" d.build_dir = "." d.ports = ["8081:80"] end endend

Vagrantfile sample

Docker + Vagrant provider

The vagrant provider has some drawbacks under Mac OS X and Windows:

It uses boot2docker images, that don’t allow ports to be forwarded when using the ports parameter (may change in the future). To correct this behaviour, Vagrantfile descriptors are to be designed for each machine that need such a configuration, a somewhat heavy overload for something designed to be run easily and independently of the host OS.

There are vagrant projects designed to fix this issue, but the Vagrantfile configuration gets heavier:

https://vagrantcloud.com/yungsang/boxes/boot2dockerhttps://vagrantcloud.com/parallels/boxes/boot2docker

Docker + Vagrant provider

Vagrant.configure("2") do |config| config.vm.define "jenkins" do |a| a.vm.provider "docker" do |d| d.name = "jenkins" d.build_dir = "." d.ports = ["8081:80"] d.vagrant_vagrantfile = "./Vagrantfile.jenkins" d.vagrant_machine = "dockerhost" end endend

Vagrantfile

Vagrant.configure("2") do |config| config.vm.provision "docker" # The following line terminates all ssh connections. Therefore # Vagrant will be forced to reconnect. # That's a workaround to have the docker command in the PATH config.vm.provision "shell", inline: "ps aux | grep 'sshd:' | awk '{print $2}' | xargs kill" config.vm.define "dockerhost" config.vm.box = "ubuntu/trusty64" config.vm.network "forwarded_port", guest: 8081, host: 8081 config.vm.provider :virtualbox do |vb| vb.name = "dockerhost" endend

Vagrantfile

Docker + Vagrant + PHPStorm

Docker as a vagrant provisioner

PHPStorm has a Vagrant plugin that allows to execute its basic actions. The « start SSH session… » command offers the option to connect on the Vagrant VM.

Docker + other tools

•Swarm: docker clustering https://github.com/docker/swarm•Fig: docker groups http://www.fig.sh•Ansible: orchestration http://www.ansible.com•Serf: orchestration, clustering https://www.serfdom.io•Etcd: orchestration, service discovery https://github.com/coreos/etcd•Zookeeper: orcherstration http://zookeeper.apache.org•HAProxy: load balancer http://www.haproxy.org

Hints: no GUI during build

• To automatically confirm the package installationapt-get install –y package

• To disable console interactive GUIs when installing a package (default choices)DEBIAN_FRONTEND=noninteractiveIn the Dockerfile: ENV DEBIAN_FRONTEND noninteractive

• To let composer make default choicescomposer install --no-interaction

• To pre-answer to interactive questionsecho "xxx yyy"|debconf-set-selections

• To reactivate console interactive GUIs when logged in:echo 'export DEBIAN_FRONTEND=dialog' >> /etc/profileecho 'export TERM=linux' >> /etc/profile

Hints: port forwarding

To help reaching linked container, it is often convenient to forward local network communication towards this container. There are many tools for that, I use socat to avoid struggling with iptables.

If a container is linked, our images redirect localhost:<port> towards the corresponding container.

MySQL:if [ ! -z $MYSQL_PORT_3306_TCP_ADDR ]; then mkdir -p /var/run/mysqld socat UNIX-LISTEN:/var/run/mysqld/mysqld.sock,fork,reuseaddr,unlink-early,user=root,group=root,mode=777 TCP:mysql:3306 &fi

Memcache:if [ ! -z $MEMCACHED_PORT_11211_TCP_ADDR ]; then socat TCP-LISTEN:11211,fork,reuseaddr,user=root,group=root,mode=777 TCP:memcached:11211 &fi

Graylog:if [ ! -z $GRAYLOG_PORT_12201_UDP_ADDR ]; then socat UDP4-RECVFROM:12201,fork UDP4-SENDTO:graylog:12201 &fi

Hints: Github/Bitbucket API limit

Github and Bitbucket have API limits that disallow to load too much packages when not identified. Project builds may easily hit these limits. To overcome it when using composer, choose to load packages form source.

composer --dev install --prefer-source

Note that the –prefer-source option also include the git metadata.

To disable the git SSH warnings, create a file ssh_config in /etc/ssh, containing:StrictHostKeyChecking no

Hints: Phusion insecure SSH

Enabling SSH may be useful in development, as it allows to run commands directly on the container, ignoring the intermediate virtual machine. The IDE for instance may run remote PHPUnit tests and gather its results directly through SSH.

To easily enable it on the container, create a script:

Call it from a container startup script if the environment variable SSH_INSECURE is set:if [ -e /var/docker/startup-ssh-insecure.sh ] && [ ! -z $SSH_INSECURE ] && [ $SSH_INSECURE -eq 1 ]; then /var/docker/startup-ssh-insecure.sh && rm /var/docker/startup-ssh-insecure.shfi

Keep in mind that the Docker environment variables won’t be at disposal during an SSH session (so it also concerns the locale environment variables that should be set in a bash startup file).

#!/bin/sh

rm -f /etc/service/sshd/down/etc/my_init.d/00_regen_ssh_host_keys.sh/sbin/my_init --enable-insecure-key &

startup-ssh-insecure.sh

Hints: RUN

• RUN may be between brackets or not–RUN ["echo hop"]–RUN echo hop

• If no brackets, the script is executed with sh –c• The executed sh –c command doesn’t behave exactly as expected... The command seems to be somewhat quoted or escaped.

• To remind: the sh interpreter has a limited syntax (no accolade, some shell functions like echo have different/less functions)

Won’t display anythingRUN ["echo hop"]

Will display « hop »RUN echo hop

Won’t write as expected in a file

RUN echo foo && echo bar > test

Will write as expectedRUN (echo foo && echo bar) > test

Hints: ENTRYPOINT/CMD

• CMD is executed only if ENTRYPOINT is between brackets.• If ENTRYPOINT is between brackets, the file existence is checked with stats• Behaves differently of RUN• CMD/ENTRYPOINT are designed to accept only a single command

CMD is ignoredENTRYPOINT /var/docker/startup.sh lsCMD uname

Error: stat /var/docker/startup.sh ls: no such file or directoryENTRYPOINT ["/var/docker/startup.sh ls"]CMD uname

Only ls is executedENTRYPOINT ["/var/docker/startup.sh"]CMD ls && uname

Its worksENTRYPOINT ["/var/docker/startup.sh"]CMD ["apache2ctl -d /etc/apache2-f /etc/apache2/apache2.conf -e info-DFOREGROUND"]

Hints: explore an image

Docker images are always open (if their repository is available).To explore an image, for instance to observe the content of a file before it is removed during the build process:

• Search for a relevant version in the historydocker history test/anyimage:1.3.2

IMAGE CREATED CREATED BY SIZEc89094ec2bf4 4 days ago /bin/sh -c #(nop) COPY file:7978b485faf6af1cc 1.544 kB602e5ea7bbaf 4 days ago /bin/sh -c rm interesting-file 11 kB8419f2b0e486 4 days ago /bin/sh -c #(nop) CMD [mysqld] 0 B8e72632db815 4 days ago /bin/sh -c #(nop) EXPOSE map[3306/tcp:{}] 0 B7834ba645aeb 4 days ago /bin/sh -c #(nop) ENTRYPOINT [/entrypoint.sh] 0 B7d64a14cac57 4 days ago /bin/sh -c sed -Ei 's/^(bind-address|log)/#&/ 1.822 kB299556685f43 4 days ago /bin/sh -c { 206 MB

• Run bash on this version by bypassing any entrypoint. This method is also ideal to debug a container.docker run -ti –u root --entrypoint bash 8419f2b0e486

Hints: good practices

• A startup script (runtime) should be repeatable, as it may be executed more than once if the container is stopped and restarts:

–Don’t delete a line in a file, but comment it/uncomment it–Test if a file exists before calling it–Check and kill side running processes before launching them againpidof php5-fpm; if [ $? -ne 1 ]; then killall php5-fpm; fi

–etc…• The scripts used at build time don’t need to be repeatable, and it’s a good idea to suppress them, as they will be of no use:/var/docker/install-devtools.sh && rm /var/docker/install-devtools.sh• Do environment variable tests and use the error output in startup scripts:if [ ! -z $MONGODB_URL ]; then sed -i "s%'db.host'.*$%'db.host' => 'mongodb:\/\/$MONGODB_URL',%" /var/www/xhprof/config/config.phpelse >&2 echo "Undefined MongoDB URL, specify the host with '-e MONGODB_URL=1.2.3.4:27017'"fi• In the Dockerfile, keep the long operations (package installation) at the top of the file, as it will save time if the image needs to be rebuilt after a modification that comes after in the file

Conclusions so far

Despite its appearant simplicity, its intuitive interface and Dockerfile syntax, working with Docker revealed an higher complexity than expected.

Working with Docker mainly mixes the complexity of Linux scripting/commands with the Docker concepts and voluntary restrictions, sometimes turning the fact of thinking/deploying apps as containers into a challenge:• running background apps as foreground apps• isolating interdependent apps not designed for that, communication via socket/volumes• environment variable parameters to influence shell scripts/alter configuration• security parameters, SSH key handling, API limits, …• and so on…

However containerizing applications is often a great step towards scalibility (the remaining is a matter of service discovery).

Docker is a recent and popular tool, growing fastly. Its features are going to expand, and its restrictions might get leveraged in the future. We should keep a close eye on its upcoming evolutions, side tools and potential forks.

Conclusions so far

Use cases so far:

• Project development: not as flexible as using a VM (built by provisioning), but allows to be sure that the same configuration runs everywhere (packages, headers). Good if the developer just focuses on the sources, less interesting if the developer wants for instance to tweak his configuration or use custom tools. However the ephemeral nature of the container may still be a good point in such a case.

• Side development depending on the project: if the project is a pre-requisite to a development (an API), Docker is a good solution as it allows to deploy the complete stack quickly. Vagrant is recommended in this case.

• Continuous integration: using docker is a very good solution in this case, as it avoids dealing with the configuration of a development server, and allows to return a clean server after the operations have run.