Docker in Production: Reality, Not Hype
-
Upload
bridgetkromhout -
Category
Technology
-
view
579 -
download
1
Transcript of Docker in Production: Reality, Not Hype
![Page 1: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/1.jpg)
@bridgetkromhout
Docker in Production Reality, Not Hype
Bridget Kromhout
![Page 2: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/2.jpg)
@bridgetkromhout
Bridget Kromhout
Operations Engineer @DramaFeverMinneapolis, Minnesota
@devopsdays@devopsdaysMSP
@arresteddevops
bridgetkromhout.com
![Page 3: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/3.jpg)
@bridgetkromhout
K-dramas since 2009. Docker in prod since October 2013.
![Page 4: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/4.jpg)
@bridgetkromhout
Why Docker?
consistent development repeatable deployment
![Page 5: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/5.jpg)
How?
not: a tutorial
but: repeatable
@bridgetkromhout
![Page 6: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/6.jpg)
private registry: the official party line
https://github.com/docker/docker-registry#quick-start
S3 is a storage engine optionbut… a central registry server didn’t scale well for us
@bridgetkromhout
![Page 7: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/7.jpg)
private registry: dramafever
@bridgetkromhout
# this goes in /etc/default/docker to control
docker's upstart config
DOCKER_OPTS="--graph=/mnt/docker --insecure-
registry=local-repo-alias.com:5000"
● local-repo-alias.com in DNS with A record to 127.0.0.1● OS X /etc/hosts: use the boot2docker host-only network IP
![Page 8: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/8.jpg)
registry upstartdocker pull public_registry_image
docker run -p 5000:5000 --name registry \
-v /etc/docker-reg:/registry-conf \
-e DOCKER_REGISTRY_CONFIG=/registry-conf/config.yml \
public_registry_image
@bridgetkromhout
![Page 9: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/9.jpg)
config.yml s3_region: us-east-1 s3_access_key: <aws-accesskey> s3_secret_key: <aws-secretkey> s3_bucket: <bucketname> standalone: true
@bridgetkromhout
![Page 10: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/10.jpg)
what even is flate?!Pulling repository local-repo-alias.com:5000/www4dda2b433370: Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671 flate: corrupt input before offset 54393671d497ad3926c8: Error downloading dependent layers2014/12/07 02:34:54 Error pulling image (prod) from local-repo-alias.com:5000/www, flate: corrupt input before offset 54393671
@bridgetkromhout
![Page 11: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/11.jpg)
registry rewrite coming!
DOCKER_OPTS="--graph=/mnt/docker --insecure-registry=local-repo-alias.com:5000 -e STORAGE_REDIRECT=true"
...until we get to the promised go lan(d|g), there’s a workaround for the flate errors we’re seeing:
@bridgetkromhout
![Page 12: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/12.jpg)
Achievement unlocked:distributed privateDocker registry
@bridgetkromhout
![Page 13: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/13.jpg)
@bridgetkromhout
Next up:build pipeline
starringeveryone’s
favorite butler
![Page 14: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/14.jpg)
weekly base builds
FROM local-repo-alias.com:5000/www-base
● include infrequently-changing dependencies○ ubuntu packages○ pip requirements○ wheels
● other builds can start from these images (so they’re faster):
@bridgetkromhout
![Page 15: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/15.jpg)
sudo docker build -t="a12fbdc" .
sudo docker run -i -t -w /var/www -e DJANGO_TEST=1 --
name test.a12fbdc a12fbdc py.test -s
sudo docker tag a12fbdc local-repo-alias.com:
5000/www:'dev'
sudo docker push local-repo-alias.com:5000/www:'dev'
@bridgetkromhout
www-master build
![Page 16: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/16.jpg)
2014/10/30 21:35:31 Error getting container init rootfs b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f3108f86386aab67 from driver devicemapper: open /dev/mapper/docker-9:127-14024705-b528d54a0458a8cd8a798309930adb45cb5e1a7430e981e0f3108f86386aab67-init: no such file or directorymake: *** [build-django] Error 1Build step 'Execute shell' marked build as failure
@bridgetkromhout
breaking builds
![Page 17: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/17.jpg)
https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin
@bridgetkromhout
![Page 18: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/18.jpg)
@bridgetkromhout
Retry the build…...only if a specific regex appears
![Page 19: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/19.jpg)
@bridgetkromhout
useful for unattended base builds
need to change how it reports to Slack
![Page 20: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/20.jpg)
@bridgetkromhout
tag for staging
tag for prodout of ELBrestart upstartback in ELB
Ship it!
![Page 21: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/21.jpg)
What about local development?
@bridgetkromhout
![Page 22: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/22.jpg)
before summer 2014Vagrant for local development
chef-solo provisioner
17 minutes to install everything
@bridgetkromhout
![Page 23: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/23.jpg)
now: boot2docker
devs pull down images built on jenkinsmysql image is built with fixturescan run master or qa image (or even prod)
can build new local images from Dockerfiles
@bridgetkromhout
![Page 24: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/24.jpg)
local registry for devdocker run -d -p 5000:5000 --name
docker-reg -v ${DFHOME}:${DFHOME} -e
DOCKER_REGISTRY_CONFIG=${DFHOME}
/config/docker-registry/config.yml
public_registry_image
@bridgetkromhout
![Page 25: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/25.jpg)
$ boot2docker ssh date -u
Mon Nov 24 16:09:02 UTC 2014
$ date -u
Tue Nov 25 01:43:49 UTC 2014
@bridgetkromhout
time is what turns kittens into cats
![Page 26: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/26.jpg)
S3 requires clock sync$ docker pull local-repo-alias.com:5000/mysqlPulling repository local-repo-alias.com:5000/mysql2014/11/24 19:44:31 HTTP code: 500
$ boot2docker ssh sudo date --set \"$(env TZ=UTC date '+%F %H:%M:%S')\"
@bridgetkromhout
![Page 27: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/27.jpg)
Devs can use their preferred editing environment:
-v ${DFHOME}/www:/var/www
We still want logs, too, so we expose those for the dev here:
-v ${DFHOME}/www/run:/var/log
volume mounting & our fork
@bridgetkromhout
Until 1.3 we ran a forked boot2dockerWe needed to mount local files into the VM
![Page 28: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/28.jpg)
containerizing front-enduseful for building front-end apps on Jenkinsalso allows consistent testing
RUN apt-get install -y nodejs nodejs-legacy npmRUN npm install -g [email protected] npm install -g [email protected] npm install -g [email protected] bower.json /var/www/dependencies/bower.jsonRUN cd /var/www/dependencies && bower install --allow-root --config.interactive=false --force
@bridgetkromhout
![Page 29: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/29.jpg)
@bridgetkromhout
django: image: local-repo-alias.com:5000/www:dev ports: - "8000:8000" links: - mysql - redis environment: - PYTHONPATH=/var/local - DJANGO_ENVIRON=LOCAL - DB_PORT_3306_TCP_ADDR=mysql command: /var/local/config/local/start-django-local volumes: - ${DFHOME}/www/run:/var/log - ${DFHOME}/www:/var/local
mysql: image: local-repo-alias.com:5000/mysql:dev expose: - "3306:3306"
![Page 30: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/30.jpg)
for persistent instances# remove stopped containers
@daily docker rm `docker ps -aq`
# remove images tagged "none"
@daily docker rmi `sudo docker images | grep none
| awk -F' +' '{print $3}'`
@bridgetkromhout
![Page 31: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/31.jpg)
failure modes
cron zombiesout of memory errors
race conditions
@bridgetkromhout
![Page 32: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/32.jpg)
what isolation?-v /var/log/containers:/var/log
@bridgetkromhout
Host instances moving into
![Page 33: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/33.jpg)
through a container darkly: monitoring
@bridgetkromhout
![Page 34: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/34.jpg)
containers building (lighter) containers
easier with statically linked binaries
go microservicesandroid apk
@bridgetkromhout
![Page 35: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/35.jpg)
$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
local-repo-alias.com:5000/mysql dev b0dc5885f767 2 days ago 905.9 MB
local-repo-alias.com:5000/www dev 82cda604a4f1 2 days ago 1.092 GB
local-repo-alias.com:5000/micro local bed20dc84ea1 4 days ago 10.08 MB
google/golang 1.3 e3934c44b8e4 2 weeks ago 514.3 MB
public_registry_image 0.6.9 11299d377a9e 6 months ago 454.5 MB
scratch latest 511136ea3c5a 18 months ago 0 B
$
ever-smaller images
@bridgetkromhout
![Page 36: Docker in Production: Reality, Not Hype](https://reader030.fdocuments.in/reader030/viewer/2022032616/55a779691a28ab580a8b4909/html5/thumbnails/36.jpg)
@bridgetkromhout
www.dramafever.com/company/careers.html
Thank you!(and we’re hiring!)