Building containers

Go-TorontoNov/2016

@lxfontes

#bfcm

Continuous Delivery Pipeline 1-1

BuildGit Push Test Deploy

Your company keeps growing

Growing

Growing MOAR...

Clone repository

docker build

- Install RubyGems

- Install NPM Packages

- Compile static assets

- Seal Docker image

docker push

- Upload to Docker Registry

Building a Rails Container

InertiaYou get used to it

Your build time only goes up

Number of workers only goes up

“Our build is complex”

“We already use Docker, the Cloud. This is as fast as it gets!”

“It is automated, we don’t care it takes 30 minutes”

“The XYZ team should fix it”

it was a reality check20 minutes per build

Flakiness

Resource Starvation

Really expensive

* where do you put secrets??????

Why is it slow?

push webhook

worker bootstrap

docker build cache not reused between builds

cold asset precompilation

2nd build

peed

Where are we: Daily stats

+200Devs

+700Builds

+42kTests

per build

~3Images

per build

Commitment

Container

BuildGit Push

Automated

TestsDeploy

5s 90s 200s 300s

From Git Push to Deployed in ~10 minutes

Every developer can deploy to productionGoals:

Testing Rails at

Scale by Emil

How We Deploy

Shopify by KatThis right here

push 30 sec POLL

Pull / Build

Coordinator+ API

zookeeper

workers

registry

branch affinity

ohai bootstrap

Clone repository

docker build

- Install RubyGems

- Install NPM Packages

- Compile static assets

- Seal Docker image

docker push

- Upload to Docker Registry

Building a Rails Container

Dockerfile

mo layers, mo problems

* apparently docker 1.13can squash these

Look ma, no Dockerfilesprepare

Install OS packages

precompile

Load (ejson) secrets

Populate artifact cache

compile

Combine app + artifact cache

Seal image

Daily seed 20min

Builds 2minInheritCache

InheritCache

Docker LayersBase

Ubuntu-ish

Prepare

OS packages

Intermediate (Daily seed)

App + Libraries

Final

Delta between Intermediate … Current

Locutus trade offsFAST

Local caches

Secure

Stable

But needs love

We need to maintain:

Infrastructure

Orchestration

Web UI

Scripts Copy & Pasta

Dockerfiles / Buildpacks :(

Troubleshooting :(

Pipa* Kite in Portuguese(and sounds cool)

Provider agnostic

Disposable

Secure

Sandboxed Docker Daemon (dind)

Pre-made recipes

Buildpack

Dockerfiles

Locutus Assembly

Pipa: RequirementsBuild whatever you want

Cache whatever you want

Repeatable

Run locally if needed

As fast as Locutus

Is there an app fo dat?Orchestration / UI

Docker first

On-premise

Worker fully sandboxed

Concurrency Primitives

“Only 3 builds for app X at same time”

Parallelism

“Send job to 30 workers”

BuildkiteOn-premise

Orchestration *only*

Github Hooks

UI / Reporting

API

Agent is a single binary (Go)

Also distributed as docker container

Concurrency Primitives

Parallelism

Per Build artifacts

Per Build shared key-value store

Switch workers!!

No batteries included … on purpose

Buildkite Pipeline Sample - buildkite/sample-pipelines

Waits for previous stepto complete

Requires InteractionUI/API/slack

Branch filtering

Can it scale?

BuildkiteGitHub BK PipaDocker

RegistryBK Tests

Webhook Build Container

Switch Worker

Start Tests

Fetch Image

Tests DoneShip It!

Tests

Upload Image

100 cores 8820 coresPipa: Goal

Pipa: Architecture goalsEverything exposed via environment variables

PIPA_SSH_KEY / PIPA_APP_COMMIT_ID

Chainable (simplify IF/ELSE ops)pipa build --unless v1 -- docker build -t myimage:v1 .

Configurable via environment, config file, argument switchesPIPA_APP_NAME=myapp pipa build ...

pipa build --app-name myapp …

Isolated Docker Daemon (docker-in-docker)Reset after each buildKubernetes: external SSD / Local: give it a directory

Environment / Process Tree

pipa

wrapper

AWS / GCS / SSH Keys

Kubernetes secretsenv vars with paths

APP Name

Buildkite / Jenkins / etcGIT (local)

APP SHA

Buildkite / Jenkins / etcGIT (local)

Your code

Signals

Prototype: I’m going to write this all in bash

Single Binary

Shell files as Assets

using go-bindata

docker / aws-cli / rsync

LOL

BUT, normalized environment is solid!

Provider agnostic

Disposable

Secure

Sandboxed Docker Daemon (dind)

Pre-made recipes

Buildpack

Dockerfiles

Locutus Assembly

Pipa: RequirementsBuild whatever you want

Cache whatever you want

Repeatable

Run locally if needed

As fast as Locutus

Toolbox: spf13/cobraCLI handlingdocker, kubernetes, rkt, etcd

ABUSE: func init()

Toolbox: kelseyhightower/envconfig *spf13/cobra deficiency

Fills struct with env vars

Why both?

spf13/cobra

kelseyhightower/envconfig

Result

Google Cloud Storage

Amazon S3

JSON Asymmetric Encryption

Shopify/ejson

Registry Interaction

push / pull / exists

mmap diff 2 dirs

burke/treediff

Environment normalizer (entrypoint)

Using itEverything is namespaced!

pipa cache pull /tmp/artifacts

pipa build --if base -- docker build -t $PIPA_IMAGE_FULL_NAME .

pipa image exists --local --tag basepipa image exists --remote --tag base

Downloads s3://artifacts/app_group/app_env/app_name/cache.tar to /tmp/artifacts

Only runs command if registry/app_group/app_env/app_name:base exists

Checks if registry/app_group/app_env/app_name:base exists (locally/remotely)

Pipeline InferenceHow should pipa build your app?

Customize it in your repo; or

Select a template; or

Let pipa figure it out!

Built-in pipelines

Pipeline Output

PipelineParallelization

3 parallel builds

Pipeline SelectionCustom script in repo

Custom pipeline in repo

Default pipeline in repo

User requested specific pipeline

Check for common file locationsrepo/Dockerfile? -> dockerrepo/borg? -> borg

Default to Heroku Buildpack

PIPA_PIPELINE_CMD

PIPA_PIPELINE_FILE

PIPA_PIPELINE_TEMPLATE

repo/.buildkite/pipeline.yml

High Level

push webhook

Agent pollSSL

dockerimages

cacheartifacts

QueueAPI

User Interface

ssh-keygcloud-keybuildkite-tokendocker-auth

Sec: Kubernetes Layout

EJSONpriv/pub keypairs

/builder-secrets

/app-secrets

/var/lib/docker (dind)

/builds (git clone)

/cache (artifacts)

6x nodes

24 pods

/mnt/disks/ssd0/builder1/docker

Provider agnostic

Disposable

Secure

Sandboxed Docker Daemon (dind)

Pre-made recipes

Buildpack

Dockerfiles

Locutus Assembly

Pipa: RequirementsBuild whatever you want

Cache whatever you want

Repeatable

Run locally if needed

As fast as Locutus?

Gems

Tar + Gzip

Go + Docker = fsouza/go-dockerclient

Respect environment variables

Reads $HOME/.docker/config.json or $HOME/.dockercfg

Rocker - grammarly/rockerDockerfile on steroids

Herokuish - gliderlabs/herokuishIf you like Heroku, you will want this

Packaged as docker container or single binary

alpinelinux.org

based on musl libc and busybox

Lots of *up-to-date* packages (yes debian, looking at you)

Demo Time