Shipping Manifests, Bill of Lading and Docker Metadata and Container

Gareth RushgroveSenior Software Engineer, Puppet Labs

@garethr

Senior Engineer at Puppet Labs

Creator of the Puppet Docker module

This TalkThe introduction

Shipping containers are cool

But nothing without all the paper work

A manifest or ship's manifest is a document listing the cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials.

A bill of lading is a document issued by a carrier which details a shipment of merchandise and gives title of that shipment to a specified party.

State of the Software Supply Chain

—State of the Software Supply Chain 2015

A once safe component may be found to be vulnerable at any time”

“

Vulnerable or defective components… end up in our software largely unnoticed”

—State of the Software Supply Chain 2015

“

Lets apply the same principles to a different type of container

Docker LabelsDocker builtin metadata capabilities

Added in 1.6One meta data to rule them all

Labels on Docker Engines

Provide information about the host

$ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"

Labels to guide Swarm scheduling

$ docker run -d -P \ -e constraint:storage==ssd --name db mysql

Labels on Docker images

LABEL [<namespace>.]<key>[=<value>] ...

Dockerfile Label instruction

Don’t do this - new layer per label

LABEL vendor=ACME\ IncorporatedLABEL com.example.version.is-betaLABEL com.example.version="0.0.1-beta"LABEL com.example.release-date="2015-02-12"

Better - only one layer

LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"

$ docker inspect 4fa6e0f0c678..."Labels": { "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12"}... Access labels via inspect

Containers inherit labels from the image

Add additional labels to containers

$ docker run \ -d \ --label com.example.group="webservers" \ --label com.example.environment="production" \ busybox \ top

Query based on labels with filters

Filter images by label

$ docker images --filter "label=com.example.is-beta"

Filter containers by label

$ docker ps --filter "label=com.example.is-beta"

Added in v1.17 Lots of label support in the API

But what to store in Labels?

A quick asidePackage Managers

Post from 2011I like system packages

The power of system packages lies not in the file format but in the metadata

DPKG and RPM

Debian New Maintainers’ Guide

Fedora packaging guidelines

Summary: A CD player app that rocks!Name: cdplayerVersion: 1.0Release: 1Copyright: GPLGroup: Applications/SoundSource: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer-1.0.tgzURL: http://www.gnomovision.com/cdplayer/cdplayer.htmlDistribution: WSS LinuxVendor: White Socks Software, Inc.Packager: Santa Claus <sclaus@northpole.com>

%descriptionIt slices! It dices! It's a CD player app thatcan't be beat. By using the resonant frequencyof the CD itself, it is able to simulate 20X

Example RPM spec file

Given standard metadata what can we do?

$ dpkg -L lynx/./usr/usr/share/usr/share/doc/usr/share/doc/lynx/usr/share/doc/lynx/copyright/usr/share/doc/lynx/changelog.gz/usr/share/doc/lynx/changelog.Debian.gzList files from package

What installed that file?

$ rpm -qf /usr/bin/mysqlaccessMySQL-client-3.23.57-1

Find unmet dependencies

$ apt-cache unmetPackage libdataobjects-sqlite3-ruby1.9.1 version 0.10.1.1-1 has an unmet dep: Depends: libdataobjects-ruby1.9

$ rpm -qdf /usr/bin/mysqlaccess/usr/share/man/man1/mysql.1.gz/usr/share/man/man1/mysqlaccess.1.gz/usr/share/man/man1/mysqladmin.1.gz/usr/share/man/man1/mysqldump.1.gz/usr/share/man/man1/mysqlshow.1.gz

Find documentation

$ apticronThe following packages are currently pending an upgrade:

xfree86-common 4.3.0.dfsg.1-14sarge3libice6 4.3.0.dfsg.1-14sarge3libsm6 4.3.0.dfsg.1-14sarge3xlibs-data 4.3.0.dfsg.1-14sarge3libx11-6 4.3.0.dfsg.1-14sarge3libxext6 4.3.0.dfsg.1-14sarge3libxpm4 4.3.0.dfsg.1-14sarge3Find outdated packages

StandardsThe power of agreement

Docker official label guidance

All (third-party) tools should prefix their keys with the reverse DNS notation of a domain controlled by the author. For example, com.example.some-label.

1

The com.docker.*, io.docker.* and org.dockerproject.* namespaces are reserved for Docker’s internal use. 2

Keys should only consist of lower-cased alphanumeric characters, dots and dashes (for example, [a-z0-9-.]). 3

Keys should start and end with an alpha numeric character.

4

Keys may not contain consecutive dots or dashes.

5

Keys without namespace (dots) are reserved for CLI use.

6

How widely adhered to?

< 20% from a small sample

But some folks care

Unify the labels for Docker images

Merged

The problem with inconsistent metadata

Without complete metadata we can’t trust the tools built on top

Docker Label Inspector

Check against Docker guidelines

$ dli lint========> Check all labels have namespaces [WARN] Label 'vendor' should use a namespace based on reverse DNS notation========> Check labels don't use reserved namespaces========> Check labels only use valid characters========> Check labels start and end with alpanumeric characters========> Check labels for double dots and dashes

$ dli validate========> Check labels based on schema in 'schema.json' [ERROR] u'com.example.is-beta' is a required property

Check against a schema

{ "title": "Dockerfile schema", "type": "object", "properties": { "com.example.release-date": { "type": "string" }, "com.example.is-beta": { "type": "string" }, "com.example.version": { "description": "Version", "type": "integer", "minimum": 0 } }, "required": ["com.example.is-beta", "com.example.version"]}

Define labels in JSON schema

Stand well backLIVE DEMO

Runtime MetadataA missing piece, and some ideas

What temperature is a refrigerated shipping containers at?*Not all shipping containers are refrigerated

The need for runtime metadata

docker exec as an API

Dockerfile example

FROM alpineLABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv"RUN apk add --update bash && rm -rf /var/cache/apk/*COPY Dockerfile /

Discover out API

$ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq{ "net.morethanseven.dockerfile": "/Dockerfile", "net.morethanseven.exec.packages": "apk info -vv"}

Read the Dockerfile

$ docker run -i -t garethr/alpine cat /DockerfileFROM alpineLABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv"RUN apk add --update bash && rm -rf /var/cache/apk/*COPY Dockerfile /

$ docker run -i -t garethr/alpine apk info -vvmusl-1.1.11-r2 - the musl c library (libc) implementationbusybox-1.23.2-r0 - Size optimized toolbox of many common UNIX utilitiesalpine-baselayout-2.3.2-r0 - Alpine base dir structure and init scriptsopenrc-0.15.1-r3 - OpenRC manages the services, startup and shutdown of a hostalpine-conf-3.2.1-r6 - Alpine configuration management scripts

List installed packages

Second time luckyLIVE DEMO

R.I.Pienaar

ToolingWhat could we build atop our metadata?

Documentation discovery

License verification

Package search

Tempting fateLIVE DEMO

ConclusionsIf all you remember is…

Step 1 Step 2 Step 3

Metadata! Something… Profit

Share schemas and namespaces

Build agreement

Build tooling

Extract standards

Thank you!Gareth Rushgrove@garethr gareth@puppetlabs.com