28/1-14 23.24Things in Flow | Helping teams and individuals to get things done in flow

Page 1 of 2http://thingsinflow.dk/

Search …

Archives

January 2014

Meta

Log in

Things in Flow

THINGS IN FLOW APS · PARKVEJ 3 · 2830 VIRUM · CVR 35632506 · EMAIL: INFO@THINGSINFLOW.DK · THEME: MOTIF BY AUTOMATTIC.

HOME SAMPLE PAGE

1 COMMENT

Hello world!

12/01/2014

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

How to autodeploy your R solution to DigitalOcean using

Docker, GitHub and CircleCI

CopenhagenR - useR Group Meetup IT University of Copenhagen, Oct. 11 2016Presenter: Niels Ole Dam, Things in Flow

28/1-14 23.24Things in Flow | Helping teams and individuals to get things done in flow

Page 1 of 2http://thingsinflow.dk/

Search …

Archives

January 2014

Meta

Log in

Things in Flow

THINGS IN FLOW APS · PARKVEJ 3 · 2830 VIRUM · CVR 35632506 · EMAIL: INFO@THINGSINFLOW.DK · THEME: MOTIF BY AUTOMATTIC.

HOME SAMPLE PAGE

1 COMMENT

Hello world!

12/01/2014

Welcome to WordPress. This is your first post. Edit or delete it, then start blogging!

Things in Flow

About me

• Physics and Communications at Roskilde University

• Previously: • Project Manager at Experimentarium science center • Science Journalist at Illustrated Science magazine • Digital Business Developer at Bonnier Publications • Project Manager and Drupal Teamlead at Peytz & Co • Games Producer at Cape Cph

• Since 2014: • Independent consultant at Things in Flow

• I collect tools and help people get their stuff done digitally and with the right amount of efficient processes so that the work still feels right

2

Things in Flow

Coursera courses to recommend

• Data Science Specialisation by Johns Hopkins University • I did 8 of them a couple of years ago – they where great.

• Process Mining by Eindhoven University of Technology • I did this one too – and I’m actually an official Disco partner.

• By The Way: • I’m currently looking for a good Process Mining case. • Please contact me if you have that kind of log file data and

business itch that needs scratching.

3

Tip: All underlined links are clickable in the pdf-version of this slideshow – click to get more info on the subjects.

Things in Flow

Why I chose R

• Feel the raw data under my fingertips (again)

• Visualie raw data

• Deliver prototypes to colleagues/clients

• Deliver widely available solutions to clients

• Bonus: Test new tools without polluting my computer

4

Things in Flow

Why I chose Docker

• Production ready for teams and small companies

• Makes it possible to follow the Continuous Integration (CI) paradigme in my own production

• Open Source – dedication and low cost

• Docker with R is Awesome! :-)

5

Hang on! A lot of steps and things to grasp but easy to execute in practice

;-)

Things in Flow

Ingredients to shop for• GitHub account:

• Free if Open Source – otherwise starting at $7/mo

• Docker Hub account (optional): • Free if Open Source and only 1 private account, otherwise starting at $7/mo

• CircleCI: • Free if Linux and only 1 container – otherwise starting at $39-50/mo

• R + RStudio locally: • Free

• Cloud Server: • Variable, typically starting at $5/mo

• Total: • Starting with Free (Open Source) or $7/m (1 private project) + Cloud server

7

GitHub +

RStudio

Things in Flow

GitHub: How it works

• Clone code from GitHub

• Commit your own changes

• Git keeps track on additions and removals automatically

• Push your code back to GitHub

9

Source: betterexplained.com

Things in Flow

Using Git with RStudio

• Create a repo at GitHub

• Clone the repo to a local RStudio project

• Commit to GitHub regularly using RStudio

Use branching for trying out alternative solutions: • Open a terminal via the RStudio-link and run:

git checkout -b [branchname] • …do your stuff with local commits… • Switch to Master, open a terminal via the RStudio-link and run:

git merge --no-ff [branchname] • …the branch is now merged back into master with all it’s Git history…

10

Things in Flow

Demo 1: GitHub + RStudio

• Connect RStudio project with GitHub

• Make a RStudio Shiny App project

• Make some code changes

• Push changes to GitHub repo

• Optional: Make a local branch

11

Things in Flow

New GitHub repo

12

Copy and use for RStudio

Things in Flow

Create new RStudio project

13

Things in Flow

RStudio is now connected with GitHub

14

Docker

Things in Flow

Docker: How it works

16

Traditional Virtual Machines:

Full OS for each VM

Docker:

OS code is partly shared

Things in Flow

The ideal Docker workflow

17

Source: docker.com

Things in Flow

Another take on the same

18

Source: terranillius.com

Things in Flow

Using Docker

• Install and start Docker for Mac (or Windows)

• Verify that Docker is running

• Search for Docker images on Docker Hub • … either via http://dockerhub.com • …or via Kitematic (installed as part of Docker for Mac)

• Start your first Docker container i.e. hello-world-nginx

• You are now running Docker locally

19

Things in Flow

Demo 2: Deploy Shiny App on local VM

• Start Docker for Mac (or Windows)

• Verify that Docker is now running

• Add Dockerfile to project

• Add docker-compose.yml file to project

• Open terminal via RStudio-link

• Run: docker-compose up -d

• Shiny App is now running on local Docker VM

• …and can be accessed via http://localhost

20

Things in Flow

Docker cheatsheetdocker run <container> <command>

docker run -v LOCAL_DIR:CONTAINER_DIR <container> <command>

docker exec -it <container_id> <command>

docker images

docker logs <container_id>

docker-compose up -d --build

docker-compose start

docker rmi [imageID]

docker ps [-a]

docker run ubuntu /bin/echo 'Hello world'

21

Things in Flow

Manage data in containers

• If you have some persistent data that you want to share between containers, or want to use from non-persistent containers, it’s best to create a named Data Volume Container, and then to mount the data from it.(from docker.com)

22

Things in Flow

Using Compose is basically a three-step process

1. Define your app’s environment with a Dockerfile so it can be reproduced anywhere.

2. Define the services that make up your app in docker-compose.yml so they can be run together in an isolated environment.

3. Lastly, run docker-compose up and Compose will start and run your entire app.(from docker.com)

23

Things in Flow

Compose can mange the whole lifecycle

• Start, stop and rebuild services (start, stop, up, down)

• View the status of running services

• Stream the log output of running service (logs)

• Run a one-off command on a service (run)(from docker.com)

24

Things in Flow

Volume data preserved when containers are created

• Compose preserves all volumes used by your services. When docker-compose up runs, if it finds any containers from previous runs, it copies the volumes from the old container to the new container. This process ensures that any data you’ve created in volumes isn’t lost.(from docker.com)

25

Things in Flow

Moving Compose files between environments

• Compose supports variables in the Compose file. You can use these variables to customize your composition for different environments, or different users. See Variable substitution for more details.

• You can extend a Compose file using the extends field or by creating multiple Compose files. See extends for more details.(from docker.com)

26

From Local to Cloud

Things in Flow

Cloud Deploy: How it works

28

Source: circleci.com

Things in Flow

The Workflow

29

bit.ly/R-Docker-workflow

R-Docker Workflow for teams and small companies

Loca

lLo

cal V

M (D

ocke

r)C

loud

(Circ

leC

I/Dig

italO

cean

)

v1, Niels Ole Dam, 5/10-2016

Start

Client

GitHub, Docker Hub and Cloud Server (i.e. DigitalOcean, Amazon, Azure etc.)

http://thingsinflow.dk/

Show/Hide Messages

Develop R solution + test locally

Test passed?

NoTest/fix on local VM

Test passed?

NoSetup initial deploy to

Cloud

Deploy to local VM

Test/iterate with client

Test passed?

No

Initial solution

deployed

Build on local VM

docker-compose up

Build on Cloud server

Develop, test, verify and deploy

docker-compose up

1. 2. 3.

Setup auto deploy on CircleCI Solution

updated4.

Switch prod server

5.

6.

7.

GitHub Docker CircleCIRstudio Project

Git branch

bit.ly/R-Docker-workflow = clickabel version of this document

Docker Cheat Sheet:- docker run <container> <command>- docker run -v

LOCAL_DIR:CONTAINER_DIR <container> <command>

- docker exec -it <container_id> <command>- docker images- docker logs <container_id>- docker-compose up -d- docker-compose start- docker rmi [imageID]- docker ps [-a]- docker run ubuntu /bin/echo 'Hello world'

Request solution

Request change

Try out new frameworks on local VM

8.

Test php/

Spark?

Click on the blue numbers or doc icons for more info on that step/subject.

CircleCI

(or Jenkins or TravisCI etc.)

Things in Flow

CircleCI: How it works

• Connects to GitHub

• Listens for any changes on a specific GitHub repo

• Looks for a circle.yml script in the root of the GitHub repo

• Runs the circle.yml script every time the repo changes

• Can be used for automatical running of: • …code tests • …docker image builds for Docker Hub • …deployment(s) to Cloud server(s)

31

Things in Flow

Demo 3-4: Setup CircleCI for auto deploy

• CircleCI connects to GitHub and Cloud via environment variables (=secure because credentials are not shown in code)

• Environment variables could be: • $DOCKER_USER • $DOCKER_PASS • $DEPLOY_TOKEN • $DOCKER_EMAIL

• Test that the circle.yml script is working as planned • Now everything is set up for auto deploy on push to GitHub!

32

Things in Flow

Known issues with CircleCI

• Docker 1.12 is currently NOT supported by CircleCI

• Docker 1.10 is currently NOT installed out-of-the-box by CircleCI! (=compose must be installed explicitly)

• But should not be a too big issue

• …if it is, then use TravisCI (or own Jenkins etc.) instead

33

Things in Flow

Demo 5-6: Auto deploy to DigitalOcean server

• Make a code change to the RStudio project

• Commit and push to Master (GitHub)

• Wait 10 min – you’ll also get a mail

• A new version of the cloud server has now been build on DigitalOcean with a random IP

34

Things in Flow

Demo 7: Switch prod server and setup A record

• New cloud server has now been build

• Log into DigitalOcean controlpanel

• Switch the floating IP address from old to new server

• Requires: A subdomain (i.e. http://meetup.thingsinflow.dk) has been setup on your website (A Record)

35

Things in Flow

Docker Swarm Mode: How it works

• 30 sec demo:

• www.johnzaccone.io/3-node-cluster-in-30-seconds/

• Using docker-compose with docker swarm • https://docs.docker.com/compose/swarm/

• Swarm mode overview: • https://docs.docker.com/engine/swarm/

36

Bonus: Docker for local try outs

Things in Flow

Try out new frameworks safely

• Test out new frameworks in a local VM without: • …risking the health of your main computer • …messing with the initial configuration • …thinking too much about documenting your steps

• Quickly test out new integrations

38

Things in Flow

Demo 8: Try out some images

• hello-world-nginx

• Jenkins

• php

• Java

• MySQL

• Spark

• Hadoop

• …

39

Summing Things Up

Things in Flow

Summing up: Steps needed

• Connect RStudio to GitHub – see this link at Preferences -> Git/SVN

• Make your Shiny App

• Add docker-compose.yml and/or Dockerfile

• Test Docker setup locally

• Connect GitHub to Docker Hub

• Connect Docker Hub to CircleCI (or other CI service)

• Connect CircelCI to a cloud server (i.e. Amazon, DigitalOcean)

• Push to GitHub and watch the magic unfold

Tips: • Dockerfile and circle.yml: Use one-liners or shell script • Beware of absolute vs. relative paths in all scripts

41

Things in Flow

Stuff not touched today

• Partly update of individual containers on cloud server via GitHub and CircleCI (i.e. only on container at a time)

• Sharing of volumes between Docker container and local machine

• How to add a nice html frontend page (index.html) ;-)

42

Things in Flow

Key Take Aways

• Docker is production ready for teams and small companies

• Docker makes it possible to use the Continuous Integration (CI) paradigme for your R projects

• Docker with R is Awesome! :-)

43

Things in Flow

Links• Getting started with Docker:

• http://www.docker.com/products/overview • https://docs.docker.com/engine/getstarted/

• Manage Data in containers: • https://docs.docker.com/engine/tutorials/dockervolumes/#/creating-

and-mounting-a-data-volume-container • Docker Compose:

• https://docs.docker.com/compose/overview/ • Automated Docker Deployments with CircleCI:

• http://www.mikeheijmans.com/docker/2015/09/07/automated-docker-deployment-with-circle/

• Continuous Integration and Delivery with Docker-CircleCI: • https://circleci.com/docs/docker/

44

Things in Flow

Questions?

Things in Flow

Niels Ole Dam – nd@thingsinflow.dk thingsinflow.dk – Mob. +45 2712 9314

Thanks! :-)CopenhagenR - useR Group, Meetup at ITU, Oct. 11 2016

This presentation can be found at thingsinflow.dk/R-Docker-workflow