Continuous Integration & Continuous Delivery with OpenSource Tools

Annotated Version

• The grey slides are additional slides added to the presentation to make it easier to understand on Slideshare etc.

@kaktusmimi lihsmi.ch

Inspired by…

Jo

Dani

Paddy

Seb

Daniel

Christiane

Inspiration

• This talk was inspired a lot by the work of my colleagues at punkt.de

• And a presentation of Sebastian Helzle at FOSSASIA 2014

• http://www.slideshare.net/Sebobo/continuous-delivery-with-open-source-tools

Motivation

TestsVersion Control

JenkinsDeploymentConfiguration Management

Monitoring

Summary & Further Reading

Server

Developer

Delivery

Motivation

• We want to deliver our Software from Development to Production

• In small increments

Feedback

Motivation

• As a developer we want to improve constantly

• Build - Measure - Learn is a Mantra for Learning

• Feedback supports Learning

• Continuous Integration and Delivery is a constant source of Feedback

• The earlier the Feedback - the better

FTP to live Server

Live Server

FTP to test Server

Test Server

FTP to live Server

FTP to test Server

Test Server

FTP to live Server

All developers work onsingle Dev Server

instance

FTP to live Server

Dev Server

Live Server

Version Control Build

gitlab triggers

Unit TestsCommit Stage

build succeeds

UI TestsAcceptance Stage

DB Tests

tests pass

tests pass

Deployment Stage

tests pass

build succeeds

Build locally

Demo Stage

Production Stage

rsync

rsync

git push

Developer

Feedback

Evolution of Software Delivery

• One developer ships his software to live server using FTP

• We start to use a Test Server to protect the production server

• Multiple developers make the scenario more complex

• Introduction of a developer server

• Setup of a sophisticated deployment chain

① Git

First Step Towards Continuous Integration

if you don’t use it yet

USE GIT

Gitlab Server

git pull / push

Developer

Reviewer

Merge / DeclineFeature Branch

Master Branch

Master Feature

git checkout -b feature��

�

pull� !

Merge Request Workflow

• No developer is allowed to directly push to master

• Each developer opens a merge request for a finished feature

• Any other developer can review the request and merge it into master

• Provides a first feedback loop!

② Tests

Functional Testing

• Test that your code is working

• Not how it is working

• More black-box testing

• Incorporate your Database

• End-to-End testing

/** @test */public function createCustomerCreatesExpectedCustomer() { $this->customerService->createCustomer('4711', 'MickeyMouse'); $this->persistenceManager->persistAll(); $this->assertSame( '4711', $this->customerRepository->findAll()->getFirst()->getId() ); }

Use Integration Tests

• Unit tests might be the wrong tool to test, whether your application is working

• Integration Tests provide a full stack test that detect defects easier

Feature: Language menu In order to switch the language on the website As a website user I need to be able to select the language in a menu Background: Given I am in "desktop" layout And I am on "/" Scenario: Switching the language from english to german When I follow "Language" And I wait for 500 milliseconds And I follow "Deutsch" Then I should be on "/de.html"

/** * Given I am in "desktop" layout * * @When I am in :layout layout */public function iResizeTheWindowToLayout($layout) { if (array_key_exists($layout, $this->screenSizes)) { $currentLayout = $this->screenSizes[$layout]; $this->getSession()->getDriver()->resizeWindow( $currentLayout['width'], $currentLayout['height'], 'current'); return TRUE; } throw new \Exception(sprintf('Layout "%s" not defined', $layout)); }

Frontend Tests with Behat

• Behat Tests provide full-stack tests including the frontend

• Behat uses easy-to-read language „Gherkin“ for writing tests

• Selenium is used to run the tests

③ Jenkins

Jenkins

• Java Application

• Can be deployed into Tomcat or Standalone

• Many Plugins available

• Basically a Task Runner

Deployment Stage

* Demo Deployment

* Production Deployment

Acceptance Stage

* Functional Tests * Frontend Tests

Commit Stage

* Clone Repository * Build Project * Unit Tests

Triggeredby

Commit

green green

Jenkins Stages• Commit Stage

• Fast feedback: project builds, unit tests run?

• Acceptance Stage

• Slower tests: Integration tests and Frontend tests

• Deployment Stage

• Responsible for the Deployments to Staging and Production

TL;DR

Do not „programm“ your CI / CD tasks in Jenkins - use scripts and make them part of your projects!

④ Surf

TYPO3 Surf

• A Remote Server Automation and Deployment Tool

• Written in PHP

• Based on the TYPO3 Flow Framework

• Can be deployed as a Flow Package or Standalone

Deployment Workflow

• Build Locally

• Run Tests

• Ship / Transfer

• Run Smoketests

• Only switch Release if Tests pass

Surf Concepts

$workflow set up HOW to deploy

$node set up WHERE to deploy

$application set up WHAT to deploy

$deployment glue it all together

<?php$workflow = new \TYPO3\Surf\Domain\Model\SimpleWorkflow(); $node = new \TYPO3\Surf\Domain\Model\Node('staging'); $node->setOptions(array( 'username' => '<username>', 'composerCommandPath' => '/usr/local/bin/composer')); $node->setHostname($host); $application = new \TYPO3\Surf\Application\TYPO3\Neos('<project_name>'); $application->setOptions(array( 'repositoryUrl' => '<git_remote_url>', 'keepReleases' => 5, 'packageMethod' => 'git', 'transferMethod' => 'rsync', 'updateMethod' => NULL)); $application->setDeploymentPath('<deployment_path>'); $application->addNode($node); $deployment->addApplication($application); $deployment->onInitialize(function() use ($workflow, $application, $project) { $workflow->afterStage('migrate', 'codecoon:importContent'); });

/var/apache/XXXXXX/staging/XXXXX/releases$ ls

20141105100124201411051029352014111209592420141118055114201411180722252014111904183520141120045634current -> ./20141119041835previous -> ./20141118072225next -> ./20141120045634

/var/apache/XXXXXX$ ls -la

htdocs -> staging/XXXXXX/releases/current/htdocs

⑤ Chef&Vagrant

Developer Machine / Laptop

vagrant up

Virtual Machine

WorkspaceProject 1

chef-solo

Gitlab Server

gitpull / push

ssh

mysql

samba nfs

sambanfs

gitssh

httphttp

chef run

ssh

Our Vagrant Approach* We have VirtualBox and Vagrant running on our laptop * We start a virtual machine and run Chef inside this machine * Chef sets up

* Our services (Apache, PHP, MySQL, …) * Our projects / webspaces

* We can now use our familiar tools to work on our projects * It feels like „working locally“ although we have a Sandbox

* We cannot crash the host OS when crashing the Dev-Environment * Think about packages * Think about different software versions for different projects * Think about how long it takes to re-install your laptop…

WorkspaceProject 1

WorkspaceProject 2

WorkspaceProject 2

Prov

ision

Pro

ject

s

Prov

ision

Jenk

ins S

erve

r

Confi

gure

Apa

che

Confi

gure

PH

P&

MyS

QL

We set up our Projects on Jenkins

Automatically!!!

Chef & Jenkins

There shall be 2 Chef runs 1. The one that provisions our Jenkins Server [not yet finished…] 2. The one that provisions the projects inside our Jenkins Server Those Projects are „publicly“ available * We can use them as „normal“ Website (e.g. for Review and Manual Testing) * We can run UI tests on them

In the future we want to change this to a master/slave approach.

⑥ Monitoring

Monitoring Stage

• Check for Website to be alive

• Send Notifications

• Email, Jabber, SMS

• Ops: Nagios — Devs: Jenkins

Performance Stage

• Check Realtime Performance of your Website

• Gatling is a great Tool for writing Performance Tests

• Visualize Results on Dashboard

Summary

Version Control Build

gitlab triggers

Unit TestsCommit Stage

build succeeds

UI TestsAcceptance Stage

DB Tests

tests pass

tests pass

Deployment Stage

tests pass

build succeeds

Build locally

Demo Stage

Production Stage

rsync

rsync

git push

Developer

Feedback

Gitlab Server

Jenkins Server

WorkspaceProject 1

WorkspaceProject 2

WorkspaceProject 2

prov

isio

n

Selenium Serverhttp / RESTful Services

RepositoryProject 1

WorkspaceProject 2

WorkspaceProject 3

git shell / ssh

http / UI Testing

git clone

Production 1

WebspaceProject 1

Production 2

WebspaceProject 2

Production 3

WebspaceProject 3

ssh / rsync

www.codecoon.comMake it fun to code again

Check it out

Questions

https://about.gitlab.com/

http://jenkins-ci.org/

https://phpunit.de/

https://www.getchef.com/

https://www.vagrantup.com/

http://www.seleniumhq.org/

http://typo3.org/additional-products/surf

http://www.sourcetreeapp.com/download/

http://dashing.io/

http://gatling.io/

http://www.martinfowler.com/articles/continuousIntegration.html

http://www.thoughtworks.com/insights

http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html

http://www.rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste.pdf

http://www.mind-the-seb.de/blog/codereview-made-simple.html

https://www.atlassian.com/git/tutorials/comparing-workflows/forking-workflow

http://nvie.com/posts/a-successful-git-branching-model/