Post on 11-Aug-2020
THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY
MICHIEL ROOK
@michieltcs
▸ Developer, consultant, trainer, speaker
▸ @michieltcs
@michieltcs
@michieltcs
Large aging monolith
@michieltcs
Large aging monolith
Generates income
@michieltcs
Large aging monolith
Slow & complex
Generates income
@michieltcs
Large aging monolith
Slow & complex
Generates incomeTechnical debt
FRONTEND
MYSQL DB
BACKEND
LOAD BALANCERS / VARNISH
ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK
FRONTEND FRONTENDFRONTEND
BACKEND BACKEND BACKEND
MEMCACHE FTP EXT. SERVICES
SOLR
@michieltcs
@michieltcs
@michieltcs
Manual releases
@michieltcs
Manual releases
Fragile tests
@michieltcs
Manual releases
Frequent issues
Fragile tests
@michieltcs
Manual releases
Frequent issues
Fragile testsLow velocity
GOALS
@michieltcs
GOALS
@michieltcs
REDUCEISSUES
1
GOALS
@michieltcs
REDUCEISSUES
REDUCELEAD TIME
1 2
GOALS
@michieltcs
REDUCEISSUES
REDUCELEAD TIME
INCREASE PRODUCTIVITY
1 2 3
GOALS
@michieltcs
REDUCEISSUES
REDUCELEAD TIME
INCREASE PRODUCTIVITY
INCREASE MOTIVATION
1 2 3 4
REFACTOR? REBUILD?
APPROACH
▸ API first
▸ Services per domain object (job, jobseeker, ...)
▸ Migrate individual pages
@michieltcs
ORIGINAL MONOLITH
DB
@michieltcs
ORIGINAL MONOLITH
PROXY
SERVICEORIGINAL MONOLITH
DB
DBDB
@michieltcs
ORIGINAL MONOLITH
PROXY
SERVICEORIGINAL MONOLITH
ORIGINAL MONOLITH
SERVICE SERVICE
SERVICE
PROXY
DB
DBDB
DB
DB DB
@michieltcs
PROXY
@michieltcs
RewriteEngine On
# Serve feature from new service for internal network RewriteCond expr "%{HTTP:X-FORWARDED-FOR} -ipmatch '192.168.0.0/24'" RewriteRule ^/feature/(.*)$ ${NEW_SERVICE_URL}/$1 [P,L]
# Proxy everything else to legacy application RewriteRule ^/(.*) ${LEGACY_URL}/$1 [P] ProxyPassReverse / ${LEGACY_URL}/
ARCHITECTURE
@michieltcs
FRONTENDS ARE SERVICES
ARCHITECTURE
@michieltcs
FRONTENDS ARE SERVICES
SERVICES BEHIND LOAD BALANCERS
ARCHITECTURE
@michieltcs
FRONTENDS ARE SERVICES
SERVICES BEHIND LOAD BALANCERS
ACCESS LEGACY DB'S
ARCHITECTURE
@michieltcs
FRONTENDS ARE SERVICES
SERVICES BEHIND LOAD BALANCERS
ACCESS LEGACY DB'S
SERVICES IN CONTAINERS
LEGACY
ELASTIC SEARCH
DB
JOB SERVICE
RMQ
ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK
MONGO DB
ITBANEN
JOBSEEKER SERVICE
NVBINTERMEDIAIR
@michieltcs
PROCESS
CONTINUOUS EVERYTHING
CD?
DEV BUILD / TEST
CONTINUOUS INTEGRATION
@michieltcs
DEV BUILD / TEST ACCEPTANCE PRODUCTION
CONTINUOUS DELIVERY
@michieltcs
DEV BUILD / TEST STAGING / ACCEPTANCE PRODUCTION
CONTINUOUS DEPLOYMENT
@michieltcs
@michieltcs
SMALL STEPS
@michieltcs
SMALL STEPS EARLY FEEDBACK
@michieltcs
SMALL STEPS EARLY FEEDBACK
REDUCE TIME TO RECOVER
@michieltcs
SMALL STEPS
EXPERIMENTS!
EARLY FEEDBACK
REDUCE TIME TO RECOVER
@michieltcs
EVERY COMMIT GOES TO PRODUCTION
ONLY COMMIT TO MASTER
NO BRANCHES
NO BRANCHES
REALLY.
PAIR PROGRAMMING
FOCUS ON VALUE
FEATURE TOGGLES, A/B TESTS
@michieltcs
BOY SCOUT RULE
GATES
100% CODE COVERAGE *
DEVOPS
MONITORING
BUILDPIPELINE
AUTOMATE REPEATABLE THINGS
CONTINUOUS TESTING
DEFENSE IN DEPTH
UNIT TESTS
INTEGRATIONTESTS
ACCEPTANCETESTS
UI TESTS
@michieltcs
DEFENSE IN DEPTH
UNIT TESTS
INTEGRATIONTESTS
ACCEPTANCE
UI TESTS
@Test public void jobCannotBeFound() { when(jobRepository.getById(EXPECTED_JOB_ID)) .thenReturn(null); JobService jobService = new JobService(jobRepository); assertNull(jobService.getById(EXPECTED_JOB_ID)); verify(jobRepository).getById(EXPECTED_JOB_ID); }
@michieltcs
DEFENSE IN DEPTH
UNIT TESTS
INTEGRATIONTESTS
ACCEPTANCETESTS
UI TESTS
@Test public void shouldFindJob() { expectedJob = loadFixture('active_job.yml'); actualJob = repository.getById(expectedJob.getId()); assertThat(actualJob, isA(Job.class)); assertEquals(expectedJob.getId(), actualJob.getId()); }
@michieltcs
DEFENSE IN DEPTH
UNIT TESTS
INTEGRATIONTESTS
ACCEPTANCETESTS
UI TESTSScenario: Link to related job Given a job exists And there are related jobs available When that job is viewed Then a list of related jobs is shown And each related job links to the detail page of the related job
@michieltcs
DEFENSE IN DEPTH
UNIT TESTS
INTEGRATIONTESTS
ACCEPTANCETESTS
UI TESTS
@michieltcs
DEFENSE IN DEPTH
TESTER?
UNIT TESTS
INTEGRATIONTESTS
ACCEPTANCETESTS
UI
@michieltcs
CONTINUOUS TESTING
UNIT TESTS
INTEGRATION TESTS
ACCEPTANCE TESTS
UI TESTS
SMOKETESTS
Cost Speed
Exploratorytesting Monitoring
@michieltcs
PIPELINE AS CODEnode { stage('Run tests') { sh "phpunit" sh "behat" } stage('Build docker image') { sh "docker build -t jobservice:${env.BUILD_NUMBER} ." sh "docker push jobservice:${env.BUILD_NUMBER}" } stage('Deploy staging') { sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i staging deploy.yml" } stage('Deploy production') { sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER} -i prod deploy.yml" } }
@michieltcs
DEPLOYING: ROLLING UPDATE
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker pull
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker run
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
wait_for: port=8080 delay=5 timeout=15
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
uri: url: http://localhost:8080/health status_code: 200 timeout: 30
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg
service: name=haproxy state=reloaded
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
template: src=haproxy.cfg.j2 dest=/etc/haproxy/haproxy.cfg
service: name=haproxy state=reloaded
@michieltcs
DEPLOYING
PULL IMAGE
START NEW CONTAINER
WAIT FOR PORT
SMOKE TESTS / HEALTH CHECKS
ADD NEW CONTAINER TO LB
REMOVE OLD CONTAINER FROM LB
STOP OLD CONTAINER
docker stop
docker rm
@michieltcs
BUILD PIPELINE
@michieltcs
FEEDBACK!
RESULTS
Build per service
< 10 min.
1 4
@michieltcs
Build per service
< 10 min.
1
50+ deploys per day
2
@michieltcs
Build per service
< 10 min.
1
50+ deploys per day
Reduced number of
issues
2 3
@michieltcs
Build per service
< 10 min.
1
50+ deploys per day
Reduced number of
issues
Improved page load
times
2 3 4
@michieltcs
Improved metrics & audience statistics
5
@michieltcs
Improved metrics & audience statistics
5
Learning new
technology
6
@michieltcs
Improved metrics & audience statistics
5
Learning new
technology
Increased confidence, velocity &
fun
6 7
@michieltcs
Team acceptance
@michieltcs
Team acceptance
New technology
@michieltcs
Team acceptance
New technology
Docker stability
@michieltcs
Team acceptance
New technology
Docker stability
Pipeline stability
@michieltcs
Feature toggle cap
@michieltcs
Feature toggle cap
Business alignment
@michieltcs
Feature toggle cap
Business alignment
Focus on replacing legacy
@michieltcs
Reduce cycle times. React faster to the market and test product ideas. Make things!
We make Agile, DevOps and
Continuous Delivery accessible!
Hands-on, results-oriented approach to
get you where you need to be.
Modern infrastructure and
pipelines in minutes.