How we sped up delivery
using feature branches
Agilia Conference 2018
Jan Vilímek, Platform Architect
Oriflame Software
About Oriflame
Complexity we face
Challenges we had when we
were growing
How to get effectivity and
flexibility back?
Take away
AGENDA
2Copyright ©2018 by Oriflame Cosmetics Global SA
3
BUT FIRST.. A FEW QUESTIONS FOR YOU..
Copyright ©2018 by Oriflame Cosmetics Global SA
Do you use agile for
developing software?
4Copyright ©2018 by Oriflame Cosmetics Global SA
Which source control do
you use?
5Copyright ©2018 by Oriflame Cosmetics Global SA
Are you already using
feature branches?
6Copyright ©2018 by Oriflame Cosmetics Global SA
About Oriflame
Complexity we face
Challenges we had when we
were growing
How to get effectivity and
flexibility back?
Take away
WHERE ARE WE NOW?
7Copyright ©2018 by Oriflame Cosmetics Global SA
8
Whole globe spread
250+ million online visits per
year
Direct selling company
3 millions active
consultants
Copyright ©2018 by Oriflame Cosmetics Global SA
~300 people across Europe
Provides IT solutions exclusively to Oriflame
Dynamic team with a family spirit in a wonderfull town of Olomouc, branch in Brno
Partners in Ukraine, Poland
Multinational
Center of technology innovations
ORIFLAME SOFTWARE
9Copyright ©2018 by Oriflame Cosmetics Global SA
Enterprise Resource Planning / Invoicing / Warehouse
systems
Data Warehouse / Business Intelligence
Product catalogues DTP, pricing
Digital e-commerce platform
See next slides
DEVELOPING & SUPPORTING
10Copyright ©2018 by Oriflame Cosmetics Global SA
E-COMMERCE CONSTANT TRANSFORMATION
11
1990 - 2001 2002 - 2013 2014-2015 2016-2019
Orico Orisales Online V4 Cumulus
1-2 Servers Farm of servers Elastic cloud computing
WebDigital
MarketingDigital Business
Autonomous
Business
+Smart
Machines
2000 2005 2017 2025
2020-2025
Combination of people,
connected things, analytics and
algorithms, providing a platform
for business operations
Operators only
(~100s users)
Online orders
growing…99% orders done online (3M users)
Copyright ©2018 by Oriflame Cosmetics Global SA
E-COMMERCE
Copyright ©2018 by Oriflame Cosmetics Global SA 12
SEVERAL MOBILE APPLICATIONS
Copyright ©2018 by Oriflame Cosmetics Global SA
14 cross-functional distributed development teams
2 weeks sprint
Every team:
Front-End & Back-End developers
Test Analyst and Engineer
Product Owner, Scrum Master, Architect (guilds)
Owns specific business domains
E-COMMERCE TEAMS
14Copyright ©2018 by Oriflame Cosmetics Global SA
About Oriflame
Complexity we face
Challenges we had when we
were growing
How to get effectivity and
flexibility back?
Take away
WHERE ARE WE NOW?
15Copyright ©2018 by Oriflame Cosmetics Global SA
60 countries (legal / functional requirements) in one monolith
Central driven content
Many 3rd party systems (sms / payment providers)
Legacy sales system integration
24/7 site availability, many parts (x6 regions) => complex deployment
Cloud (specific development)
OUR SITUATION
16Copyright ©2018 by Oriflame Cosmetics Global SA
SALES PEAK HANDLING
17
One month
One day
Page views per hour in one country
One of the reasons
to use cloud..
Copyright ©2018 by Oriflame Cosmetics Global SA
QA ENVIRONMENT
18
Cluster in the South
Central US
Cluster in East Asia
Copyright ©2018 by Oriflame Cosmetics Global SA
About Oriflame
Complexity we face
Challenges we had when we
were growing
How to get effectivity and
flexibility back?
Take away
WHERE ARE WE NOW?
19Copyright ©2018 by Oriflame Cosmetics Global SA
20
LIVEDEVTEST
User Story 2
User Story 1
MAIN
Smoke and functionaltesting
“Regression" testingLive deploy + post deploy
support
PRE-LIVE
Release
Once per 2 weeks full merge
HotfixMerge based
on label
What was in MAIN
branch in time of the
merge was eventually
pushed to Live
Is it OK to
check-in now or
should I wait?
WHEN WE HAD ONLY SEVERAL TEAMS, DEVELOPMENT WAS EASY…DEVELOPMENT LIFECYCLE OVERVIEW
Copyright ©2018 by Oriflame Cosmetics Global SA
More developers, but also testers, analysts
More teams
Wheel re-inventing
How to coordinate?
WE HAVE BEEN GROWING…
21
Architect guild was
established
Copyright ©2018 by Oriflame Cosmetics Global SA
Once per week Architect Synchro
Dual track agile
Architecture track
Delivery track
SOFTWARE ARCHITECTS GUILD
22
LEAD DEV
ARCHITECT
LEAD DEV
ARCHITECT
LEAD DEV
ARCHITECT
ARCH LEAD
DEV MANAGER
Architect Lead DEV
Copyright ©2018 by Oriflame Cosmetics Global SA
No stable code in Main
Needs for formal Quality Assurance
Quality gate:
User Story can’t be deployed to production unless it’s tested
Part of Definition of Done
WE HAVE BEEN GROWING…
23Copyright ©2018 by Oriflame Cosmetics Global SA
Team Foundation Server
No environment for testing
Environments too complex (Tamagotchi)
SHOULD WE USE FEATURE BRANCHES?
24
So… why not do
some branching?
Copyright ©2018 by Oriflame Cosmetics Global SA
DEPLOYMENT WITH QA GATE
25
QA LIVEDEVTEST
User Story 2
User Story 1
DEV
Smoke testingFunctional testing +
acceptanceRegression testing
Live deploy + post deploy support
QAcherry
picking
Release/HF X.YMain
cherry
picking
new
branch
PRE-LIVE
Copyright ©2018 by Oriflame Cosmetics Global SA
Developer
• Implement and test locally
Developer
•Check-in to Dev branch
Developer
•Ask for CODE REVIEW
•Test on DEVTEST environment
• (daily deploys)
Developer
•Code Review OK?
•Task [Implementation] ⇒Done
Quality Assurance
•Smoke test on DEVTEST environment
• If OK, assign MERGE task to Cherry picker
Cherry picker
•Merge to QA
•Task [Merge QA] ⇒Done
• (daily deploys)
Quality Assurance
•Full test on QAenvironment
• If OK, assign MERGE task to Cherry picker
Cherry picker
•Merge to Main
•Task [Merge Main]
⇒ Done
ACTIVITY FLOW: REGULAR DEVELOPMENT
Copyright ©2018 by Oriflame Cosmetics Global SA
IT SOUNDS GREAT
SO WHERE IS THE
ISSUE?
27Copyright ©2018 by Oriflame Cosmetics Global SA
TOO MANY CONFLICTS..
28
MAIN MAINQADEV
Former: Now:
Copyright ©2018 by Oriflame Cosmetics Global SA
MERGE HELL
29
DEV
Ch01 (US01)
Ch02 (US02)
Ch03 (US01)
Ch04 (US02)
Ch05 (US01)
QA
Ch02 (US02)
Ch01 (US01)
Ch03 (US01)
Ch05 (US01)
Ch04 (US02)
MAIN
Ch02 (US02)
Ch01 (US01)
Ch03 (US01)
Ch04 (US02)
Ch05 (US01)
ChXX: changeset XX
USXX: User Story XX
Code in Main not stable..
Copyright ©2018 by Oriflame Cosmetics Global SA
Development is not able to quickly deliver new increment
respectively get customer feedback
TIME MATTERS…
30
1st day
• Commit to DEV
• Code review with peer, no changes
2nd day
•Deployed to DEVTEST
•Smoke tests (of the feature + code integration)
•Merge to QA
3rd day
• Deployed to QA Env
• Functional + integration tests
• Merge to Main
If issue found in DEV => +1 day (+possibly broken environment)
If issue found in QA => +2 days (+possibly broken environment)
Copyright ©2018 by Oriflame Cosmetics Global SA
3 days to deliver …… but usually 6 days or more
… in 10 days sprint
©Oriflame Cosmetics AG, 2018 31
About Oriflame
Complexity we face
Challenges we had when we
were growing
How to get effectivity and
flexibility back?
Take away
WHERE ARE WE NOW?
©Oriflame Cosmetics AG, 2016 32
HOW TO GET EFFECTIVITY AND FLEXIBILITY BACK?
33
Change starts from the people
Retro, Yammer, Slack, E-mails…
Software Architects Guild
Technical solution
Scrum Masters Guild
VSTS Epic & Feature boards on
Global level
Copyright ©2018 by Oriflame Cosmetics Global SA
FEATURE BRANCHES DEPLOYMENT MODEL
34
QA LIVEDEVTEST
Master
Feature full scope testing
Code integrationRegression testing
+ acceptanceManual + auto
regression testingLive deploy + post
deploy support
Release X.Ynew branch
Feature 2
Branches
Environment
mergingFeature 1
Feature 2
PRE-LIVE
Feature 1
Branches
Environment
Separated
environments per FB
Just one Master
branch
Copyright ©2018 by Oriflame Cosmetics Global SA
35
MAINQADEV
Do you remember?
Copyright ©2018 by Oriflame Cosmetics Global SA
MAINQADEV
NO CONFLICT IN MASTER BRANCH
36
Feature01
Master
Feature01 Feature01 Feature01
Copyright ©2018 by Oriflame Cosmetics Global SA
You remember this…
TIME MATTERS… WITH PREVIOUS SETUP
37
1st day
• Commit to DEV
• Code review with peer, no changes
2nd day
• Deployed to DEVTEST
• Smoke tests (of the feature + code integration)
• Merge to QA
3rd day
• Deployed to QA Env
• Functional + integration tests
• Merge to Main
Copyright ©2018 by Oriflame Cosmetics Global SA
FROM 3-6 DAYS TO 3 HOURS
38
1st hour
• Create branch
• First commit
• Code review with peer
• Start build + new environment
2nd hour
• Deploy changes
• Smoke test
• Run integration (auto) tests
3rd hour
• Functional tests (Quality Assurance )
• Merge to Master
Copyright ©2018 by Oriflame Cosmetics Global SA
39
THE BENEFITS
Copyright ©2018 by Oriflame Cosmetics Global SA
Deployment from 3-6 days to 3 hours
Faster feedback from Product Owners / customers
=> Less feature redevelopment
Higher team responsibility
End of the merging hell, more stable environments
WHAT HAS THE SOLUTION BROUGHT US?
40Copyright ©2018 by Oriflame Cosmetics Global SA
Isolated Continuous Integration environment per feature branch
Branch master
Process change
Teams involvement
Mindset change
WHAT DO WE NEEDED FOR FEATURE BRANCHING?
41Copyright ©2018 by Oriflame Cosmetics Global SA
42
CHALLENGES
Copyright ©2018 by Oriflame Cosmetics Global SA
How big should a feature be?
Environment costs money + time
Let’s limit environments per team (2-3)
FEATURE SIZE MATTERS...
43Copyright ©2018 by Oriflame Cosmetics Global SA
We needed to mock 3rd parties
Legacy sales systems
Sms/email/payment providers
It’s good:
you develop independently
Ultimate responsibility for environment => team
it’s testable
But: the mock needs to be developed and kept up to date
ISOLATION OF DEVELOPMENT (MOCKS)
44Copyright ©2018 by Oriflame Cosmetics Global SA
Environment (VM, storage…) creation with one click
No image, but install scripts
Database/storage snapshot
Build
Deployment
Testing
AUTOMATION (SELF SERVICE)
45Copyright ©2018 by Oriflame Cosmetics Global SA
About Oriflame
Complexity we face
Challenges we had when we
were growing
How to get effectivity and
flexibility back?
Take away
WHERE ARE WE NOW?
46Copyright ©2018 by Oriflame Cosmetics Global SA
47
Every change come with people
Communicate
Automate everything
Separate concerns, independent
development
Use feature branches ☺
Copyright ©2018 by Oriflame Cosmetics Global SA
49