How we build features at Serv.sg

in a Pandan Leaf...

January 06th 2016

Nicolas MasNicolas@serv.sg

Nicolas.Mas@gmail.com

https://www.linkedin.com/in/nicolasmashttps://github.com/nicolasmas

Acknowledgment

“Alone we can do so little; together we can do so much”

Sandeep Isac Marco Yang Alex Hayden

Alexander Bulat Andrei Ping Yang

<ME>

Nicolas MasNicolas@serv.sg

Nicolas.Mas@gmail.com

https://www.linkedin.com/in/nicolasmashttps://github.com/nicolasmas

</ME>

What this talk won’t do

⬗ Entertain you with funny

pictures (sorry). This will be

the only one.

⬗ Describe the tools being

used, unless there is a

specific request (let’s keep

the focus).

⬗ Guarantee that Murphy’s law

won’t impact the demo.

So what is it going to be?

⬗ What do we do at Serv.sg?

⬗ Our (current) technology stack

⬗ Problem we solve

⬗ Feature build (+ DEMO)

⬗ Back to our initial problem...

⬗ Limits

⬗ Enhancements

⬗ B Tracks: if we really have time

Disclaimer: I am a DevOps padawan, what is described tonight is probably not the best solution ever. Any suggestion is welcome~

What we do at Serv.sg

✓ Real time Mobile workforce location tracking✓ Automated scheduling of service appointments✓ Automated email / SMS reminders of upcoming appointments and contracts.✓ Manage customer data and all history of service appointments✓ Manage Services offered, including recurring contracts and packages, etc.✓ Advanced reports on employee utilization as well as business

Technologies (Stack… for now)

Key backend technology

Data storage

Key frontend technology

Technologies (DevOps)

For now… (bis)

The Problem We Hope to solve...

Feature A

Feature B

… Feature N

PRODUCT BACKLOG

QA / UAT / PRODUCTION

Build - All or Nothing

Feature A

Feature B

Monolithic App Development

Overview

JIRA-456

JIRA-456.serv.sgJIRA-456.db.serv.sg

/serv-branch-build

Branch JIRA-456

JOB JIRA-456

On “New Branch”

JOB Monitor Branches

Overview

On “Branch Deleted”

JIRA-456

JIRA-456.serv.sgJIRA-456.db.serv.sg

/serv-branch-build

Branch JIRA-456

JOB JIRA-456

JOB Monitor Branches

It all starts in Jira

⬗ The developer takes ownership of, let’s say the item “JIRA-456” during a

sprint.

⬘ Actually, he does not do it in JIRA, but from github by branching out the

codebase.

⬘ The Branch HAS TO BE named after the jira item, e.g. JIRA-456

It continues on Slack

⬗ The moment the branch is created, we can actually build the

infrastructure:⬘ In slack: “/serv-branch-build”

⬗ Slack is asking a Jenkins job to monitor the github repository for

new/deleted branches

What happens in the jenkins monitoring job

⬗ The Jenkins job will do a few things:⬘ 1. It will connect to github and get all the branches

⬘ 2. It will detect our new branch

⬘ 3. A custom jenkins job will be created for that branch

⬘ Terraform will be invocated

⬗ It will also keep you updated of the progress in Slack

The Infra is built...

...But not yet provisioned !!

Note: we use a pre-baked OS image for the instances

Building the branch

⬗ In Jenkins

Provisioning the branch

⬗ Ansible playbook⬘ Site.yml dynamically set

⬗ Roles⬘ Hostname set

⬘ Supervisord

⬘ Yum additional packages

⬘ Web conf

⬘ DB conf

⬘ DB Load

⬘ (...)

End Result

Destroying the build

⬗ The features has been validated or ditched, the branch is either merged

or destroyed on github

Note: If we need a few back and forth between the product and the dev for the feature, it’s just a matter of running the jenkins branch job to deploy again

Back in Slack

⬗ “/serv-branch-build”

⬗ AWS quick check.. the instances are terminated

Looking back at our initial problem...

⬗ Deploying by feature is not new. Immutable infra seems to be less

common. The two combined? Anyone doing it?

⬗ From a business/product perspective⬘ More responsive to bug fixing, features deliveries

⬘ No confusion on what has been deployed

⬘ The branch can be hotfixed => one step closer to CD

⬗ From a technical perspective⬘ Leveraging the cloud infrastructure, programmatically

⬘ No environment recycling or reconfiguring with the related uncertainty

⬘ As close to production as we can

⬘ A good way to craft baked OS with the latest update and test it

Limits/Challenges

⬗ Hard to keep track of the full cycle, from the OS baking to the feature

deployed:

⬘ Packer: need to manually describe the target provisioned OS. Hard to test,

need to manage the produced artifacts

⬘ Ansible: need to manually write the roles, hard to test and a pain to

maintain

⬘ Terraform: need to manually describe the infra, hard to test, painful to

maintain and evolve. Some bugs with route 53

⬘ Dependency on Jenkins

⬘ A fair amount of templating (jinja2 style) with a risk induced by the dynamic

aspect of the build (hostnames, EIP...)

⬗ Many pieces in movement, dependency

⬗ Need to be in sync with the product team to check and act upon the

feature as soon as it is available. We don’t want a stack running for ages..

Enhancement

⬗ Use a tool to orchestrate the branch + infra build aspect (Spinnaker.io?

StackStorm.io?) or build a simple micro service to do it

⬗ Choose the branch to build in slack instead of all the branches with the

pattern “JIRA-XYZ”

⬗ Improve the provisioning and better manage the ansible playbook

(Tower.io?)

⬗ How is this compatible with our next architecture steps? (micro services)

Bonus Track - Did you know ?

“The mother of all demos” http://en.wikipedia.

org/wiki/The_Mother_of_All_Demos

⬗ Douglas Engelbart's December 9, 1968

⬗ 90-minute live public demonstration of the online system, NLS, they had

been working on since 1962

⬗ 1000 attendees and a demo of almost all features we use today⬘ Mouse

⬘ hypertext, object addressing and dynamic file linking

⬘ shared-screen collaboration involving two persons at different sites

communicating over a network with audio and video interface.

http://web.stanford.

edu/dept/SUL/library/extra4/sloan/mousesite/1968Demo.html

https://www.youtube.com/playlist?list=PL76DBC8D6718B8FD3

Watch it !!

Bye Bye !