Continuous Deployment

by Smartling, Inc

#SmartlingDevLove

Andrey Akselrod

Smartling, Founder & CTO

@chelya

Max Sogin

Director, Software Development

What do we do?

● Cloud-based enterprise software platform for

translation management

● Used for websites and dynamic Web

applications, desktop & mobile apps,

business documents, and all kinds of digital

content

Clients

Smartling Operational Scale

3active data centers: US

East, US West, Ireland

400+servers

~200production

deployments a month

2B+page views/month

84K+metrics monitored

17K+triggers for alerts

Tech Team

New York: 20

Dnepropetrovsk, Ukraine: 29

Kiev, Ukraine: 3

Technologies

Java+

Everything you can imagine

Continuous Deployment == Continuous

Releases

Continuous Deployment != Continuous

Integration

Life prior to Continuous

Deployment

● A lot of code is being released

○ Bugs are pretty much guaranteed

○ It is difficult to understand which change caused

issues when a lot of new code is deployed.

○ It is difficult and sometimes impossible to roll code

back

● Quick fixes sometimes wait weeks to be deployed

● There is pressure to wrap up the feature at all costs

before scheduled deployment. Quality takes a hit.

● No one works during the release day. Everyone

expects a sleepless night.

Can we deploy on Friday night?

|

NO!

|

But we only need…

|

HELL NO!

● User feedback is coming late - after substantial

investment of time and effort into development

● The absence of early feedback increases the load

on planning. Heavy upfront planning makes

development process closer to waterfall rather than

agile

● Feedback results are waiting for the next release

which significantly slows down product

development cycle.

OODA Loop

Transition period - Team

Transition Challenges

● Management support is essential

o Transition will take time and resources

o Complete transition may take 6-8 months

● Team culture will change substantially

o Find champions and make sure you have their

support

o It is better to get support from other departments in

the company by explaining the benefits they get at

the end

Transition Challenges

● Excellent reason to create DevOps

● Do not wait until ideal implementation. Start with one

simple component and one button.

● Celebrate every single win

What we did

● Folks on the team were passive-aggressive and did not

want to do anything

● We were assuring, discussing and blackmailing the

team before starting Continuous Deployment

● Weekly releases got cancelled

● We were using stickers for every deployment: smiles

and asses. Smile was a successful deployment, ass -

was a deployment with a problem.

● At first folks were afraid of deploying, but eventually

they got used to it.

Smiles & Asses ;)

Continuous Deployment Rules

● Master (git) or Trunk (svn) is always stable

● Every commit to master (trunk) is deployed to prod asap

● Continuous integration

● Automated testing => where is QA Team?

● Single button deployment

● Monitoring

● Zero downtime software update

● Communication. Every deployment is in chat. Every

deployment should be confirmed. Skype => Slack

● Developer is responsible for every feature end to end

QA

● Automation - as much as possibleo unit testing

o integration testing

o front end testing - balance is needed!

● Developers are responsible for manual

testing, QA is not involved!

● QA helps with automation

Continuous Integration

● Bamboo or Jenkins

● Every branch is built and tested on every

commit

● Only green builds could be deployed

● Build time is critical (fighting for < 10min)

Bamboo

CDeploy

Slack communication

Monitoring

● Examples:o RPS

o avg resp time & percentile

o error rates

o CPU Utilization

o transaction rates

● Correlate Deployments with Metrics

● Splunk, Zabbix, Graphite, New Relic, Loggly

Splunk

● Stack trace by erroridSpeeds up investigation of problems. Easy to communicate. Secure.

● “Log duty”Keep logs clean, by continuous monitoring and elimination of errors. Noise

in logs makes monitoring difficult.

Splunk

New Relic

DB Updates

● Use DB version control system. DBMaintain

● Every change to DB is reviewed by two leads

● There is a single button deploy of DB changes

● Two phase DB updates

o Phase 1: new structures, backward compatible

New columns always nullable, etc.

o Phase 2: cleanup of elements no longer in use

Results

● Continuous Deployment becomes addictive

● Results of your work are immediately

available for your users to use. It is pretty

cool.

● Management is happy - there is a constant

flow of new features being deployed. Users

are happy with new features as well.

● Quick feedback and quick iterations lead to a

product that is much better adapted to the

market

● Your competition does not understand what

happened and how you can move so quickly

● Other parts of the company respect your

results

● Near zero employee turnover in the team

Discussion

#SmartlingDevLove