6 ways DevOps helped PrepSportswear move from monolith to microservices

Andi GrabnerPerformance Advocate@grabnerandi

Richard DominguezDeveloper in OperationsPrep Sportswear

The Promise of

Confidential, Dynatrace, LLC

Goal: Becoming a software defined business

2013 2014 2015

$108m

$400m

$2bn est.

Source: “Creative destruction in the S&P500 index,” Jan 2014; "Uber Expands Funding Round as Revenue Growth Acceleratesm," Wall Street Journal, Feb 2015. See more discussion in “The Three Horsemen of the Digital Apocalypse Considered.”

52% of G500 since 2000 GONE Uber rumored net revenue

What they really want!

700 deployments / YEAR

10 + deployments / DAY

50 – 60 deployments / DAY

Every 11.6 SECONDS

Not only fast delivered but also delivering fast!

-1000ms +2%

Response Time Conversions

-1000ms +10%

+100ms -1%

60% Rate performance/response time as the #1

mobile app expectation- ahead of features and functionality -

Source: Forrester Research 2014

Why most (will) fail!

Agenda and learning journey

2005

CEO started with new

business idea

Most successful shopping season in

company history

Building a DevOps Culture

Building a website Growing the monolith

Increased deployments

resulted in more failed deployments

Our pipeline: why we deploy less

frequently today!

Breaking the monolith -

deploy more frequent

Started with monitoring

Response time and availabilty

improvements

. . . 2014 2015 20172016

Let’s begin! PrepSportswear 10 years ago . . .

• SportsWear Inc. started out with our CEO Chad Hardvigson personally pressing shirts• The demand for specially made sport attire was high•No one was providing this type of specialty service

Technology ramp-up to the rescue . . .

•We needed a website• Automation was needed to handle large demand•New techniques in printing automation was quickly developed to meet increasing demand

Rapid development put stability in the back seat

• Fast development lead to continual instability• Instability lead to consistent breaks• Consistent breaks lead to developer ‘blindness’

Large buildup of half-done projects

•Development team was isolated from the rest of the company• Requirements for projects were deemed by the development team only•No true concept of “done” – no monitoring of usage

Monolith started to take shape

• Its easier to develop new features on top of each other• Its easier to setup one single (though large) application• Testing individual components, however became very difficult

You are not alone in this world!

The Monolith was getting uncontrollable

• PrepSportswear was heading toward a development nightmare• Development team didn’t

want to ‘see’ this reality• Constant fixes were a

common occurrenceHOW MANY DEPLOYMENTS

DID YOU MAKE?

It‘s not about blind automation of pushing more bad code on new stacks through a pipeline

It‘s not about blindly giving everyone ops powerto deploy changes only tested locally

Hard decisions needed to be made . . .

•New Director of Information Technology was hired to “help” with the system•Old development team didn’t see a problem – this caused a lot of friction• Certain individuals were let go

PrepSportswear’s new beginning

•Hired the “right” people•DevOps mentality was soon adopted.• Moved into a two week sprint process

• Reduced the number of deployments

We needed to reduce the number of deployments

• Increased the Quality for each deployment•Needed to create proper Monitoring when Deployment occurred• Invested in External Monitoring

Performance increase when new team started

Keynote time graph (April2014 – Dec 2015)

Performance increase when new team started

Time chart showing uptime availability (April2014 – Dec 2015)

Monitoring, monitoring, monitoring

• Adopted a true APM system• Implemented proper hardware monitoring •No more “customer’s will let us know if we have issues”…

Cool dashboards & metrics for biz & dev

Dashboards & metrics for DevOpsperformance focused

Dashboard & metrics for marketing

• Overloaded Pages• Memory Leaks• Too many Database Queries• Bad External Web Service Calls• Threading Issues• Caching Issues• SEO Overuse of bots causing

larger load then nessessary• ....

Some of the top problems found ...

#1: Dev: Don’t Check In Bad Code

Step #1: Execute your Tests just as you

always do ...

Step #2: ... but DO IT WITH Dynatrace!!

#1: Dev: Don’t Check In Bad Code

Step #1: Execute your Tests just as you

always do ...

Step #2: ... but DO IT WITH Dynatrace!!

Step #3: Verify Code works as intended without leaving the IDE!

#1: Analyzing every Unit, Integration & REST API test

#2: Key Architectural Metrics for each test

#3: Detecting regression based on measure per Checkin

#2: CI/CD: Stop Bad Builds through Metrics

#3: Ops: Monitor your Services/Users after Deploy#1: Usage

Tip: UEM Conversion!#2: Load vs Response

Tip: See unusual spikes

#3: Architectural Metrics DB, Exceptions, Web

Service Calls

#1: Do my campaigns work?

#2: Who are my users?

#5: Biz: Understand your End Users

#6: Biz/UX: Optimize End User Behavior#1: Are they using the

features we built?

#2: Is there a difference between Premium and

Normal users?

#3: Does Performance have a Behavior Impact?

Metrics-Driven Pipeline: Stop Bad Builds Early!

Dev&Test: Personal License to Stop Bad Code when it

gets created!Tip: Dont leave your IDE!

Continuous Integration: Auto-Stop Bad Builds based on AppMetrics from Unit-, Integration, - Perf Tests

Tip: integrate with Jenkins, Bamboo ...

Prod: Monitor Usage and Runtime Behavior per Service, User Action,

Feature ...Tip: Stream to ELK, Splunk and Co ...

Automated Tests: Identify Non-Functional Problems by looking at App Metrics

Tip: Feed data back into your test tool!

PrepSportswear build pipeline

• Developer checks in code• We use release branching on Git• We run a Release Build from TeamCity

• Build Code, Verify Build, Deploy in Test Environment• Run Functional Tests & Manual Acceptance Tests

• We dogfood our code: Deploy internally and use the new code• Since it’s a monolith app, everything is in the same deployment (e-

commerce site, internal only back-end services, marketing diagnostics, designer tools, etc..)

• Watch Dynatrace for failed transactions, poor performance, etc

PrepSportswear Build Pipeline (continued)

• Pre-Release TiP – Test in Production• Deploy new code to a small set of

servers that will serve <10% traffic• Deployed servers have a higher set of

Dynatrace sensors that will be used to highly monitor incoming traffic.

• Once satisfied, we release to the rest of our external servers on the official release date

The way forward… micro services

•Why move to Micro Services?•What Micro Services mean to maintain Business Relevance•Does the Cost justify the means to move toward Micro Services?

Image server as part of the monolith app

Future: Continuing breaking up the monolith

• Identifying Key Lines of Business of your App• Figuring out what is internal only and external only•Dependency Management

Q & AAndi GrabnerPerformance Advocate@grabnerandi

Richard DominguezDeveloper in OperationsPrep Sportswear

Performance management for the digital customer age