[@IndeedEng] Engineering Velocity: Building Great Software Through Fast Iteration

Engineering VelocityBuilding Great SoftwareThrough Rapid Iteration

Dan HellerEngineering ManagerEmployer Applications

Indeed

#1 Job Site Worldwide

More people find jobs on Indeed than anywhere else

Over 100M monthly visitors

Over 3B monthly searches

Rapid IterationLeads to

Better Products

Data-DrivenProduct Design

go.indeed.com/ddpd

http://go.indeed.com/ddpd

http://go.indeed.com/ddpd

Indeed Product Development Principles

Measure Everything

Test Everything (A/B Experiments)

Iterate Quickly

Build

MeasureLearn

Lean Startup Cycle

Indeed Product Development Cycle

Deliver

MeasureLearn


Quickly!

Deliver

MeasureLearn


MeasureLearn MeasureLearn

Deliver

Engineering Velocity

MeasureLearn MeasureLearn

Deliver

Engineering Velocity (n)[en-juh-neer-ing vuh-los-i-tee]

Our ability to quickly deliver implementations of ideas to job seekers & employers

Deliver Developer Productivity

Software Architecture

Processes and Tools

Risk Management

Deliver

Service Oriented Architecture

Job SearchWeb App

Doc Retrieval

Company Info

Authentication

Location

Index Search

Job SearchWeb App Index Search

Doc Retrieval

Company Info

Location

Authentication

Doc Retrieval

Company Info

Job SearchWeb App

Authentication

Location

Index Search

Technical Challenges

High Performance● Speed is a feature● 100ms response time


Forward & Backward Compatibility● Any client and any service can be updated at

any time

High Performance SOA Framework

Boxcar

Image used with permission of O Scale Trains Magazine (oscalemag.com), photo credit Don McFall of Old Line Graphics

http://go.indeed.com/boxcar

http://oscalemag.com/




Boxcar: Load Distribution without Middleware



Represent complex data types

Extremely compact and performant

Forwards and backwards compatible

Boxcar: API Compatibility

Boxcar: API Compatibility

Request and response data encoded as Protocol Buffers

Compatible evolution of service interfaces

Reduces release order dependencies

Doc Retrieval

Company Info

Job SearchWeb App

Authentication

Location

Index Search

Boxcar!

Service Oriented Architecturefor

Web Frontends

Each page hasdifferent featuresdifferent operational requirementsdifferent owners

Each page should be its own application

Trends

Company Info

Forums

Salary

Job Search

Indeed


Provide a consistent user experience

One implementation of shared functionality

Option: Shared Code Libraries

Reduced flexibility of technology decisions

Requires a release of all projects

Occasional inconsistent user experience

Navshell

Application-level Rewriting HTTP Proxy

Navshell

Request

Web App

HTTP

<head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"></head><body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div></body>

REQUEST HEADERSCookie, User-Agent, Referer, etcX-Indeed-Shell: 1X-Indeed-Logged-In: [01]X-Indeed-Email: [email protected]: en_USX-Indeed-Geo: US

Navshell

Web App

RESPONSE HEADERSSet-CookieX-Indeed-Page-Title: Indeed CareersX-Indeed-Include-Js-Footer: /js/app.jsX-Indeed-Include-Css: /style/main.css

Navshell

Web App

Navshell

No duplicate code

Centralized navigation changes

Plain HTTP with strong contract

Decoupled front-ends

Navshell

Requests

Services Web Apps

Web-Based Delivery of Mobile App Features

Mobile

Mobile Browser Mobile App

MobileBrowsers

MobileApps

DesktopBrowsers

Job Search App

● One code base for mobile web, iOS, Android● Simultaneous release of features across

platforms

Shared Code Across Platforms

● No waiting for App Store approval● No waiting for job seekers to upgrade

Release On Our Own Schedule

Easy To Run Experiments

● A/B testing in native apps extremely challenging

● Reuse existing infrastructure for performing and evaluating experiments


Processes and Tools

Risk Management

Deliver

one search. all jobs.

una búsqueda. todos los empleos.

Jedno vyhledávání. Všechna pracovní místa.

Ein Klick. Alle Jobs.

一站搜尋，工作齊全

tek arama. bütün işler.

uma busca. todos os empregos.

één klik. alle vacatures.

あらゆる仕事まとめて検索

una ricerca. tutti i lavori.

Implement Commit

Deploy to QA

Deploy to ProductionVerify

Before Internationalization

Implement Commit

Translate


Deploy to QA

CommitExtract

Internationalization Process Overhead

+1 day (20%)

Machine Translation

English

Portuguese

Machine Translation

Low quality translations are better than no translations

But we needed something better

Human Translation Service

Tuesday afternoon: Commit Code Fetch Machine Translations

Human Translation Service

Tuesday afternoon: Commit Code Fetch Machine Translations Request Human Translations

Wednesday morning: Retrieve Human Translations

Implement Commit

Translate


Deploy to QA

CommitExtract

Localization before Translation Automation

Implement Commit


Deploy to QA

Extract

Localization with Translation Automation

Retrieve


Processes and Tools

Risk Management

Deliver

Probability of issue

Duration of issue

Number of users affected

Risk

Releases

Smaller Releases

More releases → less functionality per release


Monitoring

Duration of issue

Real-time monitoring with Datadog

● Agent reports summarized data

● System and customized business metrics

● Customizable dashboards

Healthcheck Framework

● Applications and services report internal state & health

● Divert traffic away from unhealthy instances● Disable features for failing dependencies

Monitoring and Alerting

● Identify runtime issues● Alert Ops when an issue arises

Proctor

● System for managing A/B experiments● Segment visitors and assign test groups● Manage feature visibility

Proctor

Manage Experiments

Proctor

Manage Experiments → Manage Behavior

Proctor

"rule" : "loggedIn"

"rule" : "${smartphone && 'JP' == country}"

"rule" : "${(ua.OS.family == 'ios' && ua.OS.majorVersion >= 6)}"

etc.

Proctor

if (proctor.isNewThingEnabled()) { newThing.generateResults();} else { oldThing.generateResults();}

Feature Toggles


"rule" : "fn:endsWith(confirmedUserEmail, '@indeed.com')"

Decouple Features from Releases

Duration of issue

Rollback features and releases independently

Number of users affected

Small allocation changes

0% → 1% → 5% → 50% → 100%


Processes and Tools

Risk Management

Deliver

Increased impact on products

Increased chance of success

No Failed Projects

Fail Fast

No Deadline Crunches

There’s always another release in a few days

More Transparent Schedules

Large changes hidden behind feature toggles

Incremental progress validated in production

environment

We built all of this incrementally, to meet our

own challenges

Yours are different

v = d / t

With rapid iteration & quick delivery, we can

cover more ground

[@IndeedEng] Engineering Velocity: Building Great Software Through Fast Iteration

Technology

Transcript of [@IndeedEng] Engineering Velocity: Building Great Software Through Fast Iteration