Punctuated Equilibrium,Celestial Navigation,and APIs

Competing through dynamic adaptation#W2EAPI

Sam Ramji @sramjiDan Jacobson @daniel_jacobsonMichael Hart @michaelhart

PUNCTUATEDEQUILIBRIUM

Darwin formulated his theory of evolution about 150 years ago

Based on observations he made in the Galapagos Islands 15 years earlier

A wild diversity of creatures existed in a new environment

Starting from an ancestor which looked like this

Geospiza Fulginosa

Finches evolved that looked like this

Geospiza Fortis

and this

Camarhynchus Pallidus

and this

Camarhynchus Pauper

and this

Geospiza Conirostris

and this

Certhidea Olivacea

and this

Geospiza Scandens

For many years the belief was that this change happened slowly and gradually.

In 1972, Stephen Jay Gould and Niles Eldredge proposed a new idea

that evolution is not slow and gradual

but sudden and violent.

Applying this view to the observations of finches

Geospiza

C. Parvulus

C. Pauper

Camarhynchus

C. Psittacula

C. Heliobates

C. Pallidus

G. Fortis G. Fulginosa

G. Magnirostris G. Scandens

G. Conirostris

Certhidea

C. Olivacea

“Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows.”

Charles DarwinOn Origin of Species

So while it may look slow and gradual in hindsight

Evolution is experienced in punctuated bursts.

If you’re living in a punctuated burst of evolution

it feels like a revolution

CELESTIALNAVIGATION

Exploration

like evolutionary change

only looks smooth in hindsight

Living through it is usually chaotic

Karen JamesThe Beagle Project Blog

To navigate, you need a map and instruments

Maps exist for transferring knowledge

and they too have evolved over time.

They started as oral traditionand were written down in a form called a periplus

Periplus of HannoCourtesy of Heidelberg University

Periplus of HannoCourtesy of Cornell University

Far less efficient knowledge transfer than a modern map of the same journey

Map of Hanno’s JourneyCourtesy of Bourrichon/Wikipedia

Exploration was dramatically held back for want of a map

In the two thousand years between Hanno’s journey on a Phoenician trireme

And the Mediterranean caravel of the 15th century

Maps had only evolved to be graphical descriptions of coastlines

That was a map published a few years beforeColumbus crossed the Atlantic to find India

After his crossing, his expedition shared their knowledgein a new map

Still far from perfectbut much improved.

The biggest challenge in this kind of explorationwas determining their location on the Earth

Instruments for measuring latitude had beenused and improved for centuries

Longitude was the hard problem.

You needed to know not just the angle of the sun and stars

you also needed to know the precise time.

Regardless of your sailing technologywithout the proper measurementyou were lost

We are not promising a perfect map of the new world

But it should be more like this

than this

Periplus of HannoCourtesy of Heidelberg University

and we will show you what we know how to measure.

APIS

There are more niches today than we’ve seen before, so we need to borrow from nature

If we start with an API we can explore all the niches around our business

Visualization by Apigee

The leaders of today’s Internet

clearly understand this mechanism.

They understand that the distribution model for value has changed in the Internet era.

ConsumerRetail StoreProducer

Packaged Goods

Internet Services

ConsumerDeveloperProvider App

Developers took their APIs and explored the niches for them

The providers and the developers both benefited from this adaptation

Suddenly this seems obvious to everyone.

Data from Programmable Web

2005 2006 2007 2008 2009 2010 20110

1000

2000

3000

4000

5000

6000

Open APIs from 2005-2011

And developers are racing to pack the niches.

Data from Wikipedia

0

100,000

200,000

300,000

400,000

500,000

600,000

0

2000000000

4000000000

6000000000

8000000000

10000000000

12000000000

App Store Growth 2008-2011

Apps AvailableTotal App Down-loads

This is a sudden, material shift in competition.

It only looks gradual if you’re losing.

BUSINESS MODELS

We’ve seen punctuated equilibrium in business models over the last hundred years

Direct Sales

Specialty Store

Department Store

Indirect Sales

National Chain

Big Box Retailer

App Developer

Mobile App

Web Catalog

Device App

Web Retail

What’s the environmental stress driving the current rapid change?

The first wave of the Internet demonstrated the economic impact of web-based business models.

Previous eras of business showed a normal distribution for revenue, with most firms getting most of the revenue. In the later half of the 20th century, business model innovations focused revenue in the 2nd standard deviation above the mean. The “80:20” rule became conventional wisdom.

The HTML-driven Internet showed new business models that focused revenue in the 3rd standard deviation (examples: Amazon, EBay). Reality reflected a “95:5” rule where 5% of companies dominated the transactions and profits.

The API-driven Internet is demonstrating the next concentration of power and is reflecting a “99:1” distribution (examples: Twitter, Facebook) due to the high switching costs and effective lock-in through software.

80:20

95:5

99:1

The next wave of the Internet is demonstrating the economic impact of API-based business models.

Hardt’s Theorem: The Internet Power Law

But you need to tackle it in a way that fits your business

Platform Partner

1st Party

Open

Open Open

These are complementary and distinct.Open is different for each one.

1st Party Apps

Partner Apps

Platform Ecosystem

1st party is about offering direct access to your core business via apps that you make or contract out

1st party

Here open means all the business is accessible to internal developers and contract specialists

1st party

Partner is about enabling directed development of apps that extend your business model towards your business partners

partner

Here open means existing partners have access to your business via APIs and can innovate asynchronously

partner

Platform is about enabling unknown developers to build brand new apps and businesses that will surprise and inform you

platform

Here open means enabling business models and allowing developers to support each other at massive scale

platform

Open is attractive

open

Open is Biz Dev 2.0

open

Platform Partner

1st Party

Open

Open Open

Open lets you navigate across the possible business models when your first model doesn’t work as planned

open

Platform Partner

1st Party

Open

Open Open

To get your API strategy properly grounded

John MusserProgrammable Web

“

But how?

Let’s break it down

Establish Target Segments

Engage Developer Channel

Set Industry Goal

An API should extend your core businessinto a new part of the market

target segments

Your core business already has key performance indicators

target segments

So apply your KPIs to the new market segment you’re targeting with your APIs

target segments

What is the market impact you need to create in order to succeed as a business?

target segments

What does the target segment need that it is not getting from you today?

target segments

The answer will be the foundation of your API strategy.

target segments

In most cases the channel for your API will be developers, but what do they need?

developer channel

A profit motive.

developer channel

Here are the leading profit models for developers today

developer channel

App sales

developer channel

In-app purchases

developer channel

Affiliate royalty

developer channel

Your advertising spend

developer channel

Market awareness of their offering

developer channel

If you don’t know where you’re going, you definitely won’t get there

industry goal

Partnerships and platform businesses are very different things.

industry goal

Partnerships are formed to serve a known set of entities

industry goal

A partner API should be traceable to each partner’s relationship

industry goal

And support end-to-end business processes

industry goal

A platform exists to create massive and unpredictable opportunities

industry goal

All your technology, support, and community decisions will be about surviving the scale of adoption

industry goal

That’s the strategy dimension.

The execution dimension is what you already know.

Planning.

Management.

Organization.

Putting all this in context gives us a map for our API strategy

Planning Management Organization

Target Segment(s)

Define market segment in detail including size and

user persona; specify API profile needed to satisfy top use cases for each

target segment

Establish KPI targets, traceability and dashboard

Business-led

Segment-oriented workstreams

Engage Channel

Specify business model and marketing driver for

the channel that will reach each target segment

Establish developer adoption targets,

developer marketing and channel actions

(community site, events, and communication)

Channel-led

Community, developer, and business development

workstreams

Industry Goal

Specify roadmap of API deliverables, mechanics, integration, and business

process to meet target segment needs

Implement API roadmap, adjust and report on iteration cycle, and

establish alpha developer team

Engineering-led

API, infrastructure, and developer support

workstreams

The instruments will be your KPIs and your core API metrics: performance and adoption

ARCHITECTURE

Computing

Mainframe

Minicomputer

Integrated

PersonalComputer

Smartphone

Connected Devices

Website

Client/Server

Web App

DCOM

Distributed

CORBA

N-tier

Chris AndersonWired Magazine

“The Web is Dead. Long Live the Internet.

The Web is Dead. Long Live APIs!

Twitter Traffic in 2010

Twitter APITwitter.com

Netflix traffic distribution is nearly the same

The majority of traffic comes from API-driven connected devices.

Like Columbus, Netflix started with a map of the coastline

Build an open API as a platformand let a thousand flowers bloom

But they had left the coastline far behind

And the instruments indicated that there were fewer flowers than expected

Netflix API Requests by Segment

Netflix DevicesOpen API De-velopers

But partners started building apps for connected devices and the business took off

XBox

PS3

Wii

LG TVs

Apple TV

iPad

iPhone

Roku

Samsung TVs

Architecture should reflect the business model

So Netflix has drawn the following map

XBox

PS3

Wii

Google TV

Apple TV

iPad

iPhone

Roku

LG TVs Samsung TVs

Instruments show that API traffic has grown tremendously in a short time

Growth of Netflix API

Jan-10

Feb-10

Mar-1

0

Apr-10

May-1

0Jun-10

Jul-10

Aug-10

Sep-10

Oct-10

Nov-10

Dec-10

Jan-110

5

10

15

20

25M

onth

ly R

eque

sts

in B

illio

ns

20,000,000,000 API requests per month.

Is that a cause for celebration?

Or for concern?

When you’re navigating uncharted waters,speed is not your friend.

Perhaps it’s time to slow down and avoid risking unknown reefs.

Navigating this growth challenge means that the next API revision will focus on reducing overall traffic.

Part of this redesign is reviewing conventions

Punctuation: REST

ProgrammableWeb

REST, as long as it makes sense for the business

Don’t be afraid to break convention

Tiered architecture helps you navigate different problems with agility

Recommendations

User Info

Similar Movies

Movie Metadata

Ratings

Viewing History

DataNormalization

&Resiliency

User Service

R12n Service

Similar Movie Service

USER API

iPhone Wrapper

Wii Wrapper

Xbox Wrapper

PS3 Wrapper

Roku Wrapper

Apple TV Wrapper

iPad Wrapper

PC / Mac Wrapper

TiVo Wrapper

Source Data Layer

API Repository Layer

API Layer Wrapper Layer

App LayerWeb Service Layer

SHARED

API INTERFACES

Shared Layer

Model Controller View

UN

IFIED

LIST/TITLE API

Recommendations

User Info

Similar Movies

Movie Metadata

Ratings

Viewing History

DataNormalization

&Resiliency

User Service

R12n Service

Similar Movie Service

USER API

iPhone Wrapper

Wii Wrapper

Xbox Wrapper

PS3 Wrapper

Roku Wrapper

Apple TV Wrapper

iPad Wrapper

PC / Mac Wrapper

TiVo Wrapper

Source Data Layer

API Repository Layer

API Layer Wrapper Layer

App LayerWeb Service Layer

SHARED

API INTERFACES

Shared Layer

Flexible Stable Agile

UN

IFIED

LIST/TITLE API

Server architecture should support both crests and troughs of the waves of demand

Instance Architecture Based on Specialization

List CreationDependency

Service

API METADATA CACHING LAYER

METADATA SERVICE

MetaData

Dependency Service

ELASTIC INSTANCE LAYER

Instance Architecture Based on Specialization

List CreationDependency

Service

API METADATA CACHING LAYER

METADATA SERVICE

MetaData

Dependency Service

ELASTIC INSTANCE LAYER

Handles Request/Response

Caches Dependency Data

Populates and Manages Cache

Map out your usage patterns and cache your data accordingly

Vertical Document-Based Caching

Vertical Caching

Netf

lix Full Movie

Horizontal Caching

Horizontal Caching: Netflix Basic Data

Combining horizontal and vertical caching may be the best approach when building for multiple geographies

Two-Dimensional Caching

Design for The Audience You Want

If your current audience is this size

Then design your system for an audience that is this size

You don’t have to implement for the dream audience

But you should design systems to scale to the dream audience

Versioning APIs is the default approach

In many cases, it is appropriate

Versioning means supporting multiple applications

All of which do the same thing, basically

Version-less API?1.0

1.5

2.0

api.NEXT

Today

3.0?

4.0?

5.0?

If possible, go versionless

Extend your API by extending data types

Addition is not version-worthy

Better to be incomplete than inaccurate

Withhold implementation if you are unsure

Don’t forget to support your API

With technology, people and management

With APIs emerging, we need better tools help us navigate

Service Level Agreements (SLA)

• Set reasonable SLAs– Will vary based on API– Communicate them to customers

• Set up monitors– Monitor average latencies at various percentiles– Monitor error rates and types– Trigger alerts when SLAs get broken

• Establish escalation and response policies– Who responds?– What is the escalation plan?

Use metrics to guide your decisions

They provide better navigation than anything else

Metrics / Business IntelligenceRequest-Based Metrics (default)

– What endpoints were called?– What parameters were passed in?

Response-Based Metrics– What was delivered?– If applicable, what was presented?– Was the response valid?

Request Trace– What underlying systems were called?– How did they respond?

System Health– Are the systems meeting uptime SLAs?– Are they performing with reasonable latencies?

Metrics (continued)

They can affect the future design of the API

They can inform strategic business decisions

They can reveal rogue usage

They can reveal errors in the system

DATA

Data

Flat file

Mainframe

Silos

Caching DBs

Domain-specificData APIs

RDBMS

Data APIData

Warehousing

Shared

Private Cloud DBs

Sharing your data, from silos to social

Mainframe Databases Middleware APIs

evolution in data sharing

App Org Cross-org Cross-business

loss of control (once it’s out there)

only recourse is legal

enforcement is expensive (time and reputation)

challenges in data sharing

case study: opening up movie metadata

optimize data delivery

enable larger downloads for fewer queries

more liberal retention policies fewer refreshes, less breakage

push incremental updates

limited access to richer queries

Highest rated titles from the 80's and their awards:

http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 and AverageRating gt 4&$expand=Awards

[{ "id": null, "name": null, "type": "/dining/restaurant", "/business/business_location/address": [{ "street_address": [], "citytown": { "id": "/en/toronto" } }], "cuisine": [{ "/dining/cuisine/region_of_origin": [{ "!/film/film/featured_film_locations": [{ "id": "/en/the_italian_job" }] }]

Find Toronto restaurants with cuisine from a filming location of "The Italian Job"

What data APIs should your business be using?

In

CLOSING

Periplus of HannoCourtesy of Heidelberg University

THANK YOUQuestions and ideas to:

@michaelhart@daniel_jacobson@sramji