Netflix: Amazon S3 & Amazon Elastic MapReduce to Monitor at Gigascale (BDT302) | AWS re:Invent 2013

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

Roy Rapoport

November 14, 2013

Deft Data at Netflix:Using Amazon S3 and Amazon Elastic

Friday, November 15, 13

A Word About Me …


• About 20 years in technology

A Word About Me …


• About 20 years in technology• Systems engineering, networking,

software development, QA, release management

A Word About Me …




• Time at Netflix: 1599 days

A Word About Me …




• Time at Netflix: 1599 days

A Word About Me …

(4y:4m:15d)




• Time at Netflix: 1599 days • Before at Netflix: Service Delivery in

the IT/Ops, troubleshooter, Builder of Python Things[tm]

A Word About Me …

(4y:4m:15d)






• Current role: Cloud Monitoring

A Word About Me …

(4y:4m:15d)






• Current role: Cloud Monitoring•We build platforms

A Word About Me …

(4y:4m:15d)






• Current role: Cloud Monitoring•We build platforms•Sometimes we make them easy to use

A Word About Me …

(4y:4m:15d)


A Word About Netflix …


A Word About Netflix …Just the Stats


• 16 years



• 16 years• 2000+ employees



• 16 years• 2000+ employees• 40 million users



• 16 years• 2000+ employees• 40 million users• 5x10^9 hours/quarter



A Word About Netflix …Freedom and Responsibility Culture


• Optimize speed of innovationConstrain availabilityCost will be what cost will be




• Hire smart (experienced) peopleGet out of their way




• Hire smart (experienced) peopleGet out of their way

• Anti-process bias



A Word About Netflix …Technology and Operations



•Service Oriented Architecture



•Service Oriented Architecture•Decentralized Operations. You




•Build




•Build•Test




•Build•Test•Deploy




•Build•Test•Deploy•Set up alerting and monitoring




•Build•Test•Deploy•Set up alerting and monitoring•Wake up at 2AM



• AWS-based for 100% of streaming*

Technology and Operations



• AWS-based for 100% of streaming*• Huge expansion





• Customer Growth





• Customer Growth• New markets





• Customer Growth• New markets• Metrics



In the Old Days …

Copyright: State Library of Victoria Collections. CC Attribution 2.0 License


In the Old Days …Our Old Alerting System




• Enterprise IT Solution


Copyright USAID Microlinks. CC Attribution 2.0 License



• Enterprise IT Solution• Managed by the Enterprise IT Alerting People


Copyright USAID Microlinks. CC Attribution 2.0 License



• Enterprise IT Solution• Managed by the Enterprise IT Alerting People• File Tickets


Copyright: http://www.flickr.com/photos/s_w_ellis

CC Attribution 2.0 License


http://www.flickr.com/photos/s_w_ellis

http://www.flickr.com/photos/s_w_ellis


• Enterprise IT Solution• Managed by the Enterprise IT Alerting People• File Tickets• Send alerts to NOC




• Enterprise IT Solution• Managed by the Enterprise IT Alerting People• File Tickets• Send alerts to NOC• Completely separate from telemetry system



In the Old Days …


In the Old Days …


In the Old Days …


In the Old Days …

Our Old Telemetry System


In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin


In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin• Loved by developers


In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol


In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage


In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage• Mostly Perl

Copyright: http://www.flickr.com/photos/acme



In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage• Mostly Perl• Datacenter-bound (and limited)


In the Old Days …


In the Old Days …


• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage• Mostly Perl• Datacenter-bound (and limited)• Starting to falter under metrics growth


Speaking of Growth


Speaking of Growth

By way of comparison


Speaking of Growth

By way of comparison • Every person in the world• twice


Speaking of Growth

By way of comparison • Every person in the world• twice•Every smartphone in the

world• ten times


Copyright: http://www.flickr.com/photos/76651030@N02/


So We Built Something Better


So We Built Something Better

UI

Atlas Epic CloudWatch

UI Layer Fronts Multiple Systems




So We Built Something Better UA E CClear Regional Separation

• And aggregation

global

us-east-1 us-west-1 us-west-2 eu-west-1




So We Built Something Better UA E C

glus us us e

Localized Node/Metric Identification

Before:

I think You’re Bob

Here’s a metric!

OK!

I’m Bob. Here’s

a metric!

Now:





glus us us e



glus us us eWhat’s a Metric?




• com.netflix.eds.nccp.successful.requests.uiversion.nccprt-authorization.devtypid-101.clver-PHL_0AB.uiver-UI_169_mid.geo-US





• 256 characters aren’t enough!





• 256 characters aren’t enough!• This is better:






nf.ami ami-aa5166ef






nf.app wpnf.ami ami-aa5166ef






nf.app wpnf.cluster wp-batch

nf.ami ami-aa5166ef






nf.asg wp-batch-v163


nf.ami ami-aa5166ef








nf.ami ami-aa5166ef

nf.country us








nf.ami ami-aa5166ef

nf.country us

nf.node i-097c0e52








nf.ami ami-aa5166ef

nf.country us

nf.region us-west-1nf.node i-097c0e52








nf.ami ami-aa5166ef

nf.country us

nf.region us-west-1nf.zone us-west-1b

nf.node i-097c0e52








nf.ami ami-aa5166ef

nf.country usclass nccp


nf.node i-097c0e52








nf.ami ami-aa5166ef



nf.node i-097c0e52

type request








nf.ami ami-aa5166ef



nf.node i-097c0e52

type request

uiversion UI_169_mid








nf.ami ami-aa5166ef



nf.node i-097c0e52

type request

action authorizationuiversion UI_169_mid








nf.ami ami-aa5166ef



nf.node i-097c0e52

type request

action authorizationdevtype 101









nf.ami ami-aa5166ef



nf.node i-097c0e52

type requestclver PHL_0AB










nf.ami ami-aa5166ef



nf.node i-097c0e52

type requestclver PHL_0AB



geo us



glus us us e

Copyright: Kurt Moerman




glus us us e

Powerful queries





glus us us e

Powerful queries• Make the complex possible





glus us us e

Powerful queries• Make the complex possible• Make the simple … sort of hard





glus us us e




glus us us e


http://atlas/api/v1/graph?q=nf.region,us-west-1,:eq,nf.app,employeeinfo,:eq,:and,name,employeeinfo_api,:eq,:and,:sum&e=now-5m&s=e-3h



glus us us e




glus us us e


http://atlas/api/v1/graph?q=nf.region,us-west-1,:eq,nf.app,employeeinfo,:eq,:and,name,employeeinfo_api,:eq,:and,:sum,(,nf.zone,),:by&e=now-5m&s=e-3h



glus us us e




glus us us e


http://atlas/api/v1/graph?q=sps,nf.cluster,(,nccp-legacy,nccp-modern,),:in,nccprt,(,NCCPLicense,com_netflix_streaming_nccp_request_license,),:in,:and,stat,SuccessfulRequests,:eq,:and,device.rollup,3ds,:eq,:and,:sum,:set,entering_trough,sps,:get,1h,:offset,0.95,:mul,sps,:get,:gt,:set,smoothed,sps,:get,10,0.1,0.02,:des,:set,low_volume,smoothed,:get,-0.005,:mul,0.1,:add,:set,mid_volume,smoothed,:get,-0.00125,:mul,0.1,:add,:set,base,0.06,:set,min_pct,1,smoothed,:get,20,:lt,low_volume,:get,:mul,smoothed,:get,80,:lt,mid_volume,:get,:mul,:add,entering_trough,:get,0.05,:mul,:add,base,:get,:add,:sub,10,0.1,0.02,:des,:set,sps,:get,$(device.rollup)SPS,:legend,min_pct,:get,smoothed,:get,:mul,lowerbound,:legend,sps,:get,min_pct,:get,smoothed,:get,:mul,:lt,5,:rolling-count,2,:ge,:vspan,60,:alpha,$(device.rollup),:legend


http://atlas/api/v1/graph?q=sps,nf.cluster

http://atlas/api/v1/graph?q=sps,nf.cluster


glus us us e



glus us us eRidiculous Read Volume:

• Engage

• Graphs and Dashboards




• Engage

• Graphs and Dashboards• Alerting




• Engage

• Graphs and Dashboards• Alerting• Automated Canaries




• Engage

• Graphs and Dashboards• Alerting• Automated Canaries• Capacity Analytics




• Engage

• Graphs and Dashboards• Alerting• Automated Canaries• Capacity Analytics• Special Projects




• Engage

• Graphs and Dashboards• Alerting• Automated Canaries• Capacity Analytics• Special Projects• BI



glus us us e

backendinstancebackend

instancebackendinstancebackend


instancebackendinstance

regionalendpoint

global endpoint



glus us us e

clientinstance





regionalendpoint

global endpoint



glus us us e

clientinstance

publishcluster





regionalendpoint

global endpoint


Amazon S3


glus us us e

clientinstance

publishcluster





regionalendpoint

global endpoint


Amazon S3


glus us us e

clientinstance

publishcluster





pollercluster

regionalendpoint

global endpoint


Amazon S3


glus us us e

clientinstance

publishclusterm





pollercluster

regionalendpoint

global endpoint


Amazon S3


glus us us e

clientinstance

publishclusterm





pollercluster

m

regionalendpoint

global endpoint


Amazon S3


glus us us e

clientinstance

publishclusterm





pollercluster

m

m

regionalendpoint

global endpoint


That Sounds Great!


That Sounds Great!Surely there are no problems

Copyright: http://www.flickr.com/photos/lainetrees/




•Speed is hard



•Speed is hard•Speed at volume is harder



•Speed is hard•Speed at volume is harder•We looked at spinning disks



•Speed is hard•Speed at volume is harder•We looked at spinning disks•Memory’s the way to go



•Speed is hard•Speed at volume is harder•We looked at spinning disks•Memory’s the way to go•m2.4xlarge



•Speed is hard•Speed at volume is harder•We looked at spinning disks•Memory’s the way to go•m2.4xlarge•This is operational data




•People want it available, fast




•People want it available, fast•Operations have short memories




•People want it available, fast•Operations have short memories

20,160 m2.4xlarge$32,094,720 upfront$8,005,939/month

per regionwith no redundancy




•People want it available, fast•Operations have short memories Copyright: http://www.flickr.com/photos/amenk/



That Doesn’t Sound Great!


That Doesn’t Sound Great!•If only we could reduce it …


That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?


That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?•Our granularity is two-dimensional

Step size (time)

Dim

ensi

onal

ity (t

ags)


That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?•Our granularity is two-dimensional•We can reduce on either dimension

Step size (time)

Dim

ensi

onal

ity (t

ags)


That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?•Our granularity is two-dimensional•We can reduce on either dimension•Some tags make sense for very rapid reduction

Step size (time)

Dim

ensi

onal

ity (t

ags)



•Hystrix

Step size (time)

Dim

ensi

onal

ity (t

ags)



•Hystrix•nf.node

Step size (time)

Dim

ensi

onal

ity (t

ags)




•Sometimes a lot (vhs) Step size (time)

Dim

ensi

onal

ity (t

ags)




•Sometimes a lot (vhs)•Sometimes a little (Cassandra)

Step size (time)

Dim

ensi

onal

ity (t

ags)


A Reductive Approach


A Reductive Approach•For a series of values, reduce and keep:



•minimum



•minimum•maximum



•minimum•maximum•total



•minimum•maximum•total•count




•Example:




•Example:•3,5,9,14,20: min 3, max 20, tot 51, count 5





•Allows for sense of scale





•Allows for sense of scale•Allows for arbitrary further reduction w/o loss of precision


Reduction: Policy

Copyright: http://www.flickr.com/photos/bagaball/



Reduction: Policy

•Policy-driven EMR engine




Reduction: Policy

•Policy-driven EMR engine•Four possible actions




Reduction: Policy


•preserve




Reduction: Policy


•preserve•drop




Reduction: Policy


•preserve•drop•consolidate




Reduction: Policy


•preserve•drop•consolidate•rollup




Reduction: Policy

{ "rules" : [ { "operations" : [{"op" : "drop"}], "query" : "nf.app,api,:eq,class,(,LastMinuteFailRatio,SLA,NetflixSimpleDBService,),:in,:and" }, { "operations" : [{ “config" : { "keys" : [ "nf.node", "device", "nf.country" ] }, "op" : “rollup" }], "query" : ":true" } ]}


Amazon EMR

clientinstance

publishcluster

Amazon S3

pollercluster

regionalendpoint

global endpoint

6Hcluster

EMRDriver

4Dcluster

18Dcluster

Historicalcluster

metrics

query

responsemetrics

metrics metrics

1

2 3

45 5

5


Amazon EMR

clientinstance

publishcluster

Amazon S3

pollercluster

regionalendpoint

global endpoint

6Hcluster

EMRDriver

4Dcluster

18Dcluster

Historicalcluster

as-neededcluster

as-neededcluster

as-neededcluster

metrics

query

responsemetrics

metrics metrics

1

2 3

45 5

5


Reduction: Benefits

Copyright: http://www.flickr.com/photos/dr_pete/



Reduction: Benefits

•Indefinite storage in Amazon S3




Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:



Unlocked


Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics



Unlocked


Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days



Unlocked


Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*



Unlocked


Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*•Not in critical path for visibility SLA



Unlocked


Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*•Not in critical path for visibility SLA•Firewalls accidental metric explosions



Unlocked


Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*•Not in critical path for visibility SLA•Firewalls accidental metric explosions•Huge efficiency gains



Unlocked


Reduction: Efficiency

Copyright: http://www.flickr.com/photos/sebrenner/




6H 4D 18D HISTORY

TimeHorizon

Size

Instances Per Hour

% Reduction

6 Hours 4 Days 18 Days 3 Months

600 512 180 12

100 5 0 0

0 95 100 100


Previews


Previews

Copyright: http://www.flickr.com/photos/creativealan/



Previews

•Self-service for special requests




Previews

•Self-service for special requests•Different instance types




Previews


•cr1.8xlarge




Previews


•cr1.8xlarge•hi1.4xlarge




Previews


•cr1.8xlarge•hi1.4xlarge

•Multi-tiered metric visibility




Growth Redux


5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

(M)

met

rics

2 2.5 10

Growth Redux


5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

(M)

met

rics

2 2.5 10 15

Growth Redux


5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

(M)

met

rics

2 2.5 10 15 18 30 55 90212

728

Growth Redux


5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

(M)

met

rics

2 2.5 10 15 18 30 55 90212

728

1,200

Growth Redux


Growth Redux


And a Last Word About Costs



•Priorities Reminder



•Priorities Reminder•Speed of Innovation



•Priorities Reminder•Speed of Innovation•Availability



•Priorities Reminder•Speed of Innovation•Availability•Cost




•Never intended to lower costs




•Never intended to lower costs•Cloud migration




•Never intended to lower costs•Cloud migration•Additional features




•Never intended to lower costs•Cloud migration•Additional features•Massive Performance


EMR

FTWFriday, November 15, 13

Please give us your feedback on this presentation

As a thank you, we will select prize winners daily for completed surveys!

BDT302 Thank You


Netflix: Amazon S3 & Amazon Elastic MapReduce to Monitor at Gigascale (BDT302) | AWS re:Invent 2013

Technology

Transcript of Netflix: Amazon S3 & Amazon Elastic MapReduce to Monitor at Gigascale (BDT302) | AWS re:Invent 2013