Netflix: Amazon S3 & Amazon Elastic MapReduce to Monitor at Gigascale (BDT302) | AWS re:Invent 2013

Roy Rapoport

November 14, 2013

Deft Data at Netflix:Using Amazon S3 and Amazon Elastic

Friday, November 15, 13

A Word About Me …

• About 20 years in technology

A Word About Me …

• About 20 years in technology• Systems engineering, networking,

software development, QA, release management

A Word About Me …

• Time at Netflix: 1599 days

A Word About Me …

• Time at Netflix: 1599 days

A Word About Me …

(4y:4m:15d)

• Time at Netflix: 1599 days • Before at Netflix: Service Delivery in

the IT/Ops, troubleshooter, Builder of Python Things[tm]

A Word About Me …

(4y:4m:15d)

• Current role: Cloud Monitoring

A Word About Me …

(4y:4m:15d)

• Current role: Cloud Monitoring•We build platforms

A Word About Me …

(4y:4m:15d)

• Current role: Cloud Monitoring•We build platforms•Sometimes we make them easy to use

A Word About Me …

(4y:4m:15d)

A Word About Netflix …

A Word About Netflix …Just the Stats

• 16 years

• 16 years• 2000+ employees

• 16 years• 2000+ employees• 40 million users

• 16 years• 2000+ employees• 40 million users• 5x10^9 hours/quarter

A Word About Netflix …Freedom and Responsibility Culture

• Optimize speed of innovationConstrain availabilityCost will be what cost will be

• Hire smart (experienced) peopleGet out of their way

• Anti-process bias

A Word About Netflix …Technology and Operations

•Service Oriented Architecture

•Service Oriented Architecture•Decentralized Operations. You

•Build

•Build•Test

•Build•Test•Deploy

•Build•Test•Deploy•Set up alerting and monitoring

•Build•Test•Deploy•Set up alerting and monitoring•Wake up at 2AM

• AWS-based for 100% of streaming*

Technology and Operations

• AWS-based for 100% of streaming*• Huge expansion

• Customer Growth

• Customer Growth• New markets

• Customer Growth• New markets• Metrics

In the Old Days …

Copyright: State Library of Victoria Collections. CC Attribution 2.0 License

In the Old Days …Our Old Alerting System

• Enterprise IT Solution

Copyright USAID Microlinks. CC Attribution 2.0 License

• Enterprise IT Solution• Managed by the Enterprise IT Alerting People

Copyright USAID Microlinks. CC Attribution 2.0 License

• Enterprise IT Solution• Managed by the Enterprise IT Alerting People• File Tickets

Copyright: http://www.flickr.com/photos/s_w_ellis

CC Attribution 2.0 License

• Enterprise IT Solution• Managed by the Enterprise IT Alerting People• File Tickets• Send alerts to NOC

• Enterprise IT Solution• Managed by the Enterprise IT Alerting People• File Tickets• Send alerts to NOC• Completely separate from telemetry system

In the Old Days …

Our Old Telemetry System

In the Old Days …

• Spare-time effort by a lone sysadmin

In the Old Days …

• Spare-time effort by a lone sysadmin• Loved by developers

In the Old Days …

• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol

In the Old Days …

• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage

In the Old Days …

• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage• Mostly Perl

Copyright: http://www.flickr.com/photos/acme

In the Old Days …

• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage• Mostly Perl• Datacenter-bound (and limited)

In the Old Days …

• Spare-time effort by a lone sysadmin• Loved by developers• Custom TCP protocol• RRD file back-end storage• Mostly Perl• Datacenter-bound (and limited)• Starting to falter under metrics growth

Speaking of Growth

By way of comparison

Speaking of Growth

By way of comparison • Every person in the world• twice

Speaking of Growth

By way of comparison • Every person in the world• twice•Every smartphone in the

world• ten times

Copyright: http://www.flickr.com/photos/76651030@N02/

So We Built Something Better

Atlas Epic CloudWatch

UI Layer Fronts Multiple Systems

So We Built Something Better UA E CClear Regional Separation

• And aggregation

global

us-east-1 us-west-1 us-west-2 eu-west-1

So We Built Something Better UA E C

glus us us e

Localized Node/Metric Identification

Before:

I think You’re Bob

Here’s a metric!

I’m Bob. Here’s

a metric!

glus us us e

glus us us eWhat’s a Metric?

• com.netflix.eds.nccp.successful.requests.uiversion.nccprt-authorization.devtypid-101.clver-PHL_0AB.uiver-UI_169_mid.geo-US

• 256 characters aren’t enough!

• 256 characters aren’t enough!• This is better:

nf.ami ami-aa5166ef

nf.app wpnf.ami ami-aa5166ef

nf.app wpnf.cluster wp-batch

nf.ami ami-aa5166ef

nf.asg wp-batch-v163

nf.ami ami-aa5166ef

nf.country us

nf.ami ami-aa5166ef

nf.country us

nf.node i-097c0e52

nf.ami ami-aa5166ef

nf.country us

nf.region us-west-1nf.node i-097c0e52

nf.ami ami-aa5166ef

nf.country us

nf.region us-west-1nf.zone us-west-1b

nf.node i-097c0e52

nf.ami ami-aa5166ef

nf.country usclass nccp

nf.node i-097c0e52

nf.ami ami-aa5166ef

nf.node i-097c0e52

type request

nf.ami ami-aa5166ef

nf.node i-097c0e52

type request

uiversion UI_169_mid

nf.ami ami-aa5166ef

nf.node i-097c0e52

type request

action authorizationuiversion UI_169_mid

nf.ami ami-aa5166ef

nf.node i-097c0e52

type request

action authorizationdevtype 101

nf.ami ami-aa5166ef

nf.node i-097c0e52

type requestclver PHL_0AB

nf.ami ami-aa5166ef

nf.node i-097c0e52

type requestclver PHL_0AB

geo us

glus us us e

Copyright: Kurt Moerman

glus us us e

Powerful queries

glus us us e

Powerful queries• Make the complex possible

glus us us e

Powerful queries• Make the complex possible• Make the simple … sort of hard

glus us us e

http://atlas/api/v1/graph?q=nf.region,us-west-1,:eq,nf.app,employeeinfo,:eq,:and,name,employeeinfo_api,:eq,:and,:sum&e=now-5m&s=e-3h

glus us us e

http://atlas/api/v1/graph?q=nf.region,us-west-1,:eq,nf.app,employeeinfo,:eq,:and,name,employeeinfo_api,:eq,:and,:sum,(,nf.zone,),:by&e=now-5m&s=e-3h

glus us us e

http://atlas/api/v1/graph?q=sps,nf.cluster,(,nccp-legacy,nccp-modern,),:in,nccprt,(,NCCPLicense,com_netflix_streaming_nccp_request_license,),:in,:and,stat,SuccessfulRequests,:eq,:and,device.rollup,3ds,:eq,:and,:sum,:set,entering_trough,sps,:get,1h,:offset,0.95,:mul,sps,:get,:gt,:set,smoothed,sps,:get,10,0.1,0.02,:des,:set,low_volume,smoothed,:get,-0.005,:mul,0.1,:add,:set,mid_volume,smoothed,:get,-0.00125,:mul,0.1,:add,:set,base,0.06,:set,min_pct,1,smoothed,:get,20,:lt,low_volume,:get,:mul,smoothed,:get,80,:lt,mid_volume,:get,:mul,:add,entering_trough,:get,0.05,:mul,:add,base,:get,:add,:sub,10,0.1,0.02,:des,:set,sps,:get,$(device.rollup)SPS,:legend,min_pct,:get,smoothed,:get,:mul,lowerbound,:legend,sps,:get,min_pct,:get,smoothed,:get,:mul,:lt,5,:rolling-count,2,:ge,:vspan,60,:alpha,$(device.rollup),:legend

glus us us e

glus us us eRidiculous Read Volume:

• Engage

• Graphs and Dashboards

• Engage

• Graphs and Dashboards• Alerting

• Engage

• Graphs and Dashboards• Alerting• Automated Canaries

• Engage

• Graphs and Dashboards• Alerting• Automated Canaries• Capacity Analytics

• Engage

• Graphs and Dashboards• Alerting• Automated Canaries• Capacity Analytics• Special Projects

• Engage

• Graphs and Dashboards• Alerting• Automated Canaries• Capacity Analytics• Special Projects• BI

• Engage

glus us us e

backendinstancebackend

instancebackendinstancebackend

instancebackendinstance

regionalendpoint

global endpoint

glus us us e

clientinstance

regionalendpoint

global endpoint

glus us us e

clientinstance

publishcluster

regionalendpoint

global endpoint

Amazon S3

glus us us e

clientinstance

publishcluster

regionalendpoint

global endpoint

Amazon S3

glus us us e

clientinstance

publishcluster

pollercluster

regionalendpoint

global endpoint

Amazon S3

glus us us e

clientinstance

publishclusterm

pollercluster

regionalendpoint

global endpoint

Amazon S3

glus us us e

clientinstance

publishclusterm

pollercluster

regionalendpoint

global endpoint

Amazon S3

glus us us e

clientinstance

publishclusterm

pollercluster

regionalendpoint

global endpoint

Amazon S3

glus us us e

clientinstance

publishclusterm

pollercluster

regionalendpoint

global endpoint

That Sounds Great!

That Sounds Great!Surely there are no problems

Copyright: http://www.flickr.com/photos/lainetrees/

•Speed is hard

•Speed is hard•Speed at volume is harder

•Speed is hard•Speed at volume is harder•We looked at spinning disks

•Speed is hard•Speed at volume is harder•We looked at spinning disks•Memory’s the way to go

•Speed is hard•Speed at volume is harder•We looked at spinning disks•Memory’s the way to go•m2.4xlarge

•Speed is hard•Speed at volume is harder•We looked at spinning disks•Memory’s the way to go•m2.4xlarge•This is operational data

•People want it available, fast

•People want it available, fast•Operations have short memories

20,160 m2.4xlarge$32,094,720 upfront$8,005,939/month

per regionwith no redundancy

•People want it available, fast•Operations have short memories Copyright: http://www.flickr.com/photos/amenk/

That Doesn’t Sound Great!

That Doesn’t Sound Great!•If only we could reduce it …

That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?

That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?•Our granularity is two-dimensional

Step size (time)

ity (t

That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?•Our granularity is two-dimensional•We can reduce on either dimension

Step size (time)

ity (t

That Doesn’t Sound Great!•If only we could reduce it …•“Reduce”? Get it? Get it?•Our granularity is two-dimensional•We can reduce on either dimension•Some tags make sense for very rapid reduction

Step size (time)

ity (t

•Hystrix

Step size (time)

ity (t

•Hystrix•nf.node

Step size (time)

ity (t

•Sometimes a lot (vhs) Step size (time)

ity (t

•Sometimes a lot (vhs)•Sometimes a little (Cassandra)

Step size (time)

ity (t

A Reductive Approach

A Reductive Approach•For a series of values, reduce and keep:

•minimum

•minimum•maximum

•minimum•maximum•total

•minimum•maximum•total•count

•Example:

•Example:•3,5,9,14,20: min 3, max 20, tot 51, count 5

•Allows for sense of scale

•Allows for sense of scale•Allows for arbitrary further reduction w/o loss of precision

Reduction: Policy

Copyright: http://www.flickr.com/photos/bagaball/

Reduction: Policy

•Policy-driven EMR engine

Reduction: Policy

•Policy-driven EMR engine•Four possible actions

Reduction: Policy

•preserve

Reduction: Policy

•preserve•drop

Reduction: Policy

•preserve•drop•consolidate

Reduction: Policy

•preserve•drop•consolidate•rollup

Reduction: Policy

{ "rules" : [ { "operations" : [{"op" : "drop"}], "query" : "nf.app,api,:eq,class,(,LastMinuteFailRatio,SLA,NetflixSimpleDBService,),:in,:and" }, { "operations" : [{ “config" : { "keys" : [ "nf.node", "device", "nf.country" ] }, "op" : “rollup" }], "query" : ":true" } ]}

Amazon EMR

clientinstance

publishcluster

Amazon S3

pollercluster

regionalendpoint

global endpoint

6Hcluster

EMRDriver

4Dcluster

18Dcluster

Historicalcluster

metrics

responsemetrics

metrics metrics

Amazon EMR

clientinstance

publishcluster

Amazon S3

pollercluster

regionalendpoint

global endpoint

6Hcluster

EMRDriver

4Dcluster

18Dcluster

Historicalcluster

metrics

responsemetrics

metrics metrics

Amazon EMR

clientinstance

publishcluster

Amazon S3

pollercluster

regionalendpoint

global endpoint

6Hcluster

EMRDriver

4Dcluster

18Dcluster

Historicalcluster

as-neededcluster

metrics

responsemetrics

metrics metrics

Reduction: Benefits

Copyright: http://www.flickr.com/photos/dr_pete/

Reduction: Benefits

•Indefinite storage in Amazon S3

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:

Unlocked

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics

Unlocked

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days

Unlocked

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*

Unlocked

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*•Not in critical path for visibility SLA

Unlocked

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*•Not in critical path for visibility SLA•Firewalls accidental metric explosions

Unlocked

Reduction: Benefits

•Indefinite storage in Amazon S3•Fear of commitment achievement:•Can be aggressive about hiding metrics•High granularity for special days•Automated for regular operations*•Not in critical path for visibility SLA•Firewalls accidental metric explosions•Huge efficiency gains

Unlocked

Reduction: Efficiency

Copyright: http://www.flickr.com/photos/sebrenner/

6H 4D 18D HISTORY

TimeHorizon

Instances Per Hour

% Reduction

6 Hours 4 Days 18 Days 3 Months

600 512 180 12

100 5 0 0

0 95 100 100

6H 4D 18D HISTORY

TimeHorizon

Instances Per Hour

% Reduction

600 512 180 12

100 5 0 0

0 95 100 100

6H 4D 18D HISTORY

TimeHorizon

Instances Per Hour

% Reduction

600 512 180 12

100 5 0 0

0 95 100 100

6H 4D 18D HISTORY

TimeHorizon

Instances Per Hour

% Reduction

600 512 180 12

100 5 0 0

0 95 100 100

6H 4D 18D HISTORY

TimeHorizon

Instances Per Hour

% Reduction

600 512 180 12

100 5 0 0

0 95 100 100

Previews

Copyright: http://www.flickr.com/photos/creativealan/

Previews

•Self-service for special requests

Previews

•Self-service for special requests•Different instance types

Previews

•cr1.8xlarge

Previews

•cr1.8xlarge•hi1.4xlarge

Previews

•cr1.8xlarge•hi1.4xlarge

•Multi-tiered metric visibility

Growth Redux

5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

2 2.5 10

Growth Redux

5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

2 2.5 10 15

Growth Redux

5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

2 2.5 10 15 18 30 55 90212

Growth Redux

5/11 8/11 9/11 1/12 4/12 8/1211/121/13 5/1310/131/14

2 2.5 10 15 18 30 55 90212

Growth Redux

And a Last Word About Costs

•Priorities Reminder

•Priorities Reminder•Speed of Innovation

•Priorities Reminder•Speed of Innovation•Availability

•Priorities Reminder•Speed of Innovation•Availability•Cost

•Never intended to lower costs

•Never intended to lower costs•Cloud migration

•Never intended to lower costs•Cloud migration•Additional features

•Never intended to lower costs•Cloud migration•Additional features•Massive Performance

FTWFriday, November 15, 13

Please give us your feedback on this presentation

As a thank you, we will select prize winners daily for completed surveys!

BDT302 Thank You

Netflix: Amazon S3 & Amazon Elastic MapReduce to Monitor at Gigascale (BDT302) | AWS re:Invent 2013

Technology

Transcript of Netflix: Amazon S3 & Amazon Elastic MapReduce to Monitor at Gigascale (BDT302) | AWS re:Invent 2013

City of Seattle - Seattle.gov Home€¦ · Early Design Guidance: 04.25.2012 4 context: vicinity map N 400 Proposed Troy Block Amazon Amazon Amazon Amazon Amazon Amazon Cascade Park

Amazon EKS - User Guide · Amazon EKS User Guide Amazon EKS Control Plane Architecture What Is Amazon EKS? Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that

Amazon Quantum Ledger Database (Amazon QLDB) - Developer … · Amazon Quantum Ledger Database (Amazon QLDB) Developer Guide Amazon QLDB Video What Is Amazon QLDB? Amazon Quantum

Amazon DocumentDB (with MongoDB compatibility) Deep Dive 1 - Am… · Aurora CommunityCommercial Amazon ElastiCache Amazon Keyspaces Amazon DocumentDB Time-series Ledger Amazon Timestream

Amazon WorkMail - User Guide - AWS Documentation...Amazon WorkMail User Guide Table of Contents What Is Amazon WorkMail?..... 1 Amazon WorkMail System Amazon WorkMail Amazon WorkMail

The Gigascale Silicon Research Center Edward A. Lee UC Berkeley The GSRC Semantics Project Tom Henzinger Luciano Lavagno Edward Lee Alberto Sangiovanni-Vincentelli.

€¦ · animoto inc amazon amazon money claim online amazon flag & bunting store fasthosts amazon animoto inc amazon - roomserve ltd fasthosts amazon cake stands.co.uk impact workwear

c 2011 Pradeep Ramachandran - University Of Illinoisrsim.cs.illinois.edu/Pubs/pradeep-ramachandran-thesis.pdfv the NSF grant numbers CCF 0541383 and CCF 0811693, the Gigascale Systems

SketchSort: An Efficient Nearest Neighbor Graph ...- Enable to store gigascale data in main memory - Speed up learning algorithms x=(0.2, -0.3, -1.3, 1.2, 2.2, …) s=10101011101010101

(BDT302) Big Data Beyond Hadoop: Running Mahout, Giraph, and R on Amazon EMR and Analyzing Results with Amazon Redshift | AWS re:Invent 2014

Minimum-Buffered Routing of Non- Critical Nets for Slew Rate and Reliability Control Supported by Cadence Design Systems, Inc. and the MARCO Gigascale.

BEAR: MITIGATING BANDWIDTH BLOAT IN GIGASCALE DRAM CACHES Chiachen Chou, Georgia Tech Aamer Jaleel, NVIDIA* Moinuddin K. Qureshi, Georgia Tech ISCA 2015.

Amazon EKS - Guía del usuario · Amazon EKS Guía del usuario Arquitectura de plano de control de Amazon EKS ¿Qué es Amazon EKS? Amazon Elastic Kubernetes Service (Amazon EKS)

Î Ð È ÈÄ Ï Ê ÉÇ Ä É À...AWS Lambda의다양한활용 Amazon API Gateway Amazon Alexa AWS IoT Amazon Kinesis Amazon SNS Amazon SES AWS Step Functions Amazon S3 Amazon DynamoDB

® 1 VLSI Design Challenges for Gigascale Integration Shekhar Borkar Intel Corp. October 25, 2005.

AWS Directory Service, Amazon WorkSpaces, Amazon WorkMail, and Amazon WorkDocs

CUSTOMER PROFILE - Care Analytics workflows instead of lengthy emails and Word documents. ... Amazon EC2, Amazon IAM, Amazon RDS, Amazon Certificate Manager, Amazon ELB, Amazon …

ANALYTICS ON AWS - Amazon Web Services, Inc. · Store Analyze Amazon Glacier Amazon S3 Amazon DynamoDB Amazon RDS, Amazon Aurora Big data portfolio –focus on choice AWS Data Pipeline

Amazon Web Services - isppref.com€¦ · Amazon RDS Amazon VPC AWS Auto Scaling AWS Elastic Load Balancing 2010 Amazon SNS AWS Identity & Access Management Amazon Route 53 2011 Amazon

Amazon ECR - User Guide · Amazon ECR User Guide Components of Amazon ECR What Is Amazon Elastic Container Registry? Amazon Elastic Container Registry (Amazon ECR) is a managed AWS