AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

67
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Matt Williams - Evangelist @ Datadog November 29, 2016 Monitoring, Hold the Infrastructure Getting the Most from AWS Lambda

Transcript of AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Page 1: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Matt Williams - Evangelist @ Datadog

November 29, 2016

Monitoring, Hold the Infrastructure

Getting the Most from AWS Lambda

Page 2: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

The Plan

Learn about the basics of AWS Lambda

Add AWS Lambda to your infrastructure

Monitoring 101 (the tl;dr version)

Monitoring AWS Lambda

Page 3: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

What is AWS Lambda

The next stage in a long progression...

physical

boxes

virtual

machines

on

physical

boxes

containers lambdaAWS

instances

Page 4: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Important AWS Lambda Concepts

TRIGGER

BASED

Page 5: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Important AWS Lambda Concepts

STATELESS

STATELESS

Page 6: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Important AWS Lambda Concepts

SERVERLESS

AS FAR AS YOU ARE CONCERNED

Page 7: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Important AWS Lambda Concepts

Each event will trigger a new Lambda function

There is a safety limit per region of 100

(contact support for changes)

Page 8: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

What is a AWS Lambda Function

• Your executable code

• No restrictions

• Priced based on execution time, quantity, and memory

Page 9: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

What can trigger AWS Lambda?

Files Amazon S3

Database Updates Amazon DynamoDB

Streams Amazon Kinesis

Messages Amazon SNS, Amazon SES

User Data & Authentication Amazon Cognito

Infrastructure Changes AWS CloudFormation

Logs, Events, Metrics Amazon CloudWatch

IoT Echo, Amazon API Gateway

Page 10: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

LambdaLambda Lambda Lambda

$goad

Page 11: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

http://benchling.engineering/crispr-aws-lambda/

Page 12: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Other Examples

Bittorrent TrackerHugely scalable using S3

https://blog.zappa.io/posts/zappa-bittorrent-

tracker

Chat BotsEasy to build

https://claudiajs.com/claudia-bot-builder.html

S3 Data LoadingSimpler than the typical Kinesis process

https://www.trek10.com/blog/serverless-

architectures-s3-data-loading/

Skills for the Amazon EchoMake one today in the hall

Page 13: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Best Practices from AWS

Write stateless code

Local vars

Permissions

Minimize startup

Monitor

Delete old functions

Page 14: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Adding AWS Lambda

to your infrastructure

Page 15: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

How to Create a AWS Lambda Function

1.Write the code

2.Setup on AWS

3.Debug

Page 16: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

AWS Lambda Demo

Page 17: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 18: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 19: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 20: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 21: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 22: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 23: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 24: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 25: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 26: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 27: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 28: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Frameworks

node-lambda

Page 29: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Native Gotchas

Lambda runs on an Amazon Linux container

…and your dev box probably isn’t Amazon Linux…

…so native NPM modules are problematic

Page 30: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Native npm

1. Launch an Amazon Linux ec2

2. Use a node version manager

3. Install node 4.3

4. Install the native modules you need

5. Package the modules and dl to your box

6. Deploy

Page 31: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Building the demo with node-lambda

Four files:

• package.json

• .env

• event.json

• index.js

Page 32: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 33: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 34: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 35: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 36: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 37: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 38: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 39: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 40: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 41: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 42: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 43: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 44: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 45: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Monitoring 101

the tl;dr version

Page 46: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 47: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 48: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Collecting data is

cheap

Page 49: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Collecting data is

cheap;

Not having it when you

need it can be

expensive

Page 50: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 51: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 52: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 53: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 54: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 55: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 56: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 57: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Monitoring AWS Lambda

Page 58: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

What are the AWS Lambda Metrics

aws.lambda.duration, min, max, sum

aws.lambda.errors

aws.lambda.invocations

aws.lambda.throttles

Page 59: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Three Options for Monitoring AWS Lambda

1. Add a custom metric to CloudWatch

2. Add metric directly to monitoring application

3. Add a line to the CloudWatch logs

Page 60: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

How Datadog Does It

Add line to CloudWatch logs:

MONITORING|unix_epoch_timestamp|value|type|my.metric.name|#tag1:value,tag2

Page 61: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Monitoring example

Page 62: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Monitoring example

Page 63: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Monitoring example

Page 64: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Monitoring example

Page 65: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 66: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)
Page 67: AWS re:Invent 2016: Monitoring, Hold the Infrastructure: Getting the Most from AWS Lambda (DEV205)

Matt Williams

Evangelist - Datadog

[email protected]

@technovangelist

Twitter: @list