Post on 20-Mar-2017
AWS LAMBDA @ .Juan Manuel Torres
César López-Natarén Jan. 2016
JUAN AND CÉSAR
MindTouch Engineers
JUAN MANUEL TORRES
➤ Born and raised in Bogotá Colombia
➤ Software Engineer @
➤ Love
➤ Programming
➤ Photography
➤ Lizards
➤ Traveling
➤ juant@mindtouch.com
CÉSAR LÓPEZ-NATARÉN
➤ Senior Software Engineer @
➤ Love to work on:
➤ APIs
➤ Data engineering
➤ Making code fast
➤ Infrastructure automation
➤ cesarn@mindtouch.com
Customer Success SaaS
➤ Authoring and publishing platform for Product Help
➤ We use your product’s knowledge to:
➤ Accelerate User Adoption
➤ Build Customer Loyalty
➤ We help you:
➤ Understand your Customers
➤ Accelerate your Sales Cycle
SOME MINDTOUCH CUSTOMERS
MINDTOUCH’S SCALE
➤ Up to 2M req/day
➤ Up to 5M events/day
AWS LAMBDAServerless computing
WHAT IS AWS LAMBDA?
➤ Compute service that hosts an execution environment for your code on high availability compute infrastructure.
➤ The service:
➤ Administers the compute resources (CPU, memory, network and OS maintenance).
➤ Guarantees the selected capacity.
➤ Scales your software automatically.
➤ Provides monitoring and logging services.
WHY DOES IT MATTER?
➤ Because all the tasks described before consume programmer’s time
➤ And instead of focusing on your problem domain you are dealing with those other things
➤ It is actually complicated to get all those pieces right
HOW DO I USE IT?
➤ You provide your Node.js, Java, or Python code.
➤ Configure:
➤ Event sources
➤ Memory
➤ Execution role
➤ Handler function
LET’S GET ONE PYTHON LAMBDA FUNCTION DEPLOYED
photo by Kool Cats Photography
LAMBDA FUNCTION LIFE CYCLE
➤ Develop ➤ Test ➤ Deploy ➤ Repeat
LAMBDA FUNCTION DEVELOPMENTphoto by Lachlan Hardy
PROGRAMMING MODEL / ENTRY POINT
Handler
1 def lambda_handler(event, context): 2 s3_info = event['Records'][0]['s3'] 3 ... 4 tweet_images.send_image( 5 s3_info['bucket']['name'], 6 s3_info['object']['key'], 7 cleanup=True 8 )
PROGRAMMING MODEL / LOGGING
Logging
1 import logging 2 3 logger = logging.getLogger() 4 logger.setLevel(logging.INFO) 5 6 def lambda_handler(event, context): 7 logger.info("Invoked with {0} events".format( 8 len(event['Records'])))
LAMBDA FUNCTION TESTING
TESTING OPTIONS
➤ Local testing through unit testing
➤ Make sure you can swap out implementations of components that perform I/O
➤ Mock testing through the AWS Console in the Lambda service itself
➤ Create your own simulation environment for Lambda
photo by NASA's Marshall Space Flight Center
LAMBDA FUNCTION DEPLOYMENT
DEPLOYMENT
➤ Creating the lambda function on the service
➤ Granting permissions to event sources and any other components that the functions needs to interact with
➤ Sending the packaged code to the service
➤ Configuring the application
AWS LAMBDA USAGE @ MINDTOUCH
➤ Mirror data streams across accounts and regions
➤ Archive events to S3
➤ Generate SQS messages from S3 object creation
➤ Publish Auto Scaling Group’s notifications to DataDog
CHALLENGES FACEDwhile using AWS Lambda
CHALLENGES WHILE USING AWS LAMBDA
➤ Consistent function configuration
➤ Automated function deployment on different environments (dev, test, production)
MINDTOUCH LAMBDA OPSMindTouch deployment tool for Lambda functions
MINDTOUCH LAMBDA OPS DESIGN OBJECTIVE
To enable a consistent development/test/deployment lifecycle for AWS Lambda functions.
TOOL REQUIREMENTS
➤ Developers + DevOps
➤ Configuration must be simple
• Single file
• Checked-in to version control
• Many deployments
➤ Permission management
➤ Managing Event Source Mappings
IMPLEMENTATION DETAILS
➤ Single YAML Configuration file
➤ Automatic generation of config.json including names for all sources and resources
➤ Prepares lambda function for deployment
➤ Creates a CloudFormation stack
DEMOKinesis to S3
KINESIS2S3XMLARCHIVER CONFIGURATION
1 app: 2 3 resources: 4 source: 5 type: kinesis 6 name: events-stream 7 event-source: 8 batch-size: 10000 9 starting-position: latest 10 11 s3bucket: 12 type: s3 13 name: events-archive 14 allow: 15 - s3:PutObject 16 - s3:GetObject
LIMITS
➤ Execution time is limited to 300s (5 minutes)
➤ 1.5GB memory limits
photo by Navalarp Teratanatorn
SECURITY
➤ As Jan 2016 there is no support for accessing resources on a VPC (Coming soon)
DEPLOYMENT
➤ No straight forward deployment other than the console or using a third party tool
photo by NASA's Marshall Space Flight Center
CONCLUSIONS ➤ Huge developer productivity
gains ➤ Reduces complexity in large
infrastructure systems ➤ They do what they claim to do ➤ We will choose AWS Lambda
for suitable projects in the future