2016 - Serverless Microservices on AWS with API Gateway and Lambda
-
Upload
devopsdaysaustin -
Category
Software
-
view
707 -
download
0
Transcript of 2016 - Serverless Microservices on AWS with API Gateway and Lambda
Building Microservices with API Gateway and Lambda
@mattjbarlowgithub.com/mattjbarlow
DevOps Days Austin 2016Revenge of the Devs
What we will talk aboutSwagger
API Gateway
Lambda
DynamoDB
What we will talk aboutSwagger
API Gateway
Lambda
DynamoDB
API Spec File
Proxy / Router / URL
Backend Code
Data
With Swagger you can auto-generate...Documentation Monitoring Tests
Integration Tests
Client Libraries
We Never Edit API Gateway Directly We Edit Swagger
which is imported into...
API Gateway Monitoring Docs Tests
Example Microservice
Teammates want the ability to run a Lambda Function exactly one time in the future.
Solution
At Job Microservice
A microservice that mimics the Unix at command for Lambda.
The At Command
First appears in 1979 as part of Unix Version 7.
Design
API MethodsPath Operation OperationID Description
/atq GET list_jobs List all jobs
/atq POST create_job Create an at job
/atq/{id} GET describe_job Describe an at job
/atq/{id} DELETE delete_job Delete an at job
A method is a combination of a resource path and an operation.
Objects
Name Type Format Required?
jobid string uuid yes
lambdaArn string arn no
time string dateTime no
atJob
The object will define the response that our API returns.
CodeOperationID Pseudocode
list_jobs DynamoDB BatchGetItem
describe_job DynamoDB Query on jobid
create_job Create CloudWatch Event and PutItem into DynamoDB.
delete_job Delete CloudWatch Event and DeleteItem out of DynamoDB.
These Operation IDs are defined in Swagger and passed through to Lambda code as part of the event object.
Initialize Directory
git clone [email protected]:mattjbarlow/microservice-template.git
File Description
service/service.py * Python module that will run in Lambda.
circle.yml Circle CI instructions.
deploy.yml Ansible playbook for provisioning microservice.
destroy.yml Ansible playbook for deleting microservice.
swagger.yml * Spec file that describes your API.
template.json AWS resources required by the microservice.
version.yml Ansible playbook that versions your Lambda code.
* The bulk of your edits will be in these two files.
Update Swagger Paths
Remember these?Path Operation OperationID Description
/atq GET list_jobs List all jobs
/atq POST create_job Create an at job
/atq/{id} GET describe_job Describe an at job
/atq/{id} DELETE delete_job Delete an at job
We Insert Them Directly Into SwaggerRESOURCE PATH
HTTP OPERATION
Object Definition
Then we define our objects
Deploy
ansible-playbook -e “prefix=devopsdays” deploy.yml
Ansible Deployment Playbook
1. Zips up Python module and uploads it to S3
2. Creates the API Gateway
3. Provisions AWS resources
a. Lambda
b. DynamoDB
c. IAM Roles and Policies
d. Lambda Permission
4. Adds mapping templates to API Gateway
5. Sets stage variables
6. Exports Swagger file from API Gateway
Import Swagger Into Postman
Open Postman
Click Import
Select Swagger File
Start Sending Requests
Tail Lambda Logs
Logging in Lambda
API Gateway turned our HTTP request data into an Event Object. To start with, we log the entire Event Object.
pip install awslogs
awslogs allows us to mimic tail -f behavior on our Lambda function’s Log Group.
Event Object Close-Up
Post Body
Headers
Path ParamsQuery Params
API Gateway Variables
Helpful for filtering logs.
Tracing Our Request
GET Request Lifecycle
Method Response
Integration Response
Lambda
Method Request
Integration Request
Client
Cloudwatch Logs
HTTP GET Request Is Sent
GET /dev/v1/atq HTTP/1.1Host: cg4e6xg82i.execute-api.us-east-1.amazonaws.comConnection: keep-aliveCache-Control: no-cacheUser-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36Postman-Token: 265fcfe1-e196-0114-89bf-442a34c0180dAccept: */*Accept-Encoding: gzip, deflate, sdchAccept-Language: en-US,en;q=0.8
Mapped To Event Object
Event Object Received by Lambda
Logged to CloudWatch
Editing Code
Dev WorkflowEdit Eval
Push
Request
Parse
create_job function
Push Code To Lambda
ansible-playbook -e “prefix=dod” version.yml
Test create_jobImportant!
create_job response
Persisted in DynamoDB
POST Request Lifecycle
Method Response
Integration Response
Lambda
Method Request
Integration Request
Client
Cloudwatch Logs
EventsDynamo
Removing Lambda
GET Request With LambdaStep What Happens
Send GET request API Gateway Receives HTTP Request Data from client.
Transform API Gateway Transforms Request Data into Event Object
Proxy API Gateway POSTs Event Object to Lambda
Read Lambda Reads data from DynamoDB
Return API Gateway Receives return values from Lambda
Transform API Gateway Transforms backend data into HTTP Response
Respond API Gateway Responds back to the client.
GET Request without LambdaStep What Happens
Send GET request API Gateway Receives HTTP Request Data from client.
Transform API Gateway Transforms Request Data into Event Object
Proxy API Gateway POSTs DynamoDB Query to Dynamo
Read Lambda Reads data from DynamoDB
Return API Gateway Receives return values from Dynamo
Transform API Gateway Transforms backend data into HTTP Response
Respond API Gateway Responds back to the client.
Mapping GET Request To DynamoDB
Mapping DynamoDB Response to Client Response
Automating Tests
CI WorkflowCI
Step 1 Deploy Microservice
Step 2 Validate API Calls
Step 3 Destroy Microservice
git push POST: /project/:tree/:branch
Validating API Calls With Flexschema = load('swagger.awsexport.json')validate_api_call(schema, raw_request=r.request, raw_response=r)
1. Receives Swagger spec file which is our source of truth.2. Makes an HTTP Request to the API Gateway URL.3. Ensures the response matches what we said it would in Swagger.
Loads Swagger spec into memory
AuthIt has been 1 days since we talked about auth.
Built In Auth OptionsGET /dev/v1/atq HTTP/1.1Host: cg4e6xg82i.execute-api.us-east-1.amazonaws.comConnection: keep-alivex-api-key: bkayZOMvuy8aZOhIgxq94K9Oe7Y70Hw55
Option 2: Signature Version 4 signing with IAM(Powerful, but requires client having AWS API Key)
Option 1: API Keys managed by API Gateway API(Not really useful for user auth in public APIs)
Custom Authorizers
Receive Token from IdentitySend Token to /auth endpoint Pass Through
Responds With JWT
Make request with JWT Custom Authorizer Intercept
Caches temporary policyAllows Request
Generates JWT
Validates JWT
Returns temporary IAM policy
Client Library API Gateway Lambda$:
Validates Identity Token
Links
https://github.com/mattjbarlow/microservice-template
https://github.com/mattjbarlow/at
http://editor.swagger.io/#/
http://flex-swagger.readthedocs.io/en/latest/