Paws - A Perl AWS SDK

55
Paws – A Perl AWS SDK @pplu_io 03/09/2015 - Granada YAPC::EU 2015 Jose Luis Martinez

Transcript of Paws - A Perl AWS SDK

Page 1: Paws - A Perl AWS SDK

Paws – A Perl AWS SDK

@pplu_io

03/09/2015 - Granada

YAPC::EU 2015

Jose Luis Martinez

Page 2: Paws - A Perl AWS SDK

AWS is…• Cloud Computing• Consume computing/database/queuing/etc services via

an API• Everything is an API

Page 3: Paws - A Perl AWS SDK

AWS is…

Programmers wet dream

Page 4: Paws - A Perl AWS SDK

Why?Isn’t there support for AWS on CPAN?

Page 5: Paws - A Perl AWS SDK

AWS Services on CPAN• There are a LOT• EC2, SQS, S3, RDS, DynamoDB, etc• But lots are were missing

• AWS::CLIWrapper is a generic solution too• Shells off to the oficial AWS CLI (python)

I want Perl support for ALL of them

Page 6: Paws - A Perl AWS SDK

Different authors, different opinions• Default region (eu-west-1 for some, us-east-1 for others)• Different HTTP clients• LWP, HTTP::Tiny, Furl, etc

I want explicit, required, region. Croak if not specifiedPluggable HTTP client?

Page 7: Paws - A Perl AWS SDK
Page 8: Paws - A Perl AWS SDK
Page 9: Paws - A Perl AWS SDK

Different authors, different photo• Some regions not supported due to bugs

• Subtle name changes in region endpoints• Credential handling• Module just covers their needs

I want as broad support as we can get

Page 10: Paws - A Perl AWS SDK

Credential handling• Roles in AWS help you not have to distribute credentials

(AccessKey and SecretKey)• Support depends on author of module knowing of them

/ needing them

I want support for Instance Roles, STS AssumeRole, Federation for all services

Page 11: Paws - A Perl AWS SDK

UpToDate-ness• Being up to date depends on authors needs, time, etc• AWS APIs are updated a lot

I want up to date APIs

Page 12: Paws - A Perl AWS SDK

Lets write an SDK!

Page 13: Paws - A Perl AWS SDK
Page 14: Paws - A Perl AWS SDK

Some numbers52 services

Page 15: Paws - A Perl AWS SDK

Some numbers52 services

~1600 actions

Page 16: Paws - A Perl AWS SDK

Some numbers52 services

~1600 actions

~3700 distinct input/output objects

Page 17: Paws - A Perl AWS SDK

Some numbers52 services

~1600 actions

~3700 distinct input/output objects

~12000 attributes

Page 18: Paws - A Perl AWS SDK
Page 19: Paws - A Perl AWS SDK

Write by hand?

Page 20: Paws - A Perl AWS SDK

Write by hand?

Page 21: Paws - A Perl AWS SDK

Paws is autogenerated

Page 22: Paws - A Perl AWS SDK
Page 23: Paws - A Perl AWS SDK

Paws is autogenerated• AWS has some JSON definition files in their SDKs (data-driven)

• Pick them up to generate classes for:• Actions• Inputs to actions (parameters)• Outputs from actions (outputs)

• HTML documentation -> POD

make gen-classes

Page 24: Paws - A Perl AWS SDK
Page 25: Paws - A Perl AWS SDK

Code generators• In builder-lib (not distributed on CPAN)• Paws::API::Builder

• Paws::API::Builder::EC2• Paws::API::Builder::query• Paws::API::Builder::json• Paws::API::Builder::restjson• Paws::API::Builder::restxml

• Leaves all auto-generated code in auto-lib (distributed on CPAN)• Hand-written code is in lib

Note: this is not needed if you only want to use Paws. This is intended for developers. We’ll see more internals later

Page 26: Paws - A Perl AWS SDK

Using Paws

Page 27: Paws - A Perl AWS SDK

Each AWS API is a “Service Class”• Each Action in the API is a method on the Service Class

• EC2 API -> Paws::EC2 service class• Paws::EC2 objects have methods like• RunInstances• TerminateInstances• DescribeInstances

Page 28: Paws - A Perl AWS SDK

How do I get an instance of a service class?use Paws;my $ec2 = Paws->service(‘EC2’, region => ‘eu-west-1’);my $iam = Paws->service(‘IAM’);

# $ec2 and $iam are instances of Paws::EC2 and Paws::IAM# they use Paws default config (they just work )

Page 29: Paws - A Perl AWS SDK

How do I get an instance of a service class? (II)my $paws = Paws->new(config => { region => ‘eu-west-1’, caller => ‘Paws::Net::LWPCaller’, credentials => ‘My::Custom::Credential::Provider’});

my $ec2 = $paws->service(‘EC2’);# ec2 is bound to region ‘eu-west-1’# and called with LWP# and gets it’s credentials from some custom source

Page 30: Paws - A Perl AWS SDK

Calling a method

$ec2->Method1( Param1 => ‘Something’, Param2 => 42, Complex1 => { x => 1, y => 2, z => 3 }, Complex2 => [ { x => 1, y => 2 }, { x => 2, y => 3 } ])

Page 31: Paws - A Perl AWS SDK

Calling a method

$ec2->Method1( Param1 => ‘Something’, Param2 => 42, Complex1 => { x => 1, y => 2, z => 3 }, Complex2 => [ { x => 1, y => 2 }, { x => 2, y => 3 } ])

Docs tell you that this is a Paws::Service::XXX object, but you don’t have to instance it !!!

Just pass the attributes and the values as a hashref

Page 32: Paws - A Perl AWS SDK

Calling a method: maps• Arbitrary key/value pairs• Don’t build an object either. Paws will handle it for you• $ec2->Method1( Map1 => { x => 1, y => 2, z => 3 });

Page 33: Paws - A Perl AWS SDK

Methods return objectsmy $object = $x->Method1(…)

Method1 returns Paws::Service::Method1Result has ‘X’, has ‘Y’, has ‘Complex’ => (isa => ‘Paws::Service::Complex1’)

$object->X$object->Complex->Complex1Attribute

Page 34: Paws - A Perl AWS SDK

Tricks

Page 35: Paws - A Perl AWS SDK

Tricks: CLI• Paws ships with a CLI

paws SERVICE --region xx-east-1 DescribeFoo Arg1 Val1

Uses ARGV::Struct to convey nested datastructures via command line

Page 36: Paws - A Perl AWS SDK

Tricks: open_aws_console• Opens a browser with the AWS console (using the SignIn

service)• Uses your current credentials (extends a temporary

token)

Page 37: Paws - A Perl AWS SDK

Tricks: Changing endpointsmy $predictor = $paws->service('ML', region_rules => [ { uri => $endpoint_url } ]);

• Works for any service: SQS, EC2…

Page 38: Paws - A Perl AWS SDK

Tricks: Credential providers• Default one tries to behave like AWS SDKs• Environment (AWS_ACCESS_KEY_ID and

AWS_SECRET_ACCESS_KEY)• File (~/.aws/credentials, an ini file)• From the metadata service (Instance Roles)

• Your own• Just attach Role “Paws::Credential” and get the

credentials from wherever

Page 39: Paws - A Perl AWS SDK

InternalsNote: actual implementation as of Sept 2015

Read the code / changelog to get a hold of changes

Page 40: Paws - A Perl AWS SDK
Page 41: Paws - A Perl AWS SDK

Each method has parameters• Parameters are converted into Moose objects for validation

package Paws::EC2 sub Method(Param1 => Str, Param2 => Int)

Coerces its @_ into Paws::EC2::Method (has ‘Param1’, has ‘Param2’)

Note: not using Moose coercion. Using new_with_coercions

Page 42: Paws - A Perl AWS SDK

Each method has parameters• Parameters are converted into Moose objects for validation

package Paws::EC2 sub Method(Param3 => Complex1)

Complex1 has it’s own “input class”Paws::EC2::Complex1 has [‘X’, ‘Y’, ‘Z’ ]

new_with_coercions knows how to coerce { x => 1, y => 2, z => 3 } into a Paws::EC2::Complex1

Page 43: Paws - A Perl AWS SDK

After coercing parameters into an object• $self->caller->do_call($self, $call_object)

• Service classes have a “caller”. Caller is defined when constructing the service object.• Callers are responsable for

• Getting a Paws::Net::APIRequest (via prepare_request_for_call)• Prepare_request_for_call is specialized for each type of service in Paws::Net::*Caller

roles• Doing I/O

• Paws::Net::Caller uses HTTP::Tiny (Paws default)• Paws::Net::LWPCaller uses LWP (contributed)• Paws::Net::MojoAsyncCaller uses Mojo::UserAgent (experimental)

• Passing results to handle_response

Page 44: Paws - A Perl AWS SDK

Call Object to APIRequest (prepare_request_for_call)• Looks in the call object where it has to place parameters

to the API• Headers• In a serialized body

• JSON• Query Parameters

• Arrays get coded in f(x) of the API• att.0=xxx• att.members.0=xxx

• In the body• In the URL (REST APIs)

• Signs the request (via roles that know how to sign for that service)

Page 45: Paws - A Perl AWS SDK

handle_response• Takes a look if there were error conditions in the HTTP

call• Future: should determine how to retry

• Deserializes the response• XML• JSON

• Deserializes into objects• Note: sometimes decides it wants an exception• Doesn’t throw: just creates an exception object

Page 46: Paws - A Perl AWS SDK

Callers• Do the IO• Have to handle some common logic (still)

• Asyc callers don’t need to return the result immediately• The experimental Mojo caller returns a Future • The future fails if the result was an exception

Page 47: Paws - A Perl AWS SDK

Future

Page 48: Paws - A Perl AWS SDK
Page 49: Paws - A Perl AWS SDK

Future (hint: help needed and accepted)• Testing Async stuff• Retrying

• Some APIs return temporary failures• Want automatic exponential backoff with jitter

• Paging• Some APIs return paged results

• Want a “give me all of them”• Waiters

• Wait until some condition is met• Want a call to wait until Instance is in running state

• A lot more: take a look at GitHub issues

Page 50: Paws - A Perl AWS SDK

Future (hint: help needed and accepted)• Object Oriented results• $ec2->TerminateInstances(InstanceIds => [ ‘i-

12345678’ ])• $instance->Terminate

• Special properties• En/Decode base64, URIescape, etc

• Better access to ArrayRefs• Use ArrayRef Moose trait for

• Number of elements• Get element i• Get list of elements

Page 51: Paws - A Perl AWS SDK

Future (hint: help needed and accepted)• Refactoring generator clases• MooseX::DataModel• Template::Toolkit

• Split Paws into separately instalable modules

• Rinse and Repeat• For other APIs• AWS API as a Service

Page 52: Paws - A Perl AWS SDK

Support for APIs

REST Plain05

101520253035404550

Types of APIs

Query+XML JSON EC2

Page 53: Paws - A Perl AWS SDK

Support for APIs

REST Plain05

101520253035404550

Types of APIs

Query+XML JSON EC2

Implemented

Page 54: Paws - A Perl AWS SDK

Support for APIs

REST Plain05

101520253035404550

Types of APIs

Query+XML JSON EC2

Implemented

Need love and testing

S3 not workingRoute53?Lambda?

Page 55: Paws - A Perl AWS SDK

Fork your heart outhttps://github.com/pplu/aws-sdk-perl/

Contact me:Twitter: @pplu_ioMail: [email protected]: JLMARTIN

CAPSiDETwitter: @capsideMail: [email protected]