Building an AWS SDK

Granada Perl Workshop 2014

Jose Luis Martinez

JLMARTIN on CPAN

Twitter: @pplu_io

joseluis.martinez@capside.com

About me

CTO @CAPSiDE

Training Partner for AWS

Delivering AWS official training

All four AWS certifications

Real experience with real platforms

Deliver consulting and continuous operation

Writing Perl for +10 years

And still having fun with it!

About AWS

Mostly known for EC2 and S3

But lot's more services! (+30)

Everything is an API

“Programmable datacenter”

Software Defined Everything

About AWS

Mostly known for EC2 and S3 But lot's more services! (+30) Everything is an API “Programmable datacenter”

Software Defined Everything

It’s the your wet dream

Perl & AWS

Bad news

No official SDK

Official ones: Ruby, PHP, Python, JS, .Net, Java

Note: python wasn't official initially!

Perl & AWS

Good news

CPAN

And an active community

Lots of modules for different services (sometimes more than one)

Perl & AWS: The CPAN

Inconsistencies

In method naming

Arbitrary defaults ('eu-west-1', 'us-east-1')

Not up to date with latest APIs

Keeping up with AWS is hard!!!

“Not so popular” services not covered

IAM, CloudFormation, SWF, RedShift, CloudSearch, etc

Almost no Role support

AWS::CLIWrapper is a good generic solution. Not native

So lets write an entire SDK!

Hard enough

You can end up with…

or

Challenges: # of services

30+ services

Some with 100+ API calls

Each call with it’s parameters

Want to return objects

Evolution

Challenges: # of services

So lets start classifying

Challenges: Endpoints

Endpoint Type Services

Regional Services available in a región

the rest

Single Global services 6

Challenges: WebService Types

One URL REST

Params: QueryResponse: XML

17 3

Params: JSONResponse: JSON

10 1

Challenges: Signatures

Type Services

v2 2

v3https 1

cloudfront 1

v4 the rest

Challenges: API versions

More tan one API version is live

That’s why not-up-to-date callers still work

RDS

4 versions

CloudFront

3 versions

EC2

4 versions

AWS Pace of innovation

Every month there are changes

Changes mean APIs change (slightly)

Very hard to keep up!

AWS Pace of innovation

Every week there are announcements

Changes mean APIs change (slightly)

Very hard to keep up!

Lets start writing!

Hand write classes and methods, and parameters

Hand write classes and methods, and parameters

Parse docs

After all… you would have to read them to write the classes

Parse docs

After all… you would have to read them to write the classes

No published spec

Introspect an official SDK!

After all AWS already has read the docs for you

No published spec Introspect an official SDK!

That’s your spec!

After all AWS already has read the docs for you

Data driven SDKs!

JS SDK has datastructures that define calls

Proof of concept

Win++! Boto and PHP2 too!

Later migrate to using Boto

Data driven SDKs!

JS SDK has datastructures that define calls

Proof of concept

Win++! Boto and PHP2 too!

Later migrate to using Boto

The solutions

Modern Perl to the rescue

Try to be very lazy Parameter validation: Create Moose classes that validate

the parameters passed to the call. May change in the future.

Got very quick results!

Hide all possible implementation from the user Try not to leak implementation

That way you can change it at will

Meta lets you inspect the objects! Call objects get marshalled into datastructured via meta

Result datastructures get marshalled into objects via meta

Roles

Modern Perl: Roles

All functionality gets pulled in via Roles

Code is basically in different roles, Classes are “serialized configurations” of how to call APIs

Makes it easy to

Generate service classes from datastructures

Change behaviour for tests

Roles are used for

One global endpoint vs endpoints per region

Caller (Query, JSON)

Response (XML, JSON)

Signatures (V2, V3, V4)

IO

Aws is a class factory

Dynamic class construction

User never constructs service classes directly

Aws class method uses default config

my $ec2 = Aws->service(‘EC2’)->new( . . . )

Dynamic class construction

User wants custom functionality

my $aws = Aws->new( config => AWS::SDK::Config->new( caller => 'Net::AWS::TestCaller‘ ) );

my $ec2 = $aws->service(‘EC2’)->new( . . . )

service loads Aws::EC2, creates a new class with TestCaller

Win: Hello to the async world

Still playing around in examples

my $aws = Aws->new( config => AWS::SDK::Config->new( caller => 'Net::AWS::MojoCaller‘ ) );

my $ec2 = $aws->service(‘EC2’)->new( . . . )

my $asgs = $ec2->DescribeAutoScalingGroups;

See examples/asyncAutoScaling.pl

Win: Also going “fully async”

my $aws = Aws->new( config => AWS::SDK::Config->new(caller => 'Net::AWS::MojoAsyncCaller'));

my $delay = Mojo::IOLoop->delay(sub { my ($delay, @responses) = @_; Dumper($_) for @responses;});

foreach my $region ( "us-east-1", "ap-northeast-1", "sa-east-1", "ap-southeast-1", "ap-southeast-2", "us-west-2", "us-west-1", "eu-west-1", 'invented-region') { my $ctrail = $aws->service('CloudTrail')->new( region => $region, delay => $delay, ); print "Doing a call for $region\n"; my $list = $ctrail->DescribeTrails;}

$delay->wait;

Lessons Learned

Lessons learned

First shot with MooseX::Declare

Error messages are HORRIBLE!

Change to Moops didn't go well

Finally plain old Moose

Lessons learned

Namespace finding (still in the works)

AWS vs Aws

Thanks @clintongormley for the article!

Lessons learned

Enums (still in the works)

APIs sometimes return unexpected stuff

Lessons learned

Abstract $request object passed around

Finally marshalled into the UserAgents’ taste

Still open stuff

Load time for big services (EC2)

Not all classes will be used all the time

Generate classes on demand

Calling convention

Now CamelCased

Not so perlish

Calling from objects (not service objects)

$queue->DeleteQueue(); from a queue object

Attribute Traits

Access to Arrays and HashRefs via nicer methods

Lots more!!!

Use AWS? Use Perl? Want to help?

Fork your heart out:

https://github.com/pplu/aws-sdk-perl/