Vertebra: Bringing The Cloud Down To Earth

VERTEBRABringing the Cloud Down To Earth

Jayson VantuylChief Systems Architect

Engine Yard, Inc.

☁☔

Who Is This Guy?

Jayson Vantuyl

Chief Systems Architect / Founder

Architecting Large Networks Since 1994

Engine Yard, Inc.

Large Hosting Provider focused on hosting Ruby Applications

Managing Over 2,000 Virtual Servers

The Classic Blockers

My existing resources are not in the cloud.

A critical component is not in the cloud.

Critical components don’t fit together.

I’m on a budget: small first, scale later.

My boss wants it all (or some of it) in-house.

Fog Computing

It’s a Cloud, but it’s “down to Earth”. Fog, get it?

Focus on What, Not Where

Clouds Without Borders

Don’t Rain On My Parade

A Sky Full Of Clouds

The Nuts and Bolts

Agents

XMPP

Discovery

Security

XMPP

Security

&

Discovery

Agents

Resource

Database

Workflow

Engine

XMPP

XML Messaging and Presence Protocol

IETF Standard (RFC 3920, 3921)

Each Connection Is Essentially Two Streaming XML Documents

Authentication

Transport Security

Ordered Message Delivery, Presence

Many Implementations, Designed To Be Extensible

Server-to-Server Security via Dial-Back

Application-Level Reliability via SRV DNS Records

Resources

Every point of control in the system is described by a set of a “resources” (much like a URL).

These resources need not be unique in the cloud.

They are organized into hierarchies.

They are the point of abstraction.

Some Possible Resources

Each box is an agent that offers operations on the listed resources

Geography

Organizational Structure

Security Clearance

Type Of Service

[email protected]:/us/ny/new_york/sales/copy_writer

[email protected]:/ca/qc/montreal/sales/rep

[email protected]:/us/dc/washington/research/weapons/clearance/s/clearance/c/clearance/ers/clearance/rs

[email protected]:/us/fl/tampa/ops/helpdesk

[email protected]:/net/tcpip/server/http/blog

[email protected]:/net/tcpip/server/smtp/service/mailer

[email protected]:/hr/ca

[email protected]:/security/ca

[email protected]:/security/us

[email protected]:/hr/us

mailto:[email protected]


















Operations

Fundamental Unit Of Work In Vertebra

Shaped Like Remote Procedure Call

Receives Parameters In A Namespace

Discover Agents That Provide The Resources

Request — Ack — Result* — Done

Scope (i.e. shotgun or sniper?)

Example Operations

Completely Unrealistic Example

Let’s Fire Some People, The Entire Sales Dept.

Then Blog About It






[email protected]:/manage/ca

[email protected]:/manage/us

fire(who=/manage/sales)

kick_out(who=/security/sales)

add_entry(site=/blog,message=”Fired Sales!”)

instant_message(who=/sales,msg=”Goodbye.”)





















An Ominous Message

“who” is a resource, discovery is done on /sales

We discover [email protected] and [email protected]

Both get “instant_message” operation, and the agent on their computer delivers the message, “Goodbye.”
































You’re Fired!

“who” is a resource, discovery is done on/manage/sales

We discover us_hr and ca_hr, which both offer/manage

the “fire” operation goes to the the HR departments for both countries
































Kick ‘Em Out!

“who” is a resource, discovery is done on/security/sales

We discover us_sec and ca_sec, which provide/security

Security (in both countries) gets the operation to kick out everybody that matches /sales
































Gloat!

“site” is a resource, discovery is performed on/blog

We discover [email protected], which provides /blog

The operation to post the blog entry goes to the web server agent.


































Get Your Head In The Cloud

Operations in many agents with one request

Replace the “sales” entry with “us” to fire the whole US workforce

Replace “sales” with “us/ny” to fire the New York office

Can Subdivide the HR office any time and just offer the correct resources

Get Your Head In The Cloud

The agents are abstracted.

HR in the US might be done by human with some CRM.

HR in Canada might be automated, or might be paper-only, with people doing the work.

Same for security, and the web server.

Reduced To Set NotationIn the abstract, a resource is a set that maps to a set of agents

Resources may be singly inherited

The set that maps to a resource is a subset of the set that maps to its parent

There are useful operations on sets of resources, such as “coverage”

For any operation, the set of discovered agents is the union of the sets mapped to by the intersection of the set of requested resources and set of actual resources

What Do We Get Out Of This?

Computational Patterns

map-reduce

traditional RPC

message passing

job queue

observer

Component Abstraction

Same Resources, Different Code

Same Resources, Same Code

Different Resources, Same Code

What I Didn’t Tell You About

Responses from Operations and How They Aggregate

The RBAC Security Implementation

The Workflow Engine

The Resource-Keyed Database

Intrinsic I18n

Extensible Data Marshaling

Resource Advertising

Federating Clouds

Fault-Tolerant, Distributed Core Implementation

Monitoring Protocol

Job Management Protocol

Distributed Audit System

Distributed Log System

Reliability, Ease of Use, Scalability

This innovation brought to you by

Vertebra: Bringing The Cloud Down To Earth

Technology

Transcript of Vertebra: Bringing The Cloud Down To Earth