Polyglot parallelism
-
Upload
phillip-toland -
Category
Technology
-
view
1.226 -
download
0
Transcript of Polyglot parallelism
Polyglot ParallelismA Case Study in Using Erlang
and Ruby at Rackspace
The ProblemPart 1
20,000 network devices
9 Datacenters,3 Continents
devices not designed forhigh-throughput
management
we need a high throughput solution
the time spent in I/O is the primary
bottleneck
if you want to speed things up you have to talk to more devices
in parallel
The ProblemPart II
huge blobs of data
lots of backups equals big database
ad-hoc searching is difficult but
important
customer SLA means need to restore from
backup quickly
an event must be generated for each device interaction
migrations are problematic with that much data
rigid schema made adapting to new devices difficult
each device type has different properties
“backup” means different things for each device type
need to grow with the business
Previous Solution
multiple Ruby apps
difficult to scale
vendor device managers
New Solution
the simplest thing that could possibly
work
most db writes come from
scheduled jobs
Rails
MongoDB
Erlang
ReST API
Other Clients
NetworkDevices
Joe Armstrong
AXD301ATM Switch
99.9999999%
Functional
Dynamically Typed
Single Assignment
A = 1. %=> 1A = 2. %=> badmatch
[B, 2, C] = [1, 2, 3].B = 1. %=> 1C = 3. %=> 3
ImmutableData
Structures
D = dict:new().D1 = dict:store(foo, 1, D).D2 = dict:store(bar, 2, D1).
Concurrency Oriented
-module(fact).-export([fac/1]). fac(0) -> 1; fac(N) -> N * fac(N-1).
-module(quicksort).-export([quicksort/1]). quicksort([]) -> [];quicksort([Pivot|Rest]) -> quicksort([Front || Front <- Rest, Front < Pivot]) ++ [Pivot] ++ quicksort([Back || Back <- Rest, Back >= Pivot]).
Details
jobs framework
Runner
CallbackModuleWorkers
Runner Worker CallbackModule
start
ready
process processready
.
.
.stop
item
“behaviour” is interface
behaviour_info(callbacks) -> [ {init, 1}, {process_item, 3}, {worker_died, 5}, {job_stopping, 1}, {job_complete, 2}].
running({worker_ready, WorkerPid, ok}, S) -> case queue:out(S#state.items) of {empty, I2} -> stop_worker(WorkerPid, S), {next_state, complete, S#state{items = I2}};
{{value, Item}, I2} -> job_worker:process(WorkerPid, Item, now(), S#state.job_state),
{next_state, running, S#state{items = I2}} end;
handle_info({'DOWN', _, process, WorkerPid, Info}, StateName, S) -> {Item, StartTime} = clear_worker(WorkerPid, S),
Callback = S#state.callback, spawn(Callback, worker_died, [Item, WorkerPid, StartTime, Info, S#state.job_state]),
%% Start a replacement worker start_workers(1, Callback), {next_state, StateName, S};
handle_cast({process, Item, StartTime, JS}, S) -> Callback = S#state.callback, Continue = try Callback:process_item(Item, StartTime, JS) catch throw: Error -> error_logger:error_report(Error), ok end,
job_runner:worker_ready(S#state.runner, self(), Continue),
{noreply, S}.
story time
ReSTful APIwith Webmachine
The Convention Over Configuration Webserver
HTTP Request Lifecycle Diagram
http://webmachine.basho.com
Webmachine Is Simple As Proven by the “Number of Types of Things”
Measurement of Complexity
If you know HTTP
The 3 Most Important Types of Things In Webmachine
1. Dispatch Rules (pure data--barely a thing!)2. Resources (composed of simple functions!)3. Requests (simple get/set interface!)
Dispatch Rules
GET /devices/12345
Webmachine inspects the device_resource module for defined callbacks, and sets the Request record’s “server”
value to 12345.
{ ["devices", server], device_resource, [] }
Resources
• POEM (Plain Old Erlang Module)• Composed of referentially transparent functions*• Functions are callbacks into the request lifecycle• Approximately 30 possible callback functions, e.g.:
• resource_exists → 404 Not Found• is_authorized → 401 Not Authorized
* mostly
Perma-404resource_exists(Request, Context) -> {false, Request, Context}.
Lucky Authis_authorized(Request, Context) -> S = calendar:time_to_seconds(now()), case S rem 2 of 0 -> {true, Request, Context}; 1 -> {“Basic realm=lucky”, Request, Context} end.
Resource Functions
Requests• The first argument to each resource function • Set and read request & response data
wrq:set_resp_header(“X-Answer”, “42”, Request).
RemoteIP = wrq:peer(Request).
content_types_provided(Request, Context) -> Types = [{"application/json", to_json}], {Types, Request, Context}.
to_json(Request, Context) -> Device = proplists:get_value(device, Context), UserId = get_user_id(Request),
case fe_api_firewall:get_config(Device, UserId) of
{ok, Config} -> success_response(Config, Request, Context);
{error, Reason} -> error_response(502, Reason, Request, Context) end.
Retrieving a JSON Firewall Representation
Gotchas
primitive obsession
string-ish
[<<"easy as ">>, [$a, $b, $c], " ☺\n"].
“hi how are you”
<<“hello there”>>
hashes vs records
to loop is human,to recur divine
Erlang conditionals always return a value
design for testability
don’t spawn,
use OTP
Downsides
Erlang changes very slowly
3rd party libraries
standard librarycan be inconsistent
package management
Questions
Phil: @philtolandhttp://github.com/tolandhttp://philtoland.com
Mike: @lifeinzemblahttp://github.com/msassak
http://spkr8.com/t/7806