Gearman, from the worker's perspective

GearmanFrom the Worker's Perspective

Tuesday, June 21, 2011

/usr/bin/whoami

Brian Aker

CTO/Founder Data Differential

Previously MySQL, Slashdot, Sun Microsystems


What is Gearman?


“The way I like to think of Gearman is a massively distributed fork mechanism”

-Joe Stump, Digg

“The Not Mechanical Turk” -Don MacAskill, SmugMug


do (“resize_image”)resize_image()

{ … return $image;

}


! !

!"#$%#&'()#*+


! !

!"#$%&$'()&$*&+,-./

! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9

! :45-14.$."43$64.47;)7<

" ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3

! @-.5)A.47<-4<+$)75+<145)"7

! B)&51)6-5+$C1";+&&)7<

" 0"&&)6.8$;."&+1$5"$3454

! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+&


Server

Provides Asynchronous and Synchronous Requests

Restarts Work

Durable Requests

Gearman Protocol/HTTP

Epoch Scheduling

Logging


Client# Create our client object.$gmclient= new GearmanClient();

# Add default server (localhost).$gmclient->addServer();

$result= $gmclient->do("reverse", "Hello!");

echo "Success: $result\n";


Worker

# Create our worker object.$gmw= new GearmanWorker();

# Add default server (localhost).$gmw->addServer();

$gmw->addFunction("reverse", "reverse_fn");while ($gmworker->work()) {…}


Worker Function

function reverse_fn($job){ $workload= $job->workload(); $result= strrev($workload);

return $result;}


Lots of functions...

$gmw->addFunction("resize", "resize_fn");

$gmw->addFunction("grep", "grep_fn");

$gmw->addFunction("fetch_url", "fetch_url");


gearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);

… gearman_job_send_data(job, chunk, sizeofchunk);

… gearman_job_send_status(job, 50,100);

… if (issue_warning) gearman_job_warning(job, “I'm sorry, Dave. I'm afraid I can't do that.”, size); return GEARMAN_SUCCESS;}

Function


GEARMAN_SUCCESS

GEARMAN_FATAL

GEARMAN_ERROR

GEARMAN_SHUTDOWN

Worker Return


map(list[…], reduce()); map() {…}

reduce() {…}

reduce() {…}

reduce() {…}


Map @#$@# ?


Partitioning

{A...K}

{L...Q}

{R...Z}

find()

find()

find()


gearman_return_t split_worker(gearman_job_st *job, void* /* context */){ const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job);

const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ' ') { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } }

return GEARMAN_SUCCESS;}

Partitioning


Aggregation

+ result

+ result

+ result$result

$result

$result

= sum result


Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::string string_value;

do { gearman_result_st *result_ptr= gearman_task_result(task);

string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr));

} while ((task= gearman_next(task)));

gearman_result_store_value(result, string_value.c_str(), string_value.size());

return GEARMAN_SUCCESS;}


Do we have to partition?

(What other tricks exist!)


Pipeline

Store() Resize() Publish()


CPU?poll()


multiple languages


NamespacesFoo::resize_image() { … return $image;}

Acme::resize_image() { … return $image;}


Future

0.22 Released

Custom Logging Plugins

Client/Worker Configuration

Extended Administrative Commands

SSL

Job Result Cache

Uplift!


• Gearman.info (up to date)

• Gearman.org (...)

• http://launchpad.net/gearmand/

• twitter: brianaker

• blog: blog.krow.net


http://launchpad.net/gearmand/






Gearman, from the worker's perspective

Technology

Transcript of Gearman, from the worker's perspective