Post on 15-Jan-2015
description
GearmanFrom the Worker's Perspective
Tuesday, June 21, 2011
/usr/bin/whoami
Brian Aker
CTO/Founder Data Differential
Previously MySQL, Slashdot, Sun Microsystems
Tuesday, June 21, 2011
What is Gearman?
Tuesday, June 21, 2011
“The way I like to think of Gearman is a massively distributed fork mechanism”
-Joe Stump, Digg
“The Not Mechanical Turk” -Don MacAskill, SmugMug
Tuesday, June 21, 2011
do (“resize_image”)resize_image()
{ … return $image;
}
Tuesday, June 21, 2011
! !
!"#$%#&'()#*+
Tuesday, June 21, 2011
! !
!"#$%&$'()&$*&+,-./
! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9
! :45-14.$."43$64.47;)7<
" ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3
! @-.5)A.47<-4<+$)75+<145)"7
! B)&51)6-5+$C1";+&&)7<
" 0"&&)6.8$;."&+1$5"$3454
! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+&
Tuesday, June 21, 2011
Server
Provides Asynchronous and Synchronous Requests
Restarts Work
Durable Requests
Gearman Protocol/HTTP
Epoch Scheduling
Logging
Tuesday, June 21, 2011
Client# Create our client object.$gmclient= new GearmanClient();
# Add default server (localhost).$gmclient->addServer();
$result= $gmclient->do("reverse", "Hello!");
echo "Success: $result\n";
Tuesday, June 21, 2011
Worker
# Create our worker object.$gmw= new GearmanWorker();
# Add default server (localhost).$gmw->addServer();
$gmw->addFunction("reverse", "reverse_fn");while ($gmworker->work()) {…}
Tuesday, June 21, 2011
Worker Function
function reverse_fn($job){ $workload= $job->workload(); $result= strrev($workload);
return $result;}
Tuesday, June 21, 2011
Lots of functions...
$gmw->addFunction("resize", "resize_fn");
$gmw->addFunction("grep", "grep_fn");
$gmw->addFunction("fetch_url", "fetch_url");
Tuesday, June 21, 2011
gearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);
… gearman_job_send_data(job, chunk, sizeofchunk);
… gearman_job_send_status(job, 50,100);
… if (issue_warning) gearman_job_warning(job, “I'm sorry, Dave. I'm afraid I can't do that.”, size); return GEARMAN_SUCCESS;}
Function
Tuesday, June 21, 2011
GEARMAN_SUCCESS
GEARMAN_FATAL
GEARMAN_ERROR
GEARMAN_SHUTDOWN
Worker Return
Tuesday, June 21, 2011
map(list[…], reduce()); map() {…}
reduce() {…}
reduce() {…}
reduce() {…}
Tuesday, June 21, 2011
Map @#$@# ?
Tuesday, June 21, 2011
Partitioning
{A...K}
{L...Q}
{R...Z}
find()
find()
find()
Tuesday, June 21, 2011
gearman_return_t split_worker(gearman_job_st *job, void* /* context */){ const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job);
const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ' ') { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } }
return GEARMAN_SUCCESS;}
Partitioning
Tuesday, June 21, 2011
Aggregation
+ result
+ result
+ result$result
$result
$result
= sum result
Tuesday, June 21, 2011
Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::string string_value;
do { gearman_result_st *result_ptr= gearman_task_result(task);
string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr));
} while ((task= gearman_next(task)));
gearman_result_store_value(result, string_value.c_str(), string_value.size());
return GEARMAN_SUCCESS;}
Tuesday, June 21, 2011
Do we have to partition?
(What other tricks exist!)
Tuesday, June 21, 2011
Pipeline
Store() Resize() Publish()
Tuesday, June 21, 2011
CPU?poll()
Tuesday, June 21, 2011
Tuesday, June 21, 2011
multiple languages
Tuesday, June 21, 2011
NamespacesFoo::resize_image() { … return $image;}
Acme::resize_image() { … return $image;}
Tuesday, June 21, 2011
Future
0.22 Released
Custom Logging Plugins
Client/Worker Configuration
Extended Administrative Commands
SSL
Job Result Cache
Uplift!
Tuesday, June 21, 2011
• Gearman.info (up to date)
• Gearman.org (...)
• http://launchpad.net/gearmand/
• twitter: brianaker
• blog: blog.krow.net
Tuesday, June 21, 2011