Dispatching jobs with Gearman
or
How to farm out jobs
or
Using Gearmanin Perl
Using Gearmanin Perl
gearman.org
The name "Gearman" was chosen as an
anagram for "Manager," "since it dispatches
jobs to be done, but does not do anything
useful itself"
Originally written in Perl
Rewritten in C
Clients in Perl, PHP, Phython,
Java, C# (.NET)
Clients in Perl, PHP, Phython,
Java, C# (.NET),even MySQL and PostgreSQL
WTF?
Architecture
Part I
Application
Job
Application
Job (or task)
Application
Job
gearmand
Application
Job
gearmand
Worker
Job
Application
Job
gearmand
Worker
Job Response
Application
gearmand
Worker
Worker
Worker
Application
gearmand
Worker
Worker
gearmand
Application
gearmand
Worker
Worker
gearmand
Application
Workers are scalable
Run any number you need
Run on remote servers
Application does not care
Application only talks with gearmand server
Application only talks with gearmand server
(one or more)
One or more application
One or more application
Applications throw jobs
One or more job servers
One or more job servers
Job servers dispatch jobs
One or more workers
One or more workers(clones or different)
One or more workers
Workers do jobs
One or more workers
Workers do jobsand may issue new jobs
Scalable also means redundant
Scalable also means redundant and reliable
Scalable also means redundant and reliable*
* when it works properly
Differentapproaches
to the architecture
Synchronousor
asynchronous
Worker either generates a response or works silently
Application
Job
gearmand
Worker
Job Response
Response
Synchronous
Application
Job
gearmand
Worker
Job Response
Response
Asynchronous
Application
Job
gearmand
Worker
Job Response
Response
With a response
Application
Job
gearmand
Worker
Job
With no response
“I’m done”
Application
Job
gearmand
Worker
Job
With no response
“I’m done”
DatabaseResult
Installation
Installation
1. Job server2. Client libraries3. Worker libraries
Installation
1. Job server C2. Client libraries Perl3. Worker libraries
Installation
1. Job server C2. Client libraries Perl XS3. Worker libraries
https://launchpad.net/gearmand
$./configure
$make
#makeinstall
Installing job server
#cpanGearman
Installing Perl modules
#cpanGearman::Client
#cpanGearman::Worker
or
#cpanGearman
Installing Perl modules
#cpanGearman::XS
or
Starting gearmand deamon
$gearmand‐d
Starting gearmand deamon
$gearmand‐d
or$gearmand‐d\
‐L127.0.0.1\
‐p4730
Starting worker(s)
$./worker.pl&
Starting worker(s)
$./worker.pl&
or$./workerA.pl&
$./workerB.pl&
Starting worker(s)
$./worker.pl&
or$./workerA.pl&
$./workerB.pl&
$./workerA.pl&
$./workerB.pl&
Running client
$./client.pl
Worker example#!/usr/bin/perl
usev5.10;
usestrict;
useGearman::Worker;
my$worker=newGearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo=>\&echo);
$worker‐>workwhile1;
subecho{
my$job=shift;
say'+1';
return$job‐>arg+1;
}
Worker example
my$worker=newGearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo=>\&echo);
$worker‐>workwhile1;
Worker example
my$worker=newGearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo=>\&echo);
$worker‐>workwhile1;
Connect to the server
Worker example
my$worker=newGearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo=>\&echo);
$worker‐>workwhile1;
Register worker method
Worker example
my$worker=newGearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo=>\&echo);
$worker‐>workwhile1;
Wait for incoming jobs
Worker example#!/usr/bin/perl
usev5.10;
usestrict;
useGearman::Worker;
my$worker=newGearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo=>\&echo);
$worker‐>workwhile1;
subecho{
my$job=shift;
say'+1';
return$job‐>arg+1;
}
Job method
subecho{my$job=shift;say'+1';return$job‐>arg+1;}
Receive data
Job method
subecho{my$job=shift;say'+1';return$job‐>arg+1;}
Evaluate response
Client example#!/usr/bin/perl
use v5.10;use strict;use Gearman::Client;
my $client = new Gearman::Client;$client->job_servers('127.0.0.1:4730');
for (1..10000) { my $result_ref = $client->do_task('echo', 10); say $$result_ref;}
Client example
for (1..10000) { my $result_ref = $client->do_task('echo', 10); say $$result_ref;}
Issue jobs
Asynchronous client exampleuse v5.10;use strict;use Gearman::Client;
my $client = new Gearman::Client;$client->job_servers('example.com:4730');
my $tasks = $client->new_task_set;$tasks->add_task( 'echo' => 10, { on_complete => \&completed });
$tasks->wait;
sub completed { my $result = shift; say $$result;}
my $tasks = $client->new_task_set;$tasks->add_task( 'echo' => 10, { on_complete => \&completed });$tasks->wait;
Get task set placeholder
Asynchronous client example
my $tasks = $client->new_task_set;$tasks->add_task( 'echo' => 10, { on_complete => \&completed });$tasks->wait;
Place the task
Asynchronous client example
my $tasks = $client->new_task_set;$tasks->add_task( 'echo' => 10, { on_complete => \&completed });$tasks->wait;
and register callback method
Asynchronous client example
my $tasks = $client->new_task_set;$tasks->add_task( 'echo' => 10, { on_complete => \&completed });$tasks->wait;
Wait until the job is done
Asynchronous client example
sub completed { my $result = shift; say $$result;}
Use result somehow
Asynchronous client example
XS worker exampleuse v5.10;
use strict;
use Gearman::XS::Worker;
my $worker = new Gearman::XS::Worker;
$worker->add_server('127.0.0.1', 4730);
$worker->add_function('echo', 0, \&echo, undef);
$worker->work() while 1;
sub echo {
my $job = shift;
return $job->workload() + 1;
}
XS client exampleuse v5.10;
use strict;
use Gearman::XS::Client;
my $client = new Gearman::XS::Client;
$client->add_server('127.0.0.1', 4730);
my ($ret, $result) = $client->do('echo', '15'); # arg as string!
say $result;
Async XS client exampleuse v5.10;
use strict;
use Gearman::XS::Client;
my $client = new Gearman::XS::Client;
$client->add_server('127.0.0.1', 4730);
my ($ret1, $task1) = $client->add_task('echo', '15');
my ($ret2, $task2) = $client->add_task('echo', '51');
$client->run_tasks();
Monitoring job server
TCP protocol
Connect with telnet
$ telnet localhost 4730
$ telnet localhost 4730Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.
$ telnet localhost 4730Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.status
$ telnet localhost 4730Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.statusecho2 1 1 2echo 5 2 5.
echo2 1 1 2echo 5 2 5.
echo2 1 1 2echo 5 2 5.
Known worker methods
echo2 1 1 2echo 5 2 5.
Jobs running
echo2 1 1 2echo 5 2 5.
Jobs queued
echo2 1 1 2echo 5 2 5.
Workers available
Step by step example
Part III
for(...){...}
worker.pl
worker.pl
worker.pl
worker.pl
Live demo
Top Related