Queue your work

Jurian Sluiman - uncon session PHPBenelux 2014

About me

• Jurian Sluiman• Founder @ soflomo (Delft, The Netherlands)• Web development (eCommerce, health care)• Zend Framework 2 contributor• Blogger: http://juriansluiman.nl• Twitter: @juriansluiman

Queue systems

• Execute tasks in background• Return your response fast!

Server

request

response

Queue systems

• Execute tasks in background• Return your response fast!

• Producer: piece of code pushing new jobs• Worker: piece of code consuming jobs

Server(Producer)

request

response WorkerQueue

Queue systems

Server(Producer)

request

response

Worker

Server(Producer)

request

response

Server(Producer)

request

response

Worker

Queue

Queue

Worker

Worker

So why?

Advantages• Asynchronous• Resilience• Scalable• Atomic

Disadvantages• Asynchronous• Complexity (is it?)

Types

1.Dedicated job queues• Gearman, beanstalkd, Celery

2.Message queues• RabbitMQ, ZeroMQ, ActiveMQ

3.SaaS queues• Amazon SQS, IronMQ

4.Databases• Redis, (My)SQL

A list of most of the queues: http://queues.io

Job queues

1.Gearman• Widely known • PECL extension available (http://php.net/gearman)

2.Beanstalkd• Small footprint• Very easy setup

Gearman

1. Run task in background// producer$client = new GearmanClient();$client->addServer(); // 127.0.0.1:4730$client->doBackground('email', $workload);

// worker$worker = new GearmanWorker();$worker->addServer();$worker->addFunction('email', function($job) { $params = $job->getWorkLoad(); mail($params['to'], $params['subject'], $params['msg']);};

while($worker->work());

Gearman

2. Normal tasks$client = new GearmanClient();$client->addServer();$result = $client->doNormal('generate-key');// use $result

Gearman

2. Normal tasks$client = new GearmanClient();$client->addServer();$result = $client->doNormal('generate-key');// use $result

3. Multiple servers$servers = array('192.168.1.200', '192.168.1.201');shuffle($servers);

$client = new GearmanClient();$client->addServers(implode(',', $servers));

Gearman

Advantages• Multi-tenant• Job status• Priorities• Persistent

Disadvantages• Requires compilation!• Pre-defined functions• Opt-in persistency

Beanstalkd

1. Run task in background// producer$client = new Pheanstalk_Pheanstalk('127.0.0.1');$client->put($data);

// worker$worker = new Pheanstalk_Pheanstalk('127.0.0.1');while($job = $worker->reserve()) { $data = $job->getData(); $function = $data['function']; $args = $data['args'];

call_user_func_array($function, $args);}

Beanstalkd

2. Priority & delayed task$client = new Pheanstalk_Pheanstalk('127.0.0.1');$prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY;$delay = 5 * 60;$client->put($data, $prio, $delay);

Beanstalkd

2. Priority & delayed task$client = new Pheanstalk_Pheanstalk('127.0.0.1');$prio = Pheanstalk_PheanstalkInterface::DEFAULT_PRIORITY;$delay = 5 * 60;$client->put($data, $prio, $delay);

3. Using tubes// producer$client = new Pheanstalk_Pheanstalk('127.0.0.1');$client->putInTube('image', $data);

// worker$client = new Pheanstalk_Pheanstalk('127.0.0.1');$job = $client->reserveFromTube('image');

Beanstalkd

Advantages• Fast!• Priorities & delays• Job life cycle• Time-to-run• Persistent• Flexible payload

Disadvantages• Single-tenant• Fairly unknown

How fast?

Gearman• 1mln jobs• Push:~205 seconds

~4900 ops/sec• Pull: ~ 180 seconds

~ 5500 ops /sec

Beanstalkd• 1mln jobs• Push:~120 seconds

~ 8300 ops/sec• Pull: ~ 150 seconds

~ 6600 ops/sec

Tested on a 2.5GHz VPS, 1 core, 1GB RAM – http://gist.github.com/juriansluiman/8593421

Message queues

1.RabbitMQ• AMQP implementation (Message-Oriented-Middleware)• High availability, multi-tenancy, persistency

2.ØMQ “ZeroMQ”• Higher throughput than TCP• Flexible socket library, create your own patterns

ØMQ

Example// producer$context = new ZMQContext();$producer = new ZMQSocket($context, ZMQ::SOCKET_REQ);$producer->bind(“tcp://localhost:5555”);

$producer->send($payload);

// worker$context = new ZMQContext();$worker = new ZMQSocket($context, ZMQ::SOCKET_REP);$worker->connect(“tcp://*:5555”);

while(true) { $payload = $worker->recv();}

Message queues

Advantages• Extremely flexible• AMQP• Message broker• Extreme & advanced

Disadvantages• Extremely flexible• DIY• No “best way”• Routing, pub/sub,

ventilators, channels

Software-as-a-Service

1.Amazon Simple Queue Service (SQS)• Useful within EC2 instances• http://aws.amazon.com/sqs

2.IronMQ• They say they're better than SQS• http://iron.io/mq

Advantages• No maintenance• Easy to set-up• Easy to scale

Disadvantages• HTTP latency• Distributed (SQS)• Cost

Software-as-a-Service

1.Redis• Extremely light-weight key/value store• BRPOPLPUSH• Watch back Ross Tuck @ PHPBenelux 2013

2.(My)SQL• Only if you have to

Databases

Databases

Advantages• Well known set-up• OK for shared hosting

Disadvantages• Not meant for jobs• DIY

Queue abstraction layers

1.SlmQueue (April 2012)

2.php-queue (September 2012)

3.BBQ (June 2013)

4.Laravel Queue component

SlmQueue

• Supports Beanstalkd, SQS and Doctrine• Redis and Gearman support coming• Integrated with ZF2, but not required• Packagist: slm/queue + slm/queue-beanstalkd• GitHub: http://github.com/juriansluiman/SlmQueue

SlmQueue

SlmQueue\Job\JobInterfaceinterface JobInterface{ public function execute();}

SlmQueue

SlmQueue\Job\JobInterfaceinterface JobInterface{ public function execute();}

SlmQueue\Queue\QueueInterfaceinterface QueueInterface{ public function push(JobInterface $job, $options); public function pop();}

SlmQueue

SlmQueue\Worker\WorkerInterfaceinterface WorkerInterface{ public function processQueue($name, $options); public function processJob(JobInterface $job);}

Workers

Run via ZF2 appphp public/index.php queue beanstalkd default

Configuration• Time-out for blocking calls• Maximum number of cycles• Maximum memory usage• Signal handlers for SIGTERM and SIGINT

Dependency injection

MyModule\Job\Emailclass Email extends AbstractJob{ protected $transport;

public function __construct(Transport $transport) { $this->transport = $transport; }

public function execute() { $payload = $this->getContent(); $message = new Message; $message->setTo($payload['to']); $message->setMessage($payload['message']);

$this->transport->send($message); }}

Dependency injection

MyModule\Factory\EmailJobFactoryuse MyModule\Job\Email as EmailJob;class EmailJobFactory implements FactoryInterface{ public function createService(ServiceLocator $sl) { $transport = $sl->get('MyEmailTransport'); return new EmailJob($transport); }}

module.config.php'slm_queue' => [ 'job_manager' => [ 'MyEmailJob' => 'MyModule\Factory\EmailJobFactory' ]]

Dependency injection

Example: ZF2 Controllerpublic function fooAction(){ $payload = array( 'to' => 'jurian@juriansluiman.nl', 'message' => 'Hi there!', );

$this->queue('default') ->push('MyEmailJob', $payload);}

Queue aware jobs

MyModule\Job\Fooclass Foo extends AbstractJob implements QueueAwareInterface{ use QueueAwareTrait;

public function execute() { // work here

$job = new BarJob(); $this->getQueue()->push($job); }}

Pro-tips

1.Start experimenting now!

2.Atomic jobs

3.Log everything

4.Use a control system like supervisord

Questions?

Jurian Sluiman - uncon session PHPBenelux 2014

Jobs in services

MyModule\Service\Fooclass Foo{ protected $queue;

public function __construct(QueueInterface $queue) { $this->queue = $queue; }

public function doSomething() { // work here

$job = new BarJob; $this->queue->push($job); }}

Lazy servicesclass Buzzer{ public function __construct() { sleep(5); }

public function buzz() { // do something }}

Lazy services with a Proxy pattern by Marco Pivetta (Ocramius)

Lazy servicesclass BuzzerProxy extends Buzzer{ private $sl; private $instance;

public function __construct(ServiceLocator $sl) { $this->sl = $sl; }

private function initialize() { $this->initialized = true; $this->original = $this->sl->get('Buzzer'); } public function buzz() { if (!$this->initialized) { $this->initialize(); } return $this->instance->buzz(); }}