Celery by dummy
-
Upload
dungjit-shiowattana -
Category
Software
-
view
87 -
download
1
Transcript of Celery by dummy
![Page 1: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/1.jpg)
By
Os Shiowattana (@djshiow)
![Page 2: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/2.jpg)
Who am I
● Os Shiowattana (@djshiow)● Software engineer for ~10 years● Currently at Origami Logic
○ We’re looking for great developers ;-)● Python experience: 3 months
○ Teaching is a good way to learn
![Page 3: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/3.jpg)
Outline
● What is Celery?● What is it for?● Simple Example
http://www.myhdiet.com/sharons-raw-celery-soup/
![Page 4: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/4.jpg)
What is celery?
● Task queue○ Distributed○ Asynchronous
● Create tasks and add it to the queue
● Workers pick them up & work on them
![Page 5: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/5.jpg)
How does it work?
Broker
Worker
Worker
Worker
….
Producer
ResultStore
![Page 6: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/6.jpg)
What is it for?
● Long running asynchronous task● Execute a task “reliably”
○ Asynchronous + Retries○ e.g. Interactions with external APIs
● Scheduling Periodic tasks
![Page 7: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/7.jpg)
How does Origami Logic use it?
● Fetch marketing metrics from all your social network daily○ Periodic○ Reliable (with retries) / Traceable○ Distributed (multiple machines) & Scalable○ Rate limits
![Page 8: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/8.jpg)
● Populate follower feed
>7 M followers
Source: http://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-celery-and-rabbitmq.html
![Page 9: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/9.jpg)
Simple Example
● Adding two numbers asynchronously !!● Setup:
○ Install RabbitMQ○ Install Celery
From: http://celery.readthedocs.org/en/latest/getting-started/first-steps-with-celery.html
![Page 10: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/10.jpg)
First, define Celery app & tasks from celery import Celery
#setup app
app = Celery('tasks', backend='amqp' , broker='amqp://guest@localhost//')
#Define task
@app.task
def add(x, y):
return x + y
Then, run celery worker
celery -A tasks worker --loglevel=info
Decorator to create a task class
![Page 11: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/11.jpg)
Lastly, trigger the taskfrom tasks import add
#run the add task asynchronously 3 seconds later
#passing (4,4) as paramadd.apply_async((4, 4) , countdown=3)
[2015-03-15 14:34:10,397: INFO/MainProcess] Received task: tasks.add[b68b0109-7e99-473b-8991-1da8adaa50e8] eta:[2015-03-15 14:34:13.385448+00:00]
[2015-03-15 14:34:14,836: INFO/MainProcess] Task tasks.add[b68b0109-7e99-473b-8991-1da8adaa50e8] succeeded in 0.0209855769999s: 8
![Page 12: Celery by dummy](https://reader034.fdocuments.in/reader034/viewer/2022042615/55a6b13b1a28ab4e5c8b47c3/html5/thumbnails/12.jpg)
Reference● http://www.slideshare.net/idangazit/an-introduction-to-celery● http://blogs.vmware.com/vfabric/2013/04/how-instagram-feeds-work-
celery-and-rabbitmq.html● http://celery.readthedocs.org/en/latest/