A jobs queue for processing tasks asynchronously
-
Upload
camptocamp -
Category
Technology
-
view
173 -
download
2
Transcript of A jobs queue for processing tasks asynchronously
![Page 1: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/1.jpg)
A Jobs Queue for processing tasksasynchronously
Guewen Baconnier & Leonardo Pistone
Camptocamp
1 / 32
![Page 2: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/2.jpg)
Guewen Baconnier
Developer @ CamptocampOCA committer, OCA DelegateConnector author
@guewenb
@guewen
Leonardo Pistone
Developer @ CamptocampOCA committer, OCA Delegate
@lepistone
About us
2 / 32
![Page 3: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/3.jpg)
Computers are slow!
3 / 32
![Page 4: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/4.jpg)
Computers are slow!
Humans want them to be fast!
3 / 32
![Page 5: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/5.jpg)
The problem
4 / 32
![Page 6: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/6.jpg)
User Odoo Server
Wai
ting
Hea
vy t
ask!
5 / 32
![Page 7: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/7.jpg)
Loading...
6 / 32
![Page 8: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/8.jpg)
Still loading...
7 / 32
![Page 9: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/9.jpg)
Still loading... Please be patient.
8 / 32
![Page 10: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/10.jpg)
Don't leave yet, it's still loading
9 / 32
![Page 11: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/11.jpg)
You may not believe it, but the application isactually loading...
10 / 32
![Page 12: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/12.jpg)
Take a minute to get a coffee, because it'sloading...
11 / 32
![Page 13: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/13.jpg)
Come on...
12 / 32
![Page 14: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/14.jpg)
We can try to save a few seconds
13 / 32
![Page 15: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/15.jpg)
We can try to save a few seconds
But we have more radical solutions
13 / 32
![Page 16: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/16.jpg)
User Odoo Server
Hea
vy t
ask!
Connector Runner
14 / 32
![Page 17: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/17.jpg)
Connectorodoo-connector.com
15 / 32
![Page 18: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/18.jpg)
Queue it!
Dependency on connector
16 / 32
![Page 19: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/19.jpg)
Queue it!
Dependency on connectorDeclare a job:
from openerp.addons.connector.queue.job import job
@jobdef a_heavy_task(session, model_name, record_id): # do an heavy task on record_id of model_name
16 / 32
![Page 20: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/20.jpg)
Queue it!
Dependency on connectorDeclare a job:
from openerp.addons.connector.queue.job import job
@jobdef a_heavy_task(session, model_name, record_id): # do an heavy task on record_id of model_name
Delay a job:
session = ConnectorSession.from_env(self.env)a_heavy_task.delay(session, 'res.partner', 1)
16 / 32
![Page 21: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/21.jpg)
Dequeue it!Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2 ./openerp-server --load=web,connector --workers=4
17 / 32
![Page 22: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/22.jpg)
Dequeue it!Start the server with:
ODOO_CONNECTOR_CHANNELS=root:2 ./openerp-server --load=web,connector --workers=4
ODOO_CONNECTOR_CHANNELS=root:3,root.csv:1,root.magento:3 \ ./openerp-server --load=web,connector --workers=4
17 / 32
![Page 23: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/23.jpg)
Channels
18 / 32
![Page 24: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/24.jpg)
HTT
P W
ork
ers
csv
magento
root
Running jobsCapacity
Channels
19 / 32
![Page 25: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/25.jpg)
Properties
20 / 32
![Page 26: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/26.jpg)
Priority
10 50 999
Priorityimport_order.delay(session, 1) # default is 10import_order.delay(session, 2, priority=50)import_order.delay(session, 10, priority=999)
21 / 32
![Page 27: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/27.jpg)
B C
Now + 6:00 + 12:00 +18:00 + 24:00
A
ETAimport_order.delay(session, 1) # Aimport_order.delay(session, 1, eta=6*60*60) # Bimport_order.delay(session, 2, eta=datetime.now() + timedelta(days=1)) # C
22 / 32
![Page 28: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/28.jpg)
Retriesimport_order.delay(session, 1, max_retries=3)
23 / 32
![Page 29: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/29.jpg)
Retriesimport_order.delay(session, 1, max_retries=3)
Invoke a retry@jobdef import_order(session, args): try: do_operation() except (socket.gaierror, socket.error, socket.timeout) as err: raise RetryableError( 'A network error caused the failure of the job: ' '%s' % err)
23 / 32
![Page 30: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/30.jpg)
Best Practices
24 / 32
![Page 31: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/31.jpg)
Outdating Data in jobs can become outdated.
No:
@jobdef example(session, record_id, vals): export(record_id, vals)
Yes:
@job def example(session, record_id): export(session.env['model'].browse(record_id))
25 / 32
![Page 32: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/32.jpg)
Outdating
Existence
A job can refer to a record which has been deleted. Alwayscheck if it still exists.
No:
@jobdef example(session, record_id): export(session.env['model'].browse(record_id))
Yes:
@job def example(session, record_id): record = session.env['model'].browse(record_id) if record.exists(): export(record)
26 / 32
![Page 33: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/33.jpg)
Outdating
Existence
Idempotence
A job should, when possible, produce the same resultwhen executed several times.
No:
@jobdef example(session, record_id): export(session.env['model'].browse(record_id))
Yes:
@job def example(session, record_id): record = session.env['model'].browse(record_id) if record.exists(): if not record.exported: export(session.env['model'].browse(record_id))
27 / 32
![Page 34: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/34.jpg)
Useful Patterns
28 / 32
![Page 35: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/35.jpg)
Fanout JobA job generating other jobs.
@jobdef import_file(session, filepath): with open(filepath) as f: for line in f: import_line.delay(session, line)
29 / 32
![Page 36: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/36.jpg)
Try or delayIf an operation failed, try it later.
@jobdef do_operation(session, args): # work
try: do_operation(session, args)except TimeoutError: do_operation.delay(session, args, eta=10*60)
30 / 32
![Page 37: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/37.jpg)
Extract highly concurrent tasksAnd put them in a one-by-one channel.
31 / 32
![Page 38: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/38.jpg)
Thanks!
32 / 32
![Page 39: A jobs queue for processing tasks asynchronously](https://reader034.fdocuments.in/reader034/viewer/2022042615/55c4527bbb61eb817c8b467c/html5/thumbnails/39.jpg)
Thanks!
OCA Sponsors
32 / 32