Inter-process data exchange in Python
19
COMMUNICATION BETWEEN PROCESSES LVIV.PY #5
-
Upload
yehor-nazarkin -
Category
Technology
-
view
390 -
download
0
Transcript of Inter-process data exchange in Python
COMMUNICATION BETWEEN PROCESSES
LVIV.PY #5
LVIV.PY #5
ABOUT
▸ Myself: Yehor Nazarkin
▸ Occupation: Python Software Engineer at DataRobot LLC
▸ Follow up: @nimnull, github.com/nimnull
LVIV.PY #5
IPC
▸ Client-server
▸ Signals
▸ Message queue
▸ Data Storage
SERVER
CLIENT
CLIENT
BUS
SERVER
LVIV.PY #5
SERIALISATION
▸ CPU/memory aligned
▸ Takes some time
▸ Variadic performance for different data
LVIV.PY #5
SERIALISATION
LVIV.PY #5
SERIALISATION
▸ Tested for json, ujson, msgpack, pickle
▸ first test — huge attributes number, small attribute size
json: 16.32 ms
ujson: 2.89 ms
msgpack: 1.40 ms
pickle: 1.49 ms
LVIV.PY #5
SERIALISATION
LVIV.PY #5
SERIALISATION
▸ Tested for json, ujson, msgpack, pickle
▸ second test — small attributes number, huge attribute size
json: 1.59 ms
ujson: 1.91 ms
msgpack: 1.22 ms
pickle: 1.20 ms
LVIV.PY #5
ZMQ (WHATEVER MQ)
LVIV.PY #5
ZMQ (WHATEVER MQ)
LVIV.PY #5
ZMQ (WHATEVER MQ)
▸ Matrix 1000x1000, float, ~7.68Mb
▸ Mean exchange time ~ 15.61ms
▸ Client-server
▸ Serialisation
▸ Buffer I/O
▸ Netwok latency
LVIV.PY #5
MULTIPROCESSING PIPES
LVIV.PY #5
MULTIPROCESSING PIPES
PID Command USS
# — parent process
20363 python multiproc_test.py 13368 <- keep matrix
# -- child process
20367 python multiproc_test.py 1484 <- no matrix
20367 python multiproc_test.py 10056 <- matrix passed
▸ Pipe — parent connection, child connection
▸ Memory overhead on copying data
LVIV.PY #5
OS.FORK()
LVIV.PY #5
OS.FORK()
PID Command USS PSS RSS
28181 python fork_test.py 4724 15836 29792
28182 python fork_test.py 836 11701 23560 <-- before copy on write
28182 python fork_test.py 8756 15853 23944 <-- after copy on write
▸ Copy-on-write benefits.
▸ One-way channel.
LVIV.PY #5
MEMORY-MAPPED FILES
LVIV.PY #5
MEMORY-MAPPED FILES
PID Command USS PSS RSS
16737 python mmap_test.py 26540 26914 29968 <-- before mmap write
16737 python mmap_test.py 8096 19782 34200 <-- after mmap write
16795 python mmap_test.py 5484 16920 29656 <-- child processing
▸ Limitations — string/byte ndarray type (numpy)
▸ Value data types (no refcounts, etc)
▸ Serialisation
LVIV.PY #5
QUESTIONS?
▸ Source: https://github.com/nimnull/lviv.py-5
▸ Follow me: @nimnull
