PEP-3156: Async I/O en Python
-
Upload
saul-ibarra-corretge -
Category
Technology
-
view
2.220 -
download
4
description
Transcript of PEP-3156: Async I/O en Python
![Page 1: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/1.jpg)
PEP-3156Async I/O en Python
Saúl Ibarra Corretgé@saghul
PyConES 2013Sunday, November 24, 13
![Page 2: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/2.jpg)
repr(self)
>>> from Amsterdam import saghul>>> saghul.work()VoIP, SIP, XMPP, chat, Real Time Communications>>> saghul.other()networking, event loops, sockets, MOAR PYTHON>>> saghul.languages()Python, C>>> saghul.message()Estoy encantado de participar en PyConES!>>>
Sunday, November 24, 13
![Page 3: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/3.jpg)
import open_source
• Core Team de libuv
• Tulip / asyncio
• Muchos experimentos con event loops y más cosas
• github.com/saghul
Sunday, November 24, 13
![Page 4: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/4.jpg)
import socketimport socket
server = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)server.bind(('127.0.0.1', 1234))server.listen(128)print("Server listening on: {}".format(server.getsockname()))
client, addr = server.accept()print("Client connected: {}".format(addr))
while True: data = client.recv(4096) if not data: print("Client has disconnected") break client.send(data)
server.close()
Sunday, November 24, 13
![Page 5: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/5.jpg)
except Exception
• Solo podemos gestionar una conexión
• Soluciones para soportar múltiples clientes
• Threads
• Procesos
• Multiplexores de I/O
Sunday, November 24, 13
![Page 6: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/6.jpg)
import threadimport socketimport thread
def handle_client(client, addr): print("Client connected: {}".format(addr)) while True: data = client.recv(4096) if not data: print("Client has disconnected") break client.send(data)
server = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)server.bind(('127.0.0.1', 1234))server.listen(128)print("Server listening on: {}".format(server.getsockname()))
while True: client, addr = server.accept() thread.start_new_thread(handle_client, (client, addr))
Sunday, November 24, 13
![Page 7: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/7.jpg)
except Exception
• Los threads añaden overhead
• Tamaño del stack
• Cambios de contexto
• Sincronización entre threads
• El GIL - NO es un problema
Sunday, November 24, 13
![Page 8: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/8.jpg)
Multiplexores de I/O
• Examinar y bloquearse a la espera de que un fd esté listo
• ¡Aun así la operación puede bloquear!
• El fd tiene que ser puesto en modo no bloqueante
• ¿Y Windows?
Sunday, November 24, 13
![Page 9: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/9.jpg)
Sunday, November 24, 13
![Page 10: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/10.jpg)
Multiplexores de I/O
1. Poner un fd en modo no bloqueante2. Añadir el fd al multiplexor3. Esperar un rato4. Realizar la operación bloqueante sobre
el fd si está listo
Sunday, November 24, 13
![Page 11: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/11.jpg)
def set_nonblocking
import fcntl
def set_nonblocking(fdobj): # unix only try: setblocking = fdobj.setblocking except AttributeError: try: fd = fdobj.fileno() except AttributeError: fd = fdobj flags = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK) else: setblocking(False)
Sunday, November 24, 13
![Page 12: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/12.jpg)
import select
• El módulo “select” implementa los distintos multiplexores de i/o
• select
• poll
• kqueue
• epoll
• En Python 3.4, módulo “selectors”
Sunday, November 24, 13
![Page 13: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/13.jpg)
sys.platform == ‘win32’
• Soporta select()!
• 64 fds por thread
• >= Vista soporta WSAPoll!
• Está roto: bit.ly/Rg06jX
• IOCP es lo que mola
Sunday, November 24, 13
![Page 14: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/14.jpg)
import IOCP
• Empezar la operación de i/o en modo overlapped
• Recibe un callback cuando ha terminado
• Completion Ports
• Abstracción totalmente distinta a Unix
Sunday, November 24, 13
![Page 15: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/15.jpg)
windows.rules = True
Sunday, November 24, 13
![Page 16: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/16.jpg)
Sunday, November 24, 13
![Page 17: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/17.jpg)
Frameworks
• Los frameworks nos abstraen de las diferentes plataformas
• Protocolos
• Integracion con otros event loops: Qt, GLib, ...
• Distintas APIs
Sunday, November 24, 13
![Page 18: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/18.jpg)
import twisted
• Usa select, poll, kqueue, epoll del módulo select
• IOCP en Windows
• Integración con otros loops como Qt
• Protocolos, transportes, ...
• Deferred
Sunday, November 24, 13
![Page 19: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/19.jpg)
import tornado
• Usa select, poll, kqueue, epoll del modulo select
• select() en Windows :-(
• Principalmente orientado a desarrollo web
• API síncrona con coroutines
Sunday, November 24, 13
![Page 20: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/20.jpg)
import gevent
• Usa libevent en la 0.x y libev en la 1.x
• select() en Windows :-(
• API síncrona con greenlet
Sunday, November 24, 13
![Page 21: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/21.jpg)
import asyncore
• raise RuntimeError(“NOT GOOD ENOUGH”)
• “asyncore: included batteries don’t fit” bit.ly/182HcHT
Sunday, November 24, 13
![Page 22: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/22.jpg)
Solución
Sunday, November 24, 13
![Page 23: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/23.jpg)
I’m not trying to reinvent the wheel. I’m trying to build a good one.
Guido van Rossum
Sunday, November 24, 13
![Page 24: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/24.jpg)
import tulipasyncio
Sunday, November 24, 13
![Page 25: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/25.jpg)
import asyncio
• Implementación de referencia del PEP-3156
• Componentes base para i/o asíncrona
• Funciona en Python >= 3.3
Sunday, November 24, 13
![Page 26: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/26.jpg)
Objetivos
• Implementación moderna de i/o asíncrona para Python
• Utilizar yield from (PEP-380)
• Pero no obligar a nadie
• No utilizar nada que requiera Python > 3.3
• Interoprabilidad con otros frameworks
Sunday, November 24, 13
![Page 27: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/27.jpg)
Objetivos
• Soporte para Unix y Windows
• IPv4 e IPv6
• TCP, UDP y pipes
• SSL básico (seguro por defecto)
• Subprocesos
Sunday, November 24, 13
![Page 28: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/28.jpg)
No Objetivos
• Perfección
• Reemplazar los frameworks actuales
• Implementaciones de protocolos concretos
• Reemplazar httplib, smtplib, ...
• Funcionar en Python < 3.3
Sunday, November 24, 13
![Page 29: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/29.jpg)
¿Interoperabilidad?
selectors iocp
asyncio
twisted tornado gevent ...
Sunday, November 24, 13
![Page 30: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/30.jpg)
¿Interoperabilidad?
rose / pyuv
asyncio
twisted tornado gevent ...
Sunday, November 24, 13
![Page 31: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/31.jpg)
ArquitecturaSunday, November 24, 13
![Page 32: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/32.jpg)
Componentes
Event loop, policy
Coroutines, Futures, Tasks
Transports, Protocols
Sunday, November 24, 13
![Page 33: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/33.jpg)
Event loopSunday, November 24, 13
![Page 34: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/34.jpg)
Event loop y policy
• Selecciona el mejor selector para cada plataforma
• APIs para crear servidores y conexiones (TCP, UDP, ...)
Sunday, November 24, 13
![Page 35: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/35.jpg)
Callbacks
• loop.call_soon(func, *args)
• loop.call_later(delay, func, *args)
• loop.call_at(when, func, *args)
• loop.time()
Sunday, November 24, 13
![Page 36: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/36.jpg)
Callbacks para I/O
• loop.add_reader(fd, func, *args)
• loop.add_writer(fd, func, *args)
• loop.remove_reader(fd)
• loop.remove_writer(fd)
Sunday, November 24, 13
![Page 37: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/37.jpg)
Señales Unix
• loop.add_signal_handler(sig, func, *args)
• loop.remove_signal_handler(sig)
Sunday, November 24, 13
![Page 38: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/38.jpg)
Interfaz con Threads
• loop.call_soon_threadsafe(func, *args)
• loop.run_in_executor(exc, func, *args)
• loop.set_default_executor(exc)
Sunday, November 24, 13
![Page 39: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/39.jpg)
Inicio y parada
• loop.run_forever()
• loop.stop()
• loop.run_until_complete(f)
Sunday, November 24, 13
![Page 40: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/40.jpg)
Acceso a la instancia
• get_event_loop()
• set_event_loop(loop)
• new_event_loop()
Sunday, November 24, 13
![Page 41: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/41.jpg)
Policy (default)
• Un event loop por thread
• Solo se crea un loop automágicamente para el main thread
Sunday, November 24, 13
![Page 42: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/42.jpg)
Policy
• Configura qué hacen get/set/new _event_loop
• Es el único objeto global
• ¡Se puede cambiar! (ejemplo: rose)
Sunday, November 24, 13
![Page 43: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/43.jpg)
Coroutines, Futures y Tasks
Sunday, November 24, 13
![Page 44: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/44.jpg)
Coroutines, Futures y Tasks
• Coroutines
• una función generator, puede recibir valores
• decorada con @coroutine
• Future
• promesa de resultado o error
• Task
• Future que ejecuta una coroutine
Sunday, November 24, 13
![Page 45: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/45.jpg)
Coroutines y yield fromimport asyncioimport socket
loop = asyncio.get_event_loop()
@asyncio.coroutinedef handle_client(client, addr): print("Client connected: {}".format(addr)) while True: data = yield from loop.sock_recv(client, 4096) if not data: print("Client has disconnected") break client.send(data)
@asyncio.coroutinedef accept_connections(server_socket): while True: client, addr = yield from loop.sock_accept(server_socket) asyncio.async(handle_client(client, addr))
server = socket.socket(family=socket.AF_INET, type=socket.SOCK_STREAM)server.bind(('127.0.0.1', 1234))server.listen(128)server.setblocking(False)print("Server listening on: {}".format(server.getsockname()))
loop.run_until_complete(accept_connections(server))
Sunday, November 24, 13
![Page 46: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/46.jpg)
Coroutines y yield from
• Imagina que el yield from no existe
• Imagina que el código es secuencial
• No te bases en la definición formal de yield from (PEP-380)
Sunday, November 24, 13
![Page 47: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/47.jpg)
Futures
• Parecidos a los Futures de PEP-3148• concurrent.futures.Future
• API (casi) idéntico:• f.set_result(); r = f.result()
• f.set_exception(e); e = f.exception()
• f.done()
• f.add_done_callback(x); f.remove_done_callback(x)
• f.cancel(); f.cancelled()
Sunday, November 24, 13
![Page 48: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/48.jpg)
Futures + Coroutines
• yield from funciona con los Futures!
• f = Future()
• Alguien pondrá el resultado o la excepción luego
• r = yield from f
• Espera a que esté listo y devuelve f.result()
• Normalmente devueltos por funciones
Sunday, November 24, 13
![Page 49: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/49.jpg)
Deshaciendo callbacks
@asyncio.coroutinedef sync_looking_function(*args): fut = asyncio.Future() def cb(result, error): if error is not None: fut.set_result(result) else: fut.set_exception(Exception(error)) async_function(cb, *args) return (yield from fut)
Sunday, November 24, 13
![Page 50: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/50.jpg)
Tasks
• Unicornios con polvo de hada
• Es una coroutine metida en un Future
• WAT
• Hereda de Future
• Funciona con yield from
• r = yield from Task(coro(...))
Sunday, November 24, 13
![Page 51: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/51.jpg)
Tasks vs coroutines
• Una coroutine no “avanza” sin un mechanismo de scheduling
• Los Tasks pueden avanzar solos, sin necesidad de esperar
• El event loop es el scheduler!
• Magia!
Sunday, November 24, 13
![Page 52: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/52.jpg)
Otro ejemploimport asyncio
loop = asyncio.get_event_loop()clients = {} # task -> (reader, writer)
def _accept_client(client_reader, client_writer): task = asyncio.Task(_handle_client(client_reader, client_writer)) clients[task] = (client_reader, client_writer)
def client_done(task): del clients[task]
task.add_done_callback(client_done)
@asyncio.coroutinedef _handle_client(client_reader, client_writer): while True: data = (yield from client_reader.readline()) client_writer.write(data)
f = asyncio.start_server(_accept_client, '127.0.0.1', 12345)server = loop.run_until_complete(f)loop.run_forever()
Sunday, November 24, 13
![Page 53: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/53.jpg)
Transports y Protocols
Sunday, November 24, 13
![Page 54: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/54.jpg)
Transports y Protocols
• Transport: representa una conexión (socket, pipe, ...)
• Protocol: representa una aplicación (servidor HTTP, cliente IRC, ...)
• Siempre van juntos
• API basada en llamadas a función y callbacks
Sunday, November 24, 13
![Page 55: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/55.jpg)
Clientes y servidores
• loop.create_connection(...)
• crea un Transport y un Protocol
• loop.create_server(...)
• crea un Transport y un Protocol por cada conexión aceptada
• devuelve un objeto Server
Sunday, November 24, 13
![Page 56: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/56.jpg)
Clientes y servidores
• loop.open_connection(...)
• wrapper sobre create_connection, devuelve (stream_reader, stream_writer)
• loop.start_server(...)
• wrapper sobre create_server, ejecuta un callback con (stream_reader, stream_writer) por cada conexión aceptada
Sunday, November 24, 13
![Page 57: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/57.jpg)
Interfaz Transport -> Protocol
• connection_made(transport)
• data_received(data)
• eof_received()
• connection_lost(exc)
Sunday, November 24, 13
![Page 58: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/58.jpg)
Interfaz Protocol -> Transport
• write(data)
• writelines(seq)
• write_eof()
• close()
• abort()
Sunday, November 24, 13
![Page 59: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/59.jpg)
MOAR PROTOCOL!
• ¡Esta tarde a las 17:30!
• “Tulip or not Tulip” - Iñaki Galarza
Sunday, November 24, 13
![Page 60: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/60.jpg)
¿Y ahora qué?
• Ven a la charla de Iñaki
• Lee el PEP-3156
• Implementa algún protocolo sencillo (cliente IRC por ejemplo)
• ¡Usa asyncio en tu próximo proyecto!
Sunday, November 24, 13
![Page 61: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/61.jpg)
¿Preguntas?
bettercallsaghul.com@saghul
Sunday, November 24, 13
![Page 62: PEP-3156: Async I/O en Python](https://reader034.fdocuments.in/reader034/viewer/2022051012/54068a498d7f72d8088b45ef/html5/thumbnails/62.jpg)
Referencias
• code.google.com/p/tulip/
• groups.google.com/forum/#!forum/python-tulip
• PEP-3156
• http://www.youtube.com/watch?v=1coLC-MUCJc
• https://www.dropbox.com/s/essjj4qmmtrhys4/SFMeetup2013.pdf
Sunday, November 24, 13