1

Combining Events and

Threads for Scalable Network Services

Peng Li and Steve Zdancewic

University of Pennsylvania

PLDI 2007, San Diego

2

Overview A Haskell framework for massively concurrent network

applications Servers, P2P systems, load generators

Massive concurrency ::= 1,000 threads? (easy)

| 10,000 threads? (common)

| 100,000 threads? (challenging)

| 1,000,000 threads? (20 years later?)

| 10,000,000 threads? (in 15 minutes) How to write such programs?

The very first decision to make: the programming modelShall we use threads or events?

A lazy, purely functional programming languagehttp://www.haskell.org

3

Threads vs. EventsThe multithreaded model

One thread ↔ one client Synchronous I/O Scheduling: OS/runtime libs

int send_data(int fd1, int fd2) { while (!EOF(fd1)) { size = read_chunk(fd, buf, count); write_chunk(fd, buf, size); } …

The event-driven model: One thread ↔ 10000 clients Asynchronous I/O Scheduling: programmer

while(1) { nfds=epoll_wait(kdpfd, events, MAXEVT,-1); for(n=0; n<nfds; ++n) handle_event(events[n]); …

Threads Events

Expressiveness and Abstraction

(for programming each client)

Synchronous I/O + intuitive control flow primitives

Finite state machines /

Continuation-passing style (CPS) programming

Flexibility and Control

(for resource scheduling)

Baked into OS/runtime, difficult to customize

Programmer has complete control – tailored to each application’s needs

“Why threads are a bad idea (for most purposes)” [USENIX ATC 1999]

“Why events are a bad idea (for high-concurrency servers)”

[HotOS 2003]

4

Can we get the best of both worlds?

The bridge between threads/events?(some kind of “continuation” support)

Resource scheduling: events•Written as part of the application • Tailored to application’s needs

Programming with each client: threads • Synchronous I/O• Intuitive control-flow primitives

One application program

5

Roads to lightweight, application-level concurrency

Direct language support for continuations:Good if you have them

Source-to-source CPS translationsRequires hacking on compiler/runtimeOften not very elegant

Other solutions? (no language support) (no compiler/runtime hacks)

6

The poor man’s concurrency monad “A poor man’s concurrency monad” by Koen Claessen,

JFP 1999. (Functional Pearl) The thread interface:

The CPS monad The event interface:

A lazy, tree-like data structure called “trace”

server_loop s = do { sock <- sock_accept s; sys_fork (client sock); server_loop;}client_loop sock = do { sock_send sock data; sock_close sock;}sock_send sock data = do { ... n<-sys_nbio (write_nb ...); ...; sys_epoll_wait sock EPOLL_READ; ... foo; ... n<-sys_nbio (write_nb ...); ...}

scheduler = do { ... trace <- fetch_thread; execute trace; ...}

execute trace = case trace of SYS_NBIO c -> do { cont <- c; execute cont; } SYS_FORK t1 t2 -> ...

SYS_NBIO(accept)

SYS_EPOLL_WAIT(s)

SYS_NBIO(accept)

SYS_FORK

SYS_NBIO(write_nb)

SYS_EPOLL_WAIT(sock)

Multithreaded code Trace

Thread Abstraction E

vent Abstraction

InternalRepresentation Scheduler code

CPSMonad

SYS_NBIO(write_nb)

7

Questions on the poor man’s approach

Does it work for high-performance network services?(using a pure, lazy, functional language?)

How does the design scale up to real systems? Symmetrical multiprocessing? Synchronization? I/O?

How cheap is it? How much does a poor man’s thread cost?

How poor is it? Does it offer acceptable performance?

8

Our experiment

A high-performance Haskell framework for massively-concurrent network services!!!

Supported features: Linux Asynchronous IO (AIO) epoll() and nonblocking IO OS thread pools SMP support Thread synchronization primitives

Applications developed IO benchmarks on FIFO pipes / Disk head scheduling A simple web server for static files HTTP load generator Prototype of an application-level TCP stack

We used the Glasglow Haskell Compiler (GHC)

9

Exception handling

Nested function calls

Conditional branches

Synchronous call to I/O lib

Recursion

Multithreaded code example

10

Event-driven code exampleA wrapper function to the C library call using the Haskell Foreign Function Interface

(FFI)

Put events in queues for processing in other OS threads

An event loop running in a separate OS thread

11

A complete event-driven I/O subsystem

Submit AIO request

Register event handler

ready_queue

blio_queue

worker_blio

worker_epoll

worker_aio

SYS_BLIO

/ forkworker_main

worker_main

worker_main

worker_main

Epollinterface

AIO interface

Event notification

AIO completion

Fet

ch t

hre

ad

s

OS thread pool for CPS thread execution

with event handler

Fetch thread

System call completion, thread ready to run

Context switchworker_blio

OS thread pool for blocking I/O

Haskell Foreign Function

Inteface (FFI)

Each event loop runs in a

separate OS thread

One “virtual processor” event

loop for each CPU

12

Modular and customizable I/O system (add a TCP stack if you like)

Submit AIO request

Register event handler

ready_queue

blio_queue

worker_epoll

worker_aio

SYS_BLIO

/ forkworker_main

worker_main

worker_main

worker_main

Epollinterface

AIO interface

Event notification

AIO completion

Fetc

h t

hre

ad

s

OS thread pool for CPS thread execution

with event handler

Fetch thread

System call completion, thread ready to run

Context switch

TCP stackstates

TCP User requests

worker_tcp_input

worker_tcp_timer

Request Completion

Request Completion

Blocking

worker_blio

worker_blio

OS thread pool for blocking I/O

worker_main

Define / interpret TCP syscalls (22 lines)

Event loop for incoming packets (7 lines)

Event loop for timers (9 lines)

13

How cheap is a poor man’s thread?

Minimal memory consumption: 48 bytes

Each thread just loops and does nothing Actual size determined by thread-local states

Even an ethernet packet can be >1,000 bytes… Pay as you go --- only pay for things needed

In contrast: A Linux POSIX thread’s stack has 2MB by default The state-of-the-art user-level thread system (Capriccio) use at

least a few KBs for each thread

Observation:The poor man’s thread is extremely memory-efficient(Challenging most event-driven systems)

48 bytes

14

I/O scalability test Comparison against the Linux POSIX Thread Library

(NPTL) Highly optimized OS thread implementation Each NPTL thread’s stack limited to 32KB

Mini-benchmarks used: Disk head scheduling (all threads running) FIFO pipe scalability with idle threads (128 threads running)

15

A simple web server

16

How poor is the poor man’s monad?

Not too shabby Benchmarks shows comparable (if not higher)

performance to existing, optimized systems

An elegant design is more important than 10% performance improvement

Added benefit: type safety for many dangerous things Continuations, thread queues, schedulers, asynchronous I/O

17

Related Work We are motivated by two projects:

Twisted: the python event-driven framework for scalable internet applications- The programmer must write code in CPS

Capriccio: a high-performance user-level thread system for network servers- Requires C compiler hacks

- Difficult to customize (e.g. adding SMP support)

Continuation-based concurrency [Wand 80], [Shivers 97], …

Other languages and programming models: CML, Erlang, …

18

Conclusion Haskell and The Poor Man’s Concurrency

Monad are a promising solution for high-performance, massively-concurrent networking applications:

Get the best of both threads and events!

This poor man’s approach is actually very cheap, and not so poor!

http://www.cis.upenn.edu/~lipeng/homepage/unify.html