LuaSocket behind the scenesLuaSocket behind the scenes

Diego Nehab

Short BioShort Bio

• Graduated from PUC in CS & E, 1999;

• Worked in Tecgraf 1995-2002;

• MSc in PL with Roberto, 2001;

• 3rd year PhD candidate at Princeton;

• Computer Graphics.

Outline of talkOutline of talk

• A few historical notes

• Case study: SMTP support

• Protocol abstraction

• Message abstraction

• Implementation highlights

• Conclusions

Historical notesHistorical notes

• 1.0, 1999, 1.5k C, 200 man

• 1.1, 2000, 1.5k C, 1.3k Lua, 500 man

• added protocol support for HTTP, SMTP, FTP

• 1.2, 2001, 2k C, 1.3k Lua, 900 man

• buffered input and non-blocking I/O

• UDP support

• object oriented syntax

Historical notesHistorical notes

• 1.3, 2001, 2.3k C, 1.6k Lua, 1.2k man

• streaming with callbacks

• added select function

• 1.4, 2001-2, 2.2k C, 2.2k Lua, 1.9k man

• LTN7

• added URL module

• named parameters

David Burgess

Current versionCurrent version

• 2.0, 2005, 4.6k C, 2.5k Lua, 4.7k man

• Extensible C architecture, split in modules

• LTN12 (sources, sinks and filters)

• MIME support (partial but honest)

• Multipart messages support

• LTN13 (finalized exceptions)

• Package proposal

• Improved non-blocking code, robust to signals...

Outline of talkOutline of talk

• A few historical notes

• Case study: SMTP support

• Protocol abstraction

• Message abstraction

• Implementation highlights

• Conclusions

from

rcpt

body

SMTP (RFC2821)SMTP (RFC2821)

[lua:roberto] telnet mail.tecgraf.puc-rio.br 25220 tecgraf.puc-rio.br ESMTP Sendmail 8.9.3/8.9.3helo lua250 tecgraf.puc-rio.br Hello lua, pleased to meet youmail from: <roberto@inf.puc-rio.br>250 <roberto@inf.puc-rio.br>... Sender okrcpt to: <diego@tecgraf.puc-rio.br>250 <diego@tecgraf.puc-rio.br>... Recipient okdata354 Enter mail, end with "." on a line by itselfSubject: World domination: instructions.

Commence stage two..250 RAA10452 Message accepted for deliveryquit221 tecgraf.puc-rio.br closing connection

Protocol abstractionProtocol abstraction

status, error = smtp.send {

from = "<roberto@inf.puc-rio.br>",

rcpt = "<diego@tecgraf.puc-rio.br>",

body = "Subject: World domination: instructions.\r\n\r\n" ..

"Comence stage two."

}

• What if body is large?

LTN12 sourcesLTN12 sources

function ltn12.source.file(handle)

return function()

local chunk = handle:read(BLOCKSIZE)

if not chunk then handle:close() end

return chunk

end

end

• Use callback function that produces data;

• Returns one chunk each time called;

• Signals termination returning nil.

Using sourcesUsing sources

status, message = smtp.send {

from = "<roberto@inf.puc-rio.br>",

rcpt = "<diego@tecgraf.puc-rio.br>",

body = ltn12.source.file(io.open("/mail/body", "r"))

}

• What if body is complicated?

headers

Message Format (RFC2822)Message Format (RFC2822)

From: Roberto Ierusalimschy <roberto@inf.puc-rio.br>

To: Diego Nehab <diego@tecgraf.puc-rio.br>

Subject: World domination: roadmap.

Content-Type: multipart/mixed; boundary=part

This message contains attachments

--part

Content-Type: text/plain

Please see attached roadmap.

--part

Content-Type: text/html; name="roadmap.html"

...

--part--

headers

body

part 2

bodypart 1

Message abstractionMessage abstraction

declaration = {

headers = {

subject = "World domination",

from = "Roberto <roberto@inf.puc-rio.br>",

to = "Diego <diego@tecgraf.puc-rio.br>"

},

preamble = "This message contains attachments.",

[1] = {

headers = { ... },

body = "Please see attatched roadmap."

},

[2] = {

headers = { ... },

body = ltn12.source.file(io.open("/plans/roadmap.html", "r"))

}

}

Our message APIOur message API

status, message = smtp.send {

from = "<roberto@inf.puc-rio.br>",

rcpt = "<diego@tecgraf.puc-rio.br>",

body = smtp.message(declaration)

}

• Transform declaration into an LTN12 source;

• Pass source as body to sending function.

How hard is it?How hard is it?

• Message structure is recursive;

• Need to return chunks but mantain context;

• Nightmare to write in C!;

• Use coroutines;

• Write function recursively, naturally;

• Call yield with each chunk;

• Next call resumes wherever we left.

Zoom in on attachmentsZoom in on attachments

[2] = {

headers = {

["content-type"] = 'text/html; name="roadmap.html"',

["content-disposition"] = 'attachment; filename ="roadmap.html"'

},

body = ltn12.source.file(io.open("/plans/roadmap.html", "r"))

}

• Would like to send PDF;

• Binary data has to be encoded (Base64);

• Want to encode on-the-fly.

LTN12 filters and chainsLTN12 filters and chains

• Filters process data one chunk at a time;

• MIME module provides common filters:

• base64, quoted-printable, stuffing, line-wrap...

• Can chain two filters together: factory

• Produce a filter with the composite effect

• Can chain a filter with a source: factory

• Produce a source that returns filtered data.

Zoom in on attachmentsZoom in on attachments

[2] = {

headers = {

["content-type"] = 'application/pdf; name="roadmap.pdf"',

["content-disposition"] = 'attachment; filename ="roadmap.pdf"',

["content-description"] = 'Detailed world domination plan',

["content-transfer-encoding"] = 'BASE64'

},

body = ltn12.source.chain(

ltn12.source.file(io.open("/plans/roadmap.pdf", "r")),

ltn12.filter.chain(

mime.encode("base64"),

mime.wrap("base64")

)

)

}

Creating filters: high-levelCreating filters: high-level

function ltn12.filter.cycle(low, ctx, extra)

return function(chunk)

local ret

ret, ctx = low(ctx, chunk, extra)

return ret

end

end

function mime.normalize(marker)

return ltn12.filter.cycle(mime.eol, 0, marker)

end

• Chunks can be broken arbitrarily;

• Filters have to keep context between calls;

Creating filters: low-levelCreating filters: low-level

int eol(lua_State *L) {

int ctx = luaL_checkint(L, 1);

size_t isize = 0;

const char *input = luaL_optlstring(L, 2, NULL, &isize);

const char *last = input + isize;

const char *marker = luaL_optstring(L, 3, CRLF);

luaL_Buffer buffer;

luaL_buffinit(L, &buffer);

while (input < last)

ctx = translate(*input++, ctx, marker, &buffer);

luaL_pushresult(&buffer);

lua_pushnumber(L, ctx);

return 2;

}

Creating filters: low-levelCreating filters: low-level

#define candidate(c) (c == CR || c == LF)

int translate(int c, int last, const char *mark, luaL_Buffer *buffer) {

if (candidate(c)) {

if (candidate(last)) {

if (c == last) luaL_addstring(buffer, mark);

return 0;

} else {

luaL_addstring(buffer, mark);

return c;

}

} else {

luaL_putchar(buffer, c);

return 0;

}

}

SMTP dependenciesSMTP dependencies

socket

mime

smtp

tp

ltn12

Error checkingError checking

• Function return convention

• Return nil, followed by message on error;

function metat.__index:greet(domain)

local r, e = self.tp:check("2..")

if not r then return nil, e end

r, e = self.tp:command("HELO", domain)

if not r then return nil, e end

return self.tp:check("2..")

end

• Tedious, error prone, virotic, not finalized.

LTN13 exceptionsLTN13 exceptions

• try = newtry(finalizer): factory;

• On success, try returns all arguments;

• On failure, throws the second argument;

• Calls finalizer before raising the exception.

• foo = protect(bar): factory;

• foo executes bar in a protected environment;

• Returns nil followed by any thrown error.

No 'if' statementsNo 'if' statements

function metat.__index:greet(domain)

self.try(self.tp:check("2.."))

self.try(self.tp:command("HELO", domain))

return self.try(self.tp:check("2.."))

end

• Internal functions throw exceptions;

• try calls tp.close() on error;

• External functions can be protected.

ConclusionsConclusions

• Hope you like our API, we do;

• It is easy to implement;

• Function factories + clusures, coroutines

• It is fast;

• Time critical in C, management in Lua;

• Questions?