A Theory ofHygienic Macros

PhD Thesis ProposalDavid Herman

2

The power of macros

Derived (user-defined) syntactic constructs Defined by rewriting to existing constructs Translated at compile-time

(define-macro (increment! a) (set! a (+ a 1)))

3

The power of macros

Abstractions for scope, control, program structure Powerful meta-programming idioms Allow for a minimal language core

functions, recursion, iteration, primitive datatypes

custom loop forms, derived datatypes, first-class module systems, object-orientation, test harnesses, data

constructors, laziness, short-circuit boolean operators, iteration protocols,

static analyses, parser generators, partial evaluation, debuggers,

coroutines, exceptions, threads, type checkers,

introspection, etc…

local bindings, software contracts,

languagedesigner

language user

4

The need for hygiene

(define-macro (or e1 e2) (with ([t e1]) (if t t e2)))

(with ([x 1]) (or #f x))=>(with ([x 1]) (with ([t #f]) (if t t x)))

(with ([t 1]) (or #f t))=>(with ([t 1]) (with ([t #f]) (if t t t)))

5

The need for hygiene

(define-macro (or e1 e2) (with ([t e1]) (if t t e2)))

(with ([x 1]) (or #f x))=>(with ([x 1]) (with ([t #f]) (if t t x)))

(with ([t 1]) (or #f t))=>(with ([t 1]) (with ([t′ #f]) (if t′ t′ t)))

6

What are hygienic macros?

7

What are hygienic macros?

(define-macro (or e1 e2) (with ([t e1]) (if t t e2)))

(with ([x 1]) (or #f x))=>(with ([x 1]) (with ([t #f]) (if t t x)))

(with ([t 1]) (or #f t))=>(with ([t 1]) (with ([t #f]) (if t t t)))

8

What are hygienic macros?

9

What are hygienic macros?

KFFD ’86:

Generated identifiers that become binding instances in the completely expanded program must only bind variables that are generated at the same transcription step.

i.e., not provided as the macro’s input

i.e., a single rewriting of a macro application

10

What are hygienic macros?

Clinger and Rees ’91:

1. It is impossible to write a macro that inserts a binding that can capture references other than those inserted by the macro.

2. It is impossible to write a macro that inserts a reference that can be captured by bindings other than those inserted by the macro.

again: not provided as the macro’s input

11

Why isn’t this enough?

12

Pathological macros in Scheme

Impossible with hygienic macros, right?

(loop (begin (when (prime? i) (break i)) (increment! i)))

13

Pathological macros in Scheme

Contrast with the explicit version:

(loop/explicit break (begin (when (prime? i) (break i)) (increment! i)))

provided as input

14

Pathological macros in Scheme

Petrofsky extraction:

(loop (begin (when (prime? i) (break i)) (increment! i)))

15

Pathological macros in Scheme

Dumpster diving:

(loop (begin (when (prime? i) (break i)) (increment! i)))

16

Pathological macros in Scheme

Dumpster diving:

(loop/explicit break (begin (when (prime? i) (break i)) (increment! i)))

17

Pathological macros in Scheme

(define-macro (murky a e) (begin (set! a e) (lambda (a) e)))

(let ([foo 10]) (murky foo (+ foo 1)))

18

Pathological macros in Scheme

(define-macro (indecisive ([a e]) body) (if-zero …complicated computation… ((lambda (a) body) e) ((lambda (a) e) body)))

19

What do we know about hygiene?

20

Hygiene and lexical scope

(define-macro (or e1 e2) (with ([t e1]) (if t t e2)))

(with ([x 1]) (or #f x))=>(with ([x 1]) (with ([t #f]) (if t t x)))

(with ([t 1]) (or #f t))=>(with ([t 1]) (with ([t′ #f]) (if t′ t′ t)))

=

21

Hygienic macro expansion isinsensitive to -conversion.

The essence of hygienic macros

In other words,hygienic = respects

!

22

What is the scope of with?

(define-macro (with ([a e1]) e2) …)

First try: look at results of expansion

(with ([x 1]) (or #f x))=>((lambda (x) ((lambda (t) (if t t x)) #f) 1)

(with ([t 1]) (or #f t))=>((lambda (t) ((lambda (t′) (if t′ t′ t)) #f) 1)

23

Using expansion to define scope

Goal: define hygienic macro expansion in terms of -equivalence.

Strategy: define -equivalence in terms of hygienic macro expansion.

Oops.

24

What is the scope of with?

(define-macro (with ([a e1]) e2) …)

Better idea: provide a binding specification

with consumes an argument of shape(with ([a e1]) e2) and produces an expression, where:

a is an identifier e1 and e2 are expressions a is bound in e2

25

What is the scope of with?

(define-macro (with ([a e1]) e2) …)

Better idea: provide a binding specification

with : (with ([<a> expr]) expra) → expr

26

Annotated with macro

(define-macro (with ([a e1]) e2) : expr [(a : binder) (e1 : expr) (e2 : expr a)] …)

States macro writer’s intention about scope Admits rigorous -equivalence definition Provides correctness criterion for hygienic

macro expansion

27

Thesis

The key to specifying and verifying hygienic macro expansion is specifying the binding structure of macros.

28

A Theory of Hygienic Macros

m, a model of hygienic macros

1. Macro binding specifications (types)2. Theory of -equivalence3. High-level expansion semantics4. Definition and proof of hygiene

29

A Theory of Hygienic Macros

m, a model of hygienic macros

1. Macro binding specifications (types)2. Theory of -equivalence3. High-level expansion semantics4. Definition and proof of hygiene

30

m: a model of hygienic macros

(with ([x 1]) x)

(x.x) 1

S-expressions

-calculus

31

m: a model of hygienic macros

→ (with ([x 1]) x)

→ ((lambda (x) x) 1)

→ ((x.x) 1)

→ (x.x) 1

32

m: a model of hygienic macros

let syntax fun = macro ((a) e) : ((<a>) expra) => a.e with = macro (([a e]) body) : (([<a> expr])

expra) => ((a.body) e)in (with ([f (fun (x) 42)]) (f y.y))end

33

A Theory of Hygienic Macros

m, a model of hygienic macros

1. Macro binding specifications (types)2. Theory of -equivalence3. High-level expansion semantics4. Definition and proof of hygiene

34

1. Macro binding specifications

Shape types based on Culpepper and Felleisen ’03:

with : (with ([id expr]) expr) → expr

Types for reasoning about scope inspired by nominal datatypes and nominal logic of Gabbay and Pitts ’01

35

1. Type checking

let syntax m = macro p : => …in x.eend

[m : → expr] ⊢ x.e : expr

36

1. Type checking

let syntax m = macro p : => …in x.eend

[m : → expr, x : expr] ⊢ e : expr

37

1. Meta-level scope

let syntax m = macro (e1 e2) : (expr expr) => t.(e1 e2)in …end

[t : expr], [e1 : expr, e2 : expr] ⊢ e : expr

38

1. Two dimensions of binding

: program bindings (object-level) : macro pattern bindings (meta-level)

, ⊢ s :

41

A Theory of Hygienic Macros

m, a model of hygienic macros

1. Macro binding specifications (types)2. Theory of -equivalence3. High-level expansion semantics4. Definition and proof of hygiene

42

2. Theory of -equivalence Adapted to macro applications by exploiting

shape type annotations Relies on novel shape-directed -conversion Based on Gabbay/Pitts “swapping” formulation of

-equivalence

43

2. -equivalence via swapping

let val x = x in x (fn x => x) end=

let val y = x in y (fn w => w) end

binding occurrence

44

2. -equivalence via swapping

let val z = x in x (fn x => x) end=

let val z = x in y (fn w => w) end

free

45

2. -equivalence via swapping

let val z = x in x (fn x => x) end=

let val z = x in y (fn w => w) end

46

2. -equivalence via swapping

let val z = x in z (fn z => z) end=

let val z = x in z (fn w => w) end

47

2. -equivalence via swapping

let val z = x in z (fn z′ => z′) end=

let val z = x in z (fn z′ => z′) end

48

2. -equivalence via swapping For each binder x, pick fresh name z Rename the binding occurrence of x In the body, swap all occurrences of x with z Recur on subterms Compare results syntactically

49

2. -equivalence for m

(with ([x x]) (x (fun (x) x)))

=

(with ([y x]) (y (fun (w) w)))

?

50

2. Shape-directed -conversion

(with ([x x]) (x (fun (x) x)))

(with ([<a> expr]) expra)

(with ([y x]) (y (fun (w) w)))

binding occurrence

51

2. Shape-directed -conversion

(with ([z x]) (x (fun (x) x)))

(with ([<a> expr]) expra)

(with ([z x]) (y (fun (w) w)))

not in scope of a;not converted

52

2. Shape-directed -conversion

(with ([z x]) (x (fun (x) x)))

(with ([<a> expr]) expra)

(with ([z x]) (y (fun (w) w)))

entering scope of a;swap all occurrences of x / y

with z

53

2. Shape-directed -conversion

(with ([z x]) (z (fun (z) z)))

(with ([<a> expr]) expra)

(with ([z x]) (z (fun (w) w)))

54

2. Shape-directed -conversion

(with ([z x]) (z (fun (z) z)))

(fun (<b>) exprb)

(with ([z x]) (z (fun (w) w)))

55

2. Shape-directed -conversion

(with ([z x]) (z (fun (z′) z′)))

(fun (<b>) exprb)

(with ([z x]) (z (fun (z′) z′)))

56

2. -equivalence for m

(with ([z x]) (z (fun (z′) z′)))

=

(with ([z x]) (z (fun (z′) z′)))

57

But what is hygienic expansion?

58

A Theory of Hygienic Macros

m, a model of hygienic macros

1. Macro binding specifications (types)2. Theory of -equivalence3. High-level expansion semantics4. Definition and proof of hygiene

59

3. Hygienic macro expansion

Simple substitution semantics:

let syntax x = m in e end → e[m/x]

if BV(e) ∩ FV(m) =

(m . s) → transcribe(e, match(p, s))

if m = (macro p : => e) and BV(s) ∩ FV(e) = and BV(e) # s

60

3. Expansion up to x.let syntax m = macro (e) : (expr) => (e x) in x.(m x) end

61

3. Expansion up to x.let syntax m = macro (e) : (expr) => (e x) in x′.(m x′) end

→

x.x′.((macro (e) : (expr) => (e x)) x′)

→

x.x′.(x′ x)

62

3. Confluence

(with ([x 1]) (or #f x))

((x.(with ([t #f]) (if t t x))) 1)

((x.(or #f x)) 1)(with ([x 1]) (with ([t #f]) (if t t x)))

63

3. Confluence

Theorem: Let s be a term such that , ⊢ s : . Ifs → s1 and s → s2 then there exists an s′ such that s1 →

* s′ and s2 →*

s′.

* *

s

s1 s2

s′

64

A Theory of Hygienic Macros

m, a model of hygienic macros

1. Macro binding specifications (types)2. Theory of -equivalence3. High-level expansion semantics4. Definition and proof of hygiene

AKA, paydirt

65

4. Hygiene

The expansion relation → is hygienic if for any well-typed expressions e0 and e′0 such that e0 = e′0 and fully-expanded expressions e and e′ such thate0 →

* e and e′0 →

* e′, then e = e′.

e0

e′0

e

e′

*

*

Hygienic = respects !

66

4. Remember confluence?

* *

s up to-equivalence

s1 s2

s′

67

4. Hygiene

Theorem: the relation → is hygienic.

e0

e′0

e′

e

68

Research Plan

69

Additional properties and proofs

Joint confluence with evaluation:

(with ([x ((y.y) 1)]) x)

(with ([x 1]) x)((x.x) ((y.y) 1))

((x.x) 1)

70

Evaluation of classic algorithms

KFFD ’86 Macros that Work (Clinger and Rees ’91) Proof of correctness, if possible

71

Elements of syntax-rules

Core features: Recursive macros Case dispatch

Advanced features: Macro-defining macros

Conveniences: Variable-length sequences Ellipsis patterns (Kohlbecker and Wand ’87)

72

Research plan

Formal model (Completed.) Correctness proofs (Completed.) Elements of syntax-rules

(Expected May 2008.) Evaluation of classic algorithms

(Expected June 2008.) Additional properties and proofs

(Expected August 2008.) Dissertation and defense

(Expected November 2008.)

73

Related Work

74

Related work

Hygienic macro systems Kohlbecker et al ’86: Hygienic macro expansion Kohlbecker and Wand ’87: Macro-by-example Bawden and Rees ’88: Syntactic closures Clinger and Rees ’91: Macros that work Dybvig et al ’93: Syntactic abstraction in Scheme Krishnamurthi ’01: Linguistic reuse

Models of macros Bove and Arbilla ’92: Confluent calculus of expansion Gasbichler ’06: Modules with hygienic macros

75

Related workMacro types Culpepper and Felleisen ’03: Well-shaped macros

Variables and binding Higher-order abstract syntax, de Bruijn, “locally nameless” Gabbay, Pitts: Nominal logic

Staged programming Davies and Pfenning: Modal analysis of staged computation Taha et al: Multi-stage programming, MetaML, MacroML Nanevski: Meta-programming with names and necessity. Kim et al ’06: Polymorphic modal type system for Lisp-like

multi-staged languages

76

Hygienic = respects !

Thank you.

77

Thank you.