The Difficulty of Improving Upon the Standard ML Module...

45
The Difficulty of Improving Upon the Standard ML Module System Derek Dreyer Max Planck Institute for Software Systems (MPI-SWS) Kaiserslautern and Saarbr¨ ucken, Germany MacQueen Fest Chicago, USA May 13, 2012

Transcript of The Difficulty of Improving Upon the Standard ML Module...

Page 1: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The Difficulty of Improving Uponthe Standard ML Module System

Derek Dreyer

Max Planck Institute for Software Systems (MPI-SWS)Kaiserslautern and Saarbrucken, Germany

MacQueen FestChicago, USAMay 13, 2012

Page 2: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Balancing expressive power and conceptual simplicity

Dave MacQueen, “Reflections on Standard ML” (1993):

“The pragmatic perspective tends to lead designers to addfeatures to the language to enhance its perceived expressive poweror convenience, and hence there is a danger of excessive complexityand featurism, especially if the interaction of different features isnot carefully considered. One of the hardest tasks in languagedesign is judging what should be left out of the language, becauseevery feature has its constituency. Another difficult and criticalproblem is to judge the expressiveness/complexity tradeoff for agiven feature. This is especially tricky because the use of aparticular feature in programming may involve a high mentaloverhead for programmers even though the feature’s semanticdescription is relatively straightforward (e.g., first-classcontinuations).”

Page 3: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The Standard ML module system hits a sweet spot!

The modularity mechanisms of dependent typeswithout the semantic/type-theoretic complexity

Phase separation yields decidable typechecking

Intuitive treatment of linking constraintsSharing by specification instead of parameterization

Generative semantics of type abstractionType equivalence is simple to understand

Crucial for encapsulating effects properly!

. . . (among many other things). . .

Page 4: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Dave MacQueen has made my life very difficult. . .

Research ≈ trying to improve the state of the artI chose to try to improve on Standard ML Modules

Oy vey iz mir!Very hard to improve on SML Modules without makingmajor changes to the design/semantics of the language

But I keep on truckin’. . .. . . while thinking of Dave as the skeptical angel/devil onmy shoulder, asking: “Is your newfangled language designreally any better than SML Modules?”

Thanks, Dave!

(no sarcasm intended)

Page 5: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Dave MacQueen has made my life very difficult. . .

Research ≈ trying to improve the state of the artI chose to try to improve on Standard ML Modules

Oy vey iz mir!Very hard to improve on SML Modules without makingmajor changes to the design/semantics of the language

But I keep on truckin’. . .. . . while thinking of Dave as the skeptical angel/devil onmy shoulder, asking: “Is your newfangled language designreally any better than SML Modules?”

Thanks, Dave!

(no sarcasm intended)

Page 6: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Dave MacQueen has made my life very difficult. . .

Research ≈ trying to improve the state of the artI chose to try to improve on Standard ML Modules

Oy vey iz mir!Very hard to improve on SML Modules without makingmajor changes to the design/semantics of the language

But I keep on truckin’. . .. . . while thinking of Dave as the skeptical angel/devil onmy shoulder, asking: “Is your newfangled language designreally any better than SML Modules?”

Thanks, Dave!

(no sarcasm intended)

Page 7: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Dave MacQueen has made my life very difficult. . .

Research ≈ trying to improve the state of the artI chose to try to improve on Standard ML Modules

Oy vey iz mir!Very hard to improve on SML Modules without makingmajor changes to the design/semantics of the language

But I keep on truckin’. . .. . . while thinking of Dave as the skeptical angel/devil onmy shoulder, asking: “Is your newfangled language designreally any better than SML Modules?”

Thanks, Dave!

(no sarcasm intended)

Page 8: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Two directions for “improvement”

1 Recursive modules

2 Substructural types

Page 9: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Two directions for “improvement”

1 Recursive modules

2 Substructural types

Page 10: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The “double vision” problem [Crary et al. ’99, Dreyer ’05]

In recursive modules, there may be multiple “paths”(i.e., multiple ways of referring to) the same type

Want all such paths to be considered equal

But this is hard to achieve, leading one to seetwo paths to the same type as distinct

Page 11: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Example of the “double vision” problem

module rec A : sig

type t

val f : B.u -> A.t

end = ...

and B : sig

type u

val g : A.t -> B.u

end = ...

Page 12: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Example of the “double vision” problem

module rec A : sig

type t

val f : B.u -> A.t

end = struct

type t = int

fun f x = ... (B.g 3) ...

end

and B : sig

type u

val g : A.t -> B.u

end = ...

Page 13: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Approaches to handling double vision “directly”

Force recursive modules to be transparent

[Crary et al. ’99, Russo ’01]

Too restrictive

Disallow more than one path to an abstract type

Units [Flatt-Felleisen ’98, Owens-Flatt ’06]

Can’t really encode ML module system properly

Handle special cases

OCaml [Leroy ’03], Scala [Odersky-Zenger ’05]

Don’t handle itTraviata [Nakata-Garrigue ’06]

Page 14: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Elaboration semantics

Interpret module language into a more explicit IL:

Classify ML modules via (annotated) IL types Σ

Erasure Σ◦ maps Σ to IL type

Soundness of elaboration:

Γ `ML mod : Σ ; M =⇒ Γ `IL M : Σ◦

Γ `ML sig ; Σ =⇒ Γ `IL Σ◦ type

Different elaborations of ML modules possible:

[Harper-Stone ’00]: IL = “translucent sums”

[Rossberg et al. ’10]: IL = Fω (“F-ing modules”)

Page 15: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Elaboration can be harmful to your health

From my thesis, an unmitigated disaster:

9.3. ELABORATION 227

! ! modexp ! mod :! sig

1. ! ! sigexp ! sig rec : Sig

2. !,modid!mvar rec:sig rec !stat modexp ! metasigactual

3. mvar crec "# FV(Stat(Priv(metasig actual))) $ FV(Stat(Pub(metasig actual)))

4. !,mvar actual:Stat(Pub(metasig actual)) !sub

mvar actual : Stat(Pub(metasig actual)) % Stat(sig rec) ! : tstatsig coerced

5. metasig !actual := metasigactual[mvar c

rec.visible"/mvar crec]

6. metasig static := "(mvar rec).&(mvar actual: metasig !actual ).tstatsig coerced

7. ! !can Priv(metasig static) ! mod static

8. tsig rec := s(mvar cstatic.visible" : sig rec)

9. " := !;mvar static:metasig static;modid!mvar rec:maybe(tsig rec)10. " !rec unroll(mvar static).hidden ' modexp ! mod actual : tsigactual

11. ",mvar actual:tsigactual !sub mvar actual : tsigactual % tsig rec ! pmod coerced :12. mod := let mvar static=mod static in

(rec(mvar rec:tsig rec.letmvar actual=mod actual in (pmod coerced : tsig rec)): sig rec)

! ! rec (modid : sigexp) modexp ! mod :P sig rec

(9.152)

Rule 9.152: Let us consider the premises one at a time. (I suggest the reader compare the formalsteps here with the high-level description given in Section 5.4, as they correspond quite closely.)

1. Translate the declared signature sigexp to sig rec, which need not be transparent.

2. Perform static elaboration of modexp, resulting in the meta-signature metasig actual.

3. Enforce the “dynamic-on-static” restriction on modexp by checking that the recursive modulevariable mvar rec does not appear in the static part of metasig actual. References to mvar rec

may still occur in metasig actual’s value specs (i.e., its datatype specs).

4. We need to close up references to mvar rec in metasigactual by enclosing it in an rds (cf. Fig-ure 5.17). Unfortunately, we cannot just write "(mvar rec).metasigactual, because (by step 2)metasigactual expects mvar c

rec to have kind Fst(sig rec), not Fst(metasigactual). So first wecoerce metasigactual into the shape of Stat(sig rec), which produces tstatsig coerced.

5–6. Using tstatsig coerced, we can close up metasig actual with an rds and call it metasig static (inSection 5.4, this was called CSS). For now, ignore the box around metasig !

actual. The purposeof the box will be explained below when the !rec judgment is defined.

7. Construct the canonical module mod static matching metasig static. Note: this will only succeedif Fst(Priv(metasig static)) is an expandable kind. From a programming perspective, this meansthat if in modexp there appears a total functor expression whose body contains datatype

definitions, then the argument signature of that functor must not contain any transparenttype specifications. I admit this is a rather bizarre restriction, but I see no way around it.

8. mod static will eventually be bound to mvar static (in Section 5.4, this was called Static). Wecan thus selfify the declared signature sig rec with respect to mvar static.visible", in order toobtain a transparent version of the declared signature tsig rec.

plus 37 more rules just for recursive modules! §

Page 16: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The essential problem

Good elaboration should not be too “lossy”Should “preserve meaning” of ML program

This is true of Harper-Stone and F-ing modules

But translucent sums / System F do not accountfor recursively defined ADTs

E.g. my thesis elaboration threw away all abstractioninside a recursive module

So a more expressive IL is needed

Page 17: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

RTG [Dreyer ’05]

A type system for “Recursive Type Generativity”System F + forward decl’s of abstract types

Distinguish creation of type name from definition

new α in e (α bound)

def α := τ in (e : τ ′) (α free)

“Ability to define α” treated as a linear resourceTracked using a simple effect system,and in later work, using a linear type-and-kind system

Page 18: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Curing double vision in RTG

The new’s and def’s can be inferred automatically as part ofan elaboration translation for recursive modules [Dreyer ’07].

new α, β in

module rec A : sig

type t

val f : B.u -> A.t

end = struct

type t = int

fun f x = ... (B.g 3) ...

end

and B : sig

type u

val g : A.t -> B.u

end = ...

Page 19: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Curing double vision in RTG

The new’s and def’s can be inferred automatically as part ofan elaboration translation for recursive modules [Dreyer ’07].

new α, β in

module rec A : sig

type t = αval f : B.u -> A.t

end = def α := int in struct

type t = int

fun f x = ... (B.g 3) ...

end

and B : sig

type u = βval g : A.t -> B.u

end = def β := ??? in ...

Page 20: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Greatly improved account of recursive modules

Well-formed modules: !;" ! mod : # with ! "We omit “with ! "” if ! = # (i.e., ifmod does not define any abstract types).

X :# $ "!;" ! X."s : #."s

(14)!;" ! con ! A : K

!; " ! [con ] : [[= A :K]](15)

! ! # : T !;" ! exp : #

!;" ! [exp] : [[# ]](16)

K = Tn %T !;" ! con ! A : K $ = $1, . . . , $n ! &K $ !

!; " ! [" ' con : K] : [[" : [[= ! : K]], in : [[([$]. A($) ) !($)]], out : [[([$]. !($) ) A($)]]]] with ! "(17)

!;" ! [] : [[]](18)

!;" ! mod1 : #1 with !1 " !@ !1 ";", X1 :#1 ! [" % X= mod ] : [[" : #]] with !2 "!; " ! ["1 %X1 =mod1, " % X=mod ] : [["1 :#1, " : #]] with !1, !2 "

(19)

!;" ! mod1 : #1 with !1 " ! @ !1 ";", X :#1 ! mod2 : #2 with !2 "!;" ! let X= mod1 in mod2 : #2 with !1, !2 " (20)

!;" ! sig ! *(!1 "K1).(L1;#1) !, !1 "K1, !2 &K2;", X : #1 ! mod : #2 with !2 "!; " ! &(X : sig).mod : ((!1 " K1).(L1; #1)% *(!2 "K2).(#; #2)

(21)

!;" ! P1 : ((!1 " K1).(L1; #1)% *(!2 "K2).(L2;#2) !;" ! P2 : #

! &K2 + ! ! lookup L1 in # ! ' FV(') , &(!) = # ! # - '#1

!;" ! P1(P2) : '{!2 .% !}#2 with ! " (22)

!; " ! sig ! *(! "K).(L;#) !, ! "K;", X :# !stat mod : #stat with $ "! lookup L in #stat ! ' !;", X : '# ! mod : #! with $ " ! #! - '#

!;" ! rec (X : sig)mod : #! with $ "(23)

!;" ! sig ! *(!0 "K0).(L0;#0) ! = !!, ! & K0 (L;#) = {!0 .%!}(L0;#0)

!, $ &L;" !stat mod : #stat with $ " ! lookup L in #stat ! ' FV(') , &(!) = #!!, $ & L; '" ! mod : #! with $ " ! #! - '#

!;" ! mod :> sig : # with ! " (24)

!;" ! sig ! *(! "K).(L;#) !;" ! mod : #! with $ " ! lookup L in #! ! ' ! #! - '#

!;" ! mod : sig : '# with $ "(25)

Statically well-formed modules: !; " !stat mod : # with ! "The rules defining this static judgment are precisely the same as the rules defining the regular module typing judgment (above),except with the shaded premises removed, and all occurrences of the regular module typing judgment replaced by this static judgment.

Signature matching: ! #1 - #2

! # - #(26)

#! = [[" :#]]

! #! - [[]](27)

! #!."1 - #1 ! #! - [[" :#]]

! #! - [["1 :#1, " :#]](28)

! lookup L!1 in #1 ! '1 ! #1 - '1#

!1 ! lookup L2 in '1#

!2 ! '2 ! '1#

!2 - '2#2

! ((!!1 " K!

1).(L!1;#

!1)% *(!!

2 "K!2).(L!

2;#!2) - ((!1 "K1).(L1;#1) % *(!2 "K2).(L2;#2)

(29)

Abstract type lookup: ! lookup L in # ! '

L = {!1 : K1 .% "s1, . . . , !n : Kn .% "sn} '0 = #(i $ 1..n : #."si = [[= Ai : Ki]] FV(Ai) , dom(L) + {!1, . . . , !i"1} 'i = 'i"1 / {!i .% 'i"1Ai}

! lookup L in # ! 'n(30)

Figure 6. Typing Rules for Modules

38-fold reduction in the number of typing rulesneeded to support recursive modules!

But a problem remains: Separate compilation

Page 21: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Greatly improved account of recursive modules

Well-formed modules: !;" ! mod : # with ! "We omit “with ! "” if ! = # (i.e., ifmod does not define any abstract types).

X :# $ "!;" ! X."s : #."s

(14)!;" ! con ! A : K

!; " ! [con ] : [[= A :K]](15)

! ! # : T !;" ! exp : #

!;" ! [exp] : [[# ]](16)

K = Tn %T !;" ! con ! A : K $ = $1, . . . , $n ! &K $ !

!; " ! [" ' con : K] : [[" : [[= ! : K]], in : [[([$]. A($) ) !($)]], out : [[([$]. !($) ) A($)]]]] with ! "(17)

!;" ! [] : [[]](18)

!;" ! mod1 : #1 with !1 " !@ !1 ";", X1 :#1 ! [" % X= mod ] : [[" : #]] with !2 "!; " ! ["1 %X1 =mod1, " % X=mod ] : [["1 :#1, " : #]] with !1, !2 "

(19)

!;" ! mod1 : #1 with !1 " ! @ !1 ";", X :#1 ! mod2 : #2 with !2 "!;" ! let X= mod1 in mod2 : #2 with !1, !2 " (20)

!;" ! sig ! *(!1 "K1).(L1;#1) !, !1 "K1, !2 &K2;", X : #1 ! mod : #2 with !2 "!; " ! &(X : sig).mod : ((!1 " K1).(L1; #1)% *(!2 "K2).(#; #2)

(21)

!;" ! P1 : ((!1 " K1).(L1; #1)% *(!2 "K2).(L2;#2) !;" ! P2 : #

! &K2 + ! ! lookup L1 in # ! ' FV(') , &(!) = # ! # - '#1

!;" ! P1(P2) : '{!2 .% !}#2 with ! " (22)

!; " ! sig ! *(! "K).(L;#) !, ! "K;", X :# !stat mod : #stat with $ "! lookup L in #stat ! ' !;", X : '# ! mod : #! with $ " ! #! - '#

!;" ! rec (X : sig)mod : #! with $ "(23)

!;" ! sig ! *(!0 "K0).(L0;#0) ! = !!, ! & K0 (L;#) = {!0 .%!}(L0;#0)

!, $ &L;" !stat mod : #stat with $ " ! lookup L in #stat ! ' FV(') , &(!) = #!!, $ & L; '" ! mod : #! with $ " ! #! - '#

!;" ! mod :> sig : # with ! " (24)

!;" ! sig ! *(! "K).(L;#) !;" ! mod : #! with $ " ! lookup L in #! ! ' ! #! - '#

!;" ! mod : sig : '# with $ "(25)

Statically well-formed modules: !; " !stat mod : # with ! "The rules defining this static judgment are precisely the same as the rules defining the regular module typing judgment (above),except with the shaded premises removed, and all occurrences of the regular module typing judgment replaced by this static judgment.

Signature matching: ! #1 - #2

! # - #(26)

#! = [[" :#]]

! #! - [[]](27)

! #!."1 - #1 ! #! - [[" :#]]

! #! - [["1 :#1, " :#]](28)

! lookup L!1 in #1 ! '1 ! #1 - '1#

!1 ! lookup L2 in '1#

!2 ! '2 ! '1#

!2 - '2#2

! ((!!1 " K!

1).(L!1;#

!1)% *(!!

2 "K!2).(L!

2;#!2) - ((!1 "K1).(L1;#1) % *(!2 "K2).(L2;#2)

(29)

Abstract type lookup: ! lookup L in # ! '

L = {!1 : K1 .% "s1, . . . , !n : Kn .% "sn} '0 = #(i $ 1..n : #."si = [[= Ai : Ki]] FV(Ai) , dom(L) + {!1, . . . , !i"1} 'i = 'i"1 / {!i .% 'i"1Ai}

! lookup L in # ! 'n(30)

Figure 6. Typing Rules for Modules

38-fold reduction in the number of typing rulesneeded to support recursive modules!

But a problem remains: Separate compilation

Page 22: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The trouble with separate compilation

ML’s separate compilation mechanism isthe functor.

functor Sep A (X : SIG B) :> SIG A = ...

Problem: SIG B depends on type components of A,which are not in scope.

Not obvious how to generalize functors to work inthe recursive case.

Page 23: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The trouble with separate compilation

ML’s separate compilation mechanism isthe functor.

functor Sep A (X : SIG B) :> SIG A = ...

Problem: SIG B depends on type components of A,which are not in scope.

Not obvious how to generalize functors to work inthe recursive case.

Page 24: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Symptom of a larger problem. . .

Structure formation (struct)

Signature formation (sig)

Signature inheritance (include)

Transparent type specifications (type t = typ)

Opaque type specifications (type t)

Value specifications (val v : typ)

Signature refinement (where type / with type)

Sharing constraints (sharing type)

Signature bindings (signature)

Functor abstraction (functor)

Functor application ( ( ) )

Transparent signature ascription ( : )

Opaque signature ascription ( :> )

Local definitions (let / local)

Recursive structures (struct rec)

Recursively dependent signatures (sig rec)

Question

Why are modules composed using parameterizationwhile signatures are composed using fibration?

Page 25: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Symptom of a larger problem. . .

Structure formation (struct)

Signature formation (sig)

Signature inheritance (include)

Transparent type specifications (type t = typ)

Opaque type specifications (type t)

Value specifications (val v : typ)

Signature refinement (where type / with type)

Sharing constraints (sharing type)

Signature bindings (signature)

Functor abstraction (functor)

Functor application ( ( ) )

Transparent signature ascription ( : )

Opaque signature ascription ( :> )

Local definitions (let / local)

Recursive structures (struct rec)

Recursively dependent signatures (sig rec)

Question

Why are modules composed using parameterizationwhile signatures are composed using fibration?

Page 26: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

MixML (Dreyer-Rossberg ’08)

Key idea: Compose everything by fibration!

MixML modules synthesize ML’s structure andsignature languages into one.

Consequences:

ML structures and signatures are endpoints on aspectrum of MixML modules.

Signatures and structures (and mixtures of both)are composed using the exact same constructs.

Both structures and signaturesare just a particular mode of use of modules!

Page 27: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

MixML (Dreyer-Rossberg ’08)

Key idea: Compose everything by fibration!

MixML modules synthesize ML’s structure andsignature languages into one.

Consequences:

ML structures and signatures are endpoints on aspectrum of MixML modules.

Signatures and structures (and mixtures of both)are composed using the exact same constructs.

Both structures and signaturesare just a particular mode of use of modules!

Page 28: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

The MixML module language

mod ::= X (variable)| {} (empty)| [exp] | [: typ] (term)| [typ] | [: kind ] (type)| {` = mod} | mod .` (namespaces)| (X = mod1) with mod2 (linking)| (X = mod1) seals mod2 (sealing)| [mod ] | new mod (units)

Page 29: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Some useful derived forms

Structure formation (struct)

Signature formation (sig)

Signature inheritance (include)

Transparent type specifications (type t = typ)

Opaque type specifications (type t)

Value specifications (val v : typ)

Signature refinement (where type / with type)

Sharing constraints (sharing type)

Signature bindings (signature)

Functor abstraction (functor)

Functor application ( ( ) )

Transparent signature ascription ( : )

Opaque signature ascription ( :> )

Local definitions (let / local)

Recursive structures (struct rec)

Recursively dependent signatures (sig rec)

Page 30: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

A qualified success

MixML could provide a clean core for anext-generation ML-style module system

Syntactically minimal and useful for understandingvanilla SML modules (even ignoring recursion)

But semantics is still quite complex:

Mixin’ Up the ML Module System A:23

Modules: Γ; R;β � mod : Σ

X : |Σ| ∈ Γ

Γ; {||}; ∅ � X : |Σ| (VAR)Γ; {||}; ∅ � {} : {||} (EMP)

� A ⇑ knd

Γ; [[= A]]; ∅ � [:knd] : [[= A]](ITYP)

Γ � typ ❀ A

Γ; {||}; ∅ � [typ] : [[= A]](ETYP)

Γ � typ ❀ A � A ⇑ type

Γ; {||}; ∅ � [:typ] : [[A]]−(IVAL)

Γ � exp : A � A ⇑ type

Γ; {||}; ∅ � [exp] : [[A]]+(EVAL)

Γ; R;β � mod : Σ

Γ; {|� : R|};β � {�=mod} : {|� :Σ|}(STR)

Γ; {|� : R|};β � mod : {|� :Σ, �� : |Σ�||}Γ; R;β � mod .� : Σ

(DOT)

� L1 locates α1 R1 # Σ2 Γ; R � R1 � L1;β1 � mod1 : Σ1

� L2 locates α2 R2 # Σ1 Γ, X : |Σ1|; R � R2 � L2;β2 �stat mod2 : Σ�2

� (L1;Σ1) � (L2;Σ�2) ❀ δ Γ, X : |δΣ1|; R � R2 � δL2;β2 � mod2 : Σ2

α1,α2 fresh � δΣ1 + Σ2 ⇒ Σ

Γ; R � R1 � R2;β1,β2 � (X =mod1) with mod2 : Σ(LINK)

� L1 locates α1 Γ; L1;β1 � mod1 : Σ1

� L2 locates α2 Γ, X : |Σ1|; L2;β2 �stat mod2 : Σ�2

� (L1;Σ1) � (L2;Σ�2) ❀ δ δΓ, X : |δΣ1|; δL2;β2 � mod2 : Σ2

β2,α2 fresh � δΣ1 + Σ2 ⇒ |Σ|Γ; {||};β1,α1 � (X =mod1) seals mod2 : |Σ1|

(SEAL)

Γ � mod : ΦΓ; {||}; ∅ � [mod] : [[Φ]]+

(EUN)

Γ � mod : [[∀α. ∃β. (L;Σ)]]+ dom(δ) = {α,β}Γ; δL; δβ � new mod : δΣ

(NEW)

Complete Modules: Γ � mod : Σ

Γ; {||};β � mod : |Σ| β fresh β �∈ fv(Σ)

Γ � mod : |Σ| (COMPL)

Units: Γ � mod : Φ

Γ; L;β � mod : Σ � L locates α α,β fresh

Γ � mod : ∀α. ∃β. (L;Σ)(UNIT)

Fig. 5. Typing Rules for MixML

ACM Transactions on Programming Languages and Systems, Vol. V, No. N, Article A, Publication date: January YYYY.

Actively working with my student Scott Kilpatrick onsimplifying it further

Page 31: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

A qualified success

MixML could provide a clean core for anext-generation ML-style module system

Syntactically minimal and useful for understandingvanilla SML modules (even ignoring recursion)

But semantics is still quite complex:

Mixin’ Up the ML Module System A:23

Modules: Γ; R;β � mod : Σ

X : |Σ| ∈ Γ

Γ; {||}; ∅ � X : |Σ| (VAR)Γ; {||}; ∅ � {} : {||} (EMP)

� A ⇑ knd

Γ; [[= A]]; ∅ � [:knd] : [[= A]](ITYP)

Γ � typ ❀ A

Γ; {||}; ∅ � [typ] : [[= A]](ETYP)

Γ � typ ❀ A � A ⇑ type

Γ; {||}; ∅ � [:typ] : [[A]]−(IVAL)

Γ � exp : A � A ⇑ type

Γ; {||}; ∅ � [exp] : [[A]]+(EVAL)

Γ; R;β � mod : Σ

Γ; {|� : R|};β � {�=mod} : {|� :Σ|}(STR)

Γ; {|� : R|};β � mod : {|� :Σ, �� : |Σ�||}Γ; R;β � mod .� : Σ

(DOT)

� L1 locates α1 R1 # Σ2 Γ; R � R1 � L1;β1 � mod1 : Σ1

� L2 locates α2 R2 # Σ1 Γ, X : |Σ1|; R � R2 � L2;β2 �stat mod2 : Σ�2

� (L1;Σ1) � (L2;Σ�2) ❀ δ Γ, X : |δΣ1|; R � R2 � δL2;β2 � mod2 : Σ2

α1,α2 fresh � δΣ1 + Σ2 ⇒ Σ

Γ; R � R1 � R2;β1,β2 � (X =mod1) with mod2 : Σ(LINK)

� L1 locates α1 Γ; L1;β1 � mod1 : Σ1

� L2 locates α2 Γ, X : |Σ1|; L2;β2 �stat mod2 : Σ�2

� (L1;Σ1) � (L2;Σ�2) ❀ δ δΓ, X : |δΣ1|; δL2;β2 � mod2 : Σ2

β2,α2 fresh � δΣ1 + Σ2 ⇒ |Σ|Γ; {||};β1,α1 � (X =mod1) seals mod2 : |Σ1|

(SEAL)

Γ � mod : ΦΓ; {||}; ∅ � [mod] : [[Φ]]+

(EUN)

Γ � mod : [[∀α. ∃β. (L;Σ)]]+ dom(δ) = {α,β}Γ; δL; δβ � new mod : δΣ

(NEW)

Complete Modules: Γ � mod : Σ

Γ; {||};β � mod : |Σ| β fresh β �∈ fv(Σ)

Γ � mod : |Σ| (COMPL)

Units: Γ � mod : Φ

Γ; L;β � mod : Σ � L locates α α,β fresh

Γ � mod : ∀α. ∃β. (L;Σ)(UNIT)

Fig. 5. Typing Rules for MixML

ACM Transactions on Programming Languages and Systems, Vol. V, No. N, Article A, Publication date: January YYYY.

Actively working with my student Scott Kilpatrick onsimplifying it further

Page 32: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Two directions for “improvement”

1 Recursive modules

2 Substructural types

Page 33: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Major theme in types research

Enriching type systems with information aboutpurity and effects

Want types (e.g., τ1 → τ2) to “mean more”Monads useful for isolating effectful computations

Want to use types for compositional verification ofhigher-order imperative programs

Substructural type systems have been developed to animpressive extent, e.g., Hoare Type Theory (HTT)

Page 34: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Richly expressive types are not always a clear win

Rich types can be too informativeHard to hide local uses of state (e.g., memoization)that are not externally observable

As Bob Harper likes to say, “Monads seem like a greatidea on Monday, Wednesday and Friday. . . ”

“Poor” old SML types are great at hiding stateGenerative semantics plays a crucial role here

Page 35: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Using SML generativity to hide local state

signature SYMBOL = sigtype tval eq : t * t -> boolval insert : string -> tval lookup : t -> string

endfunctor Symbol () :> SYMBOL = structtype t = intval size = ref 0val table = ref nilfun eq (x,y) = x = yfun insert str = (

size := !size + 1;table := str :: !table;!size

)fun lookup n =

List.nth (!table, !size - n)end

Figure 1. Generativity Example

given string to the table and returns a fresh symbol mapped to it;and a function lookup, which looks up a given symbol in the tableand returns the corresponding string.The functor Symbol implements the symbol type t as an integer

index into a (mutable) list of strings. When applied, Symbol createsa fresh table (represented as a pointer to an empty list) and amutable counter size (representing the size of the table). Theimplementations of the various functions are straightforward, andthe body of the functor is sealed with the signature SYMBOL, thushiding access to the local state (table and size).The call to List.nth in the lookup function might in general

raise a Subscript exception if the input n were an arbitrary inte-ger. However, we “know” that this cannot happen because lookupis exported with argument type t, and the only values of type t thata client could possibly have gotten hold of are the values returnedby insert, i.e., integers that are between 1 and the current size oftable. Therefore, the implementation of the lookup function neednot bother handling the Subscript exception.This kind of reasoning is commonplace in modules that encap-

sulate local state. But what justifies it? Intuitively, the answer istype generativity. Each instantiation of the Symbol functor createsa fresh symbol type t, which represents the type of symbols that arevalid in its own table (but not any other). Were Symbol not gen-erative, each application of the Symbol functor would produce amodule with distinct local state but the same symbol type. It wouldthen be easy to induce a Subscript error by accidentally passinga value of one table’s symbol type to another’s lookup function.1While this intuition about the importance of generativity is very

appealing, it is also completely informal. The goal of this paper isto develop a formal framework for reasoning about the interactionof generative type abstraction and mutable state.In the case of an example like the Symbol functor, we will

be able to show that the implementation of Symbol shown inFigure 1 is contextually equivalent to one whose lookup functionis replaced by:

fun lookup n =if n > 0 andalso n <= !size

andalso !size = length(!table)then List.nth (!table, !size - n)else "Hell freezes over"

1 This is the case, for example, in OCaml, which only supports applicative(i.e., non-generative) functors [15].

In other words, there is no observable difference between the orig-inal Symbol functor and one that dynamically checks the variousinvariants we claim to “know.” Hence, the checks are unnecessary.This kind of result can be understood as an instance of represen-

tation independence, albeit a somewhat degenerate one in that theADTs in question share the same type representation. As with mostsuch results, the proof hinges on the construction of an appropriaterelational interpretation of the abstract type t, which serves to im-pose an invariant on the possible values of type t. In this case, wewish to assert that for a particular structure S defined by Symbol(),the only values of type S.t are integers between 1 and the currentsize of S’s table. This will allow us to prove that any range checkon the argument to S’s lookup function is superfluous.The problem is that the relational interpretation we wish to

assign to S.t depends on the current values stored in S’s localstate. In effect, as S’s insert function is called repeatedly overtime, its table grows larger, and the relational interpretation ofS.t must grow accordingly to include more and more integers.Thus, what we need is an account of state-dependent representationindependence, in which the relational interpretations of abstracttypes are permitted to grow over time, in a manner that is tightlycoupled with changes to some local state.

1.2 OverviewIn this paper, we present a novel method for proving state-dependentrepresentation independence results. Our method extends previ-ous work by Ahmed on syntactic step-indexed logical relationsfor recursive and quantified types [1]. We extend her techniquewith support for reasoning about local state, and demonstrate itseffectiveness on a variety of small but representative examples.Although our primary focus is on proving representation indepen-dence for ADTs that exhibit an interaction of existentials and state,our method also handles several well-known simply-typed exam-ples from the literature on local state (such as Pitts and Stark’s“awkward” example [21]) that have proven difficult for previouslogical-relations-based methods to handle.In order to reason about local state, we build into our logical

relation a notion of possible worlds. While several aspects of ourpossible worlds are derived from and inspired by prior work, otheraspects are quite novel:1. We enrich our possible worlds with populations and laws,which allow us to evolve the relational interpretation of anabstract type over time in a controlled, state-dependent fashion.For instance, we can use a population to grow a set of val-ues (e.g., the integers between 1 and some n), together with alaw that explains what the current population implies about thecurrent machine state (e.g., that the symbol table has size n).

2. Second, our method provides the ability to reason locally aboutreferences to higher-order values. While ours is not the firstmethod to handle higher-order state, our approach is novel andarguably simpler than previous accounts. It depends criticallyon step-indexing in order to avoid a circularity in the construc-tion of possible worlds.The remainder of the paper is structured as follows. In Sec-

tion 2, we present Fµ!, our language under consideration, whichis essentially System F extended with general recursive types andgeneral ML-style references. In Section 3, we explain at a high levelhow our method works and what is novel about it. In Section 4, wepresent the details of our logical relation and prove it sound (but notcomplete) with respect to contextual equivalence of Fµ! programs.In Section 5, we show how to use our method to prove a num-ber of interesting contextual equivalences. Finally, in Section 6, weconclude with a thorough comparison to related work, as well asdirections for future work.

2

Page 36: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Key observation

SML generativity (which can be modeled using ∃’s)enables hiding of monotonic state

State that undergoes “irreversible” changes

Across a series of papers (POPL’09, ICFP’10, POPL’12),Ahmed, Birkedal, Hur, Neis, Rossberg, Vafeiadis and Ihave explored the semantic underpinnings of this

But what about hiding of non-monotonic state?What does that even mean?

We need a story about hiding in a substructural setting!

Page 37: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Key observation

SML generativity (which can be modeled using ∃’s)enables hiding of monotonic state

State that undergoes “irreversible” changes

Across a series of papers (POPL’09, ICFP’10, POPL’12),Ahmed, Birkedal, Hur, Neis, Rossberg, Vafeiadis and Ihave explored the semantic underpinnings of this

But what about hiding of non-monotonic state?What does that even mean?

We need a story about hiding in a substructural setting!

Page 38: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Example of non-monotonic state

Specification of realistic memory allocator:Internal invariant A depends on set of allocated locations

∃A : LocSet→ Type.

init cap : A(∅)⊗ malloc : !∀L : LocSet. A(L) (

∃X : Loc. ptr X ⊗ cap X 1⊗ A(L ] {X})⊗ free : !∀L : LocSet. ∀X : Loc.

ptr X ⊗ cap X 1⊗ A(L ] {X}) ( A(L)

Problem: Interface pollution for clientsA client must thread the “capability” A(L) through itsinterface to guard against interference from other clients

Page 39: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Superficially substructural types(Submitted – Joint work with Krishnaswami, Turon, Garg)

We propose a new sharing rule:Enables A(L) to be split into “fictionally disjoint” pieces,so clients can be oblivious to one another’s existence

split : ∀L1, L2 : LocSet. A(L1 ] L2) ( A(L1)⊗ A(L2)join : ∀L1, L2 : LocSet. A(L1)⊗ A(L2) ( A(L1 ] L2)

This can be done for any commutative monoid!Each ADT can pick whatever monoid it wants,so long as all its operations are frame-preserving

Builds on Birkedal et al.’s work on separation logic andDreyer et al.’s Kripke logical relations for SML

ML-style “hidden state” supported as a mode of use

Page 40: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

But several problems remain. . .

Dealing with unsafe reentrancyCurrently, we use locks to forbid reentrancy,but this will not scale well to concurrency

Possible idea: synthesize with Pottier’s anti-frame rule

Scaling to full-blown dependent typesWe have shown soundness of the sharing rule in thecontext of a hybrid of Dependent ML and L3

But we want to scale to an HTT-like language

Doing strictly better than SML Modules is hard!

Page 41: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Fond memories of Chicago. . .

Page 42: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major
Page 43: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major
Page 44: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Thanks, Dave. . .

. . . for being such a tough act to follow!

Page 45: The Difficulty of Improving Upon the Standard ML Module Systemmacqueenfest.cs.uchicago.edu/slides/dreyer.pdf · Oy vey iz mir! Very hard to improve on SML Modules without making major

Thanks, Dave. . .

. . . for being such a tough act to follow!