.

.

. ..

.

.

Development of SML# –making ML an ordinary practical languaage

Atsushi Ohorijoint work with Katsuhiro Ueno

, Tohoku University

Atsushi Ohori Development of SML# – making ML an ordinary practical language 1 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers wouldbe:

We should be taking about the glorious future of ML or otherfunctional languages ...

ML has already been a super-advanced and highly practicallanguage.

What’s the issue?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers wouldbe:

We should be taking about the glorious future of ML or otherfunctional languages ...

ML has already been a super-advanced and highly practicallanguage.

What’s the issue?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers wouldbe:

We should be taking about the glorious future of ML or otherfunctional languages ...

ML has already been a super-advanced and highly practicallanguage.

What’s the issue?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers wouldbe:

We should be taking about the glorious future of ML or otherfunctional languages ...

ML has already been a super-advanced and highly practicallanguage.

What’s the issue?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

This talk is about SML#

Its motivation:

to make ML an ordinary practical language.

Some immediate reactions from eminent ML/FP researchers wouldbe:

We should be taking about the glorious future of ML or otherfunctional languages ...

ML has already been a super-advanced and highly practicallanguage.

What’s the issue?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 2 / 59

Motivation and backgrounds

Backgrounds

ML is indeed a great language. it is:

Highly Productive

We all know it.

Highly Reliable, andPolymorphic type inference is the only verification tool made intopractice for programmers’ daily use.

SafeIt is type safe and memory safe; “buffer overflow” is not even an issue.

You get all of them for free by just using ML ....

Then, everyone should have been using ML by now,...

Atsushi Ohori Development of SML# – making ML an ordinary practical language 3 / 59

Motivation and backgrounds

Backgrounds

ML is indeed a great language. it is:

Highly Productive

We all know it.

Highly Reliable, andPolymorphic type inference is the only verification tool made intopractice for programmers’ daily use.

SafeIt is type safe and memory safe; “buffer overflow” is not even an issue.

You get all of them for free by just using ML ....

Then, everyone should have been using ML by now,...

Atsushi Ohori Development of SML# – making ML an ordinary practical language 3 / 59

Motivation and backgrounds Weaknesses of ML

Weaknesses of ML

Despite these advantages, ML has not yet been among the popularproduction languages of the industry.

There may be historical or cultural backgrounds/prejudice.

It is a toy invented in academia...

Functional language is difficult/inefficient/impractical...

But, more fundamentally, several important practical problems have beenleft unsolved.

improper treatment of records

lack of interoperability with C

lack of database support

lack of native thread support

lack of separate compilation and linking

lack of tools · · ·Atsushi Ohori Development of SML# – making ML an ordinary practical language 4 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (1)： improper treatment of records

Record is the most basic data constructor, which must be fullysupported.

However, record operations are not quite first-class citizens in ML.In SML:- fn x => map #name x;stdIn:1.1-1.17 Error: unresolved flex record(can’t tell what fields there are besides #name)

This is actually better than some of the other systems.

A minor comment: there are several solutions. I nonetheless address thisissue first, since our solution to this problem turns out to be an importantstep in the development of SML#.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 5 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (2) : lack of interoperability with C

Seamless (or at least smooth) interoperability with C is aprerequisite for a new language to be accepted in practice.

Unfortunately, in current ML, using C libraries requires:to understand runtime representations of ML objects,to write a low-level stub for data conversion, andperhaps to call mysterious coordination-functions for GC

which make using C not only cumbersome but also dangerous.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 6 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (3) : lack of database support

There are few real-world applications that do not use databases.

The current (ordinary) practice is string interface to SQL, where theprogrammer

generates an SQL command string,sends it to an SQL server, andconverts the result to appropriate data structure.

This is cumbersome, error prone and unsafe.

ML is worse: programmers cannot even use string interface to SQL.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 7 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (4) : lack of native thread support

Exploiting emerging multi/many-core CPUs is the key to futurehigh-performance application development.

In the excising typical ML compilers, threads are implemented bytheir runtime system using a timer.

This implies that only one core can be used, no matter how manythreads the program code generates.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 8 / 59

Motivation and backgrounds Weaknesses of ML

Weakness (5) : lack of separate compilation

In a large software project, separate compilation and linking isessential.

Unfortunately, no ML compiler supports separate compilation andlinking in the ordinary sense, i.e. the process:

to compile a source file into an object file consisting ofa set of machine code blocksexternal name definitions of the exported variablesexternal name refereces of the imported variables

and to link an object file withother ML object files independenly compiled, andsystem libraries (libc, libm etc)

There seem to be no theoretical foundation for separately compilingStandard ML (except for typecheking.)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 9 / 59

Motivation and backgrounds Sources of the weakness

Probable sources of these weakness

Widely accepted assumptions/beliefs/folklore around ML:Parametric polymorphism is the principle.Copying collector is the choice.Interfaces are Types

Taking them as axioms, ML attempts to re-construct the world.

This attempt may work well in theory, but creates some seriousproblems in practice.

Let us examine their implication and impact.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 10 / 59

Motivation and backgrounds Sources of the weakness

Parametric Polymorphism Principle

Since ’a in ’a -> ’a ranges over any types, we should only have onerepresentation, namely 32/64 bit word (possibly with an inline tag).

In the ordinary daily computation (on any hardware), however, we have:

Some programmers would surely give up polymorphism for these data.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59

Motivation and backgrounds Sources of the weakness

Parametric Polymorphism Principle

Since ’a in ’a -> ’a ranges over any types, we should only have onerepresentation, namely 32/64 bit word (possibly with an inline tag).

In the ordinary daily computation (on any hardware), however, we have:

Some programmers would surely give up polymorphism for these data.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59

Motivation and backgrounds Sources of the weakness

Parametric Polymorphism Principle

Since ’a in ’a -> ’a ranges over any types, we should only have onerepresentation, namely 32/64 bit word (possibly with an inline tag).

In the ordinary daily computation (on any hardware), however, we have:

Some programmers would surely give up polymorphism for these data.Atsushi Ohori Development of SML# – making ML an ordinary practical language 11 / 59

Motivation and backgrounds Sources of the weakness

Copy GC Folklore

Functional programs require fast allocation (true, indeed), on whichfolklore has it that:

“Cheney’s copying GC is the only option for primary (minor) GC”.

This implies that

there is one allocation pointer for the thread-shared global heap, andobjects are moved at any unpredictable time.

In such an absurd environment, one reasonable strategy for the MLruntime is to act as a monitor for thread scheduling and memory allocation.

However, if we are free from the folklore, then there is a possibility of:

having a non-moving GC with multiple allocation pointers, anddirectly using POSIX threads provided by the OS kernel.

Then you get true efficient multi-core threads for free.This is the way it should be; OS kernels are designed to serve us.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 12 / 59

Motivation and backgrounds Sources of the weakness

Copy GC Folklore

Functional programs require fast allocation (true, indeed), on whichfolklore has it that:

“Cheney’s copying GC is the only option for primary (minor) GC”.

This implies that

there is one allocation pointer for the thread-shared global heap, andobjects are moved at any unpredictable time.

In such an absurd environment, one reasonable strategy for the MLruntime is to act as a monitor for thread scheduling and memory allocation.

However, if we are free from the folklore, then there is a possibility of:

having a non-moving GC with multiple allocation pointers, anddirectly using POSIX threads provided by the OS kernel.

Then you get true efficient multi-core threads for free.This is the way it should be; OS kernels are designed to serve us.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 12 / 59

Motivation and backgrounds Sources of the weakness

Interface as Types Assumption

There are some proposals for (a theory of?) “separate compilation”system for ML, but as far as I can tell they only talk about type-checking,and do not address separate compilation.

In the ordinary understanding, compilation is the process to translate asource program into an object file consisting of

machine code blocks,a set of defined symbols for the set of exported variables, anda set of external symbols (to be resolved by a linker).

Separate compilation is a process to compile (in the above sense) asource without consulting any source programs that they are referenced.The reslulting object files are linked by a system linker (ld) with

other object files separately compiled, andsystem libraries (libc, libm etc).

In this ordinary sense, there is no system that can separately compileStandard ML (other than SML#.)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 13 / 59

Motivation and backgrounds Sources of the weakness

Interface as Types Assumption

There are some proposals for (a theory of?) “separate compilation”system for ML, but as far as I can tell they only talk about type-checking,and do not address separate compilation.

In the ordinary understanding, compilation is the process to translate asource program into an object file consisting of

machine code blocks,a set of defined symbols for the set of exported variables, anda set of external symbols (to be resolved by a linker).

Separate compilation is a process to compile (in the above sense) asource without consulting any source programs that they are referenced.The reslulting object files are linked by a system linker (ld) with

other object files separately compiled, andsystem libraries (libc, libm etc).

In this ordinary sense, there is no system that can separately compileStandard ML (other than SML#.)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 13 / 59

Motivation and backgrounds Sources of the weakness

Interface as Types Assumption II

For example, consider compiling the following into linkable machine code:

functor F (type foodatatype bar = A of foo * int) =

struct datatype hoge = X of int * X.barfun f (a, b) = X (a, X.A (b, a))fun g (X (_, X.A c)) = c

end

It is rather trivial to do the following:...1 separately typecheck sources and keep them (or elaboration of them)...2 when all the programs become available, then do code generation

from the sources (or their elaborated intermediate forms).but I would not consider such process as separate compilation.

Blurring separate compilation with typecheking would curtail theopportunity for making ML an ordinary practical language.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 14 / 59

Motivation and backgrounds The SML# Project

My comments on the weakness of ML

Of course researchers have been aware that

one-bit tag in integers causes problems

heap-allocating floating point numbers is undesirable

threads should be supported by the OS thread library, · · ·On this situation, two comments I would like to make:

...1 They are ingenious mechanisms researchers have developed torealize ML, to which my thorough respects are due.

...2 Overcoming them has been shown to be quite difficult. They are theprices to pay for ML to obtain its reliability and productivity.

Unfortunately, these prices are too high to make ML an ordinary practicallanguage.

As researchers and hackers∗ who love ML, we cannot stop here!(*) A person with an enthusiasm for programming or using computers as an end in itself.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59

Motivation and backgrounds The SML# Project

My comments on the weakness of ML

Of course researchers have been aware that

one-bit tag in integers causes problems

heap-allocating floating point numbers is undesirable

threads should be supported by the OS thread library, · · ·On this situation, two comments I would like to make:

...1 They are ingenious mechanisms researchers have developed torealize ML, to which my thorough respects are due.

...2 Overcoming them has been shown to be quite difficult. They are theprices to pay for ML to obtain its reliability and productivity.

Unfortunately, these prices are too high to make ML an ordinary practicallanguage.

As researchers and hackers∗ who love ML, we cannot stop here!(*) A person with an enthusiasm for programming or using computers as an end in itself.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59

Motivation and backgrounds The SML# Project

My comments on the weakness of ML

Of course researchers have been aware that

one-bit tag in integers causes problems

heap-allocating floating point numbers is undesirable

threads should be supported by the OS thread library, · · ·On this situation, two comments I would like to make:

...1 They are ingenious mechanisms researchers have developed torealize ML, to which my thorough respects are due.

...2 Overcoming them has been shown to be quite difficult. They are theprices to pay for ML to obtain its reliability and productivity.

Unfortunately, these prices are too high to make ML an ordinary practicallanguage.

As researchers and hackers∗ who love ML, we cannot stop here!(*) A person with an enthusiasm for programming or using computers as an end in itself.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 15 / 59

Motivation and backgrounds The SML# Project

We must develop a new ML language that can be a possible choice of theindustry for serious software development.

The SML# Project

Its goal is to develop a new language in the ML family that is:a proper extension of Standard ML:

record polymorphism (and others, e.g rank-1 polymorphism, andlimited form of overloading)complete downward compatibility with Standard ML

with practically important features:seamless interoperability with Cseamless database integrationseparate compilation and linkingdirect and light-weight native thread supporteasy-to-use parallelism (being developed)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 16 / 59

Motivation and backgrounds The SML# Project

SML# team and collaborators

SML# Development Team:

Atsushi Ohori (RIEC, Tohoku University)

Katsuhiro Ueno (RIEC, Tohoku University)

in (past) collaboration with (current affiliation):

Kiyoshi Yamatodani (Sanpu Kobo Inc.)

Nguen Duc Huu (Hanoi University of Science and Technology)

Liu Bochao (CNCERT, China)

Satoshi Osaka (Advantest)

with contributions from many peoples including:

Nobuaki Yoshida, Isao Sasano, Yutaka Matsuno,Kwanghoon Choi

Atsushi Ohori Development of SML# – making ML an ordinary practical language 17 / 59

Quick overview of SML# through examples Record polymorphism

Rest of the Talk

.. .1 Motivation and backgrounds

.. .2 Quick overview of SML# through examples

Record polymorphismInteroperability with CSeamless database integrationSeparate compilation

.. .3 Outline of the Technical Developments

Atsushi Ohori Development of SML# – making ML an ordinary practical language 18 / 59

Quick overview of SML# through examples Record polymorphism

Record polymorphism: basic record operations

fn x => map #X x;val it = fn : [’a#{X:’b},’b.’a list -> ’b list]

# fun getX x = #X x;val getX = fn : [’a#{X:’b},’b.’a -> ’b]

# fun setX x y= x # {X = y};val setX = fn : [’a#{X:’b},’b.’a -> ’b -> ’a]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 19 / 59

Quick overview of SML# through examples Record polymorphism

Record polymorphism: objects

val methods ={getX = fn self => #X (!self),setX = fn S => fn x => S := (!S # {X = x}),getY = fn S => #Y (!S),setY = fn S => fn x => S := (!S # {Y = x}),getColor = fn S => #Color (!S),setColor = fn S => fn x => S := (!S # {Color = x})

};localval state = ref { X = 0.0, Y = 0.0 }

in val p1 = fn method => method methods stateend;p1 # setX 1.0;p1 # getX;

Atsushi Ohori Development of SML# – making ML an ordinary practical language 20 / 59

Quick overview of SML# through examples Record polymorphism

Record polymorphism: polymorphic variants

val P1 = fn M => #CPoint M {x=10.0, y = 10.0};val P2 = fn M => #PPoint M {r=14.1421356, theta = 45.0};val myPointList = [P1, P2];val distance ={CPoint = fn {x,y,...} => Real.Math.sqrt (x *x + y* y),PPoint = fn {r, theta,...} => r};

P1 distance ;P2 distance ;fun iterator L f = map (fn x => x f) L;val myPointLstIter = fn x => iterator myPointList x;myPointLstIter distance;

See (Ohori, 1995) for formal accounts of polymorphic variants.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 21 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (1): the “Hello World”

SML# source program hello.sml:val print = _import "printf" : string -> intval _ = print "こんにちは，日本へようこそ．\n"

Compile, link and execution:% smlsharp hello.sml% ./a.outこんにちは，日本へようこそ．

The above “printf” is the printf in libc library, of course.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 22 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (2): printf

Source program printf.sml:val printReal = _import "printf"

: (string, real) -> intval printTriple = _import "printf"

: (string, real, real, real) -> intval _ = printReal("%f\n", 3.14)val _ = printTriple("if a=%f, b=%f then a/b=%f\n,

3.14, 1.41, 3.14/1.41)

Compile, link and execute:% smlsharp printf.sml% ./a.out3.140000if a=3.140000, b=1.410000 then a/b=2.226950

Atsushi Ohori Development of SML# – making ML an ordinary practical language 23 / 59

Quick overview of SML# through examples Interoperability with C

Some comment on “printf”

There are some proposals for printf in ML. They are elegant,entertaining, and educational. They also roughly works as analternative to printf, but not quite.

val printYen = _import "printf" : (string,int) -> intval _ = printYen("%dYen\n", 1024)val _ = printYen("%10dYen\n", 1024)val _ = printYen("%-10dYen\n", 1024)val _ = printYen("%+10dYen\n", 1024)val _ = printYen("%#10xYen\n", 1024) ...

and many more alternatives.

Remember what you have wanted: it’s printf.Then use printf.

It’s free and fully functional (of course since it is printf).

Atsushi Ohori Development of SML# – making ML an ordinary practical language 24 / 59

Quick overview of SML# through examples Interoperability with C

Some comment on “printf”

There are some proposals for printf in ML. They are elegant,entertaining, and educational. They also roughly works as analternative to printf, but not quite.

val printYen = _import "printf" : (string,int) -> intval _ = printYen("%dYen\n", 1024)val _ = printYen("%10dYen\n", 1024)val _ = printYen("%-10dYen\n", 1024)val _ = printYen("%+10dYen\n", 1024)val _ = printYen("%#10xYen\n", 1024) ...

and many more alternatives.

Remember what you have wanted: it’s printf.Then use printf.

It’s free and fully functional (of course since it is printf).Atsushi Ohori Development of SML# – making ML an ordinary practical language 24 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (3): a small real example

mt.sml using Mersenne Twister:

val printWord = _import "printf" : (string, word) -> intval printReal = _import "printf" : (string, real) -> intval init = _import "init_genrand" : (int) -> unitval genWord = _import "genrand_int32" : () -> wordval genReal = _import "genrand_res53" : () -> realval _ = init 0val _ = printWord ("%d\n", genWord ())val _ = printReal ("%lf\n",genReal ())

% gcc -c mt19937ar.c -o mtlib.o # compile Mersenne Twister% smlsharp mt.sml mtlib.o% a.out23571360440.592845

Atsushi Ohori Development of SML# – making ML an ordinary practical language 25 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions I

(Just for your entertainment)

In SML#:

val f1 = _import "f1": (((((()->unit)->unit)->unit)->unit)->unit)->unit

val g1 = _import "g1" : ()->()->()->()->unitval () = f1 (fn h1 => (print "h1\n";

h1 (fn h2 =>(print "h2\n";h2 (fn h3 => print "h3\n")))))

val g2 = g1 ()val g3 = g2 ()val g4 = g3 ()val () = g4 ()

Atsushi Ohori Development of SML# – making ML an ordinary practical language 26 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions II

In C:

void f4(void){ printf("f4\n");}

void f3(void(*f)(void(*)(void))){printf("f3\n"); f(f4); }

void f2(void(*f)(void(*)(void(*)(void(*)(void))))){printf("f2\n");f(f3); }

void f1(void(*f)(void(*)(void(*)(void(*)(void(*)(void(*)(void)))))))

{printf("f1\n");f(f2);}

Atsushi Ohori Development of SML# – making ML an ordinary practical language 27 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions III

void g4(void){printf("g4\n");}

void (*g3(void))(void){printf("g3\n");return g4;}

void (*(*g2(void))(void))(void){printf("g2\n");return g3;}

void (*(*(*g1(void))(void))(void))(void){printf("g1\n");return g2;}

Atsushi Ohori Development of SML# – making ML an ordinary practical language 28 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (4): higher-order functions IV

Compile, link and execute:

% smlsharp callback.sml callback.o% gcc -c callback.c% smlsharp callback.sml callback.o% ./a.outf1 h1 f2 h2 f3 h3 g1 g2 g3 g4

Atsushi Ohori Development of SML# – making ML an ordinary practical language 29 / 59

Quick overview of SML# through examples Interoperability with C

Interoperability with C (5): polymorphic functions

(For your further entertainment)

val libc = DynamicLink.dlopen libcval c_qsort = DynamicLink.dlsym (libc, "qsort")fun qsort (a, f) =

_ffiapply c_qsort (a : ’a array,Array.length a : int,_sizeof(’a),f : (’a ptr, ’a ptr) -> int) : unit

In the interpreter mode:

# use "qsort.sml";...

val qsort =fn : [’a. ’a array * (’a ptr * ’a ptr -> int) -> unit]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 30 / 59

Quick overview of SML# through examples Seamless database integration

Seamless database integration in SML#

SML# achieves seamless databases integration whereSQL expressions are first-class citizens in the language, andthey are evaluated by a database server.

As an example, consider the following SQL command:

Q = SELECT name, ageFROM peopleWHERE age <= 25

which is a shorthand for the following verbose syntax:

Q = SELECT person.name AS name, person.age AS ageFROM people AS personWHERE (person.age <= 25)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 31 / 59

Quick overview of SML# through examples Seamless database integration

Simple demo: seamless database integration

# val Q = _sql db => select #person.name as name,#person.age as age

from #db.people as personwhere SQL.<= (#person.age, 25);

val q = fn: [’a#{people:’b},’b#{age:int, name:’d},’c,’d::{int, word, char, string, real, ’e option},’e::{int, word, char, bool, string, real}.(’a, ’c) SQL.db -> {age: int, name: ’d} SQL.query]

Of course, you get a lot of type errors when you first write acomplicated query.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 32 / 59

Quick overview of SML# through examples Seamless database integration

Simple demo: seamless database integration

# val Q = _sql db => select #person.name as name,#person.age as age

from #db.people as personwhere SQL.<= (#person.age, 25);

val q = fn: [’a#{people:’b},’b#{age:int, name:’d},’c,’d::{int, word, char, string, real, ’e option},’e::{int, word, char, bool, string, real}.(’a, ’c) SQL.db -> {age: int, name: ’d} SQL.query]

Of course, you get a lot of type errors when you first write acomplicated query.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 32 / 59

Quick overview of SML# through examples Seamless database integration

Simple demo: seamless database integration

# val server = _sqlserver "host=127.0.0.1 dbname=test": {people: {name: string, age: int}};

val server = "host=127.0.0.1 dbname=test": {people: {age: int, name: string}} SQL.server

# val db = SQL.connect server;val db = <conn> : {people:{age: int, name: string}} SQL.conn

# val r = _sqleval q db;val r = <rel> : {age: int, name: string} SQL.rel

# val x = SQL.fetchAll r;val x = [{age = 25, name = "Bob"}]: {age: int, name: string} list

Atsushi Ohori Development of SML# – making ML an ordinary practical language 33 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation in SML#

SML# compiles a sequence of any oftype and datatype bindingsvariable and exception bindingsstructure bindingsfunctor bindings

into an object code in the standard ELF format, and links it withSML# runtime library,C libraries (linbc.a, libm.a, libgmp.a, etc), anduser compiled C functions

by invoking the system linker (ld) to produce an executable file.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 34 / 59

Quick overview of SML# through examples Separate compilation

SML# compiler command

Some typical usages:

smlsharp -c foo.smlcompiles foo.sml to an object file foo.o, consulting with its interfacefile specified implicitly or explicitly.

smlsharp -c foo.smi -o foolocates the object files referenced in foo.smi, invokes the systemlinker to produce an executable file foo.

smlsharp foo.smlcompiles foo.sml to foo.o, links it with all the files referenced in foo.smiand generates a.out.

An interface file foo.smi of foo.sml specifies:

the resources required by foo.sml and,

those foo.sml provides.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 35 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example I

top.sml:

structure B = F(type foo = realdatatype bar = A of foo * int)

val printPair = _import "printPair" : real * int -> unitval _ = printPair (B.g (B.f (2011, 3.14)))

top.smi:

_require "functor.smi", "printPair.smi"

Atsushi Ohori Development of SML# – making ML an ordinary practical language 36 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example II

functor.smi:

functor F(type foo; datatype bar = A of foo * int) =struct datatype hoge = X of int * X.bar

val f : int * X.foo -> hogeval g : hoge -> X.foo * int

end

functor.sml:

functor F (type foodatatype bar = A of foo * int) =

struct datatype hoge = X of int * X.barfun f (a, b) = X (a, X.A (b, a))fun g (X (_, X.A c)) = c

end

Atsushi Ohori Development of SML# – making ML an ordinary practical language 37 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example III

printPair.sml:

val printPair = _import "c_printPair" : (real,int) -> int

printPair.smi

val printPair : real * int -> int

C file

struct PAIR {double a; int b;};void c_printPair(struct PAIR *X) {printf ("(%4.2lf,%4ld)\n", X->a, X->b);}

Atsushi Ohori Development of SML# – making ML an ordinary practical language 38 / 59

Quick overview of SML# through examples Separate compilation

Separate compilation example IV

Compile, link and execute:

% smlsharp -c top.sml% smlsharp -c functor.sml% smlsharp -c printPair.sml% gcc -c printPair.c% smlahrp top.smi printPaur.o% a.out(3.14, 2011)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 39 / 59

Outline of the Technical Developments

Contents of the Talk

.. .1 Motivation and backgrounds

.. .2 Quick overview of SML# through examples

.. .3 Outline of the Technical Developments

Polymorphic Record CalculusNatural data representationSeparate compilationSeamless database integrationEfficient Non-moving GC

Atsushi Ohori Development of SML# – making ML an ordinary practical language 40 / 59

Outline of the Technical Developments

Technical Development of SML#

SML# is realized by combining the following...1 Polymorphic record calculus...2 Natural data representation...3 Separate compilation...4 Polymorphic type system for databases...5 Efficient non-moving GC

Atsushi Ohori Development of SML# – making ML an ordinary practical language 41 / 59

Outline of the Technical Developments Polymorphic Record Calculus

The key technique: polymorphic record compilation

The key insight underlying the development of SML# compiler is

“Polymorphism” is just an abstraction, and not acomputational reality. So it must be compiled out.

The first example is the polymorphic record calculus and itscompilation (Ohori 1992, 1995).

Consider the term:

fn x => #name x

This is apparently polymorphic, but it is also apparent that this mustbe compiled out to efficient load instruction.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 42 / 59

Outline of the Technical Developments Polymorphic Record Calculus

Compiling polymorphic operation using types

First, we reconstruct a typed term:

val f : [’a,’b#{name:’a}. ’a -> ’b]= [’a, ’b#{name:’a}. fn )x:’b) => #name x]

f {name:string, age:int} {name = "smlsharp", age=7}

where ’b#{name:’a} is a type variable with a record kind.We then insert abstraction and application to pass necessaryattributes for compilation.

val f : [’a,’b#{name:’a}. index(’b,name) -> ’b -> ’a]= [’a, ’b#{name:’a}.fn (I:index(’b,name)) => fn (x:’b) => x[I]]

f {name:string, age:int} 2 [| 7,"smlsharp" |]

Atsushi Ohori Development of SML# – making ML an ordinary practical language 43 / 59

Outline of the Technical Developments Polymorphic Record Calculus

Review of record compilation

It is based on the following ideas:...1 Information needed to compile a polymorphic function is

abstracted at type abstraction and applied at type application....2 The needed information is encoded as a type that denotes the

unique needed value. In the example, index(τ, name)denotes the index value of the name field of a record type τ.

This mechanism is far-reaching in compiling out various aspects ofpolymorphism, which we have used to develop SML# throughout.

Note: These concepts have later been variously called such as“dictionary passing”“evidences”“singleton types”

Although literature did not seem to properly compare or even mention, theessence is the same as the one first presented in the 1992 article.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 44 / 59

Outline of the Technical Developments Polymorphic Record Calculus

Review of record compilation

It is based on the following ideas:...1 Information needed to compile a polymorphic function is

abstracted at type abstraction and applied at type application....2 The needed information is encoded as a type that denotes the

unique needed value. In the example, index(τ, name)denotes the index value of the name field of a record type τ.

This mechanism is far-reaching in compiling out various aspects ofpolymorphism, which we have used to develop SML# throughout.

Note: These concepts have later been variously called such as“dictionary passing”“evidences”“singleton types”

Although literature did not seem to properly compare or even mention, theessence is the same as the one first presented in the 1992 article.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 44 / 59

Outline of the Technical Developments Natural data representation

Natural data representation

In SML#, we have:

This is obviously desired. How the pointer-tracing GC works?

Atsushi Ohori Development of SML# – making ML an ordinary practical language 45 / 59

Outline of the Technical Developments Natural data representation

Natural data representation

In SML#, we have:

This is obviously desired. How the pointer-tracing GC works?Atsushi Ohori Development of SML# – making ML an ordinary practical language 45 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: object bitmap

We make the following heap blocks.

-th bit

0 1

non-

pointerpointer

-th word

object pointer

header(includingsize )

words

layout bitmap

bits

Note: This is only for records (tuples); atomic data (except forstrings) are always unboxed.

How such a bit map can be computed for polymorphic functions?

We adopt the idea of record compilation.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: object bitmap

We make the following heap blocks.

-th bit

0 1

non-

pointerpointer

-th word

object pointer

header(includingsize )

words

layout bitmap

bits

Note: This is only for records (tuples); atomic data (except forstrings) are always unboxed.

How such a bit map can be computed for polymorphic functions?

We adopt the idea of record compilation.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: object bitmap

We make the following heap blocks.

-th bit

0 1

non-

pointerpointer

-th word

object pointer

header(includingsize )

words

layout bitmap

bits

Note: This is only for records (tuples); atomic data (except forstrings) are always unboxed.

How such a bit map can be computed for polymorphic functions?

We adopt the idea of record compilation.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 46 / 59

Outline of the Technical Developments Natural data representation

Natural data representation: compilation

We introduce the typestag(τ) to denote one bit indicating whether τ is boxed or notsize(τ) to denote the size of τ

The rest is essentially the same as record compilation, but requirescertain amount of careful development, especially in

closure conversion that creates records which require bitmaps,a call frame that also requires a bitmap,properly aligning record fields.

See (Nguen Duc Huu and Atsushi Ohori, 2006) for the details.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 47 / 59

Outline of the Technical Developments Separate compilation

Separate compilation

I’d repeat:

to make ML an ordinary practical language, it is essential to haveseparate compilation and linking in the ordinary sense, which isthe process

to compile a source file into an object file consisting ofa set of machine code blocks,external name definitions of the exported variables, andexternal name refereces of the imported variables,

and to link an object file withother ML object files independenly compiled, andsystem libraries (libc, libm etc)

As far as I can see, separate compilation in this ordinary sense has neverbeen done before for Standard ML or equivalent one.Indeed, separately compiling Standard ML declarations into an object fileis inherently difficult.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 48 / 59

Outline of the Technical Developments Separate compilation

Issues in separately compiling Standard ML (I)

To compile a program separately from those it references, it must beclosed both in its type and in its representation.

Suppose we compile a source program that references a structure

S :> sigtype fooval x : foo

end

Since foo is unknown, we must specify its interface such as

structure S =structtype foo (= real * int) (* opaque type interface *)val x : foo

end

Atsushi Ohori Development of SML# – making ML an ordinary practical language 49 / 59

Outline of the Technical Developments Separate compilation

Interface language

By this way, we design an interface language whose syntax is roughly:

<itopdec> ::= <idecl> | <ifundecl><idecl> ::= val <valdesc>

| type <itypbind>| eqtype <tyvarSeq> id (= <ty>)| datatype <idatbind>| datatype <id> = datatype <longid>| exception <exbind>| structure <id> = <istrexp>

<itypbind> ::= <tyvarSeq> <id> = <ty>| <tyvarSeq> <id> ( = <ty> )

<idatbind> ::= <tyvarSeq> <id> = <conbinds> [and <idatbind>]| <tyvarSeq> <id> ( = <conbinds> [and<idatbind>])

<istrexp> ::= struct <idecl-seq> end<ifundec> ::= functor <id> ( <id> : <sigexp> ) = <istrexp>

Atsushi Ohori Development of SML# – making ML an ordinary practical language 50 / 59

Outline of the Technical Developments Separate compilation

Separately compiling a functor

Then how to deal with functor, which is by definition open in its typeparameters.

functor F (X:sigtype foodatatype bar = A of foo * int

end) =structdatatype hoge = X of int * X.barfun f (a, b) = X (a, X.A (b, a))fun g (X (_, X.A c)) = c

end

Since foo’s is a parameter, its representation varies.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 51 / 59

Outline of the Technical Developments Separate compilation

Lifting functor arguments

We have solved this problem based roughly on the following idea:

lifting static (type) parameters as type abstractions,

lambda lift dynamic parameters (variables)

A functor becomes a polymorphic function and functor application becomeapplication with type instantiation.

We can then use the idea of record compilation again to pass necessarycompile attributes through type abstraction and type application.

The example code is transformed to the code of the form:

val ’foo F = fn (id : ’foo -> ’foo) =>let fun f (a, b) = ...

fun g v = ...in (f, g) end

Atsushi Ohori Development of SML# – making ML an ordinary practical language 52 / 59

Outline of the Technical Developments Separate compilation

Other issues in separate compilation

There are a number of other issues that had to be worked out, including

flattening nested structures (mostly mechanical),

complete signature checking before type inference

proper treatment of exceptions

statically allocating constants and records in top-level declarations,

generating initialization code, and

generating system linkable object files.

We have solved all the problems and have a separate compile system thatcompile any sequence of declarations.

The details will (hopefully) be published somewhere in near future,we hope.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 53 / 59

Outline of the Technical Developments Separate compilation

Separate compilation demo : lexgen

We have wrote interface file for each file in the basis library.basis.smi

include "basis/main/ArrayStructures.smi"include "basis/main/BinIO.smi"include "basis/main/Bool.smi"....

lexgen benchmark can be compiled by writing the interface file

_require "basis.smi"structure Main = struct val doit : unit -> unit end

without consulting any of the actual source files.The resulting object file can be liked with those of the basis to form anexecutable file by the command line:

% smlsharp lexgen.smi

Atsushi Ohori Development of SML# – making ML an ordinary practical language 54 / 59

Outline of the Technical Developments Seamless database integration

Seamless database integration in SML#

SML# achieves seamless databases integration where

SQL expressions are first-class citizens in the language that can befreely combined with all the other components of SML#, and

they are evaluated by a database server.

using the following results and new technical developments:

database type system a la Machiavelli [Ohori, et.al. 1989, 1996],

record polymorphism,

C FFI,

light-weight first-class overloading (through record compilation again),

limited introduction of existential types, and

a uniform approach to implement a DSL, bridging static and dynamicsemantics.

The details at the talk tomorrow.Atsushi Ohori Development of SML# – making ML an ordinary practical language 55 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time.

Indeed, there are many objects you can’t move....

My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.

Near the C-5 exit ofOtemachi st. About 20min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time.

Indeed, there are many objects you can’t move....

My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.

Near the C-5 exit ofOtemachi st. About 20min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time.

Indeed, there are many objects you can’t move....

My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.

Near the C-5 exit ofOtemachi st. About 20min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

It is absurd to assume that objects are moved at any unpredictable time.

Indeed, there are many objects you can’t move....

My suggestion on ahistorical interest nearthe ICFP 2011 venue:Masakado-zuka, theresting place of Tairano Masakado’s head.

Near the C-5 exit ofOtemachi st. About 20min. walk from here.

Photography by Katsuhiro Ueno

Atsushi Ohori Development of SML# – making ML an ordinary practical language 56 / 59

Outline of the Technical Developments Efficient Non-moving GC

Efficient non-moving GC

If we are sure that objects once created will not be moved (as in anordinary decent world), then interoperability with C and native threadsupport become lot easier.

We have successfully developed a non-moving GC with generationalextension that is as efficient as the legendary Cheney’s generationalcopying collector.

The details at the talk this afternoon by Ueno san.

Atsushi Ohori Development of SML# – making ML an ordinary practical language 57 / 59

Outline of the Technical Developments Efficient Non-moving GC

Conclusions

SML# is available fromSML# 0.62 the interactive compilerSML# 0.90 the separate compiler (released on 2011-09-14)

We hope to release SML# 1.00 in near future by integrating SML#0.62 into SML# 0.90.

Near Future Plan64 bit support (easy)concurrent GCoptimization (type propagation)

Atsushi Ohori Development of SML# – making ML an ordinary practical language 58 / 59

Outline of the Technical Developments Efficient Non-moving GC

Thank You

Atsushi Ohori Development of SML# – making ML an ordinary practical language 59 / 59