A Cluster of Languages for A Cluster of Languages for Mathematical ComputingMathematical Computing

1

Stephen M. Watt

Department of Computer ScienceWestern University London Ontario, Canada

DIKUUniversity of Copenhagen7 September 2012

Moving Windows AroundMoving Windows AroundAdd a borderAdd a scroll barRespond to a button

….

Derp, derp, …

We have harderproblems now.

2

Declaration of PrejudicesDeclaration of PrejudicesKey problem:

How to cascade efficient, effective abstractions.

…

3

Mathematics as a Mathematics as a Programming Language Programming Language CanaryCanary

4

Why?Why?Complex problems with many

partsComplex interactions among the

partsMany different levels of

abstractionPrecise definitionCan tell if an answer is right or

wrong

5

ExamplesExamplesGarbage collection

◦Lisp underground Java etcAlgebraic expressions

◦FortranBig integer

◦CryptoGenerics

◦ Java, C++, ***

6

Computer AlgebraComputer AlgebraSolve problems in terms of symbolic

parameters, rather than numerically.

Having the computer figure out the formulasrather than using formulas given by humans.

Algorithms – computational mathematics

Software – mathematical computation

7

Computer AlgebraComputer Algebra

Start with symbols and compute with symbols =>

Exact resultsHopefully, insightful results

Finding an AnswerFinding an AnswerOne day an individual went to the horse races. Instead of counting the number of humans and horses, she counted 74 heads and 196 legs. How many humans and horses were there?

humans + horses = 74humans × 2 + horses × 4 = 196

Finding an AnswerFinding an AnswerOne day an individual went to the horse races. Instead of counting the number of humans and horses, she counted 74 heads and 196 legs. How many humans and horses were there?

humans + horses = 74humans × 2 + horses × 4 = 196

horses = 24humans = 50

Finding an AnswerFinding an AnswerOne day an individual went to the horse races. Instead of counting the number of humans and horses, she counted H heads and L legs. How many humans and horses were there?

humans + horses = Hhumans × 2 + horses × 4 = L

horses = ‒H + L/2 humans = 2 H ‒ L/2

Computer AlgebraComputer AlgebraA couple of research problems of

personal interest◦Symbolic-numeric algorithms◦Symbolic exponents

12

Approximate PolynomialsApproximate Polynomials

Symbolic ExponentsSymbolic Exponents

ExamplesExamplesMapleAxiomAldorMathMLInkML

Warning: 3x too much stuff here.We will skip to what the audience wants.

15

Language 1: MapleLanguage 1: MapleWaterloo: 1980 onGeddes & Gonnet initiators.University, then company.

Collaboration.

Dynamically typed, interpreted language for scripting computer algebra programs.

16

An Example (small)An Example (small)

25

26

MapleMaple• Compiled kernel, interpreted library• What was compiled was hand-chosen• Support many students on shared 1980s hw

• Easy to lay down code, quick library growth• Language not very structured, so limitations

• Commercially viable project• Company focus education and CAE

27

Example 2: AxiomExample 2: Axiom• 1984 moved from Waterloo to IBM

Research

• Scratchpad in-house research project• Jenks and Trager initiators.• 1991 released as commercial product

by NAG

28

AxiomAxiom• Main idea: code re-use through

abstraction• Generic algorithms based on structures

of modern algebra (groups, rings, algebras, fields).

• The language is the thing• Compiled programming language for

writing libraries “in the large”• Syntactically similar, dynamically typed

interpreted language for scripting.

29

Type Inference in Type Inference in InterpreterInterpreter

30

More Complicated TypesMore Complicated Types

31

32

33

34

35

AxiomAxiom• Great concept for building well-structured

and flexible libraries.

• Not enough “dogfooding.”• Top-level tried to hide types from user,

but was not sufficiently successful at doing that.

• Powerful and flexible, but too complex for most users.

• Now open source.

36

Example 3: AldorExample 3: AldorRe-design of Axiom language 1984

on.Initiator Watt.

The language is the thing, writ largeEfficiency, elegance, take no prisonersNothing special about built-in types

Dependent types everywhereInteroperability with C and Lisp

37

Aldor and Its Type SystemAldor and Its Type SystemTypes and functions are values

◦ May be created dynamically◦ Provide representations of mathematical sets and functions

The type system has two levels◦ Each value belongs to a unique type, its domain, known

statically. ◦ This is an abstract data type that gives the representation.◦ The domains are values with domain Domain.

◦ Each value may belong to any number of subtypes of its domain.

◦ Subtypes of Domain are called categories.

Categories◦ specify what exports (operations, constants) a domain

provides.◦ fill the role of OO interfaces or abstract base classes.

Why Two Levels?Why Two Levels?OO inheritance pb with multi-argument

fns:

class SG { “*”: (SG, SG) -> SG; }DoubleFloat extends SG ...Permutation extends SG ...

x, y ∈ DoubleFloat ⊂ SGp, q ∈ Permutation ⊂ SG

x * y ✓p * q ✓p * y ✓ ??? Bad, Bad, Bad

Why Two Levels?Why Two Levels?OO inheritance pb with multi-argument

fns:

SG == ... { “*”: (%, %) -> %; }DoubleFloat: SG ...Permutation: SG ...

x, y ∈ DoubleFloat ∈ SGp, q ∈ Permutation ∈ SG

x * y ✓p * q ✓p * y ✗

Parametric PolymorphismParametric PolymorphismPP is via category- and domain-producing functions.

-- A function returning an integer.

factorial(n: Integer): Integer == { if n = 0 then 1 else n*factorial(n-1) }

-- Functions returning a category and a domain.

Module(R: Ring): Category == Ring with { *: (R, %) -> % }

Complex(R: Ring): Module(R) with {

complex: (%,%)->R; real: %->R; imag: %->R; conj: % -> %; ...

} == add {

Rep == Record(real: R, imag: R); 0: % == … 1: % == … (x: %) + (y: %): % == ...

}

Dependent TypesDependent TypesGive dynamic typing, e.g.

f: (n: Integer, R: Ring, m: IntegerMod(n)) -> SqMatrix(n, R)

Recover OO through dependent products:

prodl: List Record(S: Semigroup, s: S) == [[DoubleFloat, x],[Permutation, p],[DoubleFloat, y]

]

With categories, guarantee required operations available:

f(R: Ring)(a: R, b: R): R == a*b + b*a

Multi-sorted AlgebrasMulti-sorted AlgebrasCategory signature as a dependent product

type.

ArithmeticModel: Category == with {

Nat: IntegralDomain;

Rat: Field;

/: (Nat, Nat) -> Rat;}

Aldor and Its Type SystemAldor and Its Type SystemType producing expressions may be

conditional

UnivariatePolynomial(R: Ring): Module(R) with {

coeff: (%, Integer) -> R;

monomial: (R, Integer) -> %;

if R has Field then EuclideanDomain;

...

} == add {

...

}

Post facto extensions allow domains to belong to new categories after they have been initially defined.

Without Post Facto Extension Without Post Facto Extension forforStructuring LibrariesStructuring LibrariesDirectProduct(n: Integer, S: Set): Set with {

component: (Integer, %) -> S;

new: Tuple S -> %;

if S has Semigroup then Semigroup;

if S has Monoid then Monoid;

if S has Group then Group;

...

if S has Ring then Join(Ring, Module(S));

if S has Field then Join(Ring, VectorField(S));

...

if S has DifferentialRing then DifferentialRing;

if S has Ordered then Ordered;

...

} == add { ... }

Post Facto Extension forPost Facto Extension forStructuring LibrariesStructuring LibrariesDirectProduct(n: Integer, S: Set): Set with {

component: (Integer, %) -> S;

new: Tuple S -> %;

} == add { ... }

extend DirectProduct(n: Integer, S: Semigroup): Semigroup == ...extend DirectProduct(n: Integer, S: Monoid): Monoid == ...extend DirectProduct(n: Integer, S: Group): Group == ......extend DirectProduct(n: Integer, S: Ring): Join(Ring, Module(S)) == ...extend DirectProduct(n: Integer, S: Field): Join(Ring, VectorField(S)) == ......extend DirectProduct(n: Integer, S: Field): Join(Ring, VectorField(S)) == ...extend DirectProduct(n: Integer, S: DifferentialRing): DifferentialRing == ...extend DirectProduct(n: Integer, S: Ordered): Ordered == ......

Normally these extensions would all be in separate files.

Higher Order OperationsHigher Order OperationsE.g. Reorganizing constructionsPolynomial(x) Matrix(n) Complex R ≈ Complex Matrix(n) Polynomial(x) R

Slightly simpler example

List Array String R ≈ String Array List R

Higher Order OperationsHigher Order OperationsAg ==> (S: BasicType) -> LinearAggregate S;

swap(X:Ag, Y:Ag)(S:BasicType)(x:X Y S):Y X S == [[s for s in y] for y in x];

al: Array List Integer := array(list(i+j-1 for i in 1..3) for j in 1..3);

la: List Array Integer := swap(Array, List)(Integer)(al);

Phew!Phew!

Using GenericityUsing GenericityLinearOrdinaryDifferentialOperator(

A: DifferentialRing,

M: LeftModule(A) with differentiate: % -> %

) : MonogenicLinearOperator(A) with {

D: %;

apply: (%, M) -> M;

...

if A has Field then {

leftDivide: (%, %) -> (quotient: %, remainder: %);

rightDivide:(%, %) -> (quotient: %, remainder: %);

… // rgcd, lgcd

}

} == ...

Using GenericityUsing GenericityLinearOrdinaryDifferentialOperator(

A: DifferentialRing,

M: LeftModule(A) with differentiate: % -> %

) : ...

== SUP(A) add {

...

if A has Field then {

Op == OppositeOperator(%, A);

DOdiv == NonCommutativeOperatorDivision(%, A);

OPdiv == NonCommutativeOperatorDivision(Op,A);

leftDivide (a,b) == leftDivide(a, b)$DOdiv;

rightDivide(a,b) == leftDivide(a, b)$OPdiv;

}

...

}

Design Principles IDesign Principles INo compromises on flexibilityNo compromises on efficiencyUse optimization to bridge the gap.

Compilation. Separate compilation.Generated intermediate code is platform

independent, even though word-sizes, etc, vary.Libraries can be distributed, if desired, as binary

only.

Be a good citizen in a multi-language framework.◦ Call and be called by C/C++/Fortran/Lisp/Maple◦ Functional arguments◦ Cooperating memory management

Design Principles IIDesign Principles IILanguage-defined types should have no privilege

whatsoever over application-defined types.◦ Syntax, semantics (e.g. in type exprs),

optimization (e.g. constant folding)

Language semantics should be independent of type.◦ E.g. named constants overloaded, not functions

Combining libraries should be easy, O(n), not O(n2).◦ Should be able to extend existing things with new

concepts without touching old files or recompiling.

Safety through optimization removing run-time checks, not by leaving off the checks in the first place.

The Compiler as an The Compiler as an ArtefactArtefactWritten primarily in C (C++ too immature in 1990)1550 files, 295 K loc C + 65 K loc Aldor Intermediate code (FOAM):

◦ Primitive types: booleans, bytes, chars, numeric, arrays, closures

◦ Primitive operations: data access, control, data operationsRuntime system:

◦ Memory management◦ Big integers◦ Stack unwinding◦ Export lookup from domains◦ Dynamic linking◦ Written in C and Aldor

Example of OptimizationExample of OptimizationFrom the domain Segment(E: OrderedAbelianMonoid)generator(seg:Segment E):Generator E == generate {

(a, b) := (low seg, hi seg);

while a <= b repeat { yield a; a := a + 1 }

}

From the domain List(S: Set)generator(l: List S): Generator S == generate {

while not null? l repeat { yield first l; l := rest l }

}

Client codeclient() == {

ar := array(...); li := list(...);

s := 0;

for i in 1..#ar for e in l repeat { s := s + ar.i + e }

stdout << s

}

How Generators WorkHow Generators Workgenerator(seg:Segment Int):Generator Int== generate { a := lo seg; b := hi seg; while a <= b repeat { yield a; a := a + 1 }}

client() == { ar := array(...); s := 0; for i in 1..#ar repeat s := s + a.i; stdout << s}

Example of Optimization Example of Optimization (again)(again)From the domain Segment(E: OrderedAbelianMonoid)generator(seg:Segment E):Generator E == generate {

(a, b) := (low seg, hi seg);

while a <= b repeat { yield a; a := a + 1 }

}

From the domain List(S: Set)generator(l: List S): Generator S == generate {

while not null? l repeat { yield first l; l := rest l }

}

Client codeclient() == {

ar := array(...); li := list(...);

s := 0; -- NOTE PARALLEL TRAVERSAL.

for i in 1..#ar for e in l repeat { s := s + ar.i + e }

stdout << s

}

InlinedInlinedB0: ar := array(...); l := list(...); segment := 1..#ar; lab1 := B2; l2 := l; lab2 := B9; s := 0; goto B1;B1: goto @lab1;B2: a := segment.lo; b := segment.hi; goto B3;B3: if a > b then goto B6; else goto B4;B4: lab1 := B5; val1 := a; goto B7;B5: a := a + 1 goto B3;B6: lab1 := B7; goto B7;B7: if lab1 == B7 then goto B16; else goto B8B8: i := val1; goto @lab2;B9: goto B10B10: if null? l2 then goto B13; else goto B11B11: lab2 := B12 val2 := first l2; goto B14;B12: l2 := rest l2 goto B10B13: lab2 := B14 goto B14B14: if lab2 == B14 then goto B16; else goto B15B15: e := val2; s := s + ar.i + e goto B1;B16: stdout << s

Clone Blocks for 1Clone Blocks for 1stst Iterator Iterator

DataflowDataflow

[lab1 == B2, lab1 == B5, lab1 == B7]

Resolution of 1Resolution of 1stst Iterator Iterator

Clone Blocks for 2Clone Blocks for 2ndnd Iterator Iterator

Resolution of 2Resolution of 2ndnd Iterator Iterator

client() == { ar := array(...); l := list(...); l2 := l; s := 0; a := 1; b := #ar; if a > b then goto L2L1: if null? l2 then goto L2 e := first l2; s := s + ar.a + e a := a + 1 if a > b then goto L2 l2 := rest l2 goto L1L2: stdout << s}

Aldor vs C Aldor vs C (non-floating pt)(non-floating pt)

Aldor vs C Aldor vs C (floating point)(floating point)

Follow-on Research Follow-on Research ProjectsProjectsGeneric library inter-operabilityLocalized garbage collectionDynamic abstract data typesPerformance analysis of genericsEtc, etc

66

Lessons LearnedLessons Learned It is possible to be elegant, abstract and high-level

without sacrificing significant efficiency. Well-known optimization techniques can be

effectively adapted to the symbolic setting.Optimization of generated C code is not enough.

Procedural integration, dataflow analysis, subexpression elimination and constant folding are the primary wins.

Compile-time memory optimization, including data structure elimination, is important.◦ Removes boxing/unboxing, closure creation, dynamic

allocation of local objects, etc. Can move hot fields into registers.

Aldor LessonsAldor LessonsLanguage design 20+ years old.

◦ In the mean time, many of the ideas now mainstream.

◦ Many still are not.

Mathematics is a valuable canary in the coal mine of general purpose software.◦ The general world lags in recognizing needs.

It has to be free.◦ Free1 is the standard price. ◦ Free2 is required for engagement.

Example 4: MathMLExample 4: MathMLFirst XML application, ever.Language for exchange of

mathematical data.Initially

70

Example 4: MathMLExample 4: MathML

MathMLMathMLOpenMath effort initiated 1993 for data

exchange.Unfulfilled <math> element in HTML 3.2 Jan

1997.

Initial, unchartered Math WG defining microsyntax for <math>.

Internecine rivalry between syntax and semantics camps coming from TeX, Mathematica and SGML.

MathMLMathMLConvened “HTML-native” math group to

form unified proposal.

First ever XML application.XML proposed recommendation December

1997. MathML proposed recommendation February

1998.

Supported in major browsers, computer algebra systems, incorporated in HTML 5.

Example 5: InkMLExample 5: InkML Ink Messaging

Annotation

Archival

Pen-Based MathPen-Based Math

Input for CAS and document processing.

2D editing.Computer-mediated collaboration.

Pen-Based MathPen-Based MathDoes not require learning a special

language:\sum_{i=0}^r g_{r-i} X^i

sum(g[r-i]*X^i, i = 0..r);

Pen-Based MathPen-Based MathDifferent than natural language

recognition:◦ 2-D layout is a combination of writing and

drawing.◦ No fixed dictionary. ◦ Many similar few-stroke characters. ◦ Well segmented.◦ Highly ambiguous

Digital Ink FormatsDigital Ink Formats

• Collected by surface digitizer or camera

• Sequence of (x,y) points sampled at some known frequency

• Possibly other info (angles, pressure, etc)

• Grouping into traces, letters, words + labelling

InkML ConceptsInkML ConceptsTraces, trace groupsDevice information: sampling rate,

resolution, etc.Pre-defined and application defined

channelsTrace formats, coordinate

transformationsStreaming and archivalAnnotation text and XML

InkML EvolutionInkML EvolutionStarted as low-level language for traces

and hardware description. Explicitly disavowed semantics.

Wanted base language sufficiently rich to support full range of digital ink applications. Semantic grouping added, annotation, etc.

W3C StandardBuilt in to Microsoft Office 2010

81

Various Language ProjectsVarious Language ProjectsReflexAlmaJava/Aldor/C++ interopAbstract ObjectsLocal GCWWW GC

82

Research: Symbol RecognitionResearch: Symbol RecognitionMain idea:

Represent coordinate curves as truncated orthogonal series.

Advantages:◦ Compact – few coefficients needed◦ Geometric

– the truncation order is a property of the character set– gives a natural metric on the space of characters

◦ Algebraic – properties of curves can be computed algebraically (instead of numerically using heuristic parameters)

◦ Device independent – resolution of the device is not important

Choose a functional inner product, e.g.

⟨ f, g = ⟩ ∫ f(t) g(t) w(t) dt

This determines an orthonormal basis in the subspace of polynomials of degree d.Determine using GS on {1, t, t2, t3, ...}.

Can then approximate functions in subspaces

Inner Product and Basis Inner Product and Basis FunctionsFunctions

[a, b]

Like Symbols form CloudsLike Symbols form Clouds

ProblemsProblemsWant fast response –

how to work while trace is being captured.

Low RMS does not mean similar shape.

Pb 1. On-Line InkPb 1. On-Line InkThe main problem:

In handwriting recognition, the human and the computer take turns thinking and sitting idle.

We ask:Can the computer do useful work while the user is writing and thereby get the answer faster after the user stops writing?

We show:The answer is “Yes”!

On-Line Series On-Line Series CoefficientsCoefficientsIf we choose the right basis functions, then

the series coefficients can be computed on line.[Golubitsky+SMW CASCON 2008, ICFHR 2008]

The series coefficients are linear combinations of the moments, which can be computed by numerical integration as the points are received.

This is the Hausdorff moment problem (1921) , shown to be unstable by Talenti (1987).

It is just fine, however, for the orders we need.

Pb 2. Shape vs VariationPb 2. Shape vs VariationThe corners are not in the right places.

Work in a jet space to force coords & derivatives close.

Use a Legendre-Sobolev inner product

1st jet space => set μi = 0 for i > 1.Choose μ1 experimentally to maximize reco rate.Can be also done on-line.

[Golubitsky + SMW 2008, 2009]

Distance Between CurvesDistance Between CurvesApproximate the variation between curves

by some fn of distances between points.May be coordinate curves

or curves in a jet space.

Sequence alignmentInterpolation (“resampling”)

Why not just calculate the area?This is very fast in ortho series representation.

Distance Between CurvesDistance Between Curves

Comparison of Candidate to Comparison of Candidate to ModelsModelsUse Euclidean distance in the coefficient space.

Just as accurate as elastic matching.

Much less expensive.

Linear in d, the degree of the approximation.< 3 d machine instructions (30ns) vs several thousand!

Can trace through SVM-induced cells incrementally.

Normed space for characters gives other advantages.

Distance-Based Distance-Based ClassificationClassification

Distance-Based Distance-Based ClassificationClassification

GeometryGeometry

C = (1‒ t) A + t B

• Can compute distance of a sample to this line• Convex hull of a set of models• SVM separating planes

• Linear homotopies within a class

Distance-Based Distance-Based ClassificationClassification

Distance-Based Distance-Based ClassificationClassification

Error Rates as Fn of Error Rates as Fn of DistanceDistance

SVM Convex Hull

Error rate as fn of distance gives confidence measure for classifiers [MKM – Golubitsky + SMW]

Recognition SummaryRecognition SummaryDatabase of samples => set of LS

pointsCharacter to recognize =>

◦Integrate moments as being written◦Lin. trans. to obtain one point in LS space◦Classify by distance to convex hull of k-NN.

InkML allows natural representation of annotated database and real-time input.

Overall ConclusionsOverall ConclusionsMathematical problems provide

excellent challenges for language design.◦Rich, complex, hard◦Well-defined◦Performance matters – a lot!

Don’t be put off by the loud, confident proclamations of mass-market language designers.

100