Linear Typing for GoA loose adaption of Capabilities for Sharing

Masterarbeit

Julian Andres Klode

Philipps-Universität MarburgFachbereich Mathematik und Informatik

Betreuer: Prof. Dr. Christoph Bockisch

20.12.2017

Go is a programming language aimed at writing highly-concurrent software. In con-current programs, exchanging data between concurrent calculations (goroutines) isessential.

Go embraces a culture of “Don’t communicate by sharing memory, share memory bycommunicating.” (Rob Pike), that is, sending pointers to memory between goroutines.It does not, however, provide a way to validate that there is no other reference to thesame memory location, which could result in two goroutines trying to write to thesame location in parallel, for example.

This thesis tries to approach the problem by introducing annotations for linearity intoGo programs, effectively allowing a programmer to state that a given memory locationcan only be referenced by exactly one reference, which can then be moved betweengoroutines as needed.

Auf Deutsch:

Go ist eine Programmiersprache, welche auf das Schreiben von nebenläufiger Softwareabzielt. In nebenläufigen Programmen ist Datenaustausch zwischen nebenläufigenBerechnungen (goroutine in Go) essenziell.

Eines der Go Sprichwörter ist „Don’t communicate by sharing memory, share memoryby communicating.“ (Rob Pike), was so viel heißt wie: Pointer werden zwischen denBerechnungen verschickt. Es gibt aber keine Möglichkeit auszuschließen, dass es keineandere Referenz auf den gleichen Speicherbereich gibt, was dazu führen kann das zweigoroutinen z.B. gleichzeitig auf den gleichen Speicherbereich schreiben.

Diese Arbeit versucht das Problem anzugehen durch die Einführung von Annotationenfür Linearität in Go. Dies ermöglicht es einen Programmierer anzugeben, das einbestimmter Speicherbereich nur genau eine Referenz haben kann. Diese Referenzenkönnen verschoben werden, sodass die Kommunikation zwischen Goroutinen ermöglichtwird.

Contents

1 Introduction 4

2 Basics 62.1 The Go programming language . . . . . . . . . . . . . . . . . . . . . . 62.2 Approaches to dealing with mutability . . . . . . . . . . . . . . . . . . 122.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3 Permissions for Go 163.1 Structure of permissions . . . . . . . . . . . . . . . . . . . . . . . . . . 163.2 Assignment operations . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.3 Conversions to base permissions . . . . . . . . . . . . . . . . . . . . . . 243.4 Other conversions and merges . . . . . . . . . . . . . . . . . . . . . . . 313.5 Creating a new permission from a type . . . . . . . . . . . . . . . . . . 343.6 Handling cyclic permissions . . . . . . . . . . . . . . . . . . . . . . . . 343.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4 Static analysis of Go programs 374.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.2 The store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.3 Expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.4 Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

5 Evaluation 705.1 Completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705.2 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.3 Preciseness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 715.4 Usability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725.5 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 725.6 Code coverage / Unit testing . . . . . . . . . . . . . . . . . . . . . . . 73

6 Conclusion 81

References 83

3

1 Introduction

Go is a strongly statically typed programming language created by Google. It placesa heavy emphasis on goroutines, a form of concurrent processes, ligthweight threads,that can commmunicate with each other via channels - a concurrent-safe queue.

Being statically typed, Go can prevent certain type mismatches at compile time; forexample, a integer cannot be passed where a string is expected. It is also stronglytyped, that is there are no implicit conversions between incompatible types - a weaklytyped language could allow you to pass an integer where a string is expected, andimplicitly convert the integer to a string.

Go’s support for static typing is very rudimentary: There are few primitive base types,structures, pointers, and interfaces - a collection of methods, essentially. Importantly,there is no support for read-only objects - all objects are writable.

Placing a heavy emphasis on concurrency and communication between goroutines,one of Go’s proverbs is “Don’t communicate by sharing memory, share memory bycommunicating.” (Rob Pike), meaning that instead of having a pointer that is visibleto all goroutines, send the pointer via a channel. For example, given a channel of intpointers, we can send an int pointer on it:

aChannelOfIntPointers <- anIntPointer

But anIntPointer and its target are always writable. By sending the value to thechannel, two goroutines can now have access to it, and there could be race conditionsif we are not careful. It would be nice to be able to send values over channels withouthaving to fear race conditions like that.

A first useful step would be to introduce read-only permissions:// Requires: anIntPointer points to read-only memoryaChannelOfIntPointers <- anIntPointer

But how can we ensure that there is no other pointer pointing to the same locationas anIntPointer, but writable? Restricting read-only pointers to be created only fromread-only locations seems like a bad idea. Surely I want to be able to build a struct(for example) in a mutable fashion, but then return it as a read-only value.

We could use monads, a concept introduced by Haskell, to encapsulate mutability of aspecific value. But no, monads require generic types for implementation, and Go doesnot provide them, so we have to look for something different.

4

So this seems to generally boil down to aliasing: If we can ensure that the value weare sending over the channel does not have any (writeable) aliases, we can implementa solution that moves the value through the channel instead of copying it. We needsomething that says this:

// Requires: anIntPointer must not have any aliases// Ensures: anIntPointer cannot be used afterwardsaChannelOfIntPointers <- anIntPointer

Linear types allow doing just that: We can declare that an object of a given type onlyhas a single reference to it. We can easily check linearity at runtime if we use referencecounting: When a linear value is expected, but the reference count is larger than 1,we throw a runtime error. But this seems like a bad choice: Go is statically typed,and if we can check linearity statically, we can prevent a lot of race conditions fromeven compiling, reducing the amount of bugs in a running program, even in the face ofuntested code paths.

The remaining chapters of the thesis are structured as follows: Chapter 2 gives amore in-depth introduction to Go, monads, linear types, and some generalisations likecapabilities and permissions; chapter 3 introduces an approach to permissions for Go;chapter 4 introduces an abstract interpreter for statically checking permissions; andchapters 5 and 6 discuss how the implementation was tested, and what the issues are.

What properties should our implementation optimally achieve?

An important one is completeness: All syntactic constructs are tested. Or rather: Anunknown syntactic construct should be rejected.

Another is correctness: Only valid programs are allowed. That means that if somethingis wrong, the program should not be considered valid.

There is also preciseness: We do not reject useful programs because our rules are toobroad. As a non-Go example, a literal value of “1” should be able to be stored in allinteger sizes, not just whatever type a literal has.

A further property is usability: Error messages should be understandable. If program-mers do not understand the error, it makes it hard to fix it.

We should also consider compatibility - A compiling Lingo program behaves the sameas a Go program with all Lingo annotations are removed. By storing all annotationsin comments, and just passing the program to the Go compiler after performing ourchecks, we can easily ensure that.

Finally, the implementation should be well tested. As a useful metric we can usecode coverage. Optimally, it should be 100%, but that might not be entirely realistic- sometimes unreachable code needs to be written to future proof code, for example,checking that a method returns true that currently always returns true but mighteventually return false.

5

2 Basics

This chapter gives a quick overview of the Go programming language in order to makethe later abstract interpretation of it reasonably understandable, before discussingapproaches to solving the problem of aliases of mutable objects, and especially lineartypes and permissions, which the remainder of the thesis is based on.

2.1 The Go programming language

Go1 is an imperative programming language for concurrent programming created atand mainly developed by Google, initially mostly by Robert Griesemer, Rob Pike, andKen Thompson. Design of the language started in 2007, and an initial version wasreleased in 2009; with the first stable version, 1.0 released in 2012 [7].

Go has a C-like syntax (without a preprocessor), garbage collection, and, like itspredecessors devloped at Bell Labs – Newsqueak (Rob Pike), Alef (Phil Winterbottom),and Inferno (Pike, Ritchie, et al.) – provides built-in support for concurrency usingso-called goroutines and channels, a form of co-routines, based on the idea of Hoare’s‘Communicating Sequential Processes’ [9].

Go programs are organised in packages. A package is essentially a directory containingGo files. All files in a package share the same namespace, and there are two visibilitiesfor symbols in a package: Symbols starting with an upper case character are visible toother packages, others are private to the package:func PublicFunction() {

fmt.Println("Hello␣world")}

func privateFunction() {fmt.Println("Hello␣package")

}

1https://golang.org - The Go gopher was designed by Renee French.(http://reneefrench.blogspot.com/) The design is licensed under the Creative Commons3.0 Attributions license. Read this article for more details: https://blog.golang.org/gopher

6

Types

Go has a fairly simple type system: There is no subtyping (but there are conversions),no generics, no polymorphic functions, and there are only a few basic categories oftypes:

1. base types: int, int64, int8, uint, float32, float64, etc.

2. struct

3. interface - a set of methods

4. map[K, V] - a map from a key type to a value type

5. [number]Type - an array of some element type

6. []Type - a slice (pointer to array with length and capability) of some type

7. chan Type - a thread-safe queue

8. pointer *T to some other type

9. functions

10. named type - aliases for other types that may have associated methods:type T struct { foo int }type T *Ttype T OtherNamedType

Named types are mostly distinct from their underlying types, so you cannotassign them to each other, but some operators like + do work on objects of namedtypes with an underlying numerical type (so you could add two T in the exampleabove).

Maps, slices, and channels are reference-like types - they essentially are structs contain-ing pointers. Other types are passed by value (copied), including arrays (which have afixed length and are copied).

Conversions

Conversions are the similar to casts in C and other languages. They are written likethis:

TypeName(value)

Constants

Go has “untyped” literals and constants.

7

1 // untyped integer literalconst foo = 1 // untyped integer constantconst foo int = 1 // int constant

Untyped values are classified into the following categories: UntypedBool, UntypedInt,UntypedRune, UntypedFloat, UntypedComplex, UntypedString, and UntypedNil (Go callsthem basic kinds, other basic kinds are available for the concrete types like uint8). Anuntyped value can be assigned to a named type derived from a base type; for example:type someType int

const untyped = 2 // UntypedIntconst bar someType = untyped // OK: untyped can be assigned to someTypeconst typed int = 2 // intconst bar2 someType = typed // error: int cannot be assigned to someType

UntypedNil is critical because it needs special handling later, for example, in section 3.1on page 18, section 3.2 on page 24, and section 3.4 on page 32.

Interfaces and ‘objects’

As mentioned before, interfaces are a set of methods. Go is not an object-orientedlanguage per se, but it has some support for associating methods with named types:When declaring a function, a receiver can be provided - a receiver is an additionalfunction argument that is passed before the function and involved in the functionlookup, like this:type SomeType struct { ... }

func (s *SomeType) MyMethod() {}

func main() {var s SomeTypes.MyMethod()

}

An object implements an interface if it implements all methods; for example, thefollowing interface MyMethoder is implemented by *SomeType (note the pointer), andvalues of *SomeType can thus be used as values of MyMethoder. The most basic interfaceis interface{}, that is an interface with an empty method set - any object satisfiesthat interface.type MyMethoder interface {

MyMethod()}

8

There are some restrictions on valid receiver types; for example, while a named typecould be a pointer (for example, type MyIntPointer *int), such a type is not a validreceiver type.

Control flow

Go provides three primary statements for control flow: if, switch, and for. Thestatements are fairly similar to their equivalent in other C-like languages, with someexceptions:

• There are no parentheses around conditions, so it is if a == b {}, not if (a ==b) {}. The braces are mandatory.

• All of them can have initialisers, like this

if result, err := someFunction(); err == nil { // use result }

• The switch statement can use arbitrary expressions in cases

• The switch statement can switch over nothing (equals switching over true)

• Cases do not fall through by default (no break needed), use fallthrough at theend of a block to fall through.

• The for loop can loop over ranges: for key, val := range map { do something}

Goroutines

The keyword go spawns a new goroutine, a concurrently executed function. It can beused with any function call, even a function literal:func main() {

...go func() {

...}()

go some_function(some_argument)}

Channels

Goroutines are often combined with channels to provide an extended form of Commu-nicating Sequential Processes [9]. A channel is a concurrent-safe queue, and can bebuffered or unbuffered:

9

var unbuffered = make(chan int) // sending blocks until value has been readvar buffered = make(chan int, 5) // may have up to 5 unread values queued

The <- operator is used to communicate with a single channel.valueReadFromChannel := <- channelotherChannel <- valueToSend

The select statement allows communication with multiple channels:select {

case incoming := <- inboundChannel:// A new message for me

case outgoingChannel <- outgoing:// Could send a message, yay!

}

The defer statement

Go provides a defer statement that allows a function call to be scheduled for executionwhen the function exits. It can be used for resource clean-up, for example:func myFunc(someFile io.ReadCloser) {

defer someFile.close()/* Do stuff with file */

}

It is of course possible to use function literals as the function to call, and any variablescan be used as usual when writing the call.

Error handling

Go does not provide exceptions or structured error handling. Instead, it handles errorsby returning them in a second or later return value:func Read(p []byte) (n int, err error)

// Built-in type:type error interface {

Error() string}

Errors have to be checked in the code, or can be assigned to _:

10

n0, _ := Read(Buffer) // ignore errorn, err := Read(buffer)if err != nil {

return err}

There are two functions to quickly unwind and recover the call stack, though: panic()and recover(). When panic() is called, the call stack is unwound, and any deferredfunctions are run as usual. When a deferred function invokes recover(), the unwindingstops, and the value given to panic() is returned. If we are unwinding normally andnot due to a panic, recover() simply returns nil. In the example below, a function isdeferred and any error value that is given to panic() will be recovered and stored inan error return value. Libraries sometimes use that approach to make highly recursivecode like parsers more readable, while still maintaining the usual error return value forpublic functions.func Function() (err error) {

defer func() {s := recover()switch s := s.(type) { // type switch

case error:err = s // s has type error now

default:panic(s)

}}

}

Arrays and slices

As mentioned before, an array is a value type and a slice is a pointer into an array,created either by slicing an existing array or by using make() to create a slice, whichwill create an anonymous array to hold the elements.slice1 := make([]int, 2, 5) // 5 elements allocated, 2 initialized to 0slice2 := array[:] // sliced entire arrayslice3 := array[1:] // slice of array without first element

There are some more possible combinations for the slicing operator than mentionedabove, but this should give a good first impression.

A slice can be used as a dynamically growing array, using the append() function.slice = append(slice, value1, value2)slice = append(slice, arrayOrSlice...)

11

Slices are also used internally to represent variable parameters in variable lengthfunctions.

Maps

Maps are simple key-value stores and support indexing and assigning. They are notthread-safe.someValue := someMap[someKey]someValue, ok := someMap[someKey] // ok is false if key not in someMapsomeMap[someKey] = someValue

2.2 Approaches to dealing with mutability

Purely functional programming is a form of programming in which side effects donot exist. That is, there is no such things as mutable data structures, or even I/Ooperations; only pure transformations from one data structure to another.

We will take a look at two approaches to dealing with mutability: Monads [11], madepopular by Haskell, and Linear Types, which are becoming increasingly more popularin mainstream programming with the Rust programming language.

Linear Types are usually defined per language, and the different languages might haveslightly different semantics for the same name or different names for the same semantics.We will consider two generalisations of linear types: ‘Capabilities for Sharing’ [5] andfractional permissions [4]. The former attemps to be a framework for describing typelinearity very generically, whereas the latter is a more limited (or focused) approach onread-only vs writable values.

2.2.1 Monads and Linear Types

There is a way to express mutability or side effects in functional programming: Haskelland some other languages use a construct called monads [11]. A monad is a datastructure that essentially represents a computation. For example, an array monadcould have operations ‘modifying’ an array that compute a new monad that when‘executed’ produces a final array - it is essentially a form of embedded language. Forexample, in Haskell, a monad is defined like this:class Monad m where

return :: a -> m a(>>=) :: m a -> (a -> m b) -> m b

12

return returns a value in the monad, and >>= takes a value in the monad, and passes itto a function which expects a raw value and returns a new value in the monad, yieldingthe new value in the monad.

Monads solve the problem of referential transparency: A function with the same inputwill produce the same output (a function operating on a monad produces a new monaddescribing any computations to be made, and is thus pure). They have one majordrawback however: They are not easily combinable: As soon as you have more thanone monad, you need to ‘lift’ monad operations into other monads in order to use themtogether. This makes it hard to read code.

An alternative approach to representing mutability are linear types. A value of a lineartype can only be used once. In fact, traditionally a linear value must be used exactlyonce. So an array can be implemented as a linear type with operations consuming onelinear value and returning another. The compiler can then optimize the operation touse the same array, because it knows that nobody else is accessing the ‘old’ array.

Consuming and returning linear values is a bit annoying, which is why some pro-gramming languages started introducing shortcuts for it: A parameter with a specialannotation serves as both an input and an output parameter. For example, in Mercury,an efficient purely declarative logic programming language [14][6, section 2.2], anoperation could look like this::- module hello.:- interface.

:- import_module io.:- pred main(io::di, io::uo) is det.:- implementation.main(!IO) :-

write_string("Hello,␣world!\n", !IO).

The exclamation mark here is equivalent to an input and output parameter, so it isthe same as:main(IO0, IO) :-

write_string("Hello,␣world!\n", IO0, IO).

With such a notation, we immediately reach a level where the code basically just lookslike imperative code but with all the same guarantees of purely functional code. Wecan also make this the only notation, effectively gaining an imperative language withthe same guarantees as a functional one.

There are various names describing the same or fairly similar concepts as this: linear[2], unique [1][3], free [10][13], or unsharable [12].

One programming language using linear types is Rust2. In Rust, the input/output2https://www.rust-lang.org

13

annotation is basically the only variant - it looks and works like an imperative language.Linear values can be created and ‘borrowed’ for passing them to another function, forexample. Rust has no garbage collector, but a system of lifetimes where each functionparameter can be associated a named lifetime and the result can then refer to thenames of the parameters. This allows it to be used even without a heap, at least intheory. Rust does not use linear types for I/O which is a bit unfortunate.

2.2.2 Capabilities for Sharing

The several implementations of linear types in different programming languages areall slightly incompatible with each other, which is why ‘Capabilities for Sharing’ [5]introduces a common system for describing linearity.

It describes a simple reference based language with objects containing fields. A capabilityis a pair of an address and a permission - a set of the following flags:

• R - read• W - write• I - identity - the address can be compared• R - exclusive read - no other capability can read that address• W - exclusive write - no other capability can write that address• I - exclusive identity - no other capability has identity access to the object• O - ownership

Exclusive read and write do not imply their non-exclusive counter parts; for example,an object RW prevents others from writing, but cannot write itself - it is essentially aread-only object. Permissions also must be asserted: Other capabilities can have theirconflicting access rights stripped at run-time (or it can be ensured statically that thereare no conflicting access rights). Asserting the exclusive permissions of an unownedcapability strips away incompatible permissions from other unowned capabilities, butasserting on an owned capability strips away all incompatible permissions on all othercapabilities - so for example, if there are two capabilities A and B for the locationx, both with RW , asserting the permissions one one will strip away the permissionfrom the other. If not asserted, exclusive permissions mean nothing: There could bemultiple capabilities with exclusive reads for the same object in the program.

The O flag determines ownership. If an capability has the O bit it is owned, otherwiseit is commonly described as borrowed. In a lot of languages, borrowed objects cannotbe stored, but are just temporary aliases to the initial object.

They provide small-step semantics of a tiny language which operates on a store whichmaps addresses and (object, field) pairs to capabilities. One thing follows from thatapproach:

Given four objects a, b, x, y, with a, b each having a field x referencing x, and x havinga field referencing y, the capability for a.x.y and b.x.y have to be the same:

14

• Evaluating a.x.y:1. (A,X)→ (X,permissions for X in A)2. (X,Y )→ (Y, permissions for Y in X)

• Evaluating b.x.y:1. (B,X)→ (X,permissions for X in B)2. (X,Y )→ (Y, permissions for Y in X)

This means that while a and b can have different views of x, they must have the sameone for y. It is unclear if this was intended to keep things simple, or if it was notconsidered that it might be useful to have different permissions for y.

Permissions might also be overly flexible: Should we really care about exclusive identity,or values that have no permission at all? There are 7 flags with two values each, so weend up with 27 = 128 possible permissions.

2.2.3 Fractional permissions

Another approach to linear values is fractional permissions [4] and fractional permissionswithout fractions [8]. Fractional permissions are of course, only permissions, they needto be associated with values somehow, compared to capabilities which also abstractaway the object. In the fractional permission world, an object starts out with apermission of 1, and each time it is borrowed, the permissions are split. A permissionof 1 can write, other permissions can only read.

Fractional permissions have one advantage over the permission approach outline inthe previous section: They can be recombined. They are also less flexible, offeringonly linear writeable and non-linear read-only kinds of values, rather than 27 possiblecombinations, which might be an advantage or not, depending on what the requirementsare.

It seems possible to extend fractional permissions with some non-linear writeableobject: Introduce infinity as a valid value, and define fractions of infinity as infinity;and defining a writeable object as having a permission ≥ 1, rather than equal to 1. Thisway, there could be an infinite number of references to a writeable object. Likewise, alinear read-only value could perhaps be introduced by introducing a special fractionthat cannot be divided into smaller fractions.

2.3 Summary

We learned how Go looks and works, and what kinds of control structures it has. Wealso learned about Monads and linear types, and how to generalize linear types tocapabilities (and their permissions) and fractional permissions. In the next chapter, wewill discuss which of these approaches works for Go (hint: permissions), and adjustthem to work with Go.

15

3 Permissions for Go

In the previous chapter, we saw monads, linear types, and the two generalisationsof linear types as capabilities and fractional permissions. This chapter introducespermissions for Go based on the concepts from ‘Capabilities for Sharing’ [5], andcertain operations that will be useful to build a static analyser that checks permissionson a Go program:

1. Operations for checking whether certain types of assignments are allowed2. Operations for ensuring consistency and allowing to specify incomplete annota-

tions for variables.3. Operations to merge permissions from different branches of the program4. An operation to create a permission for a given type

In the github.com/julian-klode/lingolang reference implementation, the permissionsand operations are provided in the permission package.

The reasons for going with a capabilities-derived approach are simple: Monads donot work in Go, as Go does not have generic types; and fractional permissions areless powerful, and we also need to deal with legacy code and perhaps could use someother permissions for describing Go-specific operations, like a permission for allowing afunction to be executed as a goroutine.

3.1 Structure of permissions

This approach to linear types in Go is called Lingo (short for linear Go). Permissionsin Lingo are different from the original approach in ‘Capabilities for Sharing’ in a fewpoints. First of all, because Go has no notion of identity, since it uses pointers, theright for that is dropped. We end up with 5 rights, or permission bits: r (read), w(write), R (exclusive read), W (exclusive write), and o (ownership).

A permission is called linear iff it contains an exclusive right matched with its baseright, that is, either rR or wW. Compared to the introduction of linear types, the onesto be introduced are not single-use values, but rather may only have a single referenceat a time, which is conceptually equivalent to having the same parameter act as bothinput and output arguments, where passing a value makes a function use it and thenwrite back a new one. A linear object can only be referenced by or contained in otherlinear objects, in order to preserve linearity. For example, in the following code, b is an

16

Listing 3.1: Permission syntaxmain <- inner EOFinner <- '_' | [[basePermission] [func | map | chan | pointer | sliceOrArray]

| basePermission]basePermission ('o'|'r'|'w'|'R'|'W'|'m'|'l'|'v'|'a'|'n')+func <- ['(' param List ')'] 'func' '(' [paramList] ')'

( [inner] | '(' [paramList] ')')paramList <- inner (',' inner)*fieldList <- inner (';' inner)*sliceOrArray <- '[' [NUMBER|_] ']' innerchan <- 'chan' innerchan <- 'interface' '{' [fieldList] '}'map <- 'map' '[' inner ']' innerpointer <- '*' innerstruct <- 'struct' '{' fieldList '}'

array of mutable pointers. If the array were non-linear, we could copy it to b, creatingtwo references to each linear element, which is not allowed.

var a /* @perm orR [] owW * owW */ = make([]int)var b = a

We will see later that this actually would not be a problem, as the checks are recursiveand would prevent such an object from being copied, but it makes no real sense tohave an object marked as non-linear contain a linear one - it would be confusing to thereader, as it does not convey anything useful.

Ownership plays an interesting role with linear objects: It determines whether areference is moved or borrowed (temporarily moved). For example, when passing alinear map to a function expecting an owned map, the map will be moved inside thefunction; if the parameter is unowned, the map will be borrowed instead. Coming backto the analogy with in and out parameters, owned parameters are both in and out,whereas unowned parameters are only in.

Instead of using a store mapping objects, and (object, field) tuples to capabilities,that is, (object, permission) pairs, Lingo employs a different approach in order tocombat the limitations shown in the introduction: Lingo’s store maps a variable to apermission. In order to represents complex data structures, it does however not justhave the permission bits introduced earlier (from now on called base permission), butalso structured permissions, which are similar to types. These structured permissionsconsist of a base permission and permissions for each child, target, etc. - a “shadow”type system essentially.

There is one problem with the approach of one base permission and one permission perchild: Reference types like maps or functions actually need two base permissions: The

17

permission of the reference (as in, “can I assign a different map to this variable”) andthe permission of the referenced value (as in, “can I insert something into this map”).We will see later in section 3.3 on page 26 that this causes some issues.

Apart from primitive and structured permissions, there are also some special permis-sions:

• The untyped nil permission, representing the nil literal (following section 2.1 onpage 8).

• The wildcard permission, written _. It is used in permission annotations wheneverthe default permission for a type should be used.

There also are some shortcuts for some common combinations:

• m, for mutable, is equivalent to rwRW• v, for value, is equivalent to rW• l, for linear value, is equivalent to rRW and a linear variant of value• n, for none, is equivalent to, well, none bits set• a, for any, is equivalent to all non-exclusive bits set, that is orwRW.

The syntax for these permissions (except for nil, and tuple permissions - these makeno sense to actually write) is given in listing 3.1. The base permission does not needto be specified for structured types, if absent, it is considered to be om.

In the rest of the chapter, we will discuss permissions using a set based notation: Theset of rights, or permissions bits is R = {o, r, w,R,W}. A base permission is a subset⊂ R of it, that is an element in 2R. The set P is the infinite set of all permissions:

P = 2R ∪ {p struct {P0, ..., Pn}|p ⊂ R, Pi ∈ P} ∪ . . . ∪ {nil,_}

Compare the syntax chart in listing 3.1 for which permissions are possible.

Base permissions like b ∈ 2R are usually denoted by lower case, other permissions (orgenerically, all permissions) are denoted by uppercase characters like P ∈ R.

3.1.1 Excursus: Parsing the syntax

In the implementation, base permissions are stored as bitfields and structured per-missions are structs matching the abstract syntax. Permission annotations are storedin comments attached to functions, and declarations of variables. A comment lineintroducing a permission annotation starts with @perm, for example:var pointerToInt /* @perm om * om */ *int

Go’s excellent built-in AST package (located in go/ast) provides native support forassociating comments to nodes in the syntax tree in a understandable and reusable way.We can simply walk the AST, and map each node to an existing annotation or nil.

18

The permission specification itself is then parsed using a hand-written scanner anda hand-written recursive-descent parser. The scanner operates on a stream of runes(unicode code points), and represents a stream of tokens with a buffer of one token forlook-ahead. It provides the following functions to the parser:

• func (sc *Scanner) Scan() Token returns the next token in the token stream• func (sc *Scanner) Unscan(tok Token) puts the last token back• func (sc *Scanner) Peek() Token is equivalent to Scan() followed by Unscan()• func (sc *Scanner) Accept(types ...TokenType) (tok Token, ok bool) takes a

list of acceptable token types and returns the next token in the token streamand whether it matched. If the token did not match the expected token types,Unscan() is called before returning it.

• func (sc *Scanner) Expect(types ...TokenType) Token is like Accept() but er-rors out if the token does not match.

Error handling is not done by the usual approach of returning error values, becausethat made the parser code hard to read. Instead, when an error occurs, the built-inpanic() is called with a scannerError object as an argument. This makes the scannernot very friendly to use outside the package, but it simplifies the parser, which callsrecover in its outer Parse() method to recover any such error and return it as an errorvalue.

Listing 3.2: The outer Parse() function of the parserfunc (p *Parser) Parse() (perm Permission, err error) {

defer func() {if r := recover(); r != nil {

switch rr := r.(type) {case scannerError:

perm = nilerr = rr

default:panic(rr)

}}

}()perm = p.parseInner()// Ensure that the inner run is completep.sc.Expect(TokenEndOfFile)return perm, nil

}

With these functions, it is easy to write a recursive descent parser. For example, thecode for parsing <basePermission> * <permission> for pointer permission is just this:func (p *Parser) parsePointer(bp BasePermission) Permission {

p.sc.Expect(TokenStar)

19

rhs := p.parseInner()return &PointerPermission{BasePermission: bp, Target: rhs}

}

Internally the scanner is implemented by a set of functions:

• func (sc *Scanner) readRune() rune returns the next Unicode code point fromthe input string

• func (sc *Scanner) unreadRune() moves one rune back in the input stream• func (sc *Scanner) scanWhile(typ TokenType, acceptor func(rune) bool)

Token creates a token by reading and appending runes as long as the givenacceptor returns true.

The main Scan() function calls readRune to read a rune and based on that rune decidesthe next step. For single character tokens, the token matching the rune is returneddirectly. If the rune is a character, then unreadRune() is called to put it back and sc.scanWhile(TokenWord, unicode.IsLetter) is called to scan the entire word (includingthe unread rune). Then it is checked if the word is a keyword, and if so, the properkeyword token is returned, otherwise the word is returned as a token of type Word(which is used to represent permission bitsets, since the flags may appear in any order).Whitespace in the input is skipped:for {

switch ch := sc.readRune(); {case ch == 0:

return Token{}case ch == '(':

return Token{TokenParenLeft, "("}...case unicode.IsLetter(ch):

sc.unreadRune()tok := sc.scanWhile(TokenWord, unicode.IsLetter)assignKeyword(&tok)return tok

case unicode.IsDigit(ch):sc.unreadRune()return sc.scanWhile(TokenNumber, unicode.IsDigit)

case unicode.IsSpace(ch):default:

panic(sc.wrapError(errors.New("Unknown␣character␣to␣start␣token:␣" +string(ch))))}

}

20

3.2 Assignment operations

Some of the core operations on permissions involve assignability: Given a sourcepermission and a target permission, can I assign an object with the source permissionto a variable of the target permission?

As a value based language, one of the most common forms of assignability is copying:var x /* @perm or */ = 0var y = x // copy

Another one is referencing:var x /* @perm or */ = 0var y = &x // referencevar z = y // still a reference to x, so while we copy the pointer, we also

reference x one more time

Finally, in order to implement linearity, we need a way to move things:var x /* @perm om */ = 0 // this was or before (!!!)var y = &x // have to move x, otherwise y and x both reach xvar z = y // have to move the pointer from y to z, otherwise both reach x

(Though, since we do not modify the semantics of Go, we actually just pretend to movestuff and just mark the original value as unusable afterwards.)

In the following, the function assµ : P ×P → bool describes whether a value of the leftpermission can be assigned to a location of the right permission. µ is the mode, it canbe either cop for copy, ref for reference, or mov for move.

The base case for assigning is base permissions. For copying, the only requirementis that the source is readable (or it and target are empty). A move additionallyrequires that no more permissions are added - this is needed: If I move a pointer toa read-only object, I cannot move it to a pointer to a writeable object, for example.When referencing, the same no-additional-permissions requirement persists, but bothsides may not be linear - a linear value can only have one reference, so allowing tocreate another would be wrong.

assµ(a, b) :⇔

r ∈ a or a = b = ∅ if µ = cop

b ⊂ a and (r ∈ A or a = b = ∅) if µ = mov

b ⊂ a and and not lin(a) and not lin(b) if µ = ref

where lin(a) :⇔ r,R ∈ a or w,W ∈ a

21

Listing 3.3: Base case of assignability, in code formfunc (perm BasePermission) isAssignableTo(p2 Permission, state

assignableState) bool {perm2, ok := p2.(BasePermission)if !ok {

return false}switch state.mode {case assignCopy:

return perm&Read != 0 || (perm == 0 && perm2 == 0) // Either Areadable, or both empty permissions (hack!)case assignMove:

return perm2&^perm == 0 && (perm&Read != 0 || (perm == 0 && perm2 ==0)) // No new permission && copycase assignReference:

return perm2&^perm == 0 && !perm.isLinear() && !perm2.isLinear() //No new permissions and not linear}panic(fmt.Errorf("Unreachable,␣assign␣mode␣is␣%v", state.mode))

}

The r ∈ a or a = b = ∅ requirement for mov is not entirely correct. There really shouldbe two kind of move operations: Moving a value (which needs to read the value), andmoving a reference to something as b ⊂ a (like moving a pointer om * om does notrequire reading the om target). The latter is essentially similar to a subtype relationshipif permissions were types.

In the code, the function is implemented like in listing 3.3:

Next up are permissions with value semantics: arrays, structs, and tuples (tuples areonly used internally to represent multiple function results). They are assignable if alltheir children are assignable.

assµ(a [_]A, b [_]B) :⇔ assµ(a, b) and assµ(A,B)assµ(a struct {A0; . . . ;An},

b struct {B0; . . . ;Bm}):⇔ assµ(a, b) and assµ(Ai, Bi) ∀0 ≤ i ≤ n

assµ(a (A0, . . . , An), b (B0, . . . , Bm)) :⇔ assµ(a, b) and assµ(Ai, Bi) ∀0 ≤ i ≤ n

Channels, slices, and maps are reference types. They behave like value types, exceptthat copying is replaced by referencing.

22

assµ(a chan A, b chan B) :⇔{assref (a, b) and assref (A,B) µ = cop

assµ(a, b) and assµ(A,B) else

assµ(a []A, b []B) :⇔{assref (a, b) and assref (A,B) µ = cop

assµ(a, b) and assµ(A,B) else

assµ(a map [A0]A1, b map [B0]B1) :⇔

assref (a, b) and assref (A0, B0)and assref (A1, B1) µ = cop

assµ(a, b) and assµ(A0, B0)and assµ(A1, B1) else

Interfaces work the same, but methods are looked up by name.

assµ(a interface {A0; . . . ;An},b interface {B0; . . . ;Bm})

:⇔{assref (a, b) and assref (Aidx(Bi,A), Bi) µ = cop

assµ(a, b) and assµ(Aidx(Bi,A), Bi) elsefor all 0 ≤ i ≤ m

where idx(Bi, A) determines the position of a method with the same name as Bi in A.

Function permissions are a fairly special case. The base permission here essentiallyindicates the permission of (elements in) the closure. A mutable function is thus afunction that can have different results for the same immutable parameters. Thereceiver of a function, its parameters, and the closure are essentially parameters ofthe function, and parameters are contravariant: I can pass a mutable object when aread-only object is expected, but I cannot pass a read-only object to a mutable object.For the closure, ownership is the exception: An owned function can be assigned to anunowned function, but not vice versa.

assµ(a (R) func (P0 . . . , Pn)(R0, . . . , Rm),b (R′) func (P ′0 . . . , P ′n)(R′0, . . . , R′m))

:⇔

assref (a ∩ {o}, b ∩ {o})and assref (b \ {o}, a \ {o})and assmov(R′, R)and assmov(P ′i , Pi)and assmov(Rj , R′j) µ = cop

assµ(a ∩ {o}, b ∩ {o})and assµ(b \ {o}, a \ {o})and assmov(R′, R)and assmov(P ′i , Pi)and assmov(Rj , R′j) elsefor all 0 ≤ i ≤ n, 0 ≤ j ≤ m

23

mov is used for the receiver, parameters, and return values, due to containing thesub-permission-of semantic. That it also contains the read requirements was unintended,and can cause some issues here as hinted before in section 3.2 on page 22: A functionwith an unreadable argument cannot be copied (usually, unless both source and targetparameter permissions are simply n), for example.

Pointers are another special case: When a pointer is copied, the pointer itself is copied,but the target is referenced (as we now have two pointers to the same target).

assµ(a ∗A, b ∗B) :⇔{assµ(a, b) and assref (A,B) µ = cop

assµ(a, b) and assµ(A,B) else

There is one minor deficiency with this approach: A pointer a with permission ol * omcannot be moved into a pointer b with permission om * om, due to the rule aboutnot adding any permissions. But that’s not always correct, which brings us backto the moving values vs moving references: b = a should be possible, but it shouldnot be possible to assign a pointer to a (e.g. om * ol * om) to a pointer to b (e.g.om * om * om) - now we could access b with more permissions than we created it with.This issue means that a function should probably accept ol * om pointers rather thanom * om, but that seems a minor issue.

Finally, we have some special cases: The wildcard and nil. The wildcard is notassignable, it is only used when writing permissions to mean “default”. The nilpermission is assignable to itself, to pointers, and permissions for reference andreference-like types.

assµ(_, B) :⇔ falseassµ(nil, a ∗B) :⇔ true assµ(nil, a chan B) :⇔ true

assµ(nil, a map [B]C) :⇔ true assµ(nil, a[]C) :⇔ trueassµ(nil, a interface {. . .}) :⇔ true assµ(nil,nil) :⇔ true

3.3 Conversions to base permissions

Converting a given permission to a base permission essentially replaces all base per-missions in that permission with the specified one, except for some exceptions likefunctions, which we’ll see later in this section. Its major use case is specifying anincomplete type, for example:var x /* @perm om */ *int

24

It is a pointer, but the permission is only for a base. We can convert the defaultpermission for the type (we’ll discuss type default permissions later in section 3.5 onpage 34) to om, giving us a complete permission. And in the next section, we’ll extendconversion to arbitrary prefixes of a permission.

Another major use case is ensuring consistency of rules, like:

• Unwriteable objects may not embed any writeable objects• Non-linear unwriteable objects may contain pointers to non-linear writeable

objects• Linear unwriteable objects may point to linear writeable objects.

(That is, while unwriteable objects cannot contain writeable objects directly, they canpoint to them as long as linearity is respected)

As every specified permission will be converted to its base type, we can ensure thatevery permission is consistent, and we do not end up with inconsistent permissions likeor * om - a pointer that could be copied, but pointing to a linear object.

The function ctb : P × 2R → P is the convert-to-base function. Its simple cases areconversions from a base permission or wildcard, yielding the target base permission,and conversions from nil, yielding nil.

ctb(a, b) := b

ctb(_, b) := b

ctb(nil, b) := nil

For comparison, this is how the first case looks in the reference implementation:func (perm BasePermission) convertToBaseBase(perm2 BasePermission)

BasePermission {return perm2

}

Otherwise, apart from functions, interfaces, and pointers, ctb is just applied recursively.

ctb(a chan A, b) := ctb(a, b) chan ctb(A, ctb(a, b))ctb(a []A, b) := ctb(a, b) []ctb(A, ctb(a, b))

ctb(a [_]A, b) := ctb(a, b) [_]ctb(A, ctb(a, b))ctb(a map[A]B, b) := ctb(a, b) map[ctb(A)]ctb(B, ctb(a, b))

ctb(a struct {A0; . . . ;An}, b) := ctb(a, b) struct {ctb(A0, ctb(a, b)); . . . ;ctb(An, ctb(a, b))}

ctb(a (A0; . . . ;An), b) := ctb(a, b) (ctb(A0, ctb(a, b)); . . . ; ctb(An, ctb(a, b)))

25

The rules are problematic in some sense, though: All children have the same basepermission as their parent. This kind of makes sense for non-reference values likestructs containing integers - after all, they are in one memory location; but for referencetypes, it is somewhat confusing: For example, a struct cannot have both a mutable(om map...) and a read-only map (or map...) as their base permissions are different. Asmentioned before in section 3.1 on page 18, these really need a second base permissionfor the object being referenced (like a pointer, see below). Then both maps couldbe (linear) read-only references, one referencing a mutable map, one referencing aread-only map.

Functions and interfaces are special, again: methods, and receivers, parameters, resultsof functions are converted to their own base permission.

ctb(a (R) func (P0, . . . , Pn)(R0, . . . , Rm), b):= ctb(a, b) (ctb(R, base(R))) func

(ctb(P0, base(P0)), . . . , ctb(Pn, base(Pn)))(ctb(R0, base(R0)), . . . , ctb(Rm, base(Rm)))

ctb(a interface {A0; . . . ;An}, b):= ctb(a, b) interface {ctb(A0, base(A0)) . . . ; ctb(An, base(An))}

The reason for this is simple: Consider the following example:var x /* om */ func(*int) *int

x should be om, but this does not mean that it should be om func (om * om) om justbecause the closure might be mutable - a function parameter should have the leastpermissions possible, so you can pass as many things as possible into it. The defaultalso should be unowned, so a function does not consume it if it is linear, but releases itagain later, so it can be used again in the caller.

For pointers, it is important to add one thing: There are two types of conversions:Normal ones and strict ones. The difference is simple: While the normal one combinesthe old target’s permission with the permission being converted to, strict conversionjust converts the target to the specified permission. Strict conversions will becomeimportant when doing a type conversion (recall section 2.1 on page 7), for example,value to interface:var x /* om * or */ *intvar y /* om interface {} */ = xvar z /* om * om */ = y.(*int) // um, target is mutable now?

Converting to an interface is a lossy operation: We can only maintain the outerpermission. But we cannot allow the case above to happen: We just converted a pointer

26

to read-only data to a pointer to writeable data. Not good. One way to solve this is toensure that a permission can be assigned to it is strict permission, gathered by strictlyconverting the type-default permission to the current permissions base permission:

y = x⇔ assµ(perm(x), ctbstrict(perm(typeof(x)), base(perm(x)) and assµ(base(x), base(y))

The rules for converting a pointer permission to a base permission are therefore a bitcomplicated – basically, if the base permission becomes non-linear, the target becomesnon-linear as well.

ctb(a ∗A, b) : = a′ ∗ ctb(A, t \X)where a′ = ctb(a, b)(= b)

t = (base(A) \ {o}) ∪ (a′ ∩ {o})

X =

{R,w,W} if R 6∈ a′ and t ⊃ {r,R,w,W}{w,W} else if R 6∈ a′ and t ⊃ {w,W}{R} else if R 6∈ a′ and t ⊃ {r,R}∅ else

ctbstrict(a ∗A, b) : = ctbstrict(a, b) ∗ ctbstrict(A, ctbstrict(a, b))

In the formal notation, t replaces the owned permission bit from the old target withthe owned flag from the given base permission. This is needed to ensure that we do notaccidentally convert ‘om * om‘ to ‘m * om‘. Keeping ownership the same throughoutpointers also simplifies some other aspects in later code. X ensures consistency: Ifour new target is not linear, we strip any linearity from the target; thus only a linearpermission can have linear inner permissions.

Pointer examples

Assuming we have a function that accepts a pointer:func foo(*int) {

...}

The default pointer permission here would be m * m. With the rules defined, we canwrite an incomplete annotation and get good results for the useful cases:

1. @perm func(om) pointer is now om * om (case else)2. @perm func(r) pointer is now r * r (case 1)

Some cases are weird, though. Converting it to rw yields a non-linear writable pointerwith a readonly target (but that’s the only safe choice, really). Converting it to l onlymakes the pointer linear but does not modify the target.

27

1. @perm func(rw) pointer is now rw * r (case 1) - sure we could do rw * rw instead,but if we then assigned a om * om value to it, it could end up with multiple writereferences.

2. @perm func(l) pointer is now l * m (case else) - this probably makes no realsense.

Theorem: ctbb(A) = ctb(A, b) is idempotent

Theorem: Conversion to base, ctb is idempotent, or rather ctbb(A) = ctb(A, b) is. Thatis, for all A ∈ P, b ∈ 2R: ctbb(A) = ctb(A, b) = ctb(ctb(A, b), b) = ctbb(ctbb(A)).

Background: This theorem is important because we generally assume thatctb(A, base(A)) = A for all A ∈ P that have been converted once (what is calledconsistent, and is the case for all permissions the static analysis works with).

Proof. This only shows the proof for ctb(), not ctbstrict(), but the only difference is thepointer case, which can be proven like channels below.

1. Simple cases:

ctb(ctb(a, b), b) = ctb(b, b) = b = ctb(a, b)ctb(ctb(_, b), b) = ctb(b, b) = b = ctb(_, b)ctb(ctb(nil, b), b) = ctb(nil, b) = nil = ctb(nil, b)

2. Channels, slices, arrays, maps, structs, and tuples basically have the same rules:All children are converted to the same base permission as well. It suffices to showthe proof for one of them. Let us pick channels:

ctb(ctb(a chan A, b), b)= ctb(ctb(a, b) chan ctb(A, ctb(a, b)), b) (def chan)= ctb(ctb(a, b), b) chan ctb(ctb(A, ctb(a, b)), b) (def chan)= ctb(b, b) chan ctb(ctb(A, b), b) (ctb(a, b) = b)= b chan ctb(A, b) (ctb(a, b) = b, other case)= ctb(a, b) chan ctb(A, ctb(a, b)) (ctb(a, b) = b, other case)= ctb(a chan A, b) (def chan)

3. Functions and interfaces convert their child permissions to their own bases. Wecan proof the property for the special case of an interface with one methodwithout loosing genericity, since these are structured the same.

28

ctb(ctb(a interface {A0}, b), b)=ctb(ctb(a, b) interface {ctb(A0, base(A0))}, b) (def)= ctb(ctb(a, b), b)︸ ︷︷ ︸

=ctb(a,b)

interface {ctb(ctb(A0, base(A0)), base(ctb(A0, base(A0))))︸ ︷︷ ︸=base(A0)(trivial)

} (def)

=ctb(a, b) interface {ctb(ctb(A0, base(A0)), base(A0))︸ ︷︷ ︸case of ctb(ctb(A, b), b)

}

=ctb(a, b) interface {ctb(A0, base(A0))}=ctb(a interface {A0}, b) (def)

4. Pointers are more complicated:

Recall that ctb(a, b) = b for all a, b ∈ 2R. Thus for all A ∈ P, b ∈ 2R, it followsthat

ctb(a ∗A, b) = b ∗ ctb(A, t \X)ctb(ctb(a ∗A, b), b) = ctb(b ∗ ctb(A, t \X), b) = b ∗ ctb(ctb(A, t \X), t′ \X ′)

where t,X and t′, X ′ are the helper variables for these equations as defined insection 3.3 on page 27.

For t and t′ it follows that (directly replaced a′ with b in the definition forreadability)

tdef= (base(A) \ {o}) ∪ (b ∩ {o})

t′def= (base(ctb(A, t \X)) \ {o}) ∪ (b ∩ {o})= (t \X) \ {o}) ∪ (b ∩ {o})= ((base(A) \ {o}) ∪ (b ∩ {o}) \X) \ {o}) ∪ (b ∩ {o})o 6∈X= ((base(A) \X) \ {o}) ∪ (b ∩ {o})o 6∈X= ((base(A)) \ {o}) ∪ (b ∩ {o}) \Xo 6∈X= t \X

If we show that X ′ ⊂ X, then we know that t′ \X ′ = (t \X) \X ′ = t \X, andthus it would follow that:

29

ctb(ctb(a ∗A, b), b) = . . . = b ∗ ctb(ctb(A, t \X), t′ \X ′)= b ∗ ctb(ctb(A, t \X), t \X) due to X ′ ⊂ X= b ∗ ctb(A, t \X) due to base case= ctb(a ∗A, b) per definition

Substituting t for t \X and a for b in the definition of X yields X ′:

X ′ =

{R,w,W} if R 6∈ b and t \X ⊃ {r,R,w,W}{w,W} else if R 6∈ b and t \X ⊃ {w,W}{R} else if R 6∈ b and t \X ⊃ {r,R}∅ else

To show that X ′ ⊂ X, let’s first exclude R ∈ b. If R ∈ b, then X ′ = ∅ and thusX ′ ⊂ X. Now, assuming R 6∈ b. We actually will show that X ′ = ∅, by proof bycontradiction for the other cases:

a) Assume that t \X ⊃ {r,R,w,W}.

⇒ t ⊃ {r,R,w,W} def. X====⇒ X = {r, w,W} ⇒ t \X 6⊃ {r,R,w,W}

b) Assume that t \X ⊃ {w,W}.

⇒ t ⊃ {w,W} def. X====⇒ X = {w,W} or X{R,w,W} ⇒ t \X 6⊃ {w,W}

c) Assume that t \X ⊃ {r,R}.

⇒ t ⊃ {r,R} def. X====⇒ X = {R} or X{R,w,W} ⇒ t \X 6⊃ {r,R}

d) Therefore, the else case applies and X ′ = ∅.

Thus ctb(ctb(a ∗A, b), b) = ctb(a ∗A, b).

In conclusion, ctb(ctb(A, b), b) = ctb(A, b) for all A ∈ P, b ⊂ R, as was to be shown. Italso follows that ctbstrict(ctbstrict(A, b), b) = ctbstrict(A,B) because the functions arethe same, except for the diverging pointer case, but that one is trivial to proof (likechannels).

30

3.4 Other conversions and merges

The idea of conversion to base permissions from the previous paragraph can be extendedto converting between structured types. When converting between two structuredtypes, replace all base permissions in the source with the base permissions in the sameposition in the target, and when the source permission is structured and the target isbase, it just switches to a to-base conversion.

There are two more kinds of recursive merge operations: intersection and union.These are essentially just recursive relaxations of intersection and union on the basepermissions, that is, they simply perform intersection and union on all base types inthe structure. A static analyser could use intersections to join the results of differentbranches, for example:if (...) {

myfun = function expecting mutable value} else {

myfun = function expection read-only value}

myfun = intersect(myfun in first branch, my fun in second branch)

In this example, after the if/else block has been evaluated, the permissions of myfunare an intersection of the permission it would have in both branches.

The function performing merges and conversions is mergeµ : P×P → P . µ is the mode,which can be either union (∪), intersection (∩), conversion (ctb), or strict conversion(ctbstrict).

In essence, mergeµ just extends an underlying function µ : 2R × 2R → P (∩ and ∪)or µ : P × 2R → P (ctb and ctbstrict) to a function P × P → P. In the latter case, wedirectly use µ(A, b) for all structured permissions A and base permissions b, so thefunction can do special handling for the structured permission in the first argument.

mergeµ(A, b) := µ(A, b) ={ctb(A, b) if µ = ctb

ctbstrict(A, b) if µ = ctbstrict

for all µ : P × 2R → P and A ∈ P \ 2R

The wildcard exists just as a placeholder for annotation purposes, so merging it withanything should yield the other value. For nil permissions, merging them with a nilablepermission (a chan, func, interface, map, nil, pointer, or slice permission) yields theother permission.

31

mergeµ(_, B) := _ mergeµ(A,_) := _mergeµ(N,nil) := N mergeµ(nil,N) := N for all nilable N ∈ P and N = nil

Regarding the soundness of the merging nils with nilable permissions for non-conversionmodes:

• For union, the question is: Can merge∪(N,nil) = N be used in place of bothN and nil? Technically the answer is no, because N cannot be used where nilis expected. But nil permissions are only ever used for nil literals (they cannoteven be specified, there is no syntax for them), so we never reach that situation.N can of course be used where N can be used.

• For intersection, the question is: Can values of N or nil be assigned tomerge∩(N,nil) = N . Yes, they can be, nil is assignable to every pointer, and Nis assignable to itself (at least if readable, but it makes no sense otherwise)

Otherwise, the base case for a merge is merging primitive values: Just call µ.

mergeµ(a, b) := µ(a, b) =

ctb(a, b) if µ = ctb orctbstrict(a, b) if µ = ctbstrict

a ∩ b if µ = ∩a ∪ b if µ = ∪

In the code, this is implemented as a function on some special mergeState type (listing3.4). This state happens to record the mode of operation for the merge function, sothe recursion does not need to be duplicated for each of them. It also has another usecase, to which we will get back later, at the end of the chapter.

Listing 3.4: Base case of merge, in code formfunc (state *mergeState) mergeBase(p1, p2 BasePermission) BasePermission {

switch state.action {case mergeConversion, mergeStrictConversion:

return p1.convertToBaseBase(p2) // call ctb base case for typereasonscase mergeIntersection:

return p1 & p2case mergeUnion:

return p1 | p2}panic(fmt.Errorf("Invalid␣merge␣action␣%d", state.action))

}

32

Pointers, channels, arrays, slices, maps, tuples, structs, and interfaces are trivial (structsand interfaces must have same number of members / methods) - mergeµ just recurses.

mergeµ(a ∗A, b ∗B) := mergeµ(a, b) ∗mergeµ(A,B)mergeµ(a chan A, b chan B) := mergeµ(a, b) chan mergeµ(A,B)

mergeµ(a[_]A, b[_]B) := mergeµ(a, b)[_]mergeµ(A,B)mergeµ(a[]A, b[]B) := mergeµ(a, b)[]mergeµ(A,B)

mergeµ(a map[A0] A1, b map[B0] B1) := mergeµ(a, b) map[mergeµ(A0, B0)]mergeµ(A1, B1)

mergeµ(a(A0, . . . , An), b(B0, . . . , Bn)) := mergeµ(a, b)(mergeµ(A0, B0),. . . ,

mergeµ(An, Bn))mergeµ(a struct {A0, . . . , An},

b struct {B0, . . . , Bn}) := mergeµ(a, b) struct {mergeµ(A0, B0),. . . ,

mergeµ(An, Bn)}mergeµ(a interface {A0, . . . , An},

b interface {B0, . . . , Bn}) := mergeµ(a, b) interface {mergeµ(A0, B0),. . . ,

mergeµ(An, Bn)}

Functions are more difficult: An intersection of a function requires union for closure,receivers, and parameters, because just like with subtyping (in languages that haveit), parameters and receivers are contravariant: If we have a func(orw) and a func(or), a place (like a function (parameter)) that needs to accept functions of bothpermissions, needs to accept func(orw \cup or) = func(orw) - passing a writeableobject to a function only needing a read-only one would work, but passing a read-onlyvalue to a function that needs a writeable one would not be legal.

For that, let

mergeContraµ(A,B) :=

merge∩(A,B) if µ = ∪merge∪(A,B) if µ = ∩mergeµ(A,B) else

be a helper function that merges contravariant things after swapping union andintersection modes.

33

Then merging functions is:

mergeµ(a(R) func (P0, . . . , Pn)(R0, . . . , Rn), b(R′) func (P ′0, . . . , P ′n)(R′0, . . . , R′n)):=mergeContraµ(a, b)(mergeContraµ(R,R′)) func

(mergeContraµ(P0, P′0), . . . ,mergeContraµ(Pn, P ′n))

(mergeµ(R0, R′0), . . . ,mergeµ(Rn, R′n))

Theorem: mergeµ is commutative for commutative µ.

An interesting property of mergeµ is that it is commutative if µ is commutative, thatis for the intersection ∩ and the union ∪.

This follows directly from the structural definitions given above - they just recursivelycall mergeµ until they reach a base case for which µ can be called. For example, forchannels:

mergeµ(a chan A, b chan B) = mergeµ(a, b) chan mergeµ(A,B)= mergeµ(b, a) chan mergeµ(B,A)= mergeµ(b chan B, a chan A)

The other cases are trivial as well, and therefore no complete proof will be shown.

3.5 Creating a new permission from a type

Since permissions have a shape similar to types and Go provides a well-designed typespackage, we can easily navigate type structures and create structured permissions forthem with some defaults. Currently, it just places maximum m permissions in all basepermission fields. And the interpreter, discussed in the next section, converts to ownedas needed, using ctb().

One special case exists: If a type is not understood, we try to create the permissionfrom it is underlying type. For example, type Foo int is a named type, but we do notsupport named types, so we use the underlying type, int, for creating the permission.

3.6 Handling cyclic permissions

So far, we have only looked at permissions without cycles. In the real world, permissionscan have cycles, because types can have cycles too, for example, type T []T is a type

34

that is a slice of itself. The functions discussed so far transparently handle cycles witha simple caching mechanism. Essentially, all functions seen so far recurse via a wrapperfunction that first checks the cache for the given arguments and returns the cachedvalue if it exists, and only calls the real function if the arguments were not seen yet.

For predicate functions, that is, the assignability functions, this wrapper function doesall the work, including registering the arguments in the cache, as listing 3.5 shows forthe implementation of the ass family.

Listing 3.5: Cycle helper for assignability functionfunc assignableTo(A, B Permission, state assignableState) bool {

key := assignableStateKey{A, B, state.mode}isMovable, ok := state.values[key]

if !ok {state.values[key] = trueisMovable = A.isAssignableTo(B, state)state.values[key] = isMovable

}

return isMovable}

For producer functions, that is, functions producing permissions, it is similar. Forexample, listing 3.6 shows the wrapper function for convertToBase.

Listing 3.6: Cycle helper for convert-to-base functionfunc convertToBase(perm Permission, goal BasePermission, state *

convertToBaseState) Permission {key := mergeStateKey{perm, goal, state.action}result, ok := state.state[key]if !ok {

result = perm.convertToBase(goal, state)}return result

}

The actual registering of the expected output in the cache does not happen here,though. We need to do this in the concrete methods, as we need to construct a newpermission first. Hence, all merge and convertToBase methods start with somethinglike in listing 3.7.

This also means that the proof for ctb holds even in the face of cycles. For example,if we have the permission A := a[]A corresponding to our type above, then, whenconverting this to b, the inner ctb(A, b) will be the result of the outer permission. As

35

Listing 3.7: Registering return values for cycles in convertToBasefunc (p *SlicePermission) convertToBase(p2 BasePermission, state *

convertToBaseState) Permission {next := &SlicePermission{}state.register(next, p, p2)

// convertToBase(p, p2, state) returns next now

long as the rule holds for a cycle free permission, it thus also holds for a permissionwith cycles.

Also noteworthy: For merge, it is this wrapper function that handles the fallbackto convertToBase by checking if we are converting a non-base permission to a base-permission and then calling convertBase instead. This avoids having to implement a“to-base” case for each type.

3.7 Summary

We have introduced:

• the set of base permission bits R = {o, r, w,R,W}• the set of permissions P consisting of 2R and structured permissions like a ∗A

(a ∈ 2R, A ∈ P)

We also defined operations for

• checking whether assignments of values with these permissions are legal: asscop,assref and assmov, for copying, referencing, and moving.

• ensuring consistency and completing incomplete annotations: ctb and ctbstrict, aswell as mergectb and mergectbstrict

.• merging permissions from values in different program branches: merge∩ andmerge∪

• creating permissions from types

We also saw that permissions can have cycles in practice, and that these cycles arehandled in the implementation in a generic way.

36

4 Static analysis of Go programs

Based on the operations described in the previous section, a static analyser can bewritten that ensures that the rules of linearity are respected. This static analysis canbe done in the form of an abstract interpreter; that is, an interpreter that does notoperate on concrete values, but abstract values and tries to interpret all possible pathsthrough a program.

We will introduce a store mapping variables to permissions with some operations, anexpression evaluator , and a statement evaluator →. There will also be severalimportant helper functions like moc which moves or copies a value, depending on whichaction is applicable, and defineOrAssign which takes care of defining and assigningvalues. Most of the chapter will be in the form of a operational semantics, thoughsome more complex cases will be discussed in code form for better readability.

In the github.com/julian-klode/lingolang reference implementation, the permissionsand operations are provided in the package called capabilities for historic reasons. Abetter name might have been interpreter.

4.1 Overview

The abstract interpreter acts on a store which maps variables to permissions. Evaluatingan expressions in a store yields a permission for the object returned by the expression,a new store, and an owner and dependencies - these two deserve some explanation:

An owner here means the variable from which the object the evaluation evaluates tohas been reached, and its effective permission at the time it was reached. When anidentifier is evaluated, the variable and its effective permission become the owner, andthe original variable is rendered unusable (by converting it to n). The owner will beimportant in some places later on. For example, when a part of an object is madeimmutable, we also want to make the whole object - the owner - immutable (or morespecifically, we would like to note that the part is now immutable, but we need theowner anyway). Not every expression has an owner: For example, an addition neverhas an owner: The returned value is freshly created, it is not part of another value.

A dependency is similar to an owner. It is another variable that has been used in theexpression, alongside its effective permission. A list of dependencies could also containowners: For example, in a[b], the owners and dependencies of b would be some of thedependencies of the complete expression (the others are the dependencies of a).

37

Owners and dependencies can be released back to the store when they are no longerneeded. For example, when a + b is evaluated, the owners and dependencies for a andb can be released after a + b has been evaluated. Sometimes, a value is moved, thenits owner and dependencies are not released - they become consumed by the operation,so to speak.

The abstract interpreter for statements evaluates the statement in a store and acurrent function – the latter is needed for some lookups, like return values. Multiplebranches are handled by returning a set of stores, rather than just one. Statements alsohave early exits like return statements. We handle these by including the statementthat abnormally terminated a statement alongside the store. For example, a returnstatement would have one exit: a pair of its store and itself. A block statement with areturn statement would have the exit of the return statement, and maybe others.

4.2 The store

The store is an essential part of the interpreter. It maps variable names ∈ V to:

1. An effective permission2. A maximum permission3. A use count

The maximum permission restricts the effective permission. It can be used to preventre-assignment of variables by revoking the write permission there. The uses count willbe used later when evaluating function literals to see which values have been capturedin the closure of the literal.

The store has several operations:

1. GetEffective, written S[v], returns the effective permission of v in S.2. GetMaximum, written S[v], returns the maximum permission of v in S.3. Define, written S[v := p], is a new store where a new v is defined if none is in

the current block, otherwise, it is the same as S[v = p].4. SetEffective, written S[v = p], is a new store where v’s effective permission is

set to merge∩(S[v], p)5. SetMaximum, written S[v = p], is a new store where v’s maximum permission is

set to p. It also performs S[v = merge∩(p, S[v])] to ensure that the effectivepermission is weaker than the maximum permission.

6. Release, written S[= D], where D is a set of tuples V ×P is the same as settingthe effective permissions of all (v, p) ∈ D in S. We call that releasing D, becauseD will be a set of dependencies we borrowed from the store.

7. BeginBlock, written S[+], is a store where a new block has begun8. EndBlock, written S[−], is S with the most recent block removed9. Merge, written S ∩ S′, where S and S’ have the same length and variables in the

same order, is the result of intersecting all permissions in S with the ones at the

38

same position in S’.

In the code, the store is a slice of structs where each struct contains a name, an effectivepermission, a maximum permission, and the number of times the variable has beenreferenced so far. Defining a new variable or beginning a new block scope prepends tothe store.type Store []struct {

name stringeff permission.Permissionmax permission.Permissionuses int

}

The beginning of a block scope is marked by a struct where the fields have their zerovalues, that is {"", 0, 0, 0}. More specifically, such a block marker is identified bychecking if the name field is empty. When exiting a block, we simply find the first suchmarker, and then create a slice starting with the element following it.

4.3 Expressions

The function : Expr × Store → Permission × (V ariable, Permission) ×set of (V ariable, Permission)× Store (also called VisitExpr in the code) abstractlyinterprets an expression in a store, checking permissions. It yields a new permission,an owner, a set of variables borrowed by the expression, and a new store.

The types Borrowed and Owner are pairs of a variable name and a permission, asmentioned before. There is a special NoOwner value of type Owner that represents thatno owner exists for a particular expression.

The Owner vs Borrowed distinction is especially important with deferred function callsand the go statement. We will later see that the owner is the function (which may bea closure with a bound receiver), while any owners and dependencies of the argumentsare forgotten.

There also is a sister function, VisitExprOwnerToDeps which does not return a owner,but instead inserts the owner into the list of dependencies. This is helpful in placeswhere the owner is not interesting (it is not used in the formal notation, but will beseen in some code excerpts later).

In the following, we will look at the individual expressions and check how they evaluate.

Identifier: id

There are three cases of identifiers:

39

Listing 4.1: Abstract interpreter for identifiersfunc (i *Interpreter) visitIdent(st Store, e *ast.Ident) (permission.

Permission, Owner, []Borrowed, Store) {if e.Name == "nil" {

return &permission.NilPermission{}, NoOwner, nil, st}if e.Name == "true" || e.Name == "false" {

return permission.Mutable | permission.Owned, NoOwner, nil, st}perm := st.GetEffective(e.Name)if perm == nil {

i.Error(e, "Cannot␣borow␣%s:␣Unknown␣variable␣in␣%s", e, st)}owner := Owner{e, perm}dead := permission.ConvertToBase(perm, permission.None)st, err := st.SetEffective(e.Name, dead)if err != nil {

i.Error(e, "Cannot␣borrow␣identifier:␣%s", err)}return perm, owner, nil, st

}

1. nil evaluates to the nil permission, it was created just for nil literals, since nilliterals can be assigned to any nilable value (compare section 3.2 on page 24).

2. true and false evaluate to the om permission, since they are just primitive values.They could just as well evaluate to any other (readable) base permission, but omhas all bits set, making it the “strongest” one.

3. Any other identifier id evaluates to the effective permission e in the store. Theeffective permission in the store is replaced with an unusable one (converted ton), and the owner becomes (id, e). The owner can later be released when thevariable is no longer needed.

〈nil, s〉 (nil,NoOwner, ∅, s) (P-Nil)〈true, s〉 (om,NoOwner, ∅, s) (P-True)〈false, s〉 (om,NoOwner, ∅, s) (P-False)〈id, s〉 (s[id], (id, s[id]), ∅, s[id = ctb(s[id], n)]) (P-Ident)

For a comparison, the code implementing this is shown in listing 4.1.

40

Star Expression: *E

The star expression dereferences a pointer. Therefore we must evaluate the expressionE and then dereference the permission it returns, that is, return the target permission.For example:// E has permission om * l*E // permission l

〈E, s〉 (a ∗A, o, d, s′) for some a ⊂ R, A ∈ P〈∗E, s〉 (A, o, d, s′) (P-Star)

Binary expression: A op B

First a is evaluated in s, yielding (Pa, oa, da, sa). The borrowed objects in oa and daare released, yielding s′a, and then b is evaluated in s′a, yielding: (Pb, ob, db, sb). Itsdependencies and owner are released as well, yielding s′b.

For logical operators, that is op ∈ {&&, ||}, the resulting store is generated by inter-secting the store s′a with s′b. This ensures that any code executed after A && B is validboth if only A was executed (and false), and if both A and B were executed (because Awas true).

〈A, s〉 (Pa, oa, da, sa) 〈B, sa[= da ∪ {oa}]〉 (Pb, ob, db, sb) r ∈ Pa, Pb〈A op B, s〉 (om,NoOwner, ∅, sa[= da ∪ {oa}] ∩ sb[= db ∪ {ob}])

(P-Logic, for all short-circuiting binary operators op)

For other operators, that is op 6∈ {&&, ||}, the resulting store is just the store afterexecuting B, as there is no “short circuiting”.

〈A, s〉 (Pa, oa, da, sa) 〈B, sa[= da ∪ {oa}]〉 (Pb, ob, db, sb) r ∈ Pa, Pb〈A op B, s〉 (om,NoOwner, ∅, sb[= db ∪ {ob}])

(P-Binary, for all not short-circuiting binary operators op)

In both cases, the result has no owner and no dependencies, and the permission om, asall binary operators produce primitive values, either boolean or numeric.

Examples:5 + 5 // om, no ownera + b // om, no owner

41

Index expression: A[B]

The index operator indexes an array, a slice, or a map (by the key type of the map). Itcan appear on the left-hand side of an assignment expression, and it is also addressable(except for maps): It is legal to take its address with the & operator. Having it appearon the left-hand side means that a map expression must move or copy the key into themap - we are storing a new value after all.someMap[someKey] = someValue // someKey is either moved or copied into

the map, as is someValue

We first evaluate the left side, then the index. If the left side is an array or a slice, theright side is a primitive value, so we can release its owner and dependency if any (it isjust an offset into an array), and then return the permission for the elements in thearray or slice, with A being the owner (as A[B] is part of A).

〈A, s〉 (pa[]Pa, oa, da, sa) 〈B, sa〉 (Pb, ob, db, sb) r ∈ pa, pb〈A[B], s〉 (Pa, oa, da, sb[= ob ∪ db])

(P-SIdx)

〈A, s〉 (pa[_]Pa, oa, da, sa) 〈B, sa〉 (Pb, ob, db, sb) r ∈ pa, pb〈A[B], s〉 (Pa, oa, da, sb[= ob ∪ db])

(P-AIdx)

In order to handle indexing maps, we need to take care of the left-hand-side situationmentioned above: The key must be copied or moved into the map. Since we donot know whether the expression is on a left-hand or right-hand side, we must beconservative and assume it is on the left-hand side.

We can define a helper function, called moveOrCopy (listing 4.2), or short moc (caseschecked in order, top to bottom). moc tries to copy first, and then falls back to movingit or (in case of assigning an object of mutable linear permission to a non-linear one),making it immutable and then copying the value.

moc(st, F, T, o, d) :=

(st[= d ∪ {o}], NoOwner, ∅) if asscop(F, T )⊥ if not assmov(F, T )(st, o, d) if o 6∈ base(T )(st[= d ∪ {unlinear(o)}], if lin(F )

and not lin(T )(st,NoOwner, ∅) else

where unlinear(o) :={o if o = NoOwner

(v, ctb(p, base(p) \ {R,W,w})) else, ∃v,p: o = (v, p)

42

Explanations for each case:

1. The permission F is copyable to T . This means that we can release the ownerand dependencies back to the store - they no longer need to be borrowed.

2. The permission F is not movable. This is an error case, all later cases requiremovability.

3. The target permission is unowned. We are just temporarily borrowing the object,so we keep the owner and dependencies around

4. We are moving a linear value to a non-linear value. This means we need to“freeze” the value, and any object containing it, that is make them immutableand non-linear.

5. We are doing any other kind of move. Since we handled copying in case 1, thismeans that we are moving a linear value to a linear value (a non-linear valuewould be copyable). Therefore, the owner and dependencies borrowed for F willbe forgotten, ensuring we cannot reach F via an alias once we moved it to T .

With moc defined, we are able to define for indexing maps. After evaluating themap and the key, we use moc to copy or move the key into the map, and the permissionfor values of the map is returned (and the owner is the map).

〈A, s〉 (pamap[K]V, oa, da, sa) 〈B, sa〉 (Pb, ob, db, sb) r ∈ pa, pb〈A[B], s〉 (V, oa, da, s′b) where s′b, o′b, d′b = moc(sb, Pb,K, ob, db)

(P-MIdx)

As can be seen, if any owner and dependencies are remaining for B after moc, theseare forgotten too. This only affects maps with unowned keys, as moc otherwise alwaysreturns no owner and an empty dependency set.

One thing missing is indexing strings, which yields characters. Strings are representedas base permissions currently, but probably deserve their own permission type.

Unary expressions

We have already seen one unary expression, the star expression. For unknown reasons,it is its own category of syntax node in Go, while all the other unary expressions sharea common type.

The first unary expression to discuss is &E, the address-of operator. Taking the addressof E constructs a pointer to it, currently simply by wrapping it in an om *.

〈E, s〉 (Pe, oe, de, se)〈&E, s〉 (om ∗ Pe, oe, de, se)

(P-Addr)

There is a problem with that approach and how assignment is handled: Given avariable v that is mutable, &v moves the permission from the store to the expression,

43

Listing 4.2: The essential moveOrCopy helper functionfunc (i *Interpreter) moveOrCopy(e ast.Node, st Store, from, to permission.

Permission, owner Owner, deps []Borrowed) (Store, Owner, []Borrowed,error) {switch {// If the value can be copied into the caller, we do not need to borrowitcase permission.CopyableTo(from, to):

st = i.Release(e, st, []Borrowed{Borrowed(owner)})st = i.Release(e, st, deps)owner = NoOwnerdeps = nil

// The value cannot be moved either, error out.case !permission.MovableTo(from, to):

return nil, NoOwner, nil, fmt.Errorf("Cannot␣copy␣or␣move:␣Needed␣%s,␣received␣%s", to, from)

// All borrows for unowned parameters are released after the call is done.case to.GetBasePermission()&permission.Owned == 0:// Write and exclusive permissions are stripped when converting a valuefrom linear to non-linearcase permission.IsLinear(from) && !permission.IsLinear(to):

if owner != NoOwner {owner.perm = permission.ConvertToBase(owner.perm, owner.perm.

GetBasePermission()&^(permission.ExclRead|permission.ExclWrite|permission.Write))

}st = i.Release(e, st, []Borrowed{Borrowed(owner)})st = i.Release(e, st, deps)owner = NoOwnerdeps = nil

// The value was moved, so all its deps are lostdefault:

deps = nilowner = NoOwner

}

return st, owner, deps, nil}

but only the effective permission is moved, as we saw in the definition for identifiersbefore. This causes problems later when introducing assignment statements: Theycheck the maximum permissions. What should happen here is that we also takeaway the maximum permission of the owner, preventing any future re-assignment andrestoration of its effective permissions .

44

The next operation is <-E, the channel receive operation. The expression E is a channel,and the next value in it is to be retrieved. The owner and the dependencies of thechannel are essentially irrelevant: The value received from the channel is not owned bythe channel, but its owned by whatever is receiving it; the owner and dependenciescan thus be released immediately after evaluating the expression.

〈E, s〉 (pe chan Pe, oe, de, se)〈← E, s〉 (Pe, NoOwner, ∅, se[= de ∪ {oe}])

(P-Recv)

Finally we have the “boring” case of other unary operators, like plus and minus. Theseare just working on primitive values, so we can just return a new primitive ownedmutable permission om and have no owner, since these construct new primitive values.

〈E, s〉 (Pe, oe, de, se)〈op E, s〉 (om,NoOwner, ∅, se[= de ∪ {oe}])

(P-Unary)

Basic literals

A basic literal lit just evaluates to an om permission, and obviously has no owner ordependencies.

〈lit, s〉 (om,NoOwner, ∅, s) (P-Lit)

Function calls

A function call E(A0, . . . , An) is fairly simple: First of all, the function is evaluated,then the arguments, from left to right. Owners and dependencies are collected untilthe end of the call, when they can be released. A special case in a function call are gostatements and defer statements: Here, no permissions are released, and the owner andthe dependencies of the function become the owner and dependencies of the functionexpression. This is needed: For deferred statement, the arguments are bound in placeof the statement, but the function is only executed when unwinding the call stack,hence these parameters need to be unreachable in the function where the statementis located. The Go statement is similar, except that execution is not done when thestack unwinds, but on a different goroutine. We define go/defer for these cases, whichis identical to , except for the call expression, as specified below.

45

〈E, s〉 (e func (P0, . . . , Pn)(R0, . . . , Rr), o, d, s−1)〈Ai, si−1〉 (PAi , o

′i, d′i, s′i)

si, oi, di = moc(s′i, PAi , Pi, o′i, d′i)

〈E(A0, . . . , An), s〉 go/defer (results, o, d, sn) if deferred or go〈E(A0, . . . , An), s〉 (results,NoOwner, ∅, rel(sn)) else

where rel(s) = s

[= {o} ∪ d ∪

n⋃i=0

(di ∪ {oi})]

results ={R0 if r = 0(R0, . . . , Rn) else

One seemingly odd result of the rules is that an argument is moved into a deferredcall or go call even if the parameter is unowned. Under normal circumstances bindingan argument to an unowned parameter would just temporarily borrow it. But with adeferred call or go call, there is no clear point at which the call ends, hence it is notpossible to release them again:// Case 1: deferred function call or go statementdefer function(arg) // arg is linear and not copyable

// should not be able to use arg here, even if function parameter is unowned

// Case 2: Normal function callfunction(arg) // arg is linear and not copyable

// arg released (if parameter is unowned), should be able to use it

Slice expressions

The slice expression A[L:H:M] (where L - low, H - high, M - maximum (capacity of theslice) are optional) is simply evaluated left to right. For the purpose of this definition,let us assume that they all are specified. If either are unspecified, their owner would beNoOwner, their dependencies empty, and the store identical to the left-hand-side store.

Therefore, if:

〈A, s〉 (Pa, oa, da, sa)〈L, sa〉 (Pl, ol, dl, sl) and r ∈ base(Pl)〈H, sl〉 (Ph, oh, dh, sh) and r ∈ base(Ph)〈M, sm〉 (Pm, om, dm, sm) and r ∈ base(Pm)

46

Then, f Pa is an array (has the form a [_]E), the result becomes a slice of Es:

〈A[L : H : M ], s〉 (om[]E, om, dm, sm[= dl ∪ dh ∪ dm ∪ {ol, oh, om}])

Otherwise, if Pa is a slice already, it stays one:

〈A[L : H : M ], s〉 (Pa, om, dm, sm[= dl ∪ dh ∪ dm ∪ {ol, oh, om}])

This is overly pessimistic: All owners and dependencies are only released at the end,they could be released earlier.

Selector expression

The selector expression E.I, where E is an expression and I is an identifier seems simple.But Go allows embedding structs in others and accessing the fields of embedded structswithout explicitly referencing the embedded struct:type T struct { x int }type U struct { T }

var u U

// u.x is u.T.x

Hence any such expression actually has to traverse a path.

Go defines three types of selections:

1. FieldVal, field values - a value indexed by a field name2. MethodVal, method values - a value indexed by a method name3. MethodExpr, method expressions - a type indexed by a method name

The type of selection really only applies to the last element in the path though: Whenselecting a path like u.T.x above, T must be a struct - a function (which the other caseswould produce) cannot be selected.

It also provides a library function to translate a selection expression to a path ofintegers, referencing the fields / methods in the object that is being selected from. Letthis function produce a path indexi (0 ≤ i < n) and selectionKind, one of the threeselection kinds.

First of all, if we can evaluate the statement:

〈E, s〉 (P−1, o−1, d−1, s−1)

And next, can, for each step i ≥ 0 in the path, perform a selection:

47

(Pi, oi, di, si) : = selectOne(si−1, Pi−1, indexi, kindi, oi−1, di−1)

where kindi ={selectionKind if i == n− 1FieldVal else (as explained before)

and selectOne() will be defined later

Then:〈E.I, s〉 (Pn−1, on−1, dn−1, sn−1)

Now, selectOne is tricky. Field values are simple, with one complication: Pointers arehandled transparently - selecting a field in a pointer selects a field in the struct, as thelanguage requires that.

selectOne(s, a ∗A, idx, FieldVal, o, d):= selectOne(s,A, idx, FieldVal, o, d)

selectOne(s, a struct {A0, . . . , An}, idx, FieldVal, o, d):= (Aidx, o, d, s)

Method expressions simply find the method, and prepend the receiver permission tothe parameter permissions

selectOne(s, a interface {A0, . . . , An}, idx, MethodExpr, o, d):= (recvToParams(Aidx), NoOwner, ∅, s[= d ∩ {o}])

where recvToParams(a(R) func (P0, . . . , Pn)(R0, . . . , Rr)):= a func (R,P0, . . . , Pn)(R0, . . . , Rr)

For method values, we reuse the moc function defined earlier to move or copy the lhsinto the receiver. If we are binding an unowned receiver, the bound method value willbe unowned too, to ensure we do not store an unowned value in an owned functionvalue, as they have different lifetimes.

selectOne(s,=In︷ ︸︸ ︷

a interface {A0, . . . , An}, idx, MethodVal, o, d):= (stripRecv(maybeUnowned(Aidx)), o′, d′, s′)

where stripRecv(a(R) func (P0, . . . , Pn)(R0, . . . , Rr)):= a func (P0, . . . , Pn)(R0, . . . , Rr)maybeUnowned(In,R)

:={In if o ∈ base(R)ctb(In, base(In) \ {o}) if o 6∈ base(R)

(s′, o′, d′) := moc(s, In,R, o, d)

48

Method values and method expressions also exist on named types, but named typesdo not have an equivalent permission yet, and therefore, they are not implementedfor these types. Introducing a permission equivalent to named types would havecaused significant changes in the abstract interpreter and it was too late for that.An alternative would have been to associate method sets from named types with thepermissions for their underlying types (for example, a pointer to struct). It is unclearhow they would have to be handled on assignments, though. Can they be just ignored?More work is needed here.

Composite Literals

A composite literal T {E_0, ..., E_n} is fairly similar to a function calls. All valuesare moved or copied to their relevant position in the generated struct. However,dependencies are not released. There naturally is no owner, since the result is not partof a value reached from a variable.

There is one complication: The individual expressions may also be key-value expressions.Let’s just assume that we have two functions index(Ei) returning the index in thegenerated struct (which is i if it is not a key-value expression, and value(Ei) representingthe permission of the value of the expression (which is just Ei if it is not a key-valueexpression). These functions can be trivially implemented by looking at the typeinformation provided by the go/types package.

The final algorithm looks like this:

1. Evaluate the type2. Evaluate each argument, and move it to the corresponding position in the struct3. Return the type’s permission, with dependencies collected from the parameters.

〈T, s〉 (=P︷ ︸︸ ︷

a struct {A0, . . . , Ak}, o−1, d−1, s−1) k ≥ n〈value(Ei), s′i−1〉 (Pi, oi, di, si)

〈T{E0, . . . , En}, s〉

P,NoOwner, i≤n⋃i=0

d′i ∪ {o′i}, s′n

where (s′−1, o

′−1d

′−1) = (s−1[= d−1 ∪ {o−1}], NoOwner, ∅)

(s′i, o′i, d′i) = moc(si, Pi, Aindex(Ei), oi, di)

An alternative approach that was considered is to construct the struct permissionsimply using the arguments, without the type info - just evaluate each argumentand make a list of their permissions, essentially. This fails in two ways however: Acomposite literal may be incomplete, so the struct would have less fields then expected,and would thus not be assignable to the permission for the real type. Also, key-value

49

expressions could not be handled without type info: It would not be clear where toput the permissions in the struct permission.

Function literals

Function literals will be handled in the statement section, since they require a lot ofknowledge about statements.

4.4 Statements

Statements are slightly more complex than expressions, (1) because they allow intro-ducing new variable bindings in the current scope; and (2) because they allow jumping:There can be return, goto, break, fallthrough and all other kinds of statements inthere.

There are two common ways to handle “early” exits in an interpreter:

1. Raise an exception (for example, a ContinueException, or BreakException)2. Return a value for a block statement describing where the statement exited, and

pass that through

In an abstract interpreter, option 1 does not work - there may be multiple exits involved;some leaving the block normally (by falling out of it), some with a branching statement.The second option is applicable, with the change that instead of returning one valuewe return multiple ones. Each statement visitor returns pairs of

1. a new store, with the changes the statement made2. an indicator of how the block was left (in this implementation, it is either nil or

a pointer to the ReturnStmt or BranchStmt (goto, break, continue, falltrough))

Most statements return just one such pair, but if control flow is involved, there mightbe multiple, representing the individual paths.

Evaluating statements also needs access to the current function’s permission. As such,we evaluate triplets of statement, store, and a function permission. The followingsections refer to the statement evaluation function as

→: Stmt× Store× FuncPermission︸ ︷︷ ︸active function

→ set of ( Store︸ ︷︷ ︸resulting store

× (Stmt ∪ {nil})︸ ︷︷ ︸jump/return, if any

).

Some parts (assignments, blocks, and loops) are not easily to define in such a formalnotation. They are explained as Go functions in literate programming style.

50

Simple statements

An expression statement simply evaluates the expressions, releases owners and depen-dencies afterwards, and returns the new store yielded by the expression.

〈E, s〉 (P, o, d, s′)〈ExprStmt(E), s, f〉 → {(s′[= d ∪ {o}], nil)} (P-ExprStmt)

An increase/decrease statement like E++ needs read and write permissions for theexpression E. It evaluates E, then releases the owner and dependencies.

〈E, s〉 (P, o, d, s′) r, w ∈ base(E)〈E + +, s, f〉, 〈E −−, s, f〉 → {(s′[= d ∪ {o}], nil)} (P-IncDecStmt)

A labeled statement x: S for a statement S just evaluates S.

〈S, s〉 → X

〈x : S, s, f〉 → X for all label names x (P-LabeledStmt)

An empty statement does nothing.

〈, s, f〉 → {(s, nil)} (P-EmptyStmt)

Assignments and declarations

There are essentially two forms of assignment and declarations:

1. Assign statements: a := b and a = b (the former defines the variable if notdefined in the current block)

2. Declaration statement: var a = b, var a (the latter creates zero values)

Both of them share most of the implementation in the form of two functions:defineOrAssign which handles a single LHS expression1 and a single RHS permission,and defineOrAssignMany which takes care of defining or assigning multiple (or zero)RHS to one or more LHS values.

The function defineOrAssign is the core function responsible for evaluating definitionsand assignment. The function starts by checking that the left-hand side is an identifierand defining it as necessary (or, if the identifier is _, by returning). Afterwards itevaluates the left-hand side (which is now defined), and then performs a move-or-copyfrom the right permission to the left.

1While the abstract syntax tree just has expressions in general on the left-hand side of an assignment,only certain expressions are allowed in practice (variables, indexing, field access, pointer dereference,wildcard).

51

func (i *Interpreter) defineOrAssign(st Store, stmt ast.Stmt, lhs ast.Expr,rhs permission.Permission, owner Owner, deps []Borrowed, isDefine bool,allowUnowned bool) (Store, Owner, []Borrowed) {var err error

<<define or assign lhs>>

// Ensure we can do the assignment. If the left-hand side is anidentifier, this should always be// true - it is either Defined to the same value, or set to somethingless than it in the previous block.

perm, _, _ := i.visitExprOwnerToDeps(st, lhs) // We just need to knowpermission, do not care about borrowing.if !allowUnowned {

i.Assert(lhs, perm, permission.Owned) // Make sure it is owned, so wedo not move unowned to it.}

// Input deps are nil, so we can ignore them here.st, owner, deps, err = i.moveOrCopy(lhs, st, rhs, perm, owner, deps)if err != nil {

i.Error(lhs, "Could␣not␣assign␣or␣define:␣%s", err)}

log.Println("Assigned", lhs, "in", st)

return st, owner, deps}

The code handling defining or assigning the permission of the lhs first handles theunderscore case, and then handles the define or assign:<<define or assign lhs>>=// Define or set the effective permission of the left-hand side to the right-

hand side. In the latter case,// the effective permission will be restricted by the specified maximum (

initial) permission.if ident, ok := lhs.(*ast.Ident); ok {

<<handle _>>

if isDefine {<<define value>>

} else {<<set value>>

}

if err != nil {i.Error(lhs, "Could␣not␣assign␣or␣define:␣%s", err)

52

}} else if isDefine {

i.Error(lhs, "Cannot␣define:␣Left-hand␣side␣is␣not␣an␣identifier")}

Handling the underscore case is simple: An assignment to _ would be equivalent tojust executing the expression on the right-hand side, that is, we can release owner anddependencies - it is not going to be moved anywhere.<<handle _>>=if ident.Name == "_" {

i.Release(stmt, st, []Borrowed{Borrowed(owner)})i.Release(stmt, st, deps)return st, NoOwner, nil

}

If we are evaluating a define statement, an annotation may be present. In that case,the actual RHS permission is converted (recall section 3.3 on page 24 and section 3.4on page 31) to the annotated permission to create the LHS permission. Otherwise,the LHS permission is the RHS permission (possibly subject to limits if the variable isalready defined and we are in fact reassigning).<<define value>>=log.Println("Defining", ident.Name)if ann, ok := i.AnnotatedPermissions[ident]; ok {

if ann, err = permission.ConvertTo(rhs, ann); err != nil {st, err = st.Define(ident.Name, ann)

}} else {

st, err = st.Define(ident.Name, rhs)}

Otherwise, when assigning rather than defining, we just set the effective permission toeither the maximum permission the variable can hold (if it can be copied to it) or tothe RHS permission (limited by the maximum permission, see section 4.2 on page 38).The maximum case allows us to add permissions when copying, which is a propertycopying was designed to have (see section 3.2 on page 21).<<set value>>=if permission.CopyableTo(rhs, st.GetMaximum(ident.Name)) {

st, err = st.SetEffective(ident.Name, st.GetMaximum(ident.Name))} else {

st, err = st.SetEffective(ident.Name, rhs)}

The function defineOrAssign allows us to do 1:1 definitions and assignments, that is,cases where we have one value and one variable. There are more cases though: Tuples

53

representing multiple return values can be unpacked, and there might simply be novalues at all when defining (which would create zero values for their respective type,like null for a pointer type).

The function defineOrAssignMany takes care of that. It takes multiple LHS expressionsand multiple RHS expressions, and then unpacks the RHS expressions into a list ofpermissions and a list of dependencies. If there are less RHS expressions than LHSexpressions, the missing ones are substituted with permissions for zero values, so wecan handle cases like var x int where no values are specified. Finally, defineOrAssignis called for each pair of LHS expression and RHS permissions.func (i *Interpreter) defineOrAssignMany(st Store, stmt ast.Stmt, lhsExprs []

ast.Expr, rhsExprs []ast.Expr, isDefine bool, allowUnowned bool) Store {var deps []Borrowedvar rhs []permission.Permissionif len(rhsExprs) == 1 && len(lhsExprs) > 1 {

<<unpack tuple>>} else {

<<unpack multiple>>}

<<fill rhs up with zero values>>if len(rhs) != len(lhsExprs) {

i.Error(stmt, "Expected␣same␣number␣of␣arguments␣on␣both␣sides␣of␣assignment␣(or␣one␣function␣call␣on␣the␣right):␣Got␣rhs=%d␣lhs=%d", len(rhs), len(lhsExprs))}

for j, lhs := range lhsExprs {st, _, _ = i.defineOrAssign(st, stmt, lhs, rhs[j], NoOwner, nil,

isDefine, allowUnowned)}

st = i.Release(stmt, st, deps)

return st}

Unpacking a tuple is simple: We evaluate the single RHS expression, and then take thetuple elements as the RHS dependencies. Since a tuple is the result of a function call(you might have noticed that is the only evaluation that can cause a tuple permissionto be created) we do not really need to care about owners or dependencies, so we justcollect them for releasing later.<<unpack tuple>>=// These really cannot have owners.rhs0, rdeps, store := i.visitExprOwnerToDeps(st, rhsExprs[0])st = storetuple, ok := rhs0.(*permission.TuplePermission)

54

if !ok {i.Error(stmt, "Left␣side␣of␣assignment␣has␣more␣than␣one␣element␣but␣right␣hand␣only␣one,␣expected␣it␣to␣be␣a␣tuple")

}deps = append(deps, rdeps...)rhs = tuple.Elements

When unpacking multiple RHS expressions, the idea is that all RHS values are asssignedto a temporary variable, and the temporary variables are assigned to the final oneslater. The reason for that is that an operation like a, b = b, a – or (a,b) = (b,a)in languages with more syntax – swaps the variables, rather than making both be bafterwards.<<unpack multiple>>=for _, expr := range rhsExprs {

log.Printf("Visiting␣expr␣%#v␣in␣store␣%v", expr, st)perm, ownerThis, depsThis, store := i.VisitExpr(st, expr)log.Printf("Visited␣expr␣%#v␣in␣store␣%v", expr, st)st = storerhs = append(rhs, perm)

// Screw this. This is basically creating a temporary copy or (non-temporary, really) move of the values, so we// can have stuff like a, b = b, a without it messing up.store, ownerThis, depsThis, err := i.moveOrCopy(expr, st, perm, perm,ownerThis, depsThis)st = store

if err != nil {i.Error(expr, "Could␣not␣move␣value:␣%s", err)

}

deps = append(deps, Borrowed(ownerThis))deps = append(deps, depsThis...)

}

Finally, we fill up missing elements on the right with default permissions for their type- these will be zero values, as mentioned before.<<fill rhs up with zero values>>=// Fill up the RHS with zero values if it has less elements than the LHS.

Used for var x, y int; for example.for elem := len(rhs); elem < len(lhsExprs); elem++ {

var perm permission.Permission

perm = i.typeMapper.NewFromType(i.typesInfo.TypeOf(lhsExprs[elem]))perm = permission.ConvertToBase(perm, perm.GetBasePermission()|permission.Owned)

55

rhs = append(rhs, perm)

}

Now, as for the actual statements:

Assign and define (= and :=) are equivalent to defineOrAssignMany. They set theisDefine parameter to true if the token of the statement was :=, otherwise it is false.Unowned values are not allowed.func (i *Interpreter) visitAssignStmt(st Store, stmt *ast.AssignStmt) []

StmtExit {return []StmtExit{{i.defineOrAssignMany(st, stmt, stmt.Lhs, stmt.Rhs,stmt.Tok == token.DEFINE, false), nil}}

}

Declaration statements of the form var x, y = a, b or just var x, y are more compli-cated. The declaration has a list of specifications, and each specification has a list ofnames and values. We therefore need to call defineOrAssignMany once per specification(there could also be other specifications). Unowned values are not allowed, and isDefineis always true.func (i *Interpreter) visitDeclStmt(st Store, stmt *ast.DeclStmt) []StmtExit

{<<boring setup code>>

for _, spec := range decl.Specs {switch spec := spec.(type) {case *ast.ValueSpec:

names := <<convert spec.Names into a slice of expressions>>st = i.defineOrAssignMany(st, stmt, names, spec.Values, true,

false)default:

continue}

}return []StmtExit{{st, nil}}

}

Special function calls: Go / Deferred

The go and defered statements are fairly simple. They consist of a call expressionwhich is evaluated in go/defer mode. If you recall the definition of function calls fromsection 4.3 on page 46, you will remember that the dependencies of a deferred call arethe dependencies of the function.

56

Now, there is one difference between go and defer calls: While neither release theirowner, a defer call releases unowned dependencies. As we have established before insection 4.3 on page 46, the dependencies of a go or defer call are the dependencies ofthe function.

This has one important effect: If the function called is a function literal, the dependencieswill correspond to the captured variables. We can thus continue to work with unownedparameters in the function containing the defer statement, even if the deferred call’sliteral captures the parameter.// @perm func(m * m)function fooDefer(bar *int) {

defer function() {... bar ... // captured

}... bar ... // still works

}// @perm func(m * m)function fooGo(bar *int) {

go function() {... bar ... // captured

}... bar ... // does not work: bar is unusable after the go statement

}

This is legal because any unowned parameter will still be usable when the functionexits.

〈E(A0, . . . , An), s〉 go/defer (P, o, d, s′)〈go E(A0, . . . , An), s, f〉 → {(s′, nil)} (P-GoStmt)

〈E(A0, . . . , An), s〉 go/defer (P, o, d, s′)〈defer E(A0, . . . , An), s, f〉 → {(s′[= {(v, p) ∈ d|o 6∈ p}], nil)} (P-DeferStmt)

Branch and return statements

A branch statement is one of the statements break, continue, goto, or fallthrough.Evaluating it has no effect on the store, but the statement is the statement in theresult pair of →.

〈B, s, f〉 → {(s,B)} for all branch statements B (P-BranchStmt)

A return statement is more complex: It might have 0 or more arguments, correspondingto the number of return values of the current function. Each argument is moved-or-

57

copied into the return “position” before the next argument is evaluated. The result isone pair: the store after evaluating the last argument, and the return statement.

〈Ai, si−1〉 (Pi, o′i, d′i, s′i) si, oi, di = moc(s′i, Pi, receivers(f)i, o′i, d′i)〈return A0, . . . , An︸ ︷︷ ︸

=B

, s−1, f〉 → {(sn, B)} (P-ReturnStmt)

Block statements

A block is generally evaluated like this: First begin a new block scope, then evaluatethe statement list, and finally remove the block scope from all exits:

(s[+], [S0, . . . , Sn], false) visitStmtList−−−−−−−−→ exits

〈{S0; . . . ;Sn}, s, f〉 → {(s[−], b)|(s, b) ∈ exits} (P-BlockStmt)

Now, interpreting the list of statements is difficult: The presence of goto statementscan cause us to jump within the block, and break, continue, or return statements canjump out of the block. Instead of just iterating the statements in the block, we maintaina work stack and a seen set, each containing pairs of store and position (we call thatthe block manager. The block manager uses the seen set to determine if it should addwork to the work stack, it will only add work it has not seen yet). As mentioned before,formal definition of visitStmtList will not be provided, only a literate programmingstyle one.

The execution of blocks, switches, and loops all follow the same approach: First somesetup is performed, and then, while there is work, the statement refered to by thework item is executed (after some pre and before some post code). One example isvisitStmtList which handles non-loop lists of statements (block bodies, list of caseclauses in switches and select statements):func (i *Interpreter) visitStmtList(initStore Store, stmts []ast.Stmt,

isASwitch bool) []StmtExit {var bm blockManager // A helper type providing helper stuff

<<setup>>for bm.hasWork() {

item := bm.nextWork() // pop a work item from the stack<<pre>>exits := i.visitStmt(work.Store, <<stmt>>) // interpret the statement<<post>>

}

return bm.exits}

58

The <<post>> part usually splits the list of statement exits into exits of the block we areevaluating and further work, which gets added to the work stack (if not seen already).This means that it will evaluate the block for all possible inputs: For example, if weare evaluating a block with a conditional jump back to the beginning, it will followthe jump back and evaluate the block again for the state of the store before the jump.Since only unseen entries are added, the code will also terminate eventually.

Details:

The <<setup>> part just exits with the initial store if it there are no statements toexecute. If it is a switch/select statement, the entry point is added as an exit, and eachstatement (which corrrespond to case clauses) is added to the work state. If it is just anormal block, we enter statement 0 in the block. We also collect all labels, building amap of strings to indices in the statement list:if len(stmts) == 0 {

return []StmtExit{{initStore, nil}}} else if isASwitch {

bm.addExit(StmtExit{initStore, nil})for i := range stmts {

bm.addWork(work{initStore, i})}

} else {bm.addWork(work{initStore, 0})

}labels := collectLabels(stmts)

The <<pre>> part is empty.

The <<stmt>> is stmts[item.int], that is, the statement in the list of statements atthe index stored in the work item.

The <<post>> part is exciting. Each exit of the evaluated statement (inner exit) eitherbecomes additional work or an exit of the current block. It depends on what the branchstatement is:

switch branch := exit.branch.(type) {<<case 1>><<case 2>><<case 3>>

}

The cases are:

1. If the inner exit contains no branch or return statement, the next statement isadded as work if we are not in a switch/select and there is at least one morestatement in the list. Otherwise, the inner exit becomes an exit.case nil:

59

if len(stmts) > item.int+1 && !isASwitch {bm.addWork(work{exit.Store, item.int + 1})

} else {bm.addExit(StmtExit{exit.Store, nil})

}

2. If the inner exit contains a return statement, it becomes an exit of the currentblock:case *ast.ReturnStmt:

bm.addExit(exit) // Always exits the block

3. If the inner exit contains a branch statement:

1. We are in a switch, it is a break, and the break applies to the current branch:Add the inner exit (without the branch statement) as an exit.

2. We are in a switch, it is a fallthrough: The next case clause is added aswork.

3. It is a goto: If the target is in the current block, add it as work; otherwise,add the inner exit as an exit

4. Otherwise, add the inner exit as an exit.case *ast.BranchStmt:

branchingThis := (branch.Label == nil || branch.Label.Name == "" /*| TODO current label */)switch {case isASwitch && branch.Tok == token.BREAK && branchingThis:

bm.addExit(StmtExit{exit.Store, nil})case isASwitch && branch.Tok == token.FALLTHROUGH:

bm.addWork(work{exit.Store, item.int + 1})case branch.Tok == token.GOTO:

if target, ok := labels[branch.Label.Name]; ok {bm.addWork(work{exit.Store, target})

} else {bm.addExit(exit)

}default:

bm.addExit(exit)}

The for loop

The evaluation of loops, either for or range statementsfor x := 0; x < 5; x++ { ... }for key, val := range something { ... }

60

works the same way as visitStmtList from blocks, just with different <<setup>>,<<pre>> and <<post>> phases.

For the for statement, the <<setup>> consists of beginning a new block, and thenvisiting the initializing statement (x := 0 in the example above).

initStore = initStore.BeginBlock()

// Evaluate the container specified on the right-hand side.for _, entry := range i.visitStmt(initStore, stmt.Init) {

if entry.branch != nil {i.Error(stmt.Init, "Initializer␣exits␣uncleanly")

}bm.addWork(work{entry.Store, 0})

}

The <<pre>> part consists of evaluating the condition in the store of the iteration,releasing its owner and dependencies (visitExprOwnerToDeps simply merges owner intodependencies), and adding the state after condition evaluation as an exit.

// Check conditionperm, deps, st := i.visitExprOwnerToDeps(st, stmt.Cond)i.Assert(stmt.Cond, perm, permission.Read)st = i.Release(stmt.Cond, st, deps)// There might be no more items, exitbm.addExit(StmtExit{st, nil})

The <<stmt>> that will be evaluated is always stmt.Body, the body of the for loop. Theposition stored in the work item is thus not used at all.

The <<post>> part consists of classifying exits of the loop body into more iterationsand direct exits, running the post statement for each next iteration in its store, andadding the direct exits to the loop’s list of exits, after ending the block.

nextIterations, exits := i.collectLoopExits(exits)for _, nextIter := range nextIterations {

for _, nextExit := range i.visitStmt(nextIter.Store, stmt.Post) {if nextExit.branch != nil {

i.Error(stmt.Init, "Post␣exits␣uncleanly")}bm.addWork(work{nextExit.Store, 0})

}}i.endBlocks(exits)bm.addExit(exits...)

The collectLoopExits function splits the exits of a loop’s body into further iterationsand exits of a loop.

61

func (i *Interpreter) collectLoopExits(exits []StmtExit) ([]work, []StmtExit){

var nextIterations []workvar realExits []StmtExit

for _, exit := range exits {switch branch := exit.branch.(type) {

<<case 1>><<case 2>><<case 3>>

}}

return nextIterations, realExits}

The cases are:

1. No branch statement: The exit is another iterationcase nil:

nextIterations = append(nextIterations, work{exit.Store, 0})

2. A return statement: The exit is a real exitcase *ast.ReturnStmt:

realExits = append(realExits, exit)

3. A branch statement. A break of the current loop is an exit, a continue of thecurrent loop a next iteration, anything else an exit.case *ast.BranchStmt:

branchingThis := branch.Label == nil || branch.Label.Name == "" /* |TODO current label */

switch {case branch.Tok == token.BREAK && branchingThis:

realExits = append(realExits, StmtExit{exit.Store, nil})case branch.Tok == token.CONTINUE && branchingThis:

nextIterations = append(nextIterations, work{exit.Store, 0})default:

realExits = append(realExits, exit)}

The range loop

The range statement is extremely similar to the for statement. It shares thecollectLoopExits function, but there are a few differences related to it assigning values

62

from a container. Notably, the block begins new for each iteration, and is released forall exits in the statement.

The <<setup>> part adds an exit for not entering the range (which will not evaluatethe right-hand side at all, which might be wrong). It then evaluates the right-handside, and defers the release of its dependencies until the end of the block. Finally, thepermissions for key and values are extraced from the RHS, and the initial work isadded.bm.addExit(StmtExit{initStore, nil})

// Evaluate the container specified on the right-hand side.perm, deps, initStore := i.visitExprOwnerToDeps(initStore, stmt.X)defer func() {

if canRelease {for j := range rangeExits {

rangeExits[j].Store = i.Release(stmt, rangeExits[j].Store, deps)}

}}()i.Assert(stmt.X, perm, permission.Read)

var rkey permission.Permissionvar rval permission.Permission

<<rhs permissions>>

bm.addWork(work{initStore, 0})

where <<rhs permissions>> determines the permissions for the key and value of whatis being iterated:switch perm := perm.(type) {case *permission.ArrayPermission:

rkey = permission.Mutablerval = perm.ElementPermission

case *permission.SlicePermission:rkey = permission.Mutablerval = perm.ElementPermission

case *permission.MapPermission:rkey = perm.KeyPermissionrval = perm.ValuePermission

}

The <<pre>> part begins a new block, and then uses defineOrAssign to define or assign(depending on whether it is a for k, v := range or a for k, v = range loop) thevariables representing keys and values. This also modifies a canRelease variable that isused in the function deferred in <<setup>> - basically, if the LHS are owned, they arebound indefinitely, and we cannot release the container we are iterating over later.

63

st = st.BeginBlock()if stmt.Key != nil {

st, _, _ = i.defineOrAssign(st, stmt, stmt.Key, rkey, NoOwner, nil, stmt.Tok == token.DEFINE, stmt.Tok == token.DEFINE)if ident, ok := stmt.Key.(*ast.Ident); ok {

log.Printf("Defined␣%s␣to␣%s", ident.Name, st.GetEffective(ident.Name))

if ident.Name != "_" {canRelease = canRelease && (st.GetEffective(ident.Name).

GetBasePermission()&permission.Owned == 0)}

} else {canRelease = false

}}if stmt.Value != nil {

st, _, _ = i.defineOrAssign(st, stmt, stmt.Value, rval, NoOwner, nil,stmt.Tok == token.DEFINE, stmt.Tok == token.DEFINE)if ident, ok := stmt.Value.(*ast.Ident); ok {

log.Printf("Defined␣%s␣to␣%s", ident.Name, st.GetEffective(ident.Name))

if ident.Name != "_" {canRelease = canRelease && (st.GetEffective(ident.Name).

GetBasePermission()&permission.Owned == 0)}

} else {canRelease = false

}}

As with for, the <<stmt>> is just the loop’s body: stmt.Body.

The <<post>> step is shorter than for for loops, as there are no post statements. We canimmediately end the block we began in <<pre>> (meaning the key and value variablesare gone again), collect the exits. One difference here is that every next iteration isalso a valid exit, since we might have just “reached” the end of the container.i.endBlocks(exits)nextIterations, exits := i.collectLoopExits(exits)bm.addExit(exits...)// Each next iteration is also possible work. This might generate duplicate

exits, but we have// to do it this way, as we might otherwise miss some exitsfor _, iter := range nextIterations {

bm.addExit(StmtExit{iter.Store, nil})}bm.addWork(nextIterations...)

64

If statements and switches

An if statement consists of an initializer, a condition, a body, and an else case:if foo := 5; foo > bar {

then} else ... // the ... is a statement as well. An else is optional

An if statement is evaluated as follows:

1. Create a new block2. Evaluate the initializing statement. It must produce one exit (the syntax does

not allow more, this simplifies thing)3. Evaluate the condition expression. It must be readable, and its owner and

dependencies are released.4. Evaluate body and else part in the store that resulted from evaluating the

condition.5. Combine the exits of body and else to form the result, but remove the block we

started at the beginning from their stores.

〈S0, s[+]〉 → {(s0, b0)} 〈E1, s0〉 (P1, o1, d1, s1) r ∈ P1

〈S2, s2, f〉 → exitsthen 〈S3, s2, f〉 → exitselse

〈if S0;E1S2S3, s, f〉 → {(s[−], e)|(s, e) ∈ exitsthen ∪ exitselse}

(P-IfStmt)

where s2 = s1[= d1 ∪ {o1}]

A switch statement switch INIT; TAG { BODY }, where BODY is a list of case clauses, isevaluated like this:

1. Create a new scoping block2. Evaluate the initializing statement.3. For each possible exit: Evaluate the tag in the exit’s store, and the body (with the

cases being not entered, and entered for each case clause) in the store resultingfrom evaluating the tag.

4. For each exit of the body: Release the dependencies of the tag, and undo theblock from the start.

〈INIT, s[+]〉 → {(s0, b0), . . . , (sn, bn)}〈TAG, si〉 (Pi, oi, di, s′i) ∀0 ≤ i ≤ n, r ∈ Pi

(s′i, BODY, true)visitStmtList−−−−−−−−→ exits′i exitsi := {(s[= di ∪ {oi}][−], e)|(s, e) ∈ exits′i}〈switch INIT ;TAG{BODY }, s, f〉 →

⋃0≤i≤n

exitsi

(P-SwitchStmt)

65

Listing 4.3: Abstract interpreter for switch statementsfunc (i *Interpreter) visitSwitchStmt(st Store, stmt *ast.SwitchStmt) []

StmtExit {var exits []StmtExit

st = st.BeginBlock()for _, exit := range i.visitStmt(st, stmt.Init) {

st := exit.Storeperm, deps, st := i.visitExprOwnerToDeps(st, stmt.Tag)if stmt.Tag != nil {

i.Assert(stmt.Tag, perm, permission.Read)}

for _, exit := range i.visitStmtList(st, stmt.Body.List, true) {exit.Store = i.Release(stmt.Tag, exit.Store, deps)exits = append(exits, exit)

}}

for i := range exits {exits[i].Store = exits[i].Store.EndBlock()

}return exits

}

It might be more readable in code form, as shown in listing 4.3.

A case clause has the form case E_0, ..., E_n: BODY for some expression E and somestatement BODY. The expressions are evaluated from left to right and the resultingstores are merged. Then the body is executed in the resulting store. Alternatively, thebody could be executed for each store, but executing it once should be faster thanexecuting it n+ 1 times.

〈Ei, si−1〉 (P, o′i, d′i, s′i) r ∈ P si = s′i[= d′i ∪ {o′i}]( ⋂0≤i≤n

si, BODY, false

)visitStmtList−−−−−−−−→ exits

〈case E0, . . . , En : BODY, s, f〉 → exits

(P-CaseClause)

Channel statements

The easiest channel statement is the send statement. Given a writable channel, and anobject, we move-or-copy the object permission into the channel element permission.Any dependency and owner left over by moc are then released.

66

〈E0, s〉 (c chan C, o0, d0, s0)〈E1, s〉 (V, o′1, d′1, s′1) s1, o1, d1 := moc(s′1, V, C, o′1, d′1)

〈E0 <- E1, s, f〉 → {(s1[= d1 ∪ {o1}], nil)}

(P-SendStmt)

There is arguably a bug here. We can send to a unowned channel containing unownedvalues, and the dependencies would be released. A channel containing unowned valuesshould not exist in the first place, though, but sadly, it is entirely possible to defineone at the moment.

The select statement allows waiting for non-blocking send/receive availability on a setof channels. It is thus similar to (and probably named after) the select(2) system callfrom 4.2BSD and POSIX.1-2001. The resulting exits will be the initial store (for thenot-entered case), and those gained by jumping into each of the comm clauses in thebody from the store (which is what visitStmtList does).

(s[+], BODY, true) visitStmtList−−−−−−−−→ exits

〈select {BODY }, s, f〉 → ∪{(s[−], e)for(s, e) ∈ exits})} (P-SelectStmt)

A comm clause has the form case STMT: BODY for some statements STMT and BODY.First the statement is evaluated, and then the body is evaluated for each exit of thestatement.

〈STMT, s, f〉 → exits0 exits :=⋃

e∈exits0visitStmtList(store(e), BODY, false)

〈case STMT : BODY, s, f〉 → exits

(P-CommClause)

Back to expressions: The function literal

Now that we have introduced statements, it’s time to come back to the function literalexpression which we deferred to a later point at the end of the expression section. Afunction literal looks like a function, but without a name.

The first step is to get the type of the function literal to look up things like parameternames, swapping the currently active function (represented by the third element in theformal notation), and then calling a helper function that builds a function.func (i *Interpreter) visitFuncLit(st Store, e *ast.FuncLit) (permission.

Permission, Owner, []Borrowed, Store) {oldCurFunc := i.curFuncif i.typeMapper == nil {

67

i.typeMapper = permission.NewTypeMapper()}typ := i.typesInfo.TypeOf(e).(*types.Signature)i.curFunc = i.typeMapper.NewFromType(typ).(*permission.FuncPermission)defer func() {

i.curFunc = oldCurFunc}()return i.buildFunction(st, e, typ, e.Body)

}

The helper function buildFunction has been extracted because it will also be usefulwith function declarations in a package. It defines receivers and parameters, interpretsthe function body, and then collects all captured variables and moves them “into” theclosure, so they cannot be modified by the parent function. They are also added asdependencies, so the caller could still release them back - if we recall, that’s whathappens for unowned variables if a function literal is deferred.func (i *Interpreter) buildFunction(st Store, node ast.Node, typ *types.

Signature, body *ast.BlockStmt) (permission.Permission, Owner, []Borrowed, Store) {var deps []Borrowedvar err errororigStore := stperm := i.typeMapper.NewFromType(typ).(*permission.FuncPermission)perm.BasePermission |= permission.Owned

st = st.BeginBlock()<<define stuff>>

exits := i.visitStmtList(st, body.List, false)st = append(NewStore(), origStore...)for _, exit := range exits {

<<collect captures>>}

return perm, NoOwner, deps, st}

Defining receivers and parameters is a rather boring affair. This should actually alsodefine any named return values, but that’s not<<define stuff>>=if len(perm.Receivers) > 0 {

for _, recv := range perm.Receivers {st, err = st.Define(typ.Recv().Name(), recv)if err != nil {

i.Error(node, "Cannot␣define␣receiver␣%s", err)}

}

68

}

params := typ.Params()for j := 0; j < params.Len(); j++ {

param := params.At(j)st, err = st.Define(param.Name(), perm.Params[j])if err != nil {

i.Error(node, "Cannot␣define␣parameter␣%d␣called␣%s:␣%s", j,param.Name(), err)

}}

Collecting the captured variables is a rather simple affair. We just visit every variablein the exit store that has a higher use count than in the input store, and move it to thedependencies of the function permission. We also adjust the permission of the literal:If we capture an unowned variable, the function becomes unowned too so we have thesame narrower life time (since unowned objects cannot be stored).<<collect captures>>=

exit.Store = exit.Store.EndBlock()<<verify that store has same length as before>>

for j := range exit.Store {<<several safety checks>>if exit.Store[j].uses <= origStore[j].uses {

continue}log.Printf("Borrowing␣%s␣=␣%s", st[j].name, exit.Store[j].eff)deps = append(deps, Borrowed{ast.NewIdent(st[j].name), st[j].eff

})st[j].eff = permission.ConvertToBase(st[j].eff, 0)log.Printf("Borrowed␣%s␣is␣now␣%s", st[j].name, st[j].eff)st[j].uses = exit.Store[j].uses

if exit.Store[j].eff.GetBasePermission()&permission.Owned == 0 &&perm.GetBasePermission()&permission.Owned != 0 {

log.Printf("Converting␣function␣to␣unowned␣due␣to␣%s", exit.Store[j].name)

perm = permission.ConvertToBase(perm, perm.GetBasePermission()&^permission.Owned).(*permission.FuncPermission)

}}

The use counting actually happens in the store, but it is buggy: It only happens fordefinitions and assignments, not for read accesses (because the store is passed by value,and while we increase the count, that change is thus not visible outside the functiondoing the lookup).

69

5 Evaluation

Having discussed the design and implementation, the time has come to evaluate theproject according to our criteria from section 1 on page 5, that is:

Completeness All syntactic constructs are testedCorrectness Only valid programs are allowedPreciseness We do not reject useful programs because our rules are too broad.Usability If there is a problem, is it understandableCompatibility A compiling Lingo program behaves the same as a Go program with all

Lingo annotations are removedCoverage The implementation should be well-tested with unit tests

5.1 Completeness

The implementation is unfortunately not complete.

We only have support for checking expressions and statements (including functionliterals), but we do not have support for declaring global variables, functions, orimporting other packages. Adding support for global declarations requires handlingglobal state. While we could just create a “package” permission holding permissionsfor all objects declared in it, and make the current package a store, it is unclear howany mutable global state should interact with functions. There does not seem to be asafe solution for global mutable state: If two functions want to access the same global,mutable, variable, how do we handle that? Marking each function’s closure as mutableis not enough: Their closure is the same. Two solutions might be possible: Identifygroups of functions with the same closure and only allow one of them to be usedat a time, or just forbid two functions from using the same global mutable variable;essentially making global mutable state function-specific.

The support for expressions and statements is slightly incomplete as well: Type assertion,conversion between named types and interfaces, and type switches are missing. Theserequire gathering a set of methods for a given named type. But we do not have anequivalent to named types in permissions, and adding it does not seem feasible anymore,as it would require substantial changes in the interpreter. An alternative would beto simply attach a list of methods to the unnamed permissions, and when converting,take the left set of methods. When a conversion to an interface is required, we could

70

just build an interface out of these methods, and check if that interface is assignable tothe interface.

Also as mentioned in section 4.3 on page 43, strings can be indexed, but we do notsupport it - strings are just base permissions, but should be their own permission kind.This seems easily solvable however.

5.2 Correctness

The implementation is shallow: When looking at the requirements for operands, itonly looks at the base permission of the operands. So, for example, if an expressionrequires an operand to be readable, we only check whether its base permission containsthe r bit. That should be fine so far, as we ensure consistency at some point in theprogram by converting each permission to its own base permission, and thus a readableobject cannot have unreadable members. But it falls short if we actually want toallow it. There might be an option: Instead of checking if r ∈ base(A) we can check ifassmov(A, ctb(A, r)), that is, we create a new structure where all base permissions arereplaced with {r} and then we can check if the value is movable to it, which recursivelychecks that each base permission is movable to {r}.

As shown in section 4.3 on page 44, taking the address of a pointer does not modifythe store, except for the evaluation of the pointer, of course. As mentioned there, thisis wrong: Taking the address of a pointer should (usually) consume the maximumpermission of the owner. Let’s look at an example: Let’s say we have a variable athat is om * om, and we take a pointer to it: p = &a - p is now om * om * om. But acan be reassigned: a = a new pointer and regain its effective permission, meaning wenow have two usable references to a. This happens because assignment checks againstthe maximum permission (section 4.4 on page 53), and thus, taking the maximumpermission away instead of the effective permission would solve the issue.

5.3 Preciseness

The implementation is coarse: If I borrow anything uniquely referred to by a variable(for example, when freezing it, as in section 4 on page 43), then the entire variable ismarked as unusable, rather than just the part of the permission that was borrowed. Asolution to this problem would be to collect a path from a variable to an object (like,select field x, select field y), when evaluating an expression - then we could create anew permission where the borrowed part is replaced by an unusable permission. Butthis then leads to the shallowness problem mentioned above.

The implementation is ambiguous: Permissions for types and values are stored in thesame namespace. This is not a real problem, though, as Go ensures that we cannot

71

use type in value contexts and vice versa.

As seen in section 4.3 on page 40, moving a value out of the store also happens fornon-linear values. This is overly broad: If a value is not linear, it should be copyable.Therefore it might make sense to leave that out.

We have seen in section 4.3 on page 43 that indexing a map also moves the key intothe map (its dependencies are forgotten) if the map has unowned keys (the map isunowned). This might break some code that should actually work.

In section 4.3 on page 47 we saw that slicing only releases the permissions of theslice arguments after they have all been evaluated, which is overly coarse. It preventsa call like a[x:x] for some linear x. Instead, slicing should release each argument’sdependencies as soon as it has been evaluated - the result will be an integer and itsdependencies thus do not matter.

In section 3.2 on page 22 we saw that there should actually be two types of move oper-ations, and section 3.2 on page 24 showed that, because we only have the one definitionthat requires the source to be readable, functions with parameters of permission n cannot be assigned elsewhere. Also, section 3.2 on page 24 showed that we have similarproblems for pointers: A ol * om cannot be passed to function expecting an om * ompointer, although this would be harmless - both are linear pointers, and the target isthe same in both cases.

5.4 Usability

Usability is bad, really bad. The structured permissions are incredibly powerful, butthis power comes at a price: Error messages are not readable. There are two reasons forthat: First of all, there can be a lot of nesting and a lot of wide permissions (like structswith a lot of elements), leading to long and hard to read permissions. Secondly, therecan be cycles, and the cycles do not always appear at the same stage: For examplea permission “A = om * A” could be stored as A = om ∗ A or it could be stored asA′ = om ∗ om ∗ A - they are still compatible. This makes it hard to figure out theactual error when two permissions are incompatible.

5.5 Compatibility

On the semantic front, if a Lingo program compiles it will behave exactly like a Goprogram. This is a side effect of going with comment-based annotations and a simplechecker that does not generate a modified program.

On the actual use part, while interfacing with legacy code could be made possiblesimply by using n permissions for parameters, and om or orw for return values, this

72

seems a bit unsafe. Permissions should be annotated with an unsafe bit, and conversionsbetween unsafe permissions and safe permissions should produce a warning. Thereshould also be a way to annotate that a certain conversion is safe.

5.6 Code coverage / Unit testing

The implementation, since the beginning, has been subject to rigorous unit testingwith continuous integration on travis-ci.org1 and code coverage reports on codecov.io2.

Figure 5.1: code coverage chart

The code coverage chart shows that it started out slightly below 100% line coverage,eventually reaching 100%, only to drop again - when the interpreter started comingtogether - there are quite a few places in the interpreter code that are unreachableconditions and would require constructing a lot of illegal AST objects to test.

To be precise, the coverage for the permission package itself stayed at 100%, and thestore also has 100% coverage, the interpreter, however only has about 92% coverage.

Unfortunately, the Go tools only provide line coverage, and not branch or path coverage.This is somewhat problematic: For example, if we have an if statement without anelse part, we can test if the if has been taken, but we usually cannot check whether ithas not been taken: The if statement would eventually fall out of its block and backinto the parent block, and thus all lines are executed.

Like the code, this section is split in two parts: First, we will discuss how the permissionspackage was tested - the package includes the parser and the permission op erationsdiscussed in chapter 3. Afterwards, we will discuss testing the implementations of thestore and abstract interpreter described in chapter 4.

1https://travis-ci.org/julian-klode/lingolang2https://codecov.io/gh/julian-klode/lingolang

73

5.6.1 Testing the permissions package

The permissions package contains the parser and the rules for permissions described inthe section Permissions for Go. Coverage is 100%.

The parser

The first functional component to be introduced were the scanner and the parser, alongwith its test suite. Actually, only the parser has a test suite initially: It already testedmost of the scanner, except some error formatting code and the function to render atoken type as a string for an invalid token type value - but these were also fixed later,leading to 100% coverage for both of them.

The commits after that enabled continuous integration. Test cases in Go are usuallyconstructed in a table driven fashion: You define a table of test inputs and outputs,and a function iterating over them. In the case of the parser, the tests simply are amap from string in the permission syntax (listing 3.1) to the permission object thatshould have been parsed:

Listing 5.1: Parser test casesvar testCasesParser = map[string]Permission{

"123": nil, // ..."\xc2": nil, // incomplete rune at beginning"a\xc2": nil, // incomplete rune in word"oe": nil,"or": Owned | Read,// ... other combinations ..."n": None,"m␣[": nil, // ..."m␣[]␣a": &SlicePermission{

BasePermission: Mutable,ElementPermission: Any,

},"m␣[1]␣a": &ArrayPermission{

BasePermission: Mutable,ElementPermission: Any,

},// .. more tests ..."m␣struct␣{v;␣l}": &StructPermission{

BasePermission: Mutable,Fields: []Permission{

Value,LinearValue,

},},"_": &WildcardPermission{},

}

74

As can be seen, the map starts with some error cases, followed by base permissions,and then followed by more complex permissions (with error cases again). Due to theparser being recursive descent, we do not have to test every possible base permissioncombination at every level, but testing it once is enough, so the test for structuredpermissions can just pick any base permission and are still representative. Combinedwith 100% line coverage (and no short circuiting logic operators in the code), that meansthat we can parse all valid permissions. But the line coverage is also misleading: Thecode uses Expect(tokenType, tokenType, ...) and Accept(tokenType, tokenType), sothe error checking is not represented in additional lines, so there could of course beerrors lingering somewhere.

There are two slight issues with this specific table: First, a map has no order; andsecond, the tests have no names. The first is a bit annoying to deal with, especially ifyou have logging to the code being tested, but it is not a real problem: Even if testsfail, the function running the tests executes all of them:

Listing 5.2: Parser test runnerfunc TestParser(t *testing.T) {

for input, expected := range testCasesParser {input := inputexpected := expectedt.Run(input, func(t *testing.T) {

perm, err := NewParser(input).Parse()if !reflect.DeepEqual(perm, expected) {

t.Errorf("Input␣%s:␣Unexpected␣permission␣%v,␣expected␣%v␣-␣error:␣%v", input, perm, expected, err)

}})

}

For the names, t.Run() is passed input as the first argument - which determines thename of the subset being run. Unfortunately, Go replaces spaces with underscores,so searching for tests that failed is not that easy. Later tests for the interpreter havenames for the test cases, as they just got to complex to use the input.

Type mapping

The type mapper was a rather simple piece of code. It had 100% coverage as well, butit was written with some wrong assumptions. For some background: Basic types, likeinteger types, untyped constant types (untyped int, untyped float, untyped nil) arerepresented as one types.Basic type in Go’s go/types package, and were thus handledby a single case statement in the switch statement switching over the type of the typerepresentation, as seen in listing 5.3.

As can be seen, it assumed that basic types always map to base permissions, which isnormally true - but one basic type is untyped nil. So, untyped nil values received a

75

Listing 5.3: NewFromTypefunc (typeMapper TypeMapper) NewFromType(t0 types.Type) (result Permission) {

// ... insert some code handling recursion by using a map ...switch t := t0.(type) {case *types.Array:

return typeMapper.newFromArrayType(t)// ... other cases ...case *types.Basic:

return basicPermission // constant defined elsewhere

permission for atom types like integers, and thus could not be assigned to pointers,maps, and other nil-able values. This led to the nil permission being added, and ofcourse a regression test that ensures that an untyped nil basic type always translatesto an untyped nil permission.

Testing the other cases was mostly a matter of picking one specific example: Considerthe pointer:func (typeMapper TypeMapper) newFromPointerType(t *types.Pointer) Permission

{perm := &PointerPermission{BasePermission: basicPermission}typeMapper[t] = permperm.Target = typeMapper.NewFromType(t.Elem())return perm

}

it was tested by this:"*interface{}": &PointerPermission{

BasePermission: Mutable,Target: &InterfacePermission{

BasePermission: Mutable,},

},

Assignability tests

The tests for assigning were done as a simple quintuplet: Source permission, targetpermission, result for move, result for reference, result for copy; as the example inlisting 5.4 show.

And again, since the function were defined recursively, and we have successfully testedthe base combinations, all that was left to do was to test some possible combinations:For example, a function with no parameters, a function with parameters (thoughusually just one). The code is not precise, though: A function with two parameters

76

Listing 5.4: Test cases for assignabilityvar testcasesAssignableTo = []assignableToTestCase{

// Basic types{"om", "om", true, false, true},{"ow", "ow", false, true, false},// ...{&NilPermission{}, "om␣chan␣om", true, true, true}, // ... and all othernilable targets ...// Incompatible types{"om", "om␣func␣()", false, false, false}, // ... and so on ...// tests with a helper type for tuples, since there is no syntax for them.{tuplePermission{"om", "om"}, tuplePermission{"om", "ov"}, true, false,true},// ....

}

can be assigned to a function with one parameter, for example. The code relies on Gofiltering such impossible cases out earlier when doing type checking, and hence theyare not always handled, and thus not tested.

Merge and conversion

Merge and conversion tests look like this: The type of merge to perform, the two inputparameters, the expected output or nil, and an error string (or empty string if no errorshould occur). For example:{mergeIntersection, "om␣*␣om", "om", nil, "Cannot␣merge"}

The tests are fairly similar to the assign tests: Both assignment and merge andconversions have their own recursing of the data structures, and thus we need to testpossible combinations of things like empty lists, non-empty lists, and so on. Merge andconvert are even safer than assign: They actually check that the length of functionparameters are the same, rather than relying on Go doing it for them. And thus theyalso have tests checking that you cannot merge a 2-parameter function with a oneparameter function, for example.

One thing is special about tests for intersection and union: Since both are commuta-tive, the test runner actually checks each test case in both directions, ensuring thatA merge∩ B = B merge∩ A (and the same for ∪) (for tested A,B) without twice thenumber of tests.

77

5.6.2 Testing the interpreter package

The interpreter contains the store and abstract interpreter described in the sectionentitled Static analysis of Go programs, coverage is about 93%.

This subsection is split on 3 pages: This page covers the store, the second coversexpressions, and the third covers statements.

Testing the store

The store really was some special thing to test. No table driven tests were used here,but rather some testing functions were written. The reason is simple: While the otherstuff to be tested were single functions with lots of cases; the store is a lot of smallfunctions with few cases per function.

An example test case for the store is the definition of values, as shown in listing 5.5.We simply define a variable, check that the permission is correct, and then redefineit, which should reset the value (since := can both define new variables, but also setexisting variables defined in the same block).

Listing 5.5: Test case for definining values in the storefunc TestStore_Define(t *testing.T) {

store := NewStore()store, _ = store.Define("a", permission.Mutable)if len(store) != 1 {

t.Fatalf("Length␣is␣%d␣should␣be␣%d", len(store), 1)}if store.GetEffective("a") != permission.Mutable {

t.Errorf("Should␣be␣mutable,␣is␣%v", store.GetEffective("a"))}store, _ = store.Define("a", permission.Read)if len(store) != 1 {

t.Errorf("Length␣is␣%d␣should␣be␣%d", len(store), 1)}if store.GetEffective("a") != permission.Read {

t.Errorf("Should␣be␣read-only,␣is␣%v", store.GetEffective("a"))}

}

The store has 100% line coverage, and all error handling is explicit with if statements,so the coverage should be more conclusive here than for the parser, for example, as allerror cases need to be explictly tested here to achieve this level of coverage.

78

Abstract expression interpretation

The expression tests are table-driven again. As we can seen, we check a given expression(or a scenario, which contains a setup code part and an expression part), with a name,and a permission for a, a permission for b, the result permission, the result owner, theresult dependencies, and the after state of the a and b permissions in the store.

Listing 5.6: Test case for expressionstestCases := []struct {

expr interface{} // expression/scenario to testname string // name for test caselhs interface{} // permission for 'a' in storerhs interface{} // permission for 'b' storeresult interface{} // permission returnedowner string // owner returneddependencies []string // dependencies returnedlhsAfter interface{} // permission of a in store after

testrhsAfter interface{} // permission of b in store after

test}{

....{"a[b:2:3]", "sliceMin", "om␣[]ov", "om", "om␣[]ov", "a", []string{},

"n␣[]n", "om"},{"a[1:2:b]", "sliceMax", "om␣[]ov", "om", "om␣[]ov", "a", []string{},

"n␣[]n", "om"},{"a[1:2:b]", "sliceInvalid", "om␣map[ov]ov", "om", errorResult("not␣

sliceable"), "a", []string{}, "n␣[]n", "om"},// TODO//{scenario{"", "func() {}"}, "funcLit", "om", "om", "", "", nil, nil

, nil},{"a.(b)", "type␣cast", "om", "om", errorResult("not␣yet␣implemented")

, "", nil, nil, nil},

// Selectors (1): Method values{scenario{"var␣a␣interface{␣b()}", "a.b"}, "

selectMethodValueInterface", "ov␣interface{␣ov␣(ov)␣func␣()␣}", "_", "ov␣func␣()", "", []string{}, "ov␣interface{␣ov␣(ov)␣func␣()␣}", "_"},

{scenario{"var␣a␣interface{␣b()}", "a.b"}, "selectMethodValueInterfaceUnowned", "ov␣interface{␣ov␣(v)␣func␣()␣}", "_", "v␣func␣()", "", []string{}, "ov␣interface{␣ov␣(v)␣func␣()␣}", "_"},

....}

Every implemented expression is tested, but the coverage of each expression is not fully100%.

79

Abstract statement interpretation

Statement interpretation testing is table-driven as well. Each test case consists of thename, an description of an input store, some code to execute (in the form of a function),a slice of exit descriptions (a store description and a position of the branch statementthat is returned - as described in section 2 on page 50), and an error string that ischecked against a returned error (empty string means no error expected).

type testCase struct {name stringinput []storeItemDesc // structs variable name -> permissioncode stringoutput []exitDesc // struct with []storeItemDesc and int for

positionerror string

}

One test case is the empty block - It simply falls through, hence the result has no exitstatement (indicated by position being -1):

{"emptyBlock",[]storeItemDesc{

{"main", "om␣func␣(om␣*␣om)␣om␣*␣om"},},"func␣main()␣{␣␣}",[]exitDesc{

{nil, -1},},"",

},

A more complicated statement is if. In the following example we have two exits, oneis the return inside the if, the other is the return after it:

{"if",[]storeItemDesc{

{"a", "om␣*␣om"},{"nil", "om␣*␣om"},{"main", "om␣func␣(om␣*␣om)␣om␣*␣om"},

},"func␣main(a␣*int)␣*int␣{␣ifa␣!=␣nil{␣return␣a␣};␣return␣nil}",[]exitDesc{

{[]storeItemDesc{{"a", "n␣*␣r"}}, 54},{[]storeItemDesc{{"a", "om␣*␣om"}}, 66},

},""},

Every implemented statement is tested, again not for all possible cases.

80

6 Conclusion

We have seen that linearity can be used to model mutability in a safe way, and thatit can prevent certain race conditions when used in concurrent programs. We notedthat Go is a good language to adapt linear permissions for, as it is both small, andfeatures native primitives like channels and lightweight “goroutines” for constructingconcurrent programs.

Our adaption of linear types happened in the form of permissions. We defined a setof base permissions consisting of read, write, exclusive read, and exclusive write, andextended that to the shape of types like a struct A,B for some base permission aand some other permissions A,B. This allows us to describe permissions for an entireGo program in a way that closely resembles types, and could probably be nativelyintegrated into the native type system.

The adaption was designed in a sort-of bottom-up approach. First, we created thepermission package and operations related to assigning values. It quickly became clearthat an abstract interpreter seems like a good approach for checking the permissions,and we extended the permission package with further functionality that could becomeuseful for the interpreter.

The bottom-up approach led to some mismatches: Most importantly, the inability torender parts of an object unusable - when a part like v.x.y is borrowed, v is currentlycompletely borrowed. On the other hand, it led to a very well tested permission packagewhich can easily be extended with new things without having to fear breaking existingfunctionality greatly.

Another mismatch is the named type and interface handling: While Go allows convertingnamed types to interfaces and vice versa, the permissions have no support for that. Therealisation that a named permission could be needed came too late in the process, aftermost of the interpreter had already been written, and would have required significantchanges in the interpreter which were not feasible anymore in the time alloted to thisthesis.

The bottom-up approach also was the reason for requiring read permissions on thesource of a copy or move assignment. This was added to fix a problem in the interpreter,but ended up breaking a few other use cases.

The implementation did not reach completion. While we can interpret almost allstatements, we do not have support for complete functions, packages, and imports,meaning that an entire Go program cannot be checked yet. Support for annotations is

81

mostly theoretical, too. While the annotation parsing works just fine and is heavilyrelied upon in testing, we did not actually combine the stage that retrieves annotationswith the interpreter that checks the permissions - we only have a map from identifierto annotation so far (seen in section 4.4 on page 53), but we never filled it.

Apart from completing the implementation, further work is also needed in the usabilitydepartment: Deepy nested permissions, permissions with a lot of direct children, andcycles in permissions means that errors are hard to understand. it is not entirely clearhow to optimise that, though.

There’s also the question on how to handle compatibility: While it is possible tohandle existing Go calls in an unsafe manner (just specify parameters to request nopermissions, and return the maximum permission for return values), this does notseem satisfactory: If we can just have unsafe parts like that, the checking can easily becircumvented. Adding an “unsafe” base permission would help here.

Overall, we can be happy with the permission package of the implementation - it is ahigh quality code base with 100% code coverage, and a fairly clear approach on whatcan be done with permissions. It seems easily reusable in a better abstract interpreteror other checking framework. it is also easily extensible, especially when writing newfunctions that check a property for two permissions (like “can I assign A to B”) orthat produce a new permission from two permissions (like, “intersect A and B”), astheir implementation is very generic and support for some new operations can easilybe inserted.

We cannot be as happy with the interpreter package: it is not always easy to follow, itis not fully tested, and it has quite a few bugs. But that was to be expected, giventhat it was written after the permission package, and time eventually ran out. Theapproach of using an abstract interpreter, and a store with effective and maximumpermissions seems worth pursuing, though.

82

References

[1] Peter Achten, John Van Groningen, and Rinus Plasmeijer. “High level specificationof I/O in functional languages”. In: Functional Programming, Glasgow 1992.Springer, 1993, pp. 1–17. url: http://repository.ubn.ru.nl/bitstream/handle/2066/111106/111106.pdf?sequence=1.

[2] Henry G. Baker. ““Use-once” Variables and Linear Objects: Storage Management,Reflection and Multi-threading”. In: SIGPLAN Not. 30.1 (Jan. 1995), pp. 45–52.issn: 0362-1340. doi: 10.1145/199818.1998601. url: http://doi.acm.org/10.1145/199818.199860.

[3] John Boyland. “Alias burying: Unique variables without destructive reads”.In: Software: Practice and Experience 31.6 (2001), pp. 533–553. url:http://onlinelibrary.wiley.com/doi/10.1002/spe.370/abstract ; jsessionid=DAD9D44968C9A2BAC0EFF603C90EB498.f04t01.

[4] John Boyland. “Checking Interference with Fractional Permissions”. In: Pro-ceedings of the 10th International Conference on Static Analysis. SAS’03. SanDiego, CA, USA: Springer-Verlag, 2003, pp. 55–72. isbn: 3-540-40325-6. url:http://dl.acm.org/citation.cfm?id=1760267.1760273.

[5] John Boyland, James Noble, and William Retert. “Capabilities for Sharing:A Generalisation of Uniqueness and Read-Only”. In: Proceedings of the 15thEuropean Conference on Object-Oriented Programming. ECOOP ’01. London,UK, UK: Springer-Verlag, 2001, pp. 2–27. isbn: 3-540-42206-4. url: http://dl.acm.org/citation.cfm?id=646158.680004.

[6] Tyson Dowd et al. Using impurity to create declarative interfaces in Mercury.Tech. rep. Technical Report 2000/17, Department of Computer Science, Universityof Melbourne, Melbourne, Australia, 2000. url: http://www.mercurylang.org/documentation/papers/purity.ps.gz.

[7] Frequently Asked Questions (FAQ) - The Go Programming Language. url: https://golang.org/doc/faq#history.

[8] Stefan Heule et al. “Fractional Permissions Without the Fractions”. In: Proceedingsof the 13th Workshop on Formal Techniques for Java-Like Programs. FTfJP ’11.Lancaster, United Kingdom: ACM, 2011, 1:1–1:6. isbn: 978-1-4503-0893-9. doi:10.1145/2076674.20766752. url: http://doi.acm.org/10.1145/2076674.2076675.

1https://doi.org/10.1145/199818.1998602https://doi.org/10.1145/2076674.2076675

83

[9] Charles Antony Richard Hoare. “Communicating sequential processes”. In: Theorigin of concurrent programming. Springer, 1978, pp. 413–443. url: https://ora.ox.ac.uk/objects/uuid:833f1ea8-feba-4d81-b419-83e6f5f24e81/datastreams/ATTACHMENT01.

[10] John Hogg. “Islands: Aliasing protection in object-oriented languages”. In: ACMSIGPLAN Notices. Vol. 26. 11. ACM. 1991, pp. 271–285.

[11] John Launchbury and Simon L Peyton Jones. “State in haskell”. In: Lisp andsymbolic computation 8.4 (1995), pp. 293–341. url: https://pdfs.semanticscholar.org/5768/f243d9d91cf3225ae6ca1a89193f5b5ee423.pdf.

[12] Naftaly H Minsky. “Towards alias-free pointers”. In: European Conference onObject-Oriented Programming. Springer. 1996, pp. 189–209. url: http://www.cs.rutgers.edu/~minsky/public-papers/unique-paper.ps.

[13] James Noble, Jan Vitek, and John Potter. “Flexible alias protection”. In:ECOOP’98—Object-Oriented Programming (1998), pp. 158–185. url: http ://www.lirmm.fr/~ducour/Doc-objets/ECOOP/papers/1445/14450158.pdf.

[14] Zoltan Somogyi, Fergus J Henderson, and Thomas Charles Conway. “Mercury, anefficient purely declarative logic programming language”. In: Australian ComputerScience Communications 17 (1995), pp. 499–512. url: http://www.mercurylang.org/documentation/papers/acsc95.ps.gz.

The Go code of the implementation, as well as the Markdown (and minor amounts ofLATEX) used to generate this thesis can be retrieved from GitHub at:

https://github.com/julian-klode/lingolang

84

Erklärung

Ich versichere hiermit eidesstattlich, dass ich die vorliegende Arbeit selbstständigverfasst, ganz oder in Teilen noch nicht als Prüfungsleistung vorgelegt und keineanderen als die angegebenen Hilfsmittel benutzt habe.

Sämtliche Stellen der Arbeit, die benutzen Werken im Wortlaut oder dem Sinn nachentnommen sind, habe ich durch Quellenangaben kenntlich gemacht. Dies gilt auch fürZeichnungen, Skizzen, bildliche Darstellungen und dergleichen sowie für Quellen ausdem Internet. Bei Zuwiderhandlung gilt die Masterarbeit als nicht bestanden.

Ich bin mir bewusst, dass es sich bei Plagiarismus um schweres akademisches Fehlver-halten handelt, das im Wiederholungsfall weiter sanktioniert werden kann.

Ort, Datum Unterschrift

85