INTEGRATING IDEINTEGRATING IDEssWITH DOTTYWITH DOTTY

- EPFLGuillaume Martres

1

WHAT IS DOTTY?WHAT IS DOTTY?Research compiler that will become Scala 3Type system internals redesigned, inspired by DOT,but externally very similarMore info:

Recent blog posts on dotty.epfl.ch

scala-lang.org

2

A CHANCE TO REDESIGNA CHANCE TO REDESIGNCOMPONENTSCOMPONENTS

Improved incremental compilation (avoidundercompilation)Better pattern matching checks (

)algorithm now

reused in Swi�!

3

TOOLINGTOOLINGA good developer experience requires good tools:

A REPL (with syntax highlighting!)Dottydoc (used to generate

)IDE supportdotty.epfl.ch/docs

4

STATE OF THE ARTSTATE OF THE ARTBased on the

Scala-IDEENSIME

Reimplementation of the Scalatypechecker

Scala plugin for IntelliJ IDEA

Scala Presentation Compiler

5

STATE OF THE ARTSTATE OF THE ARTBased on the (3KLOC)

Scala-IDE (66 KLOC)ENSIME (server: 15 KLOC, emacs client: 10KLOC)

Reimplementation of the Scala typecheckerScala plugin for IntelliJ IDEA (230 KLOC)

Scala Presentation Compiler

6

DESIGN PRINCIPLESDESIGN PRINCIPLES1. Code reuse2. Editor-agnosticity3. Easy to use (and to

install!)

7

DESIGN PRINCIPLESDESIGN PRINCIPLES1. Code reuse2. 3.

Editor-agnosticityEasy to use (and toinstall!)

8

QUERYING THE COMPILERQUERYING THE COMPILER

Each phase progressively simplify trees until theycan be emitted as JVM bytecode

9

QUERYING THE COMPILERQUERYING THE COMPILER

Each phase progressively simplify trees until theycan be emitted as JVM bytecode

10

SOURCE CODESOURCE CODE

Cursor position = 🚩Query: jump todefinition

final val elem = 1

val foo = elem🚩 + 1

11

TREE AFTER TREE AFTER TyperTyper

Every tree node has a type and apositionQuery can be answered

final val elem: 1 = 1

val foo: Int = elem + 1

12

TREE AFTER TREE AFTER FirstTransformFirstTransform

Information lost by constantfoldingImpossible to answer query

final val elem: 1 = 1

val foo: Int = 2

13

QUERYING THE COMPILERQUERYING THE COMPILERStore trees before FirstTransformRespond to IDE queries by traversing treesWhat about code that has already beencompiled?

14

PICKLINGPICKLING

In Scala 2: store methods signatures (for separatecompilation)In Dotty: store full trees

15

TASTY: TYPED ASTTASTY: TYPED ASTSERIALIZATION FORMATSERIALIZATION FORMAT

Original motivation: solve the

Always use JVM bytecode: breaks when compilerencoding changesAlways recompile source code: breaks when thetypechecker changes

Can also be used to provide interactive features:deserialize and query trees

binary compatibilityproblem

16

INTERACTIVE APISINTERACTIVE APISConvenience methods for tree traversals, compilerlifecycle managementUsed both in the IDE and the REPL (e.g., forcompletions)In the future: interruption handling, partialtypechecking, ...Less then 1 KLOC

17

DESIGN PRINCIPLESDESIGN PRINCIPLES1. 2. Editor-agnosticity3.

Code reuse

Easy to use (and toinstall!)

18

THE IDE PORTABILITY PROBLEMTHE IDE PORTABILITY PROBLEMGetting m IDEs to support n programming languagesrequires n*m IDE plugins.

19

THE LANGUAGE SERVERTHE LANGUAGE SERVERPROTOCOLPROTOCOL

20

BASICS OF THE LSPBASICS OF THE LSPFirst implemented in Visual Studio CodeJSON-RPCIDE notifies the language server about user actionsLS maintains internal representation of codeLS notify IDE about warnings/errorsIDE can send requests usually triggered by useractionsAsynchronous, cancelable

21

IMPLEMENTING THE DOTTYIMPLEMENTING THE DOTTYLANGUAGE SERVERLANGUAGE SERVER

Low-level message handling done by

Relies on interactive APIs0.5 KLOC

EclipseLSP4J

22

override def definition(params: TextDocumentPositionParams) =

}

23

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

}

24

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

}

25

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

}

26

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

}

27

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

val pos = sourcePosition(driver, uri, params.getPosition)

}

28

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

val pos = sourcePosition(driver, uri, params.getPosition)

val uriTrees = driver.openedTrees(uri)

}

29

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

val pos = sourcePosition(driver, uri, params.getPosition)

val uriTrees = driver.openedTrees(uri)

val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)

}

30

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

val pos = sourcePosition(driver, uri, params.getPosition)

val uriTrees = driver.openedTrees(uri)

val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)

val classTree =

SourceTree.fromSymbol(sym.topLevelClass.asClass).toList

}

31

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

val pos = sourcePosition(driver, uri, params.getPosition)

val uriTrees = driver.openedTrees(uri)

val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)

val classTree =

SourceTree.fromSymbol(sym.topLevelClass.asClass).toList

val defTree = Interactive.definition(classTree, sym)

}

32

override def definition(params: TextDocumentPositionParams) =

computeAsync { cancelToken =>

val uri = new URI(params.getTextDocument.getUri)

val driver = driverFor(uri)

implicit val ctx = driver.currentCtx

val pos = sourcePosition(driver, uri, params.getPosition)

val uriTrees = driver.openedTrees(uri)

val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)

val classTree =

SourceTree.fromSymbol(sym.topLevelClass.asClass).toList

val defTree = Interactive.definition(classTree, sym)

defTree.map(d => location(d.namePos)).asJava

}

33

DESIGN PRINCIPLESDESIGN PRINCIPLES1. 2. 3. Easy to use (and to

install!)

Code reuseEditor-agnosticity

34

SBT INTEGRATIONSBT INTEGRATIONAnalyze the build to find DottyprojectsCompile these projectsGenerate configuration filesInstall the Dotty VSCode extensionLaunch VSCode

35

CONFIGURATION FILESCONFIGURATION FILES.dotty-ide-artifact, used by the IDEextension to launch the Dotty Language Server:

ch.epfl.lamp:dotty-language-server_0.8:0.8.0-RC1

36

CONFIGURATION FILESCONFIGURATION FILES.dotty-ide.json, used by the DLS to launchcompiler instances:[

{

"id" : "root/compile",

"compilerVersion" : "0.8.0-RC1",

"compilerArguments" : [ ],

"sourceDirectories" : [ "src/main/scala" ],

"dependencyClasspath" : [ ... ],

"classDirectory" : "target/scala-0.8/classes"

},

{

"id" : "root/test",

...

},

...

]

37

BUILD SERVER PROTOCOLBUILD SERVER PROTOCOLInstead of making plugins for build tools to extractinformation, ask them!We also need a discovery protocol: "How do I start abuild server for this project?"

38

DESIGN PRINCIPLES, REVISITEDDESIGN PRINCIPLES, REVISITED1. Code reuse

Compiler APIs for interactive usage2. Editor-agnosticity

Implemented the LSP3. Easy to use (and to install!)

One command (but we can dobetter!)

39

GOING FURTHER: DEBUGGERGOING FURTHER: DEBUGGERSUPPORTSUPPORT

Based on the Most features "just work"(Not actually merged in Dottyyet)Challenge: expressionevaluation

Java Debug Server

40

EXPRESSION EVALUATIONEXPRESSION EVALUATIONclass Hello {

def foo(implicit y: Context): String = { /*...*/ }

def bar(implicit y: Context) = {

/*...*/

}

}

41

EXPRESSION EVALUATIONEXPRESSION EVALUATIONclass Hello {

def foo(implicit y: Context): String = { /*...*/ }

def bar(implicit y: Context) = {

🚩 /*...*/

}

}

42

EXPRESSION EVALUATIONEXPRESSION EVALUATIONclass Hello {

def foo(implicit y: Context): String = { /*...*/ }

def bar(implicit y: Context) = {

🚩 foo /*...*/

}

}

43

RUN THE COMPILER PIPELINERUN THE COMPILER PIPELINEclass Hello {

def foo(y: Context): String = { /*...*/ }

def bar(y: Context) = {

🚩 this.foo(y) /*...*/

}

}

44

EXTRACT TO A STATIC METHODEXTRACT TO A STATIC METHODobject Global {

def liftedExpr($this: Hello, y: Context) =

$this.foo(y)

}

45

EXTRACT TO A STATIC METHODEXTRACT TO A STATIC METHODobject Global {

def liftedExpr($this: Hello, y: Context) =

$this.foo(y)

def exec(self: Object, localVariables: Map[String, Object]) =

liftedExpr(

self.asInstanceOf[Hello],

localVariables("y").asInstanceOf[Context]

)

}

46

ON THE DEBUGGING VMON THE DEBUGGING VMCompile Global to a classfileLoad it in the debugged VMCall Global.exec with the rightarguments

47

ON THE DEBUGGED VMON THE DEBUGGED VMIn the standardlibrarypackage dotty.runtime

object DebugEval {

def eval(classpath: String,

self: Object,

localVariables: Map[String, Object]): Any = {

val cl = new URLClassLoader(Array(new URL("file://" + classpath)

val cls = cl.loadClass("Global")

val instance = cls.newInstance

val exec = cls.getMethod("exec")

exec.invoke(instance, self, names, args)

}

}

48

ON THE DEBUGGING VMON THE DEBUGGING VMWe want to remotely execute:

val classpath = <classpath for the compiled Global class>

val self = <this in the stackframe>

val localVariables = <map of local variables in the stackframe>

dotty.runtime.DebugEval(classpath, self, localVariables)

49

ON THE DEBUGGING VMON THE DEBUGGING VMWe use the Java Debugging Interface APIs:

val vars = stackFrame.visibleVariables

val mapCls =

vm.classesByName("java.util.Map")

.get(0).asInstanceOf[ClassType]

val localVariablesRef = mapCls.newInstance()

// Skipped: store `vars` into `localVariablesRef`

val debugCls =

vm.classesByName("dotty.runtime.DebugEval")

.get(0).asInstanceOf[ClassType]

val eval =

debugCls.methodsByName("eval").get(0)

debugCls.invokeMethod(

thread, eval,

List(vm.mirrorOf(classpath), stackFrame.thisObject,

localVariablesRef)

)

50

FUTURE WORKFUTURE WORKOptimizationsMore features

Documentation on hoverBetter build tool integration (Build ServerProtocol!)

51

CONCLUSIONCONCLUSIONDesign your compiler with interactivity in mindDesign your build tool with interactivity in mindInteractivity should go beyond what IDEs and REPLscurrently offer

Type Driven Development with Idris

52

QUESTIONS ?QUESTIONS ?More info: Come chat with us:

Contributors welcome!

dotty.epfl.ch

gitter.im/lampepfl/dotty

53