Post on 20-May-2020
INTEGRATING IDEINTEGRATING IDEssWITH DOTTYWITH DOTTY
- EPFLGuillaume Martres
1
WHAT IS DOTTY?WHAT IS DOTTY?Research compiler that will become Scala 3Type system internals redesigned, inspired by DOT,but externally very similarMore info:
Recent blog posts on dotty.epfl.ch
scala-lang.org
2
A CHANCE TO REDESIGNA CHANCE TO REDESIGNCOMPONENTSCOMPONENTS
Improved incremental compilation (avoidundercompilation)Better pattern matching checks (
)algorithm now
reused in Swi�!
3
TOOLINGTOOLINGA good developer experience requires good tools:
A REPL (with syntax highlighting!)Dottydoc (used to generate
)IDE supportdotty.epfl.ch/docs
4
STATE OF THE ARTSTATE OF THE ARTBased on the
Scala-IDEENSIME
Reimplementation of the Scalatypechecker
Scala plugin for IntelliJ IDEA
Scala Presentation Compiler
5
STATE OF THE ARTSTATE OF THE ARTBased on the (3KLOC)
Scala-IDE (66 KLOC)ENSIME (server: 15 KLOC, emacs client: 10KLOC)
Reimplementation of the Scala typecheckerScala plugin for IntelliJ IDEA (230 KLOC)
Scala Presentation Compiler
6
DESIGN PRINCIPLESDESIGN PRINCIPLES1. Code reuse2. Editor-agnosticity3. Easy to use (and to
install!)
7
DESIGN PRINCIPLESDESIGN PRINCIPLES1. Code reuse2. 3.
Editor-agnosticityEasy to use (and toinstall!)
8
QUERYING THE COMPILERQUERYING THE COMPILER
Each phase progressively simplify trees until theycan be emitted as JVM bytecode
9
QUERYING THE COMPILERQUERYING THE COMPILER
Each phase progressively simplify trees until theycan be emitted as JVM bytecode
10
SOURCE CODESOURCE CODE
Cursor position = 🚩Query: jump todefinition
final val elem = 1
val foo = elem🚩 + 1
11
TREE AFTER TREE AFTER TyperTyper
Every tree node has a type and apositionQuery can be answered
final val elem: 1 = 1
val foo: Int = elem + 1
12
TREE AFTER TREE AFTER FirstTransformFirstTransform
Information lost by constantfoldingImpossible to answer query
final val elem: 1 = 1
val foo: Int = 2
13
QUERYING THE COMPILERQUERYING THE COMPILERStore trees before FirstTransformRespond to IDE queries by traversing treesWhat about code that has already beencompiled?
14
PICKLINGPICKLING
In Scala 2: store methods signatures (for separatecompilation)In Dotty: store full trees
15
TASTY: TYPED ASTTASTY: TYPED ASTSERIALIZATION FORMATSERIALIZATION FORMAT
Original motivation: solve the
Always use JVM bytecode: breaks when compilerencoding changesAlways recompile source code: breaks when thetypechecker changes
Can also be used to provide interactive features:deserialize and query trees
binary compatibilityproblem
16
INTERACTIVE APISINTERACTIVE APISConvenience methods for tree traversals, compilerlifecycle managementUsed both in the IDE and the REPL (e.g., forcompletions)In the future: interruption handling, partialtypechecking, ...Less then 1 KLOC
17
DESIGN PRINCIPLESDESIGN PRINCIPLES1. 2. Editor-agnosticity3.
Code reuse
Easy to use (and toinstall!)
18
THE IDE PORTABILITY PROBLEMTHE IDE PORTABILITY PROBLEMGetting m IDEs to support n programming languagesrequires n*m IDE plugins.
19
THE LANGUAGE SERVERTHE LANGUAGE SERVERPROTOCOLPROTOCOL
20
BASICS OF THE LSPBASICS OF THE LSPFirst implemented in Visual Studio CodeJSON-RPCIDE notifies the language server about user actionsLS maintains internal representation of codeLS notify IDE about warnings/errorsIDE can send requests usually triggered by useractionsAsynchronous, cancelable
21
IMPLEMENTING THE DOTTYIMPLEMENTING THE DOTTYLANGUAGE SERVERLANGUAGE SERVER
Low-level message handling done by
Relies on interactive APIs0.5 KLOC
EclipseLSP4J
22
override def definition(params: TextDocumentPositionParams) =
}
23
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
}
24
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
}
25
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
}
26
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
}
27
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
val pos = sourcePosition(driver, uri, params.getPosition)
}
28
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
val pos = sourcePosition(driver, uri, params.getPosition)
val uriTrees = driver.openedTrees(uri)
}
29
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
val pos = sourcePosition(driver, uri, params.getPosition)
val uriTrees = driver.openedTrees(uri)
val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)
}
30
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
val pos = sourcePosition(driver, uri, params.getPosition)
val uriTrees = driver.openedTrees(uri)
val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)
val classTree =
SourceTree.fromSymbol(sym.topLevelClass.asClass).toList
}
31
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
val pos = sourcePosition(driver, uri, params.getPosition)
val uriTrees = driver.openedTrees(uri)
val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)
val classTree =
SourceTree.fromSymbol(sym.topLevelClass.asClass).toList
val defTree = Interactive.definition(classTree, sym)
}
32
override def definition(params: TextDocumentPositionParams) =
computeAsync { cancelToken =>
val uri = new URI(params.getTextDocument.getUri)
val driver = driverFor(uri)
implicit val ctx = driver.currentCtx
val pos = sourcePosition(driver, uri, params.getPosition)
val uriTrees = driver.openedTrees(uri)
val sym = Interactive.enclosingSourceSymbol(uriTrees, pos)
val classTree =
SourceTree.fromSymbol(sym.topLevelClass.asClass).toList
val defTree = Interactive.definition(classTree, sym)
defTree.map(d => location(d.namePos)).asJava
}
33
DESIGN PRINCIPLESDESIGN PRINCIPLES1. 2. 3. Easy to use (and to
install!)
Code reuseEditor-agnosticity
34
SBT INTEGRATIONSBT INTEGRATIONAnalyze the build to find DottyprojectsCompile these projectsGenerate configuration filesInstall the Dotty VSCode extensionLaunch VSCode
35
CONFIGURATION FILESCONFIGURATION FILES.dotty-ide-artifact, used by the IDEextension to launch the Dotty Language Server:
ch.epfl.lamp:dotty-language-server_0.8:0.8.0-RC1
36
CONFIGURATION FILESCONFIGURATION FILES.dotty-ide.json, used by the DLS to launchcompiler instances:[
{
"id" : "root/compile",
"compilerVersion" : "0.8.0-RC1",
"compilerArguments" : [ ],
"sourceDirectories" : [ "src/main/scala" ],
"dependencyClasspath" : [ ... ],
"classDirectory" : "target/scala-0.8/classes"
},
{
"id" : "root/test",
...
},
...
]
37
BUILD SERVER PROTOCOLBUILD SERVER PROTOCOLInstead of making plugins for build tools to extractinformation, ask them!We also need a discovery protocol: "How do I start abuild server for this project?"
38
DESIGN PRINCIPLES, REVISITEDDESIGN PRINCIPLES, REVISITED1. Code reuse
Compiler APIs for interactive usage2. Editor-agnosticity
Implemented the LSP3. Easy to use (and to install!)
One command (but we can dobetter!)
39
GOING FURTHER: DEBUGGERGOING FURTHER: DEBUGGERSUPPORTSUPPORT
Based on the Most features "just work"(Not actually merged in Dottyyet)Challenge: expressionevaluation
Java Debug Server
40
EXPRESSION EVALUATIONEXPRESSION EVALUATIONclass Hello {
def foo(implicit y: Context): String = { /*...*/ }
def bar(implicit y: Context) = {
/*...*/
}
}
41
EXPRESSION EVALUATIONEXPRESSION EVALUATIONclass Hello {
def foo(implicit y: Context): String = { /*...*/ }
def bar(implicit y: Context) = {
🚩 /*...*/
}
}
42
EXPRESSION EVALUATIONEXPRESSION EVALUATIONclass Hello {
def foo(implicit y: Context): String = { /*...*/ }
def bar(implicit y: Context) = {
🚩 foo /*...*/
}
}
43
RUN THE COMPILER PIPELINERUN THE COMPILER PIPELINEclass Hello {
def foo(y: Context): String = { /*...*/ }
def bar(y: Context) = {
🚩 this.foo(y) /*...*/
}
}
44
EXTRACT TO A STATIC METHODEXTRACT TO A STATIC METHODobject Global {
def liftedExpr($this: Hello, y: Context) =
$this.foo(y)
}
45
EXTRACT TO A STATIC METHODEXTRACT TO A STATIC METHODobject Global {
def liftedExpr($this: Hello, y: Context) =
$this.foo(y)
def exec(self: Object, localVariables: Map[String, Object]) =
liftedExpr(
self.asInstanceOf[Hello],
localVariables("y").asInstanceOf[Context]
)
}
46
ON THE DEBUGGING VMON THE DEBUGGING VMCompile Global to a classfileLoad it in the debugged VMCall Global.exec with the rightarguments
47
ON THE DEBUGGED VMON THE DEBUGGED VMIn the standardlibrarypackage dotty.runtime
object DebugEval {
def eval(classpath: String,
self: Object,
localVariables: Map[String, Object]): Any = {
val cl = new URLClassLoader(Array(new URL("file://" + classpath)
val cls = cl.loadClass("Global")
val instance = cls.newInstance
val exec = cls.getMethod("exec")
exec.invoke(instance, self, names, args)
}
}
48
ON THE DEBUGGING VMON THE DEBUGGING VMWe want to remotely execute:
val classpath = <classpath for the compiled Global class>
val self = <this in the stackframe>
val localVariables = <map of local variables in the stackframe>
dotty.runtime.DebugEval(classpath, self, localVariables)
49
ON THE DEBUGGING VMON THE DEBUGGING VMWe use the Java Debugging Interface APIs:
val vars = stackFrame.visibleVariables
val mapCls =
vm.classesByName("java.util.Map")
.get(0).asInstanceOf[ClassType]
val localVariablesRef = mapCls.newInstance()
// Skipped: store `vars` into `localVariablesRef`
val debugCls =
vm.classesByName("dotty.runtime.DebugEval")
.get(0).asInstanceOf[ClassType]
val eval =
debugCls.methodsByName("eval").get(0)
debugCls.invokeMethod(
thread, eval,
List(vm.mirrorOf(classpath), stackFrame.thisObject,
localVariablesRef)
)
50
FUTURE WORKFUTURE WORKOptimizationsMore features
Documentation on hoverBetter build tool integration (Build ServerProtocol!)
51
CONCLUSIONCONCLUSIONDesign your compiler with interactivity in mindDesign your build tool with interactivity in mindInteractivity should go beyond what IDEs and REPLscurrently offer
Type Driven Development with Idris
52
QUESTIONS ?QUESTIONS ?More info: Come chat with us:
Contributors welcome!
dotty.epfl.ch
gitter.im/lampepfl/dotty
53