OSCON 2013: Apache Drill Workshop > Runtime Compilation
-
Upload
apachedrill -
Category
Technology
-
view
1.026 -
download
2
description
Transcript of OSCON 2013: Apache Drill Workshop > Runtime Compilation
1
Apache Drill: Compilation WorkshopJacques Nadeau, OSCON July 23, 2013
[email protected] |@intjesus
2
Runtime Compilation is Faster
JIT is smart but more gains can be had with runtime compilation
From http://bit.ly/16Xk32x
3
How do you do runtime compilation?
Runtime Compilation Tools javax.tools.Compiler (since 1.6): Wrapper to compile string into
bytecode using javac Janino: New BSD licensed Java-based Java compiler
Which One? Simple benchmarks shows Janino performs substantially faster
than javax.tools.Compiler Janino doesn’t support Annotations nor Generics (ugh.)
4
Code Play Time
Get Latest Drill git clone git://git.apache.org/incubator-drill.git git checkout master cd incubator-drill/sandbox/prototype mvn install
Download OSCON Drill examples git clone https://github.com/jacques-n/oscon-drill.git
cd oscon-drill mvn install cd compile
5
Exercise 1: Runtime Compilation Example
Goal Generate and evaluate your first runtime compiled code leveraging
Janino
Overview Janino provides a simple ExpressionEvaluator interface– Takes array of Objects as parameters– Returns single Object that can be casted to appropriate output class
Simplest way to use ExpressionEvaluator is by using a variable holder class– Variables in the exercise
Code src/test/java/org/apache/drill/oscon/compile/E1JaninoTest.java
6
Conclusions
Runtime compilation is easy Managing and building strings is painful Object interface is less than elegant
7
Simplification is necessary
CodeModel to the rescue– CodeModel provides a simplified interface for programmatically generating
Java source code– Extremely expressive, supporting all major constructs– Apache Licensed (yay!)
Use interfaces to manage things– Define an interface– Generate an entire class instead of just an evaluation block– Ensure class implements interface then generate new instance of runtime
generated class
8
Exercise 2: CodeModel + Interfaces
Goal Clean up the previous implementation to make it more repeatable
Overview JCodeModel provides interface to generate new class JExpr, JFieldVar, JMethod and other classes used via invocations or
statically Use Drill’s QueryClassLoader to help inject bytecode into
Classloader
Code src/test/java/org/apache/drill/oscon/compile/E2CodeModelTest.java
9
Conclusions
Things are better…but– Large blocks of code are going to be painful to do with CodeModel
Isn’t there some way to merge compile time generated code with runtime generated code?
10
Solution: Runtime Bytecode Merging
CodeModel to generate runtime specific blocks Janino to generate runtime bytecode Precompiled bytecode templates Use ASM package to merge the two distinct classes into one
runtime class
Loaded ClassASM Bytecode Merging
Janino compilation
CodeModel Generated
Code
Precompiled Bytecode Templates
11
Exercise 3: Template Merging Approach
Goal Leverage best of generated and pre-compiled world
Overview Drill’s ClassTransformer class does the dirty work, taking a TemplateDefinition
and InternalInterface source code ClassTransformer also marks class final and all methods private final except
external ones to maximize likelihood of JVM inlining. Drill’s QueryClassLoader is again used
Code src/test/java/org/apache/drill/oscon/compile/E3TemplateMergeTest.java src/main/java/org/apache/drill/oscon/compile/InsideInterface.java src/main/java/org/apache/drill/oscon/compile/OutsideInterface.java src/main/java/org/apache/drill/oscon/compile/OutsideTemplate.java
12
Exercise 4: Drill example
Goal– Apply generalized knowledge to Drill
Overview– Drill utilizes a combination of templating and code generation to build
query level operators– We also use source code rewriting to simplify implementation of scalar
functions– To minimize function overhead, all scalar function evaluations are merged
into large evaluation blocks.– FunctionDefintion and DrillFunc combine to provide implementation– drill-module.conf is used to inform Drill of available extension
Code– src/main/java/org/apache/drill/oscon/compile/AbsoluteFunction.java– src/test/java/org/apache/drill/oscon/compile/E4AbsTest.java
13
Drill Implementation: Expression Compilation
Best of all worlds: Runtime Bytecode Merging Balance development effort and performance needs Interpretation overhead during record batch setup Compile time operation for large code blocks