“Paranormal Tracktivity”Transparent instrumentation and software telemetry using GhostWriter
Norbert Sram
2
Who am I?
Norbert Sram Software engineer at AdNovum Member of application engineering
3
The importance of tracing!
4
Why tracing?
5
Why tracing?Because debuggers are not your best
friend! Period.
6
“... for most of the real problems (...) a debugger doesn't much help. And the real problems are what I worry about. The rest is just details.“
-- Linus Torvalds "Debuggers are evil"
7
"The kinds of bugs I have to troubleshoot are easily isolated by my unit tests, and can be quickly found through inspection and a few judiciously placed print statements”
-- Robert C. Martin ("Uncle Bob") "Debuggers are a wasteful
Timesink"
8
Why tracing?Because debuggers are not your best only
friend!
9
Why not just use ...?
Categorization factors Low or high level Manual or automated Runtime or compile time Online service or standalone solution Open source or proprietary
Is there a gap?
10
?
Low level Automated and
Manual Compile time Online and
Offline Open and
Closed
Popular solutions
High level Automated and
Manual Runtime Online Closed
Low level Automated Online Runtime Closed
11
Low level Automated and
Manual Compile time Online and
Offline Open and
Closed
Popular solutions
High level Automated and
Manual Runtime Online Closed
Low level Automated Online Runtime Closed
12
How did we get here?
13
14
15
16
Partial tracing is not tracing!
17
Requirements
Same performance profile No boilerplate Reuse existing logging API
18
No source mangling!
19
JSR-269 + OpenJDK
Magic annotation!
20
We reached our goals! … or did we?
21
22
JSR-269 + OpenJDK
JSR-269 used for hookingInternal OpenJDK API for everything else• Create tracing AST• Mold it into the AST of the compiled application
GhostWriter
OpenJDK
JSR-269
23
“Here be dragons”
I’m not proud of this!
24
@AdnTrace - configuration by exception Getting started with GhostWriter:
The two main factors: Easier application to existing projects Partial tracing is no longer an issue out of the box!
25
GhostWriter vs. Accidental Complexity
We’ll never know what we did here…
26
GhostWriter vs. Accidental Complexity
Experimental feature: state tracing
Unless we trace the initializations!
27
Ok. Nice. But how does tracing help me in a production environment?
28
GhostWriter in “Production Land”
Pareto principle applied to tracing:Capture always, serialize and store in case of errors
Detect and capture unhandled error
Delegate to runtime
29
GhostWriter Runtime
Job of the runtime is to serialize and store the snapshots provided by the compile time component.
Simple extension point:
30
GhostWriter in the cloud
Look to the future! Support the handling of extensive amounts of tracing.
GW - Runtime GW - Tracing
31
GhostWriter in the cloud
Look to the future! Support the handling of extensive amounts of tracing.
… …
32
Impact on non-functional requirementsHow does it affect my application? What if I pull the plug?
Determinism, resilience ... am I asking for too much?
33
Minimizing impact
Dedicated thread, fixed memory consumption, low latency
Isolated, compartmentalized runtimes
34
“Use the source, Luke!”ghostwriter.io
35
Are we there yet?Stable release planned for the end of this
year.
36
Demo:Bootstrapping GhostWriter
37
Contributors:Gábor Ginter
György KovácsNorbert SramPál Könyves
38
Thank you for your attention!