“Paranormal Tracktivity”Transparent instrumentation and software telemetry using GhostWriter

Norbert Sram

2

Who am I?

Norbert Sram Software engineer at AdNovum Member of application engineering

3

The importance of tracing!

4

Why tracing?

5

Why tracing?Because debuggers are not your best

friend! Period.

6

“... for most of the real problems (...) a debugger doesn't much help. And the real problems are what I worry about. The rest is just details.“

-- Linus Torvalds "Debuggers are evil"

7

"The kinds of bugs I have to troubleshoot are easily isolated by my unit tests, and can be quickly found through inspection and a few judiciously placed print statements”

-- Robert C. Martin ("Uncle Bob") "Debuggers are a wasteful

Timesink"

8

Why tracing?Because debuggers are not your best only

friend!

9

Why not just use ...?

Categorization factors Low or high level Manual or automated Runtime or compile time Online service or standalone solution Open source or proprietary

Is there a gap?

10

?

Low level Automated and

Manual Compile time Online and

Offline Open and

Closed

Popular solutions

High level Automated and

Manual Runtime Online Closed

Low level Automated Online Runtime Closed

11

Low level Automated and

Manual Compile time Online and

Offline Open and

Closed

Popular solutions

High level Automated and

Manual Runtime Online Closed

Low level Automated Online Runtime Closed

12

How did we get here?

13

14

15

16

Partial tracing is not tracing!

17

Requirements

Same performance profile No boilerplate Reuse existing logging API

18

No source mangling!

19

JSR-269 + OpenJDK

Magic annotation!

20

We reached our goals! … or did we?

21

22

JSR-269 + OpenJDK

JSR-269 used for hookingInternal OpenJDK API for everything else• Create tracing AST• Mold it into the AST of the compiled application

GhostWriter

OpenJDK

JSR-269

23

“Here be dragons”

I’m not proud of this!

24

@AdnTrace - configuration by exception Getting started with GhostWriter:

The two main factors: Easier application to existing projects Partial tracing is no longer an issue out of the box!

25

GhostWriter vs. Accidental Complexity

We’ll never know what we did here…

26

GhostWriter vs. Accidental Complexity

Experimental feature: state tracing

Unless we trace the initializations!

27

Ok. Nice. But how does tracing help me in a production environment?

28

GhostWriter in “Production Land”

Pareto principle applied to tracing:Capture always, serialize and store in case of errors

Detect and capture unhandled error

Delegate to runtime

29

GhostWriter Runtime

Job of the runtime is to serialize and store the snapshots provided by the compile time component.

Simple extension point:

30

GhostWriter in the cloud

Look to the future! Support the handling of extensive amounts of tracing.

GW - Runtime GW - Tracing

31

GhostWriter in the cloud

Look to the future! Support the handling of extensive amounts of tracing.

… …

32

Impact on non-functional requirementsHow does it affect my application? What if I pull the plug?

Determinism, resilience ... am I asking for too much?

33

Minimizing impact

Dedicated thread, fixed memory consumption, low latency

Isolated, compartmentalized runtimes

34

“Use the source, Luke!”ghostwriter.io

35

Are we there yet?Stable release planned for the end of this

year.

36

Demo:Bootstrapping GhostWriter

37

Contributors:Gábor Ginter

György KovácsNorbert SramPál Könyves

38

Thank you for your attention!