S ess ion 3280

Experimental Tools for Serviceability

Tony Printezis Kenneth RussellL ecturer Member of T ech S taffUnivers ity of G lasgow S un Microsystems

S ess ion 3280 2

Overall Presentation Goal

G et an ins ight into examining, debugging, and profiling different aspects of a J ava™ technology-based V irtual Machine (J V M™ /J ava V M) us ing two new experimental tools

S ess ion 3280 3

Learning Objectives

• A s a result of this presentation, you will be convinced that:– J ava V M debugging is hard!– C urrent debugging tools are not always

adequate– T here is a need for specialised tools to help

the J ava V M implementor...– ...which might also be used to help application

programmers too

S ess ion 3280 4

Speakers’  Qualifications

• T ony P rintezis– L ecturer, Univers ity of G lasgow– He has been writing garbage collectors

for J ava V Ms for four years

• K enneth R ussell– Member of tech staff, S un Microsystems– He has worked on the J ava HotS pot™

runtime and compiler for over two years

S ess ion 3280 5

Bug Hunting Ain’t Fun!

How many all-nighters have you pulled because of a bug in your native code?

Now imagine debugging the G C or the J ava V M itself...

S ess ion 3280 6

“Java VM Debugging Facility”

S ess ion 3280 7

Some Proper Debugging Tools!

• G C spy– T ony P rintezis – Heap visualisation for J ava V Ms• L ow–impact, high–level visualisation

• J ava HotS pot™ S erviceability A gent– K enneth R ussell– E xaminer and debugger for J ava HotS pot V M• P ost–mortem analys is of J ava V M

S ess ion 3280 8

GCspy

• A daptable heap visualisation framework– Incorporated into several J ava V Ms

• D eveloped to visualise the behaviour of G arbage C ollectors (G C s )

• S calable– L arge heap s izes– V ery long–running applications

• D ynamic– Works while J ava V M is executing

S ess ion 3280 9

Why Heap Visualisation?

• G C s have (almost) chaotic behaviour!– O ne small change in the application behaviour

can have a large effect in the G C performance

• G C debugging and profiling facilities– S till very primitive

• V isualisation provides a convenient way to get an ins ight into the G C ’s behaviour– D ebugging and profiling

S ess ion 3280 10

Visualisation Granularity

• C oarse granularity– Heap blocks , not objects– S maller data volumes– F aster to collect / send / store data

• B ut...– L ower level of detail– Information loss

• C an adjust block s ize to deal with scalability and screen s ize limitations

S ess ion 3280

S ess ion 3280 12

Client–Server Model

• C lient–S erver– J ava V M is the server– V isualiser is the client

• F lexibility– C an be run as different processes– C an be run on different machines– C an connect / disconnect the visualiser

during J ava V M execution

S ess ion 3280 13

Portability

• F ramework is J ava V M and G C independent– But (small) modifications to the J ava V M are

needed

• V isualiser operates over some abstractions

• Need to map the G C ’s s tate to those abstractions– A G C spy D river does this

• O nly the server–s ide of the framework needs to be customised

S ess ion 3280 14

Architecture

Visualiser

SpaceRenderer

0

SpaceRenderer

1

Space 0 Space 1

Client Infrastructure

Stream 0

Stream 1

Stream 0

Stream 1

Stream 2

Comms

Java Application

Java VM

Comms

Server Infrastructure

Used

Roots

Objects

Cards

Marked

S-SDriver

M&CDriver

S-S GC M&C GC

Data Collection

Socket

ClientServer

S ess ion 3280 15

Performance Impact

• D ata collection can take a long time– S everal seconds for each G C in the case

of large heaps

• J ava V M performance is affected

• No performance penalty when the visualiser is not connected!

S ess ion 3280 16

Trace Storing / Replaying

• G oal– S tore transmiss ions from a J ava V M– R eplay them at a later time

• R eplay tool behaves exactly like the J ava V M

• C ompact trace files– C ompress ion

S ess ion 3280

S ess ion 3280 18

Current Implementations

• T hree J ava™ technology–based V Ms– J ava HotS pot™ V M– S un’s R esearchV M– IBM’s J ikes R V M

• A t least eight different G C s

• O ther systems too– Not only limited to J ava V Ms!

S ess ion 3280 19

Future Work

• B etter, more flexible visualisations

• O ther visualisation styles– e.g. S catter plots , Histograms

• R ewind facilities

• C onditional breakpoints

S ess ion 3280 20

Java HotSpot Serviceability Agent

• E xaminer and debugger for J ava HotS pot virtual machine

• O riginally des igned for post–mortem analys is

• R uns no code in target J V M

• R ecovers high–level J ava programming language s tate

S ess ion 3280 21

Motivation

• J ava HotS pot V M group needed better debugging tools

• C urrent debuggers not targeted toward debugging the J V M– C /C ++ debuggers• R equire debug info not present in product• D o not understand J ava language, objects

– J ava programming language debuggers• G ather information via J V M D I• D o not work with crashed J ava V Ms

S ess ion 3280 22

Mechanisms

• J ava HotS pot S erviceability A gent (S A )– Understands layouts of key HotS pot data

s tructures• T hread lis t• S tack frames

– C ontains code to walk these data s tructures

• D ebugger tuned for J ava HotS pot V M

• R ecovers same information as J V M D I / J D WP - based debuggers

S ess ion 3280 23

Features

• J ava programming language A P Is– Implemented almost entirely in J ava

programming language• A ll the way down to “read process memory”• E nables cross–O S and cross–C P U debugging

– C an be bound to existing J D I interfaces in J P D A (in progress ; more later)

S ess ion 3280 24

Features

• F ine–grained heap analys is– O bject–by–object iteration– Iteration through objects

• S tack walking– A ccess to local variables , express ion

stack, monitors– F allback mechanisms for analyzing crashes

in interpreter or in compiled code

S ess ion 3280 25

Architectural Diagram

TargetHotSpot

JVM

dbx

import module

SA'sJava VM

Dbx APIs Socket

/proc

• Note that no code is being run in target J ava V M– C urrent J P D A runs J V M D I-based code there

S ess ion 3280 26

Evolution

• F irs t debugger– A ttach to running J V M or open core file– E xamine all J V M state at very low level– P rocess suspended during examination

• T ools for developers of J V M™ technology– S tack memory dump with annotations– O bject inspection

• E nd user tools– Heap dumps, liveness analys is

S ess ion 3280

S ess ion 3280 28

Evolution

• F irs t debugger– Most useful to developers of J ava HotS pot

technology– Most useful on the

S olaris™ O perating E nvironment• Built on dbx• “E scape hatch” to see non–J V M state

– P orted to Win32– No interaction with process• No suspend, resume• No breakpoint setting, exception catching

S ess ion 3280 29

Observation

• S A built on very low–level debugging primitives– “read process memory”– “fetch thread register set”

• S ame primitives as C /C ++ debuggers

• Why not extend to look at more than just J ava programming language s tate?

S ess ion 3280 30

Extensions

• S A A P Is extended:– F etch all O S –level threads– F etch all dynamically loaded objects

• More platform–dependent “J ava code”:– C - level s tack walking (all frames on stack)– P ars ing of C /C ++ debug information

• E nable complete process examination system to be written in J ava programming language

S ess ion 3280 31

Extensions

• F urther extens ions:– R esume/suspend process– P oll for event (seg fault, etc.)– S et breakpoint at machine instruction

• R elatively little additional code

• E nable mostly–complete C /C ++ debugger to be written in J ava programming language

S ess ion 3280 32

Extensions

• C ommunication with J V M D I module– S tart with -X run– E nables J ava programming language breakpoint

setting and exception catching

• T his is the only code run in target J ava V M

• Most functionality available without it

S ess ion 3280 33

Observation

• Now have all primitives necessary to build both C /C ++ and J ava programming language debuggers

• S ystem was originally des igned for post–mortem analys is of J ava V M

• Ideally posed for building mixed–language debugger– C an handle crashes in native code gracefully

S ess ion 3280 34

Current Status

• E xperimental C /C ++ and J ava programming language debugger has been built– Written in J ava programming language

• S upports breakpoints in both C and “J ava code” (J ava™ technology–based code)

• S upports exception catching

• D egrades gracefully– Most of debugger s till works even if target

process crashes

S ess ion 3280

S ess ion 3280 36

Availability

• S A is currently available to source licensees of J ava HotS pot™ virtual machine

S ess ion 3280 37

Future Work

• C omplete ports to all S un–supported platforms

• J D I binding in development– A llow current tools to access J ava programming

language state of J ava HotS pot core files via S A

• D iscuss prototype C /C ++ debugging A P Is with tool vendors– O ptional extens ions to J D I– E nable all tool vendors to build

mixed–language debuggers

S ess ion 3280 38

Presentation Summary

• J ava V M debugging is hard!

• We presented two new debugging tools– Heap visualisation with G C spy– P ost–mortem analys is with the J ava HotS pot

S erviceability A gent

• T hey are used by J ava V M implementors

• T hey operate at two different levels

• T hey improve debugging effectiveness

S ess ion 3280 39

Call to Action

• Need feedback on these tools and A P Is– Whether they would be useful– F eatures which are needed– S hould we ship them (unsupported)

• F eedback on G C spy– tony@dcs.gla.ac.uk

• F eedback on the J ava HotS pot S erviceability A gent– sa- feedback@sun.com

S ess ion 3280

S ess ion 3280