Software maintenance is a tedious and challenging job. As long as the

software functions as expected, it looks great. Imagine the situation, your

pager keeps buzzing in the midnight (Not a happy feeling, right?).

Any software system, no matter how well it has been built and quality tested

might get into run-time performance issues. The reasons can be within the

functional boundary or can be from the external environment. Software

systems are built on certain assumptions and pre-conceived notions.

However, when they go live, the assumptions may turn out to be false

causing the system to malfunction.

In enterprise J2EE systems which generally have very large user base and

involve interactions between heterogeneous systems, one of the general

runtime issues reported is the system slowdown or system "hang". In such

situations, the general trouble shooting pattern would be to analyze the

Java thread dumps to isolate the threads which are causing the slow-down

or hang. This article discusses the Java stack traces; anatomy of Java

Threads and how to read thread dumps in general.

Exceptions and Stack Traces

All of us encounter and have encountered exceptions during our

learning/development phases. Exception is the way Java reports a runtime

error. Exceptions have two parts. Message and the stack trace. Message

tells you what has gone wrong. Stack Traces provide a complete call flow

through all the classes involved top-down as part of the runtime error.

The following is an example stack trace for an

ArrayIndexOutOfBoundsException

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4

at Test.run(Test.java:13)

at Test.<init>(Test.java:5)

at Test.main(Test.java:20)

In the above exception, the first line "Exception in thread "main"

java.lang.ArrayIndexOutOfBoundsException: 4" tells you that the JVM

encountered the reported exception when it tried to access the value at

array index 4. The java thread which encountered the exception is "main"

thread.

Next let's read through the stack trace. The rule of thumb for exceptions is

to read from the top line (message line) to understand what the exception is

and keep going down to understand the call flow. In the above example, the

call flow started at Test.java line number 20 (main() method) and then it

called the constructor of the Test class. Constructors are represented with

<init> in the stack traces. Then it moved on to the run() method in the Test

class and encountered the reported exception at line number 13.

From the above stack trace, we can conclude that in Test.java, an attempt is

made to read past the size of the Array.

The Java Thread Dump

Java Thread dump can be defined as a snapshot of all threads running inside

a Java Virtual Machine (JVM) at a given point of time. A thread dump might

contain a single thread or multiple threads. In multi threaded environments

like J2EE application servers, there will be many threads and Thread

Groups. Each of these threads will have its own call stack and will be

performing a separate function at a given time. Thread Dump will provide

the stack traces of all the JVM threads and much more information about a

particular thread.

The Java VM Process and Java Threads

The Java Virtual Machine or JVM is an operating system level process. Java

Threads are child processes or Light-Weight Processes (in Solaris terms) of

the JVM process.

Generating Java Thread Dumps

Thread Dump is generated by sending a SIGQUIT signal to the JVM process.

There are different ways of sending this signal to the process.

In Unix, use "kill -3 <pid>" where pid is the Process ID of the JVM.

In Windows, press CTRL+BREAK on the window where the JVM is running

Java Thread State

Each Java Thread can be in one of the four states during its lifetime.

Runnable - The thread is either running or ready to run when it gets its

CPU turn. JRockit thread dumps refer to this state as ACTIVE.

Waiting on Monitor - The thread is either sleeping or waiting on an object

for said period of time or waiting to be notified by another thread. This will

happen when any of the sleep() methods on Thread object or wait() method

on an Object is called.

For example, in WebLogc server the idle execute threads are in this

condition and they wait till a socket reader thread notify them of some new

work to be done. The stack trace will look like this:

"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5

tid=0x1752F040 nid=0x180c in Object.wait() [1887f000..1887fd8c]

at java.lang.Object.wait(Native Method) waiting on <04134D98> (a

weblogic.kernel.ExecuteThread)

at java.lang.Object.wait(Object.java:426)

at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:126)

locked <04134D98> (a weblogic.kernel.ExecuteThread)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:145)

Some other versions of the JVM call this state as CW, Object.wait() (as

above). JRockit refer to this as state as WAITING.

Waiting for Monitor Entry - The thread is waiting to get the lock for

an object (some other thread may be holding the lock). This happens if two

or more threads try to execute synchronized code. Note that the lock is

always for an object and not for individual methods.

Sample stack trace of a thread in this condition:

"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5

tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]

at weblogic.cluster.replication.ReplicationManager.createSecondary

(ReplicationManager.java:908)

- waiting to lock <6c4b9130> (a java.lang.Object)

at weblogic.cluster.replication.ReplicationManager.updateSecondary

(ReplicationManager.java:715)

at weblogic.servlet.internal.session.ReplicatedSessionData.syncSession

(ReplicatedSessonData.java:459)

- locked <6c408700> (a

weblogic.servlet.internal.session.ReplicatedSessionData)

at weblogic.servlet.internal.session.ReplicatedSessionContext.sync

(ReplicatedSessionContext.java:134)

- locked <6c408700>

(aweblogic.servlet.internal.session.ReplicatedSessionData)

at weblogic.servlet.internal.ServletRequestImpl.syncSession

(ServletRequestImpl.java:2418)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet

(WebAppServletContext.java:3137)

at weblogic.servlet.internal.ServletRequestImpl.execute

(ServletRequestImpl.java:2544)

at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)

In the above stacktrace, you can see that this thread holds the lock an

object (6c408700) and waiting to lock another object (6c4b9130)

Some other JVMs may not give the object Ids with lock information in the

stack trace. The same state may also be called as ‘MW'. JRockit refers to

this state as LOCKED.

Anatomy of a Java Thread

To be able to read/analyze thread dump, it is important to understand

various parts of a Thread dump. Let's take a simple Thread Stack example

and read various parts of it.

"ExecuteThread: '1' " daemon prio=5 tid=0x628330 nid=0xf runnable

[0xe4881000..0xe48819e0]

at com.vantive.vanjavi.VanJavi.VanCreateForm(Native Method)

at com.vantive.vanjavi.VanMain.open(VanMain.java:53)

at

jsp_servlet._so.__newServiceOrder.printSOSection( __newServiceOrder.java:

3547)

at jsp_servlet._so.__newServiceOrder._jspService

(__newServiceOrder.java:5652)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet

(ServletStubImpl.java:265)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet

(ServletStubImpl.java:200)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet

(WebAppServletContext.java:2495)

at weblogic.servlet.internal.ServletRequestImpl.execute

(ServletRequestImpl.java:2204)

at weblogic.kernel.ExecuteThread.execute (ExecuteThread.java:139)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

In the above Thread Dump, the interesting part to is the first line. The rest

of the stuff is nothing more than a general stack trace. Lets analyze the first

line here

Execute Thread : 1 This indicates the name of the thread.

daemon Indicates that this Thread is a daemon thread

prio=5 The priority of this thread (default is 5)

tid The Java Thread Id (Unique identifier for this thread in the running JVM

instance).

nid Native Identifier of the thread. Indicates LWP id in Solaris. Identifier for

this process

at the OS level.

runnable Indicate the Thread state (See above)

[x..y] Indicates the range of addresses in the heap where this thread is

executing.

The remaining part of the thread dump show the call flow. In this example,

the thread (Execute Thread 1) is a OS daemon which is currently executing

a native method vanCreateForm().

Putting Thread Dumps to Use

In this section I describe few use cases in which Thread dumps are very

useful.

High CPU consumption

Diagnosis

The application seems to consume almost 100% CPU and the throughput

has come down drastically. Starts to perform very poorly with high load on

the CPU.

Thread Dump

All threads in the dump generally show one or more threads struck doing

the same operation through all the thread dumps.

Solution

Take a series of thread dumps (around 5-7) for one particular call flow

(a web form submit for example) before the call is completed.

Looking for "runnable" threads in the Thread Dump. If each such

thread seems to move (the method call in each thread dump is

different), then the thread is progressing and probably not a culprit. If

the thread is executing the same method (same line number) through

all the thread dumps, then it is almost for sure the culprit. Go to the

code and do the code level analysis. You are almost certain to catch

an issue there.

Low CPU consumption and Poor Response time

Diagnosis

This happens in a high I/O bound systems under high load. The CPU

consumption is low with only few threads consuming reasonable CPU

chunk. However the application response time will be very slow.

Thread Dump

Some or all runnable threads seem to struck performing an I/O operation

like a File read/write or a database operation.

Solution

Understand the I/O operation profile of your application. Use caches where

applicable to reduce DB interactions.

Application/Server Hang

Diagnosis

An application or an server JVM hosting the application will hang (become

unresponsive).

Thread Dump

All runnable threads are struck doing the same operation through all

the thread dumps taken. The server runs out of any more threads as

all the runnable threads "never" complete their operation.

There might be many threads waiting for monitor entry. This can

happen if a "runnable" thread is holding a lock on an object and never

returns it while others are waiting for the same lock.

Solution

Check for dead lock. JVM generally detects a deadlock if it is simple

enough (Thread A waiting for Thread B and vice versa). However, you

need to understand the lock profile at the given moment to see if

there is any complex deadlock situation involved.

Revisit the synchronized methods/block in the code. Reduce the size

of synchronization area to the extent possible.

One issue could be too long a timeout while accessing a remote

resource/component. Implement a reasonable timeout on the remote

object clients so that the throw appropriate exception when a remote

system does not respond in a reasonable amount of time.

If all the threads are waiting for a resource (like an EJB/DB

connection), consider increasing the object pool size for those

resources.

Tools

There are both commercial and open source tools available for thread dump

analysis. One such tool is Samurai. It is a light weight open source tool

which runs as a Java Web Start application as well as from your command

prompt. For more information and documentation on Samurai, visit

http://yusuke.homeip.net/samurai/en/index.html

Conclusion

Maintaining J2EE enterprise application in a production environment is a

tough job. As the business dynamics change, J2EE application environments

change which might cause runtime instability in the production application.

One of the primary factors that effect an running J2EE application is high

load. While most systems are designed to be scalable, environmental

limitations might cause them to become non-responsive.

Java Thread Dumps is an excellent mechanism to identify, diagnose, detect

and resolve typical production issues. While application profiling and other

mechanisms do exist, analyzing Java Thread dumps will provide us an clear

and early understanding of popular production level issues there by saving

time and helping us provide better user experience with the production

applications.

Of Thread dumps and stack traces …Tags: core, debug, java, performance — Rajiv @ 10:45 am UTC

Thread dumps and stack traces are probably some of the least understood features of java. Why else would I come across developers who have no clue what do do after looking at an Exception stack trace? Street Side Programmer?!Anatomy of a stack traceInnovative usage of stack tracesThread dump 101Debugging run away CPUDebugging performance issuesDebugging “hang” problems

Street Side Programmer?!An ex-colleague of mine, Manoj “The Anger” Acharya, had coined the phrase Street Side Programmer [a la Server Side Programmer] and he would dole out this title to all those who would come to him with annoying questions. Nothing annoyed him more than having some one come and ask him I am getting some exception when I do *blah* *blah*. His typical answer *bleep*’ing Street Side Programmers … what is some exception supposed to mean?! Doesn’t it have a name? Doesn’t it have a stack trace??I was reminded of him the other day, when a trainee learning java came to me saying My program is not running … there seems to be some problem … can you come and take a look?. The kid is quite sweet, so instead of telling him about Anger, I just went to his seat. The command prompt had something like this:

C:\learn\classes>java TestException in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5 at Test.run(Test.java:11) at Test.<init>(Test.java:4) at Test.main(Test.java:19)

I wonder if the book he was reading had any section on reading stack traces. [Monsieur Bruce Eckel ... are you listening?!] I really think any introduction to java book should have this as one of the earliest chapters … right after defining a class and method! Some one makes an error while trying out samples, or is tinkering around with the code, which typically results in an exception … what is one supposed to do next?!Anatomy of a Stack TraceWell I explained that an exception stack trace is java’s way of telling you exactly what went wrong and where it went wrong. The first line of the stack trace gives you the exception name and the exception message and what follows is the “stack trace”. The stack trace is to be read from top to bottom, line by line. Each line has the name of the class and the name of the method being executed followed by the file name and line number in parentheses.  In this case a java.lang.ArrayIndexOutOfBoundsException with the message “5” was raised. To know where it was raised, we look at the next line. It tells us that the exception was raised while executing the run method of the Test class at line number 11 in Test.java file. The next line tells us that the run method was called by the constructor [the stack trace shows Constructors as <init> and static blocks in a class as <clinit>] of the Test class at line number 4 in Test.java file. The next line tells us that the constructor was called by the main method of Test class at line number 19 of Test.java file. So the java stack trace would read in English would be like: You accessed an array with an index of 5, however the array does not have 6 elements [thanks to zero based index]. This happened when I was executing the run method of Test class which happens to be in line number 11 of Test.java file. The run method was called by the constructor of Test class at line number 4 of Test.java file. The constructor was called by the main method of the Test class at line number 19 of Test.java file.  Well… there is a wealth of information here. It tells you exactly what the VM was doing when the exception was raised. Let us see how to

debug the issue given all this information. The Test.java file looks like this: 

1 public class Test2 {3 public Test(int[] nums){4 run(nums);5 }67 private void run(int[] nums)8 {9 int n = nums.length;10 for (int i = 0; i < nums.length; i++) {11 int num = nums[n];12 System.out.println(num);13 }14 }1516 public static void main(String argv[])17 throws Exception18 {19 new Test(new int[]{1,3,5,7,9});20 }21 }

Stepping through the code [as per the stack trace], we called the main method, which invoked the constructor at line number 19, which in turn called the run method at line number 4. Hey the stack trace was correct after all! Now we look at line number 11 where the exception was raised. The exception says that we accessed an array with incorrect index. The only array we are accessing at line number 11 is the num array. The index being used to access the array was n. So what the VM is trying to tell you is that the n is larger that the size of the array nums. Which is in fact true. n happens to be the length of the array. So it IS greater than the last index of the array. What the user really wanted to do was use i as the loop index and not n. Another common exception raised is the java.lang.NullPointerException. The NPE!  A NullPointerException is Java’s way of telling a user that a null object reference was being used. Take a look at the following lines from a stack trace [snipped for brevity]:

java.lang.NullPointerException: at foo.bar.MyServlet.doGet(MyServlet.java:36)

So now we know that a null object was being referened at line number 36 of MyServlet.java. The code for the servlet looks something like:

35 String userNameParam = request.getParameter("username");36 if(userNameParam.equals("root"))37 {

The only object reference being used in line number 36 happens to be userNameParam. So it was null when the VM was executing that line. Now we track down what values were assigned to the userNameParam. Line number 35 happens to be the only assignment in this case. It assigns the value of userNameParam to request.getParameter("username"). Since the VM told us that the userNameParam was null, it means that the method request.getParameter("username") returned a null value. Looking at the documentation of the method we know that the method may return null. So the users of the method need to code taking that into consideration. In this case we would change the condition like so:  

35 String userNameParam = request.getParameter("username");36 if(userNameParam!=null && userNameParam.equals("root"))37 {

Thanks to stack traces some one who is not even aware of the code can pin point the exact location of the error. In most of the cases a stack trace is definitely starting points for debugging erroneous behavior. Who wants messycore dumps anyways when you have readable stack traces?! Innovative uses of stack tracesOnce you know what a stack trace provides, there are a lot of innovative ways to use it. Basically answer questions like how did I get here or to record the location of an event.Recently a customer noticed that the VM was performing Full GC’s very frequently. This would happen even when the application is completely idle. Looking at java -verbose:gc -XX:+PrintGCTimeStamps ... , we

realized that the Full GC would occur every one minute … on the dot. We then tried adding the -XX:+DisableExplicitGC option and voila no more full GCs! So looks like some one was doing a System.gc somewhere at every one minute.So how do we find out who is calling it?! You would extract the System.java file from <jdk-home>/src.zip!/java/lang/System.java and edit it like so:

736 public static void gc() {737 new Exception("Some one triggered Full GC from here").printStackTrace();738 Runtime.getRuntime().gc();739 }740

Compile the modified file and prepend it to your bootclasspath using the option -Xbootclasspath/p:outputDir. Next time we ran the application, we got the stack trace:

java.lang.Exception: Some one triggered Full GC from here at java.lang.System.gc(System.java:737)

at sun.misc.GC$Daemon.run(GC.java:92)

Adding one more stack trace to GC.java [You will not find sources for the com.sun.* and sun.* packages in the src.zip that comes with your jdk. You will have to download it from Sun's Community Source site.] we get to know that sun.rmi.transport.ObjectTable is triggering the full GC based on an interval specified by the system property sun.rmi.dgc.server.gcInterval. The default value for the property happens to be one minute.So using the printStackTrace method we could debug where Full GC was being triggered explicitly. You couldofcourse do the same by setting a method break point for the System.gc method. Or you could bea smart google’er and stumble upon the “Other considerations” section of the GC options page!

Instead of doing a new Exception(…).printStackTrace(), you could alternatively do a Thread.dumpStack() which internally does the same. The only disadvantage is that Thread,dumpStack() does take a message as itsparameter.Some times it makes sense to create an exception object and hold a reference to it until a later point in time. Suppose you have a class which looks like:

1 import java.io.IOException;23 /** 4 * A class that represents a heavy weight resource.

5 */6 public class Resource7 {8 private boolean closed;910 public void close() throws IOException{11 if(closed)12 throw new IOException("Resource already closed.");13 //resource cleanup

14 closed=true;15 }16 //code snipped for brevity ...

The class throws an exception when a user invokes close on an already closed resource. The stack trace of the IOException is going to tell you where in the code you tried to close the already closed connection. For example the following output tells you that when you called close on the Resource at line 41 of ResourceTest.java it was already closed.

C:\learn\classes>java ResourceTestjava.io.IOException: Resource already closed. at Resource.close(Resource.java:12) at ResourceTest.closeResource(ResourceTest.java:37) at ResourceTest.run(ResourceTest.java:26) at ResourceTest.main(ResourceTest.java:50)

But now what if you want to know where did you close it the first time?! You would change the code like so:

1 import java.io.IOException;23 /** 4 * A class that represents a heavy weight resource. 5 */

6 public class Resource7 {8 private boolean closed;910 private Exception closedAt;1112 public void close() throws IOException{13 if(closed) {14 closedAt.printStackTrace();15 throw new IOException("Resource already closed.");16 }17 //resource cleanup

18 closed=true;19 closedAt=new Exception("Resource closed here the first time.");20 }21 //code snipped for brevity ...

The output would after making the changes would look like …

C:\learn\classes>java ResourceTestjava.lang.Exception: Resource closed here the first time. at Resource.close(Resource.java:19) at ResourceTest.useResource(ResourceTest.java:32) at ResourceTest.run(ResourceTest.java:25) at ResourceTest.main(ResourceTest.java:50)java.io.IOException: Resource already closed. at Resource.close(Resource.java:15) at ResourceTest.closeResource(ResourceTest.java:41) at ResourceTest.run(ResourceTest.java:26) at ResourceTest.main(ResourceTest.java:50)

So now from the stack traces we know that a close was called first at line 32 of ResourceTest.java and later at line 41 we called a close on the same resource for the second time.There are a lot of multi threaded problems [NullPointers] which we were not able to debug with a debugger because the whole application would become too slow to simulate the problem scenario. However, by using Exception objects to track threads which were setting the fields to null,

we were able to resolve the issues. A word of caution though … creating exception objects is resource intensive. Creating too many exception objects takes lot of CPU. And if you are holding references to all the objects it requires memory too! Thread dump 101If the stack trace which gives the information on what a thread was doing at that moment can help us in so many ways, just imagine the possibilites if you could find out what every single thread in the Java VM is doing at any given moment! A Full Thread Dump or a thread dump for short gives us exactly that information. Consider the following source

1 public class Test2 {3 public Test(char[] chars){4 System.out.println("New line at "+findNewLine(chars));5 }67 private int findNewLine(char[] chars)8 {9 int i = 0;10 char aChar;11 do{12 aChar = chars[i];13 }while(aChar!='\n');14 return i;15 }1617 public static void main(String argv[])18 throws Exception19 {20 new Test("Hello World!\nHowz goin?!">.toCharArray());21 }22 }

The method findNewLine is supposed to return the first index of a new line character in a given char array. [Purists please don't mail me with the list of reasons why this approach is not right ... the idea here is not really to write the best way to find a new line character!] Now when you run the program it just won’t print the result. One look at top in unix or the task manager in windows we get to know that VM has taken the CPU for a spin…. 100% CPU consumption for ever! Now wouldn’t you

want to know what the VM is doing. Why is it taking all this CPU and not printing the output it is supposed.  One way to do this would be rerun the program in debug mode. Use the debugger and debug the application. However, many a times you come across such the situation on a live system after running the app for a long duration. Since it is a live system and we hit the issue only after running the application for a long duration we can not leave it in debug mode for ever. The first line of defense under such circumstances is the thread dump. Run the program from the command prompt and when the CPU peaks take a thread dump. You can get a thread dump by pressing the following at the command prompt: Ctrl+\ for unices or Ctrl+Break for windows machines. If you are running your application as a back ground process in unix, you could execute kill -SIGQUIT <pid> from another command prompt. The above signals the VM to generate a full thread dump. Sun’s VM prints the dump on the error stream while IBM’s JDK generates a new file with the thread dump every time you send the signal. In our case the thread dump would look something like this:

C:\learn\classes>java TestFull thread dump Java HotSpot(TM) Client VM (1.4.2_04-b05 mixed mode):

"Signal Dispatcher" daemon prio=10 tid=0x0091db28 nid=0x744 waiting on condition [0..0]

"Finalizer" daemon prio=9 tid=0x0091ab78 nid=0x73c in Object.wait() [1816f000..1816fd88] at java.lang.Object.wait(Native Method) - waiting on <0x10010498> (a java.lang.ref.ReferenceQueue$Lock)

at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x10010498> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x009196f0 nid=0x738 in Object.wait() [1812f000..1812fd88] at java.lang.Object.wait(Native Method) - waiting on <0x10010388> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Unknown Source) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x10010388> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x00234998 nid=0x4c8 runnable [6f000..6fc3c] at Test.findNewLine(Test.java:13) at Test.<init>(Test.java:4) at Test.main(Test.java:20)

"VM Thread" prio=5 tid=0x00959370 nid=0x6e8 runnable

"VM Periodic Task Thread" prio=10 tid=0x0023e718 nid=0x74c waiting on condition"Suspend Checker Thread" prio=10 tid=0x0091cd58 nid=0x740 runnable

The thread dump generated here is on Sun’s JDK 1.4.2. Though the output differs from version to version and from vendor to vendor, the basic structure is the same. The output is somewhat like going over all the threads and doing a Thread.dumpStack in each of them. In this case we can see that, at the time we took the thread dump, there were seven threads:

1. Signal Dispatcher 2. Finalizer 3. Reference Handler 4. main 5. VM Thread 6. VM Periodic Task Thread 7. Suspend Checker Thread

Each thread name is followed by whether the thread is a daemon thread or not. Then comes prio the priority of the thread [ex: prio=5]. I am not sure what the tid and nid are. My best guess is that they are the Java thread id and the native thread id. Would love if someone could comment on that. Then what follows the state of the thread. It is either:

Runnable [marked as R in some VMs]: This state indicates that the threadis either running currently or is ready to run the next time the OS threadscheduler schedules it.

Suspended [marked as S in some VMs]: I presume this indicates that thethread is not in a runnable state. Can some one please confirm?!

Object.wait() [marked as CW in some VMs]: indicates that the thread iswaiting on an object using Object.wait()

waiting for monitor entry [marked as MW in some VMs]: indicates that thethread is waiting to enter a synchronized block

What follows the thread description line is a regular stack trace.  Debugging run away CPU

When we are trying to debug a run away CPU, as in this case, what we need to look at is the set of Runnable threads  in the thread dump. The question to ask is: What was the thread which was consuming CPU doing? At the instant we took the above thread dump, the thread was at line 13 of Test.java. Well … looks like it was checking the condition for the while loop. But eventually it should have returned right?! So we take a few more thread dumps. Each time it shows us the thread is within the while loop. This definitely indicates from the first time you took a dump to the last time you took a dump, the thread never got out of the loop. The problem is narrowed down that loop. Putting the loop under the magnifying glass, we realize that the counter i was never being incremented. Well … if you have a single class in your application it is no big deal! But when you have gazillions of classes, narrowing down the problem to a single loop within single class is a big saver! I have found this a useful tool even when I am using a debugger. It helps me choose a good location to set my first break point! Debugging performance issuesIts the night before the release and your application is not performing good enough. You really don’t have enough time to run the app through a profiler. Take heart! Like Ramesh says … there are always some low hanging fruits! The way a java profiler works is, it takes snapshots of what the CPU was doing at frequent intervals and generates a statistical report on where most of the CPU time was being spent during the run. If your application is performing so poorly that you could take say 10-12 thread dumps before an operation completes, you would get a rough idea of distribution of CPU time. Some of the easy kills I can think of:

Symptom: High CPU consumption and poor response time

Thread dump profile: Most of the dumps show the same thread in the same method or same class

Solution: The method/class is the one which is definitely taking a lot of CPU. See if you can optimize these calls. Some of the REALLY easy kills we have had in this category is using a Collection.remove(Object) where the

backend collection is a List. Change the backed collection to be a HashSet. A word of caution though: There have been times when the runnable threads are innocent and the GC is the one consuming the CPU.

Symptom: Low CPU consumption most of which is kernel time and poor response time

Thread dump profile: Most thread dumps have the runnable threads performing some IO operations

Solution: Most likely your application is IO bound. If you are reading a lot of files from the disc, see if you can implement Producer-Consumer pattern. The Producer can perform the IO operations and Consumers do the processing on the data which has been read by the producer. If you notice that most IO operations are from the data base driver, see if you can reduce the number of queries to the database or see if you can cache the results of the query locally.

Symptom: Medium/Low CPU consumption in a highly multithreaded application

Thread dump profile: Most threads in most thread dumps are waiting for a monitor on same object

Solution: The thread dump profile says it all. See if you can: eliminate the need for synchronization [using ThreadLocal/Session-scopeobjects] or reduce the amount of code being executed within the synchronized block.

Symptom: Medium/Low CPU consumption in a highly multithreaded application

Thread dump profile: Most threads in most thread dumps are waiting for a resource

Solution: If all the threads are choked for resources, say waiting on the pool to create EJB-bean objects/DB Connection objects, see if you can increase the pool size.

Debugging “hang” problemsA textbook case of deadlock is the easiest to debug with the newer JDKs. At the end of the thread dump you will find something like this:

Found one Java-level deadlock:============================="Thread-1": waiting to lock monitor 0x0091a27c (object 0x140fa790, a java.lang.Class), which is held by "Thread-0"

"Thread-0": waiting to lock monitor 0x0091a25c (object 0x14026800, a java.lang.Class), which is held by "Thread-1"

Java stack information for the threads listed above:==================================================="Thread-1": at Deadlock$2.run(Deadlock.java:48) - waiting to lock <0x140fa790> (a java.lang.Class) - locked <0x14026800> (a java.lang.Class)"Thread-0": at Deadlock$1.run(Deadlock.java:33) - waiting to lock <0x14026800> (a java.lang.Class) - locked <0x140fa790> (a java.lang.Class)

Found 1 deadlock.

But many a times we come across hang’s which are not deadlocks. One thing that easily comes to my mind is a resource limit. For example in an EJB container you have set the maximum bean pool size to 1000. Now say two threads have started executing a finder each returning a collection of 1000 odd beans. Assuming a decent CPU time slice distribution it could happen that the first thread iterates over 500 beans and the next thread iterates over the 500 beans. At this moment both the threads need more beans to proceed further. However the container will not create new beans as the bean pool limit has been reached. So both the threads wait for some beans to be release to the pool … which is not going to happen. We have a hung app here…. however it is not a java-level deadlock. It is an artificial deadlock introduced due to resource limitation. When your app is not responding and your CPU consumption is 0%, take a thread dump. If it does not have a java level dead lock, then take multiple thread dumps. If all of them show that the threads are waiting

for resources [EJBs or DBConnections] see if you can increase the pool limit or decrease the number of resources required within a transaction.  FinallyThread Dumps and stackTraces are really good tools … they may not replace a debugging/profiling tools but are definitely good starting points and huge time savers. Unfortunately, I think they are undersold. Classes don’t teach you about them, Books don’t talk about them and tools don’t support them. I mean I can run any class from my IDE. I has buttons to Start/Pause and Stop the app from within the IDE. But why can’t I have a button for “Generate full Thread dump”. Every time I need to generate a thread dump, I have to rerun the application fromcommand line.Well … maybe things are not so bad after all. What if the IDEs don’t support generation of a thread dump?! Most of them now open up the file and line number if you double click on a line in the exception stack trace obtained on runninga program! And what if the books don’t talk about it? People like Ashman make sure anyone joining the support team gets their dope on thread dump from me!