1. Matthew Kilner IBM Java L3 Service Core team lead 23rd September 2013Windows Debugging Tools 2013 IBM Corporation

2. Important Disclaimers THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBMS CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS2 2013 IBM Corporation 3. About me Matthew Kilner Work for IBM 13 years working on IBM Java Memory Management Class Sharing RAS Currently leading the Core customer support team. Contact info kilnerm@uk.ibm.com Twitter: @IBMJTC Youtube: IBM_JTC Visit the IBM booth #5112 and meet other IBM developers at JavaOne 2013 3 2013 IBM Corporation 4. What should you get from this talk? Awareness of some excellent windows tools: A high level understanding of what it is they let you do, Details of where to obtain the tools, Details of where to find out how to use them. Some useful tips on working with the tools 2013 IBM Corporation 5. General tips Most Microsoft tools require symbol files, often known as .pdb, to enable method names and source lines in stack traces. Always keep your symbols in one place and keep a pre-configured Symbol path to hand. I will use generic paths for the JDK symbol files and overlay those paths with the correct level of symbol for the JDK I am debugging at the time. JDK symbol files ship in the majority of the IBM JDK packaging formats. Remember to include the MS symbol server: SRV**http://msdl.microsoft.com/download/symbols5 2013 IBM Corporation 6. PDB files PDB files are an essential part of effective debugging The following link provides deailed information on how to generate them: http://msdn.microsoft.com/en-us/library/aa260783%28v=vs.60%29.aspx Requires the addition of compile and link time flags.6 2013 IBM Corporation 7. IBM Presentation Template Full VersionThe tools Windows SDK tools: UMDH ADPlus Sysinternals tools: VMMAP Procdump Standard tools: Performance Monitor Stand alone tools: Debug Diagnostics7 2013 IBM Corporation 8. Usefulness is contextual A tools usefulness is limited to a number of problem types or tasks you are working on. Most of these tools have been discovered during analysis of real world problems. Predominantly related to Virtual addressing problems. Several of the tools have capabilities beyond the use described today.8 2013 IBM Corporation 9. Monitoring the system / application There are two common problem scenarios where it is desirable to understand more about how a system or application is performing: Virtual Addressing problems Performance problems Slow responsiveness, High CPU usage, etc.... 9 2013 IBM Corporation 10. Performance Monitor Where to get it: Included in you windows installation What is it: A powerful monitoring tool What does it let us do: Monitor and analyse performance data in real time or from a log file. Can track almost every aspect of system performance. How do you use it: Launched by: Entering perfmon in your run dialog, or Via the Performance Monitor entry on the start menu Read the in-built help documentation.10 2013 IBM Corporation 11. Performance Monitor monitoring process size The most common use is for accurate measurement of a processes size while debugging virtual addressing issues. Define a data collector Set to record the following counters on the process object: Virtual bytes Private bytes 11 2013 IBM Corporation 12. Performance Monitor monitoring process size For live monitoring add the same counters to the performance monitor view When loading Data collection sets back into the tool you will see similar output. 12 2013 IBM Corporation 13. Performance Monitor other uses We primarily use it for monitoring process sizes which only scratches the surface of its practical uses. The ability to monitor nearly anything you want on one or more processes such as: Monitoring per thread CPU consumption across a process. Monitoring the power consumption of individual CPU's.13 2013 IBM Corporation 14. Capturing Dumps Occasionally we see problems where the JDK does not write dumps when we expect it to. This provides a real challenge for continuing problem analysis. We use two tools that help capture dumps in this scenario: ADPlus (Windows SDK) Procmon (Sysinternals) 14 2013 IBM Corporation 15. ADPlus Where to get it: http://msdn.microsoft.com/en-us/library/windows/desktop/hh852363.aspx Part of the Windows SDK What is it: A useful tool for generating system dumps Can also be used in hang situations What does it let us do: Monitor a process and generate system dumps on: User request Process termination First and second chance exceptions. How do you use it: Documentation is included in the install directory Windows Kits8.0Debuggersx64ADPlus.doc 15 2013 IBM Corporation 16. Procdump Where to get it: http://technet.microsoft.com/en-us/sysinternals/dd996900 What is it: A useful tool for generating system dumps What does it let us do: Generate system dumps in a multitude of scenarios: The process exceeding a threshold on any performance counter On process exit On first and second chance exceptions. How do you use it: Command line utility See the link above for examples, or Run the tool with no options. 16 2013 IBM Corporation 17. Procdump usage examples Example: Write up to 3 dumps of a process named 'consume' when it exceeds 20% CPU usage for five seconds to the directory c:dumpconsume with the name consume.dmp: C:>procdump -c 20 -s 5 -n 3 -o consume c:dumpconsume Example: Write 3 dumps 5 seconds apart: C:>procdump -s 5 -n 3 notepad.exe notepad.dmp Example: Launch a process and then monitor it for excessive CPU usage: C:>procdump -c 30 -s 10 -x consume.exe consume.dmp17 2013 IBM Corporation 18. Understanding Virtual Memory Growth A common problem we encounter is the virtual address space of processes growing and either: Causing process termination due to an inability to allocate memory, or Causing severe performance issues due to paging There are three excellent tools available on Windows to help understand what is causing the growth: UMDH (Windows SDK) Debug Diag ( Stand alone) VMMAP (Sysinternals) 18 2013 IBM Corporation 19. UMDH Where to get it: http://msdn.microsoft.com/en-us/library/windows/desktop/hh852363.aspx Part of the Windows SDK What is it: A tool that tracks and analyzes windows heap allocations within a process What does it let us do: Take snapshots of memory footprint within a process broken down by allocation stack. Compare snapshots and summarize the key differences. Identify stacks for specific allocation sizes. How do you use it: Basic usage instructions: http://msdn.microsoft.com/en-us/library/windows/hardware/ff558947%28v=vs.85%29.aspx More detail is available in this Microsoft knowledge base article: http://support.microsoft.com/kb/268343 19 2013 IBM Corporation 20. UMDH sample output // Connecting to process 3312 ... // Process 3312 opened handle=48. // Loaded modules: Snapshot files contain the raw data from a point of execution of the process, you will see: A summary of loaded modules A summary fo the data format: A series of allocations: When you have multiple snapshots you run the compare tool.//Base Size Module//13A0000//773B0000 1A9000 C:windowsSYSTEM32ntdll.dll35000 C:Program FilesIBMCodeRallyjdkjrebinjavaw.exe// Process modules enumerated. // _NT_SYMBOL_PATH set by default to C:windowssymbols // Debug library initialized ... DBGHELP: ntdll - export symbols *- - - - - - - - - - Start of data for heap @ 310000 - - - - - - - - - REQUESTED bytes + OVERHEAD at ADDRESS by BackTraceID STACK if not already dumped. *- - - - - - - - - - Heap 310000 Hogs - - - - - - - - - 28 bytes + 28 at 311310 by BackTrace720C0 7744BBED 773DC2E2 773F357E 773F1937 773DC34E20 2013 IBM Corporation 21. UMDH comparing snapshots To run a comparison you execute: UMDH > Once the compare completes the output file contains a list of the call stacks that have differing memory usage between the dumps sorted by overall size: + 550376 ( 550776 - 400) 12 allocsBackTrace33599F60 + 11 ( 12 1) BackTrace33599F60 allocations ntdll!MD5Final+A63D J9PRT26!j9mem_allocate_memory+74 (c:cygwinhomeforemansandboxjvm-binsrcj9portcommonj9memtag.c, 184) j9vm26!allocateMemoryForSegment+A0 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 216) j9vm26!allocateVirtualMemorySegmentInListInternal+E6 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 357) j9vm26!allocateFixedMemorySegmentInList+A8 (c:cygwinhomeforemansandboxjvm-binsrcj9vmsegment.c, 319)21 2013 IBM Corporation 22. UMDH important considerations The tool uses snapshots of outstanding allocations so a degree of additional interpretation is needed. Do not assume the top consumer between two snapshots is the cause of the problem Also consider the volume of allocations, particularly for small chunks which have implications in terms of fragmentation pre Vista/Server 2008. Best used in conjunction with performance monitor and over several iterations of snapshots. Longer durations between snapshots often make it easier to identify leaks22 2013 IBM Corporation 23. Debug Diagnostic Tool Where to get it: http://www.microsoft.com/en-us/download/details.aspx?id=26798 What is it: A tool providing a range of capability for analysis of crash, hang or native memory isssues. What does it let us do: Analysis of native memory issues. Basic analysis of crash or hang scenarios. Attaches to live processes and can be used to generate dumps at request or by rule configuration. Designed for minimal performance impact How do you use it: Basic Instructions: http://support.microsoft.com/kb/2580960 Read the whitepaper: http://www.microsoft.com/download/en/details.aspx?id=23521 Use the in-built help documentation. 23 2013 IBM Corporation 24. Debug Diagnostic Tool native memory analysis Our main use of this tool is for debuging native memory issues. The key thing to remember is to always inject the leak track dll, without this the reports you generate will be basic. The optimal method of analysis involves taking multple dumps of the process as its virtual size grows, understanding the rate of growth between dumps is important and can be monitored using the performance monitor tool. Dumps can be generated directly from the UI or rules can be configured to generate dumps after a set amount of time or increase in memory usage. Once dumps are generated you load them into the tool and select to run the Memory pressure script. When complete the tool will open the report in a browser window. 24 2013 IBM Corporation 25. Debug Diagnostic Tool native memory report The report is broken into distinct sections: Virtual Memory Analysis Report Heap Analysis Report Leak Analysis Report (only with leak tracking) The following extracts are taken from results of monitoring a 64 bit process which has a maximum address space of 8 Tbytes. Each report starts with a summary which contains some key pieces of information: 25 2013 IBM Corporation 26. Debug Diagnostic Tool Virtual Memory Analysis Report Contains: Virtual Memory Summary Virtual Memory Details Virtual Allocation Summary Loaded Module Summary Thread Summary The most pertinent details are the first three elements 26 2013 IBM Corporation 27. Debug Diagnostic Tool Heap Analysis Report The heap analysis report contains information on the Heaps in use within the process. Not to be confused with the JVM's heap. 27 2013 IBM Corporation 28. Debug Diagnostic Tool Heap Details For each heap you get: A summary Information on the Heap segments Information on allocations The Summary: 28 2013 IBM Corporation 29. Debug Diagnostic Tool Heap Details The Segment info:29 2013 IBM Corporation 30. Debug Diagnostic Tool Heap Details The Allocation Summary:30 2013 IBM Corporation 31. Debug Diagnostic Tool Leak Analysis Report Only available when the leaktrack dll is injected to the process being monitored. The most useful aspect is the summary of allocations at the module level: By Count: 31 2013 IBM Corporation 32. Debug Diagnostic Tool Leak Analysis Report and by Size: 32 2013 IBM Corporation 33. Debug Diagnostic Tool Module Summary For each module we get a more detailed breakdown including a summary: 33 2013 IBM Corporation 34. Debug Diagnostic Tool Module Summary We get a summary of functions by count and allocated size: 34 2013 IBM Corporation 35. Debug Diagnostic Tool Module Summary And for each function additional detail: 35 2013 IBM Corporation 36. Debug Diagnostic Tool Module Summary Allocation count and size breakdown: 36 2013 IBM Corporation 37. Debug Diagnostic Tool Module Summary And sample call stacks for prominent sizes: 37 2013 IBM Corporation 38. Debug Diagnostic Tool Drawing conclusions The tool provides a lot of information, but doesn't do comparison of multiple data sets. Using a spreadsheet to capture the pertinent data points highlighted over the previous slides will build a picture that allow you to see the key areas that are causing growth. You can even plot the data for easier visualization 38 2013 IBM Corporation 39. VMMAP Where to get it: http://technet.microsoft.com/en-us/sysinternals/dd535533 What is it: A tool for analysis the virtual and physical memory usage of a windows process What does it let us do: Attaches to live processes for run time analysis and allows for export of process information in a variety of formats. Provides a facility to launch a process and trace it which utilizes additional instrumentation that enables us to identify the stacks allocating into specific regions of memory. Allows us to compare the contents of the address space between two different points in time, when tracing is enabled you can see exactly who is allocating into regions. At any time you can flush the physical memory the process is using meaning this is a great tool for seeing just how much memory a particular feature of your application uses. 39 2013 IBM Corporation 40. VMMAP Having attached or launched a process you will see a window like this: Each time you refresh the view a new snapshot is created40 2013 IBM Corporation 41. VMMAP Comparing snapshots First you have to open the timeline window: Next you left click and drag to select the timeframe you wish to view: Snapshots are represented by the small black lines:41 2013 IBM Corporation 42. VMMAP Comparing snapshots Once selected you will see:42 2013 IBM Corporation 43. VMMAP Comparing snapshots Address ranges that are in the most recent snapshot but not in the earliest are highlighted in bright green. Address ranges that were in the earliest snapshot but not in the current are highlighted in bright red. If you started the process under full tracing you can select a HEAP and view the allocations made within it between the snapshots, for each allocation you can also view the call stack. 43 2013 IBM Corporation 44. VMMAP other features When tracing is enabled you have the option to view a trace of key memory operations by selecting the trace option which lists: Allocation sites Number of allocations for the site The number of bytes allocated The type of operation You can also select any site and choose to view its stack.44 2013 IBM Corporation 45. Q&A Questions45 2013 IBM Corporation 46. IBM@JavaOnehttp://ibm.co/JavaOne201346 2013 IBM Corporation