.NET Debugging Techniques

Bala Subra

Software Bugs are Expensive

Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.- Brian Kernighan

Why Debugging?

We need Reliable Software Users Choose Reliability over Price “Software bugs or errors are so prevalent and

so detrimental that they cost the US economy an estimated $59 billion annually, or about 0.6 percent of the GDP.” www.nist.gov/public_affairs/releases/n02-10.htm

Most Commonly Found Defects

http://scan.coverity.com/report/Coverity_White_Paper-Scan_Open_Source_Report_2009.pdf

Technical Requirements

Understand unmanaged (native) vs. Managed debugging.

Explore production-environment debugging. Deep-dive into CLR fundamentals. Understand .NET memory management. Debug hangs, crashes, and memory leaks.

Agenda

Importance of debugging Available Tools Basic Tasks and Walkthroughs Postmortem Debugging

Common Traits for good debuggers Willingness to venture outside of your "own"

code Curiosity Patience Treat dependent code as just that - code Ability to see patterns

Importance of debugging

Perfect code is an Illusion Legacy Code Deeper Understanding Helps you learn & write better code in the

future

Debugging Basics What are you trying to find

and fix? Two main types of code

errors Syntax: Compiler catches

most if not all of these for you.

Semantic or logical: Syntactically correct yet program may “crash and burn” at run-time!

Autos Locals Watch Call Stack Command Window QuickWatch Dialog Breakpoints window Threads Modules Processes Memory Disassembly Registers

Execution Control: Breakpoints Stepping through your

code Starting / Stopping Breaking Stepping through your

application (F10, F11 or Toolbar

buttons) Run to a specific location

Run To Cursor (right-click menu)

Situations under which breakpoints are not feasible Timing sensitive issues Breakpoint triggers too

often Live Debug not possible Debugging production

systems

Tools available for Debugging .NET Visual Studio 2008 (& soon 2010)

2010: Historical Debugging Visualizations for locks, threads

CorDBG / MDbg Debugging Tools for Windows

Focus for Today Used by everybody for everybody

Visual Studio 2008: Debugger Tips and Tricks Execution Control Breakpoints Symbols Multi-Threaded debugging Post Mortem Debugging Behind the Debugger Magic

VS Debugger Overview

Debugs many different code Environments Native Windows

X86 X64 IA64

Managed Code Windows (32 & 64 bit) SQLCLR

Script T-SQL Native Device Programs

VS Debugger Architecture

VSDebugPackageVSDebugPackage

SDMSDM

CPDE(Managed)

CPDE(Managed)

NatDbgDE(Native)

NatDbgDE(Native)

Your EngineHere

Your EngineHere

Your EE HereYour EE Here

http://msdn.microsoft.com/en-us/library/bb161718.aspx

Execution Control

Step Filtering for Managed Code Adds support for automatically stepping over

simple properties Right click to “Step into Specific”: pick Step into

Target Switch off: How?

Breakpoints Tracepoints

Print a variety of program state types without stopping Program Location, including Stack Expressions (including @clk in native for quick timing) Thread Info <Your Message Here>

Run a Macro Useful on stop on certain condition that is difficult to

express as a ‘Watch window’ expression. For eg. “Stop if this method is in Call Stack”

Visual Studio 2010

Tagging Filtering Import from others & Export Breakpoints

Symbols

Ensure Symbols are switched on in Final/Retail/Optimized Builds

Archive Symbols using Symbol Server VS 2010

Team Build support for Symbol Server Add Symbol & Source indexing into the Build’s

Workflow

Symbols: Reference Source Support

Symbols: Visual Studio Options

Symbol Loading Internals VS Never loads Mismatched symbols Path Plan:

Where the EXE think it is On the path we create from the “Symbols” Dialog On the path at HKLM/HKCU

Software\Microsoft\VisualStudio\MSPDB\SymbolSearchPath

On the path @ Any of these: _NT_ALT_Symbol_Path _NT_Symbol_Path SystemRoot

Threads Thread Categories Flagging Threads in the List (for tracking) Using Breakpoint Filters Freezing & Thawing Stack Tips Naming Threads

Managed: Thread.Name Native: Use the SetThreadName execution

wrapper http://blogs.msdn.com/stevejs/archive/2005/12/19/

505815.aspx

Post Mortem Analysis

Windows Error Reporting http://winqual.microsoft.com http://msdn.microsoft.com/en-us/library/aa939342.a

spx

Debugging Open the Disassembly Window Open the Autos window to see pertinent Registers http://blogs.msdn.com/greggm/archive/2004/12/15/315673.

aspx

Trust Statics/Globals Trust Stacks when you have Symbols

VS2010: Managed/Interop Support Support for reading & writing Minidumps from

Processes with Managed Code Support for Mixed Mode debugging on x64

How does the Debugger do Minidumps Use dbgHelp.dll Method:

MiniDumpReadDumpStream to read streams from MiniDump File

Read the following Streams SystemInfoStream ThreadListStream ModuleListStream MemoryListStream

Create Container Objects in the debugger that wrap the instances from the MiniDump

Wrap memory as needed by StackWalking or Data Inspection

Benefits of Debugging Tools for Windows Small Footprint

XCopy Enabled Ideal for debugging problems on machines that are

locked down Frequent releases

Updated for new versions of Windows Which debuggers does it include?

User Mode Debuggers: windbg/ntsd/cdb Kernel Mode Debugger: kd

Powerful Extensions & Instrumentation Extensible by us

Debugging: Package Content

Symbol Indexing Tools Source Indexing Tools Stand-alone Tools

AgeStore AdPlus BreakIN DbgSrv GFlags TList Remote

Installing Debugging tools for Windows Download Point:

www.microsoft.com/whds/devtools/debugging/default.msp

Default options sufficient By Default installs into

C:\Program Files\debugging Tools for Windows Directory Listing

Debugger Interaction: 1st Steps Command Mode or GUI? User mode prompt can get us a Head Start Get the Exception Code Understand the Environment Set the Correct Symbols Start from the Current Execution Context Check the Loaded Module

Basic Tasks : Running the debugger Attaching to Process

By Process ID: -p <process id> By Process Name: -pn <process name> TList Command

Running Under the Debugger NTSD.EXE <Command Line> NTSD.EXE C:\Windows\notepad.exe Caveats: Various Components may go into Debug

Mode

Demos

Working with the Target Last Event Registers Memory Variables Stack Unassemble Process Information Thread Information Address Information

Basic Tasks : Symbols

Additional Metadata about the Code Managed Types far more self-descriptive Private vs Public Symbols Microsoft Public Symbol Server How to tell the debugger the Location

Pointing to MS Public Server: .symfix Pointing to additional Paths: .sympath+ Reloading Symbols: .reload

Custom Symbol Servers

Walkthrough for Symbols

Symbol Server Large Store of Symbol & Binary Files Files are organized based on properties:

Name Type Time stamp Size of the Image RSDS Signature

Binary files can be stored in different location Files can be compressed

Building a Symbol Server Tools:

PdbCopy.exe BinPlace.Exe (WDK) SymStore.exe Extending the build process (Batch Files)

Basic Tasks : SOS

Powerful managed code debugger extension Introspect on the internal state of the CLR Son of Strike

Loading SOS .NET 2.0 : .loadby sos mscorwks .NET 4.0 : .loadby sos clr

Help Command !help displays all commands !help <command> displays help for specific command

SOSEX is another useful debugger extension http://www.stevestechspot.com

Debugger Extension walkthroughs

Basic Tasks : Thread

Basic Tasks : Thread Basic Unit of code Execution Before you launch a new thread, think twice Sync-blocks

Plethora of information about objects SOS Thread commands

!Threads: List all managed Thread !ClrStack: Displays Managed Callstack for currently active

Thread ~<ThreadNum>s: Switches currently active Thread ~*e!ClrStack: Shows Callstack for all managed threads !syncBlk

Deadlock problem walkthrough

Basic Tasks: Managed Heap & Garbage collection Automatic Memory Management

Sits on top of the Windows Memory Manager Currently consists of 3 generations (0, 1, 2) Caveat: Native resources must be explicitly cleaned up

SOS Commands DumpHeap DumpObj (do) GCRoot

Visualizing Runtime Object Graphs http://www.lovettsoftware.com/blogengine.net/post/

2010/01/15/Visualizing-Runtime-Object-Graphs.aspx

Walkthrough for managed heap commands

Resource Leaks What is a Resource?

Handles File Object Process Object Thread Object Isolation layer between User Mode code & Kernel

Synchronization Primitives Heap Memory Allocator Virtual Memory Allocator COM Allocator

Tools for Heap Memory Tracking UMDH

Tracks Heap based Memory Requires OS Instrumentation to be Enabled (gflags)

LeakDiag Uses Microsoft Detours Library Tracks different types of Memory Allocators

Heap Allocator Virtual Memory Allocator COM Allocator C Runtime Allocator

Debugger Command: !heap Static Source Code Analysis Tools: Prefast (WDK)

Memory Leaks: C Run-Time Functions _CrtDumpMemoryLeaks()

Performs leak checking where called. You want to place this call at all possible exits of your app.

_CrtSetDbgFlag () Sets debugging flags for the C run-time library. _CRTDBG_REPORT_FLAG Gets current flag(s) _CRTDBG_LEAK_CHECK_DF Perform

automatic leak checking at program exit through a call to _CrtDumpMemoryLeaks

Memory Leaks: Visual Studio _CRTDBG_MAP_ALLOC_

Memory allocation number (inside curly braces) Block type (normal, client or CRT) Memory location in hex Size of block in bytes Contents of the first 16 bytes in hex File name Line number

Heap Corruptions Violate the Integrity of Memory allocated on

the Heap Stray Pointers Overruns Underruns Over-Deletion Reuse after Deletion

One of the toughest problem to Debug

Windows Memory Architecture

ApplicationApplication

Virtual Memory ManagerVirtual Memory Manager

Heap ManagerHeap Manager

Default Process

Heap

Default Process

Heap

C RuntimeHeap

C RuntimeHeap Other HeapsOther Heaps

Heap Block Structure

Current Size

Current Size

Previous Size

Previous Size

SegIndexSeg

Index FlagsFlags UnusedUnused Tag IndexTag

Index

Pre-allocation MetadataPre-allocation Metadata Post-allocation MetadataPost-allocation Metadata

User accessible partUser accessible part

Pre-allocation Metadata

Suffix BytesSuffix Bytes

Fill Area(debug)Fill Area(debug)

Heap ExtraHeap Extra

Post-allocation Metadata

User accessible partUser accessible part

User accessible partUser accessible part

Tools for Debugging Heap Corruptions

Goal is to Break when the corruption occurs AND not after

PageHeap helps with that goal Annotates heap blocks to trigger fault at the time

of write Light PageHeap uses Fill Patterns Full PageHeap uses Fill Patterns and Guard

Pages Very Memory Intensive

Postmortem Debugging

Scenarios Live debugging not feasible Reproducing the problem is difficult

Static Snapshot of a Live Process Use the same debugger to debug offline Limitations

It is a snapshot; so you can't control execution Depending on type of dumpfile, some SOS

commands may not work.

Postmortem Debugging: How to generate dumpfiles Using the debuggers

.dump /mf c:\CoreDump.dmp Automatic

ADPlus Windows Error Reporting

https://winqual.microsoft.com Available to everyone

Windows Error Reporting (WER) Architecture

Error SentError SentDr.

WatsonDr.

WatsonProcessCrash

ProcessCrash

Crash data over HTTPSCrash data over HTTPS

Fault response over HTTPS

Windows Error Reporting Service

ISVISV

Query Fault Data

Postmortem Debugging: How to debug Dumpfiles? Slightly different than Native code debugging

The data Access Layer (DAC) Implemented in mscordacwks.dll Different for each version of the CLR

Debugging Dump files use the -z switch with path to the dump file

Walkthroughs with ADPlus & Postmortem debugging

When Not to use Native Debugging During Code Development Tracing the Code 100% managed code Need frequent variable inspection Need frequent references to the source files Debugging Partial Dumps Kernel Mode Debugging

Some pages are paged out

Summary

Importance of debugging Be aware of Magic Tools available for Debugging .NET Basic debugging Tasks

Running the debuggers

Questions?

Books Advanced .NET Debugging: Mario Hewardt Windows Internals: Mark E. Russinovich, David A.

Solomon with Alex Ionescu Windows via C/C++: Jeffrey M. Richter,

Christophe Nasarre Blogs

http://blogs.msdn.com/ms_joc/ http://www.wintellect.com/cs/blogs/jrobbins/default

.aspx