Windows Debuggingand Troubleshooting

Daniel PearsonDavid Solomon Expert Seminars

AgendaIntroduction to the Debugging Tools for WindowsUnderstanding Windows and x86/x64 ArchitecturesUnderstanding Application CrashesIntroducing Application VerifierAdvanced Debugging Techniques

Daniel Pearson7 years working at MicrosoftSenior Escalation Lead in the Windows base operating system teamLead in the Mobile Internet sustained engineering team

3 years at Digital Equipment CorporationSupporting Intel and Alpha systems running Windows NT

Instructor with David SolomonDavid, co–author of the Windows Internals series

Introduction to the Debugging Tools for Windows

Types of Windows DebuggersThe Debugging Tools install four debuggersOf those, cdb, ntsd, kd and WinDbg, only three can effectively be used to debug applicationsAll the debuggers share a common debugging engine, dbgeng.dll

Support for all architectures supported by WindowsMatch the architecture of the installed system, e.g. a 64–bit debugger for an x64–based processorThe debuggers can support cross–architecture debugging in certain scenarios

WinDbg is a Windows–based debugging toolAble to perform both application and system level debugging

Starting the DebuggerSeveral ways to select a debugging targetCan be configured to attach to a running process, spawn a new process or open a crash dumpThe target application ends when the debugging session ends

Must know the name or the identifier of the targetTools such as Task Manager, tasklist or tlist can be used to display the target identifier

Support for noninvasive debuggingNoninvasive debugging minimizes the debugger's interference with the target applicationUseful for situations where the application or debugging interface is not responding

Configuring the Windows DebuggersWinDbg supports the use of workspacesAllows customization of the debugger including the look and feel and handling of eventsAbility to create named workspaces that can be reused or shared with other users

Support included for a command line interfacePossible to configure the handling of the debugger and any initial commands that are sentUseful in automating the debugger for hard to track down or low–rate occurrence issues

Access to symbols to perform debuggingMost third party application vendors don’t make symbols for their applications available

Understanding SymbolsA collection of symbols contained within a single fileSymbols are the named units of code or data within a module, e.g. function names or local variablesThe debugger can interpret code and data using memory locations or by resolving symbol names

#include <stdio.h>

int main(int argc, char *argv[]){ char *s = "Hello";

printf("%s, %s\n", s, argv[1]); return 0;}

0:000> uf hello!main00321180 55 push ebp00321181 8bec mov ebp,esp00321183 8b450c mov eax,dword ptr [ebp+0c]00321186 ff7004 push dword ptr [eax+4]00321189 68c4103200 push offset hello!`string' (003210c4)0032118e 68bc103200 push offset hello!`string' (003210bc)00321193 ff1570103200 call dword ptr [hello!_imp__printf]00321199 83c40c add esp,0c0032119c 33c0 xor eax,eax0032119e 5d pop ebp0032119f c3 ret

Configuring SymbolsCan be challenging to locate the required symbolsSymbols need to match the version of the target system including any service packs or hotfixesUsing a symbol server can simplify the configuration process

Set the system wide environment variable_NT_SYMBOL_PATH=srv*C:\SYMBOLS*http://msdl.microsoft.com/download/symbols

Troubleshoot symbol loading errors with !sym noisyTurns on additional tracing information inside of symsrv.dll, the symbol server DLL

Cache

Getting AssistanceThe most useful information is the Help fileThe Debugging Tools documentation contains information for supported debugging commands Install the latest Debugging Tools for Windows for the most recent version of the documentation

Use the .hh command from within the debuggerAny text will be searched for in the index of the Debugging Tools for Windows documentation

Discovering commands with auto–completePressing the Tab key in the command window cycles through the debugger commands

Demo

Understanding Windows and 32–bit and 64–bit Architectures

CPUs and RegistersRegisters, small areas of extremely fast storageUsually measured by the number of bits they holde.g. a “32–bit” register or a “64–bit” register

x86 architecture provides 16 basic program registersEAX, EBX, ECX, EDX, EDI, ESI, EBP, ESP CS, DS, SS, ES, FS, GS EFLAGS EIP

x64 adds an additional 8 general–purpose registersRAX, RBX, RCX, RDX, RDI, RSI, RBP, RSP, R8–R15

General–purpose Segment

Displaying RegistersAccessible using the r debugger command0:001> rrax=000007fffffdb000 rbx=0000000000000000 rcx=000007fffffdf000rdx=0000000077347ec0 rsi=0000000000000000 rdi=0000000000000000rip=00000000772a0530 rsp=000000000230fa38 rbp=0000000000000000 r8=0000000000000000 r9=0000000077347ec0 r10=0000000000000000r11=0000000000000000 r12=0000000000000000 r13=0000000000000000r14=0000000000000000 r15=0000000000000000iopl=0 nv up ei pl zr na po nccs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246ntdll!DbgBreakPoint:00000000`772a0530 cc int 3

Virtual MemoryWindows provides support for a flat addressed virtual environmentThe processor, with support from the operating system, manages virtual memory via a mechanism called paging

Linear address space is divided into fixed–size pagesx86 and x64 CPUs support a minimum page size of 4 KB

2 GB systemspace

2 GB userspace

x86 virtual address space layout

Virtual MemoryWindows provides support for a flat addressed virtual environmentThe processor, with support from the operating system, manages virtual memory via a mechanism called paging

Linear address space is divided into fixed–size pagesx86 and x64 CPUs support a minimum page size of 4 KB

8 TB systemspace

8 TB userspace

x64 virtual address space layout

Displaying Virtual MemoryAccessible using the d debugger commands0:000> db 00000000`ff18000000000000`ff180000 4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00 MZ..............00000000`ff180010 b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00 ........@.......00000000`ff180020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................00000000`ff180030 00 00 00 00 00 00 00 00-00 00 00 00 e8 00 00 00 ................00000000`ff180040 0e 1f ba 0e 00 b4 09 cd-21 b8 01 4c cd 21 54 68 ........!..L.!Th00000000`ff180050 69 73 20 70 72 6f 67 72-61 6d 20 63 61 6e 6e 6f is program canno00000000`ff180060 74 20 62 65 20 72 75 6e-20 69 6e 20 44 4f 53 20 t be run in DOS 00000000`ff180070 6d 6f 64 65 2e 0d 0d 0a-24 00 00 00 00 00 00 00 mode....$.......

Threads and ProcessesProcess, an instance of a programA container that includes a private virtual address space, executable code and dataContains at least one unit of execution, a thread

Thread, a unit of execution within the systemIncludes the contents of a volatile set of registers that represent the state of the processorScheduled by the Windows kernel for execution

A unique identifier is assigned to bothAllocated from a shared table within system address space

Displaying Threads and ProcessesUsing the !teb debugger commandEach thread within a process contains a Thread Environment Block, linked the process blockViewable using the !teb debugger command

Using the !peb debugger commandEach process contains a single Process Environment Block, viewable using the !peb command

Using the inbuilt ~ commandThe ~ command is used to identify threads, ~* represents all the threads within a processThe ~s command can be used to switch between threads in a debugger

Thread StacksA storage location used by threadsUsed to store information such as parameters, local variables and return addressesThe amount of storage per thread is configurable by the application developer

Useful to identify the flow of code in an applicationUnderstanding the flow of code can assist in troubleshooting why an application crashed or is hungUsing the stack pointer register as a base is useful when viewing a stack trace is not successful

A unique stack is allocated to each threadTwo stacks are assigned to application threads, the other in system address space

Displaying Thread StacksAccessible using the k debugger commands0:000> kChild-SP RetAddr Call Site00000000`0021fa58 00000000`77169e9e USER32!NtUserGetMessage+0xa00000000`0021fa60 00000000`ff181064 USER32!GetMessageW+0x3400000000`0021fa90 00000000`ff18133c notepad!WinMain+0x18200000000`0021fb10 00000000`76ce652d notepad!DisplayNonGenuineDlgWorker+0x2da00000000`0021fbd0 00000000`7727c521 kernel32!BaseThreadInitThunk+0xd00000000`0021fc00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Demo

Understanding Application Crashes

Why Windows Applications CrashThe result of an unhandled exceptionAn event that occurs that requires the execution of code outside the normal flow of controlCan be initiated by either software or hardware during execution

Windows uses structured exception handlingRaising an exception causes the exception dispatcher to search for an exception handlerAllows the application to be given control when an exception occurs

Unhandled exceptions are passed to a system filterThe kernel filter UnhandledExceptionFilter, attempts to report the fault to the system

What Happened to the Doctor?Dr Watson replaced with WerFault in Windows VistaWindows Error Reporting enables users to notify Microsoft of application and kernel faultsAllows Microsoft to provide end users with troubleshooting information, solutions or updates

A central location is now provided for usersThe Action Center in Windows 7 or Problem Reports and Solutions in Windows Vista allows users to check for new solutions, manage reporting history and view details of reports

Additional support for non–critical eventsIncluding unresponsive applications, performance issues and application specific events

Configuring WerFaultDefault configuration is to not take a full dumpConfigurable using HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumpsPossible to change the default location, maximum storage count and type of dump created

Ability to exclude reports on a per application basisPrevents Windows from sending problem reports for those applications that are listedUseful for internal applications, those that sensitive by nature or are under development

Doesn’t affect applications with their own supportApplication such as Microsoft SQL Server use their own external utilities to write crash dumps

Attaching to a Crashed ApplicationApplication not terminated until the filter returnsIn most cases there’s a window in which a debugger can be attached to the process

Must know the name or the PID of the applicationWerFault.exe is passed a reference to the PID of the faulting application as a parameterC:\Windows\System32\WerFault.exe –u –p 4668 –s 180

Allows a user to create a dump of the applicationUseful when the system isn’t configured by default to save crashes or the default crash options don’t contain enough information to diagnose the issue you’re attempting to troubleshoot

PID

Demo

Introducing Application Verifier

Application VerifierA runtime verification tool for native codeUseful for identifying errors that can be difficult to diagnose under normal conditionsWorks by monitoring the application’s interaction with the operating system

Available as a separate download from MicrosoftSearch for Application Verifier from http://microsoft.com/downloads

Injects verification DLLs into the applicationBase support built into the operating system with additional support from verification DLLsThe number of DLLs loaded depends on the verification tests selected by the user

Configuring Application VerifierConfigurable using the Application Verifier toolEnables the user to choose which tests are run against the selected application and to view logsConfiguration information is written to the HKEY_LOCAL_MACHINE registry hive

Certain verification layers require a debuggerAll of the basic tests require that the application be configured to run under a debuggerThe user is reminded when selecting any of the tests from the Basics category

Support for using a command line interfaceUseful for pushing changes to multiple applications or those that require further configuration

Demo

Advanced Debugging Techniques

Taking a Dump of an ApplicationPossible to force dump creation of an applicationTaking a dump is useful as it allows you to restart the application while you perform further analysis

Using the built in Windows Task ManagerSelect the Processes tab, right–click on the application and select Create Dump FileThe resulting dump file is written to the directory defined by the user’s TEMP variable

Using the Debugging Tools for WindowsAfter attaching to the process, create a dump using one of the .dump commandsAllows for more control over what information is included, e.g. .dump /mA notepad.dmp

Attaching and Using a Kernel DebuggerSupport for redirection using a kernel debuggerSupported transports include a serial, USB or IEEE 1394 cable, named pipes or a network connectionNetwork support is to be included with Windows 8 and requires a supported network adapter

The system must be started in debugging modeConfigurable by modifying the boot configuration database, e.g. bcdedit /debug on

Useful in several advanced scenariosControl required of the Windows kernel or access to kernel debugging featuresDebugging service initialization when no user is logged into the system

Demo

Further InformationWindows Internals, 5th EditionRussinovich, Mark, and David Solomon. 2009. Microsoft Press

Advanced Windows DebuggingHewardt, Mario, and Daniel Pravat. 2007. Addison–Wesley Press

Windows via C/C++, 5th EditionRichter, Jeffrey, and Christophe Nasarre. 2007. Microsoft Press

Further InformationMemory Dump, Software Trace, Debugging, Malware and Intelligence Analysis Portalhttp://dumpanalysis.org

Advanced Windows Debugging and Troubleshootinghttp://blogs.msdn.com/ntdebugging

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.