Using Xperfinfo and Xperf

UsingUsingxperfinfo/xperfxperfinfo/xperf

Cristian LevcoviciCristian Levcovici

Windows Client PerformanceWindows Client Performance

OutlineOutline

IntroductionIntroduction Overview of xperfOverview of xperf Overview of xperfinfoOverview of xperfinfo ExtensibilityExtensibility ReferencesReferences

IntroductionIntroduction

What is xperfinfo/xperf?What is xperfinfo/xperf? What is ETW?What is ETW?

xperfinfo/xperfxperfinfo/xperf

Extensible performance analysis toolsetExtensible performance analysis toolset Based on ETW instrumentationBased on ETW instrumentation Performs high-level decodingPerforms high-level decoding

Cross-platform (XPSP1+, W2K3, LH)Cross-platform (XPSP1+, W2K3, LH) Cross-architecture (x86, x64, ia64)Cross-architecture (x86, x64, ia64)

Capture-anywhere process-anywhereCapture-anywhere process-anywhere

Introduction

Main strengthMain strength

High level control and decoding of a large High level control and decoding of a large number of ETW events built into the NT kernelnumber of ETW events built into the NT kernel

Introduction

– process lifetimeprocess lifetime– thread lifetimethread lifetime– image lifetimeimage lifetime– sample profilesample profile– context switchcontext switch– DPC DPC (Deferred Procedure (Deferred Procedure

Call)Call)

– ISR ISR (Interrupt Service (Interrupt Service Routine)Routine)

– driver delaydriver delay

– disk I/Odisk I/O– file I/Ofile I/O– registryregistry– hardfaulthardfault– pagefaultpagefault– virtual allocationvirtual allocation– heapheap– TCP/UDPTCP/UDP

… and many others

Stackwalking support (LH only)

What is ETW?What is ETW?

Event Tracing for WindowsEvent Tracing for Windows

– High performance, low overhead, highly scalable High performance, low overhead, highly scalable tracing facility provided by the Windows OS (Win2K+)tracing facility provided by the Windows OS (Win2K+)

Extensively used by the NT kernel for self-Extensively used by the NT kernel for self-instrumentationinstrumentation

– On a typical server setup, TraceEvent() takes 1,500 On a typical server setup, TraceEvent() takes 1,500 to 2,000 cyclesto 2,000 cycles

<2% CPU overhead for a sustained rate of 20,000 <2% CPU overhead for a sustained rate of 20,000 events/sec on a 2GHz processorevents/sec on a 2GHz processor

– Uses an efficient buffering and logging mechanismUses an efficient buffering and logging mechanism Non-blocking and uses per-processor buffers that are Non-blocking and uses per-processor buffers that are

written to disk by a separate writer threadwritten to disk by a separate writer thread

Introduction

What is ETW? (contd.)What is ETW? (contd.)

Event Tracing for WindowsEvent Tracing for Windows

– Fast, reliable, and versatile set of features for Fast, reliable, and versatile set of features for logging events raised by user-mode applications logging events raised by user-mode applications and kernel-mode driversand kernel-mode drivers

– Turn tracing on/off dynamically without requiring Turn tracing on/off dynamically without requiring reboots or application restartsreboots or application restarts

– The disk log is a binary fileThe disk log is a binary file Event format is encoded by GUID + type + version Event format is encoded by GUID + type + version Trace must be post processedTrace must be post processed

– events must be correlated with context events and events must be correlated with context events and domain specific knowledge for high-level decodingdomain specific knowledge for high-level decoding

Introduction

What is ETW used for?What is ETW used for?

Debug application bugs including hangs, crashes, or unexpected behavior

Diagnose performance problems

Track computing resource consumption at application transaction level for capacity planning

Introduction

ETW ArchitectureETW Architecture

ProviderProvider– Provides event traces. Can be user-mode Provides event traces. Can be user-mode

app, kernel-mode driver, or the kernel itselfapp, kernel-mode driver, or the kernel itself– Providers are instrumented with ETW APIs Providers are instrumented with ETW APIs

to register with the ETW framework to send to register with the ETW framework to send event traces from various points in the event traces from various points in the code.code.

– When enabled dynamically by the trace When enabled dynamically by the trace controller application, the provider sends controller application, the provider sends event traces to a specific trace session event traces to a specific trace session designated by the controller.designated by the controller.

ControllerController– Assists in starting, stopping or updating Assists in starting, stopping or updating

trace sessions in the kernel as well as trace sessions in the kernel as well as enabling or disabling providersenabling or disabling providers

– Used to set trace session properties such Used to set trace session properties such as sequential or circular file logging or as sequential or circular file logging or direct delivery to consumersdirect delivery to consumers

ConsumerConsumer– Application that reads trace files or listens Application that reads trace files or listens

to active trace sessions and processes to active trace sessions and processes logged eventslogged events

– Not aware of the ProvidersNot aware of the Providers– Only receive event traces from the trace Only receive event traces from the trace

sessions or log filessessions or log files Event Trace Session infrastructureEvent Trace Session infrastructure

– Brokers the event traces from the provider Brokers the event traces from the provider to consumer and in the process adds to consumer and in the process adds valuable data to each event such as valuable data to each event such as TimeStamp, Thread, Process, CPUTimeStamp, Thread, Process, CPU

Introduction

Overview of xperfOverview of xperf

Detailed interactive analysis of Detailed interactive analysis of ETW tracesETW traces– Emphasis on kernel eventsEmphasis on kernel events– Support for 3rd party events, Support for 3rd party events,

primarily in conjunction with kernel primarily in conjunction with kernel eventsevents

TimelineTimeline

Overview of xperf

TimelineTimelineSelection

Overview of xperf

TimelineTimelineContext-Menu

Summary Table

Overview of xperf

CPU Usage Summary CPU Usage Summary TableTable

Status Bar Report

% of Time excluding DPC and ISR

% Total Time

Selected Time Interval

Close Summary Table

Overview of xperf

TimelineTimeline

Sidebar

Overview of xperf

TimelineTimeline

Sidebar Scrollbar

Overview of xperf

TimelineTimeline

Scrollbar

Overview of xperf

TimelineTimeline

Scrollbar

Selection

Overview of xperf

TimelineTimeline

Context-Menu

Summary Table

Overview of xperf

Disk I/O Summary Disk I/O Summary TableTable

Expand

Overview of xperf


ExpandExpand

Overview of xperf


Expand

Individual I/Os

I/O Priority

I/O SizeDisk Service Time

Close Summary Table

Overview of xperf

TimelineTimeline

Overview of xperf

TimelineTimeline

Selection

Overview of xperf

TimelineTimeline

Context-Menu

Detail Graph

Overview of xperf

Disk DetailDisk Detail

Overview of xperf

Change Disk

Disk DetailDisk Detail

Selection

Overview of xperf

Disk Detail Summary Disk Detail Summary TableTable Disk Queue

DepthFile Path

Individual I/Os in time order by completion time

Disk Service Time

Status Bar Report

Overview of xperf

TimelineTimeline

Overview of xperf

TimelineTimeline

Overview of xperf

Context-Menu

TimelineTimeline

Overview of xperf

Context-Menu

Summary Table

CPU Summary TableCPU Summary Table

Overview of xperf

Expand

TimelineTimeline

Overview of xperf

TimelineTimeline

Overview of xperf

Load Symbols

TimelineTimeline

Overview of xperf

Context-Menu

TimelineTimeline

Overview of xperf

Context-Menu

Summary Table

CPU Summary Table – CPU Summary Table – Symbol Decoding Symbol Decoding EnabledEnabled

Overview of xperf

Expand


Overview of xperf

ExpandExpand


Overview of xperf

Expand

Available GraphsAvailable Graphs

xperf provides many graphs and xperf provides many graphs and summary tablessummary tables– Sample ProfileSample Profile– CPU AvailabilityCPU Availability– CPU SchedulingCPU Scheduling– Disk CountsDisk Counts– Disk UtilizationDisk Utilization– Disk DetailDisk Detail– Process LifetimeProcess Lifetime– DPC DPC (Deferred Procedure (Deferred Procedure

Call)Call)

– ISR ISR (Interrupt Service (Interrupt Service Routine)Routine)

– Registry countsRegistry counts– Driver DelayDriver Delay– HardfaultHardfault– PagefaultPagefault– ServicesServices– Plug ’n’ PlayPlug ’n’ Play– MarksMarks– GenericGeneric

Overview of xperf

Overview of xperfinfoOverview of xperfinfo

High level control and decodingHigh level control and decoding Dumping of ETW tracesDumping of ETW traces Command line analysis of ETW Command line analysis of ETW

tracestraces– emphasis on kernel eventsemphasis on kernel events– support for 3rd party eventssupport for 3rd party events

Taking a kernel traceTaking a kernel trace

Start kernel traceStart kernel trace

Run scenarioRun scenario

Stop and merge kernel traceStop and merge kernel trace

Overview of xperfinfo

C:\analysis> xperfinfo –on base+cswitch

C:\analysis> xperfinfo –d trace.etlMerged Etl: trace.etl

C:\analysis> MyTestApp.exe

C:\analysis> xperfinfo –help providers

HintHint: You can retrieve all known kernel flags and groups : You can retrieve all known kernel flags and groups withwith

Taking a user traceTaking a user trace

Start user traceStart user trace


Stop user traceStop user trace

C:\analysis> xperfinfo –start MySession –on Kerberos+MRxSmb –f kerberos.etl

C:\analysis> xperfinfo –stop MySession


C:\analysis> xperfinfo –help providers

HintHint: You can retrieve all known providers with: You can retrieve all known providers with


Taking a kernel+user Taking a kernel+user tracetrace Start kernel and user tracesStart kernel and user traces


Stop user traceStop user trace

C:\analysis> xperfinfo –on base+cswitchC:\analysis> xperfinfo –start MySession –on Kerberos+MRxSmb –f kerberos.etl

C:\analysis> xperfinfo –stop MySession –stop –d trace.etlMerged Etl: trace.etl


Stopping kernel trace


Retrieving Trace Retrieving Trace HeaderHeaderC:\analysis> xperfinfo -i trace.etl -a tracestats

Number of Processors : 4CPU Speed : 2372 MHzOS Version : 05.01.01.00OS Build Number : 2600Clock type : PerfCounterBoot time : 2005/10/13:16:05:14.5000000Native Pointer Size : 4 (32bit)Start time : 2005/10/14:04:03:14.3388906End time : 2005/10/14:04:03:23.8073376 (+ 0:00:00:09.4684470)Total # Lost Buffers : 0Total # Lost Events : 0

Number of Traces : 1

Trace name: trace.etl Log file mode : Relogged Pointer size : 4 (32bit) Start time : 2005/10/14:04:03:14.3388906 End time : 2005/10/14:04:03:23.8073376 (+ 0:00:00:09.4684470) # Lost Buffers : 0 # Lost Events : 0


Retrieving Trace Retrieving Trace SummarySummaryC:\analysis> xperfinfo -i trace.etl -a tracestats -detail

Classic EventGuid TotalCount TotalSize Name====================== ========== =============== =========================== 66838 1703946 <All>

{01853a65-418f-4f36-aefc-dc0f1d2fd235}

58 39376 SysConfig

Type Level Version Count TotalSize Name ---- ----- ------- ---------- --------------- --------------------------- 0x0a 0x00 0x0001 1 952 SysConfig: CPUs 0x0b 0x00 0x0001 2 1224 SysConfig: Physical Disks 0x0c 0x00 0x0001 4 448 SysConfig: Logical Disks 0x0d 0x00 0x0001 1 744 SysConfig: Network Cards 0x0e 0x00 0x0001 2 5168 SysConfig: Video Adapters 0x0f 0x00 0x0001 47 30832 SysConfig: Services 0x10 0x00 0x0001 1 8 SysConfig: Power Management

...


Retrieving Action HelpRetrieving Action Help

C:\analysis> xperfinfo -i trace.etl -a tracestats /?

Action invocation:

xperfinfo -i <trace file> ... [-o <output>] -a tracestats ...

Action help:

tracestats [-timespan [actual]] [-detail]

-timespan [actual] Show information about session and traces. [default] Without parameters, -timespan requires inspection of trace headers only; no pass is performed through the traces in the session. When the parameter "actual" is specified, the actual times of the first event and the last event in the session are added to the report. In this case, a pass through the traces in the session is required.

-detail Show detailed information about providers, ids, tasks, opcodes, versions, channels and levels of events in the session along with provider and opcode friendly names. Requires a full pass through the traces in the session.

C:\analysis> xperfinfo –help tracestatsAlternativAlternativee::


Dumping a TraceDumping a Trace


C:\analysis> xperfinfo -i trace.etl –o trace.txt[1/2] 100.0%[2/2] 100.0%C:\analysis> notepad trace.txt

Dumping a Trace with Dumping a Trace with Symbol DecodingSymbol DecodingC:\analysis> set _NT_SYMBOL_PATH=srv*C:\symbols*\\symbols\symbolsC:\analysis> xperfinfo -i trace.etl –o trace_symbols.txt -symbols[1/2] 100.0%[2/2] 100.0%C:\analysis> notepad trace_symbols.txt


C:\analysis> xperfinfo –help symbols

Symbol Help:

Available ActionsAvailable Actions

xperfinfo provides many actionsxperfinfo provides many actions

– dumperdumper– tracestatstracestats– sysconfigsysconfig– marksmarks– processprocess– perfctrsperfctrs– profileprofile– cswitchcswitch– dpcisrdpcisr– registryregistry

– diskiodiskio– filenamefilename– hardfaulthardfault– pagefaultpagefault– drvdelaydrvdelay– reference_setreference_set– bootboot– suspendsuspend– shutdownshutdown

… and many others


Online HelpOnline Help

Your best friend is the online helpYour best friend is the online helpC:\analysis> xperfinfo -help

Usage: xperfinfo options ...

xperfinfo -help start for logger start options xperfinfo -help providers for known tracing flags xperfinfo -help stop for logger stop options xperfinfo -help merge for merge multiple trace files xperfinfo -help processing for trace processing options xperfinfo -help symbols for symbol decoding configuration xperfinfo -help query for query options xperfinfo -help mark for mark and mark-flush xperfinfo -help format for time and timespan formats on the command line xperfinfo -help advanced for advanced options (things you don't need to know :-))

Lists all available processing actions


ExtensibilityExtensibility

xperfinfo and xperf are built on top of an xperfinfo and xperf are built on top of an extensible core, extensible core, xperfcorexperfcore, that supports 3, that supports 3rdrd party binary addins providing customparty binary addins providing custom– Event dumpersEvent dumpers– InfosourcesInfosources– Event name databasesEvent name databases– Graphs & summary tablesGraphs & summary tables– Application driversApplication drivers– xperfinfo Actionsxperfinfo Actions

For details, please see the “Extending For details, please see the “Extending xperfcore” presentation xperfcore” presentation

ReferencesReferences

HomepageHomepage– http://toolbox/sites/xperfhttp://toolbox/sites/xperf

Tools distribution pointTools distribution point– \\ntperformance\tools\xperf\x86\latest\\ntperformance\tools\xperf\x86\latest– \\ntperformance\tools\xperf\amd64\latest\\ntperformance\tools\xperf\amd64\latest– \\ntperformance\tools\xperf\ia64\latest\\ntperformance\tools\xperf\ia64\latest

README.TXTREADME.TXT– \\ntperformance\tools\xperf\x86\latest\\\ntperformance\tools\xperf\x86\latest\

README.TXTREADME.TXT

Context SlidesContext Slides

Using Xperfinfo and Xperf

Documents

Transcript of Using Xperfinfo and Xperf