UsingUsingxperfinfo/xperfxperfinfo/xperf
Cristian LevcoviciCristian Levcovici
Windows Client PerformanceWindows Client Performance
OutlineOutline
IntroductionIntroduction Overview of xperfOverview of xperf Overview of xperfinfoOverview of xperfinfo ExtensibilityExtensibility ReferencesReferences
IntroductionIntroduction
What is xperfinfo/xperf?What is xperfinfo/xperf? What is ETW?What is ETW?
xperfinfo/xperfxperfinfo/xperf
Extensible performance analysis toolsetExtensible performance analysis toolset Based on ETW instrumentationBased on ETW instrumentation Performs high-level decodingPerforms high-level decoding
Cross-platform (XPSP1+, W2K3, LH)Cross-platform (XPSP1+, W2K3, LH) Cross-architecture (x86, x64, ia64)Cross-architecture (x86, x64, ia64)
Capture-anywhere process-anywhereCapture-anywhere process-anywhere
Introduction
Main strengthMain strength
High level control and decoding of a large High level control and decoding of a large number of ETW events built into the NT kernelnumber of ETW events built into the NT kernel
Introduction
– process lifetimeprocess lifetime– thread lifetimethread lifetime– image lifetimeimage lifetime– sample profilesample profile– context switchcontext switch– DPC DPC (Deferred Procedure (Deferred Procedure
Call)Call)
– ISR ISR (Interrupt Service (Interrupt Service Routine)Routine)
– driver delaydriver delay
– disk I/Odisk I/O– file I/Ofile I/O– registryregistry– hardfaulthardfault– pagefaultpagefault– virtual allocationvirtual allocation– heapheap– TCP/UDPTCP/UDP
… and many others
Stackwalking support (LH only)
What is ETW?What is ETW?
Event Tracing for WindowsEvent Tracing for Windows
– High performance, low overhead, highly scalable High performance, low overhead, highly scalable tracing facility provided by the Windows OS (Win2K+)tracing facility provided by the Windows OS (Win2K+)
Extensively used by the NT kernel for self-Extensively used by the NT kernel for self-instrumentationinstrumentation
– On a typical server setup, TraceEvent() takes 1,500 On a typical server setup, TraceEvent() takes 1,500 to 2,000 cyclesto 2,000 cycles
<2% CPU overhead for a sustained rate of 20,000 <2% CPU overhead for a sustained rate of 20,000 events/sec on a 2GHz processorevents/sec on a 2GHz processor
– Uses an efficient buffering and logging mechanismUses an efficient buffering and logging mechanism Non-blocking and uses per-processor buffers that are Non-blocking and uses per-processor buffers that are
written to disk by a separate writer threadwritten to disk by a separate writer thread
Introduction
What is ETW? (contd.)What is ETW? (contd.)
Event Tracing for WindowsEvent Tracing for Windows
– Fast, reliable, and versatile set of features for Fast, reliable, and versatile set of features for logging events raised by user-mode applications logging events raised by user-mode applications and kernel-mode driversand kernel-mode drivers
– Turn tracing on/off dynamically without requiring Turn tracing on/off dynamically without requiring reboots or application restartsreboots or application restarts
– The disk log is a binary fileThe disk log is a binary file Event format is encoded by GUID + type + version Event format is encoded by GUID + type + version Trace must be post processedTrace must be post processed
– events must be correlated with context events and events must be correlated with context events and domain specific knowledge for high-level decodingdomain specific knowledge for high-level decoding
Introduction
What is ETW used for?What is ETW used for?
Debug application bugs including hangs, crashes, or unexpected behavior
Diagnose performance problems
Track computing resource consumption at application transaction level for capacity planning
Introduction
ETW ArchitectureETW Architecture
ProviderProvider– Provides event traces. Can be user-mode Provides event traces. Can be user-mode
app, kernel-mode driver, or the kernel itselfapp, kernel-mode driver, or the kernel itself– Providers are instrumented with ETW APIs Providers are instrumented with ETW APIs
to register with the ETW framework to send to register with the ETW framework to send event traces from various points in the event traces from various points in the code.code.
– When enabled dynamically by the trace When enabled dynamically by the trace controller application, the provider sends controller application, the provider sends event traces to a specific trace session event traces to a specific trace session designated by the controller.designated by the controller.
ControllerController– Assists in starting, stopping or updating Assists in starting, stopping or updating
trace sessions in the kernel as well as trace sessions in the kernel as well as enabling or disabling providersenabling or disabling providers
– Used to set trace session properties such Used to set trace session properties such as sequential or circular file logging or as sequential or circular file logging or direct delivery to consumersdirect delivery to consumers
ConsumerConsumer– Application that reads trace files or listens Application that reads trace files or listens
to active trace sessions and processes to active trace sessions and processes logged eventslogged events
– Not aware of the ProvidersNot aware of the Providers– Only receive event traces from the trace Only receive event traces from the trace
sessions or log filessessions or log files Event Trace Session infrastructureEvent Trace Session infrastructure
– Brokers the event traces from the provider Brokers the event traces from the provider to consumer and in the process adds to consumer and in the process adds valuable data to each event such as valuable data to each event such as TimeStamp, Thread, Process, CPUTimeStamp, Thread, Process, CPU
Introduction
Overview of xperfOverview of xperf
Detailed interactive analysis of Detailed interactive analysis of ETW tracesETW traces– Emphasis on kernel eventsEmphasis on kernel events– Support for 3rd party events, Support for 3rd party events,
primarily in conjunction with kernel primarily in conjunction with kernel eventsevents
TimelineTimeline
Overview of xperf
TimelineTimelineSelection
Overview of xperf
TimelineTimelineContext-Menu
Summary Table
Overview of xperf
CPU Usage Summary CPU Usage Summary TableTable
Status Bar Report
% of Time excluding DPC and ISR
% Total Time
Selected Time Interval
Close Summary Table
Overview of xperf
TimelineTimeline
Sidebar
Overview of xperf
TimelineTimeline
Sidebar
Overview of xperf
TimelineTimeline
Sidebar Scrollbar
Overview of xperf
TimelineTimeline
Scrollbar
Overview of xperf
TimelineTimeline
Scrollbar
Overview of xperf
TimelineTimeline
Scrollbar
Selection
Overview of xperf
TimelineTimeline
Context-Menu
Summary Table
Overview of xperf
Disk I/O Summary Disk I/O Summary TableTable
Expand
Overview of xperf
Disk I/O Summary Disk I/O Summary TableTable
ExpandExpand
Overview of xperf
Disk I/O Summary Disk I/O Summary TableTable
Expand
Individual I/Os
I/O Priority
I/O SizeDisk Service Time
Close Summary Table
Overview of xperf
TimelineTimeline
Overview of xperf
TimelineTimeline
Selection
Overview of xperf
TimelineTimeline
Context-Menu
Detail Graph
Overview of xperf
Disk DetailDisk Detail
Overview of xperf
Change Disk
Disk DetailDisk Detail
Overview of xperf
Change Disk
Disk DetailDisk Detail
Selection
Overview of xperf
Disk Detail Summary Disk Detail Summary TableTable Disk Queue
DepthFile Path
Individual I/Os in time order by completion time
Disk Service Time
Status Bar Report
Overview of xperf
TimelineTimeline
Overview of xperf
TimelineTimeline
Overview of xperf
Context-Menu
TimelineTimeline
Overview of xperf
Context-Menu
Summary Table
CPU Summary TableCPU Summary Table
Overview of xperf
Expand
CPU Summary TableCPU Summary Table
Overview of xperf
Expand
TimelineTimeline
Overview of xperf
TimelineTimeline
Overview of xperf
TimelineTimeline
Overview of xperf
Load Symbols
TimelineTimeline
Overview of xperf
Context-Menu
TimelineTimeline
Overview of xperf
Context-Menu
Summary Table
CPU Summary Table – CPU Summary Table – Symbol Decoding Symbol Decoding EnabledEnabled
Overview of xperf
Expand
CPU Summary Table – CPU Summary Table – Symbol Decoding Symbol Decoding EnabledEnabled
Overview of xperf
ExpandExpand
CPU Summary Table – CPU Summary Table – Symbol Decoding Symbol Decoding EnabledEnabled
Overview of xperf
Expand
Available GraphsAvailable Graphs
xperf provides many graphs and xperf provides many graphs and summary tablessummary tables– Sample ProfileSample Profile– CPU AvailabilityCPU Availability– CPU SchedulingCPU Scheduling– Disk CountsDisk Counts– Disk UtilizationDisk Utilization– Disk DetailDisk Detail– Process LifetimeProcess Lifetime– DPC DPC (Deferred Procedure (Deferred Procedure
Call)Call)
– ISR ISR (Interrupt Service (Interrupt Service Routine)Routine)
– Registry countsRegistry counts– Driver DelayDriver Delay– HardfaultHardfault– PagefaultPagefault– ServicesServices– Plug ’n’ PlayPlug ’n’ Play– MarksMarks– GenericGeneric
Overview of xperf
Overview of xperfinfoOverview of xperfinfo
High level control and decodingHigh level control and decoding Dumping of ETW tracesDumping of ETW traces Command line analysis of ETW Command line analysis of ETW
tracestraces– emphasis on kernel eventsemphasis on kernel events– support for 3rd party eventssupport for 3rd party events
Taking a kernel traceTaking a kernel trace
Start kernel traceStart kernel trace
Run scenarioRun scenario
Stop and merge kernel traceStop and merge kernel trace
Overview of xperfinfo
C:\analysis> xperfinfo –on base+cswitch
C:\analysis> xperfinfo –d trace.etlMerged Etl: trace.etl
C:\analysis> MyTestApp.exe
C:\analysis> xperfinfo –help providers
HintHint: You can retrieve all known kernel flags and groups : You can retrieve all known kernel flags and groups withwith
Taking a user traceTaking a user trace
Start user traceStart user trace
Run scenarioRun scenario
Stop user traceStop user trace
C:\analysis> xperfinfo –start MySession –on Kerberos+MRxSmb –f kerberos.etl
C:\analysis> xperfinfo –stop MySession
C:\analysis> MyTestApp.exe
C:\analysis> xperfinfo –help providers
HintHint: You can retrieve all known providers with: You can retrieve all known providers with
Overview of xperfinfo
Taking a kernel+user Taking a kernel+user tracetrace Start kernel and user tracesStart kernel and user traces
Run scenarioRun scenario
Stop user traceStop user trace
C:\analysis> xperfinfo –on base+cswitchC:\analysis> xperfinfo –start MySession –on Kerberos+MRxSmb –f kerberos.etl
C:\analysis> xperfinfo –stop MySession –stop –d trace.etlMerged Etl: trace.etl
C:\analysis> MyTestApp.exe
Stopping kernel trace
Overview of xperfinfo
Retrieving Trace Retrieving Trace HeaderHeaderC:\analysis> xperfinfo -i trace.etl -a tracestats
Number of Processors : 4CPU Speed : 2372 MHzOS Version : 05.01.01.00OS Build Number : 2600Clock type : PerfCounterBoot time : 2005/10/13:16:05:14.5000000Native Pointer Size : 4 (32bit)Start time : 2005/10/14:04:03:14.3388906End time : 2005/10/14:04:03:23.8073376 (+ 0:00:00:09.4684470)Total # Lost Buffers : 0Total # Lost Events : 0
Number of Traces : 1
Trace name: trace.etl Log file mode : Relogged Pointer size : 4 (32bit) Start time : 2005/10/14:04:03:14.3388906 End time : 2005/10/14:04:03:23.8073376 (+ 0:00:00:09.4684470) # Lost Buffers : 0 # Lost Events : 0
Overview of xperfinfo
Retrieving Trace Retrieving Trace SummarySummaryC:\analysis> xperfinfo -i trace.etl -a tracestats -detail
Classic EventGuid TotalCount TotalSize Name====================== ========== =============== =========================== 66838 1703946 <All>
{01853a65-418f-4f36-aefc-dc0f1d2fd235}
58 39376 SysConfig
Type Level Version Count TotalSize Name ---- ----- ------- ---------- --------------- --------------------------- 0x0a 0x00 0x0001 1 952 SysConfig: CPUs 0x0b 0x00 0x0001 2 1224 SysConfig: Physical Disks 0x0c 0x00 0x0001 4 448 SysConfig: Logical Disks 0x0d 0x00 0x0001 1 744 SysConfig: Network Cards 0x0e 0x00 0x0001 2 5168 SysConfig: Video Adapters 0x0f 0x00 0x0001 47 30832 SysConfig: Services 0x10 0x00 0x0001 1 8 SysConfig: Power Management
...
Overview of xperfinfo
Retrieving Action HelpRetrieving Action Help
C:\analysis> xperfinfo -i trace.etl -a tracestats /?
Action invocation:
xperfinfo -i <trace file> ... [-o <output>] -a tracestats ...
Action help:
tracestats [-timespan [actual]] [-detail]
-timespan [actual] Show information about session and traces. [default] Without parameters, -timespan requires inspection of trace headers only; no pass is performed through the traces in the session. When the parameter "actual" is specified, the actual times of the first event and the last event in the session are added to the report. In this case, a pass through the traces in the session is required.
-detail Show detailed information about providers, ids, tasks, opcodes, versions, channels and levels of events in the session along with provider and opcode friendly names. Requires a full pass through the traces in the session.
C:\analysis> xperfinfo –help tracestatsAlternativAlternativee::
Overview of xperfinfo
Dumping a TraceDumping a Trace
Overview of xperfinfo
C:\analysis> xperfinfo -i trace.etl –o trace.txt[1/2] 100.0%[2/2] 100.0%C:\analysis> notepad trace.txt
Dumping a Trace with Dumping a Trace with Symbol DecodingSymbol DecodingC:\analysis> set _NT_SYMBOL_PATH=srv*C:\symbols*\\symbols\symbolsC:\analysis> xperfinfo -i trace.etl –o trace_symbols.txt -symbols[1/2] 100.0%[2/2] 100.0%C:\analysis> notepad trace_symbols.txt
Overview of xperfinfo
C:\analysis> xperfinfo –help symbols
Symbol Help:
Available ActionsAvailable Actions
xperfinfo provides many actionsxperfinfo provides many actions
– dumperdumper– tracestatstracestats– sysconfigsysconfig– marksmarks– processprocess– perfctrsperfctrs– profileprofile– cswitchcswitch– dpcisrdpcisr– registryregistry
– diskiodiskio– filenamefilename– hardfaulthardfault– pagefaultpagefault– drvdelaydrvdelay– reference_setreference_set– bootboot– suspendsuspend– shutdownshutdown
… and many others
Overview of xperfinfo
Online HelpOnline Help
Your best friend is the online helpYour best friend is the online helpC:\analysis> xperfinfo -help
Usage: xperfinfo options ...
xperfinfo -help start for logger start options xperfinfo -help providers for known tracing flags xperfinfo -help stop for logger stop options xperfinfo -help merge for merge multiple trace files xperfinfo -help processing for trace processing options xperfinfo -help symbols for symbol decoding configuration xperfinfo -help query for query options xperfinfo -help mark for mark and mark-flush xperfinfo -help format for time and timespan formats on the command line xperfinfo -help advanced for advanced options (things you don't need to know :-))
Lists all available processing actions
Overview of xperfinfo
ExtensibilityExtensibility
xperfinfo and xperf are built on top of an xperfinfo and xperf are built on top of an extensible core, extensible core, xperfcorexperfcore, that supports 3, that supports 3rdrd party binary addins providing customparty binary addins providing custom– Event dumpersEvent dumpers– InfosourcesInfosources– Event name databasesEvent name databases– Graphs & summary tablesGraphs & summary tables– Application driversApplication drivers– xperfinfo Actionsxperfinfo Actions
For details, please see the “Extending For details, please see the “Extending xperfcore” presentation xperfcore” presentation
ReferencesReferences
HomepageHomepage– http://toolbox/sites/xperfhttp://toolbox/sites/xperf
Tools distribution pointTools distribution point– \\ntperformance\tools\xperf\x86\latest\\ntperformance\tools\xperf\x86\latest– \\ntperformance\tools\xperf\amd64\latest\\ntperformance\tools\xperf\amd64\latest– \\ntperformance\tools\xperf\ia64\latest\\ntperformance\tools\xperf\ia64\latest
README.TXTREADME.TXT– \\ntperformance\tools\xperf\x86\latest\\\ntperformance\tools\xperf\x86\latest\
README.TXTREADME.TXT
Context SlidesContext Slides
Top Related