Dtrace Overview
-
Upload
jerome-gauthier -
Category
Technology
-
view
966 -
download
2
description
Transcript of Dtrace Overview
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
Jérôme GauthierMay 2011
DTRACEOverview
AGENDA
1. Context
2. DTrace Framework
3. System inspection and analysis
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
2
4. DTrace Toolkit
5. Demo
Context
DTrace Framework
System inspection and analysis
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
3
System inspection and analysis
DTrace Toolkit
Demo
CONTEXT
We need tools to debug and observe different situations …
Analyze the systemperformance
Understand the system utilization or
saturation
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
4
Debug anapplication
saturation
Debug a system/kernel crash
CONTEXT
Processcontrol
… in several areas
Processstatistics
Processdebugging
pgreppkill
pstoppargs pflags mdbtruss
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
5
Kerneldebugging
Kernelstatistics
Systemstatistics
iostatvmstat
cpustat kstat kmdblockstat
Examples of some Solaris commands
CONTEXT
• Some existing utilities are process based
• Some are only inspecting certain parts of the system
• Disks
• Virtual memory
• Network
• Kernel
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
6
• Kernel
Need a tool to simplify the analysis of all parts of a system
This tool must not make the system any slower
DTrace Framework
Context
System inspection and analysis
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
7
System inspection and analysis
DTrace Toolkit
Demo
DTRACE FRAMEWORK
• DTrace : A facility for dynamic instrumentation of production systems
� Another way to debug and observe the entire system and understand the big picture
• Open source (CDDL = Common Development and Distribution License)
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
8
• Supported platforms : Unix-like systems :
License)
DTRACE FRAMEWORK
Operating System Programs
Kernel level
User level
DTRACE
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
9
• Examine how the system works
• Track down performance problems across many layers of software
• Locate the cause of aberrant behavior
Does not replace or retire other system utilities
• A framework for real-time analysis and observation
• Designed for live production systems
� A totally safe way to inspect live data on production systems
� No need to stop or restart applications
DTRACE FRAMEWORK
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
10
� Live measurement and interpretation
• Over 30 000 data monitoring points spread in all system
• Zero effect when disabled, safe and no system slowdown when enabled
• Everything about DTrace is virtualized per consumer
� There is no limit on the number of concurrent DTrace consumers
• Possibility to create custom programs to dynamically instrument the system
DTRACE FRAMEWORK
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
11
• Example of problems that can be detected and analyzed:
� High CPU utilization
� Acute memory pressure
� Abnormal I/O activity
� Excessive network traffic
System inspection and analysis
Context
DTrace Framework
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
12
System inspection and analysis
DTrace Toolkit
Demo
• Probes are programmable sensors placed all over the system
• A probe fires when the event happens
� Then, certain actions are executed only if the predicate expression is true
SYSTEM INSPECTION AND ANALYSISPROBES
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
13
• General form of a probe clause :probe description
/predicate/
{
actions
}
SYSTEM INSPECTION AND ANALYSISPROBES
probe description
/predicate/
{
actions
}• A predicate allow action to only be taken when certain conditions are met
� Example : Look only for a process which has the pid=1203
Match a process which has the name firefox-bin
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
14
• An action can be “ record a stack trace, a timestamp, or the argument to a function ”
• When some probes fire, DTrace gathers the data and reports it back to you
� If no action specified, DTrace will just take note of each time the probe fires
SYSTEM INSPECTION AND ANALYSISPROBES
Type of actions Example Explanation
Data recordingtrace() Records the result of trace to the buffer
printf() Traces a D expression
• Actions are used to record data to a DTrace buffer
• Different types of actions:
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
15
stack() Records a kernel stack trace
Destructivestop() Stops the process which has executed the probe
raise() Signal a process at a precise point during execution
panic() Force a crash dump
Specialexit() Stop tracing and exits
copyin() Creates a buffer and returns its address
strlen() Returns the length of a string in bytes
• Each probe is uniquely identified by a 4-tuple :
SYSTEM INSPECTION AND ANALYSISPROBES
< provider, module, function, name >
ModuleThe name of the module in which the probe is located� Either the name of a kernel module or the name of a user library
probe description
/predicate/
{
actions
}
Probe description
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
16
FunctionIf this probe correspond to a specific program location, it’s the name of the program function in which the probe is located.
NameThe final component of the tuple is the string name of the probe that give you some idea of the probe’s semantic meaning.
• Some probes does not have a module and function (e.g. : BEGIN, END)
� We can identify a probe only by its name
• DTrace probes come from a set of kernel modules called providers
� Which correspond to a particular kind of instrumentation
SYSTEM INSPECTION AND ANALYSISPROVIDERS
< provider, module, function, name >
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
17
• We can list the probes that a provider can transmit to the DTrace Framework
• A provider pass the control to DTrace when you decide to enable one of its probes
• Example of some providers :
SYSTEM INSPECTION AND ANALYSISPROVIDERS
SYSCALL
• Holds the entire communication from userland to kernel space• Every system call on the
PROC
• Handle process creation and termination• Sending and handling signals
SCHED
• CPU scheduling: sleeping, running threads• CPU time: which threads are run by which CPU and
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
18
• Every system call on the system
signals
IO
• I/O system• Disk input and output requests• I/O by device, process, size, filename
MIB
• Counters for management information bases• IP, IPv6, ICMP, IPSec
are run by which CPU and for how long
PROFILE
• Time based probing at specific interval of times• To sample anything in the system every unit time
• A D program consist of one or more probe clauses
• D Language is like C language with some constructs similar with awk :
• Support ANSI C operators
Support Strings
SYSTEM INSPECTION AND ANALYSISD LANGUAGE
/* D Program */
probe1 description
/predicate/
{
actions
}
probe2 description
/predicate/
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
19
• Support Strings
• D expressions are based on built-in variables : pid, execname, timestamps, curthread
• No control-flow constructs : loops, if statements
• Floating-point arithmetic is not permitted (only Integers)
/predicate/
{
actions
}
probe3 description
/predicate/
{
actions
}
• Used to aggregate data and look for trends
• Simple to generate report about :
Total system calls by a process or an application
SYSTEM INSPECTION AND ANALYSISD LANGUAGE : AGGREGATIONS
@name[keys] = aggfunction(args)
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
20
• Total system calls by a process or an application
• Total number of read or write by process
• Aggregating functions: • count()
• sum()
• avg()
• min()
• max()
• quantize()
SYSTEM INSPECTION AND ANALYSISD LANGUAGE : SECURITY
• D programs are compiled into a safe intermediate form that is used for execution when a probe fires
D program source files
DTrace consumer: dtrace()
DTrace providers• Programming mistakes : DTrace will
kernel
userland
DTrace safe execution environment
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
21
� It is impossible to construct a bad script that would cause damage to the system
• DTrace handle run-time errors and report them (dividing by zero, dereferencing invalid memory, …)
• Programming mistakes : DTrace will report errors and disable instrumentation
Context
DTrace Framework
System inspection and analysis
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
22
DTrace Toolkit
Demo
System inspection and analysis
DTRACE TOOLKIT
• Collection of over 100 useful documented scripts built on top of DTrace Framework
� Developed by the OpenSolaris DTrace community
• The toolkit contains :
� the scripts
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
23
� the man pages
� the example documentation
� the notes files
� the tutorials
• The script are sorted by categories
DTRACE TOOLKITCATEGORIES
DTrace ToolkitApplications
Extra, User, System
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
24
DTrace Framework
Applications
CPU
Disk
Kernel
Network
Memory
Processes
System
Extra, User, System
Context
DTrace Framework
System inspection and analysis
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
25
Demo
System inspection and analysis
DTrace Toolkit
DEMO
Command line scripts
tcpstat
Chime scripts
system calls
� System calls counter by process,
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
26
� TCP bytes received and send
httpdstat
� Real-time Apache Web Server stats
� System calls counter by process, function, module, …
php calltime
� Measure PHP elapsed times for functions
REFERENCES & ADDITIONAL INFORMATION
• PowerPoint presentations
• Joyent Performance Visualization – Brendan Gregg
• DTrace & DTrace Toolkit – Stefan Parvu
• Links
• DTrace community site:
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
28
• DTrace community site:http://hub.opensolaris.org/bin/view/Community+Group+dtrace/WebHome
• Brendan Gregg site: http://www.brendangregg.com/dtrace.html
• DTrace Blogs: http://dtrace.org/blogs/
• DTrace online guide: http://wikis.sun.com/display/DTrace/Documentation
• DTrace wiki: http://www.solarisinternals.com/wiki/index.php/DTraceToolkit
• DTrace training: http://dtracehol.com
COPYRIGHT © 2011 ALCATEL-LUCENT. ALL RIGHTS RESERVED.
29