SOLARISTM DTRACE INTRODUCTION - …df.martindorusa.eu/pdf/solaris_dtrace_an_introduction.pdf · 5...
Transcript of SOLARISTM DTRACE INTRODUCTION - …df.martindorusa.eu/pdf/solaris_dtrace_an_introduction.pdf · 5...
2
Audience
• Chief Executive Officer (CXO, CTO)• Software Developer• System Administrator• Support and Service Engineer• Rest Of The World
3
Agenda
• Troubleshooting Scenarios• What is SolarisTM DTrace• DTrace Architecture and Framework• DTrace Components (Need-To-Know)• Additional Privileges for DTrace (How-To)• Useful Links and Resources• Exercises with DTrace (Touch 'n' Feel)• Open Forum
4
Troubleshooting Scenarios
• Fatal, Reproducible and Test Case available> Traditional debugging techniques. Provide patch.
• Fatal, Non Reproducible> Crash dump, Analyze Core. Use mdb(1), dbx(1)> Postmortem, Speculate Static Snapshot. Iterative
process of providing Debug binary, studying log files etc.
• Non Fatal, Transient Failure and Unacceptable QoS> Traditional techniques of using truss(1), mdb(1)> Challenges, when a dynamic problem detection requires
Invasion, Embedding or Aggregation in production> Here, DTrace can be the Savior on Solaris10 or above
5
What Is SolarisTM DTrace
• Dynamic tracing facility framework for Solaris10• Unique in its focus on Production system, and its
integration of User-level and Kernel-level tracing• Allows tracing of arbitrary data and arbitrary
expression using the D Language• Overhead nearly 0% on System Performance• 50,000+ Probes available on a System by default• No re-compilation of application required• Safe, Powerful, Flexible, and Easy-to-Use• DTrace is Open Source under CDDL License
6
Dynamic Tracing (D Trace)
• A Comprehensive Framework for SolarisTM Operating System Environment> To Implement New DTrace Providers> To Implement fully Configurable DTrace Probes> To Implement New DTrace Consumers and Data Display
• Observability with DTrace> Aggregate arbitrary behavior of the OS and User programs> Dynamically Enable and Manage Probes> Dynamically Associate Predicates and Actions with Probes> Dynamically Manage Trace Buffers and Probe Overhead> Examine Live Production System or a Crash Dump
7
DTrace Architecture and Framework
Probes
• Need To Know> D Language> DTrace Probes> DTrace Provider> Actions for Probes> DTrace Consumer
8
D Language• 'D' Language is like 'C' and Constructs similar to awk
• Based on Actions and Predicates
• Access to Global, Thread local and Probe local variables
• Rich built-in Variable set. Associative Arrays
• Access to global Kernel variables and structures
• Support for ANSI-C Operators and Data Aggregation
• Example : Trace the pid of process “date” with syscall open(2)
#!/usr/sbin/dtrace -s
syscall::open:entry
/execname == “date”/
{trace(pid);
}
Probe Name o Entry point of open
Predicate o Process is named “date”
Action o Print the process ID
9
DTrace Probe • Defined by point of Instrumentation within OS Kernel
• Probe has a Name, Identifies the Function and Module it Instruments, Accessible through Provider
• Four Attributes: Provider, Module, Function & Name, defines a tuple to uniquely identify a Probe
probe description (provider:module:function:name)/ predicate /
{
action statements}
• Each Probe is Assigned an Integer Identifier
• Example : Type command dtrace -l
10
DTrace Provider
• A Methodology for Instrumenting Probes
• Provider registers Probes in the system
• Provider is informed by DTrace to Enable a Probe, and transfers the Control of the Probe to the DTrace
• Verify Providers : dtrace -l | moreID PROVIDER MODULE FUNCTION NAME1 dtrace BEGIN2 dtrace END3 dtrace ERROR...Output truncated
• Example of Providers : > syscall: Traces syscalls fbt : Traces in-kernel functions> pid : Traces application functions sched: Traces scheduling events> io : Traces system IO proc : Process/Thread creation, term, SIG
11
Actions for Probes
• Actions can be taken when Probe is triggered• Actions are Programmable (Very useful feature)• Most Actions record a specified State of the system• Expressions in D Language are acceptable as
Action Parameter(s)
12
DTrace Consumer
• Process that interacts with DTrace, could be a Command line or a Script (script.d)
• DTrace handles multiplexing, Supports concurrent Consumers
• dtrace(1M) is DTrace Consumer
• D Script Construction: Probe description, Predicate and ActionCreate Filename : myscript.d as below
#!/usr/sbin/dtrace -ssyscall::write:entry
/ execname == “bash”/
{ printf(“bash with pid %d called write system call\n”,pid);
}
Run the myscript.d as below
# dtrace -s myscript.d
dtrace: script 'myscript.d' matched 1 probe
13
Additional Privileges for DTrace• All Privileges for superuser root and
• Selective Privileges for non root user
• Privilege Groups provide Selective Access> dtrace_user : Provider syscall and profile> dtrace_proc : Provider pid and usdt> dtrace_kernel: Provider fbt and Kernel data structures> proc_owner : *Probe others Process which has sub-set privilege(s)
NOTE : You can observe only those processes for which you have the privilege, makes it Safe!
14
Privileges How-To
• Temporary Privilege to a running Process with PID 2596> Command ppriv -s A+dtrace_user 869
• Permanent Privilege to a User Account> Modify the file /etc/user_attr as
jack::::defaultpriv=basic,dtrace_user
> Command usermod -K defaultpriv=basic,dtrace_user jack
• Verify assigned Privileges> Command ppriv $$
Output869: bash
flags = <none>E: allI: basic,dtrace_userP: allL: all
15
Useful Links and Resources• Solaris Dynamic Tracing Guide (docs.sun.com)• BigAdmin System Administration Portal : DTrace• SDN Member Access, How To USE DTRACE from a
Solaris10 System (Excellent head start)• Advanced DTrace Tips, Tricks & Gotchas (Preso)• NetBeans or Sun Studio DTrace GUI Plugin (Refer)• DTrace Toolkit (Download)• Refer FAQ. Contact Sun PDS [email protected]• I've started blogging at http://blogs.sun.com/stripathi
16
Exercise 1: System Calls
• Look for System Call Errors# dtrace -n 'syscall:::return /errno/ {trace(execname);trace(pid);trace(errno)}'
0 318 pollsys:return Xorg 408 4
0 12 read:return gnome-terminal 660 11
0 12 read:return Xorg 408 11
0 12 read:return nautilus 650 11
...Output truncated
• Why not use truss> truss is much easier to use, and provides better information> truss will be Invasive for Production, and may not be suitable> truss only looks at one Process> DTrace looks at a System-wide Events
Advanced than Traditional Tools
17
Exercise 2: Short-lived Malloc
• Watch for Short-lived Memory Allocations in the Application. Use ready scripts in DTraceToolkit-0.99.tar.gz# dtrace -s DTraceToolkit/Proc/shortlived.d -c firefox
dtrace: script 'shortlived.d' matched 12 probesCPU ID FUNCTION:NAME 0 1 :BEGIN Tracing... Hit Ctrl-C to stop.^C 1 2 :END short lived processes:0.052 secstotal sample duration: 11.837 secs
Total time by process name, firefox 0 ms run-mozilla.sh 1 ms pwd 2 ms awk 3 ms ls 3 ms dirname 8 ms basename 9 ms sed 13 msTotal time by PPID, 6417 0 ms 6419 0 ms[1]+ Done firefox
...Output truncated
Download the Latest DTrace Toolkit
18
Exercise 3: Who's Doing I/O
• To find out who is doing I/O on the System.Use the ready script /usr/demo/dtrace/whoio.d
# dtrace -s whoio.d
^cDEVICE APP PID BYTES
sd0 picld 168 1280
sd0 fsflush 3 3072
sd0 sched 0 295936
nfs2 sched 0 786432
sd0 soffice.bin 6070 2242048
Use Default DTrace Demo Scripts or Customize
19
Exercise 4: Syscalls by the Application
• To find which Syscalls are made by the Application.Use the ready script /usr/demo/dtrace/syscall.d# dtrace -s syscall.d -c /usr/bin/ls
dtrace: script 'syscall.d' matched 232 probes
applicat.d howlong.d pri.d spec.d whoio.dbadopen.d index.html printa.d specopen.d whopreempt.d
dtrace: pid 1053 has exitedfcntl 1fsat 1getpid 1getrlimit 1gtime 1lstat64 1rexit 1stat 1close 2fstat64 2getdents64 2mmap 2munmap 2setcontext 2ioctl 3brk 6write 21
Use Options -c or -p for PID Providers or $target Macro in Scripts
20
Exercise 5: File system Workload
• Review the script /usr/demo/dtrace/io.d# cat io.d
#!/usr/sbin/dtrace -s
#pragma D option quiet
BEGIN
{printf("%-10s %10s %10s %3s %s\n","Device", "Program","I/O Size","R/W","Path");
}
io:::start
{printf("%-10s %10s %10d %3s %s\n",args[1]->dev_statname,execname,
args[0]->b_bcount, args[0]->b_flags & B_READ? "R" : "W" ,args[2]->fi_pathname);
@[execname, pid, args[2]->fi_pathname] = sum(args[0]->b_bcount);
}
END
{
printf("%-10s %8s %10s %s\n","Program", "PID", "Total", "Path");
printa("%-10s %8d %10@d %s\n",@);
}
Use Pragma options, Macros and Functions like C Programs
21
Exercise contd: File system Workload
• Use the ready script /usr/demo/dtrace/io.d# ./io.d
On a different terminal Run mkfile 2m /demo/foo
Device Program I/O Size R/W Path
ramdisk0 mkfile 8192 R <none>
ramdisk0 mkfile 8192 W /demo/foo
ramdisk0 mkfile 8192 W /demo/foo
...Output truncated
^c
Program PID Total Path
mkfile 13262 8192 <none>
fsflush 3 25088 <none>
sched 0 33792 <none>
mkfile 13262 2105344 /demo/foo
Dynamic Tracing and Monitoring
22
Exercise 6: Process Opening Files
• Shows opened files by the Process namedtrace -n 'syscall::open*:entry { printf("%s
%s",execname,copyinstr(arg0)); }'
dtrace: description 'syscall::open*:entry ' matched 2 probes
CPU ID FUNCTION:NAME
0 2596 open:entry df /var/ld/ld.config
0 2596 open:entry df /usr/lib/libcmd.so.1
0 2596 open:entry df /usr/lib/libc.so.1
0 2596 open:entry df /etc/mnttab
^c
Command line to Enable Probes and Format Output(s)
23
Exercise 7: Disk Size by Process
• To record Actual disk I/O Requests
• Application may be doing a lot more I/O which may get absorbed by the File system cache and not result in an Actual Disk I/O
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'
dtrace: description 'io:::start ' matched 6 probes
CPU ID FUNCTION:NAME
0 49944 bdev_strategy:start 3 fsflush 512
0 49944 bdev_strategy:start 0 sched 512
0 49944 bdev_strategy:start 0 sched 1024
Using System Probe to Trace System I/O
24
Exercise 8: Write Size by Processes
• Identify Write Size Distribution by Process on the Systemdtrace -n 'sysinfo:::writech { @dist[execname] =
quantize(arg0); }' dtrace: description 'io:::start ' matched 6 probes
^c
in.telnetd
value ------------- Distribution ------------- count
1 | 0
2 |@@@@@@@@@@@@@@@@@ 8
4 |@@ 1
8 |@@@@@@@@@@@@@ 6
16 | 0
32 | 0
64 | 0
128 | 0
256 |@@@@@@@@ 4
svc.configd
value ------------- Distribution ------------- count
...Output truncated
Use Data Manipulation and Display
25
Exercise 9: Disk Size Aggregation
• Using Aggregation for a Summarized Viewdtrace -n 'io:::start { @size[execname] =
quantize(args[0]->b_bcount); }'
dtrace: description 'io:::start ' matched 6 probes
^c
value ------------- Distribution ------------- count
512 | 0
1024 |@@ 37
2048 |@@@@@@@ 114
4096 |@@@@@@@ 116
8192 |@@@@@@@@@@@@@@@@@ 286
16384 |@@ 33
32768 |@@@@@ 87
65536 | 0
Use Data Aggregation Functions
26
Other Performance Tools
• Process Statscputrack : per-processor hardware
counterpargs : process argumentspflags : process flagspcred : process credentialspldd : process library dependencypsig : process signal dispositionpstack : process stack dumppmap : process memory mappfiles : open files and namesprstat : process statisticsptree : process treeptime : process micro-state timespwdx : process working directory
• Process Controlpgrep : grep for processespkill : kill processes listpstop : stop processesprun : start processesprctl : view/set process resourcespwait : wait for processpreap : reap a zombie process
• Process Tracing & Debugging
abitrace : trace ABI interfacesdtrace : trace the worldmdb : debug/control processestruss : trace functions,system calls
• Kernel Tracing & Debugging
dtrace : trace and monitor kernel lockstat : monitor locking statisticslockstat -k : profile kernelmdb : debug live and kernel cores
• System Statsacctcom : process accountingbusstat : Bus hardware counterscpustat : CPU hardware countersiostat : IO & NFS statisticskstat : display kernel statisticsmpstat : processor statisticsnetstat : network statisticsnfsstat : nfs server statssar : kitchen sink utilityvmstat : virtual memory stats