System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug...
Transcript of System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug...
![Page 1: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/1.jpg)
System Call TracingWHAT’S THAT PROGRAM DOING?
2/1
2/2
01
3
1
![Page 2: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/2.jpg)
This work is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/.
Adam [email protected] MUUG General Meeting
2/12/2013
2
![Page 3: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/3.jpg)
usermod –e 2013-02-10 dshewfelt
This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG.
Doug wrote articles for MUUG Lines starting in 1993, when he began collaborating with Arne Grimstrup.
“Complaining that C code is too difficult to read, Kenneth Iverson has ported the Unix kernel into a single line of APL.” –Sound Bytes, April 1994
Doug served on the committee organizing the 1993 MUUG/CIPS seminar.
Doug was then elected MUUG’s treasurer in 1994 and remained so until 2013.
(Note that this is all based on our digitized archives to date. If you have other information, please let me know so I can update this.)
2/12/2013
3
![Page 4: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/4.jpg)
Debugging Program Behaviour
With source code:
Symbolic Debuggers
Profiling Tools
Without source code:
Symbolic Debuggers
Call-stack tracing
Assembly-language inspection
System call tracing
2/12/2013
4
![Page 5: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/5.jpg)
System Call Tracing Options
Frameworks
DTrace
SystemTap
CTF (LTTng)
ProbeVue
Individual Tools
/[a-z]*trace/
/[a-z]*truss/
par
tusc
etc., etc., etc.
2/12/2013
5
![Page 6: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/6.jpg)
FrameworksWHO, WHEN, WHERE, WHY
(BUT NOT HOW)
2/12/2013
6
![Page 7: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/7.jpg)
DTrace
Originated at Sun, for Solaris
Ported from OpenSolaris to FreeBSD, NetBSD, Mac OS X, Linux, QNX
Oracle ported it (again) to Oracle Unbreakable Enterprise Linux
The “Gold Standard” for traceability of both userland and kernel
Vast amounts of documentation
Requires vast amounts of knowledge to use
Must write scripts in “D” (a DSL to define dtrace behaviour)
Main web site shuts down in Q1’13 as Oracle retreats further from Open Source
2/12/2013
7
![Page 8: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/8.jpg)
SystemTap
Originated at Red Hat, for Red Hat Enterprise Linux
Now supported in almost all Linux kernels
Originally designed to trace kernel activity, now includes userland
Large amounts of documentation, much of it outdated
Requires moderately large amounts of knowledge to use
Must write scripts in a DSL that is not “D”
Not as stable, still very useful, still very complex.
2/12/2013
8
![Page 9: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/9.jpg)
CTF / LTTng
Originated at ?
Now supported in almost all Linux kernels
Originally designed to trace kernel activity, now includes userland
Broad industry and tool support
Requires moderately large amounts of knowledge to use
No scripts, AFAIK
Still very complex.
2/12/2013
9
![Page 10: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/10.jpg)
ProbeVue
Originated at IBM, for AIX
Has been ported to… nothing else
IBM’s answer to DTrace
Similar features to DTrace
Similar complexity to Dtrace
Also appears to use a DSL
2/12/2013
10
![Page 11: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/11.jpg)
Individual ToolsWHERE, WHEN, WHY
(AND 2 EXAMPLES OF HOW)
2/12/2013
11
![Page 12: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/12.jpg)
Individual tool coverage (not exhaustive!)
Tool/OS Linux OUEL Solaris FreeBSD Mac OS X NetBSD OpenBSD AIX HPUX IRIX QNX
dtrace(1) Y Y Y Y Y Y Y
dtruss(1) Y Y Y
ftrace(1) Y Y
ktrace(1) Y Y Y Y
latrace(1) Y Y ~ ~ ~
ltrace(1) Y Y
par(1) Y
ptrace(2) Y Y Y Y Y Y Y Y Y Y Y
strace(1) Y Y
systrace(1) Y
trace(1) Y Y
truss(1) Y Y [dtruss] Y [tusc]
tusc(1) Y2/12/2013
12
![Page 13: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/13.jpg)
strace(1) on Linux
Displays system (i.e. kernel) calls only
Can run as a harness or attach to running process
Many options, but default is still useful
[demo]
2/12/2013
13
![Page 14: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/14.jpg)
ltrace(1) on Linux
Displays libc calls by default, can also display system calls
Can run as a harness or attach to existing process
Many options, default is still useful
Suggest using --demangle, to decode symbol names
[demo]
2/12/2013
14
![Page 15: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/15.jpg)
OK, one more KTRACE(1) ON *BSD
2/12/2013
15
![Page 16: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/16.jpg)
ktrace(1) & kdump(1) on *BSD
Available on all BSDs, including MacOS X.
dtrace(1) replaces ktrace(1) on newer versions of OS X
ktrace(1) only records to a file, does not display output
kdump(1) reads trace file, outputs human-readable(!) text
Can run as a harness or attach to running process
[demo]
2/12/2013
16
![Page 17: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/17.jpg)
Needle in a Haystack
WHAT AM I LOOKING FOR?
2/12/2013
17
![Page 18: System Call Tracing · usermod –e 2013-02-10 dshewfelt This presentation is dedicated to Doug Shewfelt, in remembrance of his 20 years of service to MUUG. Doug wrote articles for](https://reader034.fdocuments.in/reader034/viewer/2022042804/5f503ab8553e891cca709c7c/html5/thumbnails/18.jpg)
Finding the Needle in the Haystack
Key problem: sorting wheat from chaff
Know your syscalls:
connect(2)
fopen(2)
etc.
Know your syserrors:
ENOPERM
EINVAL
etc.
2/12/2013
18