Using Sysdig to Troubleshoot Like a Boss
description
Transcript of Using Sysdig to Troubleshoot Like a Boss
-
sysdig
strace tcpdump
sysdig tcpdump
tcpdump
sysdig
sysdig
sysdig
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
1 of 17 5/25/2015 12:21 PM
-
sysdig apt-get
rpm yum
sysdig
sysdig
sysdig curl
# curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -
# curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list
/etc/apt/sources.list.d/
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
2 of 17 5/25/2015 12:21 PM
-
apt-get update
# apt-get update
sysdig
dpkg
# dpkg --list | grep header
ii linux-generic 3.11.0.12.13 amd64 Complete Generic
Linux kernel and headers
ii linux-headers-3.11.0-12 3.11.0-12.19 all Header files rel
ated to Linux kernel version 3.11.0
ii linux-headers-3.11.0-12-generic 3.11.0-12.19 amd64 Linux kernel hea
ders for version 3.11.0 on 64 bit x86 SMP
ii linux-headers-generic 3.11.0.12.13 amd64 Generic Linux ke
rnel headers
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
3 of 17 5/25/2015 12:21 PM
-
linux-
generic
uname
# uname -r
3.11.0-12-generic
apt-get
uname -r
# apt-get install linux-headers-
# apt-get install linux-headers-3.11.0-12-generic
sysdig
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
4 of 17 5/25/2015 12:21 PM
-
# apt-get install sysdig
sysdig tcpdump
sysdig
tcpdump
-w
# sysdig -w
# sysdig -w tracefile.dump
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
5 of 17 5/25/2015 12:21 PM
-
tcpdump sysdig CTRL+C
sysdig
-r
# sysdig -r
# sysdig -r tracefile.dump
1 23:44:57.964150879 0 (7) > switch next=6200(sysdig)
2 23:44:57.966700100 0 rsyslogd (358) < read res=414 data=[ 3785.473354] sysdig_probe: starting cap
ture.[ 3785.473523] sysdig_probe:
3 23:44:57.966707800 0 rsyslogd (358) > gettimeofday
4 23:44:57.966708216 0 rsyslogd (358) < gettimeofday
5 23:44:57.966717424 0 rsyslogd (358) > futex addr=13892708 op=133(FUTEX_PRIVATE_FLAG|FUTEX_WAKE_OP) v
al=1
6 23:44:57.966721656 0 rsyslogd (358) < futex res=1
7 23:44:57.966724081 0 rsyslogd (358) > gettimeofday
8 23:44:57.966724305 0 rsyslogd (358) < gettimeofday
9 23:44:57.966726254 0 rsyslogd (358) > gettimeofday
10 23:44:57.966726456 0 rsyslogd (358) < gettimeofday
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
6 of 17 5/25/2015 12:21 PM
-
sysdig -A
sysdig
# sysdig -A
# sysdig -A > /var/tmp/out.txt
# cat /var/tmp/out.txt
1 22:26:15.076829633 0 (7) > switch next=11920(sysdig)
sysdig
tcpdump sysdig
sysdig -l
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
7 of 17 5/25/2015 12:21 PM
-
# sysdig -l
----------------------
Field Class: fd
fd.num the unique number identifying the file descriptor.
fd.type type of FD. Can be 'file', 'ipv4', 'ipv6', 'unix', 'pipe', 'e
vent', 'signalfd', 'eventpoll', 'inotify' or 'signalfd'.
fd.typechar type of FD as a single character. Can be 'f' for file, 4 for
IPv4 socket, 6 for IPv6 socket, 'u' for unix socket, p for pi
pe, 'e' for eventfd, 's' for signalfd, 'l' for eventpoll, 'i'
for inotify, 'o' for uknown.
fd.name FD full name. If the fd is a file, this field contains the fu
ll path. If the FD is a socket, this field contain the connec
tion tuple.
sysdig
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
8 of 17 5/25/2015 12:21 PM
-
# sysdig -r tracefile.dump proc.name=sshd
530 23:45:02.804469114 0 sshd (917) < select res=1
531 23:45:02.804476093 0 sshd (917) > rt_sigprocmask
532 23:45:02.804478942 0 sshd (917) < rt_sigprocmask
533 23:45:02.804479542 0 sshd (917) > rt_sigprocmask
534 23:45:02.804479767 0 sshd (917) < rt_sigprocmask
535 23:45:02.804487255 0 sshd (917) > read fd=3(10.0.0.12:55993->162.0.0.80:22) size=16384
fd.name
# sysdig fd.name=/dev/log
14 11:13:30.982445884 0 rsyslogd (357) < read res=414 data=[ 582.136312] sysdig_probe: starting captur
e.[ 582.136472] sysdig_probe:
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
9 of 17 5/25/2015 12:21 PM
-
# sysdig fd.name contains /etc
8675 11:16:18.424407754 0 apache2 (1287) < open fd=13(/etc/apache2/.htpasswd) name=/etc/apache2/.ht
passwd flags=1(O_RDONLY) mode=0
8678 11:16:18.424422599 0 apache2 (1287) > fstat fd=13(/etc/apache2/.htpasswd)
8679 11:16:18.424423601 0 apache2 (1287) < fstat res=0
8680 11:16:18.424427497 0 apache2 (1287) > read fd=13(/etc/apache2/.htpasswd) size=4096
8683 11:16:18.424606422 0 apache2 (1287) < read res=44 data=admin:$apr1$OXXed8Rc$rbXNhN/VqLCP.ojKu1aUN
1.
8684 11:16:18.424623679 0 apache2 (1287) > close fd=13(/etc/apache2/.htpasswd)
8685 11:16:18.424625424 0 apache2 (1287) < close res=0
9702 11:16:21.285934861 0 apache2 (1287) < open fd=13(/etc/apache2/.htpasswd) name=/etc/apache2/.ht
passwd flags=1(O_RDONLY) mode=0
9703 11:16:21.285936317 0 apache2 (1287) > fstat fd=13(/etc/apache2/.htpasswd)
9704 11:16:21.285937024 0 apache2 (1287) < fstat res=0
sysdig
sysdig
sysdig
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
10 of 17 5/25/2015 12:21 PM
-
-cl sysdig
# sysdig -cl
Category: CPU Usage
-------------------
topprocs_cpu Top processes by CPU usage
Category: I/O
-------------
echo_fds Print the data read and written by processes.
fdbytes_by I/O bytes, aggregated by an arbitrary filter field
fdcount_by FD count, aggregated by an arbitrary filter field
iobytes Sum of I/O bytes on any type of FD
iobytes_file Sum of file I/O bytes
stderr Print stderr of processes
stdin Print stdin of processes
stdout Print stdout of processes
sysdig sysdig
sysdig
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
11 of 17 5/25/2015 12:21 PM
-
-i
# sysdig -i bottlenecks
Category: Performance
---------------------
bottlenecks Slowest system calls
Use the -i flag to get detailed information about a specific chisel
Lists the 10 system calls that took the longest to return dur
ing the capture interval.
Args:
(None)
sysdig -c
# sysdig -c topprocs_net
Bytes Process
------------------------------
296B sshd
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
12 of 17 5/25/2015 12:21 PM
-
echo_fds
# sysdig -A -c echo_fds proc.name=apache2
------ Read 444B from 127.0.0.1:57793->162.243.109.80:80
GET /wp-admin/install.php HTTP/1.1
Host: 162.243.109.80
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Basic YWRtaW46ZUNCM3lyZmRRcg==
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/3
3.0.1750.152 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
echo_fds
fd.cip
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
13 of 17 5/25/2015 12:21 PM
-
# sysdig -A -c echo_fds fd.cip=127.0.0.1
------ Write 1.92KB to 127.0.0.1:58896->162.243.109.80:80
HTTP/1.1 200 OK
Date: Thu, 17 Apr 2014 03:11:33 GMT
Server: Apache
X-Powered-By: PHP/5.5.3-1ubuntu2.3
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1698
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
14 of 17 5/25/2015 12:21 PM
-
EMC PowerPath: superblock could not be read
Loving your blog so much. Reading through
article upon article. Can you elaborate on one item here,
since you are specifically discussing the passno, can
Remote Command Execution with SaltStack
Awesome article!
Building Self-Healing Applications with Saltstack
Could't agree more :)
Getting started with SaltStack by example:Automatically Installing nginx
Hey kannan, it looks like either salt couldn't
find the top.sls or you don't have a top.sls file. Make
sure you perform the steps from this part
[BENJAMIN CANE]
1 Comment 1
sysdig
Benjamin, thanks for putting this together! This is an great guide for anyone getting started with sysdig. One note
- on your network traffic example, in order to capture network traffic specifically you can use the fd.type filter:
sysdig -A -c echo_fds proc.name=apache2 and fd.type=ipv4
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
15 of 17 5/25/2015 12:21 PM
-
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
16 of 17 5/25/2015 12:21 PM
-
Using sysdig to Troubleshoot like a boss http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/
17 of 17 5/25/2015 12:21 PM