Performance Monitoring on AIX

byPeter Lee

IT Specialist

Performance Tuning Process

Indentify Workloads

Define and prioritize goals

Identify the required resoureces

Minimize resource requirements

Control the allocation of resource

Apply additional resources as indicated

Performance Analysis Tools

CPU Memory Subsystem I/O Subsystem Network

Subsystemvmstat, iostat vmstat iostat lsattr

ps lsps vmstat netstatsar svmon lsps nfsstat

gprof/prof/tprof filemon lsattr entstattime/timex bf, bfrpt lsdev netpmonnetpmon lspv/lsls/lsvg ifconfig

stem fileplace iptrace/ipreportsyscalls filemon tcpdumplockstatemstat

performance toolbox

performance toolbox

performance toolbox

performance toolbox

trace, trcrpt, utld trace, trcrpt trace, trcrpt trace, trcrpt

Performance Tuning Tools

CPUs Memory Subsystem I/O Subsystem Network

Subsystem

nice/renice vmtune vmtune no

schedtune chps/mkps chdev nfso

bindprocessor fdpr migratepv chdev

chdev chdev chlv ifconfig

setpri rmss reorgvg

Performance Tuning Flowchart

Actions

Is System CPU-Bound?

Actions

Is System Memory-Bound?

Is System Disk-Bound?

Is System Network-Bound?

Actions

Actions Additonal

Actions

NO

YES

NOYES YES

NO

NOYES

vmstat Commandvmstat Command Reports:

Summary of paging and system events since initialization (-s)Number of forks since initilization (-f)Device interrupts since initilization or per interval (-i)Periodic reports of processes, paging, real memory, and CPUUsage : vmstat [ -ifs ] [drives] [interval] [count]

vmstat 1 10kthr memory page faults cpu----- ----------- ------------------------ ------------ -----------r b avm fre re pi po fr sr cy in sy cs us sy id wa0 0 2514 24166 0 0 0 0 0 0 121 1 18 0 0 99 00 0 2514 24165 0 0 0 0 0 0 126 178 19 1 0 99 00 0 2514 24165 0 0 0 0 0 0 118 60 24 0 0 99 00 0 2514 24165 0 0 0 0 0 0 114 54 20 0 0 99 00 0 2514 24165 0 0 0 0 0 0 121 117 21 0 0 99 00 0 2514 24165 0 0 0 0 0 0 119 54 20 0 1 99 00 0 2514 24165 0 0 0 0 0 0 126 54 22 1 0 99 00 0 2514 24165 0 0 0 0 0 0 127 54 19 0 1 99 00 0 2514 24165 0 0 0 0 0 0 125 54 21 0 0 99 00 0 2514 24165 0 0 0 0 0 0 127 54 18 0 0 99 0

root@bmserver[/tmp]>vmstat -i

priority level type count module(handler)0 0 hardware 0 i_misc_pwr(b0e5c)0 24 hardware 0 i_epow(9d820)0 24 hardware 0 /etc/drivers/hscsidd(18bc578)0 48 hardware 0 i_scu(b0e50)1 2 hardware 66 /etc/drivers/rsdd(196de4c)3 4 hardware 3 /etc/drivers/middd_loadpin(19ce63c)3 5 hardware 1736 /etc/drivers/hscsidd(18bc56c)3 7 hardware 21636015 /etc/drivers/tokdd(19e78fc)3 14 hardware 54189 /etc/drivers/hscsidd(18bc56c)5 62 hardware 117295403 clock(9d670)10 63 hardware 7647 i_softoff(9d628)

vmstat Command

root@bmserver[/tmp]>vmstat -s

210376 total address trans. faults5724 page ins55944 page outs

0 paging space page ins0 paging space page outs0 total reclaims

78419 zero filled pages faults1551 executable filled pages faults

0 pages examined by clock0 revolutions of the clock hand0 pages freed by the clock

8790 backtracks0 lock misses0 free frame waits0 extend XPT waits

3760 pending I/O waits56326 start I/Os56326 iodones

21347782 cpu context switches139338639 device interrupts

0 software interrupts0 traps

1873766 syscalls

vmstat Command

fre : the number of pages of real memory on the free list at the time of the interval sample

re : the number of page reclaims per second observed in the sample interval. If a page fault occurs and this page is currently on the freelist and has not yet been reused, this is considered a reclaim since no new I/O request has to be initiated. This is not to be confused with the term repage which refers to a page which has already incurred a page-fault (the page could be currently on the freelist, filesystem, or in paging space).

pi or po : the number of page ins or page outs per second from/to paging space during the sample interval

fr/sr : the number of pages per second that were freed/examined by the page replacement algorithm during the interval

cy : the number of cycles per second of the clock algorithm

in/sy/cs : the number of device interrupts/system calls/thread context switches per second observed in the interval

CPU : CPU utilization observed in the interval, divided into us-user, sy-system, id-idle, wa-wait

disk xfer the number of disk requests per second observed to the PV

vmstat Command

topas Command

iostat Command Reports:CPU utilization is %user + %system approaching 100%?Terminal I/ODisk I/O Balanced?CD-ROM I/O

Hardware Dependent

Local disk information only

No special authority required

iostat Command

iostat Command

usage: iostat [-t] [-d] [drives] [interval [count]]-t specifies tty/cpu report only-d specifies drive report only-t and -d cannot both be specified

root@bmserver[/tmp]>iostat

tty: tin tout avg-cpu: % user % sys % idle% iowait

0.0 0.5 0.0 0.1 99.80.1

Disks: % tm_act Kbps tps Kb_read Kb_wrtnhdisk0 0.1 0.2 0.0 40264 220277hdisk1 0.0 0.0 0.0 0 0cd0 0.0 0.0 0.0 500 0

sar Command

sar Command

Data TypesAll data types collected; only specified data types reported on

Timing Intervals: the extraction interval can differ from the collection interval

Permissions: need to be in system group

sadc: back end for sar, responsible for the data collection

timex -s: displays sar statistics for the duration of the command executed by timex

sar Command

There are two forms of the sar command: the first form is used to sample, display, and/or save systemstatistics; the second form is used to process and display previously-captured data.

e.gCollection: sar -o filename 60 300 > /dev/null &Extraction: sar -A -f filenameExtraction: sar -u -f filename -s 10:20 -e 14:50Monitoring: sar -c 60 300

sar Command

Data Types:-a file access system routines-b buffer activity-c system calls-k kernel process statistics-m message and semaphore activity-q statistics on run queue and wait queue-r paging statistics-u CPU utilization-v kernel tables: i-node, process and file-w process switching activity-y reports on tty statistics-A Show all of the above-P [processor_id[,...] | ALL] Show individual processor utilization on SMP systems

# sar -u 5 319:28:02 %usr %sys %wio %idle19:28:07 0 6 0 9419:28:12 1 4 0 9519:28:07 7 4 0 90Average 3 4 0 93

# sar -c 1 319:28:25 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s19:28:26 134 36 1 0.00 0.00 2691306 151719:28:27 46 34 1 0.00 0.00 2716922 153119:28:28 46 34 1 0.00 0.00 2716922 1531Average 75 35 1 0.00 0.00 2708329 1527

# sar -q 5 319:31:42 runq-sz %runocc swpq-sz %swpocc19:31:47 1.0 100 1.0 10019:31:52 2.0 100 1.0 10019:31:57 1.0 100 1.0 100Average 1.3 95 1.0 95

sar Command

ps Command

ps CommandThe ps command writes the current status of active processes and (if the -m flag is given) associated kernel threads to standard output.A new option to ps in AIX Version 4 shows if a process is bound to a particular CPU. This option is -o THREAD. If you see the -m option as well (-mo THREAD), then you can see the thread or threads that belongs to each process.Usage : ps [-Aaedfklm] [-n namelist] [-F Format] [-o specifier[=header],...] [-p proclist][-G|-g grouplist] [-t termlist] [-U|-u userlist]Usage: ps [aceglnsuvwxU] [t tty] [processnumber]

ps Command

root@bmserver[/tmp]>ps -elF S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME CMD

200003 A 0 1 0 0 60 20 2805 252 - 0:00 init240001 A 0 2548 1 0 60 20 8831 68 5a6cf58 - 2:32 syncd40401 A 0 2814 1 0 60 20 1923 276 95378 - 0:00 errdemon240001 A 0 2902 1 0 60 20 921 244 - 0:00 srcmstr240001 A 0 3528 1 0 60 20 d21a 168 54aeccc - 0:13 cron240001 A 0 3694 2902 0 60 20 59eb 608 - 0:00 sendmail240001 A 0 3968 2902 0 60 20 c9f9 984 - 0:42 snmpd240001 A 0 4218 2902 0 60 20 a1f4 172 - 0:00 inetd240001 A 0 4468 2902 0 60 20 79ef 292 - 0:00 portmap240001 A 0 4712 2902 0 60 20 9e1 128 - 0:02 syslogd240001 A 0 4998 2902 0 60 20 2204 468 - 0:00 dpid2240001 A 0 5866 8424 0 60 20 5acb 348 pts/0 0:00 ksh240001 A 0 6054 2902 0 60 20 8a11 68 - 0:00 biod240001 A 0 6326 2902 0 60 20 4208 144 - 0:00 rpc.statd240001 A 0 6598 2902 0 60 20 f21e 68 * - 0:03 rpc.lockd240001 A 0 6858 1 0 60 20 1a23 32 9ec0170 - 0:00 install_as200001 A 0 6932 5866 11 65 20 7acf 188 pts/0 0:00 ps200001 A 0 8424 4218 2 61 20 82d0 268 - 0:00 telnetd

ps Commandroot@bmserver[/tmp]>ps -emo THREAD

USER PID PPID TID ST CP PRI SC WCHAN F TT BND COMMANDroot 1 0 - A 0 60 1 - 200003 - 0 /etc/init

- - - 463 S 0 60 1 - 400 - 0 -root 2548 1 - A 0 60 1 5a6cf58 240001 - 0 /usr/sbin/syncd 60

- - - 3325 S 0 60 1 5a6cf58 400 - 0 -root 2814 1 - A 0 60 1 95378 40401 - 0 /usr/lib/errdemon

- - - 3335 S 0 60 1 95378 400 - 0 -root 2902 1 - A 0 60 1 - 240001 - 0 /usr/sbin/srcmstr

- - - 2399 S 0 60 1 - 8400 - 0 -root 3528 1 - A 0 60 1 54aeccc 240001 - 0 /usr/sbin/cron

- - - 3795 S 0 60 1 54aeccc 400 - 0 -root 3694 2902 - A 0 60 1 - 240001 - 0 sendmail: accepting

connections- - - 4727 S 0 60 1 - 8400 - 0 -

root 3968 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/snmpd- - - 4233 S 0 60 1 - 8400 - 0 -

root 4218 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/inetd- - - 4483 S 0 60 1 - 8400 - 0 -

root 4468 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/portmap- - - 4989 S 0 60 1 - 8400 - 0 -

root 4712 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/syslogd- - - 5233 S 0 60 1 - 8400 - 0 -

root 4998 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/dpid2- - - 5519 S 0 60 1 - 8400 - 0 -

root 5866 8424 - A 0 60 1 - 240001 pts/0 0 -ksh- - - 8185 S 0 60 1 - 400 - 0 -

root 6054 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/biod 6- - - 6833 S 0 60 1 - 400 - 0 -

root 6326 2902 - A 0 60 1 - 240001 - 0 /usr/sbin/rpc.statd- - - 7105 S 0 60 1 - 8400 - 0 -

root 6598 2902 - A 0 60 3 * 240001 - 0 /usr/sbin/rpc.lockd- - - 7377 S 0 60 1 9f64c98 400 - 0 -- - - 7639 S 0 60 1 54aa988 1400 - 0 -- - - 7897 S 0 60 1 2735dc 1400 - 0 -

root 7008 5866 - A 12 66 1 - 200001 pts/0 0 ps -emo THREAD- - - 9071 R 12 66 1 - 0 - 0 -

ps Commandroot@bmserver[/tmp]>ps auxUSER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMANDroot 5522 0.0 0.0 8 16 - A Aug 20 0:00 kprocroot 5866 0.0 1.0 348 408 pts/0 A 06:01:57 0:00 -kshroot 6054 0.0 0.0 68 96 - A Aug 20 0:00 /usr/sbin/biod 6root 5264 0.0 0.0 8 12 - A Aug 20 0:01 kprocroot 4468 0.0 1.0 292 360 - A Aug 20 0:00 /usr/sbin/portmaproot 4712 0.0 0.0 128 172 - A Aug 20 0:02 /usr/sbin/syslogdroot 4998 0.0 2.0 468 548 - A Aug 20 0:00 /usr/sbin/dpid2root 7308 0.0 0.0 8 12 - A Aug 30 0:00 kprocroot 6598 0.0 0.0 68 108 - A Aug 20 0:03 /usr/sbin/rpc.lockroot 8424 0.0 1.0 268 324 - A 06:01:57 0:01 telnetdroot 7534 0.0 1.0 188 232 pts/0 A 07:27:18 0:00 ps auxroot 6326 0.0 1.0 144 196 - A Aug 20 0:00 /usr/sbin/rpc.statroot 6858 0.0 0.0 32 44 - A Aug 20 0:00 /usr/sbin/install_root 7024 0.0 1.0 208 252 pts/0 A 07:27:18 0:00 moreroot 1032 0.0 0.0 40 48 - A Aug 20 2:25 kprocroot 1896 0.0 0.0 8 12 - A Aug 20 0:00 kprocroot 2228 0.0 0.0 8 12 - A Aug 20 0:00 kprocroot 774 0.0 0.0 8 12 - A Aug 20 0:25 kprocroot 0 0.0 0.0 4 8 - A Aug 20 2:51 swapperroot 1 0.0 1.0 252 304 - A Aug 20 0:00 /etc/initroot 516 99.9 0.0 0 4 - A Aug 20 19067:57 kproc

emstat Command

Instruction Emulation can be determined

Different processors have different instruction sets

PowerPC architecture deletes about 35 POWER instruc-tions

To maintain binary compatibility, deleted instructions are emulated

Instruction emulation increases the pathlength for the instruction

Emulation can be avoided by compiling applications using Common Mode option

emstat Command

# /usr/samples/kernel/emstat 1 7emstat total count emstat interval count

21951937 2195193721967488 1555121974877 738921994354 1947722007329 1297522018171 1084222018171 0

New tool to detect instruction emulation - shipped with bos.adt.samples

Usage : /usr/samples/kernel/emstat [interval] [count]

Scheduling Policies

SCHED_FIFONon-preemptive schedulingFixed piority

SCHED_RRRound-Robin schedulingFixed piorityRun until interrupted or end of time slice

SCHED_OTHERDefault scheduling policySame as SCHED_RR but applies to threads with non-fixed pioritiesPiority degrades with CPU Usage

Process and Thread Piority

Child Process /Threads

Real-Time Process Piorities

Parent Process

User Process Piorities

fork()

Inherited Process Piority

Initial PRI = NICE + 40pe -el (to view values)ps -emo THREAD (to view threads)

Process Piorities

setp

ri( )

nice

reni

ce

High = 0

Low = 127

40

nice/renice Commands

Initial priority of non-fixed priority thread is: 40 + NICE where NICE has a default value of 20. NICE can range from 0-40.

Nice value can be set at thread creation time by using the nice command. Syntax of nice:

usage: nice [-n increment] commandusage: nice [-increment] command

Nice value can be changed after thread has already been created using the renice command. Syntax of renice:

usage: renice [-n increment] [ -g| -p| -u] ID...Nice values are reduced by using negative increment values and increased by using positive increment values. Reducing nice value makespriority more favored. Increasing nice value makes priority less favored.

schedtune Command

Modifies/Disables Memory Load Control

Modifies CPU scheduling algorithms

Modifies CPU time slices

Restores defaults

Must repeat at every boot

schedtune Command

Change the Scheduling policy from default to Round-Robin using the following command:

/usr/samples/kernel/schedtune -r 0 -d 0

Each timeslice is in units of clock ticks (10ms). Default timeslice is 1 clock tick and is tunable using

/usr/samples/kernel/schedtune -t <# of ticks>

# /usr/samples/kernel/schedtune

THRASH SUSP FORK SCHED-h -p -m -w -e -f -d -r -t -sSYS PROC MULTI WAIT GRACE TICKS SCHED_D SCHED_R TIMESLICE MAXSPIN6 4 2 1 2 10 16 16 1 1

bindprocessor Process [ ProcessorNum ] | -q | -u Process

Purpose

Binds or unbinds the kernel threads of a process to a processor.

bindprocessor Command

svmon Command

svmon Command is an AIX Specific analysis tool for virtual memory

Purpose:Captures a snapshot of the current state of memory.The display of information can be analyzed using four different reports:global [ -G] process [-P]segment [-S]detailed segment [-D]

In AIX Version 4, svmon is packaged as part of perfagent.tools fileset.

svmon Command

usage: svmon[ -G ][ -P[n|s|a] [ pid1 ... pidN ] ][ -P[n|s|a][u|p|g|r] [ count ] ][ -S sid1 ... sidN ][ -S{[n|s|a][u|p|g|r]} [ count ] ][ -D sid1 ... sidN ][ -i interval [ numintervals ] ][ -r ]

# svmon -G -i 5 3m e m o r y i n u s e p i n p g s p a c e

size inuse free pin work pers clnt work pers clnt size inuse49152 48134 1018 2613 10674 37460 0 2613 0 0 98304 810449152 48139 1013 2613 10678 37461 0 2613 0 0 98304 810549152 48139 1013 2613 10678 37461 0 2613 0 0 98304 8105

svmon Command1. To print out global statistics:

$ svmon -G

2. To print out the memory usage statistics for all processes (limit statistics to non-system segments):

$ svmon -Pn

3. To print out the memory usage statistics for processes 6746 and 10078 (includestatistics on all segments):

$ svmon -Pa 6746 10078

4. To sort system segments by the number of pages in real memory, and print outthe top 10 segments of the resulting list:

$ svmon -Ssu 10

5. To print detailed memory usage statistics on segment 340d:$ svmon -D 340d

6. To print out the global statistics, repeated 10 times at 30 second intervals:$ svmon -G -i 30 10

rmss Command

Reduced Memory System Simulator

Simulate RISC System/6000 with various size memories

Run an application over a range of memory sizes

rmss is packaged as part of perfagent.tools

usage: rmss [-s startmemsize] [-f finalmemsize] [-d deltamemsize][-n numiterations] [-o outputfile] command

rmss -c memsizermss -rrmss -p

rmss Command

To change memory size# rmss -c 32Simulated memory size changed to 32 Mb.

To display the current memory size:# rmss -pSimulated memory size is 32 Mb.

To reset the memory size to the real size:rmss -rSimulated memory size changed to 192 Mb.

To measure the performance of the command cc -O foo.c on memory size 32, 24, 16, 8MB, run and measure the command once at each memory size and write the report to the file cc.rmss.out

rmss -s 32 -f 8 -d 8 -n 1 -o cc.rmss.out cc -O foo.c

filemon Command

The filemon tool collects and presents trace data on the various layers of file system utilization including the logical file system, virtual memory segments, LVM, and physical disk layers. Data can be collected on all the layers, or layers can be specified with the -O layer option. The default is to collect data on the VM, LVM, and physical layers.

filemon will only collect data for those files opened after filemon was started unless you specify the -u flag.

Normally, only the top 20 logical files and segments are reported unless the -v (verbose) flag is used.

filemon Command

Usage: filemon [-i file] [-o file] [-d] [-v] [-u] [-O opt]-i file: open input file (default is real-time trace)-o file: open output file (default is stdout)-d: deferred trace (until ‘ trcon’ )-T num: set trace kernel buf sz (default 32000 bytes)-P: pin monitor process in memory-v: verbose mode (print extra details)-u: print unnamed file activity via pid-O opt: other monitor-specific options

valid -O options: lf,vm,lv,pv,alllf: monitor logical file I/Ovm: monitor virtual memory I/Olv: monitor logical volume I/Opv: monitor physical volume I/Oall: short for lf,vm,lv,pv

(default is: vm,lv,pv)

filemon Command

# filemon -O lv,pv -o fmon.out# dd if=/dev/rhdisk1 of=/dev/null bs=512 count=1024# cp /unix /tstc/junk# trcstop# more fmon.outFri Aug 2 15:11:21 1996System: AIX magic Node: 4 Machine: 000001455C005.450 secs in measured intervalCpu utilization: 72.6%Most Active Logical Volumes------------------------------------------------------------------------util #rblk #wblk KB/s volume description

------------------------------------------------------------------------0.36 0 1588 145.7 /dev/lv04 /tstc Compressed: Frag_Sz.=10240.00 0 16 1.5 /dev/loglv00 jfslog

Most Active Physical Volumes------------------------------------------------------------------------util #rblk #wblk KB/s volume description

------------------------------------------------------------------------0.75 1024 1604 241.1 /dev/hdisk1

netstat Command

netstat CommandShows state of interfacesShows interface statisticsShows routing statisticsShows per-protocol statisticsShows network device driver statisticsShows network memory usage statistics

netstat Commandroot@bmserver[/tmp]>netstat -iName Mtu Network Address Ipkts Ierrs Opkts Oerrs Colllo0 16896 <Link> 473 0 473 0 0lo0 16896 127 loopback 473 0 473 0 0tr0 1492 <Link>10.0.5a.c9.62.a4 10832411 0 22191 0 0tr0 1492 9.184.64 bmserver 10832411 0 22191 0 0

root@bmserver[/tmp]>netstat -I tr0 1input (tr0) output input (Total) output

packets errs packets errs colls packets errs packets errs colls10832898 0 22267 0 0 10833371 0 22740 0 0

3 0 1 0 0 3 0 1 0 07 0 1 0 0 7 0 1 0 04 0 1 0 0 4 0 1 0 05 0 1 0 0 5 0 1 0 010 0 1 0 0 10 0 1 0 07 0 1 0 0 7 0 1 0 06 0 1 0 0 6 0 1 0 011 0 1 0 0 11 0 1 0 010 0 1 0 0 10 0 1 0 09 0 1 0 0 9 0 1 0 04 0 1 0 0 4 0 1 0 09 0 1 0 0 9 0 1 0 07 0 1 0 0 7 0 1 0 07 0 1 0 0 7 0 1 0 0

netstat Commandroot@bmserver[/tmp]>netstat -m

34 mbufs in use:16 mbuf cluster pages in use72 Kbytes allocated to mbufs0 requests for mbufs denied0 calls to protocol drain routines

Kernel malloc statistics:

******* CPU 0 *******By size inuse calls failed free hiwat freed32 108 115 0 20 640 064 122 765 0 70 320 0128 111 1666 0 17 160 0256 134 21771388 0 42 384 0512 37 2470 0 11 40 01024 1 481 0 3 20 02048 0 773 0 6 10 04096 21 1510233 0 6 120 08192 0 66 0 2 10 016384 1 6 0 19 24 7

By type inuse calls failed memuse memmax mapbmbuf 34 21747482 0 8704 17920 0mcluster 16 1512910 0 65536 95744 0socket 111 698 0 18048 19008 0pcb 78 427 0 12032 12864 0routetbl 8 13 0 1312 2080 0ifaddr 5 5 0 576 576 0mblk 30 1198 0 6272 6528 0mblkdata 3 1070 0 12288 22528 0strhead 13 57 0 3776 4800 0strqueue 21 132 0 10752 13312 0strmodsw 21 21 0 1344 1344 0strosr 0 23261 0 0 768 0strsyncq 29 388 0 2944 3648 0streams 148 275 0 16000 17280 0kernel table 14 15 0 45920 46432 0temp 5 12 0 5312 9920 0

Streams mblk statistic failures:0 high priority mblk failures0 medium priority mblk failures0 low priority mblk failures

netstat Commandroot@bmserver[/tmp]>netstat -D

Source Ipkts Opkts Idrops Odrops-------------------------------------------------------------------------------ent_dev0 0 0 0 0tok_dev0 21755956 24738 58384 0

---------------------------------------------------------------Devices Total 21755956 24738 58384 0-------------------------------------------------------------------------------ent_dd0 0 0 0 0tok_dd0 21697572 24738 0 0

---------------------------------------------------------------Drivers Total 21697572 24738 0 0-------------------------------------------------------------------------------ent_dmx0 0 N/A 0 N/Atok_dmx0 10836721 N/A 10860851 N/A

---------------------------------------------------------------Demuxer Total 10836721 N/A 10860851 N/A-------------------------------------------------------------------------------IP 5171634 2654588 18061 18061TCP 26292 21120 0 0UDP 2628010 473 2 2370

---------------------------------------------------------------Protocols Total 7825936 2676181 18063 20431-------------------------------------------------------------------------------lo_if0 473 473 0 0tr_if0 10834304 22321 0 7

---------------------------------------------------------------Net IF Total 10834777 22794 0 7-------------------------------------------------------------------------------

---------------------------------------------------------------NFS/RPC Total N/A 12 0 0-------------------------------------------------------------------------------(Note: N/A -> Not Applicable)

root@bmserver[/tmp]>netstat -sip:

1079372 total packets received0 bad header checksums0 with size smaller than minimum0 with data size < data length0 with header length < data size0 with data length < header length0 with bad options0 with incorrect version number0 fragments received0 fragments dropped (dup or out of space)0 fragments dropped after timeout0 packets reassembled ok266328 packets for this host13 packets for unknown/unsupported protocol0 packets forwarded14007 packets not forwardable0 redirects sent14202 packets sent from this host0 packets sent with fabricated ip header0 output packets dropped due to no bufs, etc.0 output packets discarded due to no route0 output datagrams fragmented0 fragments created0 datagrams that can't be fragmented799024 IP Multicast packets dropped due to no receiver37 successful path MTU discovery cycles8 path MTU rediscovery cycles attempted0 path MTU discovery no-response estimates1 path MTU discovery response timeout0 path MTU discovery decreases detected39 path MTU discovery packets sent0 path MTU discovery memory allocation failures0 ipintrq overflows0 with illegal source0 packets processed by threads0 packets dropped by threads0 packets dropped due to the full socket receive buffer0 dead gateway detection packets sent0 dead gateway detection packet allocation failures

0 dead gateway detection gateway allocation failures

icmp:4559 calls to icmp_error0 errors not generated because old message was icmpOutput histogram:

echo reply: 6destination unreachable: 25

0 messages with bad code fields0 messages < minimum length0 bad checksums0 messages with bad lengthInput histogram:

echo reply: 37destination unreachable: 13echo: 6

6 message responses generatedigmp:

1 message received0 messages received with too few bytes0 messages received with bad checksum0 membership queries received0 membership queries received with invalid field(s)1 membership report received1 membership report received with invalid field(s)0 membership reports received for groups to which we belong4 membership reports sent

netstat Command

tcp:13593 packets sent

11369 data packets (4102411 bytes)89 data packets (108779 bytes) retransmitted1842 ack-only packets (1062 delayed)0 URG only packets0 window probe packets1 window update packet292 control packets

15930 packets received11390 acks (for 4104523 bytes)217 duplicate acks0 acks for unsent data9910 packets (154901 bytes) received in-sequence195 completely duplicate packets (592 bytes)0 old duplicate packets5 packets with some dup. data (1572 bytes duped)307 out-of-order packets (4588 bytes)0 packets (0 bytes) of data after window0 window probes262 window update packets0 packets received after close0 packets with bad hardware assisted checksum2 discarded for bad checksums0 discarded for bad header offset fields0 discarded because packet too short0 discarded by listeners579 ack packet headers correctly predicted3412 data packet headers correctly predicted

10 connection requests311 connection accepts316 connections established (including accepts)318 connections closed (including 5 drops)0 connections with ECN capability0 times responded to ECN5 embryonic connections dropped11180 segments updated rtt (of 11262 attempts)0 segments with congestion window reduced bit set0 segments with congestion experienced bit set0 resends due to path MTU discovery

netstat Command0 path MTU discovery terminations due to retransmits86 retransmit timeouts

0 connections dropped by rexmit timeout2 fast retransmits

0 when congestion window less than 4 segments0 newreno retransmits0 times avoided false fast retransmits6 persist timeouts

0 connections dropped due to persist timeout149 keepalive timeouts

149 keepalive probes sent0 connections dropped by keepalive

0 times SACK blocks array is extended0 times SACK holes array is extended0 packets dropped due to memory allocation failure0 connections in timewait reused0 delayed ACKs for SYN0 delayed ACKs for FIN0 send_and_disconnects0 spliced connections0 spliced connections closed0 spliced connections reset0 spliced connections timeout0 spliced connections persist timeout0 spliced connections keepalive timeout

udp:250354 datagrams received0 incomplete headers0 bad data length fields0 bad checksums4559 dropped due to no socket245436 broadcast/multicast datagrams dropped due to no

socket0 socket buffer overflows359 delivered381 datagrams output

entstat Command

The entstat command displays the statistics gathered by the specified Ethernet device driver.

This command is also invoked when the netstat command is run with the -v flag. The netstat command does not issue any entstat command flags.

usage: entstat [-drt] <device name>-d Displays all the statistics, including the device-specific statistics. -r Resets all the statistics back to their initial values. This flag can only be issued by privileged users. -t Toggles debug trace in some device drivers.

root@bmserver[/tmp]>entstat -d ent0-------------------------------------------------------------ETHERNET STATISTICS (ent0) :Device Type: IBM 10/100 Mbps Ethernet PCI Adapter (23100020)Hardware Address: 00:04:ac:17:74:a0Elapsed Time: 4 days 18 hours 8 minutes 43 seconds

Transmit Statistics: ReceiveStatistics:---------------------------------------Packets: 0 Packets: 0Bytes: 0 Bytes: 0Interrupts: 0 Interrupts: 0Transmit Errors: 6 Receive Errors: 0Packets Dropped: 1 Packets Dropped:0

Bad Packets: 0Max Packets on S/W Transmit Queue: 2S/W Transmit Queue Overflow: 0Current S/W+H/W Transmit Queue Length: 0

Broadcast Packets: 0 BroadcastPackets: 0Multicast Packets: 0 MulticastPackets: 0No Carrier Sense: 6 CRC Errors: 0DMA Underrun: 0 DMA Overrun: 0Lost CTS Errors: 0 Alignment Errors:0Max Collision Errors: 0 No ResourceErrors: 0Late Collision Errors: 0 Receive CollisionErrors: 0Deferred: 0 Packet Too ShortErrors: 0SQE Test: 0 Packet Too LongErrors: 0Timeout Errors: 0 Packets Discardedby Adapter: 0Single Collision Count: 0 Receiver StartCount: 0Multiple Collision Count: 0Current HW Transmit Queue Length: 0

entstat CommandGeneral Statistics:-------------------No mbuf Errors: 0Adapter Reset Count: 0Adapter Data Rate: 200Driver Flags: Up Broadcast Running

Simplex AlternateAddress 64BitSupportPrivateSegment DataRateSet

IBM 10/100 Mbps Ethernet PCI Adapter Specific Statistics:------------------------------------------------Chip Version: 26RJ45 Port Link Status : downMedia Speed Selected: Auto negotiationMedia Speed Running: UnknownReceive Pool Buffer Size: 384Free Receive Pool Buffers: 128No Receive Pool Buffer Errors: 0Inter Packet Gap: 96Adapter Restarts due to IOCTL commands: 1Packets with Transmit collisions:1 collisions: 0 6 collisions: 0 11collisions: 02 collisions: 0 7 collisions: 0 12collisions: 03 collisions: 0 8 collisions: 0 13collisions: 04 collisions: 0 9 collisions: 0 14collisions: 05 collisions: 0 10 collisions: 0 15collisions: 0Excessive deferral errors: 0x0

NFS Tuning ConceptsNFS Version 3 or Version 2

(Version 3 is recommended)

UDP or TCP protocol (NOTE: TCP is recommended)

Increase/decrease number of biods/nfsds

nfs socket size and sb_max

adapter transmit queues

Filesystem read/write buffer sizes

RAM for caching file pages (vmtune -P)

I/O tuning (multiple disks, separate log disk)

Tuning with nfso

nfsstat Command

Displays statistics on:Network File System (NFS)Remote Procedure Calls (RPC)

Syntax - /usr/sbin/nfsstat:-c client information-s server information-n NFS information only-r RPC information only-z reset statistics (root only)-m mount statisticsdefault: -csnr

nfsstat Command

root@bmserver[/tmp]>nfsstat -rc

Client rpc:Connection oriented:calls badcalls badxids timeouts newcreds badverfs timers1 0 0 0 0 0 0nomem cantconn interrupts0 0 0Connectionless:calls badcalls retrans badxids timeouts newcreds badverfs1448 0 12 0 12 0 0timers nomem cantsend22 0 0

nfsstat Command

# nfsstat -m/nfs/retain/pmrs from /nfs/retain/pmrs:ciaFlags:vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5Lookups: srtt=2 (5ms), dev=2 (10ms), cur=1 (20ms)All: srtt=2 (5ms), dev=2400 (12000ms), cur=600 (12000ms)

/nfs/retain/bin from /nfs/retain/bin:ciaFlags:vers=2,proto=udp,auth=unix,hard,intr,dynamic,rsize=8192,wsize=8192,retrans=5Lookups: srtt=14 (35ms), dev=6 (30ms), cur=4 (80ms)Reads: srtt=18 (45ms), dev=5 (25ms), cur=4 (80ms)All: srtt=15 (37ms), dev=7 (35ms), cur=5 (100ms)

/nfs/cust from /nfs/cust:zenFlags: vers=3,proto=tcp,auth=unix,soft,intr,link,sym-link,rsize=32768,wsize=32768,retrans=5All: srtt=0 (0ms), dev=2400 (12000ms), cur=600 (12000ms)

nfsstat Command# netstat -p udp

udp:27136572 datagrams received0 incomplete headers0 bad data length fields0 bad checksums20 dropped due to no socket8 broadcast/multicast datagrams dropped due to no socket39476 socket buffer overflows27097068 delivered177636 datagrams output

socket buffer overflows = Too few nfsd daemons and/or too small nfs_socketsize

Increase the number of nfsd daemons (NFS V2)

Increase value of nfs_socketsize and sb_max

nfso Command

nfso Command# nfso -aportcheck= 0udpchecksum= 1nfs_socketsize= 60000nfs_tcp_socketsize= 60000nfs_setattr_error= 0nfs_gather_threshold= 4096nfs_repeat_messages= 0nfs_udp_duplicate_cache_size= 0nfs_tcp_duplicate_cache_size= 0nfs_server_base_priority= 0nfs_dynamic_retrans= 1nfs_iopace_pages= 0nfs_max_connections= 0nfs_max_threads= 8nfs_use_reserved_ports= 0nfs_device_specific_bufs= 1nfs_server_clread= 1nfs_rfc1323= 0nfs_max_write_size= 0nfs_max_read_size= 0nfs_allow_all_signals= 0