PowerVM AIX Monitoring Mitec W210A 2&3

PowerVM/AIX Monitoring

MITEC Session W 210A-2 & 3

Steve [email protected]

2013 IBM Corporation2

Agenda

What This Presentation Covers PowerVM Review Monitoring Questions Topology/System Aggregation VIOS Summary Metrics/Tools

CPU Memory IO


There are a plethora of Performance Management and Monitoring products and free software. Listings and links are provided for reference later in this presentation.

This presentation does not focus on any product solution you should review your requirements and contact the vendors that interest you. Many higher end products have trial versions or can demonstrate theirfunctionality in person. Many of the free products have vast capabilities but require study, experimentation and configuration work.

This presentation is focused on the basic metrics that should be monitored for assessing performance, and where you can find them in AIX

What This Presentation Covers


Logical Partitions (LPARs) are defined to be dedica ted or shared Shared partitions use whole or fractions of CPUs (smallest increment is 0.1, can be greater than 1.0) Dedicated partitions use whole number of CPUs, traditional Unix environment Dedicated-donating can donate free cycles to a shared pool to allow higher capacity

Shared Processor Pools are a subset or all of the p hysical CPUs in a system. Desire is to virtualize all partitions to maximize exploitatio n of physical resources.

Shared Pool partitions run on Virtual Processors (V P). Each Virtual Processor maps to one physical processing unit in capacity (a physica l core in AIX).

Capacity is expressed in the form of a number of 10 % CPU units, called Entitlement Entitlement value for an active partition is a guarantee of processing resources Sum of partition entitlements must be less than or equal to physical resources in shared pool Desired: Desired size of partition at boot time Minimum: Partition will start will less than desired, but wont start if Minimum not available Maximum: Dynamic changes to desired cannot exceed this capacity

Capped vs Uncapped Capped: Entitlement is a hard limit, similar to a dedicated partitions Uncapped: Capacity is limited by unused capacity in pool and number of Virtual Processors a

partition has configured

When the pool is constrained, Variable Capacity Wei ght setting provides automatic load balancing of cycles over entitlement

PowerVM Virtualization Definitions Review


How do we monitor for shared CPU pool constraints? AIX provides metrics to show physical and entitlement utilization on

each partition AIX optionally provides the amount of the shared pool that is idle -

when you are out of shared pool, you are constrained

What about Hypervisor metrics? There are really no metrics that see into the hypervisor layer Metrics like %hypv in AIX lparstat are dominated by idle time, as idle

cycles are ceded to they hypervisor layer

This presentation will help you with determining what performance metrics are important at the partition and frame level

Questions


What product is best for interactive and short-term analysis of AIX resources? nmon provides access to most of the important metrics required for

benchmarks, proof-of-concepts and regular monitoring CPU vmstat, sar, lparstat, mpstat Memory vmstat, svmon Paging vmstat Hdisk iostat, sar, filemon Adapter iostat, fcstat Network entstat, netstat Process ps, svmon, trace tools (tprof, curt) Threads ps, trace tools (curt)

nmon Analyser & Consolidator provide free and simple trend reports Not provided in AIX/nmon

Java/GC must use java tools Transaction times application specific Database database products

Questions


Ideal for benchmarks, proof-of-concepts and problem analysis

Allows high-resolution recordings to be made while in monitoring mode Records samples at the interactive monitoring rate AIX 5.3 TL12 & AIX 6.1 TL05

Usage Start nmon, use [ and ] brackets to start and end a recording

Records standard background recording metrics, not just what is on screen.

You can adjust the recorded sampling interval with-s [seconds] on startupInteractive options - and + ( +) do NOT change ODR interval

Generates a standard nmon recording of format:__.nmon

Tested with nmon Analyser v33C, and works fine

nmon On Demand Recording (ODR)


nmon ODR


The majority of commercial products supporting PowerVM/AIX monitoring provide access to all of the important virtualization metrics

Metric naming is a problem. In many cases, different implementations display the same data, but have different naming conventions. Any customer wanting to evaluate these products will have to have an AIX specialist study the products metric definitions to map apples-to-apples.

Many products pre-package user interface views that aggregate these metrics

But usage modes for recording, post-processing, capacity planning can all vary

Many customers are using products in older dedicated modes and are not aware of differences in monitoring virtualized systems

Customers interested in a particular solution should evaluate each product for complete support. Most, if not all, of these products are available for trial evaluation.

Currency/Naming


AIX LPARKPX_memrepage_Info

KPX_vmm_pginwait_InfoKPX_vmm_pgfault_Info

KPX_vmm_pgreclm_Info

KPX_vmm_unpin_low_Warn KPX_vmm_pgout_pend_InfoKPX_Pkts_Sent_Errors_InfoKPX_Sent_Pkts_Dropped_InfoKPX_Pkts_Recv_Errors_Info KPX_Bad_Pkts_Recvd_InfoKPX_Recv_pkts_dropped_Info KPX_Qoverflow_Info

KPX_perip_InputErrs_InfoKPX_perip_InputPkts_Drop_InfoKPX_perip_OutputErrs_Info KPX_TCP_ConnInit_InfoKPX_TCP_ConnEst_Info

KPX_totproc_cs_Info KPX_totproc_runq_avg_InfoKPX_totproc_load_avg_Info KPX_totnum_procs_InfoKPX_perproc_IO_pgf_Info KPX_perproc_nonIO_pgf_InfoKPX_perproc_memres_datasz_InfoKPX_perproc_memres_textsz_InfoKPX_perproc_mem_textsz_Info KPX_perproc_vol_cs_Info

KPX_Active_Disk_Pct_InfoKPX_Avg_Read_Transfer_MS_InfoKPX_Read_Timeouts_Per_Sec_InfoKPX_Failed_Read_Per_Sec_InfoKPX_Avg_Write_Transfer_MS_InfoKPX_Write_Timeout_Per_Sec_InfoKPX_Failed_Writes_Per_Sec_InfoKPX_Avg_Req_In_WaitQ_MS_InfoKPX_ServiceQ_Full_Per_Sec_InfoKPX_perCPU_syscalls_Info KPX_perCPU_forks_InfoKPX_perCPU_execs_Info

KPX_perCPU_cs_Info

KPX_Tot_syscalls_Info

KPX_Tot_forks_Info

KPX_Tot_execs_InfoKPX_LPARBusy_pct_WarnKPX_LPARPhyBusy_pct_Warn KPX_LPARvcs_Info

KPX_LPARfreepool_Warn KPX_LPARPhanIntrs_InfoKPX_LPARentused_Info KPX_LPARphyp_used_Info

KPX_user_acct_locked_Info KPX_user_login_retries_InfoKPX_user_idletime_Info

VIOSKVA_memrepage_InfoKVA_vmm_pginwait_InfoKVA_vmm_pgfault_InfoKVA_vmm_pgreclm_InfoKVA_vmm_unpin_low_Warn KVA_vmm_pgout_pend_Infov Networking KVA_Pkts_Sent_Errors_Info KVA_Sent_Pkts_Dropped_InfoKVA_Pkts_Recv_Errors_Info KVA_Bad_Pkts_Recvd_InfoKVA_Recv_pkts_dropped_InfoKVA_Qoverflow_InfoKVA_Real_Pkts_Dropped_Info KVA_Virtual_Pkts_Dropped_InfoKVA_Output_Pkts_Dropped_Info KVA_Output_Pkts_Failures_InfoKVA_Mem_Alloc_Failures_WarnKVA_ThreadQ_Overflow_Pkts_Info KVA_HA_State_InfoKVA_Times_Primary_Per_Sec_Info KVA_perip_InputErrs_InfoKVA_perip_InputPkts_Drop_Info KVA_perip_OutputErrs_InfoKVA_TCP_ConnInit_InfoKVA_TCP_ConnEst_Infov Process KVA_totproc_cs_InfoKVA_totproc_runq_avg_Info KVA_totproc_load_avg_InfoKVA_totnum_procs_InfoKVA_perproc_IO_pgf_Info KVA_perproc_nonIO_pgf_InfoKVA_perproc_memres_datasz_InfoKVA_perproc_memres_textsz_Info KVA_perproc_mem_textsz_InfoKVA_perproc_vol_cs_InfoKVA_Firewall_InfoKVA_memrepage_InfoKVA_vmm_pginwait_InfoKVA_vmm_pgfault_InfoKVA_vmm_pgreclm_InfoKVA_vmm_unpin_low_Warn KVA_vmm_pgout_pend_Infov Networking KVA_Pkts_Sent_Errors_Info KVA_Sent_Pkts_Dropped_InfoKVA_Pkts_Recv_Errors_Info KVA_Bad_Pkts_Recvd_InfoKVA_Recv_pkts_dropped_InfoKVA_Qoverflow_InfoKVA_Real_Pkts_Dropped_InfoKVA_Virtual_Pkts_Dropped_InfoKVA_Output_Pkts_Dropped_Info KVA_Output_Pkts_Failures_InfoKVA_Mem_Alloc_Failures_WarnKVA_ThreadQ_Overflow_Pkts_Info KVA_HA_State_InfoKVA_Times_Primary_Per_Sec_Info KVA_perip_InputErrs_InfoKVA_perip_InputPkts_Drop_Info KVA_perip_OutputErrs_InfoKVA_TCP_ConnInit_InfoKVA_TCP_ConnEst_Infov Process

VIOS (cont)KVA_totproc_cs_Info

KVA_totproc_runq_avg_Info KVA_totproc_load_avg_Info

KVA_totnum_procs_Info

KVA_perproc_IO_pgf_Info KVA_perproc_nonIO_pgf_InfoKVA_perproc_memres_datasz_InfoKVA_perproc_memres_textsz_InfoKVA_perproc_mem_textsz_Info KVA_perproc_vol_cs_Info

KVA_Firewall_Info

KVA_Active_Disk_Pct_Info KVA_Avg_Read_Transfer_MS_InfoKVA_Read_Timeouts_Per_Sec_InfoKVA_Failed_Read_Per_Sec_InfoKVA_Avg_Write_Transfer_MS_InfoKVA_Write_Timeout_Per_Sec_Info

KVA_Failed_Writes_Per_Sec_InfoKVA_Avg_Req_In_WaitQ_MS_InfoKVA_ServiceQ_Full_Per_Sec_Info

KVA_perCPU_syscalls_Info

KVA_perCPU_forks_Info

KVA_perCPU_execs_Info

KVA_perCPU_cs_Info

KVA_Tot_syscalls_Info KVA_Tot_forks_InfoKVA_Tot_execs_Info

KVA_LPARBusy_pct_Warn KVA_LPARPhyBusy_pct_Warn

KVA_LPARvcs_Info

KVA_LPARfreepool_Warn

KVA_LPARPhanIntrs_Info

KVA_LPARentused_Info

KVA_LPARphyp_used_Info KVA_user_acct_locked_InfoKVA_user_login_retries_Info

KVA_user_idletime_Info

HMC

KPH_Busy_CPU_Info

KPH_Paging_Space_Full_Info

KPH_Disk_Full_Warn

KPH_Runaway_Process_InfoThe

Currency/Naming Example (Tivoli)


Excerpts from Agent definitions:

BYLS_CPU_PHYS_TOTAL_UTIL: On AIX, this metric is eq uivalent to sum of BYLS_CPU_PHYS_USER_MODE_UTIL and BYLS_CPU_PHYS_SYS_MODE_UTIL. For AIX lpars, the metric is calculated with respect to the available physica l CPUs in the pool to which this LPAR belongs to

BYLS_LS_MODE This metric indicates whether the CPU entitlement for the logical system is Capped or Uncapped.

BYLS_LS_SHARED This metric indicates whether the ph ysical CPUs are dedicated to this logical system or shared.

GBL_CPU_PHYS_TOTAL_UTIL: The percentage of time the available physical CPUs were not idle for this logical system during the interval. This m etric is calculated as GBL_CPU_PHYS_TOTAL_UTIL =GBL_CPU_PHYS_USER_MODE_UTIL + GBL_CPU_SYS_MODE_UTIL

GBL_POOL_CPU_AVAIL: The available physical processo rs in the shared processor pool during the interval.

GBL_POOL_CPU_ENTL: The number of physical processor s available in the shared processor pool to which this logical system belongs. On AIX SPLPAR, this metric is equivalent to "Active Physical CPUs in system" field of 'lparstat -i' command. On a standalone system, the value is "na".

GBL_POOL_TOTAL_UTIL: Percentage of time, the pool C PU was not idle during the interval.

Currency/Naming Example (HP)


AIX Service Tool PERFPMR:ftp://ftp.software.ibm.com/aix/tools/perftools/perfpmr/

IBMs PM for Power SystemsSubscription reports for capacity planninghttp://www-03.ibm.com/systems/power/support/pm/index.html

The AIX Community Wikihttp://www.ibm.com/systems/p/community/

AIX Wiki, use links under Performance to find nmon, nmon Analyser and other tools:

https://www.ibm.com/developerworks/community/wikis/home?lang=en#/wiki/Power%20Systems/page/AIX

IBM AIX Links


This is not an all-inclusive list, search for the VIOS Recognized here:http://www-304.ibm.com/partnerworld/wps/pub/systems/power/solutions

ATS Group Galileo http://www.theatsgroup.com/ BMC ProactiveNet Performance Management

http://www.bmc.com/products/product-listing/ProactiveNet-Performance-Management.html

CA http://www.ca.com HP GlancePlus http://www.hp.com/go/software

Metron Athene http://www.metron-athene.com/index.htmll Orsyp Sysload http://www.orsyp.com/products/software/sysload.html Power Navigator http://www.mpginc.com SAP AIX CCMS Agents http://www.sap.com Teamquest

http://www.teamquest.com/products-services/full-product-services-list/index.htm

ISV Monitoring Products


CEC Central Electronics Complex (System)

ProcessorsCPU type, frequency, count

# of processors in the shared pool, pool utilization

Total CEC utilizationNumber of unused processors

I/OSet of storage adaptersSet of network adapters

Other adaptersVirtual I/O connections

MemoryTotal amount of memory in CEC,

Memory allocated to partitionsUnallocated memory

PartitionsActive partitions,

operating systems,names and typesinactive partitions

System Topology


Virtualization allows and even encourages that a system contains multiple independent partitions, all with resources

Ideally, monitoring tools will easily determine all of the active partitions on a system and organize the data for those partitions together Normally, this requires consulting with an HMC to identify the active

partitions on a system With mobile partitions, the tools must accommodate the movement of

partitions between systems If automatic detection of partitions to systems isnt possible, some means

of organizing them manually must be applied

Various products do automatic CEC aggregation

Nmon Analyzer and Consolidator allow manual aggregation

The most important metric for CPU pool monitoring is the Available Pool Processor (APP) value. It represents the amount of a pool not currently consumed, and can be retrieved from individual partitions or calculated by aggregators

System Aggregation


We do not distinguish metrics between regular shared partitions and Virtual I/O Server partitions

All the metrics have the same meanings nmon recording works on both We treat the VIOS just like wed treat dedicated partitions hosting adapters As many VIOS host Shared Ethernet devices, we will cover this information in the

network section

Properly configured, a VIOS system can perform as well and achieve the same throughputs as a dedicated system

We tune the OS, adapters, queue depths exactly the same All IBM storage devices have Redbooks with best practices/performance sections See Best Practices section for VIOS recommendations

If you have IO performance issues, review the VIOS first Make sure you arent out of CPU Make sure you arent out of memory

A persistent complaint from customers is that there are no breakdowns of NPIV clients activity on VIOS, and each client must be individually monitored. We are working on it.

VIOS Performance Overview


VIOS overhead is not a problem

Virtual SCSI architecture between client and server is simple, a read is a read and a write is a write. These are in-memory transfers.

CPU overhead is a tiny fraction of the overall I/O time VIOS cpu entitlements are typically fractions of a physical core, more Virtual

Processors are better than running VIOS dedicated Always run production VIOS uncapped

Monitor the same CPU and memory statistics covered later in this presentation

Adjusting the VIO servers Variable Capacity Weighting (biasing a partitions access to free cycles when the shared pool is constrained) is advised for heavier production IO workloads

Hypervisor will allocate private memory for adapters depending on type and number. Allocations supported by the System Planning Toolhttp://www.ibm.com/systems/support/tools/systemplanningtool/

VIOS CPU & Memory


CPU Number of CPUs

Dedicated Shared pool Unallocated

Entitlement settings Utilization %

Dedicated consumed Pool consumed (alternatively, entitlement consumed or free)

Memory Allocated, in use, computational, non-computational Unallocated

IO Aggregated adapter totals read, write, IOPS, MB/sec

Metrics: CEC


HMC -> System Management -> Server -> Operations -> Utilization Data

Change Sampling Rate 30 seconds, 1/5/30 minutes,

1 hourView

Snapshot, Hourly, Daily, Monthly

Beginning and ending time/date

All, Utilization sample, Configuration change, Utility CoD usage

Maximum number of events

HMC Performance Information


System Utilization Views System summary Partition Processor/Memory Physical Processor, Shared Process or Shared Memory Pool

HMC Performance Information


Unix CPU buckets utilization%User: Fraction of entitlement consumed in user program%System: Fraction of entitlement consumed running in the AIX kernel

The time in the kernel could be from system calls made by user programs, interrupts, or kernel daemons

Sometimes %system is used as an indicator of health. If system becomes very high, it *might* mean that a problem exists. But, some subsystems such as the NFS server code run in system mode, so this is not always a good indicator

%Idle: Fraction of entitlement that the partition was not running any processes or threads

%I/O wait: Fraction of the entitlement that the partition was not running any processes or threads, and there was disk I/O outstanding

%I/O wait is also sometimes used as a health indicator. However, like system time, it is not always a good indicator due to the nature of the programs that may run

%User + %System + %Idle + %I/O wait = 100% of a fixed entitlement

In the dedicated world, the sum of %user + %system is the utilization percentage or busytime of the dedicated partition

Physical Busy = (%User + %System) X (# of dedicated cpus)

Metrics: Dedicated CPU Partition


CPU buckets and cpu busy percentages no longer tell us physical utilization

This is because the number of physical cores being utilized is no longer a fixed whole number, and can always be changing

User and System percentages are relative to consumption A logical CPU can be 99% busy on 0.01 to 1.0 physical cores

For shared partitions, you must understand new metrics Entitlement: physical resources guaranteed to a shared partition

Can range from 0.1 to maximum number of physical cores When entitlement is not being used, it can be ceded back to the pool

Entitlement Consumed (reported as entc% or ec%) Physical Consumed (reported as physc or pc in different tools)

Entitlement Consumed Capped partitions can only go to 100% Uncapped partitions can go over 100%

How far depends on the number of Virtual Processors (VP) Active VP count defines # of physical cores a partition can consume If a partition has an entitlement of 0.5 and 4 VPs, the maximum entitlement

consumed possible is 800% (4.0/0.5 = 8), and maximum physical consumption is 4.0

Metrics: PowerVM Shared CPU Partition


POWER6 vs POWER7 SMT Utilization

Simulating a single threaded process on 1 core, 1 Virtual Processor, utilization values change. In each of these cases, physical consumption can be reported as 1.0.

Real world production workloads will involve dozens to thousands of threads, so many users may not notice any difference in the macro scale

Whitepapers on POWER7 SMT and utilizationSimultaneous Multi-Threading on POWER7 Processors by Mark Funkhttp://www.ibm.com/systems/resources/pwrsysperf_SMT4OnP7.pdfProcessor Utilization in AIX by Saravanan Devendranhttps://www.ibm.com/developerworks/mydeveloperworks/wikis/home?lang=en#/wiki

/Power%20Systems/page/Understanding%20CPU%20utilization%20on%20AIX

busy

idle

POWER6 SMT2

Htc0

Htc1

100% busy

busy

idle

POWER7 SMT4

~65% busy

idle

idle

busy

idle

POWER7 SMT2

~70% busy

busy

busy

100% busy

busy

busy

100% busy

Htc0

Htc1

Htc0

Htc1

Htc0

Htc1

Htc0

Htc1

Htc2

Htc3

busy = user% + system%


SMT, Dispatch Behavior & Consumption

0

1

2 3

SMT Thread

Primary

Secondary

Tertiary

POWER7 processors can run in Single-thread, SMT2, SMT4 modes Like POWER6, the SMT threads will dynamically

adjust based on workload SMT threads dispatch via a Virtual Processor (VP) POWER7 threads start with different priorities on

Primary, Secondary and Tertiary instances, but can be equally weighted for highly parallel workloads

POWER5 and POWER6 overstate utilization as the CPU utilization algorithm does not account for how many SMT threads are active One or both SMT threads can fully consume a physical core and utilization is 100% On POWER7, a single thread cannot exceed ~65% utilization. Values are calibrated

in hardware to provide a linear relationship between utilization and throughput

When core utilization reaches a certain threshold, a Virtual Processor is unfolded and work begins to be dispatched to another physical core


POWER6 vs POWER7 Dispatch

There is a difference between how workloads are distributed across cores in POWER7 and earlier architectures

In POWER5 & POWER6, the primary and secondary SMT threads are loaded to ~80% utilization before another Virtual Processor is unfolded

In POWER7, all of the primary threads (defined by how many VPs are available) are loaded to at least ~50% utilization before the secondary threads are used. Once the secondary threads are loaded, only then will the tertiary threads be dispatched. This is referred to as Raw Throughput mode.

Why? Raw Throughput provides the highest per-thread throughput and best response times at the expense of activating more physical cores

POWER6 SMT2busy

idle

POWER7 SMT4

~50% busy

idle

idle

busy

busy

~80% busy

Htc0

Htc1

Htc0

Htc1

Htc2

Htc3

Another Virtual Processor is activated at the utilization values below (both systems may have a reported physical consumption of 1.0):

VirtualProcessorActivate


POWER6 vs POWER7 Dispatch

proc0 proc1 proc2 proc3

Once a Virtual Processor is dispatched, the Physical Consumption metric will typically increase to the next whole number

Put another way, the more Virtual Processors you assign, the higher your Physical Consumption is likely to be

proc0 proc1 proc2 proc3

Primary

Secondary

Tertiaries

POWER7

POWER6 PrimarySecondary


POWER7 may activate more cores at lower utilization levels than earlier architectures when excess Virtual Processors are present

Customers may complain that the physical consumption (physc or pc) metric is equal to or possibly even higher after migrations to POWER7 from earlier architectures. They may also note that CPU capacity planning is more difficult in POWER7 (discussion to follow)

Expect every POWER7 customer with this complaint to also have significantly higher idle% percentages over earlier architectures

Expect that they are consolidating workloads and may also have many more VPs assigned to the POWER7 partition.

POWER7 Consumption: A Problem?


Because POWER5 and POWER6 SMT utilization will always be at or above 80% before another VP is activated, utilization ratios (80% or 0.8 of a core) and physc of 1.0 core may be closer to each other than POWER7 environments

Physical Consumption alone was close enough for capacity planning in POWER5/POWER6 and many customers use this

This may not be true in POWER7 environments when excess VPs are present

Under the default Raw throughput mode, customers that do not want to reduce VPs may want to deduct higher idle buckets (idle + wait) from capacity planning metric(s)

Physical Busy = (User + System)% X Reported Physica l Consumption

This is reasonable presuming the workload benefits from SMT. This will not work with single-threaded hog processes that want to consume a full core.

AIX 6.1 TL8 & AIX 7.1 TL2 offer an alternative VP activation mechanism known as Scaled Throughput. This can provide the option to make POWER7 behavior more POWER6 Like but this is a generalized statement and not a technical one.

POWER7 Consumption: Capacity Planning


Metrics: Shared Pool Monitoring, Step 1

The most important metric in PowerVM Virtualization monitoring is the Available Pool Processor (APP) value. This represents the current number of free physical core resources in the shared pool.

Only partitions with this collection setting can display this pool value it is obtained via a hypervisor mechanism

The topas C command calculates this value automatically because it collects utilization values from each AIX instance.

Change is dynamic and takes effect immediately for lparstat (various products may not see the value until they restart or recycle for recording agents, typically at beginning of day)


[email protected] /tmp # lparstat -h 1 4

System configuration: type=Shared mode=Capped smt=O n lcpu=4 mem=4096 psize=2 ent=0.40

%user %sys %wait %idle physc %entc lbusy app vcsw phint %hypv hcalls

----- ---- ----- ----- ----- ----- ------ --- ---- ----- ---- - ------

84.9 2.0 0.2 12.9 0.40 99.9 27.5 1.59 521 2 13.5 2093

86.5 0.3 0.0 13.1 0.40 99.9 25.0 1.59 518 1 13.1 490

Shows the number of available processors in the shared pool. The shared pool psize is 2 processors. Must set Allow performance information collection. View the properties for a partition and click the Hardware tab, then Processors andMemory.

Shows the percentage of entitled capacity consumed. For a capped partition the percentage will not exceed 100%; however, for uncapped partitions the percentage can exceed 100%.

Shows the number of physical processors consumed. For a capped partition this number will not exceed the entitled capacity. For an uncapped partition this number could match the number of processors in the shared pool; however, this my be limited based on the number of on-line Virtual Processors.

app

%entcShared Mode Only

physc

Pool sizeLogical CPUs

Entitlement

Metrics: APP in lparstat


Metrics: nmon APP and UtilizationCPU and Entitlement Consumption

Available Pool

p display option

2013 IBM Corporation

Available PoolLocal LPAR Physical Used

Other LPAR(s) Utilization of Pool =

Pool Size APP Local Utilization

Analyser can only see:

- Pool Size

- APP

- Local Utilization

LPAR(s) View: nmon Analyser


View depends on partitions selected

Breakdowns by dedicated and shared partitions

LPAR(s) View: nmon Consolidator


Dashes represent data not available at OS level

Can be provided via command-line

Topas can be configured to collect via ssh to HMC

AIX 5.3 TL-08 topas supports

View of each shared pool, if AIX partitions

CPU cycle donations made by dedicated partitions

topas CUpper section displays aggregated CEC informationLower section displays shared/dedicated data closely mimics lparstat

Topas CEC Monitor Interval: 10 Thu Jul 28 17:04:57 2006Partition Info Memory (GB) ProcessorMonitored : 6 Monitored :24.6 Monitored :1. 2 Shr Physical Busy: 0.30 UnMonitored: - UnMonitored: - UnMonitored: - Ded P hysical Busy: 2.40Shared : 3 Available :24.6 Available : -Dedicated : 3 UnAllocated: 0 UnAllocated: - HypervisorCapped : 1 Consumed : 2.7 Shared :1. 5 Virt. Context Switch: 632Uncapped : 1 Dedicated : 5 Phantom Interrupts : 7

Pool Size : 3Avail Pool :2.7

Host OS M Mem InU Lp Us Sy Wa Id PhysB Ent %EntC Vcsw PhI-------------------------------------shared-------- -------------------ptoolsl3 A53 c 4.1 0.4 2 14 1 0 84 0.08 0.50 15.0 208 0ptoolsl2 A53 C 4.1 0.4 4 20 13 5 62 0.18 0.50 36.5 219 5ptoolsl5 A53 U 4.1 0.4 4 5 0 0 95 0.04 0.50 7.6 205 2

------------------------------------dedicated------ -------------------ptoolsl1 A53 S 4.1 0.5 4 20 10 0 70 0.30ptoolsl4 A53 4.1 0.5 2 100 0 0 0 2.00ptoolsl6 A52 4.1 0.5 1 5 5 12 88 0.10

APP = Pool Size -Shared Physical Busy

Topas Partitions & CEC View


Run queue length is another well known metric of CPU usage It refers to the number of software threads that are ready to run, but have to wait

because the CPU(s) is/are busy or waiting on interrupts The length is sometimes used as a measure of health, and long run queues usually

mean worse performance, but many workloads can vary dramatically It is quite possible for a pair of single-threaded workloads to contend for a single

physical resource (batch, low run queue, bad performance) while dozens of multi-threaded workloads share it (OLTP, high run queue, good performance)

In a dedicated processor partition or a capped shared processor partition, the run queue length will increase with higher CPU utilization

The longer the run queue length, the more time that software threads must wait for CPU, increasing response times and generally degrading the end user experience

In an uncapped shared processor partition, the run queue length may not increase with higher consumption

Extra capacity of the shared processor pool can be used This assumes shared pool resources are available and the partition has adequate

Virtual Processors assigned to fluctuate with the demand Because run queue is not a consistent indicator depending on workload type and the ability

of shared partitions to vary physical CPU resources, run queue is no longer considered a good generic indicator

Tools output vmstat reports global run queue as r mpstat reports per logical CPU run queue as rq

Metrics: Partition CPU


Context switches The number of times a running entity was stopped and replaced by another Collected for Threads (operating system) and Virtual Processors (hypervisor) There are voluntary and involuntary context switches

How Many context switches are Too Many? No rules of thumb exist Voluntary: Not an issue because it means no work for the CPU Involuntary: Could be an issue, but generally the bottleneck will materialize in a

easier to diagnosis metric; such as, CPU utilization, physical consumption, entitlement consumed, run queue

How can context switch metrics be used? Establish a baseline and compare when system encounter performance problems When benchmarking or performing a PoC, if these values blow up, this is

indicative of software scaling issues (architecture, latches, locks, etc)

Tool outputs vmstat reports total context switches as cs mpstat reports total cs and involuntary ics lparstat reports virtual processor context switches as vcsw

Metrics: Partition CPU


Other operating systems focus only on total real used and unusedmemory. This is not enough for AIX.

Because AIX memory maps files, the length of the free list (or number of free pages in the partition) is not usually a good indicator of memory utilization If software does not actually use memory pages it requested, VMM does

not allocate them AIX does not actively scrub memory. It scans and frees pages

depending on demand The free list will likely spike when a large process exits, as those pages

will become free Default AIX memory tunings typically result in total memory use being

reported as near 100%

Partition Memory (AIX)


In AIX, you must understand the difference between these types:

Computational Code and process/kernel working storage This metric is not available visible to the HMC

File Cache (sometimes referred to as non-computational) File: legacy JFS file pages in memory Client: other file pages (JFS2, NFS, etc) in memory May be broken out in some tools or reported as one file cache value Cache will not be cleared from file systems until other IO workloads

write over them or the file systems are remounted

Computational% is the only memory metric that reall y matters in AIX. Computational memory ALWAYS has priority over file cache until you reach Computational = 97%

Metrics: Partition Memory (AIX)


When computational memory becomes short, or the memory management is mistuned, paging occurs. Paging is the worst consequence of memory problems, so it should be monitored. Metric names used in vmstat: pi: Reads from paging devices po: Writes to paging devices sr & fr: Pages scanned and freed to satisfy new allocation request. A

persistently high scan rate (tens of thousands of pages per second) and low free rate typically indicate the system is struggling to allocate memory.

All physical paging is bad. If you are not near the computational threshold, you may need APARs specific to 4K or 64K memory page defects.

Advanced features like Active Memory Sharing and Active Memory Expansion (AME) require a more complex set of metrics AME adds metrics for pages in (ci) and out (co) out of a compressed pool If a requested expansion factor is too aggressive, a memory hole can be

created and may result in paging

Metrics: Partition Memory (AIX)


How can I figure out process memory? The best tool is the svmon command Use O filters to view

Process or user breakdowns of cache, kernel, shared and private memory It is most important to filter out kernel and shared segments all sophisticated

software products share kernel extension, shared/memory mapped areas (SGA, PGA, etc), libraries and some code text

Ive used nmon, topas or svmon and I cant account for all the used memory. Where is it?

Likely you a lot of file inode & metadata cached This is only available via the proc file system. Worst case, these areas in AIX 6.1

can take up to 20% of system memory (tuned lower in AIX 7.1)

# cat /proc/sys/fs/jfs2/memory_usagemetadata cache: 254160896 (values in bytes)inode cache: 57212928total: 311373824

Metrics: Process & inode


The number of threads blocked waiting for a file sy stem I/O operation to complete.bkthr

The number of threads blocked waiting for a raw dev ice I/O operation to complete.p

The number of pages scanned sr and the number of p ages stolen (or freed) fr. The ratio of scanned to freed represents relative memory activity. The ratio will start at 1 and increase as memory contention increases. Exa mine Sr # of pages to steal Fr pages. Note: Interrupts are disabled a t times when lrud is running.

fr / sr

Page Space Page In and Page Space Page Out per seco nd, which represents paging.pi / po

File pages In and File pages Out per second, which represents I/O to and from a file system. fi / fo

Page

The number of frames of memory on the free list. N ote: A frame refers to physical memory vs. a page w hich refers to virtual memory.

fre

The number of active virtual memory pages, which re presents computational memory requirements. The ma ximum avm number divided by number of real memory frames equa ls the computational memory requirement.

avmMem

ory

# vmstat -I 1

System configuration: lcpu=2 mem=912MB

kthr memory page faul ts cpu -------- ----------- ------------------------------ -- ---------- -----------

r b p avm fre fi fo pi po fr sr in sy cs us sy i d wa1 1 0 139893 2340 12288 0 0 0 0 0 200 25283 496 77 16 0 71 1 0 139893 1087 4503 0 8 733 3260 126771 415 9291 440 82 15 0 33 0 0 139893 1088 9472 0 1 95 9344 100081 191 19414 420 77 20 0 31 1 0 139893 1087 12547 0 6 0 12681 13407 207 25762 584 71 21 0 71 2 0 140222 1013 6110 1 39 0 6169 6833 160 15451 471 83 11 0 51 2 0 139923 1087 6976 0 31 2 7062 7599 183 19306 544 79 14 0 7

Key Points:

Computational (avm)

Paging Rates

Scanning Rates

Memory Monitoring with vmstat


Where is computational memory? vmstat

# vmstat -v1048576 memory pages Total Real Memory

992384 lruable pages Memory addressable by lrud668618 free pages Free List

1 memory pools One lrud per memory pool152370 pinned pages

80.0 maxpin percentage3.0 minperm percentage

90.0 maxperm percentage11.4 numperm percentage

113546 file pages JFS-only, or reports client if no JFS11.4 numclient percentage % file cache if JFS2 only90.0 maxclient percentage

113546 client pages JFS2/NFS

25.4 percentage of memory used for computational pages

New in AIX 6.1 TL06


Where is computational memory? svmon

# svmon -Gsize inuse free pin virtual

memory 233472 125663 107809 108785 140123pg space 262144 54233

work pers clnt lpagepin 67825 0 0 4096 0in use 79725 536 4442 0

Size: Total # of Memory Frames (Frames are in 4K units)

Inuse: # of Frames in Use

Free: # of Frames on Free List

Pin: # of pinned Frames

Virtual: Computational

Working (or computational) memory = 140123

%Computational = virtual/size = 140213 / 233472 = 60%

Pers (or JFS file cache) memory = 536

Clnt (or JFS2 and NFS file cache) memory = 4442


Where is computational memory? topas

Metrics related to Memory monitoring

%Comp Working memory

%Noncomp, %Client File memory


nmon Memory (m) + Kernel (s)

Computational

Run Queue + Process Context Switches

Scans & Frees (Steals)


Where is computational memory? nmon Analyser

Computational%

nmon Analyser cannot graph computational rates over 100% (physical paging)

System+Process = Computational


Cache(vmstat v client,

svmon pers + clnt)

Available Pool(lparstat)

Context Switches(vmstat, mpstat)

Run Queue(vmstat, sar q)

User/System/Idle/Wait(vmstat, lparstat, sar)

Dedicated PagingRAMShared

Memory CPU

Pages In/Out(vmstat, vmstat -s)

In Use(lspa a, svmon)

Total Size(lsps -a)

Scan Rate & Free Rate(vmstat sr & fr)

Computational(vmstat avm, vmstat v,

svmon virtual)

Physical consumedEntitlement consumed (lparstat, vmstat, sar)

Total Size(vmstat v memory

svmon size)

Nmon, nmon Analyser and topas provide all of these metrics

Metrics: CPU & Partition Memory/Paging


The following sample threshold tables are intended to be examples and not standard recommendations by IBM

We do not maintain or advise one set of thresholds for all environments

Thresholds


Sample Thresholds: CPU

Possible entitlement under-sizing

> 100% for 60 minutes, 3X dayEntitlement %

Workload dependentRun Queue

Highly variable, monitor for unusual spikes

Workload dependentContext Swithces

Pool size >= 4 cores

Ceiling = round up func()

< CEILING(#cores in pool x .10)

for 10 minutes

Available Pool

Adjusts for idle time in physical consumed, you must compute this

80% of lpar VPs

#VP x .80 = physb_thresh

Physical busy

cpu busy% x physc

90% of lpar Virtual Processors

#VP x .90 = physc_thresh

Physical consumed

(physc or pc)

relative to physical consumption in shared partitions

Dedicated > 80%

Shared n/a

cpu busy%

(user+sys)%

ExplanationThresholdMetric


Sample Thresholds: Memory

AIX 6.1 will use all free memfor file cache (~100% in use)

noneFree Memory %

Useful for determining what percentage of memory is just cache

noneFile Cache %

Any persistent paging indicates over computational rate or defect

AnyPhysical Paging in/out

Potential to crash> 80% of page spacePhysical Paging %

High filecache turnover, likely nearing 97% computational

> 8:1 and scanning > 10K pages/sec for 10 minutes

Scan:Free ratio

Assumes AIX 6.1 Physical paging begins at 97%

Green: < 80%Yellow: 80-92%

Red: > 92%

Computational %



We need to know how many I/Os are being read/written and the total bytes read/written to know where we are relative to bandwidth

nmon, topas and iostat provide aggregate values, but not relative to bandwidth

Many tools/products support this at the hdisk layer, and can become very complicated on large systems with hundreds or thousands of hdisks.

Many tools/products collect adapter statistics, but understanding their relationship to physical adapters in a virtualized environment can get complicated. For customers using newer features like NPIV, there is no simple aggregated view available within AIX at the VIOS level at this time

Metrics: Disk/Storage Activity


%utilization (or %busy) of devices and adapters

While device utilization is a commonly used metric, it is not always a good metric of quality of service, it can indicate for simple devices (SCSI) or imply for more complex devices (e.g. FC-attached LUNs) performance issues

Disk utilization measures the fraction of time that AIX has I/Os outstanding to a device. A very fast device (say a RAID5 4+1P LUN on DS4800) can process many I/Os per second. Even if a LUN is 100% busy, it may be offering good response time and be capable of processing more requests per second

These metrics are good for one thing sorting between hundreds or thousands of active vs in-active disks



Response & service times

Provide a much better view of whether I/Os are delayed. There are two commonly used measures of response time.

One is the time the I/O is queued at the device. This measures the responsiveness of the device to I/Os.

Another is the time the I/O is queued for service, which might include time queued in the operating system or device driver. If a large number of I/Os are launched at a device, the queued time may become an important metric.

AIX is well instrumented for these metrics now and they are the primary means for assessing storage performance



If IO service times are reasonably good, but queues are getting filled up, then Increase queue depths until:

You arent filling the queues or IO service times start degrading (bottleneck at disk)

For hdisks, queue_depth controls the maximum number of in-flight IOs

For FC adapters, num_cmd_elems controls maximum number of in-flight IOs

Drivers for hdisks and adapters have service and wait queues When the queue is full and an IO completes, then another is issued

Tools used on partitions to identify queue issuesSDDPCM:# pcmpath query devstats

# pcmpath query adaptstats SDD: # datapath query devstats

# datapath query adaptstats iostat: # iostat D fcstat: # fcstat fcs*

Queue Depths


hdisk1 xfer: %tm_act bps tps bread bwrtn

87.7 62.5M 272.3 62.5M 823.7

read: rps avgserv minserv maxserv timeouts fails

271.8 9.0 0.2 168.6 0 0

write: wps avgserv minserv maxserv timeouts fails

0.5 4.0 1.9 10.4 0 0

queue: avgtime mintime maxtime avgwqsz avgsqsz sqfull

1.1 0.0 14.1 0.2 1.2 60

Virtual adapters extended throughput report (-D)

Metrics related to transfers (xfer:)tps Indicates the number of transfers per second issued to the adapter.recv The total number of responses received from the hosting server to this adapter.sent The total number of requests sent from this adapter to the hosting server.partition id The partition ID of the hosting server, which serves the requests sent by this adapter.

Adapter Read/Write Service Metrics (read:)avgserv Indicates the average time. Default is in milliseconds.minserv Indicates the minimum time. Default is in milliseconds.maxserv Indicates the maximum time. Default is in milliseconds.

Adapter Wait Queue Metrics (wait:)avgtime Indicates the average time spent in wait queue. Default is in milliseconds.mintime Indicates the minimum time spent in wait queue. Default is in milliseconds.maxtime Indicates the maximum time spent in wait queue. Default is in milliseconds.avgwqsz Indicates the average wait queue size.qvgsqsz Indicates the average service queue size Waiting to be sent to the disk.sqfull Indicates the number of times the service queue becomes full.

Cant exceed queue_depth for the disk

If this is often > 0, then increase queue_depth

All D outputs are rates, except sqfull, which is an interval delta. Recent APARs change this to rate.

Service times. For SAN environment,

Reads > 10 msec & writes > 2 msec are high

Average IO Sizes

read = bread/rps

write = bwrtn/wps

Use l option

For wide output

hdisk service times: iostat


nmon adapter (a) + hdisk busy (o) and detail (D)

D

DDD

DDDD


# fcstat fcs0FC SCSI Adapter Driver Information No DMA Resource Count: 4490


Fibre channel adapter attributes:# lsattr -El fcs0bus_intr_lvl 8355 Bus interrupt level Falsebus_io_addr 0xffc00 Bus I/O address Falsebus_mem_addr 0xf8040000 Bus memory address Falseinit_link al INIT Link flags Trueintr_priority 3 Interrupt priority Falselg_term_dma 0x1000000 Long term DMA Truemax_xfer_size 0x100000 Maximum Transfer Size Truenum_cmd_elems 200 Maximum number of COMMANDS to queue to t he adapter Truepref_alpa 0x1 Preferred AL_PA Truesw_fc_class 2 FC Class for Fabric True

The max_xfer_size attribute also controls a DMA memory area used to hold data for transfer, and at the default is 16 MB

Changing to other allowable values increases it to 128 MB and increases the adapters bandwidth Usually not required unless adapter is pushing many 10s MB/sec Change to 0x200000 based on guidance from Redbooks or tools

This can result in a problem if there isnt enough memory on PHB chips in the IO drawer with too many adapters/devices on the PHB

Make the change and reboot check for Defined devices or errors in the error log, and change back if necessary

NPIV and virtual FC adapters the DMA memory area is 128 MB at 6.1 TL2 or later

FC Adapter Attributes


Option a for all adapters or ^ for FC adapters

nmon Analyser shows IO over time

nmon FC Monitoring


fcstat NPIV Monitoring NEW!

New breakdown by World Wide Port Name fcstat -n wwpn device_name Displays the statistics on a virtual port level that is specified by the

worldwide port number (WWPN) of the virtual adapter.

#fcstat -n C050760547E90000

FIBRE CHANNEL STATISTICS REPORT: fcs0

Device Type: 8Gb PCI Express Dual Port FC Adapter ( df1000f114108a03) (adapter/pciex/df1000f114108a0)Serial Number: 1B03205232Option ROM Version: 02781135ZA: U2D1.10X5World Wide Node Name: 0xC050760547E90000World Wide Port Name: 0xC050760547E90000FC-4 TYPES:

Supported: 0x00000120000000000000000000000000000000 00000000000000000000000000Active: 0x00000100000000000000000000000000000000 00000000000000000000000000

Class of Service: 3Port Speed (supported): 8 GBITPort Speed (running): 8 GBITPort FC ID: 0x010f00Port Type: FabricSeconds Since Last Reset: 431494

Transmit Statistics Receive Statistics------------------- ------------------

Frames: 2145085 1702630Words: 758610432 187172864

http://pic.dhe.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/p7hcg/fcstat.htm


~750 MB/sn/a57744 Gbps FC adapter (dual) PCI-e

750 MB/s per port simplex, 997 MB/s duplex per port1475 MB/s simplex per adapter, 2000 MB/s duplex per

142,00057358 Gbps FC dual port PCI-e

930 MB/s per port simplex, 1900 MB/s per port duplex1630 MB/s simplex per adapter, 2290 MB/s duplex per adapter

150,000570810 Gb FCoE PCIe Dual Port

400 MB/s simplex, ~750 MB/s duplexn/a57734 Gbps FC adapter PCI-e

DDR slots: ~750 MB/s, SDR slots: ~500 MB/s

n/a57594 Gbps FC adapter (dual)

DDR slots: 400 MB/s simplex, ~750 MB/s duplex, SDR slots: 400 MB/s simplex, 500 MB/s duplex

n/a57584 Gbps FC adapter (single port)

198 MB/s simplex, 385 MB/s duplex38,46157162 Gbps FC adapter (single port)

Sustained Sequential b/wIOPS 4KFCAdapter

Adapter Performance Chart


Network monitoring is different than disk monitoring, as it is not a master/slave model Disk requests follow a start/end model Network requests may be all send, all receive, or any arbitrary mix

Measure bandwidth A 1GB Ethernet link can sustain at most 125MB/sec of bandwidth If a link approaches the bandwidth limit, it is likely a point of resource

contention Knowledge of topology is important to identify if the links are 100MB,

1GB, or 10GB per second

Metrics: Network Monitoring


Measure packet rates. Each adapter will be limited to some maximum number of packets per second for small packets (though the cumulative bandwidth may still approach the wire limits)

Related health metrics may include: Packets dropped (a variety of reasons exist), collision errors, timeout

errors, etc These health metrics may imply topology issues or logical resource limits VIOS Performance Advisor will report on these errors

Metrics: Network Monitoring


How do I know network capacity? You should be able to reach 70-80% of line speeds with 1 Gb but 10 Gb

may require special tuning (beyond tcp/send receive, rfc1323, etc) On VIOS, if you are not near limits

Review CPU and Memory, always run uncapped Review netstat s/-v and entstat for any errors Review CPU, Memory and network on problem client

On 10 GB, if you are driving hundreds of thousands of tiny/smallpackets/sec at 1500 MTU, or have very short latency requirements, tuning will be required Two POWER7 cores will be required to reach 10Gb Large Send/Receive, mtu_bypass (assuming AIX clients only) Virtual buffer tunings (backup slide) Nodelay Ack, nagle, dog threads In extreme environments, using dedicated-donating mode on VIOS

If tuning exhausted and still have issues, then likely network environment or APARs required

Network Capacity


Network requirements are the same for CPU, regardless of operation as dedicated or shared lpar Driving 10 Gb adapters to capacity for workloads can take two physical

cores Larger packets, larger MTU sizes dramatically decrease CPU utilization

Integrated Virtual Ethernet vs Shared Ethernet IVE is the performance solution, will take more memory Shared ethernet 1 Gb performance is competitive with IVE, but 10 Gb

performance is more limited for small MTU receives (~5-6 Gb/sec) without tuning

Virtual Ethernet (Switch) Reliable < 1 ms latency times, but driving 1 Gb at normal packet sizes will

consume up to two physical cores Virtual switch within hypervisor is not designed to scale to 10 Gb at 1500

MTU for a single session (usually gated by core throughput for single session)

VIOS Network


Throughput is a function of entitled capacity

*POWER5 1.65 GHz

VIOS Network: CPU Capacity = Throughput

Effect of 'capping' on VIO TCP/IP throughput

020,00040,00060,00080,000

100,000120,000140,000

1 2 4 8 16 32 64 128

256

512

1024

2048

4096

8192

1638

4

Packet Size (Bytes)

Thro

ughp

ut

(KB

ytes

/Sec

ond)

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

CPU Entitlement(capped)


Review physical adapter values seastat nmon/topas entstat/netstat VIOS Performance Advisor

Check the virtual adapter Check CPU utilization Shared Ethernet

lsattr El en# entstat topas (AIX 6.1)

Shared Ethernet Tools


SEA Monitoring: seastat

Accounting must first be enabled per devicechdev -dev ent* -attr accounting=enabled

Command line for seastatseastat -d -c [-n | -s search_criterio n=value]

shared adapter device-c clears per-client SEA statistics-n displays name resolution on the IP addresses-s search values

MAC address (mac)VLAN id (vlan)IP address (ip)Hostname (host)Greater than bytes sent (gbs)Greater than bytes recv (gbr)Greater than packets sent (gps)Greater than packets recv (gpr)Smaller than bytes sent (sbs)Smaller than bytes recv (sbr)Smaller than packets sent (sps)Smaller than packets recv (spr)


SEA Monitoring: seastat$ seastat d ent5

=================================================== =============================Advanced Statistics for SEADevice Name: ent5=================================================== =============================MAC: 32:43:23:7A:A3:02----------------------VLAN: NoneVLAN Priority: NoneHostname: mob76.dfw.ibm.comIP: 9.19.51.76Transmit Statistics: Receive Stat istics:-------------------- -------------------Packets: 9253924 Packets: 112 75899Bytes: 10899446310 Bytes: 64519 56041=================================================== =============================MAC: 32:43:23:7A:A3:02----------------------VLAN: NoneVLAN Priority: NoneTransmit Statistics: Receive Stat istics:-------------------- -------------------Packets: 36787 Packets: 349 2188Bytes: 2175234 Bytes: 27220 7726=================================================== =============================MAC: 32:43:2B:33:8A:02----------------------VLAN: NoneVLAN Priority: NoneHostname: sharesvc1.dfw.ibm.comIP: 9.19.51.239Transmit Statistics: Receive Stat istics:-------------------- -------------------Packets: 10 Packets: 644 762Bytes: 420 Bytes: 48476 4292


SEA Monitoring: topas -E Usage

chdev dev ent* attr accounting=enabled

topas E or from topas screen hit E

Topas Monitor for host: P7_1_vios1 Interval: 2 Wed D ec 15 10:09:13 2010=================================================== ====================Network KBPS I-Pack O-Pack KB-In KB-Outent6 (SEA PRIM) 38.7 5.0 29.0 1.8 36.9|\--ent0 (PHYS) 19.6 4.0 14.0 1.7 17.9|\--ent5 (VETH) 19.2 1.0 15.0 0.1 19.0\--ent4 (VETH CTRL) 0.1 0.0 3.5 0.0 0.1lo0 2.7 14.0 14.0 1.3 1.3

Note: In order for this tool to work on a Shared Ethernet Adapter, the state of the layer-3 device (en) cannot be in the defined state. If you are not using the layer-3 device on the SEA, the easiest way to change the state of the device is to change one of its parameters. The following command will change the state of a Shared Ethernet Adapters layer-3 device, without affecting bridging.chdev -l -a state=down


SEA Monitoring: nmon O on VIOS


Packet Counts

Throughput in KB/sec

nmon Analyser

V34a SEA & SEAPACKET tab

SEA Monitoring: nmon O recording option

Unfortunately, Analyser does not provide stacked gra phs for SEA aggregation views


9000 MTU1500 MTUSessionsDirectionTest

1756 MB/s

1733 MB/s

1179 MB/s

1173 MB/s

1111 MB/s

1076 MB/s

Single port

237415 TPS

26171 TPS

1914 MB/s

1712 MB/s

992 MB/s

1015 MB/s

1402 MB/s

1311 MB/s

Both ports

182062 TPS 150

13324 TPS111 byte message

TCP_Request& Response

2176 MB/s1527 MB/s4

2106 MB/s1439 MB/s 1duplex

1393 MB/s925 MB/s4

1393 MB/s785 MB/s1receive

1668 MB/s1068 MB/s4

1647 MB/s870 MB/s1sendTCP STREAM

Both portsSingle port

Host: P7 750 4-way, SMT-2, 3.3 GHz, AIX 5.3 TL12, dedicated LPAR, dedicated adapter

Client: P6 570 with two single-port 10 Gb (FC 5769), point-to-point wiring (no ethernet switch)

1Single session 1/TPS round trip latency is 75 microseconds, default ISNO settings, no interrupt coalescing

AIX 6.1 should do better with SMT4, Disk I/O will do better due to larger blocks/buffers

FC over Ethernet Adapter (10Gb FC5708)


%IO relative to bandwidth (esitimate from adapter rates)

%IO relative to bandwidth(estimate from adapter rates)

Avg Service Time(s)(iostat D, sar d)

Service Queue Full or Wait(iostat -D, sar -d)

Read/Write IOPS & KB(iostat)

%busy, IO/sec, KB/sec(iostat, sar -d)

HdiskNetworkAdapter

StorageAdapter

IO

Packet errors, drops, timeouts(netstat)

Service Queue Counters(fcstat, MPIO pkg commands)

Send/Receive MB(entstat, netstat)

Read/Write Bytes(iostat -as, fcstat)

Send/Receive Packets(entstat, netstat)

IO/sec(iostat -as)

nmon, nmon Analyser provide metrics with yellow bac kground

Metrics: IO


Sample Thresholds: IO

Indicates queue depth tuning should be reviewed

Any persistence over 3 minHdisk queue full

Assuming tuned SANRead > 15 msec

Write > 2 msec

Hdisk service time

Use IBM ATS thruput charts80% of nominal capacityFC IOPS & Bytes

Expectation of 15K RPM disk

Hdisks comprising array of physical disks can go higher

> 300 IOPS per spindle, poor service times

Hdisk IOPS

Useful for benchmarkingn/aHdisk KB in/out

High packet rates typically require advanced tuning

> 100K/sec @ 10Gb

80% of nominal capacity

Network Packet/sec

Byte rates

Busy is not a reliable relative throughput metric in modern SANs, but useful for sorting most active hdisks

n/aHdisk %busy



The following are trademarks of the International B usiness Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademar ks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes : Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area.All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.HP is a registered trademark of Hewlett-Packard Development Company in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400, e business(logo), DBE, ESCO, eServer, FICON, IBM, IBM (logo), iSeries, MVS, OS/390, pSeries, RS/6000, S/30, VM/ESA, VSE/ESA, WebSphere, xSeries, z/OS, zSeries, z/VM, System i, System i5, System p, System p5, System x, System z, System z9, BladeCenter

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market.

Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

Trademarks

PowerVM AIX Monitoring Mitec W210A 2&3

Documents

Transcript of PowerVM AIX Monitoring Mitec W210A 2&3