vSphere APIs for performance monitoring
-
Upload
alan-renouf -
Category
Technology
-
view
13.706 -
download
5
description
Transcript of vSphere APIs for performance monitoring
![Page 1: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/1.jpg)
vSphere APIs for performance monitoringLondon Workshop
October – 2010
Balaji Parimi, Staff Engineer, Ecosystem Performance, VMware, Inc.
Ravi Soundararajan, Senior Staff Engineer, Performance, VMware, Inc.
![Page 2: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/2.jpg)
Motivation
To debug performance, why deal with this...?
![Page 3: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/3.jpg)
Motivation
When you can deal with this instead?
![Page 4: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/4.jpg)
More motivation
Why look at data like this…?
Before memhog: no guest swapping After memhog, guest swaps, but Host does not!
![Page 5: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/5.jpg)
More motivation
When you can look at it like this?
![Page 6: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/6.jpg)
Even more motivation…
Why compare resource pool performance like this…?
![Page 7: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/7.jpg)
Even more motivation
When you can compare them like this…?
![Page 8: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/8.jpg)
Why?
� vSphere gives you awesome, helpful charts
� But you don’t have to rely solely on these charts
� Do you want to learn how to make your own charts?
• �Keep watching
![Page 9: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/9.jpg)
Goal
Teach you how to use our APIs for performance monitoring
![Page 10: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/10.jpg)
Agenda
� What sorts of stats are useful?
� How does vSphere retrieve them?
� How can you get these stats and use them yourself?
![Page 11: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/11.jpg)
Useful stats
� Basics of performance monitoring in virtual infrastructure
• Find underperforming resources
• Find overcommitted resources
• Identify issues due to resource sharing among VMs
� Resources we will look at
• CPU
• Memory
• Disk
• Network
![Page 12: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/12.jpg)
Resources that we often look at
� ����CPU
� Memory
� Disk
� Network
![Page 13: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/13.jpg)
CPU basics
ESX
CPU0 CPU1 CPU2 CPU3
VM0 VM1 VM2 VM3
VM4
Run (accumulating used time)Ready (wants to run, no physical CPU available)Wait: blocked on I/O or voluntarily descheduled
VM5VM6
Run
Ready
Wait/Idle
![Page 14: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/14.jpg)
Why is my VM slow?
� CPU saturated (cpu.usage.average)
� Ready time? (cpu.ready.summation)
� Latency to be swapped in? (cpu.swapwait.summation)
![Page 15: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/15.jpg)
CPU saturation
2 vCPUs2.2GHz/CPU~4.4GHz used(Look at left y-axis)
![Page 16: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/16.jpg)
Small ready time
Ready time vCPU1: 150msReal-time chart: refresh 20s150ms / 20s = 0.75% (No big deal)Right y-axis is relevant
![Page 17: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/17.jpg)
Now, turn on CPU burner on same host…
CPU burner~100% of 1 vCPU
![Page 18: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/18.jpg)
And see what happens to original VM’s ready time
SpecJBB ready time~2000ms = 10%(ps. SpecJBB perf. dropped by 10%)
![Page 19: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/19.jpg)
Latency to load in VM: cpu.swapwait.average
Sometimes there is a latency to load VM data from dis k: cpu swapwait
CPU takes 20s to load in data before VM can run!
![Page 20: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/20.jpg)
CPU issues: Summary
� CPU saturated?
� High Ready time
• Problematic if it is sustained for high periods
• Sample rule of thumb: > 20% per vCPU � investigate further
• Possible contention for CPU resources among VMs
• Workload Variability? Fix with VMotion/DRS
• Resource limits on VMs? Check Limits, reservations and shares
• Actual over commitment? Fix with Vmotion/DRS/more CPUs
� High SwapWait time
• Consider setting memory reservation (see next section, “Memory”)
![Page 21: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/21.jpg)
Resources that we often look at
� CPU
� ����Memory
� Disk
� Network
![Page 22: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/22.jpg)
Memory
� ESX must balance memory usage
• Page sharing to reduce memory footprint of Virtual Machines
• Ballooning to relieve memory pressure in a graceful way
• Host swapping to relieve memory pressure when ballooning insufficient
• Compression to relieve memory pressure without host-level swapping
� ESX allows over commitment of memory
• Sum of configured memory sizes of virtual machines can be greater than physical memory if working sets fit
� Memory also has limits, shares, and reservations
� Host swapping can cause performance degradation
![Page 23: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/23.jpg)
VM1
Ballooning, compression, and swapping (1)
� Ballooning: Memctl driver grabs pages and gives to ESX• Guest OS choose pages to give to memctl (avoids “hot” pages if possible): either free
pages or pages to swap
• Unused pages are given directly to memctl
• Pages to be swapped are first written to swap partition within guest OS and then given to memctl
Swap partition w/inGuest OS
ESX
VM2
memctl
1. Balloon
2. Reclaim
3. Redistribute
F
![Page 24: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/24.jpg)
SwapPartition (w/in guest)
Ballooning, swapping, and compression (2)
� Swapping: ESX reclaims pages forcibly• Guest doesn’t pick pages…ESX may inadvertently pick “hot” pages (�possible VM
performance implications)
• Pages written to VM swap file
VM1
ESX
VM2
VSWP(external to guest)
1. Force Swap2. Reclaim3. Redistribute
![Page 25: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/25.jpg)
ESX
CompressionCache
Ballooning, swapping and compression (3)
� Compression: ESX reclaims pages, writes to in-memory c ache• Guest doesn’t pick pages…ESX may inadvertently pick “hot” pages (�possible VM
performance implications)
• Pages written in-memory cache � faster than host-level swapping
SwapPartition (w/in guest)
VM1 VM2
1. Write to Compression Cache2. Give pages to VM2
![Page 26: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/26.jpg)
Ballooning, swapping, and compression
� Bottom line:
• Ballooning may occur even when no memory pressure just to keep memory proportions under control
• Ballooning is preferable to compression and vastly preferably to swapping
• Guest can surrender unused/free pages
• With host swapping, ESX cannot tell which pages are unused or free and may accidentally pick “hot” pages
• Even if balloon driver has to swap to satisfy the balloon request, guest chooses what to swap
• Can avoid swapping “hot” pages within guest
• Compression: reading from compression cache is faster than reading from disk
![Page 27: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/27.jpg)
Swapping in Guest! = Swapping in Host
DVDstore benchmark: SQL DB benchmark… uses lots of memor y
About to start memory hogger program in guest
![Page 28: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/28.jpg)
Force Guest swapping: No Host-level swapping
Before memhog: no guest swapping After memhog, guest swaps, butHost does not!
![Page 29: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/29.jpg)
Viewing Host-level swapping with performance charts
Setup: 2 VMs…one dvdstore, one memhog, competing for host memoryHost swaps out dvdstore VM memory to fulfill memhog VM requestsHost swaps in dvdstore VM memory to fulfill dvdstore VM requests
![Page 30: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/30.jpg)
Using Swap Rate Counters: Remember CPU SwapWait?
Cpu.swapwait.summation: CPU is waiting for memory to be swapped in
![Page 31: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/31.jpg)
Absolute Swap Counters…
Swapin, swapout (KB) show some activity but hard to detect…
![Page 32: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/32.jpg)
And Swap Rate Counters…
SwapinRate, SwapoutRate (KBps) show activity much more clearlyRule of thumb: host swapping > 1MBps is cause for con cern
![Page 33: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/33.jpg)
Resources that we often look at
� CPU
� Memory
� ����Disk
� Network
![Page 34: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/34.jpg)
ESX storage stack
� Different latencies for local disk vs. SAN (caching, switches, etc.)
� Queuing within kernel and in hardware
� vSphere shows
• Total Command Latency
• Kernel Latency
• Device Latency
• Bandwidth/IOPS
![Page 35: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/35.jpg)
Disk performance problems 101
� What should I look for to figure out if disk is an i ssue?
• Am I getting the IOPs I expect?
• Am I getting the bandwidth (read/write) I expect?
• Are the latencies higher than I expect?
• Where is time being spent?
� What are some things I can do?
• Make sure devices are configured properly (caches, queue depths)
• Use multiple adapters and multipathing
• Check networking settings (for iSCSI/NAS)
![Page 36: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/36.jpg)
Another disk example: Slow VM power on
� Trying to Power on a VM
• Sometimes, powering on VM would take 5 seconds
• Other times, powering on VM would take 5 minutes!
� Where to begin?
• Powering on a VM requires disk activity on host � Check disk metrics for host
![Page 37: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/37.jpg)
Let’s look at the vSphere client…
�Max Disk Latencies range from 100ms to 1100ms…very high! Why?(counter name: disk.maxTotalLatency.latest)
Rule of thumb: latency > 20ms isBad.Here:1,100msREALLY BAD!!!
![Page 38: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/38.jpg)
High disk latency: Mystery solved
� Host events: disk has connectivity issues ���� high latencies!
� Bottom line: monitor disk latencies; issues may not be related to virtualization!
![Page 39: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/39.jpg)
Resources that we often look at
� CPU
� Memory
� Disk
� ����Network
![Page 40: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/40.jpg)
Network performance problems 101
� What should I look for to figure out if network is an issue?
• Am I getting the packet rate that I expect?
• Am I getting the bandwidth (read/write) I expect?
• Is all traffic on one NIC, or spread across many NICs?
• [more advanced… not available through counters]: out-of-order packets?
� What are some things I can do?
• Check host networking settings
• Full-duplex/Half-duplex
• 10Gig network vs 100Mb network?
• Firewall settings
• Check VM settings: all VMs on proper networks?
![Page 41: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/41.jpg)
Network performance troubleshooting
� Customer complains about slow network• She’s running netperf on a GigE Link
• She sees only 200Mbps
• Why? I bet it’s that VMware stuff!!
• Note to reader: Please don’t blame VMware first ☺
� Where do we start?
![Page 42: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/42.jpg)
All VMs using same NIC (VM network)
All VMs using “VM Network” and sharing 1 physical NIC
![Page 43: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/43.jpg)
Where do we begin? Check VM bandwidth
� Measure VM Bandwidth (net.transmitted.average)• 200 Mb/s
• Screenshot from the vSphere client
![Page 44: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/44.jpg)
Check Host Bandwidth
� Measure Host Bandwidth (net.transmitted.average)• Host sees around 900Mbps…why is VM at 200Mbps?
• Hmm… are we sharing this NIC with multiple VMs?
![Page 45: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/45.jpg)
All traffic is going through one NIC!
� Measure per-physical-NIC traffic
� Hmm… all VM traffic is going through 1 NIC
� Let’s split the VMs across NICs
All traffic through oneNIC on this host
![Page 46: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/46.jpg)
Split VMs across multiple NICs. Bingo!
![Page 47: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/47.jpg)
Network issues: Configuration woes
Network adapter set to “full duplex, 100 Mbps”:< 0.1Mbps!
Specific combo of switch and adapter caused thisperformance degradation!
Lesson: Check specs & configuration!
Network adapter set to “autonegotiate”: 90Mbps
![Page 48: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/48.jpg)
Agenda
� What sorts of stats are useful?
� ����How does vSphere retrieve them?
� How can you get these stats and use them yourself?
![Page 49: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/49.jpg)
Stats infrastructure in vSphere
ESX
VM VM VM VM VM
vCenter Server(vpxd, tomcat)
ESX
VM VM VM VM VM
ESX
VM VM VM VM VM
DB
1. Collect 20s and 5-min host and VM stats
2. Send 5-minstats to vCenter
3. Send 5-minstats to DB
4. Rollups
![Page 50: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/50.jpg)
Rollups
DB
1. Past-Day (5-minutes) ���� Past-Week2. Past-Week (30-minutes) ���� Past-Month3. Past-Month (2-hours) ���� Past-Year4. (Past-Year = 1 data point per day)
DB only archives historical data• Real-time (i.e., Past hour) NOT archived at DB• Past-day, Past-week, etc. � Stats Interval• Stats Levels ONLY APPLY TO HISTORICAL DATA
![Page 51: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/51.jpg)
Anatomy of a stats query: Past-hour (“RealTime”) St ats
Client
ESX
VM VM VM VM VM
vCenter Server(vpxd, tomcat)
ESX
VM VM VM VM VM
ESX
VM VM VM VM VM
DB
1. Query
2. Get stats from host
3. Response
No calls to DBNote: Same code path for past-day stats within last 30 minutes
![Page 52: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/52.jpg)
Anatomy of a stats query: Archived stats
Client
ESX
VM VM VM VM VM
vCenter Server(vpxd, tomcat)
ESX
VM VM VM VM VM
ESX
VM VM VM VM VM
DB
1. Query
3. Response
No calls to ESX host (caveats apply)Stats Level = Store this stat in the DB
2. Get Stats
![Page 53: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/53.jpg)
Agenda
� What sorts of stats are useful?
� How does vSphere retrieve them?
� ����How can you get these stats and use them yourself?
![Page 54: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/54.jpg)
Phew! Ok, How do I get these stats?
You want a chart like this?
� PowerCLI
• CPU Usage for a VM for last hour:
• $vm = Get-VM –Name “Foo”
• Get-Stat –Entity $vm –Realtime –Maxsample 180 –Stat cpu.usagemhz.average
• Grab appropriate fields from output, use graphing program, etc.
![Page 55: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/55.jpg)
Looks simple… What’s going on behind the scenes?
� To get stats, this is what is going on FOR EACH GET-STAT CALL
• Retrieve PerformanceManager
• QueryPerfProviderSummary $vm � Says what intervals are supported
• QueryAvailablePerfMetric $vm � Describes available metrics
• QueryPerfCounter � Verbose description of counters
• Create PerfQuerySpec � Query specification to get the stats
• QueryPerf � Get stats
Bottom line: The PowerCLI toolkit spares you detail s…Easy to use!
![Page 56: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/56.jpg)
PowerCLI Is so easy… Why use Java / C#?
� PowerCLI is great for scripting
• Stateless
• Hides details
� But with Java / C#
• You can squeeze out more performance!
• Much higher scalability
![Page 57: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/57.jpg)
Pseudo code
Get MOREF
for each Get-Stat {
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
QueryPerf();
}
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
PowerCLI Java
perfCounter propertyOf
PerformanceManager
![Page 58: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/58.jpg)
Performance implications: Need to write scalable sc ripts!
Entities(cpu.usagemhz.average)
PowerCLI(Time in secs)
Java(Time in secs)
1 VM 9.2 14
6 VMs 11 14.5
39 VMs 101 16
363 VMs 2580 (43 minutes) 50
�Java provides opportunities for scalable, ongoing stats collection�Let’s examine Java code in more detail…
A Naïve script that works for small environments may not be suitable for large environments
Highly-tunedJava Stats Collector
![Page 59: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/59.jpg)
GetPerfStats – Main method
Get MOREF
Get CounterIds
QueryAvailablePerfMetric
QueryProviderSummary
create PerfQuerySpec
QueryPerf
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
perfCounter
![Page 60: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/60.jpg)
GetPerfStats
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
![Page 61: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/61.jpg)
Get MOREF
Get the entity MOREF
![Page 62: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/62.jpg)
GetPerfStats
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
perfCounter propertyOf
PerformanceManager
![Page 63: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/63.jpg)
Get CounterIds
Get available counterIDsfrom perfCounter property of
PerformanceManager
Map human-readable stat name to counterID(e.g., cpu.usagemhz.average ���� 101)����QueryPerf (…) requires counterID
![Page 64: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/64.jpg)
GetPerfStats
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
![Page 65: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/65.jpg)
QueryPerfProviderSummary
• All VMs have same value• All Hosts have same value etc.����Call once for a given entity type and store result
![Page 66: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/66.jpg)
GetPerfStats
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
![Page 67: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/67.jpg)
Create PerfQuerySpec
Use wild card
CSV output format
![Page 68: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/68.jpg)
GetPerfStats
Get MOREF
QueryAvailablePerfMetric();
QueryPerfCounter();
QueryPerfProviderSummary();
create PerfQuerySpec();
for each Get-Stat {
QueryPerf();
}
![Page 69: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/69.jpg)
QueryPerf
![Page 70: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/70.jpg)
So, what is Java / C# buying us?
� Avoiding redundant work
� More compact return format (CSV vs. objects)
� Low-overhead tracking of ongoing inventory changes
� Etc.
� If we dig deeper, we can optimize even more…
![Page 71: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/71.jpg)
Digging deeper: The PerfQuerySpec architecture
� To grab counters: QueryPerf(PerfQuerySpec[] querySpec)
� PerfQuerySpec: Specifies which counters to grab
� PerfQuerySpec[]: [pQs1, pQs2, pQs3, …]
� Array of PerfQuerySpec objects pQs1, pQs2, pQs2
� Can grab multiple stats using single QueryPerf call
Entity (host, VM)
Format (CSV, normal)
MetricId StartTime EndTime IntervalID(20s, 300s)
maxSample
![Page 72: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/72.jpg)
Complexities of QueryPerf
How Does vSphere Process QueryPerf(querySpec[])?
1. vCenter receives queryPerf request with querySpec[]
2. vCenter takes each querySpec one at a time
3. vCenter gets data for each querySpec before processing next one
Options for querySpec[]:
1. 1 entry � 1 stat or set of stats for a single entity (e.g., all CPU)
2. Multiple entries. Examples:
• Each entry for a different entity …
• Each entry for a different stat type, same entity
VM1,cpu.* VM2,cpu.* H3,mem.*
VM1,cpu.* VM1,net.* VM1,mem.*
pQs1 pQs2 pQs3
![Page 73: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/73.jpg)
Implications of QuerySpec
Format of QuerySpec Allows Multiple Client Options
1. Grab each stat one at a time
2. Grab a group of stats per entity at once
3. Grab all stats for all entities at once
4. Grab stats for a subset of entities at once
Some Tradeoffs:
1. Network processing (large result sets vs. small result sets)
2. Client aggregation overhead
3. vCenter processing (Each QueryPerf handled in a single thread)
![Page 74: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/74.jpg)
What about in-guest stats?
� Using VIX APIs:
• Create a script that can get what ever stats you are interested in.
• Make the script write the stats to a file.
• Copy file from the guest.
• Session covering this topic
• � PPC-15 – Guest Operations using VMware VIX APIs and Beyond
![Page 75: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/75.jpg)
Back to the Future (1)
Now I know how to I convert this… (many metrics on different charts)
![Page 76: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/76.jpg)
Back to the Future (2)
To This (CPU, Memory, Disk, and Network on the same chart)
![Page 77: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/77.jpg)
Combining metrics across VMs & Hosts
![Page 78: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/78.jpg)
Combining metrics across VMs & Hosts
![Page 79: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/79.jpg)
Comparing resource pools
Use VIX API + vSphere counters to get RP performanc e data
![Page 80: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/80.jpg)
What about VMs running on a Host?
� Memory usage of VMs on a Host
![Page 81: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/81.jpg)
Summary, Part 1: Some useful Counters to monitor
Resource Metric Host or VM?
Description
CPU Usage Both CPU % used
Ready VM Ready to run, but limit or no available physical CPU
SwapWait VM CPU time spent waiting for host-level swap-in
Memory Swapin, swapinrate
Both Memory ESX host swaps in from disk (per VM, or cumulative over host)
Swapout, swapoutrate
Both Memory ESX host swaps out to disk (per VM, or cumulative over host)
Disk commands Both Operations done during stats refresh interval
totalLatency Host End-to-end disk latency (available for reads & writes)
Usage Both Disk bandwidth utilized (available for reads & writes)
Network Packets received, transmitted
Both Operations done during stats refresh interval
Usage Both Network bandwidth used (available for reads & writes)
![Page 82: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/82.jpg)
For completeness…VM memory metrics
Metric DescriptionMemory Active (KB) Physical pages touched recently by a virtual machine
Memory Usage (%) Active memory / configured memory
Memory Consumed (KB)
Machine memory mapped to a virtual machine, including its portion of shared pages. Does NOT include overhead memory.
Memory Granted (KB) VM physical pages backed by machine memory. May be less than configured memory. Includes shared pages. Does NOT include overhead memory.
Memory Shared (KB) Physical pages shared with other virtual machines
Memory Balloon (KB) Physical memory ballooned from a virtual machine
Memory Swapped (KB) Physical memory in swap file (approx. “swap out –swap in”). Swap out and Swap in are cumulative.
Overhead Memory (KB) Machine pages used for virtualization
![Page 83: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/83.jpg)
Host memory metrics
Metric DescriptionMemory Active (KB) Physical pages touched recently by the host
Memory Usage (%)* Active memory / configured memory
Memory Consumed (KB)
Total host physical memory – free memory on host. Includes Overhead and Service Console memory.
Memory Granted (KB) Sum of memory granted to all running virtual machines. Does NOT include overhead memory.
Memory Shared (KB) Sum of memory shared for all running VMs
Shared common (KB) Total machine pages used by shared pages
Memory Balloon (KB) Machine pages ballooned from virtual machines
Memory Swap Used (KB)
Physical memory in swap files (approx. “swap out –swap in”). Swap out and Swap in are cumulative.
Overhead Memory (KB) Machine pages used for virtualization
*For a cluster, mem.usage.average = (consumed + overhead) /total mem
![Page 84: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/84.jpg)
Summary, Part 2: Cheat sheet
� Rules of Thumb
• Ready Time > 20% sustained is undesirable
• Host-level swapping is bad, > 1MBps is especially bad
• Disk latencies > 20 ms BAD
• Use IOmeter to assess disk bandwidth and latency
• Network
• run netperf to get network baselines
![Page 85: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/85.jpg)
Summary, Part 3: SDK/API Tips and tricks
� Collect static data once
• CounterIDs, metricIDs, MOREFs etc.
• Use Views to keep this data up to date.
• Reuse PerfQuerySpec as much as possible
� Use CSV format
• Reduces serialization cost and the size of metadata
� Choose metrics and query intervals carefully
• Query the real-time stats at a slower rate than the refresh rate
• Choose correct stats levels
� Use parallelism (multi-threaded clients)
![Page 86: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/86.jpg)
Conclusion
� vSphere gives a bunch of awesome charts
� If you want to see the data differently, use the API
� PowerCLI is great for simple scripts
� When designing for scalability, consider Java / C#
![Page 87: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/87.jpg)
Resources
� Developer Support
• Dedicated support for your organization when building solutions using vSphere APIs, PowerCLI, vSphere Web Services SDKs and many more VMware SDKs
• http://vmware.com/go/sdksupport
� PowerCLI Training
• 2 day instructor led training, 40% lecture, 60% lab
• http://vmware.com/go/vsphereautomation
� VMware Developer Community
• SDK Downloads, Documentation, Sample Code, Forums, Blogs
• http://developer.vmware.com
� Technology Alliance Partner (TAP) Program
• Updated partner benefits
• http://www.vmware.com/partners/alliances/programs/
![Page 88: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/88.jpg)
Disclaimer
This session may contain product features that are currently under development.
This session/overview of the new technology represent s no commitment from VMware to deliver these features in any generally available product.
Features are subject to change, and must not be inclu ded in contracts, purchase orders, or sales agreements of any k ind.
Technical feasibility and market demand will affect final delivery.
Pricing and packaging for any new technologies or feat ures discussed or presented have not been determined.
“These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”
![Page 89: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/89.jpg)
Backup slides
![Page 90: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/90.jpg)
What about VMs across resource pools?
![Page 91: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/91.jpg)
Back to the Future (2)
To This (CPU, Memory, Disk, and Network on the same chart)
![Page 92: vSphere APIs for performance monitoring](https://reader034.fdocuments.in/reader034/viewer/2022050709/54b747444a7959f72e8b4585/html5/thumbnails/92.jpg)
Combining metrics across VMs & Hosts