Horizon View Troubleshooting:

Looking under the Hood

Matt Coppinger, VMware

Jack McMichaels, VMware

EUC5291

#EUC5291

2

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these features in

any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new technologies or features

discussed or presented have not been determined.

3

Agenda

Common Issues

What can go wrong?

Domain 1: View Client Connectivity issues

Domain 2: Desktop Not Available!

Domain 3: Broken Broker

Domain 4: Why Is My Desktop So Slow!?

Resources

4

Top 6 GSS tickets

1. SSL, SSL, SSL – View 5.1/5.2 introduces security improvements

• "Obtaining SSL Certificates for VMware View Servers” PDF on vmware.com

2. Persona – Make sure permissions are set correctly, and avoid

this common misconfiguration error.

http://kb.vmware.com/kb/2019502

3. Parent VM issues – Do NOT P2V! Clean VM please…

4. Connectivity – Understand the View network requirements!

5. PCoIP sizing – PCoIP Calculator -

http://myvirtualcloud.net/?page_id=1562

6. PCoIP Log Viewer - http://mindfluxinc.net

5

Generating a Certificate Signing Request – the Right Way

Read the Manual!

Create a config file

Generate cert signing

request (CSR)

Validate CSR and Private

Key

Send CSR to CA

Receive Signed Cert

from CA

Import Cert Configure

View Server to use Cert

Test!

6

The Ultimate Resource

• http://pubs.vmware.com/view-52/topic/com.vmware.ICbase/PDF/horizon-view-52-obtaining-certificates.pdf

Obtaining SSL

Certificates

for Horizon View

7

Understanding View Connectivity Requirements

8

Broker Specifications

Common Issues resulting from undersizing a broker:

• Memory Heap Issues

• Threading issues

• Latency Issues

Recommendations:

• Always size your broker right from the beginning, avoid downtime or

issues later.

• Minimum recommended specifications:

• 2 vCPU / 10 GB RAM for normal sizing (50-500 VMs)

• 4 vCPU / 16+ GB RAM for large sizing (500+ VMs)

9

Troubleshooting Keys

Check View Administrator Dashboard

• Shows system health and any issues

Understand the client connection process (where most problems lie)

Set the Logging Level on CS

Check Connection Broker Logs

• Match and Filter SessionID, User, FSP

Check View Agent Logs

Check View Desktop PCoIP Logs

Use kb.vmware.com!

10

Tracking Sessions

Use BareTail/BareGrep or similar 3rd party text parsers

11

You’ve Deployed View. What Could Go Wrong?

You’ve got a problem:

• I can’t even connect to View

• I get disconnected randomly!

• Why is the display so bad?

• Why is my desktop not available?

• I’m seeing an error in View, what does it mean?

• vCenter is reporting an error

• My desktop is slow…

12

Identifying the Problem Domain

View Client

Network

View Manager

View Composer

vCenter Server

Compute

Storage

13

Domain 1: View Client Connectivity Issues

Common challenges

• View Client can’t connect

• Logon failure

• Black screen

• Poor quality display

• Randomly disconnected session

14

Domain 1: Failure to Communicate…

Where to look

• Connection Broker logs

• Windows 2008 - <DriveLetter>:ProgramData\Application Data\VMware\VDM\logs

• Event Database

What to look for –

• (Client connects) [SimpleAJPService] (ajp:broker:Request9) Request from

/192.168.2.1: POST /broker/xml

• (Broker authentication) [WinAuthFilter] (SESSION:7072-***-a79c mattc) Attempting

to authenticate user 'mattc' in domain 'FUTUREOFFICE’

• (User has authenticated to Broker) [AuthorizationFilter] (SESSION:7072-***-a79c)

User FUTUREOFFICE\mattc has successfully authenticated to VDM

• (Audit Entry) [Audit] (SESSION:7072-***-a79c)

BROKER_LOGON:USER:FUTUREOFFICE\mattc;USERSID:S-1-5-21-326850759-

2560684469-1780228732-1113;USERDN:CN=S-1-5-21-326850759-2560684469-

1780228732-1113,CN=ForeignSecurityPrincipals,DC=vdi,DC=vmware,DC=int;

• Event Database: BROKER_USERLOGGEDIN

15

User Experience Issues

Black screen of death – instead of desktop!

• PCoIP port blocked (TCP and UDP 4172) or SVGA Driver issue

• pcoip_server/client logs - C:\Users\All Users\VMware\VDM\logs

• Error attaching to SVGADevTap, error 4000: EscapeFailed

• MGMT_SCHAN :scnet_client_open: tera_sock_connect returned error 10060 -

Connection timed out!

• Incorrect PCoIP External URL configured for Security/Connection Servers

16

User Experience Issues

Poor quality display

• Bandwidth, latency, or QoS

• Pcoip_server logs report

• VGMAC :Stat frms: Loss=0.45%/0.21% (R/T)

• MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438

Randomly disconnected session?

• 15 min after established - wssm process hasn't started on desktop

• View Agent logs (<DriveLetter>:\ProgramData\VMware\VDM\logs)

• PENDING_EXPIRED

• Sometimes caused by daisy-chaining the GINA (WinXP)

17

Domain 2: Desktop Not Available

Common Issues

• No Desktop Available

• Pool provisioning issues – customization

• Agent not communicating with broker

• Stuck at desktop login screen (SSO)

Where to look

• Connection Broker/ View Agent logs

• Event Database

What to look for

• Broker returns list of desktops available to client

• [DesktopsHandler] (SESSION:7072-***-a79c) For user [S-1-5-21-326850759-2560684469-1780228732-1113] and pool [cn=gold-np,ou=server groups,dc=vdi,dc=vmware,dc=int] DesktopTracker returned 2 guest DNs

18

Desktop Not Available

What to look for… (walk through successful connection)

Client requests desktop

• Event Database: BROKER_DESKTOP_REQUEST

Broker allocates session to user

• [FarmImp] (SESSION:7072-***-a79c) cn=3f974017-409f-4912-83bc-

2ee794f22fab,ou=servers,dc=vdi,dc=vmware,dc=int, total session count: 0

• [FarmImp] (SESSION:7072-***-a79c) allocateNewSession - identified server

for application CN=GOLD-NP,OU=Applications,DC=vdi,DC=vmware,DC=int

• Event Database: BROKER_MACHINE_ALLOCATED

Broker attempts SSO

• [FarmImp] (SESSION:7072-***-a79c) Using domain for SSO:

FUTUREOFFICE**

• User won’t be logged on to the VM without this!

19

ADSI Edit – Check Desktop Allocated

20

ADSI Edit – Common Key Values to Inspect

pae-DisplayName

• VM name as displayed in View Admin

pae-DirtyForNewSessions

• Indicates whether the VM is “Dirty” and can be re-used in a non-persistent pool

pae-SVIVMSnapshot

• Indicates the current Snapshot that is in use

pae-VmPath

• Indicates the full Path to the VM in vCenter

pae-VmState

• Indicates the current state of the Desktop – some states are a combination of

this value and other values

21

ADSI Edit – Searching for a Desktop

Find VMs with a Snapshot:

• (&(objectClass=pae-VM)(pae-SVIVmSnapshot=/Baseline/Snapshot1/Snapshot2))

Find VMs with a Name:

• (&(objectClass=pae-VM)(pae-DisplayName=Desktop-234))

22

Desktop Not Available

What to look for…

Broker starts session on VM

• [DesktopSessionImp] (SESSION:7072-***-a79c) startSession – sending StartSession message

Agent responds…

• "DesktopManager got a StartSession message”

• Client Info should be in Agent Log along with PCoIP launch

Event Database: AGENT_PENDING

[DesktopSessionImp] (SESSION:7072-***-a79c) startSession completed:

[DesktopTracker] User FUTUREOFFICE\mattc connected to machine gold-np-2 for desktop gold-np

Client connects to VM (Agent)

• “PCoIPCnx::OnConnectionComplete Begin (PCOIP)”

• “WTS_SESSION_LOGON”

• Event Database: AGENT_CONNECTED

23

Event Database

24

Desktop Not Available

What to look for… Pool Provisioning

Desktops not available due to provisioning error?

• Check View Administrator for Pool status, check datastore capacity

• Check Event Database - BROKER_PROVISIONING_ERROR_*

• Check View Composer has network access to ESX hosts

Desktop not available due to customization?

• Check Desktop status – AGENT UNAVAILABLE

• Check View Dashboard

• Desktop Status > Preparing Desktops OR Problem Desktops

• Check Desktop connectivity to DNS/AD/Connection Server

25

Desktop Not Available

Desktop not available due to VM reset/crash?

• Check Desktop status – ALREADY USED

• Typical on refresh-on-logoff or delete-on-use desktops

• Broker never received an explicit logout message from the agent

• Missing AGENT_ENDED event in DB for VM

View Composer Issues associated with incorrect domain

credentials

• C:\ProgramData\VMware\View Composer\Logs\

• FATAL CSvmGaService - [svmGaService.cpp, 116] Domain join failed Error 5 (0x5):

Access is denied.

26

Domain 3: Broken Broker

Common Issues

• Cannot connect to vCenter

• View Composer errors/issues

• JMS connectivity

• ADAM replication failure

Where to look

• View Administrator

• Event Database

• Windows Event Logs

• View Composer Logs

• Connection Server Logs

27

Broken Broker

What to look for

ADAM Replication

• Check the Connection Broker Window Event Logs

• ADAM (VMwareVDMDS)-log

• Error: ADAM Replication

• Check ADAM replication status on Connection Server

• C:\WINDOWS\adam\repadmin.exe /showrepl localhost:389 DC=vdi,DC=vmware,DC=int

28

Broken Broker

vCenter Server Connectivity

• Admin UI will show RED status

• Check Event Database

• VC_DOWN events

• Impacts provisioning and power operations ONLY

• Check Connectivity from Connection Server to vCenter Server

• Check credentials used to connect to vCenter Server

• Attempt to login in directly to vCenter using vSphere Client

29

Broken Broker

What to look for…

View Composer

• VMs stuck in DELETING status

• VMs have been manually deleted – then pool/desktop deleted

• Causes Composer DB and VC DB to get out of sync

• Composer thinks VM already exists

• Orphaned VMs – KB-2015112 (kb.vmware.com)

• Desktop Composer Fault: 'Virtual Machine with Input Specification already exists‘

JMS Connectivity

• Split site architecture / firewall causes “split brain”

• View Dashboard shows RED status

• Connection Server Logs

• tracker REJOIN messages – JMS connectivity

• tracker RESYNC messages – messages being delayed

30

Cleaning Up Broken / Orphaned Pools

BACKUP

Disable Provisioning on Broker

Stop View Composer

• Remove Composer Database Objects

• Remove ADLDS Servers, Server Groups, and Applications

• Remove AD Computer Entries

• Remove vCenter objects

• Unprotect replicas with sviconfig

• SviConfig -operation=RemoveSviClone -VmName=replica-<guid> -AdminUser=administrator

-AdminPassword=passowrd -ServerUrl=https://localhost:18443/SviService/v2_0

Active Directory View Composer

AD LDS

31

One Query To Rule Them All (Proceed with Extreme Caution!)

Delete One VM

DELETE FROM dbo.SVI_VM_NAME WHERE NAME='replaceMe'

DELETE FROM dbo.SVI_COMPUTER_NAME WHERE NAME='replaceMe'

DELETE FROM dbo.SVI_SC_PDISK_INFO WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')

DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS WHERE PARENT_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')

DELETE FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe')

DELETE FROM dbo.SVI_REQUEST WHERE ID=(SELECT REQUEST_ID FROM dbo.SVI_TASK_STATE WHERE SIM_CLONE_ID=(SELECT ID FROM

dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'))

DELETE FROM dbo.SVI_SIM_CLONE WHERE VM_NAME='replaceMe'

Delete ALL VMs and Pools

DELETE FROM dbo.SVI_VM_NAME

DELETE FROM dbo.SVI_COMPUTER_NAME

DELETE FROM dbo.SVI_SC_PDISK_INFO

DELETE FROM dbo.SVI_SC_BASE_DISK_KEYS

DELETE FROM dbo.SVI_TASK_STATE

DELETE FROM dbo.SVI_REQUEST

DELETE FROM dbo.SVI_SIM_CLONE

DELETE FROM dbo.SVI_REPLICA

DELETE FROM dbo.SVI_DG_CUST_PROP

DELETE FROM dbo.SVI_DEPLOYMENT_GROUP

Note: Composer will auto clean replicas when no dependent SIM_CLONE objects are available.

32

Domain 4: Why is My Desktop So Slow?

Common Issues

• Storage IO bottleneck

• Memory contention

• CPU contention

• Network issues

Where to look

• vCenter Server

• ESXTOP

• vCops for View

• 3rd Party Tools?

33

Why is My Desktop So Slow?

What to look for

CPU

• Cluster/Host utilization < 90%

• VM utilization - %USED (ESXTOP)

• VM %RDY Time (ESXTOP) < 10

Memory

• Host utilization < 85%

• VM utilization

• Swapping / Ballooning SWCUR > 1 / MCTLSZ > 1 (ESXTOP)

Storage

• Disk Read Latency < 25ms

• ESXTOP DAVG or KAVG < 25ms (ESXTOP)

34

ESX Performance Snapshots

Use the following command to collect performance metrics for

8 hours on a host:

• for i in `seq 8`;do esxtop -a -b -d 5 -n 720 > $i.<hostname>.csv;done

The above command will create eight 100mb files consisting of

1 hour's worth of ESXTOP snapshots.

• -d = delay in seconds

• -n = iterations

• (-d 5 x -n 720 = 3600 seconds or 1 hour).

Useful for replaying performance data over wide time periods

for support to analyze!

35

CPU

36

Memory

37

Storage

38

vSphere Performance Management Tools – vCenter Ops Mgr

vCenter Operations Manager

• Aggregates metrics into workload, capacity and health scores

• Relies on dynamic thresholds

39

VCOPs – My Favorite Tool D

ata

sto

re L

ate

ncy

Data

sto

re T

hro

ughput

40

Getting Help

Read the product documentation

Double check your configuration!

Check kb.vmware.com for your issue

http://communities.vmware.com

Run the Support.bat to extract the logs

• Notice the diagnostic tests that run…

• http://kb.vmware.com/kb/1017939

• Check the product documentation for using VDMADMIN command for creating

various Data Collection Tool bundles

Submit a Support Request

41

Summary

Understand where the issue may lie

• Client? Agent? Server? Composer? vCenter? ESX?

Know the problem domains

Check the View Dashboard and Event Database

Identify the issue – know what a successful connections looks like

Check the logs

Use Performance Tools – ESXTOP, vCenter, vCenter Operations

for View

Get Help

42

Questions

43

Other VMware Activities Related to This Session

HOL:

HOL-MBL-1301

Horizon View from A to Z

Group Discussions:

EUC1006-GD

View with Andre Leibovici

EUC5291

THANK YOU

Horizon View Troubleshooting:

Looking under the Hood

Matt Coppinger, VMware

Jack McMichaels, VMware

EUC5291

#EUC5291