Troubleshooting in the UCCX Solution · Troubleshooting in the UCCX Solution Arunabh Bhattacharjee,...

Troubleshooting in the UCCX Solution

Arunabh Bhattacharjee, Software Engineer, Customer Success

Customer Care Business Unit

BRKCCT-1593

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKCCT-1593

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4BRKCCT-1593

What’s most important is ..

Questions ?

Evidence …

Answers !!

• Problem Isolation

• System Monitoring Tools – CLI, CUIC & RTMT

• UCCX Engine - CUCM Interaction

• Cisco Finesse

• Live Data

• Database and Reporting

• HA Failover

• Chat and Email

Agenda

Problem Isolation


Troubleshooting

7BRKCCT-1593

Understand the Problem Statement

“Configuration was done a year ago, and

now we are facing ..”

“Agents are unable to login.”

“Appadmin is not responding.”

“Agents are not receiving calls.”

“Database Replication Down !”


Asking The Right Questions

• Configuration and Script Changes

• Recent Upgrades

• Firewalls, Browsers, ESXi, CUCM, Gateways, MS Exchange

• Patches (COP files)

• Failovers• Could unveil hidden DB replication issues, network issues

• Contact Volume Increases, performance issues

• Holidays, Certain Time of Day, etc…

8BRKCCT-1593

• WHEN? WHAT Changed?


• WHO is affected by the Problem?

• Are other Sites/Locations affected by the same issue?

• WHAT ELSE in the environment is affected?

• ISOLATION – Breaking the problem down, pinning it to a single problem suspect.

• Identify the EVIDENCE needed to CONFIRM the root cause.

Asking The Right Questions

9BRKCCT-1593

Evidence - System Monitoring Tools


RTMT – Real Time Monitoring Tool

• Java based plug-in for UCCX. Installable on Windows or Linux

11BRKCCT-1593


RTMT – System Summary

12BRKCCT-1593

Memory | CPU | Disk


CCX Engine JVM HeapRTMT – Performance Charts

13BRKCCT-1593


CCX Database StatsRTMT – Performance Charts

14BRKCCT-1593


Catch Problems Before Users do !RTMT – Alert Central

15BRKCCT-1593


Several Ways to View LogsRTMT – Log Central

16BRKCCT-1593


Intermittent Issue?

17BRKCCT-1593

RTMT – Scheduled Trace Collection

System Monitoring Tools –Command Line


• CLI: show status

• CLI: utils service list

• CLI: utils diagnose test

• CLI: show network cluster

• CLI: show myself

• CLI: show process load [cpu|memory] [cont|num XX]

• CLI: utils uccx ?

• CLI: show uccx ?

• CLI: set uccx ?

19BRKCCT-1593

Command Line Tools – Basic Health Check


Command Line Tools – Tailing Logs• Currently active logs can be tailed at real-time

• CLI: show open files regexp [MIVR|MADM|MCVD|Desktop-webservices]

• CLI: file tail activelog <relative path from /common/log/taos-log location>

• file tail activelog <relative path from /common/log/taos-log location>

CLI: file tail activelog /uccx/log/MIVR/Cisco001MIVR186.log regexp SS_TEL

System Monitoring Tools - CUIC


Utilization Monitoring – License and Call Handling

• CUIC provides stock reports for this

• License Utilization Hourly report

• Application Performance Analysis report

• Application Summary Report

• Aborted and Rejected Report

22BRKCCT-1593


License Utilization Hourly Grid and Chart Views


Call Failures – Aborted and Rejected Reports

• Cause Codes :

• Reject - TRIGGER_MAX_SESSION

• Reject - REMOTE_TIMEOUT

• Reject - CHANNELS BUSY

• Aborted – Too many transfer failures

• Aborted – Max Steps Executed (1000)

Aborted Contacts

• Exception occurred in the script workflow – ABORT!

Rejected Contacts

• System reached maximum capacity – REJECT!

Reports give helpful contact details and

exact timestamps!

UCCX Engine and CUCM Interaction


UCCX and CUCM Integration

27BRKCCT-1593

Communications Manager

ClusterFinesse Agent Desktops

PSTN

Router/GWUCCX

AD/LDAP Recording

SIP (5060)

SIP/H323/MGCP

(5060/1720/2427)

LDAP (389 | 3268)

SCCP | SIP (2000 | 5060)

CTI/JTAPI (2748)

AXL (443 | 8443)

CTI (12028)


UCCX Call flow – PSTN call flow

CallManager

ClusterSupervisor

Stations

Agent

Stations

PSTN

Router/GW

UCCX

12

2343

4 6

6

555

1. Inbound PSTN Call to VGW.

2. VoIP leg to CUCM.

3. CUCM routes call to RP registered by UCCX.

4. UCCX responds with REDIRECT request to

available CTI Port.

5. UCCX triggers application, instantiates and

executes script. Call is answered upon Accept

step executing.

6. UCCX provides scripted queuing treatment or

identifies an available resource and CONSULT

TRANSFER the call to the agents device.

Agent ICD Extension: 6000

CTI Route Point 8222

CTI Ports: 9988-9990


Setting up the Logging Levels

Under Engine UCCX Engine logs (MIVR): There are multiple logging facilities. Each facility is a sub-component.

• SS_CM

• SS_RM

• SS_RMCM

• ICD_CTI

• SS_TEL

• JTAPI plugin logs: JTAPI logging is not enabled by default.

Enable it upto Debugging level

JTAPI logs: Information about the interactions with CUCM

29BRKCCT-1593


There is an easier way!

Logging Profiles. Specific logging facilities for specific problem descriptions.

• CCX Serviceability > Trace > Profile

30BRKCCT-1593


UCCX Engine in Partial Service?

31BRKCCT-1593

Manager Manager? OR Subsystem Manager? Click to Expand !


UCCX Engine in Partial Service?

• If CM Telephony Subsystem is in Partial Service, check the Port Monitor Tool

32BRKCCT-1593

• If Application Manager, then check for an INVALID Application

Appadmin > Tools > Real Time Reporting > Applications

Analyze the UCCX Engine logs


STATE LOG DESCRIPTION

RECEIVED Call.received() Call was received at a configured CTI Route Point

ASSOCIATED Call.associated() An available CTI port was assigned.

ACCEPTED Call.accepted() Call was routed to available CTI port

REJECTED Call.rejected() Call failed to be routed to a port

ATTRIBUTED Call.attributed() An application task was invoked to handle this call

ANSWERED Call.answered() Call was answered by the application (Accept Step)

ABANDONED Call.abandoned() Call was dropped by the caller or CTI port during consult

DISCONNECTED Call.disconnected() Call was terminated by the application

TRANSFERRED Call.transferred() Call was successfully transferred by the application

ABORTED Call.aborted() Call was terminated prematurely

Call States - Reference

BRKCCT-1593 34


//Call is offered to the CTI RP. cgn=1001 and cn=8222

2872713: Mar 28 09:15:31.264 IST %MIVR-SS_TEL-7-UNK:Route Connection=[8222::1/(P1-abhi_1)

GCID=(1,7057)->ACTIVE]->OFFERED, reason=1, Event= (P1-abhi_1) 7057/1 CallCtlConnOfferedEv

8222::1 [#827] Cause:100 CallCtlCause:100 CiscoCause:0 FeatReason:12, cause=100,

metacode=129, isMaster=true

//CTI Port selected

2872720: Mar 28 09:15:31.266 IST %MIVR-SS_TEL-7-UNK:Route Connection: [8222::1/(P1-

abhi_1) GCID=(1,7057)->ACTIVE]->OFFERED, CTI Port selected:

TP[id=1,implId=9989,state=IN_USE]

//Call disconnected at the CTI RP since it’s REDIRECTED

2872734: Mar 28 09:15:31.299 IST %MIVR-SS_TEL-7-UNK:RP[num=8222], conn=[8222::1/(P1-

abhi_1) GCID=(1,7057)->ACTIVE]->DISCONNECTED, event=(P1-abhi_1) 7057/1

CallCtlConnDisconnectedEv 8222::1 [#842] Cause:100 CallCtlCause:210 CiscoCause:0

FeatReason:6, cause=CAUSE_NORMAL[100], meta=META_CALL_REMOVING_PARTY[131]

//The call for the trigger 8222 is accepted at CTI port 9989

2872739: Mar 28 09:15:31.329 IST %MIVR-SS_TEL-7-UNK:CallID: 13, MediaID: 7057/1 Accepting

call for CTI Route Point: 8222 on CTI Port: 9989, ciscoCause=0

BRKCCT-1593 35


//Call is accepted and the call is ringing at the CTI port

2872740: Mar 28 09:15:31.329 IST %MIVR-SS_TEL-7-UNK:Call.accepted()

JTAPICallContact[id=13,implId=7057/1,state=STATE_ACCEPTED_IDX,inbound=true,App

name=app_new,task=null,session=19000000014,seq

num=0,cn=8222,dn=8222,cgn=1001,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null

,ocn=8222,route=RP[num=8222],OrigProtocolCallRef=0000000000001B9101685E5300000000

,DestProtocolCallRef=null,TP=9989]

2872741: Mar 28 09:15:31.329 IST %MIVR-SS_TEL-7-UNK:CallID:13 MediaId:7057/1,

TerminalConnection to Terminal: renji_9989 is RINGING, [9989::1/(P1-abhi_1)

GCID=(1,7057)->ACTIVE]->ALERTING

//TaskID created: 37000000025

2872744: Mar 28 09:15:31.330 IST %MIVR-SS_TEL-7-UNK:CallID:13 MediaId:7057/1

Task:37000000025 associated with Task ID: 37000000025

//Agent selected and the state is changed to RESERVED

2872829: Mar 28 09:15:32.657 IST %MIVR-SS_CM-7-UNK:RmCm contact 16784273[7057/1]

(12) .addAssociatedResource(abhi)

2872830: Mar 28 09:15:32.657 IST %MIVR-SS_RM-7-UNK:Rsrc: abhi New State:RESERVED

Old State:AVAILABLE Reason code:0

BRKCCT-1593 36


//Consult transfer starts


Task:37000000025, transfer(6000, 12000, ACKNOWLEDGED)

//Consult call is offered at the agent's phone

2872869: Mar 28 09:15:32.690 IST %MIVR-SS_TEL-7-UNK:OrigCall=CallID:13

MediaId:7057/1 Task:37000000025, ConsultEvent= (P1-abhi_1) 7058/1

CallCtlConnOfferedEv 6000::4 [#881] Cause:100 CallCtlCause:100 CiscoCause:0

FeatReason:12

//Consult call ringing on agent's extension


Task:37000000025, ConsultCallObserver RINGING

//Agent answers the call


Task:37000000025, ConsultCallObserver TALKING - CallCtlConnEstablished,

[6000::1/(P1-abhi_1) GCID=(1,7058)->ACTIVE]->ESTABLISHED

BRKCCT-1593 37


Common call failure scenario-1

Partition and CSS issue: it is important to note that the original calling number should be

able to reach BOTH THE CTI ROUTE POINT AND THE CTI PORTS.

In the MIVR logs, the following error message is seen:

302849: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-ROUTE_FAILED:Route failed: All

Call

ids=JTAPICallContact[id=10,implId=1481/1,state=STATE_RECEIVED_IDX,inbound=true

,App name=sample_1,task=null,session=1000000011,seq

num=0,cn=7420,dn=7420,cgn=1009,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=n

ull,ocn=7420,route=RP[num=7420],TP=null,List of Active

Connections=[7420::1/(P1-sydney_cti_1) GCID=(1,1481)->ACTIVE]-

>OFFERED,Extension=7410,Exception=com.cisco.jtapi.InvalidPartyExceptionImpl:

Attempt to redirect to an unknown destination,Failure reason= call will be

rejected, CTIERR_REDIRECT_CALL_UNKNOWN_DESTINATION=0x8ccc0034::Attempt to

redirect to an unknown

destination,Contact.Reject.reason=TRIGGER_FAIL,(SelectRouteTime,ObtainingIdleC

hannelTime,RedirectTime=0,0,16)

38BRKCCT-1593


99066: Dec 01 09:22:54.039 IST %JTAPI-

PROTOCOL-7-UNK:(P1-10.106.112.196)

[MIVR_SS_TEL_TPG_ROUTE_EXE-44-24-

ROUTE_CALL_EV] sending:

com.cisco.cti.protocol.CallRedirectReq

uest {

sequenceNumber = 148

lineCallManagerID = 2

lineID = 30

callLegCallManagerID = 1

callLegID =

32362287

redirectAddress = 8000

unconditional = true

callingSearchSpace = 0

calledAddressOption = 0

redirectReason = 0

preferredOriginalCalledParty = 8000

modifiedCallingParty = null

99016: Dec 01 09:22:53.644 IST

%JTAPI-PROTOCOL-7-UNK:(P1-

10.106.112.196) received

Event:

com.cisco.cti.protocol.NewCall

Event {

eventSequence

= 395

lineCallManagerID

= 2

lineID

= 30

lineHandleSpecified

= true

deviceName

= ACD

callCallManagerID

= 1

callLegID

= 32362287

globalCallManagerID

= 1

callID

= 1464

callingParty

= 1009

calledParty

= 4000

originalCalledParty

= 4000

99067: Dec 01 09:22:54.211 IST %JTAPI-PROTOCOL-

7-UNK:(P1-10.106.112.196) received Event:

com.cisco.cti.protocol.NewCallEvent {

eventSequence = 397

lineCallManagerID = 2

lineID = 64

lineHandleSpecified = true

deviceName = CCG1_8000

callCallManagerID = 1

callLegID = 32362289

globalCallManagerID = 1

callID = 1464

callingParty = 1009

calledParty = 8000

originalCalledParty = 4000

state = 2

origin = 1

userData = null

auxiliaryData = 1

farEndpointCallManagerID = 1

farEndpointType = 1

reason = 6

callingPartyName =

calledPartyName =

BRKCCT-1593 39


The call from the CTI Route point to the CTI

Port is a REDIRECT, therefore the original

calling party should have access to both the

route points and ports.

BRKCCT-1593 40


Remember..

Avoid Unsupported

ActionsShared Lines

MoH

(Non-Multicast)

Steps in Script (Interruptible)

Select Resource

Timeout < RNA on IPCC Line Max Calls (2)

Busy Trigger (1)

2 Second Timeout before Accept (Script)

CUCM Providers Local, CTI

Ports, RPs in separate DPs

Troubleshooting Finesse


Cisco Finesse

Finesse is the only available

Desktop in UCCX 11.6(1) release

-Browser-based

-Co-resident with CCX

- Open REST APIs

- Customizable desktop

- Cisco and third party gadgets

- Integrated administration

- E.164 support

BRKCCT-1593 43


Finesse health

• Finesse depends on the following services for its normal functioning :

• Cisco Unified CCX Engine Service

• Cisco Unified CCX Notification Service

• Cisco Finesse Tomcat (Tomcat Service exclusive to Finesse)

44BRKCCT-1593


Checking Finesse health

Service status on the GUI shows ‘STARTED’ – does this mean Finesse is usable?

NO. Service state : STARTED = Finesse Tomcat status only

For actual status of Finesse:

https://<UCCX IP>:8445/finesse/api/SystemInfo

http://<UCCX IP>:8082/finesse/api/SystemInfo


Slowness? Check Finesse Health

• Finesse diagnostic portal:

• Tomcat Web Server stats

• CTI Connection stats

• Agent and call stats

https://host.domain.com:8445/finesse-dp/rest/DiagnosticPortal/GetPerformanceInformation


Finesse Issues – Further Analysis

• From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Finesse, Cisco Unified CCX Engine, Cisco Unified CCX Notification Service

47BRKCCT-1593

Finesse Logs

• Desktop-webservices, client-logs, realm log file for issues related to Finesse

Engine Logs

• MIVR logs,

• LIB_AXL: Axlclientlogs, for login issues

Notification Service Logs

• Needs to be explicitly enabled from CLI.

• For specific login issues, state change issues on Finesse.


Failure Scenario – Finesse Monitoring Error

Problem: Monitoring Fails to Start

48BRKCCT-1593

CCX Engine Receives Request to Silent Monitor agent106CCX Engine log:%MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg

Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:3233 Msg Type =

SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3233

AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0,

SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR,

AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice

= 2002, SupervisorDevice = 2001




49BRKCCT-1593

CCX Engine detects BiB Is Enabled, Uses JTAPI to Begin Monitoring Call SetupCCX Engine log:

%MIVR-SS_RM-3-Initiating silent monitor request to JTAPI server...:'Initiating silent monitor

request to JTAPI server...‘

%MIVR-SS_RM-7-UNK:processCallCtlConnInitiatedEv featureReason:1257684:

%MIVR-SS_RM-7-UNK:RIMgrAddressCallObserver: CallCtlConnInitiatedEv received for call 16779233

[2017/1] and agent agent1




50BRKCCT-1593

JTAPI Shows the Supervisor’s Monitoring Call State Go Dialtone > Dialing > Disconnected with Error:JTAPI log:

%JTAPI-CTI-7-UNK:(P2-

rmcm){Line:SEP88755651168C:2001:NewYork:(1,100)|Call:[GCID=(2017/1),CID=19750048]}

CallStateChanged (V2) [state=DISCONNECTED cause=UNALLOCATEDNUMBER destType=UNKNOWN destCM=0

fwdDest=: SecStat=1 unCg=2001 unCd= unOrigCd= unLRP= cg=2001 …….




51BRKCCT-1593

CCX Engine Reports this Failure back to Finesse ServerCCX Engine log:

%MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1

type=CONTROL_FAILURE_CONF,invokeId=3233,failureCode=CF_INVALID_DESTINATION,errorCode=0,

text=Error from Supervisor Monitor request. Description: ICDJtapiCallControlChannel (monitor);

nested exception is:

Troubleshooting High Availability


High Availability

• LAN: Both nodes share AXL, Unified CM Telephony and RMCM Provider list configuration

• WAN: Both nodes have unique AXL, Unified CM Telephony and RMCM Provider list configuration

53BRKCCT-1593

HA over LAN (HAoLAN)

Primary CCX

CM Publisher CM Subscriber

Secondary CCX

HA over WAN (HAoWAN)

Primary CCX

CM Publisher CM Subscriber

Secondary CCX

WAN


Failover Issues – Further Analysis

• From Real-Time Monitoring Tool (RTMT) collect logs for

• Cisco Unified CCX Cluster View Daemon (MCVD) logs

Engine Failure

Engine logs – MIVR logs

Database Failure

Online logs – CCX Database

Event Viewer logs

Application logs

System logs

Finesse logs


High Availability

• Primary lost connectivity

• Master Election initiated

• Master Election completed

• Failover completed

Scenario 1: Lost Connectivity

Secondary CCX Server

Engine

Database

Reporting

Finesse

Primary CCX Server

CVDM ?? S M?

!Engine

Database

Reporting

Finesse

CVD

Secondary CVD detects missing keep-alives from Primary CVD

%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:

CVD does not receive heartbeat from node for a long period: nodeId=1,dt=2049

.....

%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:


%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash:

state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5


state=Convergence State,nodeInfo=Node id=1 ip=192.168.12.5

Master Election initiated on Secondary CVD%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:

CONVERGENCE_STARTED, name=Cisco Unified CCX Engine

7631: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-

UNK:JavaService66:

Cisco Unified CCX Engine on node 1 change master from true to false

7632: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:Post Master

Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1

Secondary CVD elects Secondary as Master%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on

node 2 change master from false to true

%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,

name=Cisco Unified CCX Engine, node=2

%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:

CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine


High Availability

• WAN network failure

• Missing Heartbeats detected

• Master Election performed


Scenario 2: Partitioned Network – ISLAND MODE


Engine

Database

Reporting

Finesse

Primary CCX Server

Engine

Database

Reporting

Finesse

Primary CVD detects missing Heartbeats

and assumes Primary has failed%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:


%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node

crash:

state=Convergence State,nodeInfo=Node id=2 ip=192.168.13.5

Secondary CVD performs Master Election%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:


%MCVD-CLUSTER_MGR-7-UNK:JavaService66:

Cisco Unified CCX Engine on node 1 change master from true to false

%MCVD-CLUSTER_MGR-7-UNK:Post Master

Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1

Secondary CVD elects Secondary as Master%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on

node 2 change master from false to true

%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,


%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:


CVD CVDM M S M?

!

Secondary CVD detects missing Heartbeats

and assumes Primary has failed%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:



state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5


High Availability

• WAN network restored

• CVDs detect dual Masters

• Master Election performed


Recovery from Network partition


Engine

Database

Reporting

Finesse

Primary CCX Server

Engine

Database

Reporting

Finesse

Primary CVD detects Secondary and keeps Master%MCVD-CVD-7-UNK:Split after network partition is detected,new nodeId=2

%MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1

because primaryEngineComputerName=UC85CCXPRI

%MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine conditional-Keep-

LocalMaster

from localServiceCisco Unified CCX Engine on node 1

Secondary CVD detects Primary and drops Master%MCVD-CVD-7-UNK:Split after network partition is detected, new nodeId=1

%MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1

because primaryEngineComputerName=UC85CCXPRI

%MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine DropLocalMaster from

localServiceCisco Unified CCX Engine on node 2, conditional=true

Secondary CVD performs Master election and drops Master%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:


%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine

on

node 2 change master from true to false

%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_DROPPED,


Secondary CVD completes Master election%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:


CVD CVDM M M S?

!


Finesse Failover Behavior before 11.6

Scenario CCX HA Behavior Finesse Server

HA Behavior

Finesse Client

Behavior

Recovery

CCX Engine Failure Failover to HA node Failover to HA node Failover to HA node Finesse follows

Engine mastership

CCX Notification

Service Failure

No Failover Finesse goes Out of

Service

Sessions closed Finesse unavailable

until Notification

Service comes online

Finesse Tomcat

Failure

No Failover Finesse goes Out of

Service


until Tomcat Service

comes online

Finesse Service OOS No Failover Finesse goes Out of

Service


until issue fixed

Island Mode Both HA nodes

become Master

Both Finesse servers

In Service

Clients connect to

either

Clients reconnect to

Master node


Finesse Failover Enhancements (11.6)


Finesse Failover Enhancements (11.6)

Failure Scenario

Unified CCX HA

Behavior Cisco Finesse - Node 1 Cisco Finesse - Node 2

Unified CCX Engine fails over from

master to Non-Master node

Engine Mastership

changes from master

to non-master.

Return to IN_SERVICE as

soon as it connects to the

new master engine.

Return to IN_SERVICE as

soon as it connects to the

new master engine.

Finesse Service Out of Service

Engine mastership is

not affected OUT_OF_SERVICE

Finesse on Node 2 will

continue to be

in IN_SERVICE

CCX Notification Service Failure

Engine mastership is

not affected OUT_OF_SERVICE


continue to be IN_SERVICE

Island Mode

Both HA nodes

become Master


continue to

be IN_SERVICE and will be

connected to Engine on

Node 1

Finesse goes Out Of Service

and will return

to IN_SERVICE as soon as it

connects to the engine on

Node 2 which is also the

master

Troubleshooting Live Data


Live Data – How it worksLive Reports on Cisco Finesse


Key Flow Points

Engine• Publishes every 3

seconds into JMS

SocketIOConsumer

• Receives through JMS all the messages

• Creates Rooms.

SocketIODispatcher

• Dispatches the updates to subscribed Web Clients (Browsers)

CUIC Live Data

Streaming Source

(ONLINE)


Logging required

• Engine logs (MIVR) – ICD_RTDM, SS_RMCM, upto Xdebugging5

• SocketIO Service logs – DEBUG mode


UCCX Engine as Live Data Source

• Engine drafts JSON updates to JMS (ActiveMQ) every 3 seconds

558501: Jan 26 18:57:26.008 EST %MIVR-ICD_RTDM-7-UNK:LiveData:

Sending JMS message :

[{"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resourceI

d":"agent1","resourceName":"agent1","resourceState":2,"durationInSt

ateMillis":148093,"nHandledContacts":0,"nPresentedContacts":0,"avgT

alkDuration":0,"longestTalkDuration":0,"avgHoldDuration":0,"longest

HoldDurati……… }}]

65BRKCCT-1593


UCCX Engine as Live Data Source

• For Event based Rooms, Engine sends empty heartbeats

558509: Jan 26 18:57:26.274 EST %MIVR-SS_RMCM-7-UNK:HeartBeatAction : Sending

JMS message : [] for destinations [topic://ChatAgentStats,

topic://ChatQueueStatistics, topic://AgentCSQStats,

topic://AgentStateDetailStats]

558510: Jan 26 18:57:26.279 EST %MIVR-SS_RMCM-7-UNK:HeartBeatAction : Sending

JMS message : [] SUCCESS

66BRKCCT-1593


SocketIO debugs

• The Socket IO Server receives messages from the JMS bus for the Topic.

• Here agent=agent1, JMS topic=ResourceIAQStats.

• Room name=ResourceIAQStats*agent1, Status=RECEIVED,

0000001076: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:

%CCBU_Camel (camel-3) thread #3 - JmsConsumer[ResourceIAQStats]-6-MessageProducer:

%[message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":

{"resourceId":"agent1","resourceName":

"agent1","resourceState":3,.................

..

room_name=ResourceIAQStats*agent1][room_prefix=ResourceIAQStats][status=RECEIVED][topic=jm

s:topic:ResourceIAQStats]: Event Detail Trace

67BRKCCT-1593


SocketIO debugs

• Socket IO Server processes the messages and maps them to the appropriate rooms.

• Room name=ResourceIAQStats*agent1, Status=ENQUEUED

0000005789: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:

%CCBU_pool-19-thread-1-6-MessageDispatcher:

%[client_count=-

1][message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":

{"resourceId":"agent1","resourceName":"agent1",".....

..

room_name=ResourceIAQStats*agent1][socket_io_server_type=WS/WSS][status=ENQUEU

ED]: Event Detail Trace

68BRKCCT-1593


SocketIO debugs

• SocketIO message dispatcher sending the update to clients

• The Message Dispatcher receives the discrete room named messages and transmits them to all the clients that subscribe to the room.

• Operation = UPDATE, Status=SENT

• 0000005791: 10.106.87.133: Jan 26 2016 18:57:27.008 +0530:

%CCBU_pool-19-thread-1-6-MessageDispatcher:

%[client_count=2][message={"id":"agent1","operation":"UPDATE","ResourceIAQSt

ats":{"resourceId":"agent1","resourceName":"agent1",...................

.....

[room_name=ResourceIAQStats*agent1][socket_io_server_type=WSS][status=SENT]:

Event Detail Trace

69BRKCCT-1593


SocketIO debugs

• 4. On the Browser Side logs (F12) we see the following:

Message received on

socket::{"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resourceId"

:"agent1","resourceName":"agent1","resourceState":3,"durationInStateMillis":

264720,"nHandledContacts":6,"nPresentedContacts":8,"avgTalkDuration":1165394

,"longestTalkDuration":2266049,"avgHoldDuration":0,"longestHoldDuration":0,"

avgHandleDuration":0,"avgWorkDuration":0,"totalTalkTime":6992368,"totalHoldT

ime":0,"maxReadyTime":10199923,"avgReadyTime":1156099,

The above message can also be tallied with Finesse side Client logs to check browser updates coming in.

70BRKCCT-1593


Live Data – Checklist

• Synchronized

• Resolution Forward and Reverse

• Diagnostic test : utils diagnose test

NTP

DNS

• Resources, NIC = VMXNET3 adapter

• Sizing – Under Limit, Agents per Team!

OVA Profile

Sizing

• Lowercase

• Mismatch, Unsupported Algorithms

Hostname

Certificates


Remember.. Browser Versions

Browser Cache

(Incognito)

Supported Sizing Limits

Disable Compatibility

mode

Field Note Awareness

UCCX Database and Reporting


Configuration Datastore

Historical Datastore

Repository Datastore

CUIC

UCOSCM DB

RHEL

CCX Application

IBM Informix

Finesse

CCX DB

UCCX DB

Platform DB

UCCX Databases

BRKCCT-1593 74


Publisher Subscriber

UCCX Replication Types

BRKCCT-1593 75


utils dbreplication status – Provides the status of Platform DB replication.

CUIC and Finesse share the platform DB.

utils uccx dbreplication status – Provides the status of the UCCX DB replication.

All the CCX historical, configuration and repository data is stored on this DB.

76BRKCCT-1593

DB Replication States

CLI: utils network connectivity <hostname of secondary>

CLI: utils diagnose test

CLI: utils ntp status


CCX Database Replication

• View from CCX Admin or CLI

• Connected / Active state shows healthy replication

• Dropped / Timed Out status indicates replication issue

77BRKCCT-1593

Tools-> Datastore Control Center->

Replication Servers

CLI Command:

show uccx dbreplication servers

admin:show uccx dbreplication serversThis operation may take a few minutes to complete. Please wait...

SERVER ID STATE STATUS QUEUE CONNECTION CHANGED-----------------------------------------------------------------------g_uc115ccxpri_uccx 1 Active Local 0g_uc115ccxsec_uccx 2 Active Connected 0 May 9 18:07:55

Command successful.admin:


Command to show the DB Replication Files

CLI: utils uccx dbreplication dump configfiles

File Purpose

/usr/local/cm/db/informix/etc/sqlhosts.uccx Entries for Informix SQLHOSTS for the UCCX Instance

/etc/hosts Entries for the hosts in the cluster

/home/informix/.rhosts Entries for replicate hosts


CCX Serviceability - UI

Driven

Reset Replication

Enable/Disable

CDS/HDS

Synchronize Data

Command Line Interface

Driven

Reset Replication

forcedatasync

dbserver integrity

79BRKCCT-1593

Database Repair - Checklist

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKCCT-1593

Database Replication Reset

• Replication can be broken if Subscriber is unavailable for too long and send queues buffer is exceeded

• Typically 3-4 days (*can vary with load)

• Alert will be raised

Alert Raised!DBReplicationStopped

Tools-> Datastore Control Center->

Replication Servers

Issuing a Reset Replication causes the following to

occur:

1. Remove database replicationutils uccx dbreplication teardown

2. Setup database replicationutils uccx dbreplication setup

3. Perform data repair processutils uccx dbreplication repair all

80


Tools -> Datastore Control Center ->

Replication Servers

81BRKCCT-1593

Disable/Enable CDS/HDS


CDS

RDS

HDS

CDS

HDS

Node 1 Node 2


Replication Servers

RDS

82BRKCCT-1593



CDS

RDS

HDS

CDS

HDS

Disable CDS/HDS

1. Tear down HDS replication.

2. Deactivate CDS on node 2.

Node 1 Node 2


Replication Servers

RDS

83BRKCCT-1593



CDS

RDS

HDS

CDS

HDS

Enable CDS/HDS

1. Setup HDS replication.

2. Merge HDS data.

3. Copy node 1 CDS to node 2.

4. Activate node 2 CDS.

Node 1 Node 2


Replication Servers

RDS

84BRKCCT-1593



Common Scenario : Configuration IssueMaking Config Changes – Error occurred while saving

• Both DBs must be AVAILABLE!

• WORKAROUND?

• Temporarily Disable CDS on Subscriber

CCX Serviceability Admin -> Tools -> DatastoreControl Center -> Replication Servers

• Disable CDS and HDS

• Enable after connectivity to Subscriber is restored

• Enable CDS and HDS

CCX Administration (MADM) Logs:

%MADM-ADM_CFG-7-UNK:ICDServlet :: Exception occurred

The SUBSCRIBER node of the HA is not available

%MADM-ADM_CFG-3-ADM_EXCEPTION:Unknown ADM Exception:

Exception=java.lang.RuntimeException: The SUBSCRIBER node of the HA is not available

Only for when Subscriber is down.

If Publisher is down, no configuration

changes are possible.


CDS

RDS

HDS

CDS

RDS

HDS


Datastores

86BRKCCT-1593

Synchronize Data - GUI


CDS

RDS

HDS

CDS

RDS

HDS

•Merges a single datastore.

•CDS not eligible (no ER).


Datastores

87BRKCCT-1593

Synchronize Data - GUI


Local DB

CDS

RDS

HDS

CDS

RDS

HDS

Remote DB

88BRKCCT-1593

Force Database Sync – CLI Command


Local DB

CDS

RDS

HDS

CDS

RDS

HDS

Remote DB

Overwrite Target = Local!

89BRKCCT-1593



Local DB

CDS

RDS

HDS

CDS

RDS

HDS

Remote DB

Overwrite Target = Local!

Cluster Reboot + Replication Reset

required!

90BRKCCT-1593



Database Integrity

utils uccx database dbserver integrity

• Uses informix “oncheck” to check database for inconsistencies

• System Catalog Tables

• Chunk and extents

• Root reserved pages

• Output logged. CLI: file view activelog /uccx/cli/DbServerIntegrity_xxxx.txt

• Repairs NULL data entries in Config Tables

• Output logged. CLI: file view activelog /uccx/cli/DbConfigIntegrity_xxxx.txt


Reporting Review - Datasources

CUIC Data Sources:

Publisher CUIC uses

Subscriber’s db_cra

Subscriber’s CUIC

Also uses Subscriber’s

db_cra


Database Issues – Further Analysis

• From Real-Time Monitoring Tool (RTMT) collect logs for

• Cisco Unified CCX Database

93BRKCCT-1593

DB REPLICATION

uccx_repl_output_util.log

file for issues related to

database replication or at a

minimum gather these logs

for TAC

DB

online.uccx.log file for

issues related to database

service


Reporting Review – User Accounts

• uccxhruser - ”Historical Reporting User”

• CUIC, Wallboards, 3rd Party Reporting Software, Workforce Management

94BRKCCT-1593

uccxwallboard

Wallboard User limited to

RtCSQsSummary and

RtICDStatistics tables

uccxworkforce: Workforce User used by Calabrio WFO.

Large queries on historical tables.

uccxhruser : Historical Reporting User for

CUIC co-resident and custom report access to

all historical tables.


Scenario: Reporting and Database Performance

• Historical Reports slow to execute..

• Overall high server CPU utilization..

• Sluggishness in the system..

• Appadmin is slow..

95BRKCCT-1593

Looking into processes

You’ll find ‘uccxoninit’ as the top process!


Reporting and Database Performance

• Review the configured CUIC reports and their frequency

• Check the historical tables size

96BRKCCT-1593



• If tables have millions of rows, review purge config:

97BRKCCT-1593

Default of 90 months can be reduced

Individual table purge is new in version 11


• DBPerf Script:

• DBPerf log contains:

• Timestamps,

• Full CPU snapshot: top,

• Users: onstat –u,

• Top Session information: onstat –g ses 0,

• Memory: onstat –g mem,

• Disk: onstat –d,

• Locks: onstat- k,


98BRKCCT-1593

2 parameters - total duration and interval at which to get data

Troubleshooting Chat and Email


Internet DMZ CorporateMS Exchange

UCCX

Finesse Client

End User

SocialMiner

UCM

Voice Gateway

Chat and Email Topology

BRKCCT-1593 100


Exchange-Finesse Email Architecture with Social Miner – note that the exchange can be in the DMZ too.

Cloud Mail

Customer chat

Social Miner

UCCX

ExchangeDMZIntranet

Finesse Client

1

2

3

4

5

From Customer to Agent

BRKCCT-1593 101


Shared BOSH Connection

Interacts w/ Private REST APIs

HTTP/REST Traffic Proxied via Finesse Tomcat

Reply Template JSP loaded via SocialMiner Tomcat Chat XMPP Service

Interacts w/ Private REST APIs

HTTP/REST Traffic Proxied via Finesse Tomcat

Reply Template JSP HTTP/REST API Traffic

Hosted in SocialMiner

Hosted in UCCX

Server to Server

Communication

Customer Website

Exchange Server

XMPP

(BOSH)XMPPHTTP/S

(Agent)

HTTPS IMAPO

pe

nA

jax

Hub

Gadget to Gadget Communication

Gadget to Gadget Communication via HTML5

UC

CX

Socia

lMin

er

Cisco Tomcat

& Other

Services

SocialMiner

XMPP Service

Cisco Tomcat

& Other

Services

Cisco Unified

CCX

Notification

Service

Cis

co F

inesse T

om

cat

102


Office 365 and Gmail Support

• Only Socks 5 protocol is supported

• Configure Socks Proxy in SM and enable it.

• There is a Office365 email feed status in SocialMiner:

If the feed is good, it shows green

Else, a red X is seen

Hovering over the X will show the

reason for the state

BRKCCT-1593 103

Object Missing

Object Mismatch

Object Normal

Not Configured: Email Server not configured

Not Applicable: Cisco Finesse not active

SocialMiner incompatible

Email CSQ’s not created


SocialMinerMS Exchange UCCX Finesse Client

EmailFetch Emails

Notify New ContactNotify New Contact

Load Email UI

Get Email Body and Attachments

Fetch Complete EmailLoad Complete Email

Reserve Contact

Upload/Download Attachments / ReplyUpload/Download

Attachments / ReplyEmail Reply Contact Handled

Contact Handled

104BRKCCT-1593

Finesse Agent Email Message Flow


Email & Chat Fault Tolerance

• SocialMiner does not support High Availability (HA)

• Fault Tolerance is provided via UCCX HA

• SocialMiner configured to communicate with both UCCX nodes

• When new contact arrives, both CCX nodes notified

• Master node queues the request

• Secondary node ignores request

• If UCCX nodes failover

• Chat: existing sessions terminated, new contacts queued to HA node

• Email: on desk emails requeued, new contacts queued to HA node

SocialMiner

UCCX

M

UCCX

S

BRKCCT-1593 105


Reinject Email Contacts

UCCX SocialMiner Configuration Page

Email auto routed to Agent

When Agent goes to retrieve email,

network connection between SocialMiner and

Exchange broke

Email contact becomes parked in SocialMiner

Reinject Emails will send the emails back to

the Agent once error corrected

BRKCCT-1593 106


Finesse Agent Email – Failure Scenario 1

• Denotes email deleted on Exchange Server

• Verify that there are no auto-routing rules that moved the email

• Verify CSQ alias folder has not been deleted or moved

BRKCCT-1593 107


Finesse Agent Email – Failure Scenario 2

• Denotes a connectivity problem with SocialMiner Tomcat Service

• Verify SocialMiner is reachable

• Verify status of Tomcat Service

• Restart Tomcat Service

BRKCCT-1593 108


Finesse Web Chat Message Flow

Web ServerSocialMiner

UCCXFinesse Client

Chat RequestNew Contact Notification

Contact Dialog Event

Open Reply TemplateAgent Joined

Reply Template

End SessionSession Complete

Session CompleteContact Dialog Event


Failure Scenario – Non-Voice Gadget

• Cisco Unified CCX Webservices or Non-Voice Subsystem offline

• Agent not configured but gadget displayed

• Controls not enabled – SocialMiner unreachable

BRKCCT-1593 110


SocialMiner with UCCX - Checklist

Network

• NTP Synchronized.

• DNS - Forward and Reverse resolved

• Network Latency < 250 ms

• utils network host <IP/Hostname>

VM Settings

• ESXi – LRODisabled

• OVA Profile

• NIC (VMXNET3)

Certificates

• Hostnames in Lowercase

• SSL CCX and SocialMiner trust

relationship (tomcat-trust)

Firewall, Proxy

• Open ALL required ports

• Refer Port Utilization Guide


SocialMiner Logs

• The Main SocialMiner Engine (Runtime)

• RTMT – SocialMiner Runtime

• CCBU-runtime.*.startup.log, Error-runtime.*.startup.log

Runtime Logs

• Main API logs for the SocialMiner REST API

• RTMT – CCP API Logs

• CCBU-ccpapi.*.startup.log, Error-ccpapi.*.startup.log

API Logs

• Public interfacing API (Incoming Chat, Facebook, Twitter poll)

• RTMT – CCP PublicApps Logs

• CCBU-ccppublicapps.*.startup.log, Error-ccppublicapps.*.startup.log

Public App Logs


What’s most important is ..

Questions ?

Evidence …

Answers !!


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKCCT-1593


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs : LABCCT-2011

• Tech Circle

• Meet the Engineer 1:1 meetings

• Related sessions

116BRKCCT-1593


Call to action !

• UCCX solution health check (Once a month)

• UCCX EFTs (Early Field Trials) – Spark!

• UCCX Field Notices on Cisco.com

https://www.cisco.com/cisco/support/notifications.html

• Build your own checklists!

117BRKCCT-1593

https://www.cisco.com/cisco/support/notifications.html

Thank you

Troubleshooting in the UCCX Solution · Troubleshooting in the UCCX Solution Arunabh Bhattacharjee,...

Documents

Transcript of Troubleshooting in the UCCX Solution · Troubleshooting in the UCCX Solution Arunabh Bhattacharjee,...