Troubleshooting CPS vDRA - cisco.com€¦ · Troubleshooting CPS vDRA • Overview,page1 •...

Troubleshooting CPS vDRA

• Overview, page 1

• General Troubleshooting, page 1

• Diameter Troubleshooting and Connections, page 1

• Troubleshooting Basics, page 3

• Common Troubleshooting Steps, page 6

• Frequently Encountered Troubles in CPS vDRA, page 7

OverviewCPS vDRA is a functional element that ensures that all Diameter sessions established over Gx, Rx interfacesand for unsolicited application reporting, the Sd interface for a certain IP-CAN session reach the same PCRFor destined PCRF when multiple and separately addressable PCRFs have been deployed in a Diameter realm.

General TroubleshootingRun /var/qps/bin/diag/diagnostics.sh from any VM except sessionmgr. There should not be any failedoutput for this script.

Run /var/qps/bin/control/statusall.sh. All the processes should be running and monitored.

Diameter Troubleshooting and ConnectionsFor messages belonging to particular interface, CPS vDRA should be ready to make diameter connection onthe configured application port. As CPS vDRA acts as a server, it should be listening on ports for differentapplications to accept any incoming diameter requests for the application.

If you are facing problems making diameter connections, check for the following configuration:

CPS vDRA Troubleshooting Guide, Release 14.0.0 (1) 1

DRA Plug-in Configuration in DRA Policy Builder (PB)

Figure 1: DRA Endpoints

Step 1 Check status of application base port on active policy director (lb). It should be listening to diameter connections externallyon VIP and internally to Policy Servers (QNS).[root@lb01 ~]# netstat -na | grep 3868tcp 0 0 10.77.207.100:3868 0.0.0.0:* LISTENtcp 0 0 ::ffff:80.80.80.10:3868 :::* LISTEN

Step 2 Check haproxy-diameter.cfg file for proper entries:For Step 1, on page 2 and Step 2, on page 2 configuration, the entries should be as follows:

[root@lb01 ~]# cat /etc/haproxy/haproxy-diameter.cfgglobaldaemonnbproc 1 # number of processing coresstats socket /tmp/haproxy-diameter

defaultstimeout client 60000ms # maximum inactivity time on the client sidetimeout server 180000ms # maximum inactivity time on the server sidetimeout connect 5000ms # maximum time to wait for a connection attempt to a server to

succeedlog 127.0.0.1 local1 errlisten diameter-int1

bind 10.77.207.100:3868mode tcpoption tcpkabalance leastconnserver lb01-A lb01:3868 checkserver lb01-B lb01:3869 checkserver lb01-C lb01:3870 check

listen diameter-int2bind 10.77.207.100:4868mode tcpoption tcpkabalance leastconnserver lb01-A lb01:4868 checkserver lb01-B lb01:4869 checkserver lb01-C lb01:4870 check

listen stats_proxy_diameter lbvip01:5540mode httpoption httpclose

CPS vDRA Troubleshooting Guide, Release 14.0.0 (1)2

Troubleshooting CPS vDRADRA Plug-in Configuration in DRA Policy Builder (PB)

option abortonclose# enable web-statsstats enablestats uri /haproxy-diam?stats#stats auth haproxy:cisco123stats refresh 60sstats hide-version

Step 3 Listen for diameter traffic by logging into lb01 and lb02 and execute the following command:tcpdump -i any port 3868 -s 0 -vv

Troubleshooting BasicsTroubleshooting CPS vDRA consists of these types of basic tasks:

• Gathering Information

• Collecting Logs

• Running Traces

Diameter Error Codes and Scenarios

Table 1: Diameter Error Codes and Scenarios

DescriptionResult-Code ValueResult-Code

Informational

Subsequentmessages triggered by client shallalso used in Authentication and to get accessof required resources. Generally used inDiameter NAS.

1001DIAMETER_MULTI_ROUND_AUTH

Success

Request processed Successfully.2001DIAMETER_SUCCESS

Request is processed but some moreprocessing is required by Server to provideaccess to user.

2002DIAMETER_LIMITED_SUCCESS

Protocol Errors [E-bit set]

Server returns it if Diameter Command-Codeis un-recognized by server.

3001DIAMETER_COMMAND

_UNSUPPORTED


Troubleshooting CPS vDRATroubleshooting Basics


Message cannot be delivered because thereis no Host with Diameter URI present inDestination-Host AVP in associated Realm.

3002DIAMETER_UNABLE

_TO_DELIVER

Intended Realm is not recognized.3003DIAMETER_REALM_NOT

_SERVED

Shall return by server only when serverunable to provide requested service, whereall the pre-requisites are also met. Clientshould also send the request to alternate peer.

3004DIAMETER_TOO_BUSY

-3005DIAMETER_LOOP_DETECTED

In Response from Redirect Agent.3006DIAMETER_REDIRECT

_INDICATION

-3007DIAMETER_APPLICATION

_UNSUPPORTED

It is sent when a request is received withinvalid bits combination for consideredcommand-code in DIAMETER Headerstructure. For example, Marking Proxy-Bitin CER message.

3008DIAMETER_INVALID_HDR_BITS

It is sent when a request is received withinvalid flag bits in an AVP.

3009DIAMETER_INVALID_AVP_BITS

A DIAMETER server can be configuredwhether it shall accept DIAMETERconnection from all nodes or only fromspecific nodes. If it is configured to acceptconnection from specific nodes and receivesCER frommessage from any node other thanspecified.

3010DIAMETER_UNKNOWN_PEER

Transient Failures [Could not satisfy request at this moment]

Returned by Server, most likely because ofinvalid password.

4001DIAMETER_AUTHENTICATION

_REJECTED

Returned by node, when it receivesaccounting information but unable to storeit because of lack of memory.

4002DIAMETER_OUT_OF_SPACE


Troubleshooting CPS vDRADiameter Error Codes and Scenarios


Peer determines that it has lost election bycomparing Origin-Host value received inCER with its own DIAMETER IDENTITYand found that received DIAMETERIDENTITY is higher.

4003ELECTION_LOST

Permanent Failures [To inform peer, request is failed, should not be attempted again]

AVP marked with Mandatory Bit, but peerdoes not support it.

5001DIAMETER_AVP

_UNSUPPORTED

-5002DIAMETER_UNKNOWN

_SESSION_ID

User can not be authorized. For example,Comes in AIA on s6a interface.

5003DIAMETER_AUTHORIZATION

_REJECTED

-5004DIAMETER_INVALID_AVP_VALUE

Mandatory AVP in request message ismissing.

5005DIAMETER_MISSING_AVP

A request was received that cannot beauthorized because the user has alreadyexpended allowed resources. An example ofthis error condition is a user that is restrictedto one dial-up PPP port, attempts to establisha second PPP connection.

5006DIAMETER_RESOURCES

_EXCEEDED

Server has identified that AVPs are presentthat are contradictory to each other.

5007DIAMETER_CONTRADICTING

_AVPS

Message is received by node (Server) thatcontain AVP must not be present.

5008DIAMETER_AVP_NOT_ALLOWED

If message contains the a AVP number oftimes that exceeds permitted occurrence ofAVP in message definition.

5009DIAMETER_AVP_OCCURS

_TOO_MANY_TIMES

In response of CER if no commonapplication supported between the peers.

5010DIAMETER_NO_COMMON

_APPLICATION

Self explanatory.5011DIAMETER_UNSUPPORTED

_VERSION


Troubleshooting CPS vDRADiameter Error Codes and Scenarios


Message rejected because of unspecifiedreasons.

5012DIAMETER_UNABLE

_TO_COMPLY

When an unrecognized bit in the Diameterheader is set to one.

5013DIAMETER_INVALID_BIT

_IN_HEADER

Self explanatory.5014DIAMETER_INVALID

_AVP_LENGTH

Self explanatory.5015DIAMETER_INVALID

_MESSAGE_LENGTH

For example, marking AVP to Mandatorywhile message definition doesn't say so.

5016DIAMETER_INVALID_AVP

_BIT_COMBO

In response of CER if no common securitymechanism supported between the peers.

5017DIAMETER_NO_COMMON

_SECURITY

Common Troubleshooting Steps

Using TCPDUMP

Collect tcpdump packet capture from the primary dra_endpoint:tcpdump -i any -port 3868 -s0 -w filename test.pcap

In the collected trace file:

• Verify that the response message is sent back to CPS vDRA.

• Use Session-Id as filter if the Session-Id of the user's session is available.

• If Session-Id for the user is not available, use MSISDN as filter to retrieve the Session-Id. Then apply Session-Idfilter to view all the messages for the session.

• Match the request to response for Credit Control Request, CC-Request-Type attribute (Initial/Update/Terminate).


Troubleshooting CPS vDRACommon Troubleshooting Steps

CPS vDRA Logs

Step 1 Check the consolidated-qns.log file on pcrfclient01 for any exceptions with policy executions. For example,for null pointer exception.

Step 2 Filter using Session-Id.Step 3 Check the logs where the call is failing.

Counters and StatisticsCheck for statistics generated at pcrfclient01/02 in /var/broadhop/stats and counters in beans at jmxterminal.

Frequently Encountered Troubles in CPS vDRA

Redis Not Working

Step 1 Check redis status by executing the following command:[root@lb01 ~]# service redis statusredis-server (pid 22511) is running...

Step 2 Try starting redis process by executing the following command:[root@lb01 ~]# service redis start

Step 3 Check the following entries in /etc/broadhop/draTopology.ini file at policy directors (lb) and Policy Servers (QNS) forredis connecting on ports 6379, 6380, 6381, 6382:[root@lb02 ~]# cat /etc/broadhop/draTopology.inidra.redis.qserver.1=lb02:6379dra.redis.qserver.2=lb02:6380dra.redis.qserver.3=lb02:6381dra.redis.qserver.4=lb02:6382dra.redis.qserver.4=lb02:6383dra.local-control-plane.redis.1=lb02:6379dra.mongodb.binding.db.ipv6.uri=mongodb://sessionmgr01:27718dra.mongodb.binding.db.ipv4.uri=mongodb://sessionmgr01:27718dra.mongodb.binding.db.imsiapn.uri=mongodb://sessionmgr01:27718dra.mongodb.pcap.uri=mongodb://sessionmgr01:27718dra.mongodb.binding.db.session.uri=mongodb://sessionmgr01:27718[root@lb02 ~]# cat /etc/broadhop/redisTopology.inidra.redis.qserver.1=lb02:6379dra.redis.qserver.2=lb02:6380dra.redis.qserver.3=lb02:6381


Troubleshooting CPS vDRACPS vDRA Logs

dra.redis.qserver.4=lb02:6382dra.local-control-plane.redis.1=lb02:6379

Step 4 Redis process on active policy director (lb) should be established with all Policy Servers (QNS) as shown below:[root@lb01 ~]# netstat -na | grep 6379tcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTENtcp 0 0 80.80.80.10:6379 80.80.80.10:37400 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38020 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38034 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37390 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38207 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.16:50597 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.14:35703 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.14:35711 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38188 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37375 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38174 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38229 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38211 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.14:35709 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.16:50590 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38032 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38172 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.16:50605 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38204 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38213 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38223 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38044 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38187 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38205 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38211 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37672 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.14:35710 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.17:59833 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37388 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37389 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37662 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.17:59824 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.16:50596 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38210 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.15:49162 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38231 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38230 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.15:49159 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38152 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37659 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38208 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.17:59832 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.15:49161 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38206 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38212 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:38033 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.10:37650 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.15:49160 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38155 ESTABLISHED


Troubleshooting CPS vDRARedis Not Working

tcp 0 0 80.80.80.10:6379 80.80.80.10:37660 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.17:59831 ESTABLISHEDtcp 0 0 80.80.80.10:6379 80.80.80.11:38186 ESTABLISHEDtcp 0 0 :::6379 :::* LISTENtcp 0 0 ::ffff:80.80.80.10:37660 ::ffff:80.80.80.10:6379 ESTABLISHED

Gx Bindings not happening on Mongo

Step 1 Check if the binding's exceptions are coming in consolidated-qns.log file.Step 2 Check for the entry -DdraBindingTier=true in qns.conf file on all Policy Servers (QNS).Step 3 Check for the entries in /etc/broadhop/draTopology.ini file.

dra.redis.qserver.1=lb02:6379dra.redis.qserver.2=lb02:6380dra.redis.qserver.3=lb02:6381dra.redis.qserver.4=lb02:6382dra.redis.qserver.4=lb02:6383dra.local-control-plane.redis.1=lb02:6379dra.mongodb.binding.db.ipv6.uri=mongodb://sessionmgr01:27718dra.mongodb.binding.db.ipv4.uri=mongodb://sessionmgr01:27718dra.mongodb.binding.db.imsiapn.uri=mongodb://sessionmgr01:27718dra.mongodb.pcap.uri=mongodb://sessionmgr01:27718dra.mongodb.binding.db.session.uri=mongodb://sessionmgr01:27718

For example, make sure if the primary binding server is 27718 only as per above example.

Step 4 Check for the Binding Keys entries in binding key type profile and the application attached to the profile.

Rx Call Failing at CPS vDRA

Step 1 Check for the Binding key Retriever for Rx Profile.Step 2 Check if the Gx Binding is available for that Binding key.Step 3 Check the consolidated-qns.log file if CPS vDRA is able to retrieve SRK from the bindings.Step 4 Check for any exception in consolidated-qns.log file during binding retrieval.Step 5 If Rx peer is available for the same SRK at CPS vDRA, CPS vDRA should forward the Rx message to that peer.Step 6 Check the connection for that peer and proper entries in Peer Group, Peer Routing, Peer Group Peer and Rx_Routing

for Rx New session rules.


Troubleshooting CPS vDRAGx Bindings not happening on Mongo

CPS vDRA Forwarding Message to Wrong Peer

Step 1 Check the Control Center configuration in Gx_Routing for new session rules. Gx routing should have the AVP definedon the basis of which, one wants to route the traffic.

Step 2 Check whether the Control Center configuration for the Peer is bonded to correct Peer Group.Step 3 Check whether the Peer Group is assigned to correct Peer Route and Dynamic AVPs are properly aligned with Peer

Route in Gx New Session Rules.Step 4 Diameter Connection with the desired Destination Peer should be established with CPS vDRA.

PCRF Generated Messages not Reaching CPS vDRA

Step 1 Make sure PCRF has the correct entry of CPS vDRA as next hop.

Figure 2: Next Hop Routes

Next Hop definition is mandatory in PCRF to forward the messages to CPS vDRA generated by PCRF itself.

For example, Gx-RAR, Sd-TSR

Step 2 Wild Card Entry not supported in Next Hop Routing configuration.

Issues in Reaching Ports and Setup IPs

Step 1 Check firewall is running or not.Step 2 Make sure the firewall configuration is OK.

a) To check if this is the problem, then stop the firewall./etc/init.d/iptables stop


Troubleshooting CPS vDRACPS vDRA Forwarding Message to Wrong Peer


Troubleshooting CPS vDRAIssues in Reaching Ports and Setup IPs

Troubleshooting CPS vDRA - cisco.com€¦ · Troubleshooting CPS vDRA • Overview,page1 •...

Documents

Transcript of Troubleshooting CPS vDRA - cisco.com€¦ · Troubleshooting CPS vDRA • Overview,page1 •...