AR3

New AR: View AR | AR without Proprietary

History | States | Assignment | Timetracking

Add Attachment

AR Text Search | More | CARES HomeAssistance Request 1-5878714

Customer Ticket N/A

Short Repeated errors on RNCWCE241&RNCWCE242

Description

Current 27th Aug: EP5 need to be installed, feedback awaited from field

Summary team.

State Pending : Awaiting Customer Review of Proposed Corrective Action

Outage No

Severity 2

Priority 1

Service Request Remote Technical Support

Request Type Defect

Sub-Type Software

Category Software

Internal No

Assignment Events

Ask Alcatel-Lucent

Product

Product 9353 WMS

Version LR14.2.W

No Change Request is linked to this AR

Product Location

Site ETISALAT ABU DHABI

Instance 9353WMS-Etisalat : In Service

Site Company Emirates Telecommunications Corporation

City Abu Dhabi

Country United Arab Emirates

Contact

Name Karim IBRAHIM ABD EL NABY

Company Alcatel-Lucent : Open on behalf of Customer

Phone 971 56 354 4568 Email [email protected]

Request Method Email-Eng

Dates

Occurred 17-Aug-2015 14:47 Time now 29-Aug-2015 09:20 (GMT+4)

Reported 17-Aug-2015 14:47 AR Created 17-Aug-2015 14:51

Service Start 17-Aug-2015 14:47

Next Customer Contact 03-Sep-2015 21:30

Responded 17-Aug-2015 14:49 Respond Target 17-Aug-2015 15:47 SA - Calculated

Restored 17-Aug-2015 15:49 Restore Target 21-Aug-2015 19:29 SA - Calculated

Resolved -- --- ---- Resolve Target 25-Oct-2015 14:26 SA - Calculated

Targets for Restore & Resolve have been extended by 4 hours 42 minutes

to account for Pending time

Closed -- --- ----

Last Modified 27-Aug-2015 13:57 Modified By achowdar

Entitlement

Agreement 242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script This request is entitled to service.

People

Owner TSCd-WLS-WCDMA-IN : achowdar

Assignee AsgQ-WLS-WCDMA-vGlobal : unassigned

Referred 1 NorP-WLS-WCDMA-vGlobal : achowdar

Resolve Group NorP-WLS-WCDMA-vGlobal

Submitter mszyszko

Description

Attachments

From: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

<[email protected]>

Sent: 2015-08-17 12:47:09

To: ALU Support <[email protected]>

Cc: "MERAD, ABDESSAMAD (ABDESSAMAD)"

<[email protected]>;"ABDEL-HALIM, SAYED (SAYED)"

<[email protected]>;"EL-MIDANY, AHMED (AHMED)"

<[email protected]>;"WAGEEH,

WAEL (WAEL)** CTR **" <[email protected]>;"SAKR, HESHAM

(HESHAM)** CTR **" <[email protected]>;"WAGDY MANSOUR, AYMAN

(AYMAN)** CTR **" <[email protected]>;"KHEDR, MAHMOUD

(MAHMOUD)" <[email protected]>

Subject: Etisalat UAE//Repeated errors on RNCWCE241&RNCWCE242

Hello,

PROBLEM DESCRIPTION: Repeated errors on RNCWCE241&RNCWCE242

PROBLEM IMPACT:

We are observing the below type of alarms on RNC241, 242 from FMK_PM file

from WMS,

During the investigation on the AR-1-5873971 we observed this alarms and

requested from support team to trigger it with a separate AR,

We need to know when this alarms starts and what?s its impact and RC, and

soln to clear it,

2015/08/13-14:35:16,000+04:00 |

CP_APM_UTRAN_xdrCNodeMediationContext.cc:965 | Thr:33 | ERR |

component-name | class-name | CId:59593978 <CId:59593978>

Wrong instance of class:R/EM/IpIf/BwPool

observation file

name:/opt/nortel/data/utran/gpo/IN/RNCWCE242/20150813103000.gz

ne Unique Id:RNCWCE242

2015/08/13-14:35:16,169+04:00 | CP_APM_UTRAN_xdrCNodeDNHandler.cc:723 |

Thr:33 | ERR | component-name | class-name | CId:59593978 <CId:59593978>

Can NOT get the userLabel for : [RNC/RNCWCE242 RNCEquipment/0

INode/0]

2015/08/13-14:35:16,169+04:00 | CP_APM_UTRAN_xdrDNHandler.cc:284 | Thr:33 |

ERR | component-name | class-name | CId:59593978 <CId:59593978>

Failed to convert RAN DN: RNC/1 RNCEquipment/0 INode/0 EM/0 IpIf/63

BwPool/1 to Operator DN.

EXPECTED RESULTS:

RCA and Soln for this issue,

Contact Surname : Ayman

Contact Given Name : Wagdy

Contact Phone :+971 506722947

Contact Company : Alcatel-Lucent

Company : Emirates Telecommunications Corporation

Contract Number : 242895

Region : EMEA

Country : United Arab Emirates

City : Abu Dhabi

Site : ETISALAT ABU DHABI

Product Instance : 9771 WCE ,WCDMA

Severity : 2

Priority : 1

Product Version : LR14.2.8

Outage : No

Internal/External: :Internal

Thanks,

BR,

Karim Ibrahim

LTE/3G Integration Professional

Alcatel-Lucent, Dubai, Etisalat Program

M: +971(0)563544568| ONNET: 22341798

Every Success has its network

<http://www.alcatel-lucent.com/videos/video-every-success-has-its-network>

Attachments

Showing Investigation and Proprietary logs together in time order Show separately

Investigation Log and Proprietary Log

1. 17-Aug-2015 15:30 sudhakarbabu

Update to Current Summary: July 17: Awaiting Receive Log Files / Technical Info

2. 17-Aug-2015 15:56 ALCATEL-LUCENT PROPRIETARY sudhakarbabu

Last error observed was as below

2015/08/14-19:05:16,959+04:00 | CP_APM_UTRAN_xdrCNodeMediationContext.cc:965 |

Thr:35 | ERR | component-name | class-name | CId:60022649

Wrong instance of class:R/EM/IpIf/BwPool

observation file name:/opt/nortel/data/utran/gpo/IN/RNCWCE241/20150814150000.gz

ne Unique Id:RNCWCE241

Sent: 17 August 2015 17:42

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Cc: MERAD, ABDESSAMAD (ABDESSAMAD); ABDEL-HALIM, SAYED (SAYED); EL-MIDANY, AHMED

(AHMED); WAGEEH, WAEL (WAEL)** CTR **; SAKR, HESHAM (HESHAM)** CTR **; WAGDY

MANSOUR, AYMAN (AYMAN)** CTR **; KHEDR, MAHMOUD (MAHMOUD); INDIA-WCDMA-UTRAN-BE

Subject: RE: AR1-5843064 // Major // UAE- ETISALAT-ABU DHABI // 9771 WCE RNC //

LR14.2.8 // "Can NOT get the userLabel " Repeated errors on

RNCWCE241&RNCWCE242 FMK_PM file from WMS

Hi Karim,

Please provide the below logs:

1. From when this alarm is been observed.

2. RNC242 fulllog collection

3. RNC242 healthlog

4. RNC242 and RNC241 HFB report for last 15 days

5. Below logs from WMS:

WOM_COMM_SEPE_RNC-20010.log|tra*

WOM_MED_SEPE_FM-20008.log|tra*

cp_utran-20000.log|tra*

wom_pm_sepe_neProxy-20016.log|tra*

FMK_PM_server-20023.log|tra*

WTK_FTS-20017.log|tra*

Please upload the logs on china server 135.240.25.160 wftp/wftp, on the path:

/home/wcdma/traces/ETISALAT-UAE/AR1-5878714

Thanks,

Sudhakar,

Mobile Phone: +91 9342582713; Email Id: [email protected]


Update to Current Summary: July 17: Escalated to backend team, waiting for their

feedback



Sent: 17 August 2015 21:42

To: ROY, Paul (Paul)** CTR **; DAS, Sunil (Sunil)** CTR **

Cc: AGRAWAL, Ritesh (Ritesh)** CTR **; ARORA, Gaurav (Gaurav)** CTR **;

INDIA-WCDMA-UTRAN-BE




Dear Paul/Sunil,

We have escalated AR1-5878714 work group TEC-WLS-WCDMA-RNC. Please help to assign

this AR.

Please find the attached Escalation note.

Thanks,

Sudhakar,



Time Tracking Entry Added - IR01-TSA3


Escalation Note for AR1-5878714:

Product: 9971 WCE RNC

SW Version: LR14.2.8

Customer: Etisalat Abu Dubai, United Arab Emirates

Problem Short Description:

"Can NOT get the userLabel "Repeated errors on RNCWCE241&RNCWCE242 FMK_PM file

from WMS

Detailed Description:

We are observing the below type of alarms on RNC241, 242 from FMK_PM file from

WMS, We need to know when this alarms starts and what?s its impact and RC, and

solution to clear it.

CP_APM_UTRAN_xdrCNodeMediationContext.cc:965 | Thr:33 | ERR | component-name |

class-name | CId:59593978 <CId:59593978> Wrong instance of

class:R/EM/IpIf/BwPool observation file


ne Unique Id:RNCWCE2422015/08/13-14:35:16,169+04:00 |

CP_APM_UTRAN_xdrCNodeDNHandler.cc:723 | Thr:33 | ERR | component-name | class-name

| CId:59593978 <CId:59593978> Can NOT get the userLabel for :

[RNC/RNCWCE242 RNCEquipment/0 INode/0]


component-name | class-name | CId:59593978 CId:59593978


Operator DN.

1. All the steps taken before AR escalated to ALU

N/A

2. All the steps taken before the issue observed by the engineer, who reported the

AR

N/A

3. Activities After AR was escalated to L2 Technical Support:

From WMS FMK_PM_server-20023.tra log files we could observe the below errors:

We know there is no Cnode element available in WCE RNC. So there is no RNC IN

conter available and the below errors are appearing for WCE RNCs.

CP_APM_UTRAN_xdrCNodeMediationContext.cc:965 | Thr:33 | ERR | component-name |

class-name | CId:59593978 <CId:59593978> Wrong instance of

class:R/EM/IpIf/BwPool observation file


ne Unique Id:RNCWCE2422015/08/13-14:35:16,169+04:00 |

CP_APM_UTRAN_xdrCNodeDNHandler.cc:723 | Thr:33 | ERR | component-name | class-name

| CId:59593978 <CId:59593978> Can NOT get the userLabel for :

[RNC/RNCWCE242 RNCEquipment/0 INode/0]


component-name | class-name | CId:59593978 CId:59593978


Operator DN.

5 Current Status:

Still lots of repeated error message are appearing on WMS logs.

6 Attachments :

FMK_PM_server-20023 logs are attached in the cares

1. Expectations from the TEC:

Need fix for these errors


From: Lank, Karen Dianne (Karen)** CTR **

Sent: 18 August 2015 02:39

To: BABU, Sudhakar (Sudhakar)** CTR **

Cc: ROY, Paul (Paul)** CTR **

Subject: no traces for 1-5878714 Repeated errors on RNCWCE241&RNCWCE242

Hi Sudhakar,

I have been assigned AR 1-5878714. I see in the case notes you have requested

traces to be uploaded to directory /home/wcdma/traces/ETISALAT-UAE/AR1-5878714.

This directory is currently empty.

Regards,

Karen.


Sent: 18 August 2015 14:19

To: Lank, Karen Dianne (Karen)** CTR **

Cc: ROY, Paul (Paul)** CTR **

Subject: RE: no traces for 1-5878714 Repeated errors on RNCWCE241&RNCWCE242

Hi Karen,

Have attached FMK_PM_server-20023 logs in the cares, also please find the

attached escalation note.

Have uploaded some logs on server : /home/wcdma/traces/ETISALAT-UAE/AR1-5878714.

Hope we can start investigation. I have requested below more logs, we are waiting

for it.

If you require any other logs please let us know will try to get those in one go.

1. From when this alarm is been observed.

2. RNC242 fulllog collection

3. RNC242 healthlog

4. RNC242 and RNC241 HFB report for last 15 days

5. Below logs from WMS:

WOM_COMM_SEPE_RNC-20010.log|tra*

WOM_MED_SEPE_FM-20008.log|tra*

cp_utran-20000.log|tra*

wom_pm_sepe_neProxy-20016.log|tra*

FMK_PM_server-20023.log|tra*

WTK_FTS-20017.log|tra*

Thanks,

Sudhakar,



Update to Current Summary: July 18: Escalated to backend team, waiting for their

feedback


From: WAGDY MANSOUR, AYMAN (AYMAN)** CTR **

Sent: 18 August 2015 16:53


Cc: MERAD, ABDESSAMAD (ABDESSAMAD); ABDEL-HALIM, SAYED (SAYED); EL-MIDANY, AHMED

(AHMED); WAGEEH, WAEL (WAEL)** CTR **; SAKR, HESHAM (HESHAM)** CTR **; KHEDR,

MAHMOUD (MAHMOUD); INDIA-WCDMA-UTRAN-BE; SAKR, HESHAM (HESHAM)** CTR **; IBRAHIM

ABD EL NABY, Karim (Karim)** CTR **

Subject: RE: RE: AR1-5843064 // Major // UAE- ETISALAT-ABU DHABI // 9771 WCE RNC

// LR14.2.8 // "Can NOT get the userLabel " Repeated errors on


Hello Babu,

The errors were first observed on 13th of August 18, 2015

Logs uploading is ongoing to China server :

/home/wcdma/traces/ETISALAT-UAE/AR1-5878714

Please check after 2 hours

Best Regards,

Ayman Wagdy

Senior WCDMA NPI-EMEA-RBC

Alcatel-Lucent U.A.E

Cellular: +971 50 672 2947

E-mail: [email protected]

11. 18-Aug-2015 22:42 ALCATEL-LUCENT PROPRIETARY gserise

Hello Sudhakar,

i looked at the traces in FMK_PM_server-20023.tra and the error you are referring

to is purely related xdr format PM files decoding by WMS process

FMK_PM_server-20023.

It seems the FMK_PM_server-20023 is trying to decode Inode counters in WCE RNC

counter files.

Therefore it is trying to map the WCE RNC Inode object with what it has in its

database.




the problem is Inode is a 9370 RNC object (Legacy RNC). In the WCE 9771 RNC, the

INode was replaced with another object called WNode.

On 9370 , 2 Stats file are generated, one for Cnode and one for INode.

On WCE only one is generated containing counters of both CNode and WNode.

This is a non sense to look for object Inode in a WCE type RNC.

However, despite the presence of error messages in the processing of WCE stats

file in FMK_PM_server-20023.tra, there is absolutely no impact whatsoever to the

management of WCE RNC stats/FM and CM by WMS server.

This is a WMS bug. This AR should be escalated to WMS team. this issue can be

easily reproduced in a lab.

Wrong Product, Please escalate to WMS Product team.

Best Regards,

Ghislain Serise

12. 18-Aug-2015 23:17 ALCATEL-LUCENT PROPRIETARY asowala

From: "BABU, Sudhakar (Sudhakar)** CTR **" <[email protected]>

Sent: 2015-08-18 21:16:25

To: ALU Support <[email protected]>

Cc:




Dear WC,

Please change the product of this AR1-5878714 to 9353WMS.

Thanks,

Sudhakar,



Update to Current Summary: July 18: Seems to be WMS bug, WMS team to investigate

this issue.



Sent: 19 August 2015 00:53

To: CHATTERJEE, Nilanjan (Nilanjan)** CTR **

Cc: INDIA-WCDMA-UTRAN-BE

Subject: Re: AR1-5878714 // Major // UAE- ETISALAT-ABU DHABI // 9771 WCE RNC //



Hi Nilanjan,

This AR issue seems to be WMS bug. Please see the explanation from RNC backend

team.

We have the change the product to 9353WMS.

Please assignee this AR to team member to work on this issue.

Thanks,

Sudhakar,




Update to Current Summary: July 19: Seems to be WMS bug, WMS team to investigate

this issue.


Outgoing E-mail to

Workflow Status: Queued

Subject: Re: RE: AR1-5878714 // Major // UAE- ETISALAT-ABU DHABI // 9771 WCE RNC



Contact:

Agent: asowala

From: [email protected]

To: "BABU, Sudhakar (Sudhakar)** CTR **" <[email protected]>

Cc:

Bcc:

Sent on:

Dear Babu,

Please be informed that changing product to 9353 WMS and Product Instance to

9353WMS-Etisalat300 requires filling in the 'Request Update Support' field in tab

Classification but GWC cannot fill in this field.

Kind regards,

Agnieszka Sowala

ALCATEL-LUCENT

GLOBAL WELCOME CENTER

Visit OnLine Customer Support for Contact details


After consultation with shift I emailed back to the owner.


From: [email protected]

[mailto:[email protected]] On Behalf Of BABU,

Sudhakar (Sudhakar)** CTR **

Sent: 19 August 2015 01:03

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Cc: WAGDY MANSOUR, AYMAN (AYMAN)** CTR **; MERAD, ABDESSAMAD (ABDESSAMAD);

ABDEL-HALIM, SAYED (SAYED); EL-MIDANY, AHMED (AHMED); WAGEEH, WAEL (WAEL)** CTR

**; SAKR, HESHAM (HESHAM)** CTR **; KHEDR, MAHMOUD (MAHMOUD);

INDIA-WCDMA-UTRAN-BE; SAKR, HESHAM (HESHAM)** CTR **; IBRAHIM ABD EL NABY, Karim

(Karim)** CTR **; CHATTERJEE, Nilanjan (Nilanjan)** CTR **; INDIA-WCDMA-WMS-NPO-BE




Hi Karim,

While checking the traces in FMK_PM_server-20023.tra and the error you are

referring to is purely related xdr format

PM files decoding by WMS process FMK_PM_server-20023. It seems the

FMK_PM_server-20023 is trying to decode

Inode counters in WCE RNC counter files. Therefore it is trying to map the WCE RNC

Inode object with what it has in its database.




The problem is Inode is a 9370 RNC object (Legacy RNC). In the WCE 9771 RNC, the

INode was replaced with another object called WNode.

On 9370 , 2 Stats file are generated, one for Cnode and one for INode. On WCE

only one is generated containing counters of both

CNode and WNode. There is no sense to look for object Inode in a WCE type RNC.

However, despite the presence of error messages in the processing of WCE stats

file in FMK_PM_server-20023.tra,

there is absolutely no impact whatsoever to the management of WCE RNC stats/FM and

CM by WMS server.

This seems to be some issue on WMS. We have changed the product of this AR to

9353WMS. Our WMS team will look into this and

they will provide feedback as soon as possible. This issue can be easily

reproduced in a lab.

Thanks,

Sudhakar,



From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: 19 August 2015 11:50





INDIA-WCDMA-UTRAN-BE; SAKR, HESHAM (HESHAM)** CTR **; CHATTERJEE, Nilanjan

(Nilanjan)** CTR **; INDIA-WCDMA-WMS-NPO-BE




Thanks Bubu for your feedback, Please add our colleague who assigned to the AR now

to the loop

Please note we have planned migration for the WMS to EP5 as requested on another

AR , so we appreciate to receive the investigation , so in case we have to do

anything or add more patches we will take it into our consideration,

Thanks,

BR,

Karim Ibrahim

LTE/3G Integration Professional

Alcatel-Lucent, Dubai, Etisalat Program

M: +971(0)563544568| ONNET: 22341798


From: CHATTERJEE, Nilanjan (Nilanjan)** CTR **

Sent: 19 August 2015 12:26

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; BABU, Sudhakar (Sudhakar)** CTR **




INDIA-WCDMA-UTRAN-BE; SAKR, HESHAM (HESHAM)** CTR **; INDIA-WCDMA-WMS-NPO-BE




Hello Karim,

When are you planning to install Ep5?

As already identified the errors are not service affecting. Errors are generating

because WMS FMK PM server process is trying to get inode information from WCE XDR

PM files which actually does not exists.

In EP5 there is already some improvements on handling WCE XDR files. So please go

ahead installing the patch will see afterwards if the errors are still or not. If

required we will continue the investigation on latest SW load.

As per the latest release note in LR14.2 EP5 is the latest patch. So if you

install EP5 it will automatically cover with all the latest available patches.

BR

NILANJAN CHATTERJEE

Team Lead, OAM L2 Support- 2G & 3G

INDIA

On-NET:2758-8716

T:+91-22-66798716

M:+91-9820820019

[email protected]



22. 19-Aug-2015 18:13 ALCATEL-LUCENT PROPRIETARY nchatter


23. 20-Aug-2015 13:02 achowdar

Update to Current Summary: 20th Aug: EP5 need to be installed, feedback awaited

from field team.

24. 24-Aug-2015 11:36 achowdar


from field team.

25. 27-Aug-2015 13:57 achowdar


from field team.

Add to log

End of Assistance Request 1-5878714

New AR: View AR | AR without Proprietary | OLCS

History | States | Assignment | Timetracking

View Attachments | Add Attachment

AR Text Search | More | CARES Home

AR3

Documents

Transcript of AR3