AR97

New AR: View AR | AR without Proprietary

History | States | Assignment | Timetracking

AR Text Search | More | CARES HomeAssistance Request 1-5532074

Customer Ticket N/A

Short Repeated TMUs reset on RNC231

Description

Current Fix in LR13.3, LR14.2, LR14.2.5, LR15.3 loads

Summary

State Closed : Solution/Service Provided

Outage No

Severity 1

Priority 3

Service Request Remote Technical Support

Request Type Defect

Sub-Type Software

Category Software

Sub-Category Software Defect

Internal Yes

Assignment Events

Product

Product 9370 RNC

Version LR13.3.W

No Change Request is linked to this AR

Solution Source 1400075

Product Location

Site ETISALAT ABU DHABI

Instance 9370RNC-Etisalat : In Service

Site Company Emirates Telecommunications Corporation

City Abu Dhabi

Country United Arab Emirates

Contact

Name Karim IBRAHIM ABD EL NABY

Company Alcatel-Lucent : Open on behalf of Customer

Phone 20 127 109 0616 Email [email protected]

Request Method Email-Eng

Dates

Occurred 12-Jan-2015 21:27 Time now 29-Aug-2015 13:08 (GMT+4)

Reported 12-Jan-2015 21:27 AR Created 12-Jan-2015 21:30

Service Start 12-Jan-2015 21:27

Next Customer Contact 29-Jan-2016 03:00

Responded 12-Jan-2015 21:53 Respond Target 12-Jan-2015 21:57 SA - Calculated

Restored 15-Jan-2015 17:00 Restore Target 15-Jan-2015 17:14 SA - Calculated

Resolved 12-Mar-2015 11:33 Resolve Target 28-Apr-2015 15:01 SA - Calculated

Targets for Restore & Resolve were extended by 45 days

to account for Pending time

Closed 12-Mar-2015 11:35

Last Modified 10-Jul-2015 16:02 Modified By archive

Entitlement

Agreement 242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script This request is entitled to service.

People

Owner TSCr-MoA-WCDMA-SK : rories

Assignee TSCr-MoA-WCDMA-SK : rories

Copy To rories

Referred 1 TEC-MoA-WCDMA_RNC-FR : remiburie

Resolve Group TEC-MoA-WCDMA_RNC-FR

Submitter kmajda

Description

Attachments

[empty]

Showing Investigation and Proprietary logs together in time order Show separately

Investigation Log and Proprietary Log

1. 12-Jan-2015 21:30 kmajda

Update to Current Summary: TEC support to check the Repeated TMUs reset on RNC231,

After investigation on the AR 1-5531112 it’s recommended to open another ticket

with this description,

2. 12-Jan-2015 21:53 rories

From: RIES, Robert (Robert)

Sent: Monday, January 12, 2015 6:50 PM

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Cc: BERIDY, AHMED (AHMED); BOSLEY, Tim (Tim)** CTR **; KHEDR, MAHMOUD (MAHMOUD);

REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); Berky, Dusan (Dusan); MOHAMED,

GEHAD (GEHAD); Kosorin, Rastislav (Rastislav); EL-MIDANY, AHMED (AHMED); OKASHA,

TAHER (TAHER)

Subject: RE: Etisalat UAE/Repeated TMUs reset on RNC231 /1-5532074

Hi Karim

Based on our just finished conference call, you opened this ticket to support

deeper investigation of TMU issue on RNC231.

For your information - basic logs were collected so we started the investigation

procedure already. In case of another info, I will page you if needed.

Best Regards

Robert Ries

Update to Current Summary: TPS w.i.p

3. 13-Jan-2015 17:39 rories

Update to Current Summary: Escalated to TPS L3

4. 13-Jan-2015 17:39 ALCATEL-LUCENT PROPRIETARY rories

Dear TEC team

This ticket is opened based on new issue founded during outage in ETC (UAE)

network, which was solved in another AR 1-5531112.

Issue description:

During emergency recovery activity connected with almost all TMU pendings on

RNC231 (investigated in AR 1-5531112), we found out in hfb the huge amount of

alarms like this:

RNC231;NW/UTRAN RNC/RNC231 RNCEquipment/0 CNode/0;12-10-2014 14:21:18;12-10-2014

14:42:42;quality of service;warning;cleared;softwareProgramAbnormallyTerminated;>

Local value : ANO_RESTART_CONTEXT on PSFP slot 13 Processor role First PMC-TMU;>

Local value : ANO_RESTART_CONTEXT on PSFP slot 13 Processor role First PMC-TMU

Specific Problem:> Local value : ANO_RESTART_CONTEXT on PSFP slot 13 Processor

role First PMC-TMU, manu:[unavailable value], plmno:Ns-FailCause,

common:[unavailable value], AdditionalInformationFromNE:additionalInformation = {

common = { information = { postMortem = fmNormal eqp3GLocation = {

hardSlotPosition = { shelfNumber = 0 positionInShelf = 13 } hardconf = { } } } }

plmno = { information = { plmnoRestartContext = { boardType = v-pmc-boardType

cause = v-ns-FailCause overFMTreshold = v-fmtreshold-notExceeded } } } manu = {

information = "FM PMDZ is written and valid BaseOS PMDZ is empty. " "FM nb event

0. FM threshold has NOT occured. " } } ;RNC_2028_30467;

Alarms started the 10th Dec 2014 and were connected with link Iur/8, which was

therefore blocked (in order to stop TMU resets).

During outage investigation there were many alarms like this:

Lp/9 Ap/1; 2015-01-12 15:05:41.92

MSG minor equipment softwareError 00009000

ADMIN: unlocked OPER: enabled USAGE: active

AVAIL: PROC: reporting CNTRL:

ALARM: STBY: notSet UNKNW: false

Id: 0900000B Rel: Shelf Card/9

Com: TMU -- EXCEPTION:Memory Manager error[memPartFree] at block 0x4fee0

Prev Hdr [0xffffffff] Curr Hdr [0x4fed0] Next Hdr [0xffffffff]

both start and end addr is invalid.


Task:0x7d570000: UeCallp

DAR: 0x00000000

PC: 0x003a94f0 taskUnlock +0x174

LR: 0x6af9656c

_Z63ue_call_in_reloc_AWAIT_DRNC_TO_SRNC_READY_Ue_drnc_to_srnc_readyP23

CPI_UeDrncToSrncReady_tP16Uecall_

Int: 0/0/0/114; /clearlib/vobs/I-Node/sw/PS/Base/PsHAL/psFltSwr/src/ps

FaultHandling.cc; 2925; PATCH: RID37300E1953

From ppc file we can see a lot of:

Lp/5 Ap/1 ; 2014-12-30 08:24:30.53

MSG minor equipment 00009000

ADMIN: unlocked OPER: enabled USAGE: active

AVAIL: PROC: reporting CNTRL:

ALARM: STBY: notSet UNKNW: false

Id: 5000310

Com: TMU -- EXCEPTION:Memory Manager error[memPartFree] at block 0x4fee0

Prev Hdr [0xffffffff] Curr Hdr [0x4fed0] Next Hdr [0xffffffff]



We found also another ticket with similar symptons, maybe which can be useful in

your investigation - AR 1-5498787

Please can you ivestigate the probem and tell us:

a) What is a root cause of the problem ?

b) Fix delivery plan for this issue.

Appropriate logs are on CUBA server 135.120.182.10:

either in directory: /traces/OUTAGE_RCA_ER_GPS/1-5531112

or in directory: /traces/Etisatat UAE/1-5532074 Repeated TMUs reset on RNC231

Thank you

Best Regards

Robert Ries

5. 13-Jan-2015 19:14 ALCATEL-LUCENT PROPRIETARY remiburie

Hi,

CR 1434789 has been ceated and escalated to design.

Regards,

Rémi

6. 14-Jan-2015 11:35 rories

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: Monday, January 12, 2015 7:04 PM

To: RIES, Robert (Robert)


REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); Berky, Dusan (Dusan); MOHAMED,

GEHAD (GEHAD); Kosorin, Rastislav (Rastislav); EL-MIDANY, AHMED (AHMED); OKASHA,

TAHER (TAHER)


Hi Reis,

Attached a network snapshot as requested,

Action taken today:

- IP transport team action about 11 am.

- TMU reset observed with a very high trend with zero traffic on the RNC231 as

below,

- LP switchover action taken to try solve the problem ' Issue enhanced for 10 min

and back to the same behavior.

- Recommended by TSO (Reis) after the ticket opened to perform a RNC shelf reset'

issue enhanced, RNC carrying very low traffic, Alarms still exist.

- TEC invitagate the issue and find that the issue is the IUR towards RNC206.

- IUR link locked, Alarms cleared, unlocked again the alarms back.

- Proposed to lock the IUR link,

As discussed we have the same IUR link towards RNC 206 working fine with RNC235.

Thanks,

--

BR,

Karim

7. 26-Jan-2015 12:34 ALCATEL-LUCENT PROPRIETARY remiburie

Hi,

CR 1434789 is a duplicate of CR1400075 delivered in LR14, it is planned to

backprop this fix to next LR13 MNCL.

Regards,

Rémi

8. 26-Jan-2015 13:45 rories


Sent: Monday, January 26, 2015 10:22 AM



REDA, RAMY (RAMY); ABDEL-HALIM, SAYED (SAYED); MOHAMED, GEHAD (GEHAD); EL-MIDANY,

AHMED (AHMED); OKASHA, TAHER (TAHER)


Hi Karim

Preliminary answer from design:

CR 1434789 opened for this AR is a duplicate of CR 1400075 already delivered to

LR14.

There is planned back propagation to the next LR13 MNCL.

Best Regards

Robert Ries

9. 13-Feb-2015 18:22 ALCATEL-LUCENT PROPRIETARY remiburie

Hi,

Design propose as workaround to deactivate feature PM28436.3

isEnhancedIurRabConfigEnabled to be set to false.

The faulty SRB will be refused but possibility also to refuse RAB that RNC was

able to manage with this feature enabled.

Regards,

Rémi

10. 16-Feb-2015 12:46 rories


Sent: Tuesday, February 03, 2015 8:25 AM






Hi Reis,

You close this AR, and please when the CR is ready share it with us.

Thanks,

--

BR,

Karim

11. 16-Feb-2015 12:47 rories


Sent: Monday, February 16, 2015 9:46 AM






Hi Karim

Design proposed next workaround:

Please deactivate the feature PM28436.3 "isEnhancedIurRabConfigEnabled" - it needs

to be set to false.

By this WA the RNC will refuse RABs with faulty SRB. Once there will be fix (based

on CR 1434789 & 1400075) available in LR13 MNCL, I let you know.

Thanks

Best Regards

Robert Ries

12. 16-Feb-2015 12:47 ALCATEL-LUCENT PROPRIETARY rories

De : RIES, Robert (Robert)

Envoyé : lundi 16 février 2015 08:37

À : BURIE, REMI (REMI)

Objet : FW: 1-5532074 has been modified

Hi Rémi

Few questions:

Regarding the second sentence in your update - you wrote, that by deactivating of

feature PM28436.3 there will be refused RABs with faulty SRB. OK, but once there

will be available fix in LR13 MNCL, the feature PM28436.3 could be activated back

?

Regarding CR 1434789 (or CR 1400075) - customer agreed to close this AR in fact,

but he needs to be alerted once fix is available in LR13 MNCL (his mail in

attachment). Do you know when it will be ? Because in case it will be in short

time, I let the ticket open to wait for fix availability.

Thanks

Best Regards

Robert Ries

13. 16-Feb-2015 12:48 ALCATEL-LUCENT PROPRIETARY rories

From: BURIE, REMI (REMI)

Sent: Monday, February 16, 2015 9:27 AM


Subject: RE: 1-5532074 has been modified

Hi Robert,

Design is working on it now, I don't have a date for the fix already.

If I have one I will communicate it of course.

Regards,

Rémi BURIE

14. 26-Feb-2015 12:51 rories


Sent: Monday, February 16, 2015 1:52 PM




AHMED (AHMED); OKASHA, TAHER (TAHER); MERAD, ABDESSAMAD (ABDESSAMAD); BERIDY,

AHMED (AHMED)


Hello Reis,

Thanks for your feedback, Please what will be the impact the IUR behavior after

unlock, if this deactivation will enhance the KPIs? Or it's only related to TMU

reset issue?

Thanks,

--

BR,

Karim

15. 26-Feb-2015 12:52 rories


Sent: Tuesday, February 17, 2015 8:36 AM





AHMED (AHMED)


Hello Karim

If you will set the "isEnhancedIurRabConfigEnabled" to false, the RNC will refuse

RABs with faulty SRB. Impact to KPI is related only to RABs, which will be

refused. So here is assumption, that IuR traffic will start to be a little bit

lower.

Set this feature to false and monitor KPIs, please. In case there will be

unexpected impact to KPI, we will report this matter to TEC.

Best Regards

Robert Ries

16. 06-Mar-2015 21:51 rories


Sent: Friday, March 06, 2015 6:50 PM





AHMED (AHMED)


Hello Karim

You asked us for reminder once the fix is implemented. So now is fix for this

issue implemented in LR13.3, LR14.2, LR14.2.5, LR15.3 loads.

As you already authorized, kindly I will close this ticket.

Best Regards

Robert Ries

17. 12-Mar-2015 11:31 ALCATEL-LUCENT PROPRIETARY rories

Time Tracking Entry Added - IR01-TSA3

18. 12-Mar-2015 11:33 rories

Update to Current Summary: Fix in LR13.3, LR14.2, LR14.2.5, LR15.3 loads

AR97

Documents

Transcript of AR97