AR94

New AR: View AR | AR without Proprietary

History | States | Assignment | TimetrackingAR Text Search | More | CARES Home

Assistance Request 1-5547042

Customer Ticket NA

Short Etisalat UAE/Most of on air sites are suffering from high uplink

Description frame loss

Current Feb 10 : Investigation on IP link is suggested. This is

Summary transmission media issue. Customer agreed to close the AR.

State Closed : Solution/Service Provided

Outage No

Severity 2

Priority 3

Service Request Remote Technical Support

Request Type Support

Sub-Type Configuration Assistance

Category Other

Internal Yes

Assignment Events

Product

Product 9926 DBS-WCDMA

Version LR13.3.W

No Change Request is linked to this AR

Product Location

Site ETISALAT ABU DHABI

Instance 9926DBS WCDMA-Etisalat : In Service

Site Company Emirates Telecommunications Corporation

City Abu Dhabi

Country United Arab Emirates

Contact

Name Karim IBRAHIM ABD EL NABY

Company Alcatel-Lucent : Open on behalf of Customer

Phone 20 127 109 0616 Email [email protected]

Request Method Email-Eng

Dates

Occurred 21-Jan-2015 13:36 Time now 29-Aug-2015 13:02 (GMT+4)

Reported 21-Jan-2015 13:36 AR Created 21-Jan-2015 13:42

Service Start 21-Jan-2015 13:36

Next Customer Contact 27-Feb-2015 08:30

Responded 21-Jan-2015 13:42 Respond Target 21-Jan-2015 14:36 SA - Calculated

Restored 21-Jan-2015 13:45 Restore Target 23-Jan-2015 13:36 SA - Calculated

Resolved 10-Feb-2015 23:51 Resolve Target 09-Mar-2015 03:53 SA - Calculated

Targets for Restore & Resolve were extended by 16 days

to account for Pending time

Closed 10-Feb-2015 23:51

Last Modified 11-Jun-2015 16:06 Modified By archive

Entitlement

Agreement 242895 (OXIA 627085)

Covered Service TS 24x7 (Gold, Wireless Mobile Access)

Script This request is entitled to service.

People

Owner TSCd-WLS-WCDMA-IN : pthota

Assignee CloQ-WLS-WCDMA-vGlobal : pthota

Referred 1 NorP-WLS-WCDMA-vGlobal : pthota

Referred 2 TEC-MoA-WCDMA_BTS : unassigned

Referred 3 TEC-WLS-WCDMA-BTS : brajendm

Resolve Group TEC-WLS-WCDMA-BTS

Submitter mkarpuz

Description

Attachments

Most of on air sites are suffering from Edch frame loss, We need to investigate

the RC of this KPIs degradation, the issue appears on any site directly after

swap, TEC support is needed to check the RC of this KPIs degradation,

From: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

<[email protected]>

Sent: 2015-01-21 10:36:55

To: ALU Support <[email protected]>

Subject: Etisalat UAE/Most of on air sites are suffering from high uplink frame loss

Hello,

Short Description : Most of on air sites are suffering from

high uplink frame loss,

Current Summary : Most of on air sites are suffering from Edch

frame loss, We need to investigate the RC of this KPIs degradation, the

issue appears on any site directly after swap, TEC support is needed to

check the RC of this KPIs degradation,

Contact Surname : Karim

Contact Given Name : Ibrahim

Contact Phone :+971 56 354 4568

Contact Company : Alcatel-Lucent

Company : Emirates Telecommunications Corporation

Contract Number : OXIA 627085

Region : EMEA

Country : United Arab Emirates

City : Abu Dhabi

Site : ETISALAT ABU DHABI

Product Instance : 9926 DBS-WCDMA

Severity : 2

Priority : 3

Product Version : LR13.3W

Outage : No

Internal/External: :Internal

--

Best Regards,

Karim Ibrahim

LTE/WCDMA Integration Professional

META WLS Access Network (GNE EMEA)

M(UAE): +

<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatel-lucent.com%2fowa%2fUrlBlockedError.aspx>

971-56-354-4568

M(Egy): +

<https://mail.eu.alcatel-lucent.com/owa/redir.aspx?C=30449991f54743fd85063b63258fd6f2&URL=https%3a%2f%2fmail.eu.alcatel-lucent.com%2fowa%2fUrlBlockedError.aspx>

2010-668-44-810

ONNET: 2205 5534

Attachments

Showing Investigation and Proprietary logs together in time order Show separately

Investigation Log and Proprietary Log

1. 21-Jan-2015 13:45 ALCATEL-LUCENT PROPRIETARY mkarpuz

Outgoing E-mail to Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Workflow Status: Stopped

Subject: Re: Etisalat UAE/Most of on air sites are suffering from high uplink

frame loss

Contact: Karim (Karim)** CTR ** IBRAHIM ABD EL NABY

Agent: mkarpuz

From: [email protected]

To: "IBRAHIM ABD EL NABY, Karim (Karim)** CTR **"

<[email protected]>

Cc: "OKASHA, TAHER (TAHER)" <[email protected]>, "ABDEL-HALIM,

SAYED (SAYED)" <[email protected]>, "BERIDY, AHMED (AHMED)"

<[email protected]>, "NABIL GEORGES, MARTIN (MARTIN)** CTR **"

<[email protected]>, "REDA, RAMY (RAMY)"

<[email protected]>, "KHEDR, MAHMOUD (MAHMOUD)"

<[email protected]>, "ZAHABI, RAMSEY (RAMSEY)"

<[email protected]>, "WAGDY MANSOUR, AYMAN (AYMAN)** CTR **"

<[email protected]>, "EL-MIDANY, AHMED (AHMED)"

<[email protected]>

Bcc:

Sent on: January 21, 2015 10:43:50 AM CET

Dear Customer,

Thank you for contacting the Alcatel-Lucent Welcome Center.

Your request has been processed and the ticket number for your reference is AR

1-5547042

Please reference that number when contacting Alcatel-Lucent for follow-up.

An Alcatel-Lucent representative will contact you regarding your request.

Kind Regards,

Murat Karpuz

ALCATEL-LUCENT

GLOBAL WELCOME CENTER

Visit OnLine Customer Support for Contact details

2. 21-Jan-2015 13:49 pthota

Update to Current Summary: Jan 21 : Under Investigation

3. 21-Jan-2015 13:53 pthota

Update to Current Summary: Jan 21 : Awaiting Receive Log Files / Technical Info

4. 28-Jan-2015 09:57 pthota

Update to Current Summary: Jan 28 : Awaiting Receive Log Files / Technical Info

5. 29-Jan-2015 20:46 ALCATEL-LUCENT PROPRIETARY shelends

From: THOTA, Praveen-K (Praveen-K)** CTR **

Sent: Thursday, January 22, 2015 12:50 PM

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; NABIL GEORGES, MARTIN (MARTIN)**

CTR **; BERIDY, AHMED (AHMED)

Cc: INDIA-WCDMA-CdTSC-TEAM; OKASHA, TAHER (TAHER); ABDEL-HALIM, SAYED (SAYED);

WAGDY MANSOUR, AYMAN (AYMAN)** CTR **; ZAHABI, RAMSEY (RAMSEY); EL-MIDANY, AHMED

(AHMED); ELHAKIM, Mohamed (Mohamed)** CTR **; KHEDR, MAHMOUD (MAHMOUD); REDA, RAMY

(RAMY); NADIR, HAMMAD (HAMMAD)

Subject: RE: AR 1-1-5547042//Major//9926 DBS-WCDMA//Emirates Telecommunications

Corporation//UAE-Abudhabi//Most of on air sites are suffering from high uplink

frame loss

Hi Karim/Martin,

Kindly provide your inputs for the below queries...

1) What is the end user impact observed due to this KPI degradation. Also

provide the customer Daily KPI impact with degradation percentage.

2) From the NPO reports I can see there is some sites which are having high

frame loss events ex. AR_U_2614_GayathiIndustrilaII. Some other sites are having

very less events ex. EP_AR_U_2543_CityMallMzd. Please check and confirm the

geographic conditions of the high frame loss event sites.

3) Could you please confirm all these high frame loss NodeBs from single RNC or

distributed in multiple RNCs?

4) Could you please provide these counter reports for RNC level as well?

VS.EdchIubTnlCongestIndc.DelayBuildUp

VS.EdchIubTnlCongestIndc.FrameLoss

VS.EdchIubTnlCongestIndc.NoCongestion

VS.EdchIubTnlCongestIndc.Reserved

VS.EdchMacTsnReassemblyErr.DupTsn

VS.EdchMacTsnReassemblyErr.UnrecovTsn

Also please provide below details for the longer period for High frame loss events

NodeB as above mentioned.

Metrics:

- UHSUPA024_CR (EdchHarqRetransmissions_HSUPA024_CR)

- UHSUPA004_CR (HSUPAHARQBLER_HSUPA004_CR)

- UHSUPA021_CR EdchHARQBler1stRtx_HSUPA021_CR

- URB008_C_Hsupa

- UplinkCellLoad_ULOAD001_CR

- UplinkCellLoadHSUPA_ULOAD006_CR

- UplinkRSSI_ULOAD002_CR

- UplinkRSSI_ULOAD003_CR

- HsdpaToDchFallbackUnsuccess_HSDPA048_C_HsdpaEdch

Counters :

- VS.DedicatedUplinkBadPdusRlcRefCell.UlRabHsupa

- VS.DedicatedUplinkKbytesRlcActiveCells.UlRabHsupa

- VS.DedicatedUplinkKbytesRlcReferenceCell.UlRabHsupa

- VS.DedicatedUplinkSduRlcRefCell.UlRabHsupa

- VS.DedicatedUplinkMissingPduRlc.UlRabHsupa

- VS.UlDiscardNonConfTraffic.UlRabHsupa =

- VS.EdchFpRetransHarq.NharqFailInd

- VS.EdchFpRetransHarq.NsubfrmNharqEq0

- VS.EdchFpRetransHarq.NsubfrmNharqEq1

- VS.EdchFpRetransHarq.NsubfrmNharqGt1

- VS.EdchIubTnlCongestIndc.DelayBuildUp

- VS.EdchIubTnlCongestIndc.FrameLoss

- VS.EdchIubTnlCongestIndc.NoCongestion

- VS.EdchIubTnlCongestIndc.Reserved

- VS.EdchMacTsnReassemblyErr.DupTsn

- VS.EdchMacTsnReassemblyErr.UnrecovTsn

- VS_eDCHactiveUsers_10ms_users (U10905_2)

- VS_eDCHactiveUsers_2ms_users (U10905_1)

Thanks and Regards,

Praveen Thota

+91 96899 18050

----------------------------------------------------------------------------------

----------------------------------------------------------

From: SHARMA, Shelendra (Shelendra)** CTR **

Sent: Monday, January 26, 2015 8:08 AM

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; ARORA, Gaurav (Gaurav)** CTR **




(RAMY); NADIR, HAMMAD (HAMMAD); NABIL GEORGES, MARTIN (MARTIN)** CTR **; THOTA,

Praveen-K (Praveen-K)** CTR **; BERIDY, AHMED (AHMED)



frame loss

Hi Karim

From IMT logs, We can see Transmission flactuations in alarms and IMT logs. Also

please check Conectivity from PTP server, seems Node Bs not getting perfect

clock.

Also In 2 IMT logs, getting RRH related issues Kindly check RRH connectivity and

Power. Also check RTWP values for these Node Bs for all 3 sectors.

296_imt_2015-01-24T14h02m55

From logs:

2015-01-17 06:01:40:560:23EF5DF 9914.37680607 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-17 06:01:37 NO GRANT FROM PTP SERVER1

2015-01-17 06:01:40:569:247B5D5 9914.38254037 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-17 06:01:37 NO GRANT FROM PTP SERVER1

2015-01-17 06:11:26:568:894B69D8 9922.2303420888 [

2015-01-17 07:10:41:695:452CCF88 9974.1160564616 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 07:10:38 HEARTBEAT FAILURE

2015-01-17 07:10:46:600:577221D9 9974.1467097561 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-17 07:10:43 HEARTBEAT FAILURE

2015-01-17 22:25:07:995:AE122EF5 10772.2920427253 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 22:25:05 HEARTBEAT FAILURE

2015-01-17 22:25:18:296:D471A7BE 10772.3564218302 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-17 22:25:15 HEARTBEAT FAILURE

2015-01-17 22:25:51:188:4EF9CE65 10773.1324994149 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 22:25:48 HEARTBEAT FAILURE

2015-01-17 22:25:54:296:5A8DCD5B 10773.1519242587 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-17 22:25:51 HEARTBEAT FAILURE

2015-01-18 22:46:45:908:DA40BE79 12048.3661676153 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-18 22:46:43 HEARTBEAT FAILURE

2015-01-18 22:46:47:206:DF16B243 12048.3742806595 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-18 22:46:45 HEARTBEAT FAILURE

IMT logs:-

pltf/679[btimed ] pkt.c#681 2015-01-24 08:00:46:449:FEC8F926 ERR 014 23:13:20, <9029ac00> :PTP2 time server IP address has not been configed!

pltf/679[btimed ] pkt.c#681 2015-01-24 08:01:06:452:494D0FAF ERR 014 23:13:40, <9029ac00> :PTP2 time server IP address has not been configed!

pltf/679[MsgSeq ] CoreEccm.cc#2051 2015-01-24 08:01:27:127:9651D0EF ERR 014

23:14:01, <044f0015> :Eccm.05.......$0$35$35$

pltf/679[MsgSeq ] CoreEccm.cc#2051 2015-01-24 08:02:27:132:75DBABE3 ERR 014

23:15:01, <044f0015> :Eccm.05.......$0$35$35$

pltf/679[MsgSeq ] CoreEccm.cc#2051 2015-01-24 08:03:27:139:556605A4 ERR 014

23:16:01, <044f0015> :Eccm.05.......$0$35$35$

pltf/679[iub_link_c] bifmgt.c#4707 2015-01-24 08:04:09:513:F3417C25 ERR 014

23:16:43, <902bc400> :bif_getMac: Unable to find in arp cache, even after MAC

translation

pltf/679[T_IpInterf] ipif_pm.c#1859 2015-01-24 08:10:00:411:0E739C0E WAR 014

23:22:34, <902bc500> :IPIF_Ipinterface::g_IPIF_OAM_Counters Error count

exceeding:InError 0,OutError 32

2. 2551_imt_2015-01-24T14h06m47

Please Check RRH and power.

pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:790:4B49B48F WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN

RfhParserImpl::cbDetectedUnknownDataField(): found unknown data field=RX3PWR

pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:790:4B49BCBA WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9CE6B WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9DC4A WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9E7F9 WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9EFF1 WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


Also check Transmission flactuation & Clock stability.

2015-01-17 18:24:45:393:C9563BBF 27582.3377871807 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-17 18:24:43 NO GRANT FROM PTP SERVER1

2015-01-17 18:24:45:440:C982FE46 27582.3380805190 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-17 18:24:43 NO GRANT FROM PTP SERVER1

2015-01-12 13:26:26:202:E43EA5A6 21035.3829310886 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-12 13:26:22 HEARTBEAT FAILURE

2015-01-12 13:45:56:607:EC57878D 21052.3965159309 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-12 13:45:53 HEARTBEAT FAILURE

2015-01-12 14:24:36:099:AD1F0C86 21086.2904493190 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-12 14:24:32 HEARTBEAT FAILURE

2015-01-12 14:30:23:866:BCA79078 21091.3165098104 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-12 14:30:20 HEARTBEAT FAILURE

2015-01-14 14:16:44:784:6380A067 23594.1669374055 [MsgSeq/37b394b0] ETHPORT/0 (ETHERNET PORT/0) RAISED 2015-01-14 14:16:41 HW ETHERNET LINK FAILURE

2015-01-14 14:16:44:784:6380AA2E 23594.1669376558 [MsgSeq/37b394b0] IP RAN/0 (IP RAN/0) RAISED 2015-01-14 14:16:41 IP RAN FAILURE

2015-01-14 14:16:46:365:6963E583 23594.1768154499 [MsgSeq/37b394b0] ETHPORT/0 (ETHERNET PORT/0) CLEARED 2015-01-14 14:16:43 HW ETHERNET LINK FAILURE

2015-01-14 14:16:46:365:6963EF74 23594.1768157044 [MsgSeq/37b394b0] IP RAN/0 (IP RAN/0) CLEARED 2015-01-14 14:16:43 IP RAN FAILURE

2015-01-17 18:24:45:393:C9562023 27582.3377864739

3. 2902_imt_2015-01-24T14h13m39

2015-01-02 18:09:10:784:DA8615DE 22940.3666220510 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-02 18:08:36 HEARTBEAT FAILURE

2015-01-02 18:09:22:876:7922635 22941.127018549 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-02 18:08:49 HEARTBEAT FAILURE

2015-01-08 23:21:24:460:2D654A94 30757.761612948 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-08 23:21:00 HEARTBEAT FAILURE


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:32:997:66F62787 WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:32:997:66F633CE WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:32:997:66F63F78 WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN



01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:33:092:67516925 WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:33:093:675175EB WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


Regds

Shelendra

----------------------------------------------------------------------------------

-----------------------------------------------------------

From: NABIL GEORGES, MARTIN (MARTIN)** CTR **

Sent: Monday, January 26, 2015 9:07 AM

To: SHARMA, Shelendra (Shelendra)** CTR **; IBRAHIM ABD EL NABY, Karim (Karim)**

CTR **; ARORA, Gaurav (Gaurav)** CTR **




(RAMY); NADIR, HAMMAD (HAMMAD); THOTA, Praveen-K (Praveen-K)** CTR **; BERIDY,

AHMED (AHMED)



frame loss

Hello Shelendra,

First thanks a lot for your initial analysis

Kindly could you provide root cause for EDCH frame loss. Is there any needed logs

that can show us what is the root cause of this degradation.

Could you tell us what is the impact of not receiving correct clock, as far as I

understand most of the clocks received are either accepted with correct clock

settings or discarded. The Clock is vulnerable to jitter, so from KPIs no jitter

indication is found, so did you found any Jitter indication in the provided logs

Best Regards,

Martin Nabil

Radio Network department,

Regional Service delivery

EMEA

M: +201200254921 (Egypt)

----------------------------------------------------------------------------------

-------------------------------------------


Sent: Tuesday, January 27, 2015 8:32 AM

To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; NABIL GEORGES, MARTIN (MARTIN)**

CTR **; ARORA, Gaurav (Gaurav)** CTR **





AHMED (AHMED)



frame loss

Hi Karim

Analysis was based on IMT logs, which were provided by you.

If PTP server is ok and no issue in Clock, so why NO GRANT FROM PTP alarm or clock

flactuation alarm is coming?

If transport is ok, so why Transmission flactuation alarms are coming IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 22:25:48 HEARTBEAT FAILURE ?

Need to be check on those Node Bs and rectify first.

For RTWP, RTWP is the Received Total Wideband Power, the total level of noise

within the UMTS frequency band of any cell.

Generally RTWP value in ideal case is about -104.5 dBm and -105.5 dBm.

You can verify RTWP values in problematic Node Bs and also share with us.

Regds

Shelendra Sharma

+91 9930969351

----------------------------------------------------------------------------------

---------------------------------------------------

From: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **

Sent: Tuesday, January 27, 2015 9:43 AM

To: SHARMA, Shelendra (Shelendra)** CTR **; ARORA, Gaurav (Gaurav)** CTR **





AHMED (AHMED); NABIL GEORGES, MARTIN (MARTIN)** CTR **



frame loss

Importance: High

Hi Sharma,

Please for the PTP issue go through the AR I mentioned and you will find that due

to the TX instability we faced this Sync alarms.

For the TX instabilities , you can check also the AR , you will find that it's due

to TX instability,

My Question is: what's the RC of facing this KPIs trend? Is it totally the RTWP?

For the site 296 is I mentioned in my below mail the site is NOT suffering from

frame loss: and I provide the IMT to you to check what's the difference between it

and other sites, but as below the analysis done also on it we same output of

alarms, so I think the TX&PTP alarms is not the main cause.

Waiting your feedback.

Thanks,

----------------------------------------------------------------------------------

----------------------


Sent: Thursday, January 29, 2015 2:38 PM









frame loss

Hi Karim

RTWP values are not Ok for 2614.

I will add GPS team, But kindly involve RF team to check Radio conditions.

For Good Node B, RTWP values should be stable.

Ex:- -100 to -104 db

Regds

Shelendra Sharma

+91 9930969351


Dear GPS Team

I am escalating this AR for deeper investigation

1 ? DESCRIPTION TEMPLATE

SUMMARY: Most of on air sites are suffering from high uplink frame loss

Concerned RNC: 9926 DBS-WCDMA

Concerned component : 9926 DBS-WCDMA

Type of alarm: In IMT logs, Getting Transmission, PTP Grant (Synchronization),

RTWP flactuation & CPRI_LIB Alarm

Time of the issue: As per customer, High uplink frame loss in some of Node Bs

Impact : Most of on air sites are suffering from high uplink frame loss

Product release: LR13.3.W

Patch release: N/A

Reproducible: Yes Happening

Suggested solution (workaround...): Getting Transmission & Clock (PTP Grant)

alarms in Two Node Bs, & RTWP values are flactuating in Node B

PROBLEM DETAILS:

1. Action performed (feature activation, upgrade, work?order application...):

General

2. System behavior (high?level + detailed description if relevant): High uplink

frame loss in some Node Bs in network.

3. TSC/TIS analysis: Getting Below alarms in IMT logs.

Node B No 1:-

2015-01-17 06:01:40:560:23EF5DF 9914.37680607 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-17 06:01:37 NO GRANT FROM PTP SERVER1

2015-01-17 06:01:40:569:247B5D5 9914.38254037 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-17 06:01:37 NO GRANT FROM PTP SERVER1

2015-01-17 06:11:26:568:894B69D8 9922.2303420888 [

2015-01-17 07:10:41:695:452CCF88 9974.1160564616 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 07:10:38 HEARTBEAT FAILURE

2015-01-17 07:10:46:600:577221D9 9974.1467097561 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-17 07:10:43 HEARTBEAT FAILURE

2015-01-17 22:25:07:995:AE122EF5 10772.2920427253 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 22:25:05 HEARTBEAT FAILURE

2015-01-17 22:25:18:296:D471A7BE 10772.3564218302 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-17 22:25:15 HEARTBEAT FAILURE

2015-01-17 22:25:51:188:4EF9CE65 10773.1324994149 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-17 22:25:48 HEARTBEAT FAILURE

2015-01-17 22:25:54:296:5A8DCD5B 10773.1519242587 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-17 22:25:51 HEARTBEAT FAILURE

2015-01-18 22:46:45:908:DA40BE79 12048.3661676153 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-18 22:46:43 HEARTBEAT FAILURE

2015-01-18 22:46:47:206:DF16B243 12048.3742806595 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-18 22:46:45 HEARTBEAT FAILURE

IMT logs:-

pltf/679[btimed ] pkt.c#681 2015-01-24 08:00:46:449:FEC8F926 ERR 014 23:13:20, <9029ac00> :PTP2 time server IP address has not been configed!

pltf/679[btimed ] pkt.c#681 2015-01-24 08:01:06:452:494D0FAF ERR 014 23:13:40, <9029ac00> :PTP2 time server IP address has not been configed!

pltf/679[MsgSeq ] CoreEccm.cc#2051 2015-01-24 08:01:27:127:9651D0EF ERR 014

23:14:01, <044f0015> :Eccm.05.......$0$35$35$

pltf/679[MsgSeq ] CoreEccm.cc#2051 2015-01-24 08:02:27:132:75DBABE3 ERR 014

23:15:01, <044f0015> :Eccm.05.......$0$35$35$

pltf/679[MsgSeq ] CoreEccm.cc#2051 2015-01-24 08:03:27:139:556605A4 ERR 014

23:16:01, <044f0015> :Eccm.05.......$0$35$35$

pltf/679[iub_link_c] bifmgt.c#4707 2015-01-24 08:04:09:513:F3417C25 ERR 014

23:16:43, <902bc400> :bif_getMac: Unable to find in arp cache, even after MAC

translation

pltf/679[T_IpInterf] ipif_pm.c#1859 2015-01-24 08:10:00:411:0E739C0E WAR 014

23:22:34, <902bc500> :IPIF_Ipinterface::g_IPIF_OAM_Counters Error count

exceeding:InError 0,OutError 32

Node B 2:-

Please Check RRH and power.

pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:790:4B49B48F WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:790:4B49BCBA WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9CE6B WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9DC4A WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9E7F9 WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


pltf/584[RRHIL ]rfh_ParserIm#218 2015-01-24 14:06:29:891:4BA9EFF1 WAR 028

18:11:56, <0008a000> :CPRI_LIB...>WARN


Also check Transmission flactuation & Clock stability.

2015-01-17 18:24:45:393:C9563BBF 27582.3377871807 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) RAISED 2015-01-17 18:24:43 NO GRANT FROM PTP SERVER1

2015-01-17 18:24:45:440:C982FE46 27582.3380805190 [MsgSeq/37b394b0] BTS/0 (BTS EQUIPMENT/0) CLEARED 2015-01-17 18:24:43 NO GRANT FROM PTP SERVER1

2015-01-12 13:26:26:202:E43EA5A6 21035.3829310886 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-12 13:26:22 HEARTBEAT FAILURE

2015-01-12 13:45:56:607:EC57878D 21052.3965159309 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-12 13:45:53 HEARTBEAT FAILURE

2015-01-12 14:24:36:099:AD1F0C86 21086.2904493190 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-12 14:24:32 HEARTBEAT FAILURE

2015-01-12 14:30:23:866:BCA79078 21091.3165098104 [IubHeartbeat/51d794b0] IP RAN/0 (VIRTUAL INTERFACE/2) CLEARED 2015-01-12 14:30:20 HEARTBEAT FAILURE

2015-01-14 14:16:44:784:6380A067 23594.1669374055 [MsgSeq/37b394b0] ETHPORT/0 (ETHERNET PORT/0) RAISED 2015-01-14 14:16:41 HW ETHERNET LINK FAILURE

2015-01-14 14:16:44:784:6380AA2E 23594.1669376558 [MsgSeq/37b394b0] IP RAN/0 (IP RAN/0) RAISED 2015-01-14 14:16:41 IP RAN FAILURE

Node B No 3:-

2015-01-02 18:09:10:784:DA8615DE 22940.3666220510 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-02 18:08:36 HEARTBEAT FAILURE


2015-01-08 23:21:24:460:2D654A94 30757.761612948 [MsgSeq/37b394b0] IP RAN/0 (VIRTUAL INTERFACE/2) RAISED 2015-01-08 23:21:00 HEARTBEAT FAILURE



01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:32:997:66F633CE WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:32:997:66F63F78 WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN



01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:33:092:67516925 WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


pltf/593[RRHIL ]rfh_ParserIm#218 2015-01-24 14:13:33:093:675175EB WAR 040

01:58:02, <0008a000> :CPRI_LIB...>WARN


RTWP Flactuations:-

Seems RTWP values are flactuating, You can see in below graph & table, also

could you please share hourly reports for last 5 days.

UplinkRSSI_ULOAD002_CR_Max(UULOAD002_CR_Max)(dBm) 12/30/201412/31/2014 01/01/2015

01/02/2015 01/03/2015 01/04/2015 01/05/2015 01/06/201501/07/2015 01/08/2015

01/09/2015 01/10/2015 01/11/2015 01/12/2015 01/13/201501/14/2015 01/15/2015

01/16/2015 01/17/2015 01/18/2015 01/19/2015 01/20/201501/21/2015 01/22/2015

01/23/2015 01/24/2015 01/25/2015 01/26/2015 01/27/201501/28/2015 01/29/2015

AR_2614_26141_F1_A N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A -87.302 -86.802 -81.3

-78.099 -72.8 -80 -81.801 -64.7 -82.101 -79.1 -88.003 -91.208 -95.468 -91.393

-74.3 -92.506 -95.768 -92.605 -84.596 -93.192 N/A

AR_2614_26142_F1_B N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A -79.399 -78.099 -74.2

-72.8 -81.698 -64.6 -77.8 -68.7 -84.7 -66.3 -84.7 -87.607 -71.5 -73.8 -72.1

-88.108 -93.024 -87.302 -96.685 -84.197 N/A

AR_2614_26143_F1_C N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A -76.5 -82.501 -64.4

-78.7 -77 -81.602 -83.5 -67.3 -98.383 -58.6 -88.904 -96.089 -90.21 -75.7 -84.197

-89.595 -85.799 -88.799 -97.376 -80.199 N/A

AR_2614_26144_F2_A N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A -75.3 -77.699 -64.6

-81.101 -84.596 -82.101 -66.4 -70.4 -84.596 -81.201 -85.799 -82.701 -91.101 -92

-84 -89.696 -87.302 -82.2 -82.4 -81.101 N/A

AR_2614_26145_F2_B N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A -69.8 -63.4 -69.6 -74.1

-80.499 -80.401 -80.3 -72 -78.899 -70.1 -82.101 -77.6 -74.3 -81.698 -69.2 -77.901

-85.401 -80.7 -81.602 -72 N/A

AR_2614_26146_F2_C N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A -85.298 -79.6 -85.002

-83.101 -74 -82.501 -63.9 -85.595 -87.003 -78.2 -82.501 -76.101 -75.2 -72.2 -71.8

-84 -75.4 -77.699 -86.101 -90.697 N/A

CELL3G_WR2614_0 N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A N/A N/A

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A N/A N/A

WR_Gayathi_Industrila_II_U2614_A -94.366 -96.559 -94.676 -92.223 -91.706 -94.218

-91.318 -89.799 -95.768 -92.915 -92.706 N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A

WR_Gayathi_Industrila_II_U2614_B -92.315 -92.915 -95.01 -92.088 -90.697 -90.601

-86.202 -87.702 -95.565 -87.8 -94.076 N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A

N/A N/A N/A N/A N/A N/A N/A N/A N/A

WR_Gayathi_Industrila_II_U2614_C -84.798 -89.696 -94.291 -87.1 -90.508 -87.406

-82.4 -86.295 -91.318 -80.801 -85.696 N/A N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A


WR_Gayathi_Industrila_II_U2614_D -95.565 -94.924 -94.676 -95.979 -94.366 -93.675



WR_Gayathi_Industrila_II_U2614_E -91.393 -85.197 -87.901 -90.297 -93.871 -89.595



WR_Gayathi_Industrila_II_U2614_F -92.915 -94.518 -94.218 -95.01 -94.076 -93.427

-92.41 -92.506 -93.079 -92.809 -95.468 N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A


WR_Gayathi_New_Market_U2496_D N/A N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A N/A

N/A N/A N/A N/A N/A N/A N/A N/A N/A N/AN/A N/A N/A N/A N/A N/A N/A N/A

4. Reason for priority/severity choice: Most of on air sites are suffering from

high uplink frame loss so need Root cause for same.

5. Contact information:

CONFIGURATION:

1. Hardware configuration: 9926 DBS-WCDMA

2. Software configuration: LR13.3.W

IMPACT:

1. Frequency of the problem (once, x times, sporadic, systematic...): Still

Happening

2. Features impacted (iMCTA, CAC, Rate Adaptation, HSxPA over IUR, other...):

Uplink frame loss

3. Impact (call drop, board reset, KPI degradation, redundancy...): Uplink frame

loss

HISTORY:

1. Which & how many NEs impacted: As per customer most of Node Bs has impacted

2. How the issue was detected by customer or local ALU support (via KPIs

degradation, customer complaint...): Local ALU Support

3. When the issue has started: Most of on air sites are suffering from Edch frame

loss, We need to investigate the RC of this KPIs degradation, the issue appears

on any site directly after swap,

4. What actions were performed before or in time when the issue begun (feature

activation, upgrade, 3rd party actions...) and is the issue

still present: N/A

5. What recovery actions have been done & with what results: Suggest RTWP ,

Transmission & Check Synchronization

2 ? LOGS AND TRACES------------------------------ All required logs @

/srv/data202256/data/server/Traces/from_france/ETISALAT_UAE/AR 1-1-5547042

SFTP://172.25.80.47

user : ftraces

password : Elv24trf

Regds

Shelendra Sharma

7. 29-Jan-2015 20:53 shelends

Update to Current Summary: Jan 29 : Escalated this AR, High Frame loss in many

Node Bs, Need Deeper Investigation.


Jan 29 : Escalated this AR, High Frame loss in many Node Bs, Need Deeper

Investigation.

9. 02-Feb-2015 12:33 sunilkd

we need NPO counters as per attached template

10. 04-Feb-2015 14:10 pthota

Update to Current Summary: Feb 4 : Escalated this AR, High Frame loss in many Node

Bs, Need Deeper Investigation.

11. 04-Feb-2015 14:10 ALCATEL-LUCENT PROPRIETARY pthota


Sent: 02 February 2015 12:57






AHMED (AHMED); NABIL GEORGES, MARTIN (MARTIN)** CTR **; MOHAN, Brajendra

(Brajendra)** CTR **



frame loss

Adding GPS

Hi Brajendra

Any Update?

Regds

Shelendra

From: MOHAN, Brajendra (Brajendra)** CTR **

Sent: Monday, February 02, 2015 1:36 PM

To: SHARMA, Shelendra (Shelendra)** CTR **








frame loss

Hello Shelendra,

Need NPO for further analysis .

Regards

Brajendra



To: MOHAN, Brajendra (Brajendra)** CTR **; IBRAHIM ABD EL NABY, Karim (Karim)**

CTR **








frame loss

Hi Bajendra

Please find the attached RTWP related KPI reports & IMT logs.

Aside Karim

Please provide NPO reports.

Regds

Shelendra



To: SHARMA, Shelendra (Shelendra)** CTR **; MOHAN, Brajendra (Brajendra)** CTR **








frame loss

Importance: High

Dear Sharma,Brajenrda,

Please help UREGNLTY.. After I import the report you send to the NPO all old

customer and customized reports are deleted.. Please support urgently.

Thanks,

--

BR,

Karim

M: +971 (0)56-354-4568



To: SHARMA, Shelendra (Shelendra)** CTR **; MOHAN, Brajendra (Brajendra)** CTR **








frame loss

Dears,

Please update.

Thanks,

--

BR,

Karim

M: +971 (0)56-354-4568



To: SHARMA, Shelendra (Shelendra)** CTR **; MOHAN, Brajendra (Brajendra)** CTR **;

DAS, Sunil (Sunil)** CTR **








frame loss

Hello Sunil,

As discussed and after recovering the NPO reports, I cannot see the report you

sent to only under View tap , tell me on which level you need to export? And from

when? And if this file may be corrupted as discussed ,thanks to provide another

one or give another indicator names and I will export,

Thanks,

--

BR,

Karim

M: +971 (0)56-354-4568


Sent: Tuesday, February 03, 2015 3:31 PM

To: SHARMA, Shelendra (Shelendra)** CTR **; MOHAN, Brajendra (Brajendra)** CTR **;

DAS, Sunil (Sunil)** CTR **








frame loss

Dears,

Please Update.

Thanks,

--

BR,

Karim

M: +971 (0)56-354-4568

From: DAS, Sunil (Sunil)** CTR **


To: IBRAHIM ABD EL NABY, Karim (Karim)** CTR **; SHARMA, Shelendra (Shelendra)**

CTR **; MOHAN, Brajendra (Brajendra)** CTR **








frame loss

Can you please import again using add and replace option. This file is used by

many customers and they were able to send data.

Regards,

Sunil



To: DAS, Sunil (Sunil)** CTR **; SHARMA, Shelendra (Shelendra)** CTR **; MOHAN,

Brajendra (Brajendra)** CTR **








frame loss

Hi Sunil,

As discussed, we cannot proceed again with the XML file,

Attached all required KPIs, let me know if anything other required,

Waiting your feedback.

Thanks,

--

BR,

Karim

M: +971 (0)56-354-4568

Time Tracking Entry Added - IR01-TSA1

12. 04-Feb-2015 15:43 sunilkd

Karim,

Thanks for NPO counters. We have analyzed and here are our findings.

1. Radio condition is good. RSSI and RTWP is within limit.

2. There is no sever % of failure on restore indication. So, sync was achieved.

3. I have not seen any sync issue as well.

4. Overall Radio side is good.

I believe, you were referring following counters to access frame lost in network.

Please confirm.

1. VS_EdchFrmWithFrmLossIpIf(U1764)(Event)

2. VS_EdchIubTnlCongestIndc_FrameLoss

Both of counters indicate frame lost on IP network. It is detected based on FSN(

Frame sequence Number). Nodeb sends each packet with FSN. If RNC receives some

packets and does not receive some, it will indicate immediately packet lost in

IUB. This issue indicates frame lost in IP network.

3. We have also seen IPRAN alarm which also indicates IP network congestion or IP

network physically disconnected.

4. There is also PTP packet sync issue. it could be due to delay or PTP packet

lost in the network.

5. There is also HSDPA packet lost on IP network.

All of the above points indicate IP network which needs to be checked.

Nest step.

1. Please arrange IP network diagram. I would like to see nodes where we can trap

the IP counter. Is it MPLS based network?

2. Please collect

VS_EdchFrmWithFrmLossIpIf(U1764)(Event),

VS_EdchIubTnlCongestIndc_FrameLoss at 15 minutes granularity. I would like to

correlate IPRAN alarm and packet lost.

3. Find out, how many nodeBs data are going over one LSP if MPLS is used.

Regards,

Sunil

13. 09-Feb-2015 14:51 ALCATEL-LUCENT PROPRIETARY shelends








17. 10-Feb-2015 09:27 sunilkd

Karim,

Thanks for NPO counters. We have analyzed and here are our findings.

1. Radio condition is good. RSSI and RTWP is within limit.

2. There is no sever % of failure on restore indication. So, sync was achieved.

3. I have not seen any sync issue as well.

4. Overall Radio side is good.

I believe, you were referring following counters to access frame lost in network.

Please confirm.

1. VS_EdchFrmWithFrmLossIpIf(U1764)(Event)

2. VS_EdchIubTnlCongestIndc_FrameLoss

Both of counters indicate frame lost on IP network. It is detected based on FSN(

Frame sequence Number). Nodeb sends each packet with FSN. If RNC receives some

packets and does not receive some, it will indicate immediately packet lost in

IUB. This issue indicates frame lost in IP network.

3. We have also seen IPRAN alarm which also indicates IP network congestion or IP

network physically disconnected.

4. There is also PTP packet sync issue. it could be due to delay or PTP packet

lost in the network.

5. There is also HSDPA packet lost on IP network.

All of the above points indicate IP network which needs to be checked.

Nest step.

1. Please arrange IP network diagram. I would like to see nodes where we can trap

the IP counter. Is it MPLS based network?

2. Please collect

VS_EdchFrmWithFrmLossIpIf(U1764)(Event),

VS_EdchIubTnlCongestIndc_FrameLoss at 15 minutes granularity. I would like to

correlate IPRAN alarm and packet lost.

3. Find out, how many nodeBs data are going over one LSP if MPLS is used.

18. 10-Feb-2015 23:51 pthota

Update to Current Summary: Feb 10 : Investigation on IP link is suggested. This is

transmission media issue. Customer agreed to close the AR.

19. 10-Feb-2015 23:51 ALCATEL-LUCENT PROPRIETARY pthota

From: DAS, Sunil (Sunil)** CTR **

Sent: Tuesday, February 10, 2015 10:21 AM

To: DAS, Sunil (Sunil)** CTR **; IBRAHIM ABD EL NABY, Karim (Karim)** CTR **;

SHARMA, Shelendra (Shelendra)** CTR **; MOHAN, Brajendra (Brajendra)** CTR **





AHMED (AHMED); NABIL GEORGES, MARTIN (MARTIN)** CTR **; SUSARRET, Andres

(Andres)** CTR **



frame loss

Karim,

Please find RCA PPT. Let me know if you need some clarification.

Summary of investigation.

1. The issue lies in transmission media( IUB link).

2. UTRAN is working as expected.

Regards,

Sunil



To: DAS, Sunil (Sunil)** CTR **; SHARMA, Shelendra (Shelendra)** CTR **; MOHAN,

Brajendra (Brajendra)** CTR **





AHMED (AHMED); NABIL GEORGES, MARTIN (MARTIN)** CTR **; SUSARRET, Andres

(Andres)** CTR **



frame loss

Many thanks Sunil,Sharma, You can close the AR.

Thanks,

--

BR,

Karim

M: +971 (0)56-354-4568


Resolution

Investigation on IP link is suggested. This is transmission media issue.

End of Assistance Request 1-5547042

New AR: View AR | AR without Proprietary | OLCS

History | States | Assignment | TimetrackingAR Text Search | More | CARES Home

AR94

Documents

Transcript of AR94