13.03.2018 Page 1 · PDF fileMOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018...

MOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018 Page 1

Minutes of the Special Meeting on URTDSM / SCADA / Communication held on 13.03.2018

1. Introduction

A Special Meeting on URTDSM / SCADA / Communication was held on 13.03.2018 at SR-II,

PGCIL, Bengaluru. The list of participants is enclosed at Annexure-I.

In the 141st OCC meeting held on 09.03.2018, it was noted that there were frequent outage of

ICCP links and the ICCP data was not stable. APTRANSCO and TSTRANSCO had expressed

concern on the operational and commercial implications due to this. There were other

SCADA/Communication issues along-with URTDSM Project Phase-I was nearing

completion and it was agreed to take stock of the situation in a special meeting. The agenda

was taken up for deliberation.

2. URTDSM

2.1 Status of URTDSM project as on 12.03.2018

Region Approved Installed Integrated

SR-I SS 27 25 24

SR-I PMU 113 104 98

SR-II SS 41 40 38

SR-II PMU 142 140 135

Total PMU SR 255 244 233

SR-I - 9 PMU –Chittor (5) and Srikakulam (4) installation works under progress.

SR-I - 5 PMU -Srisailam (4) and Kurnool (1) data validation is not completed.

SR-I - 1 PMU -Simhadri (1) - SRLDC requested SR-1 to co-ordinate with NTPC Simhadri

to complete the pending installation works.

SR-II -2 PMU -Kaiga (2) installation works under progress.

SR-II -5 PMU -Arasur (2) and Kalpakkam (3) data validation is not completed.

2.2 Supply and installation of APS- Auxiliary power supply system as part of

URTDSM project

SRLDC vide letter dated 14-02-2018 (Annexure-II) had requested to SR-II to expedite

the supply Auxiliary Power Supply System.

SR-II had intimated that the Auxiliary Power Supply System is under manufacturing and

is expected to reach site by 20thApril 2018 and commissioning will be completed by 15th

May 2018

AP and TS also expressed concern regarding delay in supply of Auxiliary power supply

system. SR-I confirmed that the APS System is expected to reach site in April 2018 and

commissioning was expected by May 2018.


2.3 Status of SAT and SAVT of URTDSM project in SR

SRLDC - SAT completed and SAVT is yet to commence.

KSEBL - SAT and SAVT completed.

KPTCL - SAT and SAVT completed.

TNEB - SAT completed and SAVT is yet to commence

TSTRANSCO - SAT completed and SAVT has commenced from 1st March 2018

APTRANSCO - Commissioning works under progress expected to be completed by

31st March 2018.

2.4 Powergrid Analytical Application installed as part of URTDSM

SRLDC vide letter dated 02-03-18 (Annexure-III) had informed Smart Grid –

PGCIL the need to improve the performance of installed Analytical Application to

make it useful for real time grid operations

SRLDC requested the forum to consider a procedure on the lines of SAT for verifying

the working of the Analytical Applications.

It was noted that proper feedback could be given for improvement of the Analytical

Application Software. All the control centers were requested to use the software and

report the deficiencies. PGCIL agreed to make a Web group of all the Control

Centers, LD&C, IIT Mumbai, SR-I/SR-II etc for exchanging the information/usage

and for deficiencies / rectification etc.

It was noted that Chief Engineer, NPC vide letter dated 12.02.2018 (Annexure-IV)

had sought certain information on the Analytics and implementation of PMUs from

CTU.

3. SCADA and Communication Issues

3.1 ICCP links failure / not stable

APTRANSCO vide letter dated 26.02.2018 (Annexure-V) had expressed concern on

the ICCP data not being received for long durations due to apparent failure of PGCIL

communication network and redundancy failure at various Network/Server

equipment at SRLDC. Concern was expressed on maintaining the overdrawal /under

drawal without the ICCP data and the huge commercial implications. Besides that

many 765 kV Powergrid’s SS SCADA data is not good and affecting the state CGS

drawals. In view of the above, the following was suggested:

To discuss SCADA/Communication issues of Powergrid SR1 & SR2, states and also

SRLDC’s redundancy readiness of ICCP servers/Routers.

To discuss the issue that the Payments may be linked to availability and healthiness

of Powergrid’s communication network. SRLDC’s ICCP equipment and percentage

availability of ICCP data to the State.

TSTRANSCO vide letter dated 27.02.2018 (Annexure-VI) had expressed concern on

the ICCP data not being stable for most of the period during the month. On 7th & 8th


February 2018, ICCP data failed due to expiry of SSL certificates of secure ICCP.

Again on 23.02.2018, the ICCP data failed from 22:00 hrs and it was restored only at

08:30 hrs of 24.02.2018. The economical load dispatch operation solely lies on ICCP

data in real time including safety and security of the grid. It was pointed out that the

loss of real time data for long time periods affects the CGS drawal severely and

attracts penalties. To avoid such problems in future there was a necessity to discuss

these situations in detail and make a resolution.

SRLDC observation on 07.02.2018 ICCP failure are given below:

Secure ICCP certificates expired throughout SCADA systems maintained by M/s

GE (T & D) in India.

ICCP data exchange unaffected for Karnataka & TN SLDCs (connected on non

secure ICCP)

SRLDC SCADA team tried to resolve issue with known methods

Upon non restoration of ICCP data exchange to/from affected SLDCs, issue was

reported to GE Noida and ticket open for problem resolution by SRLDC.

ICCP data exchange restored in morning hours starting with AP, Kerala,

Telangana & Puducherry.

Data restored in non secure mode.

ICCP certificates were restored by GE on 15th Feb 2018.

SRLDC suggested that affected SLDCs need to raise issue with M/s GE (T&D)

directly for timely resolution and coordinate for early resolution. Further all were

requested to analyse the problem before arriving at any conclusion.

SRLDC observation on 23rd Feb 2018 (22:30 hrs onwards) ICCP failure is given below:

ICCP communication with AP & TS went down.

Multiple fibre cuts in CTU system were reported.

PGCIL was alerted in real time.

Links were restored in morning hours (around 08:30 Hrs), followed by ICCP

restoration.

Subsequently following was taken up jointly by SRLDC & APSLDC/TSSLDC :

AP & TS SCADA team informed SRLDC that in certain situations (ICCP server

2 being main at SRLDC, and ICCP server 1 being main at AP/TS end), ICCP

data is not reporting even when the ICCP links are up.

AP & TS SCADA teams claimed problem due to misconfiguration at SRLDC

servers/routers/firewalls.

FPRs raised by SRLDC for resolution of TS & AP ICCP issues.

Issue resolved for both AP & TS ICCP by attending erroneous entries in AP

firewall & TS routers.

Subsequently following was taken up jointly by SRLDC & APSLDC/TSSLDC for

ICCP Resolution :

AP ICCP Resolution steps:

AP internal firewall mis-configuration rectified.

AP ICCP application error rectified.


AP SISCO security outdated certificate configured with new certificates.

TS ICCP Resolution steps:

Encapsulation PPP enable in TS WR1 router for serial link interface.

Deleted all the static routing from both TS WR routers and TS ICCP servers

which was enforced for SRLDC ICCP LAN.

APTRANSCO expressed concern on frequent ICCP failures recently. This failure was

likely due to communication link and other issues. Due to high CGS schedule, ICCP

data is very important for state. Due to failure of ICCP, AP SLDC control room is not

aware of its OD/UD. After secure license updation one of ICCP server it was not able

to communicate on ICCP protocol. SCADA personnel should be available in shifts to

solve any issue related to SCADA. APTRANSCO requested to waive off penalty for

UD/OD, which happened due to ICCP failure. It was pointed out that many

communication channel has been configured for ICCP data communication between

AP SLDC and SRLDC. M/s Tejas Engineers have configured 6 No of communication

channel with same ID, which creates problem in troubleshooting after link outage.

Many changes have been done in communication network by Powergrid (M/s Tejas),

without information of AP Telecom. Expiry date of all licenses to be taken from M/s

GE. APTRANSCO requested to check the healthiness of server and link redundancy

for ICCP data transfer. Recently complete redundancy test was done for server,

Router, link and found to be OK.

TSTRANSCO expressed similar concern as stated by APTRANSCO. ICCP failure

outage duration on 23.02.2018 was longer, which causes penalty for states due to

OD/UD.

KPTCL informed that ICCP with SRLDC failed on 01.02.2018. On prima facie analysis

it was analyzed that patch updation was the issue. Subsequently on further analysis it

was found that number of points being handled in ICCP database was huge and it had

created problem. Matter was resolved on priority basis with the help of OEM, M/s

ABB Engineers. During above outage real time scada data available on web server

provided by SRLDC helped KPTCL in grid operation.

TANTRANSCO requested to waive off penalty for UD/OD, which happened due to

ICCP failure.

APTRANSCO / TSTRANSCO stressed on 24 hrs shift at SRLDC /SR-I / SR-II for

SCADA and communication issues in similar lines to the SLDCs. SRLDC pointed out

that 24 hrs support was being extended from office and through remote login.

It was confirmed that there were some mis-configuration at AP int-firewall. It was

rectified by M/s GE team upon ticket raised for rectification.

It was confirmed that there were some mis-configuration at TS ICCP router. The same

was attended by M/s GE Engineers on FPR raised by SRLDC.

It was noted that ICCP failure on 12.03.2018 was due to power supply interruption of

equipment at Hoody substation of KPTCL.

SRLDC stated that data is extremely important for reliable and secure grid operation

and all the concerned entities needed to resolve the issue at their end and also co-

ordinate with other entities.


SRLDC suggested AP & TS to actively raise all SCADA/communication related

complain to GE/CTU/STU and not leaving the matter only to SRLDC for resolution.

SR-I & SR-II pointed out that all ICCP link failures may not be due to communication.

ICCP problem occurred on 23.02.2018 was not due to fibre cut in Munirabad/Raichur

area.

After discussion the following was agreed:

Expiry date of all licenses would be furnished by M/s GE to the respective Control

Centers. It was noted that respective Control Center would ensure that M/s GE take

pro-active action to renew the licenses before the expiry. As pointed out by KPTCL

that there should not be any license renewal issues during the project cycle of the

project and it needed to be ensured by Control Centers (through M/s GE).

License for external firewall is going to expire in three months. M/s GE has to

update the certificates in ext firewall of AP. M/s GE to take necessary action.

M/s GE site engineer at respective Control Centers to escalate the matter to BO-

Noida, if they are unable the resolve the issue at their level. Further, inter-action

between M/s GE site engineers at SLDCs and M/s GE site engineers at SRLDC

needs to be strengthened.

Changeover to ICCP data communication through Backup SRLDC could be

explored during such conditions which would reduce data outage duration.

All the ICCP links (Main/Back up CC of SLDCs to Main/Back up CC of SRLDC) to

be checked daily to ensure healthiness.

All SLDCs to carry out HA test once in every month to a check

SCADA/Communication system healthiness.

All states to actively raise all SCADA/communication related complain to M/s

GE/CTU (M/s Tejas) /STU and not leaving the matter only to SRLDC for

resolution.

It was noted there was no provision for DSM waiver during ICCP failure as per

extant Regulations.

Updated communication network diagram to be submitted by SR-I and SR-II to all

SLDCs & SRLDC. Further any changes done in the communication network needed

to be informed to all.

3.2 Frequent data outage (Central Sector & State Sector)

SRLDC presented the detail of data outage from 01.01.2018 to 12.03.2018

(Annexure-VII). It was pointed out that most of the outage was due to

communication failure.

SRLDC requested Powergrid to provide all fibre cut/equipment outage information in

real time.

TSTRANSCO expressed concern that due to frequent failure of Powergrid station

(Nizamabad, Warangal etc.), TS drawl gets affected and real time CGS drawl becomes

erratic.

SR-I & SR-II stated that all data outages may not be due to communication issues.


SR-I & SR-II stated that DCPC healthiness also needed to be checked. For this SRLDC

replied that all DCPC patches etc. has been done and at present one of DCPC at

Narendra is having the problem, which is being looked by M/s GE for faster resolution

After deliberations the following was agreed:

PGCIL to share their observations for communication failure on the instances of

data outage beyond 8 hrs (refer annexure) put up by SRLDC.

PGCIL to share all fibre cut/equipment outage information in real time.

It was suggested to take all measure to minimize down time of data

3.3 KKNPP analog data not reporting

It was noted that presently Analog data issue has been resolved after Analog Card (as

supplied by PGCIL SR2) replacement and is under observation. The forum opined

that if the problem persists, it would be prudent for KKNPP to procure a new RTU.

3.4 Data outage after commissioning

In earlier meeting it was agreed to cross check with site daily values for newly

commissioned elements for a period of two week duration from commissioning.

3.5 Data discrepancy

The latest data discrepancy list compiled by SRLDC was presented in the forum

(Annexure-VIII).

The forum opined to delete the points which are not feasible to be telemetered.

SRLDC agreed to look into the issue.

3.5 RTU/SAS Maintenance/Upgradation

All states/Powergrid/CGS to explore option of replacement of old RTUs and AMC for

SAS, as getting spare is difficult for old RTU. The issue has already been discussed

earlier and AMC need to be provided for all RTU/SAS gateways by CTU/STU.

A life/functional assessment report and action plan by PGCIL/ISGS on old RTUs

would be prepared and submitted to SRLDC/SRPC.

3.6 Backup Control Centre

Backup SCADA functionality testing frequency needs to be done once in 6

months to ensure high availability of data/preparedness to operate from Back up CC

in event of any contingency. Compliance report may be furnished to SRLDC.

Periodical testing was being carried out by SRLDC and KSEBL. TN stated that

communication network is being strengthened to Backup CC. Once same is

completed, TN will do the testing.

APTRANSCO informed that communication link has to be established from AP CC at

Vijaywada to Back Up CC of SRLDC at Delhi. Powergrid & AP confirmed to establish

the same by 16.03.2018.


PGCIL would be floating new combined Open Tender for Backup SLDCs for AP

(Tirupati), TS (Warangal) along-with few other states.

After discussions, it was noted that backup control centre establishment for AP & TS

need to be expedited.

3.7 ICCP with NTAMC

PGCIL confirmed that as pointed out in 32nd TCC/33rd SRPC meeting, due to cyber

security concerns data would not be extended to SRLDC from NTAMC ICCP.

It was noted that functional requirement of the requisite data by SRLDC would

remain, which needed to be ensured by PGCIL. SRLDC would furnish the list of

telemetry points that have to be extended to PGCIL within 10 days, and within 1

month PGCIL would come out with a detailed plan to be completed within 6 months

(October 2018)

3.8 Modification in EMS package – TSTRANSCO

Earlier TSTRANSCO informed that they had received estimate of Rs 47 lakhs and the

contract was being firmed up.

TSTRANSCO requested PGCIL to assist in finalizing the EMS contract through M/s

GE.

SR-I stated that TSTRANSCO to interact directly with M/s GE without PGCIL

involvement.

SRPC requested that PGCIL to render assistance to TSTRANSCO if possible.

3.9 Status review of compliance of approved protocol of handling scada & communication issues (21.03.2016 communication meeting)

In the earlier meeting, it had been noted that RTU not reporting/data failure would be

communicated to the concerned entity/user with a copy to PGCIL. User would take

lead role in getting the issue resolved. User/SRLDC would ensure necessary

verification at their ends as explained in the training. Parallely, PGCIL would look into

communication issues. User/SRLDC would appraise PGCIL about the verification at

their end for PGCIL to sort out the communication issues. Official email ID for

SCADA issues to be created for each entity/User involved for easier communication

and proper delivery of information.

SRLDC pointed out that there were instances of delay in attending the faults wrt the

time schedule and requested all the constituents to attend to the faults within the

agreed time schedules.

It was noted that if the issue is not getting resolved within 8 hrs (3 hrs for ICCP) the

following action would be taken:

Shri A P Gangadharan, GM(AM), SR-II and Shri S Ravi, GM(AM), SR-I would be

informed for necessary follow up.

The issue would be escalated with M/s G E (both by SLDCs and SRLDC) and with

M/s Tejas (by SR-I & SR-II).


In case the issue was as not resolved within 24 hrs physical involvement of OEM

(M/s GE and M/s Tejas) would be ensured through effective coordination by user

/ PGCIL/ SRLDC.

Sustained efforts/involvement of all the concerned entities to be ensured till the

issue is resolved

In case of hardware replacement / major rectification it would be completed by

72 hrs.

3.10 Sparing of Fibre in Central Sector Project

TANTRANSCO again requested for sparing of fibre in Central Sector Project.

It was noted that the issue was already deliberated in the 30th TCC/31st SRPC

meetings held on 24/25.02.2017 wherein, the following was noted:

PGCIL had informed that in most of the FO, there were no spare fibres available. Six

fibres are available for grid management purposes. The balance 18 fibres in the state

or central sector are left to the entities on how to utilize. As for Central sector, 18

fibres are being leased out to M/s Powertel as per CERC formula which covered

issues like sparing of fibres, cost sharing (50% being shared by M/s Powertel), ROW

compensation (Rs.3000/km) etc. For grid operation purpose, bandwidth on 6 fibres

could be given to the states. M/s Powertel may not take all the 18 fibres in some of

the lines but those fibres are being leased out anywhere.

Keeping in view of the non availability of spare fibres in most of FO links and the

approved MoU with M/s Powertel the scheme of allocation the fibres was dropped.

SR-II stated that request for sparing of fibre in new FO links could be communicated

for consideration.

3.11 VC issues

TSTRANSCO pointed out that Video Conference issue at TSSLDC. There was

disturbance in Video Conference at TS end when connected to SRLDC.

SRLDC confirmed as per AMC Engineer no issue was observed at SRLDC end and

software upgradation is pending at TS end.

After deliberations, TSSLDC and SRLDC were requested to coordinate and sort out

the issue.

3.12 Talcher RTU

It was noted that earlier proposed meeting was not held due to overlapping holidays.

It was noted that a meeting with NTPC Talcher, SR2 ULDC and SRLDC for the same

in is proposed 3rd/4th week of March 2018. SR-II was kindly requested to address the

communication issue between Talcher-II & HVDC. NTPC as a User needed to be

proactive in this.

4. SE (O), SRPC thanked all for the interactive deliberations and requested the Entities for

concerted/sustained action to close the issues highlighted in the Meeting.

13.03.2018 Page 1 · PDF fileMOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018...

Documents

Transcript of 13.03.2018 Page 1 · PDF fileMOM of URTDSM – SCADA - COMMUNICATION – 13.03.2018...