C I R E D 20th International Conference on Electricity Distribution Prague, 8-11 June 2009

Paper 0242

CIRED2009 Session 3 Paper No 0242

TTHHEE MMIIGGRRAATTIIOONN OOFF SSEERRIIAALL TTOO EETTHHEERRNNEETT CCOOMMMMUUNNIICCAATTIIOONNSS –– WWHHYY BBOOTTHHEERR??

René MIDENCE Roger MOORE Glenn ALLEN RuggedCom Inc. - Canada RuggedCom Inc. - Canada PowerStream Inc. - Canada ReneMidence@ruggedcom.com RogerMoore@ruggedcom.com glenn.allen@powerstream.ca

ABSTRACT Serial communications has been the mainstay for communication systems for more than a decade using RS232 and RS485 as the physical layer. However, the Ethernet juggernaut is enticing systems engineers to re-evaluate their existing approach to telemetry with the potential benefits of a bigger 'pipe'. Fortunately, familiar protocols like Modbus and DNP support TCP/IP over Ethernet making the switch relatively easy for greenfield deployments. Retrofitting Ethernet and newer IEDs into existing communications systems including SCADA require supporting a hybrid network of both Ethernet and serial devices. One approach to a hybrid network is serial tunneling through serial servers as described below and includes an example illustrating the benefits of a real implementation.

INTRODUCTION [1] Today Ethernet is the predominant and most popular networking technology used in office and home environments. Since Ethernet networks are inexpensive and fairly well understood, they have quickly becoming popular in industrial and utility data communication networks, despite the fact that they were not originally developed to operate in hostile environments.

Ethernet infrastructure is usually available, or can be easily implemented. Buildings tend to have existing Ethernet networks. PLC and RTU manufacturers are starting to develop Ethernet add-ons to network their products, at a premium cost for this functionality.

Both industrial and utility networking experts are moving forward accepting the limitations of Ethernet networks and solving the problems associated with its application. Now days, new Ethernet equipment has becoming available designed to operate reliably under extreme harsh environment typically found in utility applications for SCADA, substation and distribution automation.

The use of serial servers enables legacy serial devices to migrate, with minimum investment, to Ethernet high speed communications replacing costly intermediate equipment (i.e. PLCs and RTUs) between serial IEDs and the company enterprise network. A transparent serial tunnel is created over Ethernet without changing much of the existing setup. The ability to create Virtual COM Ports eliminates the need

to upgrade the existing hardware with network add-on modules.

The migration to Ethernet not only improves communications, it also creates opportunities for other company stake holders to have access to critical information.

COMPARING SERIAL WITH ETHERNET COMMUNICATIONS Comparing the characteristics of Serial vs. Ethernet communications, the following are the most important reasons when considering one versus the other: Reliability of Data Communication

The Ethernet is a secure and proven way of data communication with error correction handled by the hardware and the TCP world standard communication protocol. The primary limiting factor for shared-bus topologies like serial links is that of high I/O and limited distance.

Speed of Data communication Using Ethernet data is moving with a minimum speed of 1 Mbps with absolute error correction compared to RS232 where the speed is at best one tenth of the Ethernet’s minimum, without or minimum error correction.

Ethernet Better and Faster Recovery Ethernet networks utilize redundant links and recovery protocols as protection techniques against link failures.

Standardized Cable and Connection For the Ethernet network the engineer is using world class unshielded twisted pair (UTP) cables and plugs however in the serial world there are no cable standards.

Distance of Data Communication and Noise immunity With Ethernet technology the data can be error free at high speed of 1mbps minimum over distances up to 50m. When using the RS232 for speeds of 110kbps the theoretical distance for error free communication is relatively very short. This means that in the majority of cases the distance alone between the IED and the PC is a key factor to choose Ethernet over serial communications. Furthermore in noisy environments the Ethernet cable is proven to work while the RS232

C I R E D 20th International Conference on Electricity Distribution Prague, 8-11 June 2009

Paper 0242

CIRED2009 Session 3 Paper No 0242

or RS485 non standard cables offer no guarantee that may corrupt data.

SERIAL TO ETHERNET MIGRATION EXAMPLE The following example describes the application of serial servers in a Distribution Automation System in which the owner, PowerStream Inc. a Distribution Utility in the province of Ontario, Canada, has successfully improved the reliability and availability of the communications network by upgrading it from serial communications via fiber optic modems to Ethernet communications by means serial servers.

[2] PowerStream Inc. is the third largest local electricity distribution company in Ontario. The company delivers power to more than 234,000 residential and business customers in the municipalities of Aurora, Markham, Richmond Hill and Vaughan. This particular case deals with the application of Serial Servers as part of the communications network for a new SCADA system in the city of Markham.

The SCADA System and Distribution Automation PowerStream Inc. has implemented Distribution Automation Systems in the Cities of Markham, Vaughan, Richmond Hill and Aurora. In 2008 PowerStream Inc. commissioned a single new SCADA system that controls all four municipalities from a brand new control room in the company’s Head Quarters located in the City of Vaughan.

The SCADA system communications run on the corporate network with all other corporate applications. It collects data from RTUs located at distribution substations and at key locations within the distribution grid, via Ethernet on PowerStream’s OC-12 SONET backbone. Fiber communication with pole top mounted RTUs was done via serial data fiber optic modems connected in a ring topology.

Original SCADA System The architecture of the original SCADA Master that was put in service in the city of Markham is shown in Figure 1. The SCADA Master is a Telegyr dual redundant system composed of two SCADA Host Computers, SCADA Host 1 and 2 as shown. Communications with all RTUs was via a

Figure 2 SCADA RTU Fiber Optic Communications - Block Diagram Before

Figure 1 Comm 13 Fiber Ring - Telegyr RTUs

C I R E D 20th International Conference on Electricity Distribution Prague, 8-11 June 2009

Paper 0242

CIRED2009 Session 3 Paper No 0242

serial port connected to a device called a “Digi Port”. Each of the two Digi Ports has 15 addressable ports. Each of the 15 ports can be programmed to communicate at different baud rates and protocols.

The fiber optic modems at the SCADA Master communicate with fiber optic modems in each of the RTUs. The fiber running between the Master Station and the RTUs is connected in a ring topology, as shown in Figure 2.

Communications network between the SCADA Master and the Distribution Substation RTUs is via DNP 3.0 over Ethernet. However, the SCADA Master was not able to communicate via DNP 3.0 over Ethernet, but it was able to communicate using DNP 3.0 via serial port. For this reason four terminal servers were used to bridge between the Ethernet and serial networks.

Reliability and Availability Issues with the Communication System The communications equipment that was chosen when the system was first installed was based on the best technology available at that time. Single Mode Fiber Optic Modems where selected which allowed PowerStream Inc. to interconnect Pole Top Mounted RTUs using serial communications via single mode fiber.

While it was the best engineering concept it lacked 100% availability due to the limitations imposed by the technology of that vintage. Not even fiber optic redundancy would ensure that the system would be available at all times.

PowerStream Inc. considered replacing some of the system components for the following reasons:

The rupture of a fiber link could potentially disconnect more than one RTU.

When a portion of the link was damaged, the location of the damaged section was not easy and time consuming, therefore the restoration time was undesirable.

Access to information by other stake holders i.e.

Engineering and Planning was limited

New SCADA System with Serial to Ethernet Servers In order to improve the availability of the communications system and to speed up the recovery when a fiber was damaged, PowerStream Inc. decided to replace the Serial Fiber Optic Modems with new technologies that would meet the following engineering criteria:

Pole top mounted RTUs and SCADA system were designed to communicate via serial communications only

Communications media is Fiber Optic – not a challenge but a requirement

Need to increase productivity when locating fiber optic failures

Communications protocols were serial protocols Need to interface with the corporate network to allow

others to have access to vital information for system planning, operations and maintenance, and troubleshooting.

Serial communications have restrictions on distances between devices thus limiting future expansions

Most of serial protocols use a client/server relation for communications instead of sharing the media between multiple devices as with Ethernet.

Serial protocols can not be carried over IP networks as raw serial data without the help of auxiliary devices to convert serial protocols to serial over IP protocols.

The ideal solution was found in devices known as serial servers, which allowed connecting serial devices that did not have Ethernet capabilities to Ethernet Networks,

PowerStream Inc. replaced the Serial Fiber Optic Modems with Serial to Ethernet Servers which provided a solution that met the technical requirements listed above. In addition, it also reduced the number of devices in the system since they replaced the through barrels, the Dymec fiber modems and the EIA 232 interfaces, hence increasing the reliability of the system. Figure 3 shows the new topology of the fiber optic ring between the Master Station and the pole top

Figure 3 Comm 5 Fiber Ring - Harris RTUs

C I R E D 20th International Conference on Electricity Distribution Prague, 8-11 June 2009

Paper 0242

CIRED2009 Session 3 Paper No 0242

Harris RTUs.

The DNP 3.0 double ring can be broken into two single rings that can be accessed separately by master station Modem A and master station Modem B maintaining the original design criteria. The ring can be broken at any point so that a defective length of fiber can be isolated without loss of communications to any RTU.

Each Serial Server is fitted with means to send an alarm if one of the fiber optic links is broken. In the event of a fiber optic failure, the adjacent Serial Servers will issue an alarm via the corresponding RTUs. Receiving alarm signals from two RTUs located at each end of the faulted fiber optic link reduces the search for the failed fiber to the portion of the network between them. The architecture of the new SCADA Master fiber communications to the DNP3.0 network is shown in Figure 4.

Recently the Telegyr SCADA Masters were upgraded to support DNP 3 over Ethernet. The RS400 that was installed to replace the Dymec Fiber Optic modems was placed on a SCADA LAN along with SCADA Host 1 & 2.

The Harris DNP 3.0 RTU data no longer goes through the Port Combiner or the Digi Ports. With the upgrade to the Telegyr Masters, Communication port 5 – DNP 3.0 protocol is now communicating via Ethernet with the GE Harris pole top RTUs through the RuggedCom RS400 Serial Server and a RuggedCom RS2100 Ethernet Switch.

The data is also placed on a 100MB Ethernet LAN on the PowerStream SONET ring allowing control to the Harris RTUs also from the SCADA master at the new backup control room.

LESSONS LEARNED Over the years, PowerStream has gained valuable experience while implementing their Distribution Automation system. The most important lessons learned with the current system regarding serial communications are as follows, which are the main reasons why PowerStream migrated to the Ethernet solutions:

Serial communication is not recommended when communicating with RTUs or PLCs over a dedicated fiber optic pair.

A broken link due to fiber optic damage means communication interruption beyond the point of rupture. PowerStream did not anticipate that the fiber optic cables would be damaged / broken as often as they were.

Due to the nature of serial communications, there was limited access to information by other groups within PowerStream.

As discussed above, serial communications over fiber optics using fiber optic modems was the best technology available

during the design and installation of the system.

BIBLIOGRAPHY [1] Roger Moore, “IEC 61850 Substation LAN - Issues to Consider when Designing and Deploying the Ethernet Network”, presented at CIGRE Canada 2007, August 2007

[2] Roger Moore, René Midence, Glenn Allen Ph. D., “Migrating from Serial to Ethernet Communications”, presented at DistribuTECH 2008, January 2008

Figure 4 SCADA RTU Fiber Optic Communications - Final Block Diagram