Using Cellular Machine-to -Machine Solutions for …...Using Cellular Machine-to -Machine Solutions...

Using Cellular Machine-to-Machine Solutions for Collection System SCADA:

Is a Small Data Plan Big Enough?

Dan Cote, PE, and Tim Baldwin, PE McKim & Creed

Outline Virginia Beach SCADA SYSTEM Telemetry Communications Modernization Project Proof of Concept Test Machine-to-Machine Data Plans DNP3 Protocol Cellular Ethernet Communications IT Issues Lessons Learned Conclusions

Virginia Beach SCADA System SCADA has been in operation in Virginia Beach for many years Water and Sewer SCADA systems were combined into one SCADA

application in 2008

SCADA Servers CitectSCADA

Water System Front-End

Water System SCADA 33 Facilities include Water Supply, Storage, Pumping and Monitoring

Water System SCADA Serial BSAP Communications on Licensed UHF and Spread Spectrum Radios

Bristol 3300 RTUs no longer available.

Toshiba EX40s are now obsolete.

Sewer System SCADA 406 Sewage Pumping Stations Including 6 Vacuum Stations

Sewer System SCADA Serial Modbus Communications on Licensed UHF and Spread Spectrum Radios

Modicon Micro PLC is now obsolete

Serial Radios Approaching Limit of Data Capacity

Telemetry Communications Modernization Project • System and Goals Assessment Highlights

• Flow Monitoring to meet Regional Standards and Consent Order. • PLC/RTU Standards for Equipment Replacement • Communications Infrastructure to meet Future Needs.

• Preliminary Engineering Report Highlights • Proposed Protocol: DNP3 • Proposed PLC Hardware: Top 3 PLCs Selected • Proposed Communications Infrastructure: Cellular M2M Ethernet

• Proof of Concept Testing • Evaluate Protocol: DNP3 • Evaluate Top 3 PLCs: Hands-on-Experience • Evaluate Cellular M2M Ethernet: Actual Data Usage Testing

Proof of Concept Test Install 3 PLCs at the same pump station to maintain identical

conditions Each PLC has its own cellular data plan for data usage comparison

SCADAPack 334E

Schneider M340

Kingfisher G30

Machine-to-Machine Machine-to-Machine Data Plans are widely used

• Plans are available as low as $1, $5, or $10 per month

Machine-to-Machine One objection to Cellular Data is the recurring cost: A 5 GB / Unlimited plan typically costs $45/month. Unattractive for

SCADA. M2M plans in the range of $10/month are attractive, compared to

building and maintaining a private SCADA radio system. The M2M plans are very small… too small?

5 GB

Conventional Tier 2 M2M

250 MB

Tier 1 M2M

25 MB

Serial UHF

12.8 MB X 2 = X 10 =

X 20 =

Data Available Per Month at Each Station

Machine-to-Machine One objection to Cellular Data is the recurring cost:

• A 5 GB / Unlimited plan typically costs $45/month. Unattractive for SCADA.

• M2M plans in the range of $10/month are attractive, compared to building and maintaining a private SCADA radio system.

The M2M plans are very small… too small?

5 GB

Conventional Tier 2 M2M

250 MB

Tier 1 M2M

25 MB

Serial UHF

12.8 MB X 2 = X 10 =

X 20 =

Data Available Per Month at Each Station

Why DNP3 Protocol? Open Industry Standard

• Not proprietary to one equipment supplier Versatility

• Highly configurable to meet your needs Efficient use of Band Width

• M2M data plans Automatic Backfill of Data

• Never lose your data Compatibility with HMI

• No third-party software

DNP3 Terminology Terminology:

EVENT: A single measurement, reading, calculation or event which is ‘Time-Tagged’ and sent to the BUFFER and later to the MASTER

Pressure 24.5 PSI 8:15 AM

Pump 1 ON

3:21 AM

Starts/Hr 6

4:00 PM

BUFFER

Holds up to 20,000 Events

PLC SCADA

DNP3 Protocol Terminology:

REQUEST EVENT DATA: Ask PLC to send event data. Called an “Event Poll”


Pump 1 ON

3:21 AM

Starts/Hr 6

4:00 PM

BUFFER


PLC SCADA Request Event Data

Every 2 minutes

Or

Once/Hour?


RESPONSE: PLC responds by sending the stored data.


Pump 1 ON

3:21 AM

Starts/Hr 6

4:00 PM

BUFFER



Response

Event Trends


CONFIRM: Once data is correctly received, a confirmation is sent.


Pump 1 ON

3:21 AM

Starts/Hr 6

4:00 PM

BUFFER



Response

Event Trends Confirm


Clear Buffer: Data has been sent. Does not need to be sent again.


Pump 1 ON

3:21 AM

Starts/Hr 6

4:00 PM

BUFFER



Response

Event Trends Confirm

Event Trend: Automatic Backfill

Event Poll

Event Poll

40 seconds later

40 seconds later

40 seconds later

Figure 2

Figure 3

Figure 4

Figure 5

Figure 1

Status changes of Pumps or Valves:

150 events per day

Event Logging: Pump Running

TIME BASED RECORDING

Actual Time and Date recorded

Pump #1 Pump #2

Store an Event once every 30 seconds: 2,880 events per day

Event Logging: Time vs Dead Band

EXAMPLE SIGNAL

TIME BASED RECORDING

1,100 Events Per Day

450 Events Per Day

Event Logging: Log by Dead Band

1 Foot Deadband

2 Foot Deadband

260 Events Per Day

Event Logging: Smart Dead Band

“SMART” Dead Band Algorithm

Create Time-Stamped Generated Events for the Past

Using DNP3 in Ethernet Ethernet Sends Data in Packets. Each packet contains ‘Overhead’ for addressing, routing, security etc.

67 Bytes - Overhead Request Events 6 Bytes - Message

Response

Confirm 6 Bytes - Message

PLC/RTU SCADA

Using DNP3 in Ethernet Ethernet Sends Data in Packets. Each packet contains ‘Overhead’, for addressing, routing, security etc.


Response 13 Bytes – Each Event 71 Bytes - Overhead


67 Bytes - Overhead

PLC/RTU SCADA


Response

PLC/RTU SCADA

1 event / 2 minutes: 5 MB/Month 11 % Effic


100 event / 2 minutes: 37 MB/Month 79% Effic



13 Bytes – Each Event 71 Bytes - Overhead


67 Bytes - Overhead


Response

PLC/RTU SCADA



100 event / 2 minutes: 37 MB/Month 79% Effic



13 Bytes – Each Event 71 Bytes - Overhead

Confirm 6 Bytes - Message 67 Bytes - Overhead

How Many Events Do I Need? Description ModBus (2 minute poll) DNP3 (adjustable) Wetwell Level 720 720 Pressure 720 2880 (30 seconds) Pump 1 Running 720 75 (actual start/stop time tagged) Pump 2 Running 720 75 Pump 1 Hourly RT 720 24 Pump 2 Hourly RT 720 24 Pump 1 Hourly Starts 720 24 Pump 2 Hourly Starts 720 24 Hourly InFlow 720 24 Hourly OutFlow 720 24 Intrusion 720 5 Other Possible Alarms 720 x 20 3 (only when they occur) Battery Voltage 720 20 (on deadband) Discharge Pressure 720 1,200 (on deadband) Rainfall 720 13 (only when changing) Subtotal 24,480 events 5,135 events/day FUTURE TAGS Pump Amperage 3,700 (every 10 seconds when running) Pump Voltage 600 (on deadband) Power Usage per Hour 24 Max Voltage per hour 24 Min Voltage per hour 24 Water System Pressure Monitor 8,640 (every 10 seconds) Total: 18,147 events/day

Ideal Communications “We were expecting:”

First Real Test Results

First Real Test Results:

We were expecting:

Initial Test Results

Initial Test Results:

Ideal Communications:

Wire Shark


Opportunity for finger Pointing:

Ideal Communications:


Initial Test Results:

Ideal Communications: 51% Efficiency

7% Efficiency

1 2 3 4 SECONDS

SECO

NDS

1

2

3

This is how TCP guarantees packet delivery:

Lessons Learned: Ethernet TCP

RTO = Response Timeout

WHAT IS CAUSING THESE TCP RETRANSMISSIONS??

RTO Doubled TCP Retransmission

TCP: Transmission Control Protocol

1 2 3 4 121 125 241 245 SECONDS

SECO

NDS

1

2

3

REQUEST:RESPONSE Note: First Request after delay has 2.5 seconds latency, because of cellular network.


Round-Trip Timeout (RTO) controls TCP Retransmissions. RTO adjusts automatically:

Ethernet TCP automatic Return Trip Time control: Lessons Learned: Ethernet TCP

1 2 3 4 121 125 241 245

SECO

NDS

1

2

3

SECONDS

SECONDS

When latency exceeds RTO: • TCP Retransmission is sent. • RTO is doubled.


1 2 3 4 121 125 241 245

SECO

NDS

1

2

3

SECONDS

Green Line is Round-Trip Timeout (RTO) RTO adjusts automatically again.


1 2 3 4 121 125 241 245

SECO

NDS

1

2

3

SECONDS

TCP Retransmissions will be sent with every event poll.


1 2 3 4 121 125 241 245

SECO

NDS

1

2

3


TCP does not work well with Cellular M2M in SCADA applications.

Answer: Use UDP instead. Since UDP does not guarantee packet delivery, the DNP3 protocol must be able to control communications quality.

Conclusions

M2M Data plans can be cost effective, but efficient operation requires careful attention to details.

A versatile protocol such as DNP3 is needed for best results.

TCP does not work well with M2M SCADA. PLC equipment and protocol must be able to guarantee data delivery using UDP.

Questions

Using Cellular Machine-to -Machine Solutions for …...Using Cellular Machine-to -Machine Solutions...

Documents

Transcript of Using Cellular Machine-to -Machine Solutions for …...Using Cellular Machine-to -Machine Solutions...