CENTRAL CONTROL FACILITY (CCF) DESIGN AND LATEST DATA CENTER PRACTICES Obrad Aleksic, Systems Specialist, Hatch Mott MacDonald, 2800 Speakman Drive, Mississauga, Canada, 905 403 3923, obrad.aleksic@hatchmott.com Craig Smith, Sr. Systems Specialist, Hatch Mott MacDonald, 5035 South Service Road, Burlington, Canada, 647 361 6081, craig.smith@hatchmott.com Djoko Corovic, PE, Senior Project Engineer, Hatch Mott MacDonald, 2800 Speakman Drive, Mississauga, Canada, 905 403 4147, djoko.corovic@hatchmott.com Number of Words: 2903

ABSTRACT Central Control Facility (CCF) typically serves both the operational and security requirements of a transit operator. It is a mission critical center that is used for post-event analysis and for operations planning while serving as a main data repository. The volumes of data generated by a modern CCF continue to increase exponentially, while very high availability becomes a critical element of the CCF’s Data Center. Today, the CCF should be designed, at a minimum, to accommodate a Class F2 Data Center. Additional performance expectations include modularization, interface management and access control that are all based on a common network infrastructure, shared resources and virtualization. This paper analyzes the configuration and connectivity issues in a typical CCF implementation for a municipal LRT infrastructure project, particularly in regards to data collection, storage and networking. It is important to maintain high data transfer rates between the main and backup core switches to ensure that the overall performance for application servers is not affected. Control room location options (local or remote to CCF) are discussed in terms of networking topology and switching requirements. Specific measures for data protection are further evaluated for surveillance applications given their high data usage and network speed requirements. This paper also discusses general data collection from the field in terms of networking efficiency and the redundancy of links to the CCF. A layered network configuration in the field increases data throughput, availability and general management of resources. CCF external link requirements are considered in terms of controlled and managed access that will be provided to and from different municipal services. INTRODUCTION A Central Control Facility (CCF) serves operational and security requirements for both the yard and main line in a typical LRT transit infrastructure. The CCF becomes a mission critical center used for normal and emergency operations (a command center) but also for post-event analysis and operations planning and training. In all cases the CCF is the main data repository node to keep, manage and distribute all data collected. The volumes of data generated by a modern CCF continue to increase exponentially while its high availability becomes a critical parameter to design CCF. By today’s practices, the CCF should comply with Class F2 Data Center design as minimum. Additional performance expectations include modularization, interface management and access control that are all based on a common network infrastructure, shared resources and virtualization. This paper analyzes the configuration and connectivity issues in a typical CCF implementation for a municipal LRT infrastructure project. It particularly focuses in regards to networking, access control, data collection and storage. Availability of key systems requires redundancy and high data transfer rates between the main and backup control nodes. Redundant core switches are often used to ensure that the overall performance for application servers is not affected.

© AREMA 2014 1

In terms of networking topology and switching requirements, additional considerations takes into account the corresponding control room position, local within the CCF or remote from it. Specific measures for data protection are further evaluated for surveillance applications given their high data rate requirements. This paper also discusses general data collection from the field in terms of networking efficiency and the redundancy of links to the CCF. A layered network configuration in the field increases data throughput, availability and general management of resources. External CCF link requirements are considered in terms of controlled and managed access that will be provided to and from different municipal services.

3-LAYER HIERARCHICAL TOPOLOGY

This paper presents solutions based on the latest IT Industry trends, a hierarchical topology of a segmented communication backbone consisting of Access, Distribution and Core layers particularly favorable for longitudinal alignment types (Figure 1) as well as legacy ring-based networking topologies. The access layer typically connects field IP devices (e.g., at stations, stop, tunnels, bridges, substations, etc.) along the corridor to its closest distribution nodes. Distribution nodes are strategically located along the corridor, most commonly in at selected TPSS locations, with a purpose to ensure access link aggregations and services optimization.

Figure 1: Typical Communication Backbone Infrastructure At the top of the networking hierarchy, core nodes are set for terminations of all the distribution connections from the field side. In the same time, core nodes provide internal links to the application servers and control room operations at CCF and external links to the 3rd parties such as police, emergency services, other transport utilities, etc. The complex and critical roles of core nodes at the CCF requires high quality and capacity switches that are fully managed and protected. Increased roles in data collection, security and transit operational safety and availability, make CCF redundancy and high data exchange standard requirements. LRT infrastructure is typically developed in phased implementations, changed requirements and impacts from fast-changing technologies. As a result, communications performance and networking topologies need to be adaptable and expandable.

© AREMA 2014 2

MODERN CCF DESIGNED AS A DATA CENTER

The modern CCF is the main repository and management points for all project data and communications. While a control room with operators may or not be a part of the CCF, a typical Data Center design guidelines apply. All backbone and external links, data processing and archival are essential parts of a data Center serving the CCF. A typically redundant configuration of data centers ensures continuous operations under all circumstances. Linear and stretched alignments that are typical for LRT infrastructures affect systems configuration strategies. Yard facilities located at opposite alignment ends are likely to house the main and backup data centers. Data center nodes are then typically mirrored for all data collection, archrival, and application servers. Each acts as a network hub with a core switch at the top. Two data centers than allow for the full redundancy and include dedicated core-to-core links. Figure 2 shows typical data center components within CCF including: 1. Core Switch – Collecting data from distribution nodes at strategic locations in the field and forwarding to

designated servers, plus managing redundancy between the main and backup data centers. 2. Application Servers – Preferably shared and virtualized servers to optimise resources and improve maintenance

efficiency. 3. Data Storage - Appliances enabling to optimize backup strategies for archival, preservation and access. Due to

the nature of CCTV files (size and growth rate), surveillance data is recommended to be archived separately. 4. Local Switch – A high-speed capable units (min 10 GBPS) to interconnect all data center components.

Redundant units with universal ports are recommended able to accept both optical and copper connections. Introduction of those switches follow the major manufacturers’ design recommendation

5. Firewall – A standard appliance to protect and isolate data center from external, potentially unsafe, environments. Redundancy and careful setup are required to maintain operational and production traffic while protecting inside networks from malicious access.

Figure 2: Data Center Main Components

© AREMA 2014 3

ANSI/BICSI 002-2011 Standard with a Class rating sufficient to meet the operational availability requirements shall apply. Recommended rating for LRT applications is Class 3, dual parameters for entire CCF, including: - dual uplinks between two network nodes; - dual major components (switch, server, power supply, links to a critical load); - auto-switchover for redundant major components; - disaster recovery assumed by dual CCF placed at different locations. Further considerations regarding redundancy also extends to auxiliary infrastructures such as; access control, power, fire protection, air-conditioning, etc.

MAIN TO BACKUP DATA CENTER LINKS

While equipment redundancy implemented within the same location may reduce facility costs, impacts to the operation redundancy may be significant. LRT infrastructure with vehicles storage locations typically at opposite alignment ends makes practical to build data center redundancy at two remote locations. In addition to the operational benefits in such solutions, placing two CCF’s remote to each other introduces additional communications, configuration and support complexities. In any case, Direct Core1-to- Core2 links are essential to establish systems overall integration and efficiency. High data rates between the main and backup sites are required to ensure the overall performance for application servers is not affected. As an example, 100Gbps per single connection between two sites are recommended based on commercially available equipment and over a long distance (up to 80 km). Data rate setup for such links should be carefully executed, so that transition operation from one data center to another goes smoothly and seamlessly. In order to optimize data speed requirements, four interconnections between two data centers (two cores) are recommended, each at 40 to 100Gbps, configurable for load-balance between the links. Splitting those four links into separate paths in groups of two will further increase overall system availability. It has been well documented that running all cables with the same embankment under city streets increase the risks from accidental damages during subsequent construction works despite facts that those cables run, most of lengths, within the protective embedded conduit runs.

Redundancy Issues Redundancy is typically implemented by having the main control and communication components and services divided between two different locations. Locating control and surveillance rooms are usually based on a number of various factors (available property, operator’s location and access, etc.) and those may or may not be adjacent to the main communication/equipment rooms. CCF configuration may have different requirements, but in this paper we will assume that the main control room and corresponding data center will be co-located as the most common case and with the following benefits: 1. Faster access to O&M personnel 2. Easier coordination with different services within the organization; 3. Benefits during emergencies due to easier access to all decision making personnel. When operations personnel are remote to a data center, cooperation and coordination between various services becomes complex and time consuming. LRT operations are time sensitive, so operating interruptions have to be minimized. In addition to the available tools for remote monitoring and access, personnel availability and proximity are the best scenarios to deal with emergencies and especially for safety-critical operations. Another factor to consider are operating costs and using 3rd party communication links when establishing remote operation controls.

© AREMA 2014 4

A recommended topology for establishing redundant operations is one that enables simultaneous and independent data collections at both the main and backup data centers so that direct main-to-backup links are only used for synchronization and watchdog operations.

Figure 3: Main to Backup Interconnections

This method is superior to classical replication methodology primary-to-backup data center specifically considering current increased surveillance requirements (viewing both real-time and recorded video) that significantly increases bandwidth requirements. The resulting redundancy scheme is shown on Figure 3, a mirrored configuration between the main and backup data center with watchdog links implemented for all major components and seamless switch over for the uninterrupted control room operations.

DATA CENTER ISSUES To minimize switch-over events between two data centers, all networking equipment is redundant within a single data center except for the core switch. If the main core switch fails most of the data flows would remain: - All the data collected from the field would still reach the backup DC, due to redundant connections; - End-users in the LRT should not experience any difficulties or service interruption notice. However, data exchange between two CCFs shall be interrupted and recovery services become necessary. In case the entire data center becomes unavailable, effects for the operators and end users would be the same as in the case of Core switch failures, the back-up center will automatically takes over all the services. Importance of careful design and services separation is obvious. General IT tends to aggregate different services and increased use of virtual networks is the current industry trend. Still, physical separation of critical life-safety services like train controls, emergency disconnects or fire alarm infrastructure remains a necessity. General services, like surveillance, public address, telephone, etc. are preferably run on the same physical lines as separate virtual networks (VLAN’s) to simplify support and maintenance and optimize resources. Mapping “one LRT service-per-one VLAN” is available, and multiple separated networks are transferred over single wire. Users may require preserving distinct separation for those services for various reasons including: equipment incompatibility, different maintenance staff for different services, common practices, etc.

© AREMA 2014 5

The most expensive and complex way to separate different services would be by adding equipment and links, which increases costs and support complexity. In such cases, requirements may be simplified by separations only at the field (access) level, while at the subsequent levels (distribution), equipment is shared between all the services. Uplinks connections to the Core should always be physically separated between critical and general services. As the result, networking infrastructure remains optimized with only uplinks from the field and distribution ends being increased. Service segregation at the data center can then be configured base on user’s preference and organization. Another IT trend affecting modern CCF design practices is services virtualization. The term virtualization describes the ability to run multiple operating systems on a single physical system and share the underlying hardware resources. Virtual machines become representation of a real machine using software that provides an operating environment which can run or host a guest operating system. It is now possible to virtualize almost every component of a data center: server, network, storage, and workstation. For example, a classical server solution is a single operating system per machine with software and hardware tightly coupled together. To run multiple services on the same physical server would require higher end machines and increased support to overcome resource conflicts. Virtualization, on the other hand, encapsulates an operating system and application in a single unit, a virtual machine, ensuring system hardware and software independence from each of the virtual machine consisting of a particular operating system and corresponding applications, as illustrated on Figure 4. Each VM (virtual machine) uses shared resources (CPU, memory, network card) from underlying hardware machines. The usage of virtual resources and monitoring of operations and control is done by the “Hypervisor” which is brain of the virtualization system. The hypervisor is installed on physical servers which provide these resources. There are few of those hypervisor programs for virtualization platforms on today’s market that include; VMware ESXi, Microsoft Windows Server with HyperV and Citrix Zen Server.

Figure 4: Virtualization concept

The multiple virtualization system benefits are summarized as follows: - Business continuity and increased system availability by minimizing downtime - Reducing the cost and complexity to ensure high availability and simpler disaster recovery - Reducing a number of servers and related IT tools - Portability with combined virtual machines easy to move from one environment to another for maintenance

efficiency and optimal resource utilizations.

© AREMA 2014 6

It is important to note that virtualization implementation is closely related to the storage architectures. Virtual machines reside in a storage array disks environment. Commonly SAN (storage area network) are used for relation databases and NAS (network attached storage) are used for all other applications. Clear separation between sensitive data kept in databases is maintained from other raw data types (e.g., CCTV video files, user documents, etc.). A copy of each virtual server machine is expected to exist in the secondary (backup) data center.

CCTV Service Impacts to Data Center Increased reliance on surveillance in LRT operations has significant impact on communications and data center architecture. Increased number of cameras and latest quality of video expected from digital architectures significantly increases bandwidth, storage and processing requirements. While multicasting optimizes bandwidth requirements for real-time video viewing, transfers of recorded video files, typically TCP/IP, introduce a significant network load. Dedicated application servers are typically used to manage real-time and recorded surveillance video. Due to the size and growth of video files, it is important to implement dedicated and scalable NAS storage devices for video with additional capacity automatically loaded and load-balanced, on-line by a simple plug-in with no on-going service interruptions. As with any other data center application, CCTV system redundancy and automatic failover is expected. Current regulatory requirements for the public utilities require the full archival redundancy. A modern networking infrastructure makes it possible and economically efficient to capture video images simultaneously at two separate locations, the main and backup data centers. Multicast for real-time digital video transfer and high capacity of data links makes such requirements possible. Network switching and automatic routing supports high-quality digital video transfers and expected increased role of the video surveillance in the future.

CONCLUSION The solutions presented in this paper address importance of careful considerations and implementations of the latest IT technologies to data center designs particular to LRT infrastructure implementations. Recommendations in terms of selecting data center architecture, location, connectivity and redundancy are discussed. Services segregation and potential virtualization are recommended. Data Centers represent vital parts in a typical LRT Control Center Facility (CCF) to guarantee safe, reliable and continuous operations. Generally, data centers become even more important for the infrastructure services where public safety and continual operations are of utmost importance. In addition to its role in coordinating all internal services, the LRT data center architecture is expected to facilitate all external interactions and exchanges to various City services (police, emergency, etc.) complying with different data exchange needs, standards of interconnections and users expectations. An open but fully protected architecture is capable of handling today’s operational challenges that are based on standardized data center architecture in a modern LRT CCF.

© AREMA 2014 7

CE

NT

RA

LC

ON

TR

OL

FAC

ILIT

Y(C

CF

)C

EN

TR

AL

CO

NT

RO

L FA

CIL

ITY

(C

CF

) D

ES

IGN

AN

D

LAT

ES

TD

ATA

CE

NT

ER

PR

AC

TIC

ES

LAT

ES

T D

ATA

CE

NT

ER

PR

AC

TIC

ES

-O

brad

Ale

ksic

Hat

chM

ottM

acD

onal

d-

Obr

ad A

leks

ic, H

atch

Mot

t Mac

Don

ald

© AREMA 2014 8

INTRODUCTION

CCF and Data Center for a typical LRT Project:

• Data repository & management roles

• Redundancy expectations

Service segregation needs and practices• Service segregation needs and practices

• Configuration and network topology

3-LAYER HIERARCHICAL TOPOLOGY

1. Access – Field IP devices (at stations, stops, etc.)2. Distribution – Along the ROW (at TPSS)3. Core – Data Center (CCF)

TYPICAL DATA CENTER APPROACH

• Backbone links, data processing and archival

• Services segregation

• Redundant configurations requirements

• Protection and accessibility• Protection and accessibility

TYPICAL CONFIGURATION

• Firewall-Core Switch-Local Switch-Data Storage-Servers-HMI

• ANSI/BICSI 002-2011/Class 3 concept

VIRTUALIZATION• Running multiple

operating systems on a single platform

• Share the underlying hardware resources

Benefits:– Business continuity and increased system availability– Reducing the cost, complexity, a number of physical servers – Portability - VMs ease move from one environment to another

MAIN TO BACKUP INTERCONNECTIONS

• Simultaneous and independent data collections at both ends• Direct main-to-backup links only for sync and switchover• Superior comparing to the classical main-backup replications

© AREMA 2014 9

SERVICES SEGREGATION

Dedicated links for critical and life-safety services

• Mapping “one critical LRT service-per-one LAN” is available for Critical services

Agglomeration for general services

• Mapping “one LRT service-per-one VLAN” for

General services

SERVICES SEGREGATION RATIONALE

a) Service criticality and impacts to LRT operations

b) Users and their needs

c) Equipment integration and compatibility

d) Maintenance strategy and available personnel

1. Identify, prioritize and group services2. Analyze data streams, application needs and responses3. Use virtualization to optimize implementations

SERVICES SEGREGATION PRACTICES

1. Each service has its own infrastructure

– Complex and expensive, minimal resource sharing, but highly independent and secure, typical for critical services

SERVICES SEGREGATION PRACTICES

2. Grouped services with a common infrastructure– Moderate and optimized due to virtual services approach

• Switches and links dedicated only at the field level • Shared switches and dedicated links at next levels (distribution/core)

SERVICES SEGREGATION PRACTICES

3. Different options to allocate Operator’s & Equipment rooms

– Decision based on security, access, operability, costs, etc.

INCREASED SURVEILLANCE NEEDS

High-quality digital surveillance and large video transfers:

• Higher communications and bandwidth requirements

• High video files size/growth needs scalable storage (e.g., NAS)

• Importance of smart network switching and automatic routing

• Multicast necessity to optimize bandwidth requirements

• Authorization and access control for video files utilization

© AREMA 2014 10

CONCLUSIONS

• Recommendation for following modern Data Center design strategies

• Guarantee safe, reliable and continuous operations

• Select architectures to fit redundancy, connectivity, economy

• Implement services segregation and virtualization

• Implement full protections: physical, logical and environmental

© AREMA 2014 11