Fundamentals of Network Design

JIM LOCKARD, PGMP, PMP, ENP

Statistics

• Approximately 30 States have begun created plans or Concept of Operations documents for the design and implementation of NG9-1-1 capabilities

• Over 25 have begun deployment of ESInet’s to support NG9-1-1

• Plus many more on the local / regional level

What Is an ESInet?

• Per as defined:◦ An ESInet is a managed IP network that is used for emergency services

communications, and which can be shared by all public safety agencies. It provides the IP transport infrastructure upon which independent application platforms and core functional processes can be deployed, including, but not restricted to, those necessary for providing NG9-1-1 services. ESInets may be constructed from a mix of dedicated and shared facilities. ESInets may be interconnected at local, regional, state, federal, national and international levels to form an IP-based inter-network (network of networks).

• Do not assume an isolated network (walled garden)

What Is ESIND?

• ESInet Design (ESIND) is the working group responsible for producing an informational document on designing an Emergency Services IP Network (ESInet)

• The current NENA document 08-506 (Emergency Services IP Network Design for NG9-1-1) released December 14, 2011

• New version finishing review NENA-INF-016.2-201X 

08-506 ESIND overview

• Covers the design of ESInets at OSI layers 1, 2, and 3

• Planning considerations, Design suggestions and implementation methodologies are discussed

• Performance requirements ◦ Service level agreements for operators ◦ Aspects of network security◦ Prioritization◦ Traffic engineering ◦ Network management and monitoring

08-506

• Informational document, not normative

• Target audience includes agencies responsible for procuring an ESInet and network architects tasked with designing it

• Provides an overview of the technologies available to build an ESInet and guidance on reliability and performance

• Does not replace proper training

• ESIND should be used IN CONJUNCTION with other standards to ensure linkage

Work group effort

• New ESIND work group opened 08-506 to specifically address:◦ Autonomous System’s◦ DNS◦ New service models◦ Refine metrics◦ Review entire document

New ESIND considerations

• How to use wireless links◦ Microwave◦ Can FirstNET impact me◦ Can I use WiFi◦ What about Satellite

• Autonomous System

• Other Internet Access at a PSAP

• Multiple service providers

• How to gauge / assess diversity and redundancy

What can make up an ESInet

• Essentially an IP network

• The Core services are:◦ IP routing◦ Security◦ Switching◦ Gateways◦ InterconnectionsLocal connectionsTelco connectionsOutside agency connections

ESInet and i3

• ESInet creates the capability for the functions to operate

• i3 functions utilize the ESInet for transport, delivery, end to end connectivity◦ BCF◦ ESRP◦ ECRF◦ LNG◦ LPG◦ LSRG

Emergency Services IP network (ESInet)

•NOT a walled garden◦ All ESInet elements assume ESInet is

the open Internet

• The ESInet IS connected to the Internet

• All public safety, not just 9-1-1

• Bottom up design – local ESInets interconnected to form state ESInet (with optional state backbone), state ESInets interconnected to form national ESInet

PSTN

INTERNET

International

National

Regional

State/Province

Regional (county etc.)

PSAP

Protected Layer

Protected Layer

Protected Layer

Protected Layer

Protected Layer

Protected Layer

Design considerations

• Connections

• Bandwidth

• Security

• Service level

• IP service (MPLS or other)

• Network Monitoring

• Performance requirements

Additional considerations

• Interconnections

• QoS

• Traffic Engineering

• Diversity

• Redundancy

• Fault Tolerance

Additional considerations

• Availability

• Reliability

• DNS operation

• DDoS mitigation

• Service Level Agreement (SLA)

• Access guidelines and rules

• Capabilities

Potential problem areas

• Providers unfamiliar with ESInet stds◦ Common enterprise standards can be different than expected ESInet standards

• ESInets not fully implemented◦ Networks that are partially implemented as IP networks

• Specific rules interfering with full implementation◦ Internal policies that conflict with an ESInet implementation

• ESInet “edge” integration◦ ESInets not utilizing common interfaces, protocols etc

•Transition and Migration◦ Testing and acceptance challenges

Potential Pitfalls

• Support staff training◦ Planning, Scheduling, System soak

• Pre-cut testing◦ Misroutes◦ Switch provisioning

• Phased system cutover◦ Delays due to actual emergencies

• Post-cut testing◦ Unresolved issues – non-production impacts (encryption etc)

• System acceptance

Layer 1: Physical

• This layer deals with cabling and communication medium

• Typical circuits to connect to an ESInet:◦ Copper (ex. T1, T3)◦ Optical Fiber◦ Radio and Satellite (ex. 3G/4G, microwave)

• Most important to consider is path diversity◦ Get connectivity from different sources◦ Circuits should follow different routes

Layer 2: Data Link

• Responsible for framing traffic on the wire

• Highly dependant on physical layer

• Availability depends on local service providers

• Examples:◦ HDLC (T1/T3): multiple circuits can be bonded to increase bandwidth

◦ Frame Relay: not recommended

Layer 2: Data Link (cont.)

• ATM: supports different classes of services:o CBR: Constant bandwidth, more expensiveo VBR: Variable bandwidth, less expensiveo UBR: Unspecified bandwidth, least expensive

• Metro Ethernet: o Ethernet over a wide areao Wide variety of providers

• MPLS:o Tends to replace ATM and Frame Relay

Layer 3: Network

•Two addressing schemes:◦ IPv4:Widely deployed (used by 98% of devices)Public addresses are soon to be exhaustedThere are mechanisms to lessen effect of exhaustion such as private IP addresses and NAT

◦ IPv6:Practically endless supply of addresses Interoperability with IPv4 difficultNot widely implemented by software vendors

◦ Best practice to design ESInet with both IPv4 and IPv6 (dual stack)

Layer 3: Network (cont.)

• Dynamic routing protocols:◦ Used to discover network topology and determining route availability

◦ Best practice to utilize a routing protocol within an ESInet◦ Autonomous System: An AS is a network or group of networks under a single administration

Best practice to configure regional ESInets as their own AS

Layer 3: Network (cont.)

◦ Interior Gateway Protocols: Used within an AS OSPF:◦ Widely deployed◦ Version 2 is for IPv4◦ Version 3 is for IPv6 (recommended)

EIGRP (Enhanced Interior Gateway Protocol):◦ Proprietary to Cisco

IS-IS (Intermediate System):◦ Commonly used on large deployments

◦ BGP- Border Gateway Protocol: Provide routing between AS Protocol of choice when dealing with untrusted networks Should be used at the state and national level

Traffic Engineering

• ESInet should provide non-blocking service to high priority traffic

• Network design must take into account not only current requirements but future as well, such as video◦ If it is not financially feasible to build the ESInet with capacity that will only be used

in the future, then it should be easy to add capacity when time comes

• When designing for video, a possible formula for determining bandwidth is 2Mbps per PSAP + 2Mbps per position that will take video

Traffic Engineering (cont.)

• When interconnecting an ESInet with the Internet, care should be taken to mitigate the impact of a Distributed Denial of Service (DDoS) attack and still provide service

• Service providers may monitor incoming traffic from an ESInet and may discard any traffic that exceeds the rate purchased◦ Traffic shaping should be applied before transmission to the service

provider so it meets the provider’s acceptable rate

Traffic Engineering (cont.)

• In legacy telephony, deployed capacity cannot be exceeded◦ Once capacity is used up, any new call gets a busy

• On the other hand, an ESInet will happily accept a new call when capacity is exceeded◦ In which case all calls will be affected as packets will be dropped

randomly◦ The ESRP should be used to perform Call Admission Control to prevent

capacity to be exceeded

Traffic Engineering (cont.)

• Quality of Service (QoS) provides different priorities to different data flows◦ Only useful if implemented end-to-end◦ Service provider’s ability to support QoS should be taken into account

◦ ESInets utilizes DSCP to implement QoS◦ NG9-1-1 Functional Elements must mark their traffic appropriately

Hardware/Network Elements

• Elements used to build an ESInet should:◦ Be highly reliable◦ Have a proven track record◦ Have a warranty◦ Have qualified personnel to support it◦ Have a vendor that provides 24/7 support◦ Have an acceptable MTTR◦ Be scalable

Network Security

• Applicable NENA standards:◦ 75-001 Security for Next-Generation 9-1-1 Standard (NG-SEC) contains a

number of sections applicable to an ESInet◦ 08-003 Detailed Functional and Interface Specification for the NENA i3

Solution – Stage 3 also contains requirements including encryption and authentication

• Border Control Functions should be deployed upstream and downstream from an ESInet◦ Logs and alerts should be monitored

Network Management and Monitoring

• An ESInet should be monitored◦ Not a regulation requirement but may well be◦ All circuits and network elements should be monitored◦ All network components should generate SNMP traps to a network

management system◦ Non-network components such as NG9-1-1 functional elements should also

utilize SNMP traps

Network Management and Monitoring (cont.)

• Proper network management requires:◦ Accurate and up-to-date documentation◦ Demarcation points◦ Contacts and escalation lists◦ SLA benchmarks◦ Capacity management/trending◦ Monitoring the state of element configuration (ex. QoS)◦ Configuration management / change control

Performance Requirements

• Packet loss, jitter and delay all affect the quality of multimedia traffic

• Packet loss:◦ Over 5% affects intelligibility◦ Should be kept below 2.5% and 1% is achievable

• Jitter:◦ Describes the variable delay packets experience crossing the network

◦ Jitter buffers are used in endpoints to smooth out variation◦ Jitter should be bounded to less then 20 ms

Performance Requirements (cont.)

• Latency:◦ Time required for a packet to reach its destination◦ When mouth to ear latency exceeds 150 ms, full duplex conversation

becomes difficult◦ Best to keep latency in ESInet less than 15 to 20 ms

Service Level Agreement

• It is a contractual agreement with a vendor on a service level commitment◦ Typically represented as uptime (ex. 99.9%, 99.99%, 99.999%)◦ Typically describes where, how and how often measurements are

made◦ Typically uses impact levels to define severity of outage◦ May include financial penalties if agreement is not met◦ Should also include response times for repairs and if on-site spares

are required

Service Level Agreement (cont.)

• ATM, Metro Ethernet and MPLS vendors usually share their network capacity among multiple customers◦ Unless an SLA is in place, there is no guarantee that ESInet traffic will have

preferential treatment

• Redundant systems should be regularly verified with deliberate fail-over tests

ESIND – i3 links

• ESInet’s◦ Physical connections - capabilities

◦ Routers / Switches◦ Interconnections◦ Redundant networks◦ Bandwidth◦ The foundation for applications and functions

• NG9-1-1 Functionso Logical internetwork -

functionso Functional elementso Internetworking -

Network addressing (IP, SIP, etc)

o Securityo Applications o Utilization of the ESInet

08-506 ESIND overview

• Meant to be used in conjunction with Standards documents, not as a substitute for them

• A summary of the core requirements for an ESInet are included in the NENA 08-003 v 2.0 Detailed Functional and Interface Specification for the NENA i3 Solution – Stage 3 are as follows:

◦ The network between the PSAP and the ESInet will be a private or virtual private network based upon TCP/IP

◦ It will have scalable bandwidth to support new enhanced services.◦ The Emergency Services IP Network shall be a conventional routed IP network

◦ MPLS or other sub-IP mechanisms are permitted as appropriate

08-506 ESIND overview (cont)

◦ The PSAP should use redundant local area networks for reliability

◦ PSAP LAN to the ESInet must be resilient, secure, physically diverse, and logically separate

◦ The ESInet shall be engineered to sustain real time traffic, including data, audio, and video

◦ Connections between the PSAP and ESInet shall be secured TCP/IP connections

◦ ESInets should be capable of operating on IPv4 and IPv6 network infrastructures

•Thank you!