1

UHG MPLS Experience

June 14, 2005

Sorell Slaymaker

Director Network Architecture & Technologies

sorell_slaymaker@uhc.com

2

Overview of United Health Group

• Diversified Health Care Company

• 41B in revenue

• 11B in acquisitions in 2004

• 6 Primary Divisions with 150 offices, primarily in the U.S.

• 42,000 employees - 5,000 in IT

• 4 primary and 11 secondary data centers

• 500 business applications, key applications are home grown

• Use technology as a competitive advantage

• Goal of IT infrastructure is to be reliable, efficient, cost effective

• In-sourced voice & data network with 70M/year savings with a 25% improvement in network availability and performance

3

Why MPLS on WAN• Any to Any Connectivity

– Performance – Optimal routing of traffic – Avoid tandem routing through a hub

– Lower Costs – FR & ATM w/ multiple PVCs, private line, and a CTI network. WAN transport costs reduced by 25% (12M to 9M - 48 sites getting less bandwidth, 29 getting more, & 70 staying the same)

• Convergence– QoS for key data applications that use to run on separate data networks– Voice – Migration to VoIP and IPT– One IP Network for all applications – Voice, data, video, remote storage, …

• Flexibility– Ability to scale – Add Bandwidth quickly over multiple layer 2 options– Traffic Engineering – Can tune how traffic routes across network

MPLS Cloud(any-to-any connectivity)

UHG Data Centers

UHG Remote Sites

Future Data Center(s)Eagan Southbury

UHG Frame Relay / ATM Cloud

UHG Data Centers

UHG Remote Sites

Future Data Center(s)Eagan Southbury

4

•Experienced only one outage at a single site during migration•Had good back out plans•Used underlying existing T1 & DS-3 circuits in most cases•Continued to experience outages using IMA (sites requiring 3-9Mbps)

UHG MPLS Migration

• Simulate all business

unit applications on a

lab-based MPLS

network

• Document results

• Simulate all business

unit applications on a

lab-based MPLS

network

• Document results

Proof of ConceptProof of Concept

Phase 1 (completed)Phase 1 (completed) Phase 2(in progress)

Phase 2(in progress)

Phase 3Phase 3 Phase 4Phase 4

• Cutover pilot sites to

public MPLS service

to ensure performance

• Finalize design

• Operationalize

• Cutover pilot sites to

public MPLS service

to ensure performance

• Finalize design

• Operationalize

PilotPilot

• Cutover all current

data WAN sites to the

MPLS network

• Decommission the

existing data WANs

• Cutover all current

data WAN sites to the

MPLS network

• Decommission the

existing data WANs

Roll-outRoll-out

• Add 2nd Carrier in an

active/active config. To

critical & large sites

• Tune QoS model

• Validate reporting

(performance & billing)

• Add 2nd Carrier in an

active/active config. To

critical & large sites

• Tune QoS model

• Validate reporting

(performance & billing)

ExtensionExtension

3Q03 4Q032Q031Q03 1Q05

Phase 5Phase 5

ScaleScale

• VoIP

• New Acquisitions

• Conferencing

• VoIP

• New Acquisitions

• Conferencing

3Q05

5

UHG MPLS Configuration

• 150 sites in the U.S. - 25 sites are large enough to require DS-3 access with VoIP and data center connectivity the primary requirements for bandwidth.

• Private IP VPN service for WAN – IPsec VPNs are used for external access for home users and 3rd party connectivity. Encryption on our core WAN is not required and we wanted to avoid the overhead of IPsec.

• We own and manage the edge routers and use Netflow & Concord to monitor performance

• We defined our private IP addressing and AS numbers

• Use BGP for the CE to PE routing (internal network has a BGP core and OSPF at each site giving us scalability and control)

6

UHG Challenges in Migrating to MPLS

• Training

– MPLS – New technology

– BGP – Virtual VPN services requires layer 3 – IP knowledge

• Different Carrier Approaches

– Who defines AS & IP numbers

– QoS Policies

– Routing Policies in an active/active mode and asymmetric traffic

• Management

– End to end measurements – Dropped packets, Latency, Jitter

– QoS policies – Validating the right traffic is in the right queue

7

UHG MPLS Best Practices

• QoS for a Site

– 60% RT – For Voice traffic– 40% NRT

– 60% - Critical Data– 30% - Normal Data– 10% - Best effort data

• ePVC not equal to port speed – Having the ability to burst

• Customer should design IP address & AS numbers, not carrier

• Invest in performance monitoring tools – We chose Concord & Netflow

• Meet with provider monthly to review availability, performance, and billing

• Adjust application timers to handle a 60 second reroute – IPagent, Citrix,

• Documentation is key – See Appendix 1

8

UHG MPLS Futures • Growth – OC-3 & Gig Ethernet to support

• 25%/year business growth• VoIP growth - centralized call center & audio conferencing• IPT conversion – UHG will be 90% IPT by end of 2006

• Efficiencies• ML-PPP – Replace ATM & IMA (Cell tax and stability)• POS & GE – Replace ATM & FR on head end pipes

• Convergence• Local & Off-net LD voice – A remote site will only have data connectivity

• External Connectivity

• Use MPLS for external network connections to share head end access while keeping the layer 3 networks separate.

9

Conclusion

• Convergence and 1 network, an IP network, is where we are going and MPLS on the WAN is our chosen technology to get there

• Negotiating price along with service levels with carrier(s) of choice enable a cost effective & reliable solution

• Converting existing circuits to MPLS is fairly straight forward. Ordering new local access for new circuits has all the local access provisioning challenges.

• MPLS is mature from a platform perspective, but expect a lot of new services to be added in the next 2 years with local and off-net voice a key one.

• Training and staffing need to be addressed up front. Having engineers who are accountable for specific sites and then one who owns entire WAN has worked well.

• WAN Bandwidth demand has doubled in 2 years due to VoIP and growth of the company. Moving to MPLS has enabled us to keep up with demand while optimizing costs.

10

Appendix 1 – Example Site Documentation

A/C Circuit AA/C Circuit B

modem

SD

Cisco 1720

BRIS/T

CONSOLE

AUXWIC 0 OK

OK

B2

B1

WIC 1 OK

DSUCPU

LNK100FDX

S3

LOOP

LP

Term Server IP

PSTN

WAN

ATTMPLS

Circuit: ATM DS3Port : atm 2/0-Circuit IDe-pvc: 10 MBMPLS AS# 13xxx IP 10.1.x.xLocal AS# 64xxx IP 10.1.x.x

MCIPIP

100Base-TX

Device DNA Name

Room UPSNo-IP

100/1000 BaseT

Server

100/1000 BaseT

Servers

G5/1G5/2

A/C Circuit BA/C Circuit A

Device DNS Name

LAN

Modem IP addressOut of band phone #

Device DNS Name

Lo0: Loopback IP

Cisco 7204VXR Cisco 7204VXR

F0/1 - IP Addr F0/1

G3/2 G3/2

F0/0 - IP Addr F0/0

G3/1 G3/1

OSPF Area x

2x Port-channel 51BB Trunk Allow all

10.15.x.n/30

Lo0: 10.15.x.x/32Vlan 100: 10.15.x.x/30Vlan 102: 10.15.x.x/22

Catalyst 6506 Catalyst 6506

DS3

G5/1G5/2

To IDFs

Circuit: DS-3 ATMPort : ATM 1/0Circuit ID:e-pvc: 20 MBMPLS AS# 13xxx IP 10.1.x.xLocal AS# 64xxx IP 10.1.x.x

Site Name, Addr, & Main #Site Contact Name & #

Net Ops Center Contact # - 24/7Site Engineer Name & #

DS3

Device DNS Name

Lo0: Loopback IP

To IDFs

Lo0: 10.15.x.x/32Vlan 100: 10.15.x.x/30Vlan 102: 10.15.x.x/22

11

Appendix 2 – QoS Measurements

IP Protocol based Quality of Service (QoS)

Application group based Class of Service (CoS)

Both IP protocol based as well as application group classification can be applied to setup MPLS label switched paths (LSP). This provides the flexibility to provide both qualitative as well as quantitative service levels