Outsourcing ResNetat Ithaca College

ResN

etRe

sNet

Ithaca CollegeIthaca CollegeRESIDENCE HALL COMPUTER NETWORKRESIDENCE HALL COMPUTER NETWORK

2

Ithaca College

• Located in upstate New York• Founded in 1892• Private, coed, residential college• Offering undergraduate and

graduate programs in 5 schools & aninterdisciplinary division.

• Over 100 degree programs

Faculty 440Staff 985Students 6,200

In Residence Halls 4,400

3

Information Technology Services

Director of ITS

Information Systems

and Services

Network and Communication

Services

Technology and Instructional

Support Services

Web, Systems & Departmental

Services

• Networks• Phones•9 people

• Field Service•User Support•Academic•14 people

• Web Dev.• Sys Admin• ITS Business•18 people

•ERP, Ops

•19 people

4

Residence Hall Network

• Phased in over 3 years in early to mid 90’s• “One port per pillow”• Today - 53 residential halls with 4,400 ports

5

ITS provided ResNet

• Flat network, 10/100 Mbps to the desktop • “Free” – no additional charge for the students• Shared main college Internet connection

– ResNet bandwidth was allocated as a percentage of our main connection

– Eventually allocated bandwidth by time of day (shifted more bandwidth to ResNet at night, less during day)

– Shaped Internet traffic (allocated very little bandwidth to identified P2P traffic)

6

ResNet Pressures Build & Change

At first… • It was all about support to get student machines configured and connected

By 2000 it was about….

• Support &• Bandwidth &• P2P Management

And then…. • File sharing (RIAA) &• Bandwidth &• P2P Management &• Security - Viruses and Worms, and their impact on network performance

7

Bandwidth Allocated to ResNet

1998 3 Mb

1999 6 Mb to 12 Mb

2000 16 Mb

2001 24 Mb

Spring 2004 38 Mb

38 Mb

3 Mb

8

Fall 2003 Approaches

• Viruses, Worms, and other self-propagating nasties– And their significant impact on network performance

• RIAA, MPAA, ESA…• Insatiable Bandwidth Desires• P2P “whack-a-mole”• Internal debate regarding the purpose of ResNet and

use of our resources, and what is appropriate use of ResNet (games, P2P, etc)

• Staff needed for other projects

9

Fall 2003 – It boils over

• August 2003 Blaster and Nachi strike and ResNet could not handle the traffic from infected machines

• On the Friday before opening weekend we turned off the entire ResNet to prevent further infections – It ended up being off for 7 days while we

developed procedures to deal with the situation and worked with students to get their machines protected and cleaned

10

11

Why was it so bad?• Flat network

– The entire ResNet was one subnet, no way to contain an infected machine

• Full 10/100Mbps Ethernet speeds– Allowed viruses and worms to spread like wildfire – An unprotected machine would be infected within

30 seconds of connecting to ResNet• No firewall protection for ResNet machines• Individuals were not used to having to worry about

computer security or patching

12

And once past opening weekend…• We continued to take incremental steps to address many of the

performance, support and network architecture issues• Had to remain vigilant and constantly turn off ports of infected

machines– One or two infected machines would have significant impact on

network performance• Saw a significant increase in P2P usage• RIAA, MPAA complaints increased significantly• Continued to experience a huge drain on support, network and

other ITS staff – many projects were put on hold• Bottom-line: throughout Fall 2003 ResNet performance was

sporadic and ITS staff stretched thin

13

Students (and Parents) React

• Student created “ResNet Sucks” Website with on-line petition and call-to-action

• Editorials and letters in student newspaper• Students deliver a petition to ITS with 1,175

signatures demanding a better ResNet• And of course, angry emails and phone calls to

the helpdesk, ITS management, and the college administration

14

www.evanm.com/resnet/indexold.htm

15

www.ithaca.edu/ithacan/articles/0311/06/news/LSresnet_revol.htm

16

Something had to change

• If we were to continue to support ResNet ourselves, we would need to:– Significantly change and upgrade the network – Install better traffic monitoring and compliance systems– Provide additional network and support staff– Find additional space for student computer repairs

• However, we were concerned about the ability to find funding and staff resources to support this for the long term

17

Thoughts turned to outsourcing

• Initial requirements:– Tiered service plan including a base no-cost option

for the students– On-site support– Increased bandwidth– Robust network architecture that could better

withstand viruses, Trojans and worms– A true partner interested in a collaborative

relationship and who understands higher education

18

We solicited proposals

• A major nation-wide telecom• The local cable-TV / broadband franchise • A small company specializing in supporting

residence hall networks

19

The nation-wide telecom

• Limited experience with this type of network and service

• Limited experience with the academic environment

20

Local cable-TV / broadband provider

• Would provide a significant discount for their broadband service – Based upon a guarantee # of subscribers from us– The college would need to make one payment to

them, and have students pay the college for the service

• Would not provide tiered or free service options

• No direct on-site support

21

Local cable-TV / broadband provider

• Would need to build an entirely new network infrastructure in the residence halls– Current cable-TV infrastructure could not support

broadband service– College would be unable to support this cable-TV

based network should the partnership end

22

Small Residence Hall Provider

• Agreed to meet all of our requirements– Tiered service options (including “free” option)– On-site support– Increased bandwidth / network redesign– Understood residence hall networks and the

academic culture• CEO very effective at talking with the students about

their concerns and issues– Interested in a partnership-type of relationship

23

Small Residence Hall Provider

• Concerns:– Located in Austin Texas with only two current

customers– We would be their largest deployment (and

farthest away)– Small company, operating “lean”

24

Selected Apogee Telecom

• We felt that these concerns were manageable• Selected the small residence hall provider,

Apogee Telecom• Felt they were the best overall fit for us

25

What we outsourced…

• Redesign of the Residence Hall network• Network monitoring, troubleshooting, repair

and replacement• Custom service levels and prices that met our

particular needs and budgets• Management of the account sign up process• Student billing• Marketing and PR

26

What we outsourced…

• An on-site technician for troubleshooting student network issues

• A call center open 7 days-a-week • The ISP of record for DMCA and other

complaints– Work collaboratively with the college for

responding to DMCA complaints– College gets copies of all notices and still may

impose judicial sanctions

27

Apogee Telecom – The Network• Redesigned our residence hall network and “over

built it”– Provided 3.5 times the bandwidth (OC-3, 155 Mbps total)– Used our switches, but replaced most other components– Installed two Allot Communications NetEnforcers to

control network traffic and bandwidth allocation– Significantly limited the number of computers on any one

subnet• Students allocated bandwidth based upon their

service level– Could use that bandwidth for whatever they wanted (any

“legal use”)

28

Fall 04 Apogee Service Plans

Plan Cost per semester

On Campus Resources Download Upload

Basic Free High speed* 64 Kbps 64 Kbps

Bronze $49.95 High speed 1 Mbps 128 Kbps

Gold $69.95 High speed 3 Mbps 256 Kbps

Internet Access

*High Speed = 10/100 Mbps to all ithaca.edu resources

29

Apogee Service Costs

• The College pays Apogee a base fee per month to cover:• 25 Mbps administrative internet service• “Free” 64 Kbps basic-level Internet service for all students• High-speed access to on-campus resources for all students• Apogee help desk support for ResNet students

• This base fee is roughly equal to the amount we used to pay to our old campus network provider for the T3 service – so in the end, there was no increase in costs to the college

30

The Implementation

March – August 2004

31

TimelineContract signed

Communicated plans to campusTechnical site visits and design workCable trenchingImplementation sub-teams created

Support procedures created

Student antivirus strategy determinedNetwork design finalizedOnsite support staff interviewed and hiredLegal / policy issues agreed upon

Site coordinator startsNetwork infrastructure upgraded

ResNet by Apogee Live

March April May June July August

32

Campus Communication

• Apogee CEO came to campus and met with administrators, student government, and conducted an open town hall meeting

• Announcement letters sent to students and parents detailing changes

• Ads placed in the student paper• On-going dialogs with student leaders and

parents to respond to concerns…

33

www.evanm.com/resnet/index2.htm

34

A petition to ITS with 1,175 signatures responding to a February 26, 2004 student newspaper article. They request that we work with a vendor such as Apogee to fix ResNet and not impose any additional fees for what they consider a basic and necessary service.

35

Account Sign-ups

• No matter what service level, people need to sign up for an Apogee ResNet account

• Advance sign-up mailings sent over the summer to all students living in residence halls

• Freshman sign-ups at orientation (June/July)• Could sign up via Web, phone or paper form• 2,892 people signed up in advance (67%)

36

Fall 04 Apogee SubscriptionsBasic 1,937

Bronze 1,120

Gold 1,214

Total Subscribers 4,271

As of December 28, 2004

Basic46%

Bronze26%

Gold28%

37

Activation Process• A student plugs their computer

into a ResNet port• Automatically directed to

Apogee login page– Includes a link to the account

sign-up page if needed• Student activates computer

using Apogee supplied username and password

• Only need to do this once per semester

38

Cutover and Move In – Here’s Murphy

• Last of the new equipment cutovers occurred shortly before opening weekend

• Major fiber-cut 60 miles away took down service the week before opening weekend

• A few days before opening weekend a network problem developed where students were not getting their subscribed-to level of service– Would result in a MAJOR PR problem if it didn’t work!– Apogee was able to put a workaround in place late at night

on the Friday of opening weekend

39

Opening Weekend

• Our helpdesk and offices were quiet (it was strange!)• Apogee’s call center was busy, on-site staff were very

busy and some students had to wait a few days for on-site help

• Overall things went smoothly– Few if any complaints ever came to us regarding problems

• ITS staff were unsettled by not having the ability to directly monitor ResNet performance or call volume to the Apogee support center

40

Early Fall

• A few scattered equipment problems • One particular residence hall experienced a

few interruptions due to someone repeatedly plugging an improperly configured wireless router into the network

• Overall, things went well

41

Other (minor) issues that came up along the way…

• Different expectations for who would be on-site during opening weekend

• ITS staff and management had some trouble “letting go”….

42

How’s it working?

• ITS has had time to focus on other things– “This has been my most productive fall in a very

long time”• Students seem to like the service

– Apogee conducted a customer survey in late fall. Out of 4,193 survey’s sent out, received 1,067 responses (25.4%)

43

“Connection is stable and reliable”

Total responses: 1,067

Agree 54%

Disagree 33%

Undecided 13%

0

100

200

300

400

500

600

700

44

“I am satisfied with the speed I signed up for”

Total responses: 1,067

Agree 69%

Disagree 21%

Undecided 11%

0

100

200

300

400

500

600

700

800

45

“The Apogee call-center support staff was knowledgeable in troubleshooting my issue”

Total responses: 425

Agree 80%

Disagree 8%Undecided 12%

0

50

100

150

200

250

300

350

400

46

And even our friend Evan, seems to like the service….

47

48

Where do we go from here….

49

Spring 2005

• Continuation “as-is”• Continue to optimize network settings and

support procedures

50

Future

• Upgrade a few more network components• Look to add possible other services in the

future:– Provide subscription-based TV-like services over

the network– Wireless in residence halls– Better automatic virus and patching updating and

quarantine services• Revisit price-points

51

Is Outsourcing ResNet For You?

• Depends on a number of factors such as:– In-house staffing levels and expertise– Your view of the roll / mission of ResNet– Ability to continually refine the network and services to

meet changing needs– Cost structure

• It’s really a decision that must fit with the values, expectations, and unique situation of each institution

• For us, so far the answer has been yes

52

Other Resources

• The Chronicle of Higher Education– “Outsourcing the Dorm Network”– December 3, 2004

53

Special thanks to David Weil

Director of Web, Systems and Departmental ServicesInformation Technology Services

for his assistance in preparing this presentation.

Outsourcing ResNetOne Institution’s Experience

ResN

etRe

sNet

Ithaca CollegeIthaca CollegeRESIDENCE HALL COMPUTER NETWORKRESIDENCE HALL COMPUTER NETWORK