CANHEIT 2012

Building the Digital University

IT Service Consolidation at the University of Waterloo

Bruce Campbell, Director Network Services

Martin Timmerman, Director Computing Systems Services

Information Systems and Technology

Format of Presentation

• Bruce Campbell

– History, Background, Motivation

– IT Task Force

– Networks

• Martin Timmerman

– AD, e-mail, web

– Committees, other

– Next steps

A bit about me

• Bruce Campbell

– Director, Network Services, Information Systems and Technology Department, 2007 to present

– Manager, Science Computing, 2005 to 2007

– Engineering Computing (various roles), 1984 to 2005

• (Approx 22 years in Faculty IT, 5 years in central IT)

• I’ve seen everything !

University of Waterloo

• Main campus located in Waterloo Ontario

– Plus campuses in Cambridge, Kitchener, Stratford, Hunstville, Dubai

• From http://uwaterloo.ca/aboutuw ...

– Founded in 1957

– 27,440 full time undergraduate students

– 3,680 full time graduate students

– 1,099 full time faculty members

– 2,184 full time staff members

– 6 faculties (Applied Health Sciences, Arts, Engineering, Environment, Math, Science)

– 4 federated and affiliated colleges

– 10 faculty based schools

IT Organization at UW

• Central IT department “Information Systems and Technology” (IST)

• Significant IT departments in all 6 faculties, and some schools (fully functioning IT departments with data centres, expertise in multiple technologies, etc)

• IT staff in some departments

• IT group in Housing recently merged with IST

• Approx 140 IT staff central out of approx 310 IT staff total.

Partial Org Chart showing central IT department and two faculty IT departments,

and University Committee on Information Systems and Technology (UCIST)

Provost

Associate Provost, IST

Dean of Eng

Assoc Dean Computing

Director, Engineering Computing

Dean of Arts

Assoc Dean Computing

Manager, Arts Computing

UCIST

Note: Computing

Technology and

Services Committee

(CTSC) formed

2009 not shown

Decentralized IT support at UW

• Historically (post mainframe era), the central IT group (IST) and largest faculty/department IT groups have each had the mandate, staff, and budget to provide most required IT services to their respective constituencies.

• Significant budgetary flexibility within faculties.

• UW has one of the most decentralized IT structures among Canadian universities.

• Approx 45% IT staff and IT expenditures central.

Group primarily involved in providing service (partial list, pre 2009)

Service Central IT Faculty IT

Learning Management

ERP

Telephones

Research Support

Computer labs

E-mail

Network

Active Directory

IT Security

Web

Desktop rollover

Storage, Backups

Printing

IT Task Force

• In November 2008 the Provost convened an IT Task Force to:

– collect information on current IT practices, and

– to make suggestions to improve the effectiveness and efficiency of IT services at UW.

• Trigger event ? Why now ?

– Cost of IT highly visible

– Duplication of effort in management of infrastructure

– Greater focus on client service desired in many areas

From the memo announcing IT Task Force...

• ...Information technology (IT) services at UW are quite decentralized...

• There are advantages to decentralization. Staff members who are regularly or permanently resident within a unit can provide timely response to local services request and are aware of requirements unique to that unit.

• There are disadvantages as well. Decentralization can lead to unnecessary duplication of services and suboptimal use of resources. It can also make it difficult to establish overall institutional directions, standards and best practices.

• ...

IT Task Force Mandate

1. Identify the collection of UW essential services

2. Identify the most efficient and effective way(s) in which those services can be provided

3. Make recommendations for changes where services are currently being delivered in ways inconsistent with 2.) above

IT Task Force Membership

Geoff McBoyle (Chair) Associate Vice President Academic (in 2008) (and former Dean)

Alan George Associate Provost Information Systems and Technology (and former Dean, Provost)

Dennis Huber Vice President Finance

Ken Coates Dean of Arts

Tom Coleman Dean of Mathematics

IT Task Force

• Consultations with campus groups

• Questionnaire

• Meetings

• etc

IT Task Force Report

• Delivered June 2009 http://uwaterloo.ca/provost/sites/ca.provost/files/uploads/files/IT-TASK-FORCE-REPORT-June-2009.pdf

• Reviewed and endorsed by Executive Council and Deans Council http://uwaterloo.ca/provost/sites/ca.provost/files/uploads/files/IT-Taskforce-memo-June-2009.pdf

• 18 recommendations

• Status of implementation at http://ist.uwaterloo.ca/istplans/ImplementationofTaskForceRecommendations.pdf

IT Task Force Report Major Recommendations

• Centralization/Consolidation of a number of services, including:

– Network management

– E-mail

– Active Directory

– Web Content Management

• Creation of Computing Technology and Services Committee

Network Management

• Network Management was centralized January 1, 2011

• (Previously, IST had been responsible for approximately 66% of campus network infrastructure funding and management)

• IST responsible for

– Architecture and design of network

– Network operations

– Funding

• Specific service provisioning activities (e.g. DNS additions, switch port configuration) may be performed by faculty/departmental IT to support local activities as efficiently as possible. (done with centrally provided tools, generally not cli access to network devices)

Transition of Network Management • Several factors contributed to success and relatively

straightforward transition...

• Central funding provided for some network infrastructure since 2006, subject to following campus standards for equipment and some practices.

• Central network management tool in use since 2005, for switch port configuration etc

• Central DNS/DHCP management tool

• Wireless already centrally managed

• Many practices already standardized (e.g. OSPF)

• Campus Network Advisory Group (CNAG) in place, to discuss practices, standards, evolution of services, etc. (since renamed Campus Network Services Committee)

Success depends on... • Support of senior administration

• Central funding

• Good service

• Commitment to tools which empower faculty/departmental IT staff to provision service and provide first level problem investigation.

• Commitment to increased self service, pre-provisioning of service.

• Documented process to augment central services.

• Clarity. Faculty/department IT staff no longer involved in network management need clear direction from their management, and other work to transition into.

IST Network Services Group

• Director

• Senior Technologist / Team Lead NOC

– 4 network support specialists

• Team Lead Cable Plant

– 4 network technicians

• Telecommunications Services and Physical Security Systems also in group (not affected by centralization of network management)

Policy

• Minimal formal policy in place.

• UW Procedure 1 contains: Any contract or agreement which includes purchase of equipment or

services with potential to impact UW's IT infrastructure must have prior endorsement of the Associate Provost IST before an agreement or contract can be executed.

• Network Management documents at: https://strobe.uwaterloo.ca/~twiki/bin/view/ISTNS/NetworkServicesResources (includes definition of network management, procedure for augmenting central services, etc)

UW’s Network Infrastructure • Cisco core, border, VPN

• HP (Procurve) switches and routers in buildings (approx 1,300 devices) – All new (or upgraded) buildings have all wall jacks live, gigabit POE.

– All new (or upgraded) buildings use VoIP phones

• Aruba wireless – deployed in all campus buildings including residences. (approx 2,000 APs)

• 2 gbps (soon to be 3 gbps) general external network service

• 1 gbps (soon to be 10) research external internet service (ORION)

• Infoblox IPAM

• Locally developed network management tool (called ONA)

• Sandvine traffic management for residence and wireless

• Juniper SRX for machine room firewall

• Juniper SRX for wireless NAT

Empowering faculty/departmental IT staff to provision service

• Our network management tool (Open Network Administrator (ona)) allows IT staff to:

– Find switch port based on IP address/name of host

– Change switch port settings, including vlan

– Disable/enable switch port

– View switch port graphs, statistics, errors, and switch syslog

– And much more

• Our IP Address Management tool (Infoblox) allows IT staff to create, change, domain names, DHCP settings, etc.

Next Steps (on network)

• Major projects under way

• Security Architecture audit

– Border firewall

– Wireless IDS, AUP display/acceptance

– Campus wide network documentation

• Disaster Recovery audit

– Increased redundancy

– Updates to fibre plant

• Service initiatives

– Wireless expansion

– External network bandwidth upgrades

– IPv6

A bit about me

• Martin Timmerman – Director, Computing Systems Services, IST Department, 2005 to present

– Staff and management in Data Processing, Department of Computing Services, and Information Systems and Technology from 1981 to 2005

• University of Waterloo for 31+ years, all in central IT

Microsoft Active Directory

• From the IT Task Force Report – Recommendation 2: The University should consolidate, to the maximum extent possible,

Active Directory forests and domains, with a preference to move to NEXUS, if feasible.

• Active Directory Consolidation and Future Governance – Merge two main Active Directory domains on campus, ADS and NEXUS

– Create a campus AD Management Committee for governance/review

– IST responsible for the single consolidated AD

– APIST should decide if new AD to be built, or merger into existing

• Consolidate by merge to NEXUS – Size of NEXUS, with more software distribution and higher workstation count

– Also who needed to do most of the work, keep burden off the Faculty IT groups and assign most of effort to IST staff

Active Directory Consolidation

• Project effort with campus participation

• Secondment of key architect from Faculty IT to lead project

• Design of new OU structure

• Addition of all users into NEXUS, plus provision from campus identity system

• Migration of workstations, 21% complete

• Change of authentication services, first Exchange email, next student facing (email, learning management, student registration, etc.)

• Introduction of servers (monitoring, database, applications)

• Change of domain level responsibility, with significant delegation

Campus Email Services

• Early selected common services – forwarding from user@uwaterloo.ca to user@mailhost.uwaterloo.ca

– spam processing, attachment and content filtering

– webmail front-end to multiple IMAP servers

• Pre-task force effort on student email (2006-07) – consider shared email service or sharing expertise to deliver a common technology

– considered enterprise email (Exchange), appliance (Mirapoint), outside service (Google Apps) and open source (sendmail/IMAP/Cyrus)

– built open source email service and invited campus groups to join

– two faculties migrated student email to shared service

• Support staff email (2008) – heavy Eudora usage, but moving to Outlook

– pilot investigation to try Exchange Server

– primarily campus support staff, but selected participation by faculty IT

Email Recommendations

• From the IT Task Force report: – Recommendation 1: The provision and management of the technical infrastructure for

the services listed above should become the responsibility of IST after consultation with the relevant local client group(s).

– Referred specifically to email services for students

– Further, IST could be responsible for delivering primary email service to all University faculty and staff

Email Services for Students

• Implementation phase – build up of infrastructure, including additional storage, automatic provisioning

– focused first on incoming class of September 2010

– existing mechanisms and support structures difficult to change

– then moved upper year students over the Fall 2010, Winter 2011 terms

– original two faculties have actually retired email servers, other four still have servers in operation, although mostly no student email load

• System statistics – 69,000 accounts

– 3.2 TB of email data spread over 4 mailbox servers

– 45,000,000 messages

– Dell servers, Redhat Enterprise Linux, Cyrus Email, some SquirrelMail usage, with Horde framework outside the cluster

Exchange Server

• Exchange Server Implementation – Pilot implementation in 2008-09, supporting only email with Exchange Server 2007

– Calendar function provided by Oracle Calendar

– BlackBerry Enterprise Server (BES) for significant usage at Waterloo

– Early 2010 investigation to migrate away from Oracle Calendar to Exchange Calendar, but wait for Exchange 2010 version

– Upgrade to Exchange Server 2010 in Fall 2010

– Migrate to Exchange Calendar on a weekend in December 2010

– Provided many in the faculty supported community with “calendar only” accounts

– With integrated email/calendar now possible, focus to migrate employees to Exchange

– Underway, but slow progress

– Variety of email clients, heavy reliance on client visits to migrate

Usage of Exchange Server

2.5 years of growth Current usage

Campus Email Services Committee

• Communication forum to discuss the services

• Promotion of tools to support users

• Plan for potential changes, including the authentication change

• Enhancements such as “sendit”, a service to distribute large files outside of email

• Started in Winter 2012

University Web Service

• From the IT Task Force: – Maintaining a state-of-the-art web site for the University involves the coordination of

content, design and technical infrastructure. The provision and management of the technical infrastructure properly belongs with IST; the design and content development of the web space belongs elsewhere within the University.

• Existing campus bodies: Web Advisory Committee and Web Steering Committee

• Campus project to choose a CMS (twice)

• Project to implement Drupal for Web Content Management System (WCMS)

• One area in IST with significant new staffing

New WCMS Service

• Web resources site at http://uwaterloo.ca/web-resources/

• Hands on planning sessions, with migration assistance by coop students hired for the purpose

• Four faculty level sites and 70 sites in total

• Not UW home page yet, coming in Fall

New IT Committee - CTSC

• Computing Technology and Services Committee

• Director level management with staff resources

• IST, Faculty IT, Library, Housing

• Encouraged to be university officers considering campus as a whole and not individual campus unit

• Regular forum for discussion

• Creation of CTSC projects and working groups (Microsoft licensing, Green IT, campus printing, AD Consolidation)

Next Steps

• Complete the initiatives (remove email servers, remove ADS, remove web servers)

• Digital Asset Management project underway

• Campus shared file service

• Mobile applications for the campus

• Helpdesk coordination, investigation of common toolset

• Mindset is starting to change, but old habits hard to change

• New CIO starts in July