Indiana University’s Virtualization Journey Rob Lowden Director, Enterprise Infrastructure.

Indiana University’sVirtualization Journey

Rob LowdenDirector, Enterprise Infrastructure

o Founded in 1820o 107,160 Students (Fall 2009)o 19,992 Faculty and Staffo $2.9B Annual Operating Budgeto $1.23B Endowmento $470M Grants / Awardso 8 Campuses Throughout Indianao Worlds Largest Music Schoolo 2nd Largest School of Medicine in the U.S.o Over 1,200 IT Staff / $110M Annual Budget

Above-Campus Sourcing Models

• Commercial Sourcing• Institutional Sourcing• Consortium Sourcing

Above-Campus ServicesShaping the Promise of Cloud Computing for Higher Education

by Brad Wheeler and Shelton Waggener

Illustration by Randy Lyhus ©2009EDUCAUSE Review, Nov/Dec 2009

IU’s Virtualization Journey

• Three quarters of a decade• Three short stories (phases)• Three quarters of an hour

In the beginningFrom: IT Manager <@indiana.edu>

Subject: ESX Trial Evaluation RequestDate: February 13, 2003 3:54:05 PM ESTTo: [email protected]

Hi, …I am the Tech Services manager for a number of areas in our IT

organization, including the mainframe and all central webserving using Linux, Tru64, and Solaris (about 35 boxes). Our immediate interest in ESX is to consolidate multiple existing (and pending) Linux servers (Redhat Advanced Server) on large 4-way and 8-way Intel platforms.

I would like to request a trial copy of VMWare ESX and perform an

evaluation of the product on a 2-way HP/Compaq DL360. Thanks…

…And so it began

From: John Acevedo <[email protected]>To: "IT Manager" <@indiana.edu>Subject: FW: VMware ESX Server Serial NumbersDate: Thu, 3 Apr 2003 11:37:04 -0800 REF: 0299810 Dear IU IT Manager, Thank you for your order.

First VM

• April 2003• CAS

– Central Authentication Service

IT Production Phase

• Began in earnest in 2006• Virtualize 100% of x86 business

systems• Standardize Supporting Architecture• Consolidate Storage• Improve HA• Maximize Cost Savings $$$$• Initially 20:1 Consolidation Ratio

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Au

g

Sep

Oct

Nov

Dec

Jan

Feb

Mar

Ap

r

May

Jun

Jul

Au

g

Sp

t

Oct

Nov

Dec

Jan

Feb

Mar

Ap

r

May

Jun

2006 2007 2008

0

50

100

150

200

250

Enterprise System Infrastructure

Linux VMs

Windows VMs

Physical Windows Servers

Quantity

IT Production Phase

• “Eat your own dog food” phase• IT confidence already existed @ IU • Customer confidence had to be

earned• Enabled Infrastructure to expand

scope of future Virtualization projects (remainder of central IT systems)

• Overall a tremendous success

Business Production Phase

• Began in 2007-2008• ERP Application / Web Servers

(2008)• Enterprise Oracle DB’s (2008)• New Data Center (2009)


• PeopleSoft Application / Web Migration

• PS Load Balancing Option• Entire Project < 6 months• Cost Savings?

– 90% over Traditional Solution

• Customer Confidence?– Unwavering


• IU Bloomington Campus Data Center

• Completed in 2009 for $32.7M• 88K Gross Square Feet• Single Story Construction

(Withstand F5)• 3 – 11K SqFt. Raised Floor Pods• Enterprise Pod (Tier 3 / Uptime

Institute)• Why So Large?

December 2008Indiana UniversityBloomington, IN (USA)

1964

2009


• Challenges– No downtime or service interruption– Accomplish over a Holiday weekend– Tier 1 Databases


• Outcomes– Never let a crisis go to waste– Virtualized 35 applications DB’s / > 100

Oracle instances– 3 Applications remained on physical

hardware– Savings?

PS/Tier 1 DB Conversion

98-99 99-00 00-01 01-02 02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10$0.00

$200,000.00

$400,000.00

$600,000.00

$800,000.00

$1,000,000.00

$1,200,000.00

$1,400,000.00

$1,600,000.00

Total

Total

IT as a Service Phase

• IU’s Intelligent Infrastructure (2007)• Demonstrated Confidence / Savings• Reduced power/cooling consumption

(Green) (< 40KW to power)• Improved flexibility, BC, HA, and DR• Web enabled provisioning and

workflow• Drive the Enzo Ferrari for a fraction

of the cost


• Standard x86 boxes across the entire platform

• One vendor, one chip set, one solution• 688 Total CPU’s/Cores• 5,504 Total GB RAM• Hypervisor Virtualization Booting from

SAN• 30-40 VM’s per physical box average• 3 year life cycle replacement


• Storage• Consolidated, Standardized, &

Virtualized• 4 Tiers of Storage

– High End Fiber– Fiber attached SAS– 2TB SATA Drives– Tape

• Thin & Dynamic Provisioning • Integrated NAS• 5 year life cycle

Hitachi USPV

196 TB raw

377 - 400GB drives

58 - 450GB drives

25 – 1TB drives

256 GB Cache

80 Channels

202TB BOS

202TB Tuning Manager

140TB BOSV

308TB Tiered Storage Manager

48TB ShadowImage

Frame License Hitachi Dynamic Provisioning

Hitachi AMS2300

97 TB raw

100 – 450GB drives

34 – 2TB high-density drives

16 GB Cache

8 Channels

Frame license Device Manager

Frame License Tuning Manager


Hitachi HNAS 3090

Clustered with IP replication

2 Brocade 48000

Director-class

128 Ports each

Hitachi AMS2300

85 TB raw


16 GB Cache

8 Channels




Hitachi AMS2300

64 TB raw


16 GB Cache

8 Channels




Bloomington Data Center

Indianapolis Data Center

Hitachi AMS2300

78 TB raw

180 - 450GB drives

16 GB Cache

8 Channels

Frame license HDvM

Frame License HTnMFrame License HDP

Hitachi AMS2300

92 TB raw

88 – 450GB drives

34 – 2TB high-density drives

16 GB Cache

8 Channels




Hitachi USPV

46 TB raw


44 - 400GB drives

256 GB Cache

80 Channels

46TB BOS

46TB Tuning Manager

52TB BOSV

98TB Tiered Storage Manager

22TB ShadowImage

100TB License Hitachi Dynamic Provisioning

Hitachi HNAS 3090

Clustered with IP replication

2 Brocade 48000

Director-class

96 Ports each

IT as a Service Phase• Backups via IBM System Storage TS3500 Tape Library, Model L23 (3584)• TS3500 Expansion Frame, Model D23 with hot swappable redundant parts• 16 slot cartridge I/O station• 12 3592-E05 / TS1120 Tape Drives • 628 3592 Extended Data Cartridges (700 GB ) - estimated capacity 1.5 TB to 4.0

TB• Library firmware - level 9590• Drive firmware - level 1E4C• Path Failover • Advanced Library Management System (ALMS)

– Virtual I/O– Logical Libraries

• TS3000 System Console - Remote Support Facility– Call Home– Remote login for Tech Support

• Tape Library Specialist web interface– Remote administration of TS3500

• IBM Tape Device Driver - lin_tape-1.27.0


• Network• 10Gbps from server rack to core• 1 Network & 21 server racks per

row• 2 rack combination for 10Gbps• 3 – 10Gbps NIC’s per server• Savings$$$$

Indiana University10G (2-rack design)

Linux Virtual Appliance Windows0

100

200

300

400

500

600

700

610

140

525

Intelligent Infrastructure March 2010

Total

Summary

• Virtualization Strategy Serving Us Well

• Growth of 6% per month (mainly new)

• Desktop Virtualization currently underway

• Savings, Flexibility, Reliability

Rob Lowden [email protected]

Thank You & Questions?

mailto:[email protected]

Copyright

• Copyright Rob Lowden, 2010. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Resources

• Intelligent Infrastructure at IU• Enterprise Oracle Linux Migration• PeopleSoft Application Server Migra

tion• Enterprise P2V• IU Fact Book

http://ii.uits.iu.edu/

http://uits.iu.edu/page/ayhe

http://uits.iu.edu/page/axes

http://uits.iu.edu/page/axes

http://uits.iu.edu/page/avzo

http://factbook.indiana.edu/

Indiana University’s Virtualization Journey Rob Lowden Director, Enterprise Infrastructure.

Documents

Transcript of Indiana University’s Virtualization Journey Rob Lowden Director, Enterprise Infrastructure.