I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Assessment of Cost, Quality, and Value in University IT Services

Christopher S. Peebles

Associate Vice President for Research and Academic Computing and Dean for Information

Technology

Indiana University

JISC/CNI Conference, 14th-16th June 2000, Moat House, Stratford-upon-Avon, England

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Assessment

• “Assessment is a process that focuses on student learning, a process that involves reviewing and reflecting on practice as academics have always done, but in a more planned and careful way.” Catherine Palomba and Trudy Banta. Assessment Essentials. Josey-Bass, San Francisco, CA, 1999, p. 1.

• Assessment effort of IT in the context of distributed education has gone hand-in-hand with the development of distance learning. For example:– John Daniel. Mega-Universities and Knowledge Media. Kogan Page,

Ltd., London, 1996

– A.W. (Tony) Bates. Managing Technological Change: Strategies for College and University Leaders. Josey-Bass, San Francisco, CA, 2000.

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Assessment Continued

• Assessment of IT in the context of bricks-and-mortar based higher education is of recent vintage. IT was always taken as a “good thing,” valuable by its mere existence. Now, where IT can consume up to 6% of a university budget, questions are asked about its cost, quality, and value to teaching, learning, research, and support of the business of the institution.

• Exemplary explorations:– Patricia Senn Brevik. Student Learning in the Information Age. ACE Oryx

Press, Phoenix AZ, 1998– Diana Oblinger and Richard Katz. Renewing Administration: Preparing

Colleges and Universities for the 21st Century. Anker Publishing, Bolton, MA, 1999

– Richard Katz and Associates. Dancing with the Devil: Information Technology and the New Competition in Higher Education. Josey Bass, San Francisco, CA, 1999

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Assessment Continued

• Assessment -- that is hard measures of output and outcomes -- in the context of cost, quality, and value measures of IT in traditional higher education settings are hard to find.

• Three useful examples:– The Flashlight Program: systematic evaluation of the consequences of the use of

IT in teaching and learning. Instruments and methods to measure learning outcomes and costs of IT. http://www.tltgroup.org/programs/flashlight.html

– CNI Assessing the Academic Network. The assessment “manual” by Charles McClure and Cynthia Lopata. Cooperative venture of assessment of IT among several universities. http://www.cni.org/projects/assessing/

– CAA Computer Assisted Assessment Center at the University of Luton, UK, http://caacentre.ac.uk/index.shtml, which is part of the Teaching and Learning Technology Program

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Cost

• “There are no results inside an organization. There are only costs.”

– Peter F Drucker, Managing the Nonprofit Organization: Principles and Practices. Harper Collins, NY, 1990. p. 120

• Activity Based Costing-Activity Based Management

– John Shank and Vijay Govindarajan. Strategic Cost Management. Free Press, NY, 1993

– Robert Kaplan and Robin Cooper. Cost and Effect. HBS Press, Boston, MA, 1998

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Quality

• Quality, def. Fitness for use and freedom from defect in a product or service. Joseph Juran, Juran’s Quality Handbook, 5th edition, 1999.

• Quality foundations (a few good works among much management rubbish)– Joseph M. Juran. Juran on Quality by Design. Free Press, New York, 1992.– W. Edwards Demming. The New Economics. MIT CAES Press, Cambridge,

MA, 1993 – Daniel Seymour. On Q: Causing Quality in Higher Education. ACE Oryx

Press, Phoenix, AZ, 1993– Daniel Seymour. Once Upon A Campus: Lessons for Improving Quality and

Productivity in Higher Education. ACE Oryx Press, Phoenix, AZ, 1995.– Roger Kaufman and Douglas Zahn. Quality Management Plus: The Continuous

Improvement of Education. Corwin Press, Newbury Park, CA, 1993

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Value

• IT and Value Creation

– It’s all about time: powers of automation and augmentation

• IT and Value Destruction

– It’s all about time: wasted time due to poor operating systems, poorly crafter applications, and mysterious, opaque user interfaces

• IT and Value Protection

– It’s all about time: time spent in support and education

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Measures of Performance and Success

• Do not have measures like EVA and “profit” as a measure for the success of university IT organizations

• Must draw exemplars from business and benchmarks from wherever they are available

• Organization performance: IBM “Adaptive Organization” and “Customer Relationship Management”

• Measurement: “The Balanced Scorecard” and “ Counting What Counts”

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

The Ferengi “First Rule of Acquisition”: Once You have their money, never, ever give it back

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

IT Organization

• Stephan Haeckel. Adaptive Enterprise. HBS Press, Boston, MA, 1999 – “business focus must shift from products to processes and competencies;

– individuals close to the firing line must be empowered;

– customers needs must receive increased attention” p. 2

• The highly flexible, modular organization that can “sense and respond” rather than “make and sell.”

• James Cortada and Thomas Hargraves. Into the networked age: How IBM and other forms are getting there now. Oxford University Press, New York, 1999– enterprise transformations based on knowledge, process, and

technology

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Harvey Thompson, The Customer Centered Enterprise: How IBM and Other World-Class Companies Achieve Extraordinary Results by

Putting Customers First. McGraw Hill, NY, 2000

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Performance Measures for All Organizations, Including University IT Organizations

• Robert Kaplan and David Norton. The Balanced Scorecard. HBS Press, Boston, MA, 1996.

• Four dimensions of retrospective and prospective measures– Financial perspective: deployment (and growth) of revenue, ABC against

internal (historical) and external benchmarks– Customer perspective: customer satisfaction measures, number of partnerships

with faculty in teaching and research, support of university business processes, support of library processes

– Internal perspective: process measures, classic IT measures of availability, cost-of-poor-quality, speed and depth of development cycles

– Learning perspective: employee satisfaction, employee development (MSCE, CCNE, etc.), personal alignment of employee goals with position

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Accountability: More than Measurement

Marc Epstein add Bill Birchard. Counting What Counts: Turning Corporate Accountability to Competitive Advantage. Perseus Books, Reading, MA, 1999

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Culture, Strategy, Organization & Structure

• Mary Douglas. How Institutions Think. Syracuse University Press, 1986. Dimensions of Grid and Group

HighGrid

anarchicisolates(marginal tosociety)

hierarchies(nested/boundedgroups)

LowGrid

individualistic(ego-focusednetwork)

sect/enclave(bounded group

LowGroup

HighGroup

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Culture, Strategy, Organization & StructureContinued

• James Cortada. Best Practices in Information Technology. Prentice Hall, NY, 1998

HighGrid

continuous

systematicproduction

improvement

LowGrid

systematiccustomization

craftculture

LowGroup

HighGroup

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Culture, Strategy, Organization & StructureContinued

• Systematic relationship among strategy, structure, and culture. Paul Bate. Strategies for Cultural Change.Butterworth Heineman, Oxford, UK, 1994.

• Systematic relationship among “levels of culture” -- Artifacts (visible organizational structures and processes), Espoused values (espoused justifications), Basic underlying assumptions (unconscious taken for granted beliefs, thoughts, feelings -- ultimate source of values and actions. Edgar Schein. The Corporate Culture Survival Guide. Josey-Bass, San Francisco, CA, 1999.

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Culture, Strategy, Organization & StructureIn Action

• Imagine combination of academic computing (craft culture), administrative computing (continuous improvement), and the university telephone services (systematic production) into a single integrated organization, which happened at IU Bloomington between 1989 and 1995.

• Imagine then the combination of this organization with a similar organization from the Indianapolis campus (another “tribe” altogether).

• Finally, try to imagine the goal of melding these parts into a combination of continuous improvement and systematic customization.

• Major cultural changes: from a “technology” organization to a “service” organization; from a “take it the way we give it” to a responsive, customer focused organization

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

IU in a nutshell

• Founded in 1820• $2B Annual Budget• 8 campuses• >90,000 students• 3,900 faculty• 878 degree programs; >1,000 majors; > 60 programs ranked within

top 20 of their type nationally• University highly regarded as research and teaching institution

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

IT@IU in a nutshell

• Academic programs in IT through computer science, library and information sciences, engineering and technology, and most notably through new School of Informatics

• CIO: Vice President Michael A. McRobbie• ~$70M annual budget• Technology services offered university-wide• UITS comprises ~500 FTE staff, organized into crosscutting unites

(e.g. finance and HR) and four technology divisions (Teaching & Learning Information Technology,Telecommunications, University Information Systems, Research and Academic Computing)

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

IU IT Strategic Plan

• 10 recommendations, 68 Actions covering all campuses and all IT areas

• Total required for implementation: $205M over 5 years

• A unique charter for Information Technology at a large university that sets the strategic course for the next five years

• http://www.indiana.edu/~ovpit/strategic/

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Activity Based Costs and Management

ABCosts Organization Products

Wages and Benefits

Training

PeopleHardware and Software (Expense or Depriciation)

QualityMaintenance and Other MeasuresContracts

Processes SERVICEExpendable Supplies

Unit Costs

Circuit Charges

Data Knowledge Video Voice

Organization Sustaining Activities

Other Costs

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Sample of ABC for UITS

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

UITS Services — Bloomington CampusFY 98-99

Report on Cost and Quality of Services

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

UITS Services — Bloomington CampusFY 98-99

Report on Cost and Quality of Services

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

UITS Services — Bloomington CampusFY 98-99

Report on Cost and Quality of Services

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Aggregate daily survey results for the IU Bloomington Support Center January - December

1999

Support Service ReceivedSolution

Received Solutionin

Timely Manner

Treated withCourtesy and

Respect

OverallAverage

General SupportLine (n=276)

96% 97% 100% 98%

AdministrativeComputing Line(n=310)

98% 100% 100% 99%

Macintosh Line(n=303)

93% 98% 100% 97%

Intel (PC)Line (n=293)

90% 96% 100% 95%

UnixLine (n=296)

94% 99% 100% 98%

IT Helpe-mail (n=311)

91% 99% 100% 98%

Walk-In(n=255)

95% 98% 99% 98%

ExtendedFace-to-Face(N=109)

98% 99% 99% 99%

TotalsCall Center 94% 98% 100% 97%Walk-In 95% 98% 99% 98%Email 91% 99% 100% 97%Grand Total 94% 98% 100% 97%

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

The 11th Annual IT User Survey at Indiana University Bloomington

1999

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

A Case Study of Activity Based Management

Reengineering E-Mail at Indiana University Bloomington

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Volume and Cost for e-mail services at Indiana University - Bloomington, 1996 - 1998 Academic Years.

1996-97 1997-98 1998-99Number of MailAccounts

66,000 48,000 46,000

Number ofMessages

307,000,000 270,000,000 179,000,000

Mb Received 1,258,000 NA 391,825,000

Cost/Message $0.001 $0.002 $0.002

Cost/Account/Year

$8.96 $9.80 $6.11

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

User perceived quality measures for five mail systems used at Indiana University - Bloomington, 1996-1998 Academic Years

1996-1997 1997-98 1998-1999

PercentageSatisfied

SatisfactionScore

PercentageSatisfied

SatisfactionScore

PercentageSatisfied

SatisfactionScore

Pine 95.1% 4.1 90.5% 3.9 85.8% 3.7

UnixMail(Elm)

82.1% 3.5 82.2% 3.5 62.2% 3.0

Eudora 96.5% 4.2 87.8% 4.1 92.3% 4.1

GroupWise

89.0% 3.8 78.4% 3.8 76.9% 3.5

Exchange 96.3% 4.2 85.0% 3.8 92.5% 4.2

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Comparative measures for e-mail support requests in Indianapolis and Bloomington during the academic year 1998-1999.

1998-99

TotalNumberof Users

Number ofCalls toSupportCenter

Call perUser

Percentageof e-mailCalls toSupportCenter

Percentageof TotalCalls toSupportCenter

Pine IU-B 44,000 1,537 1 per 28.6 23.8% 2.4%

OutlookExchangeIUB

4,500 1,773 1 per 2.5 27.4% 2.7%

Pine IUPUI 45,000 1,943 1 per 22.7 22.7% 4.2%

OutlookExchangeIUPUI

6,000 2,425 1 per 2.5 28.2% 5.3%

I

N

D

I

A

N

A

U

N

I

V

E

R

S

I

T

Y

Indiana University Bloomington