LESSONS LEARNED from a MULTI-CAMPUS DIVERSITY INITIATIVE

THE USE OF EVALUATION & ORGANIZATIONAL LEARNING FOR EFFECTIVE DIVERSITY

INITIATIVES:

“Organized Anarchy”

The American college or university is a prototypic organized anarchy. It does not know what it is doing. Its goals are either vague or in dispute. Its technology is familiar but not understood. Its major participants wander in and out of the organization. These factors do not make a university a bad organization or a disorganized one; but they do make it a problem to describe, understand, and lead [and as a result evaluate].

-- Michael D. Cohen & James G. March inLeadership and Ambiguity (1974)

Assessment:

• Systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student [and campus] learning and development.

Marchese, T. in Assessment Essentials, Palomba, C.A. & Banta, T.W. (1999)

Empowerment Evaluation:

• The use of evaluation concepts, techniques, and findings to foster improvement and self-determination…it is designed to help people help themselves and improve their programs using a form of self-evaluation and reflection.

David Fetterman in Empowerment Evaluation (1996)

Purposes of Assessment

• Test Assumptions“Our bridge program is highly successful”“We are / are not diverse because…”“Diversity is central to our mission”

• Initiate & Guide Dialogue• Identify Gaps, Disconnects & Potential

Solutions• Develop a sense of ownership and

empowerment• Develop and sustain capacity for

organizational learning

Steps in effective assessment of diversity

initiatives

1. Define the Purpose

2. Primary Audience

3. Evaluation Team

4. Identify Context5. Target Topic

6. Formulate Questions

7. Obtain Data8. Assess Data9. Analyze Data10. Report

Findings

Source: Assessing Campus Diversity Initiatives:A Guide for Campus Practitioners, Garcia, M. et al. (2001)

Utilize Existing Data

• Institutional Research Office

• Vice-President for Planning / Research

• Varied Campus Offices

• Past Strategic Plans

• Self-Studies for Accreditation Agencies

• Past Diversity Initiatives

• Integrated Postsecondary Education Data System (IPEDS) www.nces.ed.gov/ipeds/

Benefits of Varied Types of Data

• Quantitative Data– Helps us explore the “what” and the “who”

(e.g. campus-based surveys)

• Qualitative Data– Helps us understand the “why” and the “how”

(e.g. focus groups; interviews; observations; document analyses)

• Process Data– Helps us understand and explore where pockets

of resistance, inertia, or hope are located– And, whether “IT” is resistance…inertia…or

hope

Assess and Analyze the Data

• Response Rates• Recognize and Qualify Data

Limitations• Background Variables

(e.g. race/ethnicity, gender, socioeconomic status, major etc.)

• DisaggregationGuided by mission / purpose of program

or initiative

Report the Findings

• Review target audience• Format for dissemination

Written reports; oral presentations; internal/external dissemination

• Guide the discourse• Involve critics• Balanced Analysis that emphasizes the

good, the bad, and the ugly• Connect findings with overall institutional

mission

Research must function as a tool for appropriating the codes and cultural symbols of institutional power in an effort to transform institutional environments in the interest of cultural democracy…In this way, researchers who carry out their work in the interest of cultural democracy can function as social advocates, facilitating a production of knowledge that is committed to the creation of institutional conditions where people find their voices and their rightful places as full and equal participants.

-- Antonia DarderInstitutional Research as a Tool for Cultural Democracy

References

• Assessing Campus Diversity Initiatives: A Guide for Campus Practitioners by Garcia, M.; Hudgins, C.A.; Musil, C.M.; Nettles, M.T.; Sedlacek, W.E.; & Smith, D.G.

• Leadership and Ambiguity by Cohen, M.D. & March, J.G.• Assessment Essentials: Planning, Implementing, and Improving

Assessment in Higher Education by Palomba, C.A. & Banta, T.W.• Empowerment Evaluation: Knowledge and Tools for Self-

Assessment & Accountability edited by Fetterman, D.M.; Kaftarian, S.J.; & Wandersman, A.

• Institutional Research as a Tool for Institutional Democracy by Darder A. in Studying Diversity in Higher Education edited by Smith, D.G.; Wolf, L.E.; &Levitan, T.

iALTO!BREAK !!

Nuts and Bolts of Evaluation: How to Use the Process

to Advance Institutional Learning about

Diversity

http://www.aacu.org/irvinediveval/index.cfm

IRVINE FOUNDATION GOALS FOR THE CAMPUS DIVERSITY INITIATIVE

Increase the institutionalization of diversity in the context of educating all students for participation and leadership in a diverse society.

Increase the educational success of disadvantaged and ethnic minority populations.

GOALS AND PURPOSE

http://www.aacu.org/irvinediveval/index.cfm

http://www.aacu.org/irvinediveval/index.cfm

RESULTS

Comparative Six-Year Graduation Rate for 1997 Entering Cohort

0

20

40

60

80

100

CDI-1 CDI-2 CDI-3 National UC CSU

Institutional Sources

Perc

ent

All American Indian African American Latino/a AA/PI White

GOALS AND PURPOSE

PURPOSE OF AN EVALUATION EFFORT:ORGANIZATIONAL LEARNING

Provide ongoing information about the implementation so that mid-course corrections can be instituted.

Build the capacity of campuses to assess and learn from their own progress.

Identity and provide opportunities for campuses to share problems and solutions.

http://www.aacu.org/irvinediveval/index.cfm

ORGANIZATIONAL LEARNING

http://www.aacu.org/irvinediveval/index.cfm

Using institutional data to inform progress

Interrupting the usual

Engage myths that become self-fulfilling prophecies

Maintaining the link between campus diversity efforts and institutional goals for effectiveness and excellence

Taking time for reflection and self-learning

Keeping all campus constituencies informed

Building synergy among many efforts on campus

Approaches evaluation from an organizational learning point of view

Manageable for campus and capable of being maintained

Monitors key goals and elements of proposal

Focuses on institutional issues/change, not simply project-specific issues

Reveals success and problems along the way in both results and processes

Guides interim reports to senior leadership, campus,

board → →

PRINCIPLES

http://www.aacu.org/irvinediveval/index.cfm

PRINCIPLES

• Takes into account:– Institutional differences and stages with respect to

diversity– That institutions vary in mission, needs, goals and

culture– That strategies, goals and emphasis differ– The possibility of taking some risks and learning from

them– Differences within institutions (Disaggregation of

information)• Encourages institutional sharing• Uses outside resources as appropriate

INSTITUTIONAL AUDIT

Education and Scholarship

Access and Success

Institutional Viability and

Vitality

Climate and Intergroup Relations

CONTEXTGlobal Local

http://www.aacu.org/irvinediveval/index.cfm

ACCESS AND SUCCESS

http://www.aacu.org/irvinediveval/index.cfm

Transfer among fields (esp.

SMET)

Success of students -- graduation, persistence,

honors, performance

Pursuit of advanced degrees

Undergraduate/graduate

population by fields and levels

ACCESS and

SUCCESS

Disaggregated

Independent U. Student Retention Rates,

2000-03 (%)(Note: Non resident information not available.)

86%

76%

70%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall 2 Fall 3 Fall 4

All

Four & Six-year Graduation Rates by Race/Ethnicity (%)

Figure 20. Independent U. Student Retention Rates by Race/ethnicity,

2000-03 (%)(Note: Non res ident inform ation not available .)

86%

76%

70%72%

62%59%

79%

71%66%

87%

78%

72%

88%

82%79%

82%

72%67%

92%

78%

69%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Fall 2 Fall 3 Fall 4

All

American Indian

African American

Latino/a

Asian American/P I

White

Unknown

Independent U. Graduation Rates, 2000-03 (%)(Note: Non resident information not available.)

52%56%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

4-Year Grad 6-Year Grad

All

Figure 21. Independent U. Graduation Rates by Race/ethnicity, 2000-03 (%)

(Note: Non resident information not available.)

52%56%

40%

46%44%

61%

52%

63%

58%

67%

51% 51%

42%

36%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

4-Year Grad 6-Year Grad

All

American Indian

African American

Latino/a

Asian American/PI

White

Unknown

CLIMATE AND INTERGROUP RELATIONS

Type and quality of interaction among groups

Quality of experience/

engagement on campus

Perceptions of institution (climate,

commitment, engagement)

CLIMATE AND

INTERGROUPRELATIONS

Disaggregated

http://www.aacu.org/irvinediveval/index.cfm

% “Satisfied with Climate for Minorities” by Attendance in Racial

Cultural Awareness Session:

Racial cultural awareness workshop

noyes

% S

atisfa

ied

with

clim

ate

fo

r m

ino

ritie

s

100

90

80

70

60

50

40

30

20

10

0

47

57

% “Satisfied with Climate for Minorities” by Attendance in Racial Cultural Awareness

Session and Race/Ethnicity:

Race/Ethnicity

% S

atisfie

d w

ith

Clim

ate

fo

r M

ino

ritie

s

100

90

80

70

60

50

40

30

20

10

0

Racial/Cultural Wksh

yes

no

60

38

60

80

62

70

46

62

73

56

EDUCATION AND SCHOLARSHIP

Experience

Presence of diversity-related courses, requirements

Degree to which courses include diversity issues and the placement of such courses

Quantity and substance of student learning about diversity

Course taking patterns of students

Level of faculty expertise on diversity-related matters

Level & diversity of faculty participating in diversity efforts

Availability

Faculty CapacityLearning

EDUCATION AND

SCHOLARSHIP

http://www.aacu.org/irvinediveval/index.cfm

INSTITUTIONAL VIABILITY AND VITALITY

http://www.aacu.org/irvinediveval/index.cfm

Constituency perceptions of institutional

commitment to diversity

Perceptionsof access, equity and inclusion among all

constituents

Public perceptions of the institution

Centrality of diversity in the

planning process, mission

statements

Institutional Strategies and

dedicated resources

Institutional history on

diversity issues and incidentsDiversity of

faculty and staff by level

INSTITUTIONAL VIABILITY AND VITALITY

http://www.aacu.org/irvinediveval/index.cfm

RESULTS

Core Faculty by Race/Ethnicity, 2000 & 2004

0.4 3 4 7 7

83

1 10.6 4 59 8

1 1

80

0102030405060708090

100

Per

cen

t (%

)

2000

2004

Faculty Turnover Quotient

http://www.aacu.org/irvinediveval/index.cfm

RESULTS

TQ = 0%: No Turnover

TQ = 100%: 100% of URM new hires replaced URM faculty who left the institution

)]x100Hires URMNew

URM Period StartURM Period End([1TQ

http://www.aacu.org/irvinediveval/index.cfm

RESULTS

0

50

100

150

200

250

300

Turnover Quotient*

Turnover Quotient* of URM Faculty (Replacement Rate), Individual Campuses, 2000-2004

Institution

http://www.aacu.org/irvinediveval/index.cfm

FACULTYOverall --DISAGGREGATED--over timeNew hiresRetention/turnover--TQLocation --departments/fields

EVALUATION PROCESS

Institutional Goals

ProcessStrategies(Objectives)

REPORTING

Interpretations/PerceptionsLessons Learned

What has been the impact on for example, curriculum, student success, etc.?

What has been done?What has happened?What is working?What is not working?

http://www.aacu.org/irvinediveval/index.cfm

iALTO!BREAK !!

Principles of Evaluation & Institutional Learning

• Framework and Monitoring Progress

• Centrality to Mission• Leadership & Communication• Organizational Learning• Inclusive and Differentiated

Approach• Alignment