Assessing for Transformation

Gary BrownThe Center for Teaching, Learning & Technology

We should not expect the guidance for change of this magnitude—in institutional culture and values—to come from the faculty ranks. After all, faculty are deeply rooted in the traditional values of higher education. Fundamentally, this is a leadership issue.                          --Carol Barone, Vice President

Educause

Agenda

• Assumptions—the need for transformation• To arms, to arms…• About those jobs• What Transformation is NOT• What Transformative Assessment is NOT

• Stories• Goals, Activities, & Processes (GAPs)• Inquisitions and Assessments• Critical Thinking—the rubric paradigm• The Sleeping Bear & Principles of Transformative

Assessment• TAPS

To Arms, To Arms…

• PHOENIX • Phoenix Online is growing faster than

any of the university's 40 physical campuses

• 1,700 online support specialists• 7,000 mostly part-time faculty • 49,400 students

• 70% increase• $64.3 million in net income

Phoenix —from the Chronicle

• “Some critics note that Phoenix strips faculty members of their central role in higher education.” “One of the greatest risks we face as a nation in the growth of new educational providers is the unbundling of teaching, research, and service functions”—Art Levine

• However, counters Phoenix, “Professors at traditional universities who attempt online education are learning as they go, and often give students a bad experience as a result.”

Cost

• The USA currently has 3,500 colleges and universities with an enrollment of 14 million students.

• We spend $175 billion on education.• $12,500 per student • But now, eleven International Mega-

universities serve 3 million students.

Corporate Tax Support

0

20

40

60

80

100

1957 1990 2022

Corporate Share of Taxes

Job Growth: 1992-2005*• Predicted Percentage Increase…

020406080

100120140

Retail Cashiers Clerks Truck

Drivers

Nurses Aids Waiters Home Health Computers

*US Department of Labor Statistics (1993)

Job Growth: 1992-2005*

• Or in Total Numbers

0

200,000

400,000

600,000

800,000

Retail Cashiers Truck Drivers Home Health Technology

*US Department of Labor Statistics (1993)

Job Market Projections*

020406080

100

ProductionService

People Service Symbolic Analysts

*US Department of Labor Statistics (1993)

The Real StoryWe need:

• 9 cashiers for every technical worker.• 1.5 janitors for all the lawyers, accountants

investment bankers, stock brokers, & computer programmers combined.

• “The projected shift … in educational requirements… can be accomplished if those entering the labor force have…one fourth grade level more education than those retiring from the labor force.”

Economic Policy Institute

College Graduates @ Work

0

50,000

100,000

150,000

200,000

Retail Housekeeping Truck Drivers

*US Department of Labor Statistics (1993)

Already, In Fact…

% College Graduates working in Blue Collar Jobs

0

20

40

1968 1990*US Department of Labor Statistics (1993)

What Transformation Is NOT

Attrition Rates in WSU Distance Courses

0

10

20

30

40

fall1999

spring2000

fall2000

spring2001

fall2001

spring2002

Online Video

%

Student Enrollments in WSU Online Learning Spaces

0

4000

8000

12000

16000

20000

1998 1999 2000 2001 2002

Transformation?

What Transformative Assessment is NOT

Evaluations as Barriers to Improvement

• Many evaluation instruments are subtly biased in favor of traditional instruction.• They often penalize innovators,

measuring what faculty aren’t doing but failing to measure what they are doing.

Purpose of Evaluation is Not Clear

• Most evaluation focuses on personnel decisions.

• Inconsistent interpretation and use of evaluation results.

• Little focus on areas where that might best benefit from feedback

• Evaluation usually occurs at the very end of a semester

—Too late for change

Students Are in the Dark, Too

• The traditional evaluation process is not designed to help students become more involved with what will really help them learn.

• Students don’t take evaluations seriously.

Teaching-Learning Practices are NOT Linked to Learning

• Course evaluations rarely focus on practices that have been shown to produce better learning outcomes.

• Assessment rarely improves learning.

No Link Between

Evaluation & Faculty Development

• Inadequate support for addressing results. Literature from past 10-15 years emphasizes staff development as key to effective evaluation practices that promote teacher growth and improvement” (1997, Annunziata in Stronge, pp 289)

• Administrators frequently calculate an average score for all evaluation items and then rank faculty.

• The method does not foster dialogue among faculty about good practices.

The Goals, Activities, and Practices

• Invite faculty to join in the process of formulating the assessment.

• A series of three short, online surveys designed to provide formative assessment: one instructor survey and two student surveys.

The 3 Survey Goals1. To help faculty gather useful data about their

teaching goals, values and strategies…

2. To help faculty learn how those strategies address and influence their students’ goals, values, and learning behaviors…

3. And to examine the interaction between goals and practices that faculty might share with each other

—and so establish a culture of evidence.

GAPs: Encouraging Participation• Brown Bags - to initiate the conversation on assessing

teaching and learning.• We emailed each faculty approximately 10 times • Student surveys were linked to course sites of

instructors who answered the instructor survey.• We analyzed and returned results promptly...• Regular updates were sent to instructors with their class

results and overall results.• We offered and have co-authored papers with faculty...• We have talked with chairs, deans, and the assessment

coordinator’s to include GAPS in teaching portfolios.• We wore out our shoes…

Sample Findings• Significant mismatch between faculty and student goals• Online courses are significantly more effective than video based courses• Designed courses are significantly more likely than courses not formally designed to evidence principles

of good practice• Faculty motivation predicts perceptions of efficacy

• Faculty who want to keep abreast of the scholarship of T & L report that online learning is positive• Faculty who teach online primarily for the money …

• Student age and gender predict perceptions of testing efficacy

Time & CostEstimated Hours

01020

304050

607080

90

Design Development Delivery Assessment

Actual Design Target

The Assessment Gold Standard

Participants Who Used GAPs Data to Transform….

563

10655 16 2

0

300

600

Invited faculty Faculty who tookinstructor survey

Faculty w/ studentresponses

Faculty w/ >50%student response

rate

Faculty who usedthe data

However….• One of the two programs endured a budget related,

faculty initiated inquisition…..• Demonstrated, by virtue of GAPS, systematic, formative

data gathering and responsiveness to that data…• Demonstrated evidence of good practice.• Demonstrated the “creative use of technology” and gains in

critical information literacy• Demonstrated the program was cost effective

• The program improved freshman retention• Demonstrated the program improved student grades,

including “special admits” expected to struggle.

The Learning Context: Student Performance @ WSUCum GPA by Admissions Quartiles

1st Quartile2nd Quartile3rd Quartile 4th Quartile

English

GenEd

Math

FS

0

1

2

3

4English

GenEd

Math

FS

Which has resulted in….• The faculty committee that initiated the inquiry was commended by

administration for its attentiveness to issues of quality….• They have subsequently been invited to examine other programs… • Which has transformed the CTLT• We are seeing the inklings of a culture of evidence sprouting in a rough,

research dominated terrain…

small steps….

• Guide faculty grading

• Guide student learning

• Provide measures of growth

The Rubric Paradigm

Dimensions of Critical Thinking1. Identifies and summarizes the problem/question at issue (and/or the

source's position).

2. Identifies and presents the STUDENT’S OWN perspective and position as it is important to the analysis of the issue.

3. Identifies and considers OTHER salient perspectives and positions that are important to the analysis of the issue.

4. Identifies and assesses the key assumptions.

5. Identifies and assesses the quality of supporting data/evidence and provides additional data/evidence related to the issue.

6. Identifies and considers the influence of the context * on the issue.

7. Identifies and assesses conclusions, implications and consequences.

Critical Thinking and Measures of Growth

Emerging____________________________ Mastering

Identifies and summarizes the problem/question at issue (and/or the source's position).

Does not identify and summarize the problem, is confused or identifies a different and inappropriate problem.

Does not identify or is confused by the issue, or represents the issue inaccurately.

Identifies the main problem and subsidiary, embedded, or implicit aspects of the problem, and identifies them clearly, addressing their relationships to each other.

 

Identifies not only the basics of the issue, but recognizes

nuances of the issue.

Critical Thinking:8 Courses: 4 with CT, 4 w/o CT

0

1

2

3

4

5

6

Problem StudentView

Other Views

ContextEvidence

AssumptionsImplications

Average

Upper Division CT Lower Division CT No CT

Critical Thinking:8 Courses: 4 with CT, 4 w/o CT

2

2.5

3

3.5

4

Problem StudentView

Other Views

ContextEvidence

AssumptionsImplications

Average

Upper Division CT Lower Division CT No CT

Critical Thinking:One course—two semesters

0

1

2

3

4

5

6

Problem StudentView

Other Views

ContextEvidence

AssumptionsImplications

Average

Entomology--spring 99 Entomology--spring 00

Faculty Development

• The critical thinking rubric was valuable for helping faculty articulate their goals and communicate expectations to students.

• Faculty who used the rubric were enthusiastic and expressed plans to integrate the rubric more intensively in future courses.

Critical Thinking Study—Results

• Gains in courses when rubric is used—when the faculty in this project integrated the WSU Critical Thinking Rubric into their instruction and assessment, evidence of student gains in critical thinking increased dramatically.

• Gains from freshmen to junior years—Critical thinking was significantly higher among juniors than among freshmen.

• But even the writing of juniors had only a mean of 3.1 on a 6 point scale.

Additional Findings & Implications

• The dimension of least gain was in students’ abilities to articulate their own viewpoints.

• The greatest gains by juniors reflect improved abilities to analyze issues from multiple perspectives.

• Comparisons to WSU’s writing assessment— As critical thinking scores rise, writing placement

scores and portfolio exam scores sink...

• The faculty questionnaire revealed a focus on grading over fostering critical thinking for broader life-long learning.

Transformation?

The Sleeping Bear

• Those most oblivious of the teeth are often the first to fodder…

• The growling bear will elicit benchmarks & comparisons

• The bear will feast on standardization

• But bears are omnivorous…

Emerging Principles of Transformative Assessment

• Institutional leadership is imperative• Assessment focuses on institutional efforts to

provide students with rich learning experiences• Assessment includes student reports of their own

increasingly unique experiences• Qualitative measures are valued and may be

supported by quantitative measures• Emphasizes student learning defined by

development and change over time• Dissemination engages the responsibility for

shaping as well as reflecting society needs• “High Standards” is not the same as Standardization

Dimensions of Transformation

1. Purpose2. Data Acquisition3. Application4. Dissemination

Criteria for Rating Transformation

Rater: Project:

* Faculty * Designer

* Student * Assessment Specialist

* Community Colleague * Administrator

* Other _________

Rating

Assessment Purpose

Data & Data Acquisition

Application

Dissemination

Bonus—Faculty Rewarded for Assessment

Total Score

Comments:

The Scoring Form

U#1 U#2 U#3 U#4 U#5 DimensionAverage

Purpose 3 3.1 3.6 4 3 3.34Data 2.4 3.5 3.8 3.5 3 3.24Application 2.5 3.4 3.7 3.5 4.5 3.52Dissemination 3.3 3 3.5 2 4.5 3.26University Average

2.8 3.3 3.7 3.3 3.8 3.343.38

Pilot Findings

Pilot Findings

0

1

2

3

4

5

6

PurposeData

ApplicationDissemination

Average

U # 1

U # 2

U # 3

U # 4

U # 5

Key CommentsU#1

• Doesn’t stretch student learning or acknowledge what students may bring.

• Focus satisfies industry’s demand for skills.

• While faculty are included in this process, audience is clearly the upper echelon of the institution and future employers.

Key CommentsU#2

• Not many rewards for assessment.

• No mention of disseminating results beyond unit; no feedback to faculty and students.

• Limited to classes.

Key Comments U#3

• Assesses key concern: learning to learn.

• No plan to apply or fund results of findings. Faculty support neglected.

• Feedback to institution and external agencies is minimal—how are constituents engaged in process?

Key Comments U#4

• Design for infrastructure for hybrid classes—assumptions that tech alone equates with transformation

Key Comments U#5

• Good start but…