2009 Assessment Institute

8/14/2019 2009 Assessment Institute

1/34

E-Portfolios and Nationally

Standardized Tests:

Validation or Depreciation

2009 Assessment InstituteIndianapolis, IN

October 27

Richard Robles | Julie Burdick | Mark Nicholashttp://tinyurl.com/uceportfolioproject


2/34

Presenters

Richard RoblesAssistant Director for First-Year Experience and

Leadership Programs, University Honors Program

Julie BurdickDirector of Academic Planning and Assessment,

Provost Office

Mark NicholasPhD Candidate, CECH


3/34


4/34

About the University of Cincinnati

Top 20 research-intensive university

Liberal arts plus professional colleges

Tradition and culture of decentralized colleges

Current enrollment around 39,000

General Education program began in 2001, with the first

cohort graduating in June 2005


5/34

External Drivers

Spellings Report (2006)

Institutional accountability

Measurement of student outcomes Value-added assessment model

Institutional comparability

Voluntary System of Accountability CLA, MAPP, CAPP

University System of Ohio (2008) Mandate


6/34

Dual Pilot Study

To consider the applicability of learning portfolios(and especially e-portfolios) alongside the CLA among a

group of first-year honors students

Data collection began fall 2008

Applied the written communication and critical thinking

VALUE Metarubrics through the e-portfolio

Follow cohort beyond the first year


7/34

Primary Research Questions

How do the CLA results compare toresults produced via VALUE Metarubricscoring? In terms of..

Measures of individual student learning

Measures of institutional-level value added

As a tool for guiding program improvement As a response to accountability


8/34

Secondary Research Questions

Is one instrument more valuable for students

regarding their undergraduate program of study?

Do e-portfolios link these experiences /

instruments in ways that are meaningful to

faculty, students, or administrators?

What is the diagnostic utility of the two

instruments?


9/34

Methodology

Sample 111 first-year honors students

first-time baccalaureate degree seeking students

Average ACT 31

advanced standing (AP, IB, PSEOP)

Data Collection CLA and e-portfolio assignments embedded in FYE

honors seminar (fall 2008) CLA administered week 4 of the quarter

Students completed test-motivation survey aftertaking CLA


10/34

Use of the E-Portfolio

Opportunity to map progress on the GenEd baccalaureate competencies

Student activities: reflection on learningstyles, goals, accomplishments

Assessment strategy: AACUs VALUEMetarubrics for assessment of specificartifacts


11/34

Use of the E-Portfolio

Nuventives iWebfolio

Embedded in the

honors FYE seminar

Assignments

submitted via

e-portfolio template


12/34

VALUE Metarubrics

AACU collaboration with faculty and student

affairs professionals (LEAP Initiative, 2005)

Developed rubrics for 15 essential learning

outcomes

Encouraged institutions to revise to

accommodate their individual needs

Development is evolving among institutions UC employed the critical thinking and written

communication metarubrics


13/34

Written Communication Rubric

Modifications

Expanded a four scale to five

(to provide a mid-point)

Simplified terminology

Changed text in each criteria to make eachelement distinct as rubric categories overlapped


14/34

Written Communication Rubric

Assessment

Inter-rater reliability of r = 0.6964

Easier to assess because most people know

or are aware of what is good writing Can be assessed outside the context of thenature of a specific artifact or discipline in alimited sense

Norming sessions

Had an expert on the team that guided theevaluation process


15/34

Assessing Critical Thinking (CT)

Nature of Artifact

Proposal for honors experience within

community engagement, global studies,

leadership, research/creative arts

Equivalent to a proposal for an experiment

Part of assigned homework


16/34

CT Rubric Modifications

Changed the scale from four to five (to provide amid point)

Included the dispositional element of open-mindedness

Modified text in criteria to add clarity and detail

Modified the rubric to suit the nature of theartifact


17/34

CT as a construct

Critical thinking is a habit of mind characterized by the

comprehensive exploration of issues, ideas, artifacts, and

events before accepting or formulating an opinion or

conclusion. www.aacu.org

Subject to multiple interpretations which have

philosophical and disciplinary implications

Differing definitions and a lack of definition

among faculty (Paul, Elder & Bartell, 1997*)

* Paul, R., Elder, L., & Bartell, T. (1997). California teacher preparation forinstruction in critical thinking: Research findings and policy recommendations.

California: Commission on Teacher Credentialing.


18/34

CT Rubric Observations

Measures transdisciplinary CT Difficult to separate disciplinary interpretations

of CT (example) Provides guidelines on the nature of

artifacts Very helpful

Advise students on the nature of the artifactand its content (example)

Inter-rater reliability of r = 0.824


19/34

Side-By-Side Comparison

CLA VALUE Metarubrics

Grading Human and computer Human

Inter-rater Reliability Systematically achieved Achieved high IRR

through norming

Test Method Timed open-ended

response

Open ended, over time

Test Incentive Voluntary Homework

Relevance Real world application Academically connected


20/34

Side-By-Side Comparison

CLA VALUE Metarubrics

Scoring Holistically Trait-by-trait

Test time 90 minutes 10 weeks

Turnaround time 8 months As needed

Motivation Add on Embedded in context

with course


21/34

Findings From Student Survey

Usefulness of the test(waste of time or only mildly useful=50%,useful=40%, veryuseful=10%)

Describe the test experience

(unpleasant=37%, indifferent 48%, pleasant=15%)

Effort exerted (little or none=8%, some=40%, a great deal 52%)

Incentives to take the test seriously(would do it for free=8%, self-improvement=39%, financialincentive required=43%)

Experiences that prepared students to do well on the test

Prep courses for ACT and SAT Extensive extracurricular activities


22/34

Institutional CLA Data

Mean CLA

Score

Unadjusted

% Rank

Deviation

Score

Adjusted

% Rank

Total CLA Score 1352 98 1.4 95

Performance Task 1338 98 1.7 96

Analytic Writing Task 1366 98 0.8 79

Make-an-Argument 1348 98 0.3 63

Critique-an-Argument 1383 98 1.4 92


23/34


24/34


25/34

Metarubric Assignments

N Avg Std Dev Avg team

rater

correlation

Overall

Correlation

WrittenCommunication 109 22.7 3.47 0.717 0.696

Critical Thinking 55 18.98 3.91 0.772 0.824


26/34

Correlation


27/34

Interpretation and Explanation

The lack of statistically significant correlationbetween the CLA task scores and the e-portfolio assignment scores indicates adifference in evaluating student work

Although both processes measure theapplication of intellectual and practical skills,each is evaluating student outcomes atdiffering levels of context

Differences in the administration of the twoinstruments must be considered (context,incentives, timing, the ability to draft)


28/34

Findings

Use of the e-portfolio

The platform is an important feature in the use

of metarubrics for feedback and scoring

The platform is best used across the

curriculum

Use of e-portfolio technology presents ease in

evaluation


29/34

Conclusions

While the CLA is useful forINSTITUTIONAL assessment, care mustbe taken in assigning meaning to the

scores The CLA does not appear to provide data

useful for STUDENT diagnostic feedback

While the CLA may meet the requirementsof VSA, it is of limited value for individualstudent assessment


30/34

Conclusions

Use of the VALUE Metarubrics provides agood framework to develop studentdiagnostic feedback

Metarubrics provide a framework for yieldinginter-rater reliability for faculty feedback

VALUE Metarubrics provide more informationthan the CLA for continuous programimprovement


31/34

Conclusions

Institutions should employ a complex andvaried assessment strategy to understandachievement of student outcomes

Demonstrates value of the educationalexperience

Assures students are well-prepared for lifelonglearning and intellectual inquiry

Provides a framework for program feedback andimprovement


32/34

Recent Analyses

Trudy Banta, The Search for a Perfect Test Continues, in Assessment

Update (November-December 2007)

Joan Hawthorne, Accountability & Comparability: Whats Wrong with the

VSA Approach? in Liberal Education (Spring 2008)

Paul Basken, Electronic Portfolios May Answer Calls for More

Accountability, in The Chronicle of Higher Education (April 17, 2008)

Trudy W. Banta, Assessment for Accountability: A Cautionary Tale,

presented at The Ohio State University on June 19, 2008

Klein, Liu, Sconing, 2009. Test Validity Study (TVS) Report.

http://www.voluntarysystem.org/docs/reports/TVSReport_Final.pdf

Schulenberger, D., Keller, C. (2009) Interpretation of Findings from the TVSfor the VSA.

http://www.voluntarysystem.org/docs/reports/VSAabstract_TVS.pdf


33/34

Reflection

If you conducted a similar study at your

institution, what would be your next steps?


34/34

UCDual Pilot Study

http://tinyurl.com/uceportfolioproject

2009 Assessment Institute

Documents

Transcript of 2009 Assessment Institute