E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
-
Upload
richrobles -
Category
Documents
-
view
215 -
download
0
Transcript of E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
1/19
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
2/19
External Pressures for Assessment
Perceived Federal Interest: Spellings Report (2006)
Increased Attention to Higher Education in Ohio:University System of Ohio (Voluntary System of
Accountability, emphasizing the Collegiate Learning
Assessment test; subsidy tied to metrics-student
retention)
North Central Associations Higher LearningCommission: emphasizing student learning outcomes
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
3/19
Dual Pilot study at the University ofCincinnati:
To consider the applicability of learningportfolios (and especially e-portfolios)
alongside the CLA among a group of
first-year Honors students
http://tinyurl.com/uceportfolioproject
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
4/19
The Voluntary System of Accountability
(VSA)
Result of a partnership between the American Association of
State Colleges and University (AASCU) and the National
Association of State Universities and Land-Grant Colleges
(NASULGC), 2007Includes:
Student and Family Information
Student Experiences and Perceptions
Student Learning Outcomes (page 4)
Measure of Academic Proficiency and Progress (MAPP)
Collegiate Assessment of Academic Proficiency (CAAP)
Collegiate Learning Assessment (CLA)
Institution Specific Measures
http://www.voluntarysystem.org
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
5/19
The Collegiate Learning Assessment Developed by the Council for Aid to Education (CAE) in
2004 www.cae.org/cla
A standardized and nationally normed test that holistically
measures the institutional contributions (value added) tovery specific learning gains made by students
Direct assessment of student learning (claims face
validity)
critical thinking
analytical reasoning / problem solving written communication
Most frequently used of the VSA tests
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
6/19
AAC&U VALUE Metarubrics
AAC&U collaboration with faculty and student affairsprofessionals
Developed rubrics for 15 essential learning outcomes
Encouraged institutions to revise to accommodate individualneeds
Development is evolving among institutions
UC employed the Critical Thinking and Written Communicationmetarubrics:
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
7/19
Methodology Study cohort of 111 first-year Honors students
Average ACT is 31
Representing a range of UC colleges and programs
Briefed on experiment
Follow cohort over four-year time frame Administer the CLA as a course requirement
Half complete make an argument and critique an
argument, with the other half completing the problem-solving
performance task
Student post-test survey also administered at testing site During the same month, students completed assignments
designed to measure critical thinking and written communication
Graded by faculty using a variation of AAC&U meta-rubrics
,
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
8/19
Primary Research Questions
How do the CLA results compare to results produced via
VALUE-rubric scoring? In terms of
Measures of individual student learning
Measures of institutional-level value added
As a tool for guiding program improvement
As a response to accountability
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
9/19
Some Findings from the StudentSurvey
Usefulness of the test (waste of time or only mildlyuseful=50%,useful=40%, very useful=10%)
Describe the test experience (unpleasant=37%, indifferent48%, pleasant=15%)
Effort exerted (little or none=8%, some=40%, a great deal52%)
Incentives to take the test seriously (would do it for free=8%,
self-improvement=39%, financial incentive required=43%) Experiences that prepared students to do well on the test:
prep courses for ACT and SAT
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
10/19
Side-By-Side ComparisonCLA VALUE Metarubrics
Grading Human and computer Human
Inter-rater Reliability Systematically achieved Achieved high IRR
through norming
Test Method Timed, open-ended
response
Open ended, over time
Test Incentive Voluntary Homework
Relevance Real world application Academically connected
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
11/19
Side-By-Side Comparison
CLA VALUE Metarubrics
Scoring Holistically Trait-by-trait
Test Time 90 minutes 10 Weeks
Turnaround time 8 months As needed
Motivation Add on Embedded in context
with course
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
12/19
Institutional CLA Data
Mean CLA
Score
Unadjuste
d % Rank
Deviation
Score
Adjusted
% Rank
Total CLA Score 1352 98 1.4 95
Performance Task 1338 98 1.7 96
Analytic Writing Task 1366 98 0.8 79
Make-an-Argument 1348 98 0.3 63
Critique-an-Argument 1383 98 1.4 92
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
13/19
Metarubric Assignments
N Avg Std Dev Avg team
rater
correlation
Overall
Correlation
Written
Communication
109 22.7 3.47 0.717 0.696
Critical Thinking 55 18.98 3.91 0.772 0.824
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
14/19
Correlation
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
15/19
Interpretation and Explanation The lack of statistically significant correlation between the CLA
task scores and the e-portfolio assignment scores indicates a
difference in evaluating student work.
Although both processes measure the application of intellectual
and practical skills, each is evaluating student outcomes atdiffering levels of context.
The level instruments provide very different levels of
aggregation. While the VALUE rubrics provide trait-level analysis
of students, the CLA holistic score considers student
performance.
Differences in the administration of the two instruments must be
considered ( context, incentives, timing, the ability to draft).
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
16/19
Conclusions
The CLA is useful for INSTITUTIONAL assessment
National benchmarking & comparisons for CLA are likely to
be more valid than the more nuanced results of VALUE.
The CLA does not appear to provide data that is particularly
useful for STUDENT diagnostic feedback.
Program assessment will require significantly greatersampling than required by VSA.
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
17/19
Conclusions The Value Rubrics are useful for providing a
consistent framework for the evaluation of student
artifacts. They can help transform grading into the
assessment of specific Student Learning Outcomes Well received by faculty, appropriate assessment for
accreditation and program improvement
Useful in providing some comparability across faculty but
only of limited value for institutional-level reporting because
the rubrics are designed to be modified to meet specificneeds
Provides more timely and detailed feedback
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
18/19
Bottom Line Institutions should employ a complex and varied assessment
strategy to understand achievement of student outcomes
Understand achievement in relation to student outcomes
Demonstrate the value of the educational experience
Assure that students are well-prepared for lifelong learning
and intellectual inquiry Provide a framework for program feedback and improvement
-
8/14/2019 E-Portfolios and Nationally Standardized Tests: Validation or Depreciation?
19/19
Recent Analyses Trudy Banta, The Search for a Perfect Test Continues, in Assessment
Update (November-December 2007)
Joan Hawthorne, Accountability & Comparability: Whats Wrong with the
VSA Approach? in Liberal Education (Spring 2008)
Paul Basken, Electronic Portfolios May Answer Calls for MoreAccountability, in The Chronicle of Higher Education (April 17, 2008)
Trudy W. Banta, Assessment for Accountability: A Cautionary Tale,
presented at The Ohio State University on June 19, 2008
Klein, Liu, Sconing, 2009. Test Validity Study (TVS) Report .
http://www.voluntarysystem.org/docs/reports/TVSReport_Final.pdf
Schulenberger, D., Keller, C. (2009) Interpretation of Findings from the TVS
for the VSA.
http://www.voluntarysystem.org/docs/reports/VSAabstract_TVS.pdf