Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.
-
Upload
prosper-oconnor -
Category
Documents
-
view
213 -
download
0
Transcript of Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.
![Page 1: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/1.jpg)
Portfolio Assessment: A literature review
Philip Smyth
English Centre
The University of Hong Kong
![Page 2: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/2.jpg)
Overview
Historical perspectives Defining the portfolio Purposes of portfolios Issues in portfolio assessment Research in portfolio assessment My own possible research avenues
![Page 3: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/3.jpg)
Historical perspective
Portfolios widely used for many years Late 80s interest in portfolios for
assessment (Belanoff and Dickson 1991)
90s saw advent of eportfolios A shift in emphasis away from
assessment to learning?
![Page 4: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/4.jpg)
Definitions
“collection of student work that demonstrates achievement or improvement” (Stiggins 1994)
“a portfolio is a collection of evidence that is gathered together to show a person’s learning journey over time and to demonstrate their abilities” (Butler
2006)
![Page 5: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/5.jpg)
Definitions
“…student writing over time, which contains exhibits showing the stages in the writing processes a text has gone through and the stages of the writer’s growth as a writer, and evidence of the writer’s self-reflection on her/his identity and progress as a writer” (Hamp-Lyons 1996)
![Page 6: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/6.jpg)
Definitions
portfolios are “…prepared with a particular audience in mind”, “…are selective” and “call for judgments” (Calfee
and Freedman 1996)
![Page 7: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/7.jpg)
Definitions
“…a purposeful collection of student work that illustrates efforts, progress, and achievement in one or more areas [over time]. The collection must include: student participation in selecting contents, the criteria for selection, the criteria for judging merit, and evidence of self-reflection” (The Northwest Evaluation Association cited in Barret 2005)
![Page 8: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/8.jpg)
Definitions – main characteristics They are collections of work, different from
a single timed impromptu essay or a class essay carried out over a semester.
They are purposeful in that they “demonstrate”, “exhibit” or provide “evidence” of “achievement”, “improvement”, “the writer’s self reflection”, “the writing process” and “the writer’s growth”.
The degree to which these characteristics are evidenced in portfolios largely depends on their purpose.
![Page 9: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/9.jpg)
Types of portfolio
a process portfolio a showcase portfolio an assessment
portfolio A dossier portfolio A reflective portfolio A classroom portfolio
A positivist portfolio A constructivist
portfolio A personal portfolio A structured portfolio An employment
portfolio A working portfolio
![Page 10: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/10.jpg)
Where are portfolios used?
Primary and secondary classrooms In tertiary settings:
Teacher education Medicine Nursing Engineering Dentistry Psychology ESP/EAP writing classes
![Page 11: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/11.jpg)
How are portfolios used?
In a class Across more than one class Statewide Across a university curriculum
![Page 12: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/12.jpg)
Purposes
Accountability; evaluating program or curriculum effectiveness
Evaluating individual student progress; grading, certifying student accomplishment
Diagnosing students’ needs; informing classroom instructional planning
![Page 13: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/13.jpg)
Purposes
Encouraging teacher efficacy; encouraging reflective practice at the school and classroom levels; supporting teachers’ professional development
Encouraging student efficacy; promoting student self-assessment; motivating student performance
(Herman, Gearhart and Acshbacher, 1996)
![Page 14: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/14.jpg)
Issues
Different audiences (who is the portfolio for?)
Grading – who grades? (fairness) Learning and reflection get lost in
drive to measure competency (Herman and Winter 1994)
![Page 15: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/15.jpg)
Issues
Gap between psychometrics and collaborative nature of the revision process (Song and August 2002,Hamp-Lyons and Condon 2000 )
Time needed for both teachers and students (Callahan 1995, Herman and Winters 1994)
![Page 16: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/16.jpg)
Common problems
Purposes – clear to teacher and student? (Callahan 1995)Mismatch between assessment
criteria and goals of programme of study
Student anxiety and confusion (Butler 2006)
![Page 17: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/17.jpg)
How do portfolios function best?
![Page 18: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/18.jpg)
Research in portfolio assessment
A slim collection?
![Page 19: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/19.jpg)
Research in portfolio assessment
Validity and reliability Fairness Impact
![Page 20: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/20.jpg)
Validity
Vermont programCorrelation ranging from .47 to .58
between writing portfolio scores and direct writing assessments
Similar correlation between portfolio scores and multiple-choice maths test scores
(Koretz 1993 cited in Herman and Winter 1994)
![Page 21: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/21.jpg)
Validity
Gearhart and others (1993) cited in Herman and Winter (1994) found: No relationship when comparing writing
portfolios with standard writing assessments Two thirds of students classified as
“masters” on the portfolio assessment would not have been so classified on the standard assessment.
![Page 22: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/22.jpg)
Validity
Gearhart and others (1993) cited in Herman and Winter (1994) also found: When portfolios were scored in two different
ways (holistic and individual pieces scored) correlations were in the .6 range
Half the students who would have been classified as masters on the single portfolio score would not have been so classified when individual pieces were averaged
![Page 23: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/23.jpg)
Which assessment best represents an enduring capability?
![Page 24: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/24.jpg)
Validity
CUNY (Song and August 2002) 2 groups, 1 assessed by portfolio and writing
test, the other only a writing test Students twice as likely to move to the next
course when evaluated by portfolio At the end of the next course the pass rate
and grade distribution for the two groups were nearly identical
![Page 25: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/25.jpg)
Reliability
Vermont interrater reliability of .28 to .60
Pittsburgh portfolio system ranged from .6 to .7
Herman et al. (1993)found correlations of .82 in an elementary school portfolio containing final drafts of writing
![Page 26: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/26.jpg)
Reliability
Little work on other sources of portfolio reliabilityScore stability over timeStability across different rater groupsThe portfolio set in which a particular
portfolio is rated(Herman and Winter 1994, pg. 51)
![Page 27: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/27.jpg)
Reliability
Heller, Sheingold and Myford (1998) think-aloud protocol on portfolio raters to see if they fit process model of portfolio rating
Found score validity was threatened when a major process was omitted or extraneous assessment criteria were applied
![Page 28: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/28.jpg)
Reliability
Nystrand, Cohen and Dowling (1993) found reliability could be significantly improved if:Raters scored each task in response
to a prompt before moving to the next task and
Raters read several examples together to decide how they were to be rated
![Page 29: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/29.jpg)
Impact
Herman and Winter (1994) based on self-reports from teachers and others implementing portfolios appears to have positive effects on instruction
Vermont principals affirmed that the portfolio assessment program had beneficial effects on curriculum and instruction
![Page 30: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/30.jpg)
Impact
Aschbacher’s (1993) action research cited in Herman and Winter 1994 suggests teacher’s instructional practices and their attitudes towards students changed. Reported ways they thought about their own
teaching Two-thirds of teachers expected higher level
of performance from students
![Page 31: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/31.jpg)
Impact
Hirvela and Sweetland (2005) used 2 case studies showing the 2 students did not strongly endorse the portfolios as used in 2 different courses. Seemed to need more explanations of what
portfolio approaches were meant to achieve Even with a 5% final course grade students
saw the portfolio as essentially summative in nature
![Page 32: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/32.jpg)
Impact
Richardson (2000) study involved classroom observations teacher and student interviews and examination of student writing and teacher response. Found that students regard teacher
responses as directives. Were not prepared to make independent judgments largely because of the threat of grades
![Page 33: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/33.jpg)
The future?
More technical quality Issue of fairness needs to be
addressed More research on impact A move away from psychometric
measurement?
![Page 34: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/34.jpg)
Potential research
Equity of portfolio assessment Validity v reliability Nature of feedback Portfolios for employers
![Page 35: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/35.jpg)
References
Abrami, P. C., & Barrett, H. (2005). Directions for Research and Development on Electronic Portfolios. Canadian Journal of Learning and Technology, 31(3).
Belanoff, P., & Dickson, M. (Eds.). (1991). Portfolios : process and product. Portsmouth, N.H.: Boynton/Cook Publishers.
Calfee, R. C., & Freedman, S. W. (1996). Classroom Writing portfolios:Old, New, Borrowed, Blue. In R. C. Calfee & P. Perfumo (Eds.), Writing Portfolios in the Classroom. MahWah, N. J.: L. Erlbaum Associates.
Callahan, S. (1995). Portfolio expectations: Possibilities and limits. Assessing writing, 2(2), 117-151.
Case, S. H. (1994). Will mandating portfolios undermine their value? Educational Leadership, 52(2), 46-47.
![Page 36: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/36.jpg)
References
Desmet, C., & Cummings, R. (2004). Negotiating the Teaching-Assessment Cycle in Writing Programs with XML. Paper presented at the World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2004 Washington DC.
Frederiksen, J. R., Sipusic, M., Sherin, M., & Wolfe, E. W. (1998). Video Portfolio Assessment: Creating a Framework for Viewing the Functions of Teaching. Educational Assessment, 5(4), 225-297.
Hamilton, S. J. (2006). A Principle-Based ePort Goes Public (and Almost Loses its Principles). In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 434-446). Hershey: Idea Group Reference.
Hamp-Lyons, L. (1990). Second language writing: assessment issues. In B. Kroll (Ed.), Second language writing : research insights for the classroom. Cambridge: Cambridge University Press.
![Page 37: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/37.jpg)
References
Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio: Principles for Practice Theory and Research. Cresskill, NJ: Hampton Press.
Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning about Evidence in Portfolios: Cognitive Foundations for Valid and Reliable Assessment. Educational Assessment, 5(1), 5-40.
Herman, J. L., Gearhart, M., & Aschbacher, R. (1996). Writing portfolios in the classroom : policy and practice, promise and peril. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom : policy and practice, promise and peril
(pp. x, 374 p.). Mahwah, N.J.: L. Erlbaum Associates. Herman, J. L., Gearhart, M., & Baker, E. L. (1993). Assessing
writing portfolios: Issues in the validity and meaning of scores. Educational Assessment, 1(3), 201-224.
![Page 38: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/38.jpg)
References
Herman, J. L., & Winters, L. (1994). Portfolio research: A slim collection. Educational Leadership, 52(2), 48-55.
Hirvela, A., & Sweetland, Y. L. (2005). Two case studies of L2 writers' experiences across learning-directed portfolio contexts. Assessing writing, 10, 192-213.
Holt, D., & Baker, N. W. (1991). Portfolios as a follow-up option in a proficiency-testing program. In P. Belanoff & M. Dickson (Eds.), Portfolios: process and product. Portsmouth NH: Boynton/Cook.
Jafari, A., & Kaufman, C. (Eds.). (2006). Handbook of Research on ePortfolios. Hershey: Idea Group Reference.
Murphy, S., & Camp, R. (1996). Moving towards systemic coherence: A discussion of conflicting perspectives on portfolio assessment. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in the classroom: policy and practice, promise and peril. Mahwah, N.J.: L. Erlbaum Associates.
![Page 39: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/39.jpg)
References
Nystrand, M., Cohen, A. S., & Dowling, N. M. (1993). Addressing reliability problems in the portfolio assessment of college writing. Educational Assessment, 1(1), 53-70.
O'Brien, K. (2006). ePortfolios as Learning Construction Zones. In A. Jafari & C. Kaufman (Eds.), Handbook of research on ePortfolios (pp. 74-82). Hershey: Idea Group Reference.
Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment in an American college. Assessing writing, 10, 61-73.
Pullman, G. (2002). Electronic Portfolios Revisited: The efolios Project. Computers and Composition, 19, 151-169.
Richardson, S. (2000). Students' conditioned response to teachers' response: Portfolio proponents, take note! Assessing Writing, 7, 117-141.
![Page 40: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/40.jpg)
References
Song, B., & August, B. (2002). Using Portfolios to Assess the Writing of ESL Students: A Powerful Alternative? Journal of Second Language Writing, 11, 49-72.
Spalding, E., & Cummins, G. (1998). It was the best of times. It was a waste of time: University of Kentucky students' view of writing under KERA. Assessing Writing, 5(2), 167-199.
Stiggins, R. J. (1994). Student-centred classroom assessment. New York: Merrill.
Wagner, M., & Lamoureaux, E. (2006). Implementing an Outcome-Based Assessment ePortfolio. In A. Jafari & C. Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 539-550). Hershey: Idea Group Reference.
Weigle, S. C. (2002). Assessing writing. Cambridge, UK Cambridge University Press.
![Page 41: Portfolio Assessment: A literature review Philip Smyth English Centre The University of Hong Kong.](https://reader030.fdocuments.in/reader030/viewer/2022032600/56649dc85503460f94abdfaa/html5/thumbnails/41.jpg)
References
Yancey, K. B. (1996). Portfolio as genre, rhetoric as reflection: Situating selves, literacies, and knowledge. WPA 19(3), 55-69.
Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement: Issues and Practice, Spring 2008, 10-24.