Developing a Portfolio Assessment: Pacesetter Spanish · portfolio component, with the following...
Transcript of Developing a Portfolio Assessment: Pacesetter Spanish · portfolio component, with the following...
Developing a PortfolioAssessment:
Pacesetter® Spanish
ANDREA FERCSEY, CARMEN LUNA,EVA PONTE, and PABLO ALIAGA
College Board Report No. 99-2
College Entrance Examination Board, New York, 1999
College Board Report No. 99-2
Developing a PortfolioAssessment:
Pacesetter® Spanish
ANDREA FERCSEY, CARMEN LUNA,EVA PONTE, and PABLO ALIAGA
Acknowledgments
We would like to acknowledge the six Pacesetter®
Spanish teachers who participated in Phase I of thisresearch project: Eva Chapman and Ruth Ann Kerkoc,Leland High School, San Jose, California; GildaNissenberg, North Miami Beach High School, Florida;Dianne Michel Stenroos, Rutland High School,Vermont; Carol Thorp, East Mecklenburg High School,Charlotte, North Carolina; and María Elena Villalba,Miami Palmetto Senior High School, Florida.
We would also like to acknowledge the followingPacesetter Spanish teachers who participated in Phase IIof this research project: Esperanza Cobián-Bashara,Silver Creek High School, San Jose, California; SharonDyrland, Surrattsville High School, Clinton, Maryland;Rosalba García, Bellaire High School, Bellaire, Texas;Ruth Ann Kerkoc, Leland High School, San Jose,California; Dennis Lavoie, Fairport High School,Fairport, New York; Gilda Nissenberg, North MiamiBeach High School, Miami, Florida; Nereida SamudaZimic, William B. Travis High School, Austin, Texas;Carol Thorp, East Mecklenburg High School, Char-lotte, North Carolina; and María Elena Villalba, MiamiPalmetto Senior High School, Miami, Florida.
The contributions and efforts of these individualswere essential throughout all stages of the project.
The authors would like to thank Carol Myford, RickMorgan, and Gita Wilder for their help in interpretingand analyzing data, as well as for their review of anearlier draft of this paper.
Andrea Fercsey is an assessment specialist at ETS. Carmen Luna is a group leader at ETS.Eva Ponte was a 1996 summer intern at ETS and agraduate student at the University of California atBerkeley.Pablo Aliaga was a 1997 summer intern at ETS and isa graduate student at the University of Michigan.
Researchers are encouraged to freely express their pro-fessional judgment. Therefore, points of view oropinions stated in College Board Reports do not neces-sarily represent official College Board position or policy.
Founded in 1900, the College Board is a not-for-profiteducational association that supports academicpreparation and transition to higher education forstudents around the world through the ongoing collab-oration of its member schools, colleges, universities,educational systems and organizations.
In all of its activities, the Board promotes equitythrough universal access to high standards of teachingand learning and sufficient financial resources so thatevery student has the opportunity to succeed in collegeand work.
The College Board champions—by means of superiorresearch; curricular development; assessment; guidance,placement, and admission information; professionaldevelopment; forums; policy analysis; and publicoutreach—educational excellence for all students.
Additional copies of this report (item #200272) may beobtained from College Board Publications, Box 886,New York, New York 10101-0886, (800) 323-7155.The price is $15. Please include $4 for postage andhandling.
Copyright © 1999 by College Entrance ExaminationBoard and Educational Testing Service. All rightsreserved. College Board, Advanced PlacementProgram, Pacesetter, SAT, and the acorn logo areregistered trademarks of the College EntranceExamination Board.
Printed in the United States of America.
ContentsAbstract...............................................................1
PHASE I REPORT....................................................1
Introduction ........................................................1
Background .........................................................1The Pacesetter® Program ........................................1Pacesetter Spanish ...................................................2Portfolio Assessment ...............................................2
Pacesetter Spanish Portfolio Research Project (Phase I) .................................4
Method ...................................................................4Portfolio Analyses...................................................7Description of Elements..........................................7Distribution of Ratings ...........................................7Time Used to Read Portfolios.................................8Culminating Assessment .........................................9Portfolio Assessment Matrix Correlations ............10Portfolios Read Twice: Reliability Study...............11Third Rater Analysis .............................................15Selection of Benchmarks .......................................15
Summary of Findings.........................................16
Recommendations .............................................16
References .........................................................16
Phase I Appendix ..............................................37
Tables1. Distribution of Ratings ..................................72. Distribution of Ratings by Aspects
and Dimensions .............................................83. Time Used to Read Each Portfolio .................93A. Summary........................................................94. Average Time Employed to Read Portfolios
From Different Teachers ................................95. Average Time Employed by Each Rater .......106. Time Employed to Read Portfolios
at Different Sessions.....................................107. Culminating Assessment and
Dimension 2 Portfolio Ratings.....................108. Portfolio Assessment
Matrix Correlations .....................................119. Summary of Portfolio
Score Discrepancies ......................................1310. Summary of Portfolio Score Discrepancies
for Each Aspect in Each Dimension .............1311. FACETS OUTPUT (7.1.1) Students
Measurement Report (Arranged by FN) ......1412. Distributions of Raters’ Judgments ..............1513. Portfolios Read Three Times........................15
Figure1. FACETS map for the whole scale
(21 portfolios, two readers) ..............................12
PHASE II REPORT.................................................17
Rationale and Expected Outcomes....................17
Pacesetter Spanish Portfolio Research Project (Phase II)..............................18
Method .................................................................18Summary of Ratings..............................................19Time Used to Read Portfolios...............................21Distribution of Ratings for Each Rater .................21Portfolios Read Twice...........................................22Correlation Among the Ratings of Portfolios .......27Reliability Study....................................................27Analyses ................................................................29
Students ............................................................29Raters ...............................................................33Aspects and Dimensions ...................................33
Summary of Findings.........................................34
Recommendations .............................................34
References .........................................................35
Phase II Appendix.............................................67
Tables1. Frequency of Ratings in
Each Rating Category.....................................192. Frequency of Ratings in Each
Category for the Aspects of Dimension 1 .......203. Frequency of Ratings in Each
Category for the Aspects of Dimension 2 .......214. Time Employed by Each Rater in Minutes .....225. Total Count and Relative Frequency by
Category and Rater ........................................236. Comparison of Portfolios and Raters .............247. Discrepancies Between Raters.........................268. Specific Discrepancies Between Raters ............269. Bivariate Correlation Matrix ..........................27
10. Rater Severity.................................................2911. Percentage in Each Scale Category .................33
Figures1. FACETS: Partial credit model for raters .......282. FACETS: Partial credit model for aspects.....303. FACETS: Rating scale model........................313A. FACETS: Student adjusted scores .................32
1
AbstractPortfolios are one of the assessment tools used inPacesetter® Spanish. In Phase I of this study, conducted in1995–96, a first attempt was made to develop a stan-dardized portfolio assessment system. As part of thissystem, a set of guidelines and an assessment matrix wereprepared. The assessment matrix was piloted during aportfolio reading in which six teachers participated. Atthe end of the reading, some portfolios were identifiedas benchmarks to be used for training purposes.
After the reading, results were compiled and analyzed.Data were collected to investigate several aspects of theportfolio assessment system (i.e., scores, the reliability ofthe raters, correlation with the Culminating Assessment,and time needed to read portfolios). Our purpose was todetermine which components of the portfolio assessmentsystem were working as intended and which might needto be changed. To this end, we explored ways ofgathering preliminary evidence of the reliability andvalidity of the portfolio assessment system. Our findingssuggested that refinement of the matrix was needed aswell as clearer guidelines for the selection of artifacts toinclude in the portfolios.
In Phase II of our research, in consultation withparticipating teachers, a revised set of guidelines and arevised assessment matrix were distributed to allschools implementing the Pacesetter Spanish course in1997. Seven teachers (three of whom had also been apart of Phase I of this study) participated in a portfolioreading in 1997, at which both the guidelines and thematrix were further refined for immediate dissemina-tion to the schools. A new set of benchmark portfolioswas identified, and a set of sample portfolios was pre-pared for use in subsequent professional developmentsessions for Pacesetter Spanish teachers.
In general, the results of our findings are encour-aging. Initiating national and/or regional portfolioscoring sessions would greatly enhance the validity andreliability of the portfolio component as an integral partof the Pacesetter Spanish program.
PHASE I REPORT
IntroductionThe Pacesetter® program developed by the College Boardand Educational Testing Service (ETS) is an integrated setof learning outcomes, course materials, and instructionalexperiences, including various approaches to assess-ment, and is currently being offered in three subjectareas: English, Spanish, and Mathematics.
The Pacesetter Spanish program has provisions forstudent self-assessment, peer assessment, local assess-ments, and end-of-course standardized assessment, aswell as portfolio assessment. All Pacesetter Spanish stu-dents are expected to assemble portfolios with productsfrom their work throughout the course in order to showevidence of what they have learned.
From March to April 1996, the Pacesetter Spanishprogram conducted a research project on the assessmentportfolio component, with the following goals: (a) toprepare a working definition of Pacesetter Spanish port-folios as assessment tools, (b) to develop a standardizedportfolio assessment system, and (c) to utilize theresearch findings for upcoming professional develop-ment sessions for teachers. This paper describes thelatest developments in these three areas.
Background
The Pacesetter® ProgramThe Pacesetter program is part of the College Board’sefforts to promote achievement of educational excellencefor all students. The College Board defines Pacesetter asa school-based instructional assessment program thatintegrates high academic standards for all students,teacher preparation to help achieve those standards, anda broad range of assessments measuring student achieve-ment and facilitating instructional decisions.
The Pacesetter program is structured around threeessential components. First, it includes course frame-works specifying content curriculum, which detail theknowledge and skills students are expected to master.Teachers and students are encouraged to have a sharedunderstanding of what content areas are to be coveredthroughout the course. Second, it provides professionaldevelopment sessions for teachers, which include avaried range of activities. Third, the program incor-porates several types of assessment, ranging fromembedded assessments to standardized assessments.
The College Board will conduct research toexamine the program outcomes: how the programprepares students for future careers as well as how itprepares students for other College Board assess-ments, namely SAT® I: Reasoning Test, SAT II:Subject Tests, and Advanced Placement Program®
courses and examinations. As part of programimplementation, research will focus on how teachersadopt Pacesetter classroom strategies. Finally, thewhole assessment process will be monitored toprovide feedback for further revisions to instructionalmaterials and methodology.
2
Pacesetter SpanishThe aim of Pacesetter Spanish is to provide third-levelhigh school students with a contextualized approachto language mastery. The program integrates learningoutcomes, course materials and learning experiences,and various types of assessment.
Because the program embraces a contextualizedapproach to learning, course objectives go beyond thoseestablished by traditional instruction. The three mainexpected course outcomes are: (a) to use the Spanish lan-guage to acquire knowledge, (b) to understand Hispaniccultures, and (c) to use Spanish effectively to communicatewith others. These outcomes support the five goals of theNational Standards for Foreign Language Learning. Theyfocus as much on the process of acquiring new informa-tion and cultural skills as they do on acquiring specificknowledge in specific domains of the language.
Pacesetter Spanish covers a broad range of content.Students learn about the contributions of variousHispanic figures in different fields. The curriculummaterials facilitate students’ ability to make connectionsbetween different aspects of Spanish-speaking cultures,as well as to gain interdisciplinary knowledge. Theprogram has a holistic approach to language skills.
The materials of the course framework are themati-cally linked. Students read authentic texts, engage indiscussions, develop research projects, and experienceaesthetic works (music, arts, etc.). Teachers may chooseto supplement course materials in different ways, and touse additional sources of information, the resultingproducts presented as evidence of students’ learning.
The role of students in the program varies somewhatfrom their role in traditional courses. Students areencouraged to work both independently and in groups,to conduct research projects using different resources,and to approach learning from an interdisciplinaryperspective (i.e., they use knowledge and inquiry skillsfrom other areas of the curriculum to help them learnSpanish). They also take an active role in the evaluationprocess by monitoring and evaluating their own perfor-mance. Pacesetter Spanish empowers students to beautonomous, self-directed language learners.
Finally, assessment aims at both the processes andthe products of instruction. Assessment is integratedinto the course, taking alternative approaches beyondtraditional tests. Students are active participants in theprocess, conducting self-assessment through learninglogs and journal activities, and through peer assess-ments. Local assessments are designed by teachers toevaluate student performance and teacher effectiveness.The Culminating Assessment, a standardized com-ponent provided at the end of the course, may serveaccountability purposes. Students are expected to main-
tain a portfolio with samples of their work to showevidence of achievement in the various aspects of thedimensions assessed through this component.
These new approaches to instruction demand notonly a change in the curricula and the role of students,but also in the role of teachers. Teachers are expected(1) to teach a language course not centered in thelanguage itself, (2) to teach an integrated and inter-disciplinary course with a holistic approach, (3) toimplement cooperative learning approaches in theirclassrooms, (4) to focus on process as much as onproducts, (5) to be able to provide different sources ofinformation, (6) to be able to search for differentsources of evidence of students’ performance, (7) toteach students different learning strategies, (8) to use thescoring rubrics appropriately, (9) to adequately manageportfolios, and (10) to promote student self-assessmentand self-learning.
To attain this competency, participating teachershelp define (a) the knowledge and skills necessary forstudents’ success and (b) ways to find evidence ofstudents’ learning. These teachers also help createscoring rubrics. To teach Pacesetter Spanish, all partici-pating teachers must attend professional developmentsessions before the beginning of the academic year andare also expected to attend midyear meetings. Bothare designed to give ample opportunities for the par-ticipants to exchange suggestions on pedagogicalapproaches and techniques, as well as to give feedbackon the current approach to the content of coursematerials. It is to be noted that the leaders and trainersat these sessions are themselves teachers implementingthe Pacesetter Spanish program in their classrooms.
In the 1996–97 school year, the Pacesetter Spanishprogram was taught in approximately 25 districts, 60 schools, by 80 teachers, to 4,000 students.
Portfolio AssessmentThe United States educational system has faced severalproblems in the past. Resnick and Resnick (1992) havepointed out the link between testing efforts andeducational reforms in this country. This link hasfocused attention on school assessment practices andits problems. Some assessment alternatives have beenformulated by the educational community in an attemptto overcome these problems, including performance-based assessment.
Performance-based assessment involves both instruc-tional processes and products. One of its main featuresis that students create or construct complex responsesand products. Answers can no longer be considered assimply right or wrong. Alternative answers given by
3
students and teachers use evidence from different sourcesand media. Because this process generates complexresponses, human judgment is required to evaluatestudents’ responses along a continuum of achievement.Performance-based assessment can be linked to and em-bedded in instructional practice, and it is expected thatthis link will allow assessment to inform and guideinstructional practice.
Portfolio assessment is a special type of performance-based assessment. A portfolio is a selected collectionof students’ work over time evaluated by one or moreraters using clear criteria. In some cases, portfoliosmay also include students’ reflections about their ownwork and progress. Portfolios, like most types of perfor-mance-based assessments, are considered tools to guideinstructional practice. If a portfolio assessment system iscorrectly implemented, instruction should be expectedto: (1) pay more attention to the process of learning, (2)lead teachers to think, evaluate, and modify their ownpractices, (3) change students’ traditional role from apassive one to one in which they take charge of theirown learning and the evaluation of that learning, (4) beguided by current theories of development and theprocess of learning, and (5) make explicit what contentis valuable for both teachers and students.
To promote this change, portfolios should be morethan a pile of schoolwork. Gitomer and Duschl (1995)indicate that good portfolio practice requires funda-mental changes in conceptions of (scientific) knowledge,teaching, learning, and assessment. They explain indetail what these changes mean in the area of scienceeducation. Some of their claims can be applied to othereducational domains. First, curricula should reflect thenature of the discipline being taught; students shoulduse procedures, methods, and practices similar to thoseused by professionals in the discipline. The dynamicsand interactions found in the field should also be anintegral part of instruction. Second, it is necessary totake a constructivist approach to learning: to haveconceptual change as the goal of learning. Third, assess-ment should serve the needs of students and teachers aswell as others who have a stake in what happens in theclassroom. Criteria used to consider student perfor-mance should be explicit, and assessment must meetthree conditions: (a) it should focus on knowledge andskills considered important within the discipline, (b) itshould contribute to instruction and learning, and (c)it should serve accountability purposes.
Arter and Spandel (1992) state that portfolios willonly have the desired effects if they are carefullyplanned. They give a definition of portfolios thatincludes multiple elements and lists requisites for a goodportfolio system. First, it is necessary for users to have aclear idea of the purpose of the portfolio, since different
objectives will lead to quite different portfolios. Second,students should participate in the selection of theportfolio content. This selection process demands self-reflection from students, forcing them to analyze theirown work and what elements in that work best serveto demonstrate their knowledge and skills. To do this,students should have clear guidelines of what artifacts toselect as part of their portfolios. Third, assessmentcriteria must be fully and carefully defined and open toall. Finally, assessment and instruction should form anintegrated whole. Portfolios can monitor students’accomplishments as well as provide for new and betterways for students to improve on their achievements.
Portfolios define the teacher’s role in a “new” way:Teachers are considered professional researchers studyingtheir own practice. (This view of teachers was alreadyproposed by Elliot [1985].) This new definition of teachingrequires teachers to revise and change their beliefs andapproaches to instruction, learning, and assessment, andto rethink instructional goals. Without such changes itwould be difficult to attain the goals set by portfolioassessment. Sheingold, Heller, and Paulukonis (1995)identified the following five categories in which teachersreported changes: (1) using new sources of evidence, (2)sharing responsibility for learning and assessment, (3)changing goals of instruction, (4) using new ways ofevaluating evidence, and (5) changing the general viewof the relationship between assessment and instruction.
With respect to students, portfolios are beneficial inseveral ways. First, students tend to find them moti-vating and engaging. Second, through portfoliosstudents become more aware of and responsible fortheir own learning and are able to make more informedchoices about their work and how to approach it.Third, these students—more than students in traditionalassessment settings—tend to collaborate with otherstudents and with teachers. Fourth, different ap-proaches and levels of learning are allowed within theportfolio framework; students are given a broader rangeof opportunities to learn and to demonstrate theirlearning. Also, because portfolios pay more attention tohow student work compares to established standardsinstead of to the work of other students, a less compet-itive environment is created. Finally, because portfoliocontents tend to focus on what is important to learn,and for what purposes, students are better prepared forwork life and life in general.
Despite their promising outlook, portfolios offer nopanacea. Several problems may preclude their being asbeneficial for education as could be expected, the mostimportant problem one of implementation. In thisrespect, there is a big gap between theory and practice.Teachers, the principal catalysts of portfolios, must beprepared to perform their new assigned role. They must
4
be able to (a) clearly define the purpose of instruction andundergo all the above-mentioned changes, (b) clarifycomplex tasks and provide students with additionalsources and media of information, (c) be able to findevidence of students’ learning in different ways, makingcontinuous and accurate judgments, and (d) continuouslyrevise and adapt the instructional process. Some teachersfind this an enormous responsibility in terms of invest-ment of classroom time, professional development, andscoring. This may explain the reluctance of some teachersto implement portfolio assessment in their classrooms.
However, teachers are not the only ones who mayfind it difficult to pursue a portfolio assessment model.The general public is also concerned about how reliable,valid, and fair portfolio scores are for accountabilitypurposes. These issues may be even more critical whenportfolios are used in large-scale assessments thanwhen they are used as an assessment tool in the class-room. Portfolio scoring becomes an essential part of theportfolio system in terms of reliability, validity, andfairness. The first step in portfolio scoring is to developrubrics and construct an assessment matrix. The rubricsshould include both the content and the processesstudents are expected to master. Rubrics should alsodescribe what kind of evidence we should expect for thedifferent levels on the performance scale. Ideally,teachers should be involved in the process of definingand refining the rubrics. Some studies have been con-ducted to study how consistent and accurate readers arein their judgment (see Bridgeman, Chittenden, andCline (1995); Myford and Mislevy (1995); and Koretz(1992, 1994)). Reliability has usually been reportedin terms of inter-rater agreement (i.e., the correlationbetween the scores two different raters give to the samestudent work). When discrepancies between these ratersare large (the criteria for “large” depends on the scaleused), usually a third reading is done. The use of a thirdrater reduces the impact of rater severity on a student’sscores (i.e., how strict a rater is when evaluating astudent’s work), but this procedure does not resolve theproblem of rater interchangeability. For example, ifboth raters are extremely severe or lenient, thisprocedure will not be able to detect that, and the finalscore given to the student may be erroneous. Myford,Marr, and Linacre (1996) propose to calibrate ratersaccurately and precisely, adjusting examinees’ scores forrater severity differences, hence improving reliabilitycalculations by removing the errors in those scoresassociated with rater severity.
Statistics can point out the problems; however, tofind out the nature of these problems, a naturalisticstudy is needed. Mislevy and Myford (1995) conductedboth a statistical and a naturalistic analysis. They identi-fied some challenges that raters faced, as well as specific
types of portfolios that were difficult to evaluate.Among the challenges faced by raters, Mislevy andMyford included: (1) the “empathy mode,” a tendencyto evaluate students in terms of potentials; (2) the“bounce effect,” the effect the last portfolio read has onthe reading of the next one (i.e., comparison to anotherstudent is used instead of analyzing the student’swork by defined rubrics); (3) the overuse of irrelevantbackground knowledge for decision making; (4) the useof the middle levels of the scale, which the authors call“play it safe”; and (5) having many or few experienceswith the medium or style in which the student isworking. With respect to portfolios, challenging port-folios are those in which there is a lack of consistency(i.e., good ideas but not enough technique to developthem), different levels of abilities, different sources ofevidence, and unique or very special portfolios.
Pacesetter Spanish PortfolioResearch Project (PHASE I)MethodPortfolios are one among several assessment modesincluded in Pacesetter Spanish, and they provide teachersand students with a tool of great instructional andmotivational value. In Pacesetter Spanish, a portfolio isa collection of significant samples of student work overtime, accompanied by clearly stated evaluation criteriaand students’ reflections on their own learning progress.Most of the activities and final projects of the units inthe Pacesetter Spanish course materials are consideredacceptable choices for inclusion in students’ portfolios.Because of the variety of these activities, portfoliosmay contain a rich array of samples, including videosof student reports and role playing, audio excerpts ofinformal and formal interviews and presentations, draftsand final versions of written work, collages, and otherdescriptions of projects.
Portfolios can yield valuable data about students’progress and lead them effectively through theirlanguage learning. They function as guides by allowingstudents to make choices, helping them to both under-stand and demonstrate how they reason, create, and usestrategies. Also, portfolios promote students’ reflectionon their work and learning. Thus, portfolios helpstudents offer evidence of their progress toward meetingestablished outcomes, and enable them to take on theresponsibility for their own learning process.
The Pacesetter Spanish portfolio research projectgoals are: (1) to prepare a working definition of Pace-
5
setter Spanish portfolios as assessment tools, (2) todevelop a standardized portfolio assessment system,and (3) to use the project findings for training teachersin upcoming professional development sessions.
To achieve these goals, several issues were addressed:(1) the development of an assessment matrix to takeinto account course outcomes, performance indicators,and scoring rubrics for the Culminating Assessment, (2)piloting and refining the resulting assessment matrix,and (3) identifying benchmarks and developing a setof sample portfolios to be used for training purposes.
Project personnel included two ETS project staffmembers from the Pacesetter Spanish program and asummer intern. Andrea Fercsey and Carmen Luna of theAssessment Division coordinated this phase of theresearch with the help of Eva Ponte, graduate student atthe University of California at Berkeley, who was thesummer intern assigned to work on this project. SixPacesetter Spanish teachers participated in the researchproject, representing four different districts in which theprogram is being implemented. It should be noted thatthese participating teachers had a very short period oftime (two months) to devote to the implementation ofportfolios in their classes. Such a limitation posedvarious constraints, and it is expected that a full schoolyear devoted to portfolios would ease these extraneousproblems.
The first meeting was held in April 1996. Colleagueswho had worked on the Pacesetter English portfolioproject presented their experience with their program.The goals, issues, and design of the Pacesetter SpanishPortfolio Research Project were discussed and revisedas needed.
The role of portfolios as an assessment tool was clari-fied: Pacesetter Spanish portfolios would be used toevaluate classroom products and not processes. However,since portfolios are a collection of products over time, theinformation gathered through their evaluation may beused to enlighten instructional materials and methods.
Through discussions with participants, it becameapparent that teachers needed much guidance tobe better prepared to help their students select samplesfor portfolios. It was emphasized that the materialsneeded to be extremely teacher friendly, and issues ofportfolio management (especially logistics and storage)were also discussed. It was determined that the contentof Spanish portfolios for this phase of the researchproject should be two writing samples, two speakingsamples, one reading comprehension sample, onelistening comprehension sample, and the final projectsfor the units. It was made clear that this list of artifactswas only a suggestion open to revision.
A consensus was reached to work with the sameterminology used in the five strands of the Pacesetter
Spanish Culminating Assessment and end-of-yearreport. These five strands are: “Beginning,” “Develop-ing,” “Promising,” “Accomplished,” and “Advanced.”Using course outcomes as the basis for the elaborationof the assessment matrix, two dimensions were defined.The first dimension was called “Demonstrating Knowl-edge of Hispanic cultures,” and the second dimensionwas named “Using Spanish to Communicate Effec-tively.” The three aspects of Dimension 1 are notevaluated explicitly in the Culminating Assessment. Adecision was made to include Dimension 1 in portfolioassessment to better address the course outcomes. Apreliminary assessment matrix was then developed, andseveral aspects were included within each dimension. Apreliminary description of each of the five strands of thescale was developed. A cover sheet and basic guidelinesfor students were reviewed and approved. An exampleof these forms can be found in Appendix 1.
The assessment matrix included three aspects underDimension 1, “Demonstrating Knowledge of HispanicCultures”: Showing awareness of the diversity of Hispaniccultures, Identifying contributions of Hispanic figures, andMaking connections with other disciplines and ownculture(s). The second dimension, “Using Spanish to Com-municate Effectively,” included two aspects: Derivingmeaning from texts and personal interactions (receptiveskills: listening and reading) and Expressing meaning inoral and written form (productive skills: speaking andwriting). This version of the assessment matrix is includedin Appendix 2. Concerning the use of the assessmentmatrix, it was noted that some artifacts may be used toevaluate different dimensions—and even different aspectswithin those dimensions—so raters have to be prepared todo multiple readings of the same work as needed (e.g., avideotape may need to be seen twice: the first time toevaluate the student’s knowledge of Hispanic cultures,and the second time to evaluate the student’s ability tocommunicate in Spanish). For all aspects of bothdimensions, the matrix also included a category called“Not Enough Evidence to Judge.” The raters were to usethis category when they felt they were unable to give arating because of a lack of sufficient artifacts.
During April and May 1996, participating teacherscarried out a more concerted implementation of port-folio assessment in their own classrooms. Students wereinformed of the draft assessment matrix, cover sheet,and guidelines. It was pointed out to both students andtheir parents that this was a research project, and thatthe findings would facilitate new approaches andrevisions to the Pacesetter Spanish program. Prior to thesecond meeting, a scoring sheet was also developed.(This form is shown in Appendix 3.)
At the second meeting in June 1996, the assessmentmatrix was piloted, and sample portfolios were read
6
and scored. Participating teachers were asked to provide25 portfolios each. It should be noted that one partici-pant was unable to attend. However, she did send thecorresponding portfolios for her students, and thesewere integrated as part of the pool to evaluate. Duringthe reading sessions, raters were asked to identify theartifacts and contents of all portfolios in the scoringsheets. The transcription of comments written in thescoring sheets is included in Appendix 5.
This second meeting started with a session devoted tofeedback. Participants discussed the implementation ofportfolios in their classrooms and noted problems theyencountered. First, they mentioned they had too littletime to give their students a clear understanding of howportfolios work and enable the students to gather thenecessary information. These difficulties seem inherent inthis research project. The particular time constraints ofthe project should not be problematic when portfolios areimplemented in Pacesetter Spanish at the beginning of theschool year. Second, participants related some problemswith portfolio management throughout the school year.ETS project staff suggested the following stages to helpmanage portfolios. During the first marking period ofthe academic year, students should concentrate on theorganizational aspect of portfolios. During the secondmarking period, students should become fully cognizantof the rubrics. During the third marking period, a confer-ence should be held with individual students to ensurethat they can fully justify each artifact chosen at thatpoint and that they become aware of any aspects not dulyaddressed. Then, students should finish gathering allnecessary evidence for their portfolios. At the end of theprocess, there should be no glaring omissions or lack ofclarity. A final conference should then be conducted.
Some participants also indicated that several studentsviewed portfolios just as a collection of work andhad problems understanding the assessment matrix. Toaddress this issue, it was suggested that in the futurestudents could be presented with several portfoliosshowing evidence of performance for each level in thescale (i.e., from Beginning to Advanced).
Participants also mentioned some positive aspects ofimplementing portfolios. Less able students felt they“had a shot at this.” Students with average abilities saidthey felt very comfortable with this approach to learning.Also, as students were selecting pieces for their port-folios, they were given the opportunity to improve theirproducts, motivating them to go over their work. Thisprovided extremely beneficial effects in terms of boththeir learning experience and outcomes. More important,students felt they were learning to teach themselves andwere becoming more aware of their own processes.
Both teachers and ETS project staff were extremelypleased with the quantity and quality of the portfolios.
Initially some of the participants had doubted thepossibility of using a portfolio system in their class-room. While they faced difficulties during the initialstage of implementation, they felt a great sense ofaccomplishment at the end of the project. ETS projectstaff were delighted to see that both teachers andstudents were truly becoming fully involved in thedevelopment and implementation of portfolios in thePacesetter Spanish program.
In the next step of the meeting, the group as a wholeselected portfolios that were thought to be representativeof different levels of the scale. These portfolios werediscussed with the guidance of ETS project staff. Duringthese discussions, some teachers reported problems theyfaced when portfolios contained pieces at different levelsof performance. ETS project staff indicated that since itis usual for students’ learning to register different “peaksand valleys” throughout the school year, their artifactswould consequently show evidence of such differences.
During this part of the meeting, teachers alsoreported that poorly organized portfolios were verydifficult for teachers to read. ETS project staff indicatedthat students should be strongly encouraged to properlyorganize their portfolios. (For instance, when studentsworking as a group submit a video with a presentationor a tape with an interview, they should at all timesidentify themselves. Otherwise, raters who are not theirteachers will find it impossible to identify students inorder to evaluate their individual work.)
The next planned activity at the meeting was thereading and scoring of portfolios. Because of their lackof experience in similar endeavors, participants decidedto start by working in pairs. Participants and ETSproject staff then started by reading two portfolios inpairs, afterward moving on to read and score portfoliosindividually. Twenty-one portfolios were selected to beread twice. ETS project staff were used as third raters incase of large discrepancies. Originally, it was expectedthat each rater would read six portfolios; however,because some raters evaluated portfolios faster thanother raters, two raters judged five portfolios, two ratersread the expected number of portfolios (six), andanother two raters read seven portfolios.
In all, a total of 30 portfolios were read: 8 portfolioswere read in pairs, 21 portfolios were read individuallyby two different raters (double reading), and 1 was readby only one rater. The flow of portfolios for this readingwas organized so that teachers did not read their ownstudents’ portfolios. Raters rated each aspect and thengave a total score for each dimension.
During the final part of the meeting, the assessmentmatrix was discussed, taking into account knowledgegained from the reading sessions. A discussion tookplace on whether Aspect 1 (Showing awareness of the
TABLE 1
Distribution of RatingsCategory Number Category Name Frequency Percentage
1 Beginning 12 2.91
2 Developing 23 5.57
3 Promising 158 38.25
4 Accomplished 114 27.60
5 Advanced 10 2.42
0 No Evidence 96 23.24
TOTAL 413 100.00
Distribution N = 317 Mean: 3.27 SD: .80
7
diversity of Hispanic cultures) and Aspect 2 (Identifyingcontributions of Hispanic figures) could really beconsidered different. The conclusion was that they wereindeed different, but that teachers did not have a clearenough idea of what they were looking for in a port-folio. Consequently, they were not very effective incommunicating this aspect to their students. Also, ratersoften indicated that there was “not enough evidence” toevaluate students’ work on Aspect 3 of Dimension 1(Making connections with other disciplines and ownculture[s]). This may have occurred because this aspectwas inadvertently omitted from the cover sheet thatstudents were told to use to write their justification foreach selected piece. It was also noted that in most casesthere was little or no evidence to judge Dimension 1;there seemed to be a problem with both the quantityand quality of artifacts used to show evidence oflearning for Aspect 1. There was usually plenty ofevidence to judge Dimension 2, and this may be due toboth students and teachers having experience evaluatingproductive language skills in their courses.
At the end of the process, the parameters initiallyindicated for selecting the required number of pieces in astudent’s portfolio were found to be inadequate, resultingin portfolios that seemed to be lacking in the necessaryevidence to judge all aspects of both dimensions. Adecision was made: There would no longer be a setnumber of pieces to be included, and more emphasiswould be placed on the process of selecting enoughartifacts to show evidence of a student’s accomplishments.
Portfolio AnalysesAfter the second meeting, all data and information wereanalyzed. Certain limitations of this portfolio assess-ment should be considered. First, guidelines to selectmaterials were just tentative. Second, the assessmentmatrix was treated as a pilot. Third, certain aspectswere quite new to both students and teachers, and thetime constraints prevented them from becoming familiarwith those new aspects.
The group of raters was composed of five PacesetterSpanish teachers and two ETS project staff members.None of the raters had any prior experience scoring aSpanish portfolio, and only one ETS staff had priorexperience scoring portfolios—but in another contentarea (English as a Second Language).
Training was done first as a whole group, then in pairs.While reading portfolios as a group, raters were asked tohave the assessment matrix present at all times, andto think of ways in which they were using it to makedecisions. Raters were also asked to reflect on the use ofspecific evidence to justify the ratings they were assigning.
Finally, they were asked to note all the problematicaspects they found when scoring any portfolio. Whenraters were reading in pairs, the two of them discussedand decided on the ratings for each aspect and holisticallyfor the two dimensions. Next, the group as a whole dis-cussed and shared their experiences, and then the ratersstarted reading and scoring portfolios individually.
Description of ElementsAs mentioned earlier, 30 portfolios were read. Therewas a total of 59 portfolio readings: Eight portfolioswere read in pairs, 21 were read individually with tworaters per portfolio (42 readings), 8 portfolios were readby a third rater when discrepancies of 2 points or morewere found between the first two raters, and 1 portfoliowas read by only one rater.
Distribution of Ratings To analyze the distribution of ratings, we constructedtwo tables, one showing the frequencies of ratings givenacross portfolio aspects and the other showing thedistribution of ratings for each portfolio aspect anddimension. The rating scale was composed of fivecategories: “Beginning,” “Developing,” “Promising,”“Accomplished,” and “Advanced.” There is anothercategory in the analysis, named “Not Enough Evi-dence,” not included in the rating scale. (See page 8.)
Table 1 shows that the categories with the highestfrequencies are 3 and 4 (i.e., “Promising” and “Accom-plished”), which, when combined, accounted for abouttwo-thirds of all ratings given. Overall, students’ perfor-mances tend to be somewhat above average (i.e., 68percent of the ratings were 3 or higher while fewer than9 percent were 2 or lower), with more studentsperforming in the middle and upper half of the scale.The high percentage of judgments falling in the category“Not Enough Evidence” is also noticeable.
8
To study student performance and the incidence ofthe category “Not Enough Evidence” in more detail,Table 2 displays the distribution of frequencies in eachrating category for each aspect and dimension.
As expected, the rating category “Promising” has thehighest frequencies. There are only two aspects in whichthis does not occur: Aspect 3, in Dimension 1, and Aspect1, in Dimension 2. In the first case, Making connectionswith other disciplines and own culture(s), the categorywith the highest frequency is “Not Enough Evidence”(71 percent). The reason for this may be that this aspectwas not included in the “cover sheet” form given tostudents to select their work and justify their selection.In the second case, Deriving meaning from texts andpersonal interactions, the rating category with the highestfrequency is “Accomplished.” Not surprisingly, students’performance in this aspect (receptive skills) tended to be
better than in other aspects. This conforms to knownpatterns of language acquisition. In general, teachers hadmore experience evaluating both aspects of Dimension 2throughout the Pacesetter Spanish course. Also, studentshad less difficulty identifying artifacts to show evidenceof their achievements in Dimension 2. (They tend toshow their command of a language in terms of theirlistening, speaking, reading, and writing abilities.)
Included in Table 2 are the means and standarddeviation of the score distributions for each aspect andfor the total in each dimension. The statistics describingthese distributions were calculated for each aspect anddimension. When we calculated the means and standarddeviations, we did not include the category “Not EnoughEvidence.” The means ranged from 2.8 (total Dimension1) to 3.54 (Aspect 1, Dimension 2). The overall mean,without making distinctions between aspects and dimen-sions, is 3.27. This indicates, as we have seen before, thatoverall students’ performance is medium-high. When wereview the means for the aspects, we see that students’performance tends to be similar across the differentaspects and dimensions. Variability within dimension,measured by the standard deviation, is also similar acrossdifferent aspects and dimensions: It ranges from .61(Aspect 1, Dimension 2) to .94 (Aspect 3, Dimension 1).
In general, we see that most ratings fell in themedium-high categories of the scale (“Promising” to“Accomplished”). A much smaller number fell in theother categories (“Beginning,” “Developing,” and“Advanced”). Aspect 1, Dimension 1 (Showing aware-ness of the diversity of Hispanic cultures), wasthe aspect having the highest percentages in categories“Beginning” and “Developing.”
Time Used to Read PortfoliosBecause this was the first time that portfolios were readand scored, we were interested in the amount of timeused to read them (as shown in Table 3). Raters wereasked to keep track of this. Some forgot, hence themissing data in Table 3.
As indicated in Table 3A, the average time employedto read a portfolio was 46.74 minutes. The shortestamount of time needed was 20 minutes while thelongest was 120 minutes (Table 3). Several tables wereconstructed to examine whether the amount of timeneeded to read a portfolio was different for portfolioscoming from different Pacesetter Spanish teachers, fordifferent raters, and for readings in different sessions.
As indicated in Table 4, there is some variation in termsof the time raters needed to rate portfolios from differentteachers. On average, portfolios from the class of Teacher2 took the shortest amount of time to read (41.25 minutes
TABLE 2
Distribution of Ratings by Aspects and Dimensions
Dimension 1
Category Category Aspect 1 Aspect 2 Aspect 3 Total Name Freq. % Freq. % Freq. % Freq. %
1 Beginning 3 5 2 3 1 2 4 7
2 Developing 8 14 4 7 0 0 4 7
3 Promising 26 44 24 41 9 15 25 42
4 Accomplished 11 19 12 20 5 9 9 15
5 Advanced 2 3 1 2 2 3 2 3
0 No evidence 9 15 16 27 42 71 15 25
TOTAL 59 100 59 100 59 100 59 100
Aspect 1 Aspect 2 Aspect 3 Total
N 50 43 17 44
Mean 3.02 3.14 3.4 2.80
SD 0.89 0.80 0.94 0.84
Note: These statistics have been calculated without taking intoaccount the category “No Evidence.”
Dimension 2
Category Category Aspect 1 Aspect 2 Total Name Freq. % Freq. % Freq. %
1 Beginning 0 0 0 0 2 3
2 Developing 2 3 2 3 3 5
3 Promising 20 34 28 48 26 44
4 Accomplished 27 46 27 46 23 39
5 Advanced 1 2 1 2 1 2
0 No evidence 9 15 1 2 4 7
TOTAL 59 100 59 100 59 100
Aspect 1 Aspect 2 Total
N 50 58 55
Mean 3.54 3.24 3.33
SD 0.61 0.86 0.77
Note: These statistics have been calculated without taking intoaccount the category “No Evidence.”
9
TABLE 3
Time Used to Read Each PortfolioStudent Student’s Teacher Rater Date Minutes
1 3 7 St/Ev 20
1 3 6 Su/M 35
2 1 6 St/Ev 20
2 1 4 St/Ev 30
3 6 4 St/Ev 20
3 6 3 Su/M 55
4 3 8 St/Ev 65
4 3 2 Su/M 40
5 3 1 St/Ev 60
5 3 7 St/Ev 35
6 4 1 St/Ev 30
6 4 7 St/Ev 30
7 4 2 St/Ev
7 4 8 Su/M 50
8 1 4 St/Ev
8 1 3 Su/M 35
9 2 7 St/Ev 20
9 2 6 St/Ev 25
10 6 3 St/Ev 50
10 6 1 Su/M 20
11 5 3 St/Ev
11 5 1 Su/M 40
12 1 7 St/Ev 40
12 1 6 Su/M 45
13 2 6 St/Ev 45
13 2 4 St/Ev 35
14 2 4 St/Ev 40
14 2 6 40
15 5 1 St/Ev 30
15 5 7 Su/M 40
16 5 2 St/Ev 60
17 5 8 St/Ev 70
17 5 4 Su/M 55
18 3 6 St/Ev 40
18 3 4 Su/M 30
19 6 2 St/Ev 45
20 4 3 St/Ev 60
20 4 1 Su/M 35
21 6 8 Su/M 50
21 6 2
22 1 2 & 3 St/M
23 1 8 St/M 90
24 2 4 & 7 St/M 95
25 2 7 Su/M 30
26 3 4 & 7 St/M-Ev
27 3 4 & 7 St/M 55
28 6 2 & 3 St/M 120
29 5 1 & 6 St/M 100
30 4 1 & 6 St/M 80
TABLE 3A
SummaryMean time used to read portfolios: 46.74 minutes
Standard Deviation: 22.83
TABLE 4
Average Time Employed to Read Portfolios FromDifferent TeachersTeacher N Mean St. Dev. Minimum Maximum
1 6 43.33 24.43 20 90
2 8 41.25 23.26 20 95
3 9 42.22 14.81 20 65
4 6 47.50 19.94 30 80
5 7 56.42 23.58 30 100
6 7 51.42 33.50 20 20
N = Number of portfolios readMinimum = Minimum time neededMaximum = Maximum time needed
per portfolio); portfolios from the class of Teacher 5took the longest to read (56.42 minutes per portfolio).
Table 5 displays the average time each rater neededto read a portfolio. Raters worked at different paces;consequently, we combined different pairs of raters, sothat all 21 portfolios were read twice. In Table 5, Raters1 through 7 are individual raters, number 8 refers toRaters 4 and 6, number 9 to Raters 2 and 3, and num-ber 10 to Raters 1 and 5. Although the sample is small,it took more time to read portfolios in pairs thanindividually. This was expected because discussion tookplace during the reading. Within individual raters, Rater6 was the fastest rater, and Rater 7 was the slowest.
Table 6 shows the average time needed to read portfo-lios in each session. Session 1 ran from Saturday morningthrough the afternoon; Session 2 took place the afternoonand early evening of Saturday; and Session 3 took placeSunday morning. Raters took more time on average to rateportfolios during the first session. This was expected sinceit was the first Pacesetter Spanish portfolio evaluation, andbecause the readings in this first session were done mostlyin pairs. There is almost no difference in the average timeneeded to rate portfolios during Sessions 2 and 3.
Culminating AssessmentAs previously indicated, Pacesetter Spanish includes astandardized assessment component called the Culmin-ating Assessment. This assessment was developed forthe areas of reading, speaking, listening, and writing.Reading and speaking were locally scored by individualteachers, and listening and writing were centrally scored
10
at a national reading. Dimension 2 of the portfolio scor-ing sheets measures language, and it is divided into twoaspects: receptive skills (listening and reading), and pro-ductive skills (writing and speaking). Although we cannot directly compare these aspects with the areas of theCulminating Assessment, we wanted to examine whetherthe patterns of these two modes of assessments weresimilar (i.e., if a student got a high score in the Culmin-ating Assessment, we expected him or her to get a highscore in the respective aspects of the portfolio). Both theCulminating Assessment and portfolio scales contain fivestrands, and there was an attempt to maintain parallelismin the descriptors for each strand, especially for Dimen-sion 2 (where the constructs are similar).
Table 7 displays the scores obtained by students bothin the Culminating Assessment and in the portfolio. Theinformation displayed in this table indicates thatin general portfolio scores tended to be lower thanCulminating Assessment scores. This may have happenedbecause those skills not included in the CulminatingAssessment are usually harder for students to masterand for teachers to assess.
The overall correlation coefficients between theCulminating Assessment and the portfolio scores werelow (r = .15 between Culminating Assessment listening
grade and portfolio Dimension 2, Aspect 1 Derivingmeaning from texts and personal interactions; and r = .07between Culminating Assessment writing grade andportfolio Dimension 2, Aspect 2, Expressing meaning inoral and written forms). These results suggest that theCulminating Assessment and the portfolio assessmentmay be tapping different sets of knowledge and skills,even though the constructs were assumed to be similar.
Portfolio Assessment MatrixCorrelationsThe portfolio’s assessment matrix included sevenratings: one rating for each of the three aspects of
TABLE 6
Time Employed to Read Portfolios at Different SessionsSession N Mean St. Dev. Minimum Maximum
1 6 90.00 21.68 55 120
2 22 39.54 15.65 20 70
3 14 40.00 10.19 20 55
N = Number of portfolios readMinimum = Minimum time neededMaximum = Maximum time needed
TABLE 7
Culminating Assessment and Dimension 2 PortfolioRatingsStudent Culminat. Portfolio Culminat. Portfolio
Assess. Dim. 2 Assess. Dim. 2Listening Aspect 1 Writing Aspect 2 Grade Rating Grade Rating
1 3 3 3 3
2 4 2.5 3 3
3 3 2.5 3 4
4 2 4 3 3.5
5 5 4 5 3
6 3 4 3 3
7 4 4 3 3.5
8 5 3.5 4 4
9 5 4 5 4
10 5 4 4 3
11 4 4 5 4
12 5 3.5 5 3
13 4 2.5 4 3
14 5 3 4 3
15 4 3.5 3 3.5
16 5 4 4 4
17 5 4 4 5
18 5 3.5 5 3
19 5 2.5 3 3.5
20 4 4 4 3.5
21 5 4 5 4
22 5 4 4 4
23 5 3 5 3
24 4 3 5 3
25 5 3 5 3
26 4 4 4 4
27 4 3 3 3
28 5 3 4 4
29 2 3 2 3
30 5 4 4 4
Mean 4.30 3.47 3.93 3.48
TABLE 5
Average Time Employed by Each RaterRater N Mean St. Dev. Minimum Maximum
1 6 35.83 13.57 20 60
2 3 48.33 10.41 40 60
3 4 50.00 10.80 35 60
4 6 35.00 11.83 20 55
5 7 35.71 9.76 20 45
6 7 30.71 8.38 20 40
7 5 65.00 16.58 50 90
8 2 75.00 28.28 55 95
9 1 120.00 120 120
10 2 90.00 14.14 80 100
N = Number of portfolios readMinimum = Minimum time neededMaximum = Maximum time needed
11
each facet. We have three facets: students, readers, andportfolio aspects and dimensions. Some students weremore proficient than others, some readers rated moreseverely than others, and some portfolio aspects wereharder to get high ratings on than others. This variationamong students, raters, and aspects is expected; how-ever, unexpected variations may signal attention tounresolved or unnoticed problems; i.e., those parts of thedata that seem not to follow usual patterns of variability.
FACETS produces a map in which all facets of theanalysis are shown in one figure, providing the readerwith general information for each facet. We include inFigure 1 a reproduction of the FACETS map.
FACETS calibrates all facets of the analysis so thatthey can be positioned on the same equal interval scale.This scale is shown in the first column of the map.
Students, the first facet of our study, occupies thesecond column of the FACETS map. This column dis-plays the estimates of students’ proficiency on theportfolio assessment. Student measures are orderedwith more proficient students at the top of the columnand less proficient students at the bottom of the column.Student proficiency measures range from −3.2 logits to3.2 logits.
The third column in Figure 1 represents raters interms of the harshness or leniency they exercised whenrating portfolios. Harsher raters appear at the top of thecolumn, and more lenient raters appear at the bottom.Rater 6 was the harshest, and Rater 3 was the mostlenient. Rater harshness measures range from −2.3 logitsto 2.3 logits. As expected, this variation is smaller thanthe variation found in student measures.
The fourth column of the map displays the portfolioin terms of its aspects. The hardest aspects are at thetop of the column and the easiest at the bottom. Thisindicates that Aspect 3, Dimension 1 was the hardest toget high ratings on, and Aspect 2, Dimension 1 was theeasiest one to get high ratings on.
Columns 5 through 11 show the most probablerating for a student at a given level on the logit scale, as
Dimension 1, one rating for each of the two aspectsof Dimension 2, and two ratings as totals for bothdimensions. We include in Table 8 the correlationsamong all these ratings.
All correlations between aspects and dimensions aresignificant at the .06 level (except for the correlationbetween Aspect 1, Dimension 2 and Aspect 2,Dimension 1), and range from moderate (.25) to high(.72). Correlations among the three aspects underDimension 1 range from .30 to .51, while the corre-lation between the two aspects of dimension 2 is .29.Two correlations are moderately high: the correlationbetween Aspect 2, Dimension 1 and Aspect 2, dimen-sion 2 (.46), and the correlation between Aspect 3,Dimension 1 and Aspect 2, Dimension 2 (.54). Thecorrelations of Aspects 1 and 2 of Dimension 2 withthe total of Dimension 2 are .61 and .52.
Portfolios Read Twice:Reliability StudyPortfolio assessment is based on personal judgments.One common approach to studying the generalizabilityand fairness of the judgments is to ask a second personto read and score the portfolios. Due to the nature ofthe project and the stage in which the Pacesetter Spanishprogram was at that moment, and also due to timelimitations, it was only possible to obtain a smallsample of portfolios read twice.
FACETS (Linacre, 1989), a computer program basedon Rasch partial credit models, was used to analyze thedata. FACETS analyzes data from assessments that haveseveral facets (e.g., raters, students, tasks) by assigningparameters to each facet in the model. FACETS has theform of a log-linear model for main effects and estimatesthose effects in logits (logits are the logarithmic odds ofa given rating compared to the next lower one). Linearmeasures are then constructed from ordered qualitativedata, allowing the separation of the contribution of
TABLE 8
Portfolio Assessment Matrix Correlations Dimension 1 Dimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2 Dimension 2Aspect 1 Aspect 2 Aspect 3 Total Aspect 1 Aspect 2 Total
Dimension 1 Aspect 1
Dimension 1 Aspect 2 .37
Dimension 1 Aspect 3 .51 .30
Total Dimension 1 .72 .69 .68
Dimension 2 Aspect 1 .26 .13 .36 .35
Dimension 2 Aspect 2 .30 .46 .54 .43 .29
Total Dimension 2 .34 .25 .34 .43 .61 .52
12
expected from a rater with average harshness, in each ofthe portfolio aspects and dimensions. The horizontallines across a column indicate the point at which thelikelihood of getting the next higher rating begins toexceed the likelihood of getting the next lower rating.For instance, if we look at scale 1 (Aspect 1, Dimension1), we see that students with measures from −2.1 to 1.5logits are more likely to receive a 3 (“Promising”) thanany other rating on that aspect.
In an effort to study how reliable raters’ judgmentswere, and also with the aim of identifying problematicportfolios, we analyzed the discrepancies in ratingsbetween raters of the same portfolio. Table 9 includesthe number and percentage of scoring discrepancies foreach aspect of the portfolio, and in total.
As we have indicated before, the scale covers fivelevels: beginning, developing, promising, accomplished,and advanced. We also had a category for cases where
Figure 1. FACETS map for the whole scale (21 portfolios, two readers).
13
there was not enough evidence to judge. In the last row ofthe table, we have included those situations in which onerater said there was no evidence, and the other consid-ered there was enough evidence and gave a judgment.
The number of agreements is moderate; it constitutes40 percent of all paired judgments. (However, it shouldbe noted that cases where both raters considered therewas no evidence to judge have been included here as“agreements.”) The number of portfolios having only a1-point discrepancy is also moderate (33 percent of allpaired judgments). Finally, the number of discrepanciesof 2 points was small (3 percent).
An important finding is that the number of discrep-ancies between raters who judged student work and
raters who said there was not enough evidence to judgewas considerable (23 percent of all paired responses).Most discrepancies of this type occurred in Dimension 1,especially in Aspect 3.
We were also interested in seeing at which specificlevels of the scale these discrepancies occurred. Table 10displays those discrepancies for each portfolio aspectand for the portfolio as a whole.
The type of discrepancy with the highest occurrence(41 percent) is that in which one rater considered astudent’s performance promising while the other con-sidered it accomplished (discrepancy 3-4). Another typeof discrepancy with a high percentage of occurrence (22percent) was that in which one rater thought there was
TABLE 9
Summary of Portfolio Score DiscrepanciesDimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2
Aspect 1 Aspect 2 Aspect 3 Aspect 1 Aspect 2 D 1 D2 Total Percentage
Same Score 8 7 14 4 11 6 8 59 40
1–Point Disc. 6 7 10 10 7 9 49 33
2–Point Disc. 2 1 1 1 5 3
Evid. vs. Noevidence 5 6 7 6 7 3 34 23
TABLE 10
Summary of Portfolio Score Discrepancies for Each Aspect in Each DimensionDimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2 Dimension 1: Dimension 2: Total
Discrepancy Aspect 1 Aspect 2 Aspect 3 Aspect 1 Aspect 2 Total Total Frequency Percentage
No Evid.–1 1 1 1
No Evid.–2 1 2 2 5 6
No Evid.–3 3 4 3 3 4 2 19 22
No Evid.–4 3 2 1 1 7 8
No Evid.–5 1 1 2 2
1–2 1 1 1
1–3 1 1 1
2–3 2 1 1 2 2 2 10 11
2–4 1 1 2 2
3–4 3 6 9 7 5 6 36 41
3–5 1 1 2 2
4–5 1 1 2 2
TOTAL 13 14 7 17 10 15 12 88
(Discr.)
Agreement
(full scores) 8 6 2 4 11 5 9 45
Agreement
(No Evid., 0-0) 1 12 1 14
TOTAL
(Agreement) 8 7 14 4 11 6 9 60
14
no evidence to judge, and the other considered thework showed evidence of being promising (difference“No Evidence” – 3). This latter type of discrepancy ismore important and indicates that raters’ internalizationof the assessment matrix and search of evidence wasproblematic. One possible explanation is that someraters judged only those things students had indicated asevidence of their learning in a specific area, and otherraters used materials not marked by the student for thatspecific area. The only aspect in which there are nodiscrepancies of this type is Aspect 2, Dimension 2.
Aspect 3, Dimension 1 registered the fewest discrep-ancies. It should be noted, though, that most agreementswere of the type “No Evidence”–“No Evidence,”meaning that both raters judged there was not enoughevidence to evaluate a student’s performance.
Usually a 1-point discrepancy is considered smallenough to categorize the rating as an acceptablejudgment. However, problems may arise when thereis a cut point. FACETS provides a “fair” measure, whichindicates the score a student would have received had
his or her ratings been adjusted for rater effects. Evensmall adjustments can be important for those studentswhose scores lie in critical cut-score regions. Table 11shows each student’s observed average score and hisor her “fair” average score (i.e., adjusted for severity orleniency of the raters providing the ratings for thatstudent).
The reported scores of six students would havechanged after results were adjusted for rater severity orleniency. Four students (students 5, 6, 15, and 12)would have changed their reported score from “Promis-ing” to “Accomplished,” and two students (students 20and 8) would have changed their scores from“Accomplished” to “Promising.”
We also studied the behavior of each rater separately.Table 12 indicates how many times each rater usedthe judgment “No Evidence” and the other levels of thescale.
Raters 4 and 6 used the category “Not EnoughEvidence to Judge” very frequently. In each case over 35percent of their judgments were “Not Enough Evidence.”
TABLE 11
FACETS OUTPUT (7.1.1) Students Measurement Report (Arranged by FN).
15
As we can see, these raters also read more portfolios thanthe others. In future analysis, it may be interesting tostudy the relationship between time used to read portfo-lios, search for evidence, and use of the “No Evidence”category.
We can intuitively see why FACETS indicated rater 3as the most lenient: 69 percent of the ratings that thisrater gave were 4’s. No other rater gave that high apercentage of 4’s. Rater 6 was the harshest; only 8percent of the ratings this rater gave were 4’s, and therater gave no 5’s.
Third Rater AnalysisUsing only the portfolio ratings for total Dimension 1and total Dimension 2, we selected some portfolios withextreme discrepancies. We defined “extreme” as port-folios with dimension ratings 2 or more points apart, andalso portfolios with ratings of evidence (any evidence)as opposed to “Not Enough Evidence to Judge.” Dueto time constraints, not all portfolios were read twicewhen this selection was made. Eight portfolios wereselected for a third reading.
We analyzed the ratings each rater gave to eachportfolio aspect (see Appendix 5). Table 13 summarizesthe results of this analysis.
As we can see, portfolios 14 and 17 produced a lot ofdiscrepant ratings, indicating that these were problem-atic portfolios. Discrepancies mostly occurred in Aspect1, Dimension 1 (4 portfolios) and Aspect 1, Dimension2 (3 portfolios).
Selection of BenchmarksAfter the June 1996 meeting in Miami, ETS project staffselected some portfolios to be used as benchmarks. Stafftook into account the discrepancies between raters aswell as other factors such as content and variety ofartifacts when selecting the benchmarks. The selection
was made using a table in which portfolio scoreswere presented by matched pairs of raters (this table isincluded in Appendix 4; those portfolios selected asbenchmarks are indicated in bold). Besides the portfo-lios selected using the information given in thementioned table, another portfolio that was read by agroup was also selected. Thus, a packet of portfoliomaterials was prepared and used at the three professionaldevelopment sessions carried out in summer 1996 fornew Pacesetter Spanish teachers. The midyear meetingsat the end of the fall semester of 1996 also includedsessions on portfolios, and the samples from the trainingpackets served to disseminate the Pacesetter Spanishportfolio model.
Also, as an ancillary result, the rubrics for the fourlinguistic skills were revised, and a decision was made thatPacesetter Spanish would have a unified set of rubrics tobe used throughout the course. It is expected that bothteachers and students will become thoroughly familiarwith these rubrics and should encounter little difficultyapplying them to their work during the school year.
In order to analyze the information gathered fromthe portfolios, and to provide some useful feedback toteachers, all scoring sheets with all written commentswere transcribed. A copy of these transcriptions can befound in Appendix 5.
TABLE 12
Distributions of Raters’ JudgmentsBeginning Developing Promising Accomplished Advanced Not Enough Total Number
Rater 1 2 3 4 5 Evidence of Judgments
1 2 ( 5%) 23 (55%) 13 (31%) 4 ( 9%) 42
2 5 (14%) 11 (31%) 15 (43%) 4 (11%) 35
3 1 ( 3%) 1 ( 3%) 7 (20%) 24 (69%) 2 ( 6%) 35
4 1 ( 2%) 15 (31%) 4 ( 8%) 6 23 (47%) 49
5 1 ( 2%) 8 (19%) 17 (40%) 10 (24%) 6 (14%) 42
6 27 (55%) 4 ( 8%) 18 (37%) 49
7 1 ( 2%) 1 ( 2%) 18 (43%) 17 (40%) 5 (12%) 42
TABLE 13
Portfolios Read Three TimesPortfolio Ratings in Agreement Ratings 2 points apart
3 6 1
5 5 2
8 5 2
12 5 2
13 6 1
14 3 3
17 1 6
18 7 0
16
Summary of Findings• This research showed that some elements of the
delineated Pacesetter Spanish system of portfolioassessment worked as intended, while others needfurther refinement.
• Pacesetter Spanish portfolio raters very often used themedium category of the scale (“Promising”), and notthe extreme categories (“Developing” or “Advanced”).This may have been due in part to the small sample ofportfolios read. In addition, students in a level threelanguage class tend to cluster in the middle of the scale,which confirms actual classroom experience.
• Often the length of time needed to read a PacesetterSpanish portfolio depended on the variety and com-plexity of artifacts included (i.e., a portfolio with manydifferent types of media took longer to read than aportfolio with only paper-based samples). Althoughsuch variety and complexity are encouraged throughoutthe course framework, this element of time needs to betaken into account when preparing to read portfolios.
• The correlation between Dimension 2, “UsingSpanish to Communicate Effectively,” and thePacesetter Spanish Culminating Assessment was low.These findings suggest that both assessment instru-ments serve to evaluate different sets of knowledgeand skills, although the construct may appear similar.Whereas the Culminating Assessment evaluates fourseparate linguistic skills (reading, speaking, listening,and writing), Dimension 2 of the Pacesetter Spanishportfolio assessment system groups them in twoaspects: Deriving meaning from texts and personalinteractions (receptive skills) and Expressing meaningin oral and written form (productive skills).
• All correlations between aspects and dimensions of thePacesetter Spanish portfolio assessment system werefound to be significant, except for the correlation be-tween Aspect 1, Dimension 2 and Aspect 2, Dimension 1.
• The number of rating agreements was moderate.However, there was a considerable number of dis-crepancies between raters regarding the presence orlack of enough artifacts to judge students’ perfor-mance within the parameters of the PacesetterSpanish portfolio assessment system.
Recommendations• As shown by Phase I results, further research is
needed to refine the assessment matrix and produce afinal version to implement as part of the formalcomponents of the Pacesetter Spanish program.
• A set of more comprehensive and clearer instructionsshould be given to teachers and students aboutthe Pacesetter Spanish portfolio goals, processes, andlogistics. Some suggestions included requesting studentsto correctly copy and identify all samples they presentfor evidence of individual work, as well as reducing alllarge artifacts such as posters to a manageable size.
• It is necessary to discuss in depth the process ofselection of students’ work (i.e., the number of productsnecessary to demonstrate a certain level of perfor-mance). Guidelines for the selection of material forPacesetter Spanish portfolios need further clarification.
• More support should be given to teachers to betterunderstand and teach Dimension 1 as defined in thePacesetter Spanish portfolio assessment system. It isnecessary to define what constitutes evidence ofachievement in this area, how and where to findevidence, and how it can be collected by their students.
• Further discussion is warranted regarding theappropriateness of grouping the four linguistic skillsin the two aspects of Dimension 2 of the PacesetterSpanish portfolio assessment system.
• In order to apply the new revised version of thePacesetter Spanish portfolio assessment matrix, anew research project should include an expandedportfolio reading with more raters and a largernumber of portfolios. It would be beneficial for thefive original teachers who participated in the Phase Ireading to be incorporated in this expanded readingas experienced raters to help train the new teachers.
• The expanded portfolio reading should be carried outfollowing the model delineated in the correspondingsection of the Pacesetter Scoring Handbook.
• In addition to quantitative data, other qualitative datapertaining to discussion and debriefing sessions withparticipants should be collected during the PacesetterSpanish portfolio reading.
• The new assessment matrix and guidelines for port-folios should be disseminated to all schools imple-menting the Pacesetter Spanish program so thatteachers and students can use them.
• New benchmark portfolios from the expandedPacesetter Spanish portfolio reading should be identi-fied, and a set of sample portfolios should beprepared for use in future training sessions.
ReferencesArter, J.A. and Spandel, V. (1992). Using portfolios of student
work in instruction and assessment. Educationalmeasurement: Issues and practice, Spring 92, 36–44.
17
Bridgeman, B., Chittenden, E., and Cline, F. (1995). Char-acteristics of a portfolio scale for rating early literacy.Princeton, NJ: Educational Testing Service.
Educational Testing Service. (1995). Capturing the power ofclassroom assessment. Focus 28. Princeton, NJ.
Elliot, J. (1985). Facilitating action research in schools: Somedilemmas. In R. Burgess (Ed.), Field methods in the studyof education (pp. 235–242). Lewes, England: Falmer Press.
ETS Trustees’ Colloquy. (1995). Performance Assessment:Different needs, difficult answers. Princeton, NJ: Edu-cational Testing Service.
Hardy, R.A. (1995). Examining the costs of performance assess-ment. Applied measurement in education, 8(2), 121–134.
Gifford, B.R. and O’Connor, M.C. (Eds.) (1992). Changingassessments. Alternative views of aptitude, achievementand instruction. Boston: Kluwer Academic Publishers.
Gitomer, D.H. and Duschl, R.A. (1995). Moving toward aportfolio culture in science education. Princeton, NJ:Educational Testing Service.
Koretz, Daniel. (1994). The evolution of a portfolio program:The impact and quality of the Vermont portfolio pro-gram in its second year (1992–93). Los Angeles: NationalCenter for Research on Evaluation, Standards, andStudent Testing.
Koretz, Daniel. (1994). Lessons from an evolving system.Interim report: The reliability of Vermont portfolioscores in the 1992–93 school year. Los Angeles: NationalCenter for Research on Evaluation, Standards, and Stu-dent Testing.
Koretz, Daniel. (1992). Can portfolios assess student perfor-mance and influence instruction? The 1991–92 Vermontexperience. Los Angeles: National Center for Research onEvaluation, Standards, and Student Testing.
Koretz, Daniel. (1992). The reliability of scores from the 1992Vermont portfolio assessment program. Los Angeles:National Center for Research on Evaluation, Standards,and Student Testing.
Mislevy, R.J. (1995). On inferential issues arising in the California learning assessment system. Princeton, NJ:Educational Testing Service.
Myford and Mislevy. (1995). Monitoring and improving aportfolio assessment system. Princeton, NJ: EducationalTesting Service.
Myford, C.M., Marr, D.B., and Linacre, M. (1996). Readercalibration and its potential role in equating for the testof written English. Princeton, NJ: Educational TestingService.
Resnick, L.B. and Resnick, D.P. (1992). Assessing the thinkingcurriculum: New tools for educational reform. In B.R.Gifford, and M.C. O’Connor (Eds.), Changing assess-ments. Alternative views of aptitude, achievement andinstruction. Boston: Kluwer Academic Publishers.
Sheingold, K. and Fredericksen, J. (1995). Linking assessmentwith reform: Technologies that support conversationsabout student work. Princeton, NJ: Educational TestingService.
Sheingold, K., Heller, J.I., and Paulukonis, S.T. (1995): Activelyseeking evidence: Teacher change through assessmentdevelopment. Princeton, NJ: Educational Testing Service.
Shepard, L. (1989) Why we need better assessments. Educa-tional Leadership, (46 7), 4–9.
Wolf, D., Bixby, J., Glenn, J., and Gardner, H. (1991). To usetheir minds well: Investigating new forms of studentassessment. In G. Grant (Ed.), Review of EducationalResearch 17, 31–74. Washington, DC: American Educa-tional Research Association.
PHASE II REPORT
Rationale and Expected OutcomesPacesetter Spanish is a new and challenging third-levelhigh school Spanish course developed by the CollegeBoard and Educational Testing Service (ETS) thatcombines a carefully integrated set of learning outcomes,course materials, and instructional experiences, includingvarious approaches to assessment. Throughout thecourse, there is a provision for student self-assessmentthrough learning logs and related journal activities, forpeer assessment through work in pairs and small groups,for ongoing teacher-developed assessments, and for astandardized end-of-course assessment, as well as forstudents to assemble portfolios with products from theirwork during the year.
A portfolio is a collection of significant samples ofstudent work over time, accompanied by clear criteriafor evaluation, as well as the student’s own reflectionson his or her progress. Portfolios, although not replacingother types of assessment, give teachers and studentsanother tool of great instructional and motivationalvalue. The portfolios for Pacesetter Spanish may containa rich array of samples, including videos of studentreports and role playing, audio excerpts of informal andformal interviews, drafts and final versions of writtenwork, and collages and descriptions of projects. In fact,many of the activities and final projects for the units areacceptable choices for inclusion in student portfolios.
In the case of Pacesetter Spanish, portfolios are tobe used for assessment purposes, not to look at progressbut to evaluate results, thus also serving to inform andgive feedback on instructional materials and methods.Portfolios yield valuable data about students’ achieve-ment and effectively lead students through the processof acquiring a new language. They serve as a guide forstudents as they make choices and demonstrate howthey reason, create, strategize, and reflect. Ultimately,portfolios allow students to take on responsibility fortheir own learning process and offer evidence of their
18
progress toward meeting the course outcomes. Theseportfolios help teachers and students be reflective oftheir work, assess their effectiveness as teachers andlearners, and reshape their approach to the coursematerials. Sessions on assessing these student portfolioshave become an integral part of the professional devel-opment institutes for teachers taking place each summerin different areas of the country.
Research is needed to ensure that it is appropriate toinclude portfolio assessment as an important part of theintegrated instruction-based assessment component. Thisstudy is designed to facilitate the practice of developingand assessing portfolios and to ensure that the proceduresbeing taught at teacher institutes will lead to accurate, fairassessment of portfolios. The specific goals are (1) toprepare a working definition of Pacesetter Spanish portfo-lios as assessment tools; (2) to develop a standardizedportfolio assessment system; and (3) to use project resultsas a training tool for future professional developmentsessions for teachers. In summary, the ultimate goal is toenhance classroom practice and assessment.
In Phase I of the Pacesetter Spanish Portfolio ResearchProject, carried out in 1995–96, the following issues wereconsidered: (a) the role of the portfolio as an assessmenttool, its working definition and its appropriateness withinthe course framework; (b) the development of scoringrubrics and an assessment matrix, taking into accountcourse outcomes, performance indicators, and scoringrubrics for the four language proficiency skills (reading,writing, listening, and speaking); (c) the student’s per-spective (cover sheet and guidelines); (d) discussion andrevision of the assessment matrix; (e) reading portfolios;(f) debriefing session; and (g) preparation of trainingpackets for 1996 Professional Development SummerInstitutes for Teachers.
Both the 1996 Professional Development SummerInstitutes for Teachers and midyear meetings includedsessions on portfolios, and the samples chosen for thetraining packets served to disseminate the PacesetterSpanish portfolio model.
The final draft of the corresponding report was pro-duced, and the assessment matrix needed to be refinedaccordingly.
For Phase II of the Pacesetter Spanish PortfolioResearch project the following tasks were planned:
• In spring 1997, in consultation with the teacherswho participated in Phase I of this project, and takinginto account Phase I results, the assessment matrixwas refined.
• In summer 1997, during the three-day refresher sessionat the Delaware Summer Institute for Teachers, a port-folio reading was held. The new matrix was piloted bythe participating teachers. Since it is essential that readers
develop a shared understanding of the scoring criteriaand learn to apply the criteria consistently, benchmarkportfolios from Phase I were selected for training pur-poses. After the initial training, the reading was carriedout following the model delineated in the portfoliosection of the Pacesetter Scoring Handbook (seeAppendix A). In addition to tabulation of scores, otherdata (for example, notes from discussion with teachers,minutes from the debriefing after reading, etc.) werecollected during the reading. The final version of therevised portfolio scoring matrix and guidelines wasrefined for immediate dissemination to participatingschools in order for Pacesetter Spanish teachers to startapplying them in the 1997–98 school year.
• In fall 1997, the compilation of the results of thisreading was analyzed. Benchmark portfolios fromPhase II were identified, and a set of sample portfolioswas developed to be used in the 1997 midyear pro-fessional development training sessions for teachers.
After Phase I, initiated in 1995–96, a new assessmentmatrix was prepared, ready to be further refined. AfterPhase II, this new and revised matrix was applied anddisseminated for further use by Pacesetter Spanish class-room teachers.
Through discussions with participants to determinewhether the portfolios are a valid reflection of studentachievement, and by calculating how reliably the port-folios can be scored, we expect to have a basis forexpanding the use of portfolio assessment as an integralpart of the Pacesetter Spanish program.
Pacesetter Spanish PortfolioResearch Project (PHASE II)MethodAndrea Fercsey and Carmen Luna, of the AssessmentDivision, coordinated this phase of the research withthe help of Pablo Aliaga, a graduate student at the Uni-versity of Michigan, who was the ETS summer internassigned to work on this project. David Baum of theAssessment Division also assisted at the portfolio reading.
Portfolios were evaluated on the campus of theUniversity of Delaware from July 10 through 13, 1997.Six schools sent in work by students enrolled inPacesetter Spanish during the 1996–97 school year.Seven teachers participated in assessing these portfolios;two of them had prior experience scoring PacesetterSpanish portfolios. Not every school that sent in portfo-lios had teachers participating in the assessment of theseportfolios. Three others assisted in the assessment of the
19
portfolios. All three worked for ETS; two with priorexperience scoring Pacesetter Spanish portfolios.
Ninety-five portfolios were designated to be assessed atthe beginning of the grading session. Of these, a total of82 were read. Thirteen portfolios were rendered unread-able due to missing data (i.e., failure to incorporate coversheets). Cover sheets (see Appendix B) are essential to theevaluation of portfolios because they contain a rationaleof the choice of a particular document as evidence of aspecific dimension and aspect. Of the 82 portfolios thatwere read, 51 were read once, and 31 were reread by adifferent rater. Hence, a total of 113 readings were com-pleted. Ratings were provided for the three aspects ofDimension 1, “Demonstrating Knowledge of HispanicCultures” (Showing awareness of the diversity ofHispanic cultures, Identifying contributions of Hispanicfigures, and Making connections with other disciplinesand own culture(s)) and the two aspects of Dimension 2,“Using Spanish to Communicate Effectively” (Derivingmeaning from texts and personal interactions; i.e.,receptive skills such as reading and listening, andExpressing meaning in oral and written form, i.e.,productive skills such as speaking and writing). Ratersused the Pacesetter proficiency categories (“Beginning,”“Developing,” “Promising,” “Accomplished,” and “Ad-vanced”) as well as “Not Enough Evidence to Judge” and“Missing” to rate the portfolios. Appendix C provides thescoring rubrics for each aspect. Appendix D providesrater comments about rubrics and portfolios.
Summary of RatingsTable 1 shows the frequency of ratings in each category.Of the 882 ratings, 31.4 percent were “Promising.” Themajority of ratings (58.2 percent) were either “Devel-
oping” or “Promising.” Last year 66 percent of theratings fell within the “Promising” and “Accomplished”categories and 2.42 percent were “Advanced.” However,this year less than 1 percent of the students received arating of “Advanced.” Finally, 5.4 percent of the ratingsassigned were “No Evidence” due to lack of artifacts.
On average, this year’s students received lowerratings than last year’s students. In 1996, 68 percent ofthe ratings were 3 or higher. In 1997, only 45 percentof the ratings were 3 or higher. This year’s students mayhave been less able compared to last year’s students, theraters could have been more severe in their grading, orthe raters may have been better instructed in scoringthis year because of the previous year’s study.
Also, there was a significant decrease in the use of the“No Evidence” category. This suggests improvementin teacher and student understanding of what is neededto evaluate the portfolios: In a portfolio system like thisone, the selection process is complicated and revealing.Students need to develop a deep understanding aboutthe nature of quality in their work and how the selectedpieces constitute evidence for a particular dimension oraspect. Teachers also need to be fully cognizant of allpossible ways to show evidence of achievement.Consequently, the fact that portfolios in 1997 containedmore complete evidence of students’ work could beinterpreted as a sign of better implementation in theactual classrooms.
Table 2 provides the frequency of ratings for eachaspect of Dimension 1, “Demonstrating Knowledge ofHispanic Cultures.”
Aspects 1 and 3 have similar rating distributions.Further, the proportion of “No Evidence” ratings in thethree aspects of Dimension 1 are more uniform thisyear than last year. Last year “No Evidence” had quite
TABLE 1
Frequency of Ratings in Each Rating Category*
*Data include 13 portfolios that were not evaluated (reported as “Missing”).
20
different response rates. In 1996, over 70 percent of theratings of Aspect 3 were “No Evidence”; by compari-son, only 6 percent of the ratings of Aspect 3 were “NoEvidence” this year.
Table 3 shows the frequency distribution of ratingsfor Dimension 2, “Using Spanish to CommunicateEffectively.”
Aspect 1 has a higher percentage of “No Evidence”and “Beginning” ratings, and a lower percentage of “De-
veloping” ratings than Aspect 2 or the general dimensionratings. The distribution of ratings for “No Evidence” inthe three categories is quite similar to that of last year.
A comparison of Dimension 1 and 2 ratings shows thatsimilar percentages (57 percent to 60 percent) of theratings were “Developing” or “Promising.” For the“Beginning” rating, Dimension 1 has approximately 16percent compared to 8 percent for Dimension 2.For the “Accomplished” rating, Dimension 1 has approx-
TABLE 2
Frequency of Ratings in Each Category for the Aspects of Dimension 1*
*Data include 13 portfolios that were not evaluated (reported as “Missing”).Aspect 1: Showing awareness of the diversity of Hispanic culturesAspect 2: Identifying contributions of Hispanic figuresAspect 3: Making connections with other disciplines and own culture(s)
21
TABLE 3
Frequency of Ratings in Each Category for Aspects of Dimension 2*
imately 10 percent compared to 16 percent for Dimension2. More “Advanced” ratings were given for Dimension 2than for Dimension 1. Finally, more “No Evidence”ratings were given for Dimension 1 than for Dimension 2.
Time Used to Read PortfoliosTable 4 is a summary of the time spent on portfolios byeach rater. The table provides the number of portfolioseach rater read and the total time, average time,standard deviation, minimum, and maximum time ittook the raters to evaluate portfolios.
On average, raters read 11 portfolios, each portfoliotaking about 45 minutes to evaluate. Average time spenton portfolios ranged from 34 to 60 minutes, the rangeof time for each individual portfolio being 17 to 150minutes. Average time spent per portfolio was similar tolast year’s average of 47 minutes.
Distribution of Ratings for Each RaterTable 5 shows the distribution of ratings for each rater.Only two raters gave the “Advanced” proficiencyrating. This may be an indication of disparate use of the
*Data include 13 portfolios that were not evaluated (reported as “Missing”).Aspect 1: Deriving meaning from printed materials and oral discourseAspect 2: Expressing meaning in oral and written form
rubrics by raters. Rater 7 gave more than one-fifth ofthe portfolios he/she read a “No Evidence” rating. Noother rater gave more than one-sixth of the portfoliosthey read a rating of “No Evidence.” Rater 7 never gavea rating higher than “Promising.” Five raters gaveapproximately 50 percent of the portfolios they read a“Promising” rating. Eighty-five percent of the ratingsRater 5 gave were either “Promising” or “Developing.”Consequently, it appears that rater 5 tends to rate moreleniently than the other raters.
Portfolios Read TwiceTable 6 shows that of the 31 portfolios that weredouble read, 8 were given the same two total scores forboth dimensions. After evaluating all aspects of a partic-ular dimension, raters had to give a holistic rating to thewhole dimension. Of the remaining portfolios, approx-imately 39 percent had a score that differed by only onepoint on either Dimension 1 or 2 or both. Only oneportfolio had a difference in score of 3 points for a givendimension. The average absolute difference in time to
score the portfolios was 15 minutes. However, thisnumber is exaggerated by an apparent outlier of 105minutes. The average absolute difference in time afterremoving the apparent outlier is 12 minutes.
Table 7 examines the discrepancies between raterswithin each aspect of both dimensions. Overall, 39 per-cent of ratings were identical. Another 40 percent oftheir ratings differed by a single point. Thirteen percentof the ratings differed by 2 or more points. Finally, in10 percent of the cases, one rater gave a “No Evidence”rating and the other rater believed there was sufficientevidence to score the portfolio on that aspect. This isfewer than last year.
Table 8 shows where specific discrepancies betweenraters occurred. Thirty-two percent of the discrepanciesinvolved a rating of “Developing” and a rating of“Promising.” A year ago the greater number of discrep-ancies involved a rating of “Promising” and a rating of“Accomplished.” Nineteen percent of the discrepanciesinvolved ratings of “Beginning” and “Developing.” Only6 percent of the discrepancies involved a rating of “Begin-ning” and a “No Evidence” rating. Aspect 2 ofDimension 2 registered the fewest cases of discrepancies.
22
TABLE 4
Time Employed by Each Rater in Minutes
23
TABLE 5
Total Count and Relative Frequency By Category and Rater
24
TABLE 6
Comparison of Portfolios and Raters
25
*Absolute Difference = Number indicates the difference between the two ratings given by separate raters to the same portfolio.
**Undefined = Difference in rating could not be calculated because one rater considered that there was not enough evidence to judge a portfolioand the other rater gave the portfolio a rating (usually a rating of 2 or 3).
TABLE 6 (continued)
26
TABLE 8
Specific Discrepancies Between Raters
Dimension 1, Demonstrating Knowledge of Hispanic CulturesAspect 1: Showing awareness of the diversity of Hispanic culturesAspect 2: Identifying contributions of Hispanic figuresAspect 3: Making connections with other disciplines and own culture(s)Dimension 2, Using Spanish to Communicate EffectivelyAspect 1: Deriving meaning from printed materials and oral discourseAspect 2: Expressing meaning in oral and written form
TABLE 7
Discrepancies Between Raters
Dimension 1, Demonstrating Knowledge of Hispanic CulturesAspect 1: Showing awareness of the diversity of Hispanic culturesAspect 2: Identifying contributions of Hispanic figuresAspect 3: Making connections with other disciplines and own culture(s)Dimension 2, Using Spanish to Communicate EffectivelyAspect 1: Deriving meaning from printed materials and oral discourseAspect 2: Expressing meaning in oral and written form
27
Correlation Among the Ratings of Portfolios Each portfolio received seven ratings: one rating for eachof the three aspects of Dimension 1, one for each of thetwo aspects of Dimension 2, and two total ratings. Table 9summarizes the correlations among the various ratings.Correlations among the three aspects in Dimension 1range from .555 to .611, and the correlation between thetwo aspects of Dimension 2 is .615. The correlationbetween the three aspects of Dimension 1 and the totalfor Dimension 1 range from .702 to .830. The correla-tion between the two aspects of Dimension 2 and thetotal for Dimension 2 are .842 and .835, respectively.The correlation between the two dimensions ismoderately high at .653. With the exception of thecorrelations between aspect ratings and total dimensionratings, the correlations are all higher than last year.
Reliability StudyAs was done last year, FACETS (Linacre, 1989) wasused to summarize the data. FACETS constructs linearmeasures from qualitatively ordered counts by means offaceted Rasch analysis. Each observation is the outcomeof an interaction between elements of facets (e.g., a“student’s” portfolio evaluated by a “rater” on several“aspects”). Each observation provides FACETS withinformation about the elements that interact to con-struct it. From this information, FACETS estimates aquantitative measure for each element of each facet(e.g., each student, each rater, each portfolio aspect).
The measures for the elements obtained from oneanalysis are all in the same linear frame of reference onone common interval scale.
For this study, two partial credit models and a ratingscale model were used. The partial credit models wereconstrained with regard to rater and aspects of theportfolio. The partial credit model for raters allows usto compare the raters. The partial credit model foraspects allows us to compare the rating scales forthe aspects. Finally, the rating scale model uses all theinformation together to compare students, raters, andaspects of the portfolio.
The output in FACETS for these models is a set ofrulers that provides a graphical description of thevariable. A “+” or “−” before the facet name indicateswhether the facet measures are positively or negativelyoriented. The vertical axis provides a linear definition ofthe variable. Each element name is positioned accordingto its measure. Elements with extreme scores are notpositioned by measure but are placed at the extreme topor bottom of the column of their facet.
Figure 1 shows the results obtained from using thepartial credit model for raters. The figure reveals that notevery rater is rating students in the same way. It appearsthat Rater 7 (R.7) is using a scale that ranges from 1 to3. Most others are using a scale that ranges from 1 to 4.Finally, Raters 1 and 10 appear to be using the full rangeof ratings offered, 1 to 5. Figure 1 also indicates thatRaters 11 and 12 are grading most leniently, and Rater 4is grading most harshly. Figure 1 indicates that studentstend to receive higher ratings on aspects from Dimension1, “Demonstrating Knowledge of Hispanic Cultures,”than those from Dimension 2, “Using Spanish to
TABLE 9
Bivariate Correlation MatrixDimension 1 Dimension 1 Dimension 1 Dimension 1 Dimension 2 Dimension 2 Dimension 2Aspect 1 Aspect 2 Aspect 3 Total Aspect 1 Aspect 2 Total
Dimension 1 Aspect 1 1.000 .568 .555 .702 .398 .264 .429
Dimension 1 Aspect 2 1.000 .611 .830 .606 .511 .576
Dimension 1 Aspect 3 1.000 .795 .573 .359 .525
Total Dimension 1 1.000 .642 .543 .653
Dimension 2 Aspect 1 1.000 .615 .842
Dimension 2 Aspect 2 1.000 .835
Total Dimension 2 1.000
Dimension 1, Demonstrating Knowledge of Hispanic Cultures
Aspect 1: Showing awareness of the diversity of Hispanic cultures
Aspect 2: Identifying contributions of Hispanic figures
Aspect 3: Making connections with other disciplines and own culture(s)
Dimension 2, Using Spanish to Communicate Effectively
Aspect 1: Deriving meaning from printed materials and oral discourse
Aspect 2: Expressing meaning in oral and written form
Figure 1. FACETS: Partial credit model for raters.
28
not use the scoring rubrics in the same manner; sometend to rate more leniently than others.
The second model used was the partial credit modelfor aspects. The results obtained from using that modelare summarized in Figure 2. Figure 2 indicates that therating scale for one aspect functions somewhat differ-ently from another. The aspects generally fall into tworating scale groups. Group 1 consists of Aspects 1, 2, and3 for Dimension 1, and total for Dimension 1 as well asAspect 1 for Dimension 2. These aspects appear to berated on a 1 to 4 scale. All these aspects refer either toreceptive language skills (Deriving meaning from printed
Communicate Effectively.” This may be explained bythe fact that teachers have more experience ratinglinguistic skills than the concepts included in the aspectsfor “Demonstrating Knowledge of Hispanic Cultures.”
Table 10 shows each rater’s observed average (i.e.,average of the raw score rating given) and his or her“fair” average (the observed average adjusted for thedeviation of the portfolio in each rater’s sample fromthe overall mean of the portfolios across all raters). Therater separator index of 5.24 indicates that the tenraters can be separated into five statistically distinctseverity strata. These results suggest that the raters do
Dimension 1, Demonstrating Knowledge of Hispanic Cultures
Aspect 1: Showing awareness of the diversity of Hispanic cultures
Aspect 2: Identifying contributions of Hispanic figures
Aspect 3: Making connections with other disciplines and own culture(s)
Dimension 2, Using Spanish to Communicate Effectively
Aspect 1: Deriving meaning from printed materials and oral discourse
Aspect 2: Expressing meaning in oral and written form
29
TABLE 10
Rater Severity
44, 48, 52, 57, 70, 73, and 79) would have had their totalscores changed if we had adjusted their scores for raterseverity. This rate of 20 is less than the 29 rate of changelast year. It should be noted that the 14 students withasterisks in the observed score column have unusualpatterns of ratings (i.e., more than typical variations intheir ratings). In each case, there are one or more unex-pected ratings—aberrant ratings that don’t seem to “fit”with the others. A mean-square infit or outfit valuegreater than 1.5 indicates a student’s portfolio mightneed another review before a score report is issued,particularly if the student’s total score is near a criticaldecision-making point in the score distribution.
AnalysesWe ran two FACETS analyses: one on the ratings of the21 portfolios read twice in the June 1996 portfolioreading, and one on the ratings of the 31 portfoliosread twice in the June 1997 portfolio reading. We thenexamined selected pieces of output from the two anal-yses to see how they compared.
StudentsEach of the portfolios received 14 ratings (i.e., sevenratings each from two raters). For each portfolio,FACETS averaged the ratings to compute a total score.We then compared the range of total scores acrossyears. The range of portfolio scores was somewhatwider in 1997 than in 1996. The total scores for the 21portfolios rated in 1996 ranged from 2.4 to 4.3. Bycomparison, the total scores for the 31 portfolios readin 1997 ranged from 1.5 to 4.0.
materials and oral discourse) or a conceptual introspec-tion of different elements of the Spanish-speaking worldand cultures (“Demonstrating Knowledge of HispanicCultures”). In both cases, it is harder to apply judgementor show evidence for the type of documents or behaviorthat demonstrate evidence of achievement of these.Group 2 consists of Aspect 2 for Dimension 2 (Expres-sing meaning in oral and written form) and total forDimension 2, “Using Spanish to Communicate Effec-tively.” These aspects appear to be rated on a 1 to 5 scale,although a rating of 5 is difficult to achieve. These aspectsinclude productive skills, which are usually measuredand evaluated in language classrooms. Students andteachers have more experience with the kind of evidencethat is necessary to show achievement in these aspects.
The final model is the rating scale model. This modeldoes not constrain or condition any facet. This modelwould be valid if the partial-credit models showed thatthe raters used the aspect rating scales in a similarfashion, and that the aspect scales functioned similarly.In our case they don’t; consequently, the rating scalemodel is used with caution. The results obtained usingthis model are shown in Figure 3. Figure 3 indicates thatstudents tend to get higher ratings on Dimension 1 thanDimension 2. It appears that there were three distinctstrata of rater severity. Raters 4, 7, 8, and 9, who werethe most harsh, form the first stratum. Raters 2, 5, 10,and 1 form a second stratum. Raters 11 and 12, themost lenient, form a third stratum. The distribution ofstudents appears skewed, with some outliers at the top.
Figure 3A shows that differences between the students’observed averages and fair averages indicate that 20 stu-dents (12, 17, 18, 20, 23, 24, 25, 26, 30, 32, 36, 37, 40,
30
On average, students did better in 1996 than in1997. The average total score for the 21 portfolios ratedin 1996 was 3.3 (SD = 0.4); the average total score forthe 31 portfolios rated in 1997 was 2.5 (SD = 0.6).
When FACETS calculates “fair averages” for students,it adjusts each student’s total score for the severity orleniency of the two raters who rated that particular
student. On average, the amount of difference betweenthe “observed average” (i.e., the unadjusted raw scoreaverage of the raters’ ratings of a given student) and the“fair average” (i.e., the observed average adjusted forrater severity or leniency differences) was 0.2 in both1996 and 1997. The largest severity or leniency adjust-ment for any portfolio was 0.7 in 1997 and 0.5 in 1997.
Figure 2. FACETS: Partial credit model for aspects.
Dimension 1, Demonstrating Knowledge of Hispanic Cultures
Aspect 1: Showing awareness of the diversity of Hispanic cultures
Aspect 2: Identifying contributions of Hispanic figures
Aspect 3: Making connections with other disciplines and own culture(s)
Dimension 2, Using Spanish to Communicate Effectively
Aspect 1: Deriving meaning from printed materials and oral discourse
Aspect 2: Expressing meaning in oral and written form
31
Figure 3. FACETS: Rating scale model.
Dimension 1, Demonstrating Knowledge of Hispanic Cultures
Aspect 1: Showing awareness of the diversity of Hispanic cultures
Aspect 2: Identifying contributions of Hispanic figures
Aspect 3: Making connections with other disciplines and own culture(s)
Dimension 2, Using Spanish to Communicate Effectively
Aspect 1: Deriving meaning from printed materials and oral discourse
Aspect 2: Expressing meaning in oral and written form
32
Figure 3A. FACETS: Student adjusted scores.
33
RatersIn 1996, in Phase I of the portfolio study, each of theseven raters rated five portfolios, but not all raters ratedthe same set of five portfolios. For each rater, FACETScalculated the average of the ratings that rater gave thefive portfolios she or he rated. The average of theratings each rater gave ranged from 3.0 for Rater 5 to3.6 for Raters 3 and 4.
In 1997, in Phase II of the portfolio study, each of the10 raters rated from two to nine portfolios. The average
of the ratings each rater gave ranged from 1.8 for Rater4 to 3.1 for Rater 11. On average, raters tended to givelower ratings to the portfolios scored in 1997 than tothose scored in 1996.
Aspects and DimensionsWhen raters used the scoring rubrics for the variousaspects and dimensions, there were differences in 1996and 1997 in the percentages of ratings falling in eachscale category. Those differences are shown in Table 11.
Figure 3A (continued).
TABLE 11
Percentage in Each Scale Category
34
Dimension 1, “Demonstrating Knowledge of HispanicCultures”
Aspect 1, Showing awareness of the diversity ofHispanic cultures
• In 1996, raters used all five scale categories; in 1997,raters used only four scale categories (i.e., 1 to 4).
• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.
Aspect 2, Identifying contributions of Hispanic figures
• In 1996 and 1997, raters used four of the five scalecategories (i.e., 1 to 4).
• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.
Aspect 3, Making connections with other disciplinesand own culture(s)
• In 1996, raters used only three scale categories (i.e.,3 to 5); in 1997, raters used four scale categories (i.e.,1 to 4).
• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.
Dimension 1
• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used four scale categories (i.e., 1 to 4).
• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.
Dimension 2, “Using Spanish to CommunicateEffectively”
Aspect 1, Deriving meaning from texts and personalinteractions
• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used four scale categories (i.e., 1 to 4).
• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.
Aspect 2, Expressing meaning in oral and written form
• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used all five scale categories.
• In 1996, the majority of ratings were 3; in 1997, themajority of ratings were 2 or 3.
Dimension 2
• In 1996, raters used four scale categories (i.e., 2 to 5);in 1997, raters used all five scale categories.
• In 1996, the majority of ratings were 3 or 4; in 1997,the majority of ratings were 2 or 3.
Summary of Findings• This research showed that the aspects of the de-
lineated Pacesetter Spanish system of portfolioassessment worked better in Phase II than in Phase I.Students and teachers had a better understanding ofwhat constitutes evidence of achievement as well asof the selection process and organization of artifactsincluded in the portfolios.
• Pacesetter Spanish portfolio raters used the categoriesof “Promising” and “Developing” for the majority ofratings, but less than in 1996, and not the extremecategories (“Accomplished” or “Advanced”).
• The length of time to rate the portfolios in 1996 and1997 was an average of 45 to 50 minutes. Althoughvariety and complexity are encouraged throughout,this element of time needs to be taken into accountwhen preparing to read portfolios.
• Although higher than in 1996, the number of ratingagreements was moderate. However, there were aconsiderable number of discrepancies between ratersregarding the presence or lack of enough artifacts tojudge students’ performance within the parameters ofthe Pacesetter Spanish portfolio assessment system.
• The proportion of ratings of “No Evidence” forDimension 1, Aspect 3 was dramatically reduced.
• Raters differ in level of severity exercised when rat-ing portfolios. Some grade significantly more harshlythan others. About one-fourth of the students wouldhave received different total scores if their scores hadbeen adjusted for differences in rater severity.
• The rating scales for the individual aspects functionsomewhat differently from one another. Raters usedall five-scale categories for some of the rating scalesbut only used four-scale ratings for other scales.
• The rating patterns for about 20 percent of theportfolios contained one or more surprising or un-expected ratings (i.e., ratings that don’t seem to “fit”with the other ratings given to that portfolio). Whensuch variability is present in a set of ratings, it issuggested that those in charge of monitoring qualitycontrol in the scoring session give those portfolios asecond look to determine which ratings are “out ofsync” with the other ratings, and whether thoseunexpected ratings shall stand “as is” or be changed.
Recommendations• A final version of the matrix should be incorporated
as part of the formal components of the PacesetterSpanish program.
35
• A set of guidelines should be included in theTeacher’s and Student’s Edition of the PacesetterSpanish curriculum materials. The guidelines shouldinclude clear instructions of the selection process ofwork samples for the portfolio, cover sheets, rubrics,expectations, and any other pertinent information.
• Activities in the Pacesetter Spanish curriculummaterials should be designed to reflect all aspects anddimensions of the assessment portfolios so studentshave ample opportunity to demonstrate their achieve-ment of the course outcomes.
• Dissemination of the portfolio component of thePacesetter Spanish program should be carried outat the different professional development sessions.Training sessions should include an overview of thesteps to follow when implementing the system ofportfolio assessment as well as an explanation onhow to apply the matrix when evaluating portfolios.
• New benchmark portfolios from the 1997 PacesetterSpanish reading should be identified, and a set ofsample portfolios should be prepared for use intraining sessions.
• Efforts should be made to organize scoring sessionsat the national or regional level for Pacesetter Spanishportfolios so teachers can have an opportunity toapply the assessment matrix and have discussions onwhat constitutes evidence of achievement for eachaspect and dimension.
• Further research is needed to ensure that the evalua-tion of portfolios in the Pacesetter Spanish program isdone effectively and that all quality control measuresare taken.
• Additional research is necessary to evaluate the dif-ferent elements that comprise the Pacesetter Spanishportfolio assessment system. Results could guidefurther refinement of this component and clarify itsconnection to the goals of the overall program.
• Evaluative studies should be conducted to measurethe effectiveness and impact of the Pacesetter Spanishprogram in general. It is particularly important tofind out how each component contributes to thesuccess of the program and its vision of integratingstandards, professional development, instruction, andassessment.
ReferencesFercsey, A., Luna, C. and Ponte, E. (1997). Pacesetter Spanish
Portfolio Research Project: Phase I Report. [Unpublishedmonograph]. Princeton, NJ: Educational Testing Service.
Linacre, J.M. and Wright, B.D. (1994). A User’s Guide toFACETS: Rasch Measurement Computer Program.[Computer program manual]. Chicago: MESA Press.
Sheingold, K., Storms, B., Thomas, W. and Heller, J. (1997).Pacesetter English Portfolio Assessment: 1995 Report.Center for Performance Assessment. Princeton, NJ:Educational Testing Service.
Phase I Appendix
39
Appendix 1: DRAFT COVER SHEET
PACESETTER SPANISH PORTFOLIO ASSESSMENT
NAME:
Selection: Date work was done:
Why did you select this piece of work?
What does it show or tell about you?
What did you learn from doing this assignment?
Please check the dimension(s) or aspects for which you are showingevidence with this assignment:
• Dimension 1: Demonstrating Knowledge of Hispanic Cultures
Showing Awareness of the Diversity of Hispanic Cultures
Identifying Contributions of Hispanic Figures
• Dimension 2: Using Spanish to Communicate Effectively
Deriving Meaning from Texts and Personal Interactions
Expressing Meaning in Oral and Written Form
40
Appendix 1: DRAFT COVER SHEET (Continuation)
PACESETTTER SPANISH PORTFOLIO ASSESSMENT
NOMBRE:
Título: Fecha de ejecución:
¿Por qué seleccionaste esta obra?
¿Qué muestra o dice esta obra sobre ti?
¿Qué aprendiste de esta tarea?
He seleccionado esta obra para indicar evidencia en las siguientesdimensiones:
• Dimensión 1: Demostrar conocimiento de las culturas hispanas
Demostrar conciencia de la diversidad de las culturas hispanas
Identificar contribuciones de figuras hispanas
• Dimensión 2: Usar el español para comunicaciones eficientes
Comprensión de textos e interacciones personales
Expresión oral y escrita
41
Ap
pen
dix
2: A
ssessm
en
t M
atrix
PA
CE
SE
TT
ER
S
PA
NIS
H
DIM
EN
SIO
N: D
EM
ON
ST
RA
TIN
G K
NO
WL
ED
GE
O
F H
IS
PA
NIC
C
UL
TU
RE
S
Po
rtfo
lio
A
ssessm
en
t: E
valu
atin
g A
sp
ects o
f th
e D
im
en
sio
n
Beg
in
nin
g
�
Dev
elo
pin
g
�
Pro
misin
g
�
Acco
mp
lish
ed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
id
en
ce to
ju
dg
e�
Sh
ow
in
g
Aw
aren
ess o
f
th
e D
iv
ersity
of H
isp
an
ic
Cu
ltu
res
Vag
ue in
dicatio
n
frag
men
tary
, su
perficial
Lim
ited
-in
co
mp
lete
-a
few
sp
ecific
ex
am
ples
-u
ne
ve
n
-p
artia
l u
nd
ev
elo
ped
So
me ex
am
ples
-g
en
eral in
sig
hts
-m
id
dle ran
ge o
f
in
sig
hts/
ex
am
ples
Am
ple/
fu
ll, w
id
e
ran
ge o
f ex
am
ples,
deep
er/
mo
re
pro
fo
un
d
Co
nsisten
tly
in
sig
htfu
l,
va
ried
, ex
ten
ded
Beg
in
nin
g
�
Dev
elo
pin
g
�
Pro
misin
g
�
Acco
mp
lish
ed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
id
en
ce to
ju
dg
e�
Id
en
tify
in
g
Co
ntrib
utio
ns
of H
isp
an
ic
Fig
ures
List a
few
n
am
es a
nd
co
ntrib
utio
ns
brief, sk
etch
y
descrip
tio
n
So
me d
escrip
tio
n o
f
co
ntrib
utio
ns
-lim
ited
n
otio
ns o
f
co
ntrib
utio
n
So
me in
feren
ces ab
ou
t
pla
ce a
nd
im
pa
ct in
histo
ry
Reco
gn
itio
n o
f im
pact
in
h
isto
ry
-a
mp
le in
feren
ces
ab
ou
t p
lace in
h
isto
ry
Th
oro
ug
h in
feren
ces
-co
nsisten
tly
in
sig
htfu
l
an
aly
sis o
f co
ntrib
utio
ns
an
d im
pact/
ro
le in
histo
ry
Beg
in
nin
g
�
Dev
elo
pin
g
�
Pro
misin
g
�
Acco
mp
lish
ed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
id
en
ce to
ju
dg
e�
Mak
in
g
co
nn
ectio
ns
with
o
th
er
discip
lin
es
an
d o
wn
cu
ltu
re(s)
Iso
lated
-n
o lo
gical co
nn
ectio
ns
-few
, irrelev
an
t,
clu
eless, su
perficial
co
nn
ectio
ns
Lim
ited
-p
artial
-n
ot alw
ay
s relev
an
t
-u
nev
en
co
nn
ectio
ns
So
me g
en
eral n
otio
ns
with
ou
t m
uch
dep
th
/in
sig
ht
Am
ple ex
am
ples o
f
in
sig
htfu
l an
d w
id
e
ran
ge o
f co
nn
ectio
ns
-so
me in
feren
ces
Th
oro
ug
h an
d in
sig
htfu
l
co
nn
ectio
ns
-n
ew
-b
ey
on
d w
ha
t th
ey
'v
e
seen
in
co
urse
-creativ
ity
-o
rig
in
ality
-o
rig
in
al co
nn
ectio
ns
42
Ap
pen
dix
2
: A
ssessm
en
t M
atrix
(C
on
tin
ua
tio
n)
PA
CE
SE
TT
ER
S
PA
NIS
H
DIM
EN
SIO
N: U
SIN
G S
PA
NIS
H T
O C
OM
MU
NIC
AT
E E
FF
EC
TIV
EL
Y
Po
rtfo
lio
A
ssessm
en
t: E
valu
atin
g A
sp
ects o
f th
e D
im
en
sio
n
Beg
in
nin
g
�
Dev
elo
pin
g
�
Pro
misin
g
�
Acco
mp
lish
ed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
id
en
ce to
ju
dg
e�
Deriv
in
g
mean
in
g
fro
m tex
ts
an
d p
erso
nal
in
teractio
ns
(Sk
ills:
Listen
in
g
an
d
Read
in
g)
Un
dersta
nd
s few
gen
era
l id
ea
s
-co
ntin
uo
us
misu
nd
ersta
nd
in
gs
Un
ev
en
co
mp
reh
en
sio
n,
freq
uen
t/
nu
mero
us
misu
nd
erstan
din
gs
-u
nd
ersta
nd
s so
me
main
id
eas o
n fam
iliar
to
pics
Un
dersta
nd
s m
ain
id
ea
s
an
d so
me d
eta
ils o
n
so
mew
ha
t fa
milia
r to
pics
-m
ay
m
ak
e so
me
in
feren
ces d
ep
en
din
g o
n
co
mp
lex
ity
o
f tex
t
Un
dersta
nd
s m
ain
id
ea
s
an
d m
ost d
eta
ils ev
en
on
u
nfa
milia
r to
pics
-ca
n m
ak
e in
feren
ces
ap
pro
pria
te to
th
e ta
sk
Th
oro
ug
h
un
dersta
nd
in
g,
ex
tra
po
la
tes
in
fo
rm
atio
n in
n
ew
way
s, in
clu
din
g so
me
so
cia
l a
nd
cu
ltu
ra
l
referen
ces a
nd
affectiv
e o
verto
nes
Beg
in
nin
g
�
Dev
elo
pin
g
�
Pro
misin
g
�
Acco
mp
lish
ed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
id
en
ce to
ju
dg
e�
Ex
pressin
g
mean
in
g in
oral an
d
written
fo
rm
(Sk
ills:
S
peak
in
g
an
d W
ritin
g)
Co
mm
un
ica
tio
n
im
ped
ed
b
y
in
terferen
ce fro
m
an
oth
er la
ng
ua
ge
-lack
o
f flu
en
cy
-in
ad
eq
ua
te v
oca
bu
la
ry
an
d/
or p
ro
nu
ncia
tio
n
-fra
gm
en
ted
-rep
etitio
us m
essa
ge
fo
rces in
terp
reta
tio
n,
-p
ro
nu
ncia
tio
n
ham
pers
co
mm
un
ica
tio
n
Lim
ited
flu
en
cy
-co
nv
ey
s sim
ple
messag
es
-u
nco
nn
ected
d
isco
urse
or sen
ten
ces
-lim
ited
ra
ng
e o
f
vo
ca
bu
la
ry
-so
me in
terferen
ce fro
m
an
oth
er la
ng
ua
ge
-p
ro
nu
ncia
tio
n m
ay
in
terfere w
ith
co
mm
un
ica
tio
n
So
me u
nd
erly
in
g
org
an
iz
atio
n o
f th
ou
gh
ts/
id
eas/
disco
urse
-em
erg
in
g n
otio
n o
f
ad
eq
ua
te reg
ister
-a
deq
ua
te ra
ng
e o
f
vo
ca
bu
la
ry
-self-co
rrectio
ns
-p
ro
nu
ncia
tio
n m
ay
b
e
aw
kw
ard
b
ut d
oesn
't
in
terfere w
ith
co
mm
un
ica
tio
n
-lev
el o
f flu
en
cy
d
oesn
't
in
terfere w
ith
co
mm
un
ica
tio
n
Messa
ge is a
rticu
la
ted
(o
rg
an
iz
ed
ap
pro
pria
tely
) -p
ro
per
use o
f circu
mlo
cu
tio
n
-a
t tim
es crea
tiv
e id
ea
s,
-co
heren
t u
se o
f
vo
ca
bu
la
ry
-m
in
im
al in
terferen
ce
fro
m a
no
th
er la
ng
ua
ge
-m
ostly
a
deq
ua
te u
se o
f
reg
ister
Hig
hly
crea
tiv
e,
-th
oro
ug
h, m
ea
nin
gfu
l,
-ev
id
en
tly
o
rg
an
iz
ed
,
-w
ell a
rticu
la
ted
a
nd
org
an
ized
m
essag
e,
-ap
pro
pria
te u
se o
f
reg
ister
-w
id
e ra
ng
e o
f
vo
ca
bu
la
ry
-little o
r n
o in
terferen
ce
fro
m a
no
th
er la
gu
ag
e,
-h
ig
h lev
el o
f flu
en
cy
43
Appen
dix
3S
tuden
t N
um
ber
:R
ater
Nu
mb
er:
A
rtif
acts
Use
d
Dim
ensi
on
1:
Dem
on
stra
tin
g K
no
wle
dg
e o
f H
isp
anic
Cu
ltu
res
Dim
ensi
on
2:
Usi
ng
Sp
anis
h t
o C
om
mu
nic
ate
N
ote
s/
C
om
men
ts
Div
ersi
ty o
f H
isp
anic
Cu
ltu
res
Co
ntr
ibu
tio
ns
of
His
pan
ic F
igu
res
Co
nn
ecti
on
s w
ith
oth
er d
isci
pli
nes
Der
ivin
g m
ean
ing
fro
m t
exts
(L
&R
)E
xpre
ss. m
eanin
gfr
om
tex
ts (
S&
W)
S
core
Scale
: B
eg
inn
ing
, D
ev
elo
pin
g,
Pro
mis
ing
, A
cco
mp
lish
ed
, A
dv
an
ced
, N
ot
en
ou
gh
ev
iden
ce t
o j
ud
ge.
Oth
er c
om
men
ts/s
ugges
tions:
44
Appen
dix
4
Po
rt.
# R
ater
Dat
eT
ime
Use
dD
imen
sio
n 1
Asp
ect
1D
imen
sio
n 1
Asp
ect
2D
imen
sio
n 1
Asp
ect
3D
imen
sio
n 2
Asp
ect
1D
imen
sion 2
Asp
ect
2D
1D
2
11
6S
t/E
v
2
0m
Pro
mis
ing
Pro
mis
ing
No e
vid
ence
No e
vid
ence
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
15
Su
/M
3
5m
Acc
om
pli
shed
Pro
mis
ing
NA
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
25
St/
Ev
20
mD
evel
op
ing
Dev
elo
pin
gN
AP
rom
isin
gD
evel
opin
gD
evel
opin
gD
evel
opin
g
24
St/
Ev
30
mP
rom
isin
gN
o e
vid
ence
No e
vid
ence
No e
vid
ence
Pro
mis
ing
No e
vid
ence
Pro
mis
ing
34
St/
Ev
20
mN
o e
vid
ence
Pro
mis
ing
No e
vid
ence
Dev
elo
pin
gP
rom
isin
gN
o e
vid
ence
Pro
mis
ing
33
Su
/M
5
5m
Dev
elo
pin
gP
rom
isin
gA
cco
mp
lish
edA
cco
mp
lish
edA
ccom
pli
shed
Pro
mis
ing
Acc
om
pli
shed
47
St/
Ev
1h
5m
Pro
mis
ing
Beg
inn
ing
Pro
mis
ing
Acc
om
pli
shed
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
42
Su
/M
4
0m
Pro
mis
ing
Pro
mis
ing
No e
vid
ence
Pro
mis
ing
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
51
St/
Ev
1h
Pro
mis
ing
Pro
mis
ing
NA
Acc
om
pli
shed
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
56
St/
Ev
35
mP
rom
isin
gN
o e
vid
ence
No e
vid
ence
Pro
mis
ing
Pro
mis
ing
No e
vid
ence
Pro
mis
ing
61
St/
Ev
30
mP
rom
isin
gA
cco
mp
lish
edN
AA
cco
mp
lish
edA
ccom
pli
shed
Pro
mis
ing
Acc
om
pli
shed
66
St/
Ev
30
mP
rom
isin
gP
rom
isin
gN
o e
vid
ence
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
72
St/
Ev
Acc
om
pli
shed
Acc
om
pli
shed
No e
vid
ence
Acc
om
pli
shed
Acc
om
pli
shed
Acc
om
pli
shed
Acc
om
pli
shed
77
Su
/M
5
0m
Dev
elo
pin
gA
cco
mp
lish
edN
o e
vid
ence
Acc
om
pli
shed
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
84
St/
Ev
No e
vid
ence
No e
vid
ence
No e
vid
ence
Pro
mis
ing
Acc
om
pli
shed
No e
vid
ence
Acc
om
pli
shed
83
Su
/M
3
5m
Pro
mis
ing
Pro
mis
ing
Acc
om
pli
shed
Acc
om
pli
shed
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
1 P
ort
foli
o n
um
ber
s in
bo
ld i
nd
icat
e th
e p
ort
foli
o w
as s
elec
ted
as
a b
ench
mar
k.
45
Ap
pen
dix
4 -
Co
nti
nu
atio
n
Po
rt.
# R
ater
Dat
eT
ime
Use
dD
imen
sio
n 1
Asp
ect
1D
imen
sio
n 1
Asp
ect
2D
imen
sio
n 1
Asp
ect
3D
imen
sio
n 2
Asp
ect
1D
imen
sion 2
Asp
ect
2D
1D
2
96
St/
Ev
20
mP
rom
isin
gA
cco
mp
lish
edP
rom
isin
gN
o e
vid
ence
Acc
om
pli
shed
Pro
mis
ing
No e
vid
ence
95
St/
Ev
25
mA
cco
mp
lish
edA
cco
mp
lish
edN
AA
cco
mp
lish
edA
ccom
pli
shed
Acc
om
pli
shed
Acc
om
pli
shed
10
3S
t/E
v
5
0m
Beg
inn
ing
Acc
om
pli
shed
NA
Acc
om
pli
shed
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
10
1S
u/M
20
mD
evel
op
ing
Pro
mis
ing
NA
Pro
mis
ing
Pro
mis
ing
Dev
elopin
gP
rom
isin
g
11
3S
t/E
vA
cco
mp
lish
edA
cco
mp
lish
edN
AA
cco
mp
lish
edA
cco
mpli
shed
Acc
om
pli
shed
Acc
om
pli
shed
11
1S
u/M
40
mA
cco
mp
lish
edP
rom
isin
gP
rom
isin
gA
cco
mp
lish
edA
cco
mpli
shed
Pro
mis
ing
Acc
om
pli
shed
12
6S
t/E
v
4
0m
Pro
mis
ing
No e
vid
ence
No e
vid
ence
Acc
om
pli
shed
Pro
mis
ing
No e
vid
ence
Pro
mis
ing
12
5S
u/M
45
mA
cco
mp
lish
edP
rom
isin
gN
AP
rom
isin
gP
rom
isin
gA
ccom
pli
shed
Pro
mis
ing
13
5S
t/E
v
4
5m
Pro
mis
ing
Dev
elo
pin
gN
o e
vid
ence
Dev
elo
pin
gP
rom
isin
gD
evel
opin
gP
rom
isin
g
13
4S
t/E
v
3
5m
No e
vid
ence
No e
vid
ence
No e
vid
ence
Pro
mis
ing
Pro
mis
ing
No e
vid
ence
Pro
mis
ing
14
6--
--
No e
vid
ence
No e
vid
ence
No e
vid
ence
No e
vid
ence
Pro
mis
ing
No e
vid
ence
No e
vid
ence
14
4S
t/E
v
4
0m
Pro
mis
ing
No e
vid
ence
No e
vid
ence
Pro
mis
ing
Pro
mis
ing
No e
vid
ence
Pro
mis
ing
15
1S
t/E
v
3
0m
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
Acc
om
pli
shed
Pro
mis
ing
Pro
mis
ing
15
6S
u/M
40
mP
rom
isin
gN
o e
vid
ence
Pro
mis
ing
Acc
om
pli
shed
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
16
2S
t/E
v 1
hA
cco
mp
lish
edP
rom
isin
gP
rom
isin
gA
cco
mp
lish
edA
ccom
pli
shed
Pro
mis
ing
Acc
om
pli
shed
16
7N
A 1
hA
cco
mp
lish
edP
rom
isin
gP
rom
isin
gA
cco
mp
lish
edA
ccom
pli
shed
Acc
om
pli
shed
Acc
om
pli
shed
17
7S
t/E
v 1
h 1
0m
Pro
mis
ing
Pro
mis
ing
No e
vid
ence
No e
vid
ence
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
17
4S
u/M
55
mA
dvan
ced
Acc
om
pli
shed
Advan
ced
Advan
ced
Advan
ced
Advan
ced
Advan
ced
46
Ap
pen
dix
4-C
on
tin
uat
ion
Po
rt.
# R
ater
Dat
eT
ime
Use
dD
imen
sio
n 1
Asp
ect
1D
imen
sio
n 1
Asp
ect
2D
imen
sio
n 1
Asp
ect
3D
imen
sio
n 2
Asp
ect
1D
imen
sion 2
Asp
ect
2D
1D
2
18
5S
t/E
v
4
0 m
Beg
innin
gP
rom
isin
gN
AA
cco
mp
lish
edP
rom
isin
gP
rom
isin
gP
rom
isin
g
18
4S
u/M
30
mN
o e
vid
ence
Pro
mis
ing
No e
vid
ence
No e
vid
ence
Acc
om
pli
shed
No e
vid
ence
No e
vid
ence
19
2S
t/E
v
4
5m
Dev
elop
ing
Dev
elo
pin
gN
o e
vid
ence
Pro
mis
ing
Dev
elopin
gD
evel
opin
gD
evel
opin
g
19
7N
A 1
hP
rom
isin
gP
rom
isin
gN
o e
vid
ence
Acc
om
pli
shed
Pro
mis
ing
Pro
mis
ing
Pro
mis
ing
20
3S
t/E
v 1
hP
rom
isin
gA
cco
mp
lish
edA
cco
mp
lish
edA
cco
mp
lish
edA
ccom
pli
shed
Acc
om
pli
shed
Acc
om
pli
shed
20
1S
u/M
35
mP
rom
isin
gP
rom
isin
gN
AP
rom
isin
gA
cco
mpli
shed
Pro
mis
ing
Pro
mis
ing
21
7S
u/M
50
mP
rom
isin
gA
cco
mp
lish
edN
o e
vid
ence
Acc
om
pli
shed
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
21
2S
u/M
Pro
mis
ing
Pro
mis
ing
NA
Acc
om
pli
shed
Acc
om
pli
shed
Pro
mis
ing
Acc
om
pli
shed
47
Appen
dix
5
PO
RT
FO
LIO
S R
AT
ED
BY
TW
O D
IF
FE
RE
NT
PE
RS
ON
S∗
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
31938
OK
-Perf
ect
Bio
gra
fía-
tape
(D1&
2)∗
∗S
pee
ch (
D2)
Work
Shee
t (D
1&
2)
Guía
Mex
ico (
D1&
2)
Work
Shee
t (D
1&
2)
Dim
ensi
on 2
: P
rom
isin
g;
how
ever
, not
enough e
vid
ence
to j
udge
rece
pti
ve
skil
ls.
Pro
ject
(D
2)
Vid
eo (
D2)
Tap
e (D
1&
2)
Itin
erar
io (
D1&
2)
Rea
din
g (
D1&
2--
1N
E)
[?]
on f
ood (
D1&
2)
Dim
ensi
on 1
: P
rom
isin
g (
had
ther
e bee
n m
ore
evid
ence
it
mig
ht
hav
e bee
nA
ccom
pli
shed
).
Dim
ensi
on 2
: P
rom
isin
g (
not
enough e
vid
ence
!)
∗ T
his
is
a t
ran
scri
pt
of
the o
rig
inal
rati
ng
sh
eets
. In
try
ing
to
be f
ait
hfu
l to
th
e o
rig
inals
, w
e h
av
e w
rite
n b
etw
een
bra
ck
ets
(i.
e.,
[])
tho
se w
ord
s th
at
were
dif
ficu
lt t
o r
ead
fro
m t
he h
an
dw
riti
ng
.
∗∗ D
1:
Inst
rum
en
t w
as
mark
ed
un
der
on
e o
r m
ore
asp
ect
of
dim
en
sio
n 1
.
D2
: In
stru
men
t w
as
mark
ed
un
der
on
e o
r m
ore
asp
ect
of
dim
en
sio
n 2
.
D1
&2
: In
stru
men
t w
as
mark
ed
un
der
on
e o
r m
ore
asp
ect
of
each
dim
en
sio
n.
D
1--
NE
or
D1
--N
E:
Inst
rum
en
t w
as
mark
ed
un
der
the r
esp
ecti
ve d
imen
sio
n b
ut
there
was
no
t en
ou
gh
ev
iden
ce t
o j
ud
ge i
ts c
on
trib
uti
on
to
th
e r
esp
ecti
ve d
imen
sio
n.
D
1&
D2
--1
NE
or
D1
&D
2--
2N
E:
Inst
rum
en
t w
as
mark
ed
un
der
on
e o
r m
ore
asp
ect
of
each
dim
en
sio
n,
bu
t n
ot
en
ou
gh
ev
iden
ce w
as
fou
nd
fo
r o
ne o
f th
e
dim
en
sio
ns.
48
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30129
Dim
ensi
on 1
:O
K/N
ED
imen
sion 2
: O
K
Char
t (D
2)
Wo
rksh
eet
(D2
)E
ssay
(D
2)
Par
agra
ph (
D1&
2)
Pam
phle
t (D
1&
2)
Post
er (
D2)
Ess
ay (
D1&
D2)
Ess
ay (
D2)
Foto
(D
1--
NE
)T
ape
(D1&
2--
2N
E,
read
tex
t)T
ape
(D1&
2--
2N
E)
Invit
ació
n (
D2)
Foto
s L
abel
ed (
D2
--N
E)
Dim
ensi
on 1
: D
iver
sity
: pro
mis
ing/
No e
vid
ence
Dim
ensi
on 2
: P
rom
isin
g
Rea
din
g:
Popo (
D2)
Pre
senta
tion:
Invit
atio
ns
(D2)
May
as P
roje
ct (
D1
&2
)P
age:
Dia
rio (
D2)
Coll
age
(D2)
Itin
erar
io:
Méx
ico
(D
1)
Coll
age:
Cer
van
tes(
D1
)In
vit
acio
nes
/expli
c. (
D2
)D
ebat
e E
coló
gic
o (
D1
&2
)P
royec
to:
Exti
nci
ón
(D
2)
Coll
age:
La
moda
(D2
--N
E)
Dim
ensi
on 1
: D
evel
opin
gD
imen
sion 2
: D
evel
opin
g
Lit
tle
evid
ence
show
n i
n s
om
e ar
eas
yet
ther
e is
A L
OT
in t
he
port
foli
o!
but
it d
oes
n't
serv
e th
e p
urp
ose
.
49
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30831
Dim
ensi
on 1
:O
K/N
E
Dim
ensi
on 2
:
Work
Shee
t (D
2)
Work
Shee
t (D
1&
2--
1N
E)
Ess
ay C
har
t (D
1&
2)
Tap
e (D
1&
2)
Para
gra
ph
(D
2)
Post
er (
D2)
Ess
ay (
D1&
2)
Para
gra
ph
(D
2)
Pam
phle
t (D
1&
2)
Invit
ació
n (
D2)
Ess
ay (
D2)
Ess
ay (
D2)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
/ P
rom
isin
g h
ispan
ic f
igure
s
Dim
ensi
on 2
: P
rom
isin
g
Note
: C
over
shee
ts n
ot
incl
uded
.
Los
Chiv
elos[
?] (
D1
)F
usi
ón (
D1)
--ev
iden
ce s
ho
wn
on
ly o
n c
ov
er s
hee
tco
mm
ents
[?]
(D2
)Is
la P
ascu
a (D
2)
Fra
nco
(D
1)
Mar
tí/J
uár
ez (
D1)
Bio
gra
fía
(D2)
Pet
e S
ampra
s (?
)C
art
a [
?]
(D2
)P
asc
ua [
?]
(D2
)B
lasó
n (
D2)
Via
je [
?]
(D2
)P
intu
ras
(D1&
2)
Dim
ensi
on 1
: P
rom
isin
gD
imen
sion 2
: A
ccom
pli
shed
50
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
31
92
9O
K-P
erfe
ctQ
uiz
(D
1&
2)
Pla
n d
e V
uel
o (
D1
&2
)F
oll
eto
Mex
ico
(D
1&
D2
)B
anan
a S
pli
t (D
2)
Vid
eo (
D2
)P
oem
a "L
a M
ura
lla"
(D
1&
D2)
Tap
e-her
o (
D1&
D2--
1N
E)
Tap
e &
Co
llag
e (D
2)
Tap
e (
?) (
D2)
Dim
en
sio
n 1
: P
rom
isin
g
Dim
en
sio
n 2
: A
cco
mp
lish
ed
Est
a es
su
vid
a (D
2)
Mex
ico
(D
1&
D2)
Ban
ana
Sp
lit
(D1
&D
2--
2?)
Her
o o
f Y
ear
(D1&
D2--
1?)
Vid
eo -
-No
t in
clu
ded
Bo
lív
ar ?
(D
1&
D2
)H
isp
anic
s in
US
A (
D1
&D
2)
La
Mura
lla
(NE
)
Dim
ensi
on
1:
Pro
mis
ing
. H
ow
ever
, st
ud
ent
cou
ld h
ave
incl
ud
ed m
ore
pro
du
cts
.
Dim
2.:
Acc
om
pli
shed
. S
tud
ent
incl
ud
es m
ore
ev
iden
ce o
f d
imen
sio
n 2
than
dim
en
sio
n 1
.N
ote
: S
he
has
mo
re a
rtif
acts
to
pro
ve
evid
ence
of
wri
tin
g s
kil
ls.
Sh
ed
idn
't i
nclu
de a
ny
ev
iden
ce o
f li
sten
ing
co
mp
reh
en
sio
n.
31296
Dim
ensi
on 1
:N
O/N
E
Dim
ensi
on 2
: O
K
Car
tel:
Logo d
e P
aca
(D2
)C
arta
(D
2)
Tap
e: E
spañ
a (D
1&
2)
Tap
e: M
adre
Ter
esa
(D1
&2
)L
ette
r: I
sla
(D1&
2--
1N
E)
Work
Shee
ts:
Coló
n, C
ort
és, Is
abel
, D
. M
arin
a (D
1)
Wri
ting:
Auto
bio
gra
fía
(D2)
Wri
ting:
Food (
D1&
2)
Sum
mar
y:
(D1&
2--
1N
E)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: C
hoic
e of
dim
ensi
on d
idn't
alw
ays
show
evid
ence
for
mak
ing a
ppro
pri
ate
judgem
ent.
S
tuden
t's w
riti
ng a
nd s
pea
kin
g d
emonst
rate
d w
ell
arti
cula
ted m
essa
ges
.
Post
er (
D2)
Car
ta (
D2)
Car
ta/l
ectu
ra (
D1&
2)
Work
shee
t (D
1&
2)
Tap
e (D
2)
Bio
gra
fía
(D2)
Receta
(D
2)
Res
earc
h (
D1
&2
)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
Dim
ensi
on 2
: P
rom
isin
g
51
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30351
Dim
ensi
on 1
: O
K-
Per
fect
Dim
ensi
on 2
:O
K
Res
um
en:
Caj
as (
D2
)T
ape:
Mar
tha
Ander
son
(D
1&
2)
Wri
ting:
Goya/
Pic
asso
(D
1)
Wri
ting:
Popo/I
xtl
a (D
2)
Wri
ting:
Guer
ra C
ivil
(D
1&
2)
Wri
ting:
Ben
it./
Cháv
ez (
D1&
2)
Post
er:
A l
a der
iva
(D2
)P
ost
er:
La
selv
a (D
2)
Post
er:
Imm
igra
tion (
D2)
Tap
e: M
alin
che/
Isab
el (
D1
&2
)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
Post
er w
ith p
rese
nta
tion (
D2)
A l
a d
eriv
a (D
2--
NE
)T
ape
(D1)
Tap
e (D
1&
D2)
Res
um
en (
D2)
Work
Shee
t (D
2)
Ess
ay (
D1)
Ess
ay (
D2)
Ess
ay (
D1&
D2)
Ess
ay (
D1&
D2)
Dim
ensi
on 1
: P
rom
isin
gD
imen
sion 2
: P
rom
isin
g
30358
Dim
ensi
on 1
: O
K
Dim
ensi
on 2
: O
K
Com
p.:
Juár
ez/M
artí
(D
1&
2)
Isab
el/M
alic
he
(D1
&2
)B
enit
o J
uár
ez (
D1&
2)
El
Gre
co (
D1
&2
)Je
rry
Gar
cía
(D1
&2
)C
oll
age:
Popo/I
xtl
a (D
1&
2)
Auto
bio
gra
fía
(D2)
Dim
ensi
on 1
: A
ccom
pli
shed
Dim
ensi
on 2
: A
ccom
pli
shed
Post
er:
Quie
n y
o ...
(D
2)
A l
a D
eriv
a (D
2)
Ixtl
a/P
opo (
D1&
2)
Jerr
y G
arcí
a (D
2)
Isab
el/L
a M
alic
he
(D1
&D
2)
Ben
ito J
uár
ez (
D1&
2)
Culm
inat
ing A
ss. (D
2)
Esp
aña/
Gre
co (
D1)
--st
uden
t did
n't
indic
ated
that
she
wan
ted t
o u
se C
.A.
as e
vid
ence
.L
os
may
as (
D1)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g
Note
: S
ize
of
som
e of
the
pro
duct
s w
as v
ery l
arge
and d
iffi
cult
to h
andle
. "I
nte
racc
iones
per
sonal
es"
- S
tuden
t under
stood i
t as
cooper
atio
n i
nst
ead o
fsp
eak
ing
.
52
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30144
Dim
1:
NO
-NE
Dim
ensi
on 2
OK
-Perf
ect
Ess
ay (
D1&
D2)
Ess
ay/P
ost
Car
d (
D2)
Wo
rk S
hee
t-p
arag
rap
h (
D2
)P
ost
er (
D2)
Pic
ture
(D
2)
Dra
win
g (
D1&
2)
Esc
udo (
D2)
Men
u (
D2)
Párr
afo
(D
2)
Post
er (
D1&
2--
NE
)P
ost
er (
D2)
Par
agra
ph (
D1&
D2
)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
Dim
ensi
on 2
: P
rom
isin
g.
El
[? -
"ile
gib
le"]
(D
1&
D2)
Bis
(D
2)
Moai
s (D
1&
D2)
Chic
os
de
hoy (
D2
)[?
-"il
egib
le"]
(D
2)
Bio
gra
fía
(D2)
Com
ida
(D1&
D2)
[? -
"il
egib
le"]
(D
2)
Men
u (
D2)
Exti
nci
ón (
D2)
Cum
bre
(D
1&
D2--
1N
E)
Mo
da
(D2
--N
E)
Ch
ang
e[?]
(D
1-D
2)
Dim
ensi
on 1
: P
rom
isin
gD
imen
sion 2
: A
ccom
pli
shed
30460
Dim
ensi
on 1
: O
K
Dim
ensi
on 2
: O
K-
Per
fect
Ess
ay:
Guer
ra C
ivil
Esp
aña
(D1
)P
ascu
a post
card
(D
1)
C.A
. (D
2)
Deb
ate
(D1
&2
)E
ssay
(D
2)
Wo
rksh
eet
(D2
--N
E)
Esc
udo (
D1)
Cháv
ez
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
in s
pea
kin
g. N
ot
enough e
vid
ence
to f
ull
y j
udge
oth
eras
pec
ts o
f th
is d
imen
sion.
No
te:
Hav
e u
sed
to
sco
re a
spec
ts n
ot
mar
ked
by
stu
den
t.
Com
p.:
Guer
ra C
ivil
, G
oya
(D1)
Tarj
eta
(D
1)
Tap
e: O
rale
s (D
2)
Com
p.:
Per
sona
adm
iro (
D2)
Can
ción:
Chic
as d
e hoy (
D2)
Pro
yec
tos
Dim
ensi
on 1
: A
ccom
pli
shed
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: pro
ject
s did
n't
hav
e co
ver
shee
ts, but
ther
e is
ple
nty
of
evid
ence
in a
ll f
ield
s.
53
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30816
Dim
ensi
on 1
: O
K
Dim
ensi
on 2
: O
K
Com
p.:
Mar
tí,
Juár
ez,
Cháv
ez (
D2
)S
um
mar
y:
A l
a der
iva
(D2
)W
ork
shee
t: B
olí
var
/Was
h.
(D2
)S
um
mar
y (
D2)
Outl
ine:
Goya/
Pic
asso
(D
2)
Work
shee
t: H
ispan
os
(D1
)S
um
mar
y:
(D1)
Cas
sett
e: C
.A. (D
2)
Dim
ensi
on 1
: D
evel
opin
g
Dim
ensi
on 2
: P
rom
isin
g
Note
: S
tuden
t's p
ort
foli
o n
eeds
more
evid
ence
in d
imen
sion 1
. L
ack o
f pro
duct
s re
sult
in a
n i
nab
ilit
y t
o a
rriv
e at
an a
sses
smen
t oth
er t
han
dev
. D
im 2
pro
vid
es m
ore
evid
ence
and r
esult
s in
ass
essm
. of
pro
mis
ing.
Anci
anos
(D2)
Com
p.:
Mar
tí/C
háv
ez (
D1&
2)
Com
p.:
A l
a der
iva
(D2
)B
olí
var
(D
2)
[?]
(D2
)G
oya
y P
icas
so (
D1
)[?
] H
ispan
ic (
D1&
2)
Mar
tí/J
uár
ez (
D1)
Mia
mi
(did
n't
indic
ate
dim
ensi
on)
Ass
ess.
(did
n't
indic
ated
imen
sion)
Mi
her
man
a (d
idn't
indic
ate
dim
ensi
on)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
30435
Dim
ensi
on1:
OK
Dim
2:
OK
- P
erfe
ct
Pam
phle
t: a
uto
bio
gra
phy (
D1&
2)
Map
: M
exic
o (
D1
&d
2)
Outl
ine:
Popo/I
xtl
a (D
1&
2)
Cas
sett
e: A
la
Der
iva(
D1
&2
--1
NE
)S
um
mar
y:
El
maí
z (D
1&
2--
1N
E)
Let
ter:
Isl
a de
Pas
cua
(D1
&2
)S
um
mar
y:
El
Gre
co (
D1
&2
)P
oem
: B
lasó
n (
D1&
D2)
Pam
phle
t: E
spañ
a (D
1&
D2
)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: V
ery g
ood p
ort
foli
o e
spec
iall
y w
riti
ng, sp
eakin
g a
nd c
om
pre
hen
sion a
nd
read
ing. S
tuden
t does
not
show
evid
ence
of
contr
ibuti
on o
f H
ispan
ic f
igure
s but
chec
ks
the
spac
e n
ever
thel
ess.
N
eed
more
in
form
atio
n.
Juli
a (D
1&
2--
NE
)D
eriv
a (D
1&
2)
Map
a M
exic
o (
D1
&D
2--
2N
E)
Popo (
D1&
2)
Maí
z (D
1&
2--
1N
E)
Car
ta (
D1&
2)
Vam
isa?
(D
1&
2)
Pin
tura
Gre
co (
D1
&2
--2
NE
)B
lasó
n (
D1&
2)
Esp
aña
(D1&
2)
Dim
ensi
on1:
Acc
om
pli
shed
Dim
ensi
on 2
: A
ccom
pli
shed
54
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Ag
reem
ent
Art
ifac
tsC
om
men
ts
30124
Dim
ensi
on1 :
NO
-N
E
Dim
ensi
on 2
:O
K-P
erf
ect
Bio
gra
fía
(D2--
NE
)C
en
a (
D2
--N
E)
Cer
van
tes
(D1&
2)
Itin
erar
io M
exic
o (
D1
&2
)D
ebat
e E
coló
gic
o (
D1
&2
)A
nim
ales
pel
igro
ex
t. (
D2
--N
E)
La
moda
(D2)
Deri
va (
D2
)Is
la d
e P
ascu
a (D
2)
Chic
as d
e hoy (
D2)
Gru
pos
étnic
os
(D1
)D
iari
o (
D2)
Dim
ensi
on 1
: A
ccom
pli
shed
(S
hould
hav
e ex
pla
ined
more
, but
pro
ved
more
than
pro
mis
ing)
Dim
ensi
on 2
: P
rom
isin
g (
close
to a
ccom
pli
shed
but
no q
uit
e th
ere:
adeq
uat
e re
gis
ter
and
flu
ency
)
Post
er (
D1&
2--
no
t en
ou
gh
)P
ost
er (
D1&
2--
no
t en
ou
gh
)P
ost
er (
D2)
Guid
e (D
1&
2)
Post
er (
D1&
2--
no
t en
ou
gh
)C
um
bre
(D
1&
2--
s n
ot
enough)
La
Moda
(D2--
not
eno
ug
h)
A l
a der
iva
(D2)
Post
al (
D2)
Song (
D2)
Res
earc
h (
D1
&2
)D
iari
o (
D2)
Dim
ensi
on 1
: N
ot
enough t
o e
val
uat
e w
hole
dim
ensi
on (
on t
he
firs
t as
pec
t: p
rom
isin
g)
Dim
ensi
on 2
: P
rom
isin
g
55
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30476
Dim
ensi
on 1
: N
O-
NE
Ess
ay (
D2)
Ess
ay (
D2)
Lis
t (D
1&
2)
Invit
acio
nes
(D
2)
Lis
ta C
om
ida
(D1)
Ess
ay (
D2)
Post
er (
D2)
Coll
age
(D2)
Wo
rk-s
heet
(D2
)D
raw
ing (
D1&
2--
NE
)E
xpla
nat
ion-s
ket
ch (
D1
&2
--1
NE
)E
ssay
(D
1&
2)
Tap
e (D
2)
Tap
e (D
2&
2)
Ess
ay (
D1&
2)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
Dim
ensi
on 2
: P
rom
isin
g
Com
p.:
Art
ista
Lo
ca (
D2
)T
ape:
Invit
ació
n b
anq
uet
e (D
2)
Tap
e: D
ebat
e ec
oló
gic
o (
D2
)C
om
p.:
Goya/
Pic
asso
(D
1&
2--
1?)
Str
ip s
tory
: P
opo (
D1&
2--
1?)
Dib
ujo
: A
la
der
iva
(D1
&2
--1
?)C
oll
age:
Auto
bio
gra
fía
(D2)
Nam
e P
roje
ct:
(D2
)C
om
p.:
Bio
gra
fía
(D2)
Lis
ta c
om
ida:
(D
1)
Invit
ació
n (
D2)
Esl
abón (
D1&
2)
Dim
ensi
on 1
: D
evel
opin
gD
imen
sion 2
: P
rom
isin
g
Note
: T
oo m
uch
of
sam
e kin
ds
of
mat
eria
ls. D
idn't
know
how
to c
hoose
bes
t w
ork
s.N
ot
eno
ug
h e
vid
ence
in
so
me
case
s.
56
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30457
Dim
ensi
on 1
:O
K/N
O-N
E
Dim
ensi
on 2
: N
O-
NE
Tap
e (D
2)
Post
er/E
ssay
(D
1&
2)
Car
ta (
D2)
[bla
nco
en d
oc.
] (D
1&
2)
Work
Shee
t (D
2)
Post
er (
D2)
Esc
ud
o (
D2
--N
E)
Ess
ay (
D1&
2)
Invit
acio
nes
(D
2)
Vid
eo (
D1)
Post
er (
D1&
2--
2N
E)
Sk
etch
(D
2--
NE
)
Dim
ensi
on 1
: D
iver
sity
=pro
mis
ing/N
ot
enough e
vid
ence
.
Dim
ensi
on 2
: P
rom
isin
g, but
my s
ense
is
that
wit
h m
ore
evid
ence
it
would
be
acco
mp
lish
ed.
Tap
e (C
.A.)
(D
2)
Pic
ture
seq
. (D
1&
2--
1N
E)
Car
ta (
D2)
I. P
ascu
a (D
2)
C.
de
hoy (
D2)
Coll
age
(D2--
NE
)E
scu
do
(D
2--
NE
)It
iner
ario
de
Via
je (
D2
--N
E)
Lis
ta c
om
idas
(D
2--
NE
)C
ena
de
honor
(D2)
Leo
par
do
(lo
go
) (D
2--
NE
)C
oll
age
(D2--
NE
)V
ideo
(D
1--
NE
)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
Dim
ensi
on 2
: V
ery l
ittl
e ev
iden
ce, ca
n't
judge
over
all
dim
ensi
on.
30394
OK
-Perf
ect
Pla
nes
de
Vuel
o -
-no
in
clu
ye
las
ref
lexio
nes
(D
1)
Wri
ting:
Tar
jeta
Post
al (
D2)
Clo
ze (
D1&
2--
1N
E)
Pam
phle
t: A
uto
bio
gra
fía
(D2)
Pam
phle
t: V
iaje
(D
1)
Wri
ting:
Tole
do (
D2
)T
ape:
La
Sel
va
(D2
)W
riti
ng:
La
Guer
ra (
D1
&D
2)
Over
all
evid
ence
in t
his
port
foli
o i
ndic
ates
stu
den
t has
pro
gre
ssed
to t
he
level
that
show
s his
abil
ity t
o h
ave
gen
eral
insi
ght
and u
nder
stan
din
g a
nd d
ispla
y e
xam
ple
s of
dif
f.as
pec
ts o
f th
e cu
ltu
re.
Tap
e (D
1&
2)
# 1
& T
ape
# 2
(D
1&
2)
Vid
eo (
NE
)-re
ad!
Bio
gra
fía
(D2)
Bro
chure
(D
1&
D2
)R
esum
en (
D2)
# 1
& R
esum
en # 2
(D
2)
Bla
són (
D2--
NE
)P
lan d
e vuel
o (
D1
&D
2)
Dim
ensi
on 1
: P
rom
isin
g f
or
only
2 o
f th
e as
pec
ts;
not
enough e
vid
ence
to j
udge
dim
ensi
on a
s a
whole
, si
nce
contr
ibuti
ons
are
mis
sing.
Dim
ensi
on 2
: P
rom
isin
g (
gre
at d
iffe
rence
s bet
wee
n r
ecep
tive
skil
ls a
nd p
roduct
ive
skil
ls!)
57
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Ag
reem
ent
Art
ifac
tsC
om
men
ts
30396
Bla
són (
D1&
2)
Esp
aña
(D1&
2)
Los
May
as (
D1&
2)
Los
Azt
ecas
y l
os
May
as (
D1
&2
)
Pla
n d
e vuel
o (
D1
&2
)
Goya
& P
icas
so (
D2
)
Com
posi
ción:
Yo s
oy (
D2)
Com
posi
ción:
A l
a D
eriv
a (D
2)
Tap
e: D
eriv
a (D
2)
Pic
asso
& G
oya
(D1
&2
)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
No
te:
To
o m
uch
mat
eria
l.
Cas
sett
e:
A l
a D
eriv
a (D
2)
Cas
sett
e: G
oya
y P
icas
so (
D1&
D2)
Yo s
oy (
D2)
Ess
ay:
Ixtl
a y P
opo (
D1&
D2)
Let
ter:
Quer
ido A
leja
nd
ro (
D1
&D
2)
Vea
las
sel
vas
tro
pic
ales
(D
2)
Pla
n d
e vuel
o:
Los
may
as (
D1&
D2)
Coll
age:
Ess
ay:
Axte
cas,
may
as y
tolt
ecas
(D
1&
D2)
Ess
ay:
Tre
s obra
s d
e lo
s m
ayas
(D
1&
D2
)
Una
Esp
aña:
Much
as c
ult
ura
s (D
1&
D2
)
Bla
són y
cognad
os
del
bla
són (
D1&
D2)
Dim
ensi
on 1
: A
ccom
pli
shed
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: Y
o s
oy -
- I
could
n’t
judge
the
pro
duct
bec
ause
stu
den
t did
n’t
incl
ude
soru
ce.
T
res
obra
s de
los
may
as -
- I
could
n’t
fin
d t
his
docu
men
t in
the
port
foli
o.
G
ener
al:
Stu
den
t in
cluded
man
y d
ocu
men
ts a
nd p
roje
cts
to s
how
evid
ence
of
pro
gre
ss i
n D
imen
sion 1
. It
show
s th
at s
he
work
ed o
n t
his
dim
ensi
on t
hro
ughout
the
cou
rse.
58
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30390
Dim
ensi
on 1
: N
O
Dim
ensi
on 2
: O
K
Vid
eo (
D2)
Ess
ay (
D1&
2)
Ess
ay (
D1&
2)
Work
Shee
t (D
2)
Ess
ay (
D2)
Vid
eo (
D2)
3 T
apes
(D
1&
2)
Ess
ay, P
amph. (D
1&
2)
Dim
ensi
on
1:
Ad
van
ced
Dim
ensi
on
2:
Ad
van
ced
No
te:
cov
er s
hee
ts w
ere
exce
llen
t so
urc
e o
f w
riti
ng
an
d c
om
pre
hen
sio
n
Esp
aña
May
asG
uer
nic
a/3 M
ayo
Méx
ico
Bla
són
Vid
eo:
Me
llam
o S
onia
Vid
eo:
A l
a D
eriv
a
Dim
ensi
on 1
: P
rom
isin
gN
ote
: A
lthough t
he
studen
t did
n't
hav
e m
any a
rtif
acts
, I
could
judge
that
he/
she
was
at
leas
t P
rom
isin
g i
n d
emonst
rati
ng k
now
ledge
of
his
pan
ic c
ult
ure
s.N
ote
: S
tuden
t had
one
pro
duct
to s
how
evid
ence
of
the
div
ersi
ty o
f his
p. cu
lture
s.D
imen
sion 2
: A
ccom
pli
shed
Note
: B
ased
on S
/W;
how
ever
, sh
e did
n't
show
evid
ence
of
read
ing c
om
pre
hen
sion
31948
Dim
ensi
on 1
:N
O-N
E
Dim
ensi
on 2
:O
K/N
O-N
E
Dis
k:
Judge
(D1&
2)
Work
shee
t: P
[?]
(D1
&2
)C
om
p.:
Auto
bio
gra
fía
(D2
)P
royec
to (
D2)
Pro
yec
to (
D1&
2)
Tap
e (l
is.)
(D
1&
2)
Tap
e (s
pea
k.)
(D
2)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g
Note
: S
he
seem
s to
be
bet
ter
than
this
evid
ence
(bec
ause
of
the
wri
ting).
WorK
Shee
t (D
1&
2)
Ess
ay (
D1&
2)
Ess
ay d
ibujo
s (D
2)
Ess
ay (
D2)
Work
Shee
ts (
D1&
2)
Lis
t &
Par
agra
ph (
D2
)T
ape
(D2)
Tap
e (D
2)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
.
Dim
ensi
on 2
: D
eriv
ing m
eanin
g=
no e
vid
ence
/Expre
ss a
ccom
pli
shed
Note
: I
bel
ieve
she
may
be
bet
ter
than
pro
mis
ing b
ut
I don't
hav
e en
ough e
vid
ence
.
59
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Ag
reem
ent
Art
ifac
tsC
om
men
ts
30843
2 N
ewsp
aper
, A
rtic
les
(D1
&2
)
Com
p.:
C
háv
ez,
Mar
tí a
nd
Juár
ez (
D1
&2
)
A l
a D
eriv
a (D
2)
Auto
bio
gra
fía
(D2)
Vid
eo:
Ixtl
a/P
opo (
D2)
Vid
eo:
A l
a D
eriv
a (D
2)
Tap
e: T
est
(D2)
Fly
er:
Gra
cia[
?] (
D1
&2
)
Dim
ensi
on 1
: D
evel
opin
g
Dim
ensi
on 2
: D
evel
opin
g
Let
ter
(D2)
Ess
ay:
A l
a der
iva
(D2
)E
ssay
: M
artí
, J
uár
ez y
Cháv
ez(D
1&
2)
Exam
(D
2)
Note
s fr
om
pre
senta
tio
ns(
D1
&2
)L
as A
rtes
de
Mad
rid
(D1
&2
)A
rroz
con P
oll
o(D
1&
2)
[?]
Vid
eo (
D2)
Cas
sett
e (D
2)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g
Note
: A
lthough s
tuden
ts s
eem
s to
be
bet
ter
wit
h r
eadin
g a
nd l
iste
nin
g, he
only
incl
uded
1 s
ample
of
each
to p
rove
his
pro
gre
ss w
hil
e he
incl
uded
man
y p
roduct
s to
eval
uat
e his
wri
ting a
nd s
pea
kin
g s
kil
ls
30342
Dim
ensi
on 1
: O
K
Dim
ensi
on 2
: O
K
Post
er:
Bla
són (
D1&
2)
Post
er:
[?]
(D2)
[?]
Bla
són
Was
h/B
olí
var
(D
1&
2)
Car
ta (
dim
ensi
on?)
Isab
el/M
alic
he
(D1
&2
)E
spañ
ol
(D1&
2)
Car
ta (
dim
ensi
on?)
Caj
as (
dim
ensi
on?)
Mar
tí/C
háv
ez (
D1
&2
)
Dim
ensi
on 1
: A
com
pli
shed
Dim
ensi
on 2
: A
ccom
pli
shed
(All
art
if.
use
d a
s m
easu
rin
g p
erfo
rman
ce i
n b
oth
Dim
ensi
on)
Wri
ting:
Mar
tíC
háv
ezT
ape:
Mal
inch
e/Is
abel
Wri
ting:
Esp
aña
Wri
ting:
Was
h./
Bolí
var
Wri
ting:
Caj
as d
e C
arto
nP
ost
er:
Bla
són
Post
er:
Quié
n s
oy?
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g
Note
: S
tuden
t's p
ort
foli
o h
as e
vid
ence
for
all
aspec
ts o
f ea
ch d
imen
sion. V
ery s
trong
wri
ting a
nd s
pea
kin
g y
et o
ver
all
impre
ssio
n i
s of
a ver
y p
rom
isin
g p
ort
foli
o.
60
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Ag
reem
ent
Art
ifac
tsC
om
men
ts
30814
OK
-Perf
ect
Los
Paí
ses
(D1)
Fusi
on d
e C
ult
o (
D1--
NE
)T
ape:
Ju
árez
/Ch
ávez
(D
1&
D2
)L
anta
ro (
D1
&D
2)
Un
? p
oet
a (D
2)
Isab
el/L
a M
alic
he
(D2
)T
ape:
Mi
her
man
a (D
2)
Fel
iz c
um
ple
años
Mia
mi
(D2)
Culm
. A
ss.
(D2)
Dim
ensi
on 1
: P
rom
isin
g. S
tuden
t did
n't
incl
ude
enough p
roduct
s to
rea
lly s
ay t
hat
she
was
acc
om
pli
shed
on d
imen
sion 1
.
Dim
ensi
on 2
: A
ccom
pli
shed
.
Tap
e &
Vid
eo:
Sis
ter
(D2
)T
ape
& V
ideo
: P
op
o (
D2
)N
ote
s: L
os
paí
ses
(D1
)C
om
posi
tion:
J
uár
ez/C
háv
ez/M
artí
(D
1&
D2
)C
oll
age:
Lan
taro
Work
Shee
t: I
mm
igra
nts
(D
2)
Work
Shee
t: I
sabel
/ L
a M
alic
he
(
D1&
D2 -
did
n't
incl
ude
pag
e
w
hic
h r
equ
ires
in
fere
nce
s)C
om
posi
tion: F
eliz
Cum
ple
. N
EL
ette
r of
imm
igra
nt
(D2
)T
ape:
Med
io A
mb
ien
te (
D2
)T
ape:
Cum
bre
(D
2)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: put
a co
ver
on e
ver
yth
ing s
ubm
itte
d.
31296
Dim
ensi
on 1
:N
O/N
E
Dim
ensi
on 2
: O
K
Car
tel:
Logo d
e P
aca
(D2
)C
arta
(D
2)
Tap
e: E
spañ
a (D
1&
2)
Tap
e: M
adre
Ter
esa
(D1
&2
)L
ette
r: I
sla
(D1&
2--
1N
E)
Work
Shee
ts:
Coló
n, C
ort
és, Is
abel
, D
. M
arin
a (D
1)
Wri
ting:
Auto
bio
gra
fía
(D2)
Wri
ting:
Food (
D1&
2)
Sum
mar
y:
(D1&
2--
1N
E)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: C
hoic
e of
dim
ensi
on d
idn't
alw
ays
show
evid
ence
for
mak
ing a
ppro
pri
ate
judgem
ent.
S
tuden
t's w
riti
ng a
nd s
pea
kin
g d
emonst
rate
d w
ell
arti
cula
ted m
essa
ges
.
Post
er (
D2)
Car
ta (
D2)
Car
ta/l
ectu
ra (
D1&
2)
Work
shee
t (D
1&
2)
Tap
e (D
2)
Bio
gra
fía
(D2)
Receta
(D
2)
Res
earc
h (
D1
&2
)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
Dim
ensi
on 2
: P
rom
isin
g
61
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Ag
reem
ent
Art
ifac
tsC
om
men
ts
31948
Dim
ensi
on 1
:N
O-N
E
Dim
ensi
on 2
:O
K/N
O-N
E
Dis
k:
Judge
(D1&
2)
Work
shee
t: P
[?]
(D1
&2
)C
om
p.:
Auto
bio
gra
fía
(D2
)P
royec
to (
D2)
Pro
yec
to (
D1&
2)
Tap
e (l
is.)
(D
1&
2)
Tap
e (s
pea
k.)
(D
2)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g
Note
: S
he
seem
s to
be
bet
ter
than
this
evid
ence
(bec
ause
of
the
wri
ting).
WorK
Shee
t (D
1&
2)
Ess
ay (
D1&
2)
Ess
ay d
ibujo
s (D
2)
Ess
ay (
D2)
Work
Shee
ts (
D1&
2)
Lis
t &
Par
agra
ph (
D2
)T
ape
(D2)
Tap
e (D
2)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
.
Dim
ensi
on 2
: D
eriv
ing m
eanin
g=
no e
vid
ence
/Expre
ss a
ccom
pli
shed
Note
: I
bel
ieve
she
may
be
bet
ter
than
pro
mis
ing b
ut
I don't
hav
e en
ough e
vid
ence
.
30457
Dim
ensi
on 1
:O
K/N
O-N
E
Dim
ensi
on 2
: N
O-
NE
Tap
e (D
2)
Post
er/E
ssay
(D
1&
2)
Car
ta (
D2)
[bla
nco
en d
oc.
] (D
1&
2)
Work
Shee
t (D
2)
Post
er (
D2)
Esc
ud
o (
D2
--N
E)
Ess
ay (
D1&
2)
Invit
acio
nes
(D
2)
Vid
eo (
D1)
Post
er (
D1&
2--
2N
E)
Sk
etch
(D
2--
NE
)
Dim
ensi
on 1
: D
iver
sity
=pro
mis
ing/N
ot
enough e
vid
ence
.
Dim
ensi
on 2
: P
rom
isin
g, but
my s
ense
is
that
wit
h m
ore
evid
ence
it
would
be
acco
mp
lish
ed.
Tap
e (C
.A.)
(D
2)
Pic
ture
seq
. (D
1&
2--
1N
E)
Car
ta (
D2)
I. P
ascu
a (D
2)
C.
de
hoy (
D2)
Coll
age
(D2--
NE
)E
scu
do
(D
2--
NE
)It
iner
ario
de
Via
je (
D2
--N
E)
Lis
ta c
om
idas
(D
2--
NE
)C
ena
de
honor
(D2
)L
eop
ard
o (
log
o)
(D2
--N
E)
Coll
age
(D2--
NE
)V
ideo
(D
1--
NE
)
Dim
ensi
on 1
: N
ot
enough e
vid
ence
Dim
ensi
on 2
: V
ery l
ittl
e ev
iden
ce, ca
n't
judge
over
all
dim
ensi
on.
62
Ap
pen
dix
5 -
Co
nti
nu
atio
nP
OR
TF
OL
IOS
RA
TE
D B
Y T
WO
RE
AD
ER
S W
OR
KIN
G T
OG
ET
HE
R
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30393
Pla
n d
e vuel
o:
May
as (
D1&
2)
Au
tob
iog
rafí
a (D
2--
NE
)
Foll
eto (
D1)
Bla
són:
Poem
a (D
1&
2)
Act
ivid
ad:
Pai
nti
ngs
(D1&
2)
Popo/I
xtl
a (D
1)
Sto
ry:
A l
a der
iva
(D2
)
Let
ter:
Isl
a de
Pas
cua
(D2
)
Poem
(D
1&
2)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g
No
te:
Rea
ks
of
pro
mis
e m
ore
th
an [
?]
Note
: P
eaks
are
more
num
erous
in t
he
area
of
pro
mis
ing. O
ver
all
impre
ssio
n i
s of
apro
mis
ing p
ort
foli
o.
30357
Guid
e: E
spañ
a(D
1)
Wr.
Rep
ort
: M
éxic
o (
D1
&2
)
Cin
ta:
Tole
do (
D1&
2)
Sum
mar
y:I
xtl
a/P
opo (
D2)
Rep
ort
: M
artí
/Juár
ez (
D2
)
Inte
rvie
w:
Pro
feso
r L
. (D
2)
Rea
din
g:
Caj
as d
e C
arto
n (
D2)
Tap
e: L
a M
alin
che
(D2
)
Dim
ensi
on 1
: A
ccom
pli
shed
Dim
ensi
on 2
: A
ccopli
shed
Note
: O
ver
all
prd
uct
s dem
onst
rate
that
stu
den
t is
acco
mpli
shed
in b
oth
dim
ensi
ons.
No
te:
Ov
eral
l p
rdo
cut
dem
on
stra
tes
acco
mp
lish
ed p
rod
uct
ion
; h
ow
ever
, sp
eak
ing
[??
]
30131
Sum
mar
y:
Popo/I
xtl
a (D
1&
2--
1N
E)
Sum
mar
y:
Isla
de
Pas
cua
(D2
)
Car
ta (
D2)
Dia
rio (
D2)
Pro
yec
to:
Bio
gra
fía
(D2
)
Pro
yec
to:
Com
p. (D
1&
2)
Monum
ento
(D
2)
Tap
e: C
.A.
(D1&
2--
1N
E)
Deb
ate
(D1
&2
--1
NE
)
Pro
yec
to:
Anim
al (
D2
)
Deb
ate
(D
2)
Cum
bre
(D
2)
Dim
ensi
on 1
: B
egin
nin
g
Dim
ensi
on 2
: A
ccom
pli
shed
63
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30149
Los
May
as (
D1&
2)
Popo e
Ixtl
a (D
2)
Méx
ico y
Esp
aña
(D1
)
Anse
l A
dam
s (D
2)
Nues
tros
Hér
oes
(D
2)
C.A
. (D
2)
Auto
bio
gra
fía
(D2)
Cen
a de
Honor
(D2
)
Anim
ales
en p
elig
ro (
D2
)
La
mode
(D2)
Dim
ensi
on 1
: P
rom
isin
g, but
I’m
not
so s
ure
sin
ce t
her
e w
asn’t
enough e
vid
ence
to
jud
ge
two
asp
ects
.
Dim
ensi
on 2
: P
rom
isin
g, bas
ed t
wo s
ample
s.
Note
: In
gen
eral
, I
thin
k t
his
port
foli
o n
eeds
more
pro
duct
s to
hav
e en
ough e
vid
ence
to
judge
it.
31942
Tap
e (D
2)
Pár
rafo
: P
oet
a (D
2)
Ess
ay (
D2)
Spea
kin
g (
D2)
Rec
eta
(D1&
2)
Note
s fr
om
hea
din
g (
D1
&2
)
Ess
ay (
D1&
2)
Sp
eech
[?](
D2
)
Ess
ay (
D1&
2)
No
te
(D2
--N
E)
Dim
ensi
on 1
: P
rom
isin
g
Dim
ensi
on 2
: P
rom
isin
g e
xpre
sion o
nly
. N
ot
enough e
vid
ence
to j
udge
rece
pti
ve
skil
ls.
30819
Cla
ss N
ote
s (D
1&
2)
Art
ículo
(D
2)
Isab
el/L
a M
alin
che
(D1
&2
)
Work
Shee
t (D
1&
2)
Mad
rid y
las
art
es (
D2
)
Tap
e: A
la
der
iva
(D2
)
Com
p.:
Cháv
ez (
D2
)
Tap
e: P
opo (
D2)
Dim
ensi
on 1
: A
ccom
pli
shed
Dim
ensi
on 2
: A
ccom
pli
shed
No
te:
Po
orl
y o
rgan
ized
. V
ery
few
ev
iden
ce o
f in
fere
nci
ng
[?]
.
No
te:
So
me
con
nec
tio
ns
wer
e m
ade
bu
t w
ere
no
t ch
eck
ed a
s ev
iden
ce.
64
Ap
pen
dix
5 -
Co
nti
nu
atio
n
Stu
den
t #
Agre
emen
tA
rtif
acts
Com
men
ts
30470
Post
er (
D2)
Ess
ay (
D2)
Ess
ay (
D1&
2)
Ess
ay (
D1&
2--
2N
E)
Lis
ta C
om
ida
(D1
&2
--2
NE
)
Invit
acio
nes
(D
2)
Tap
e (D
2)
Tap
e: C
olu
mbia
(D
1&
2)
Outl
ine
(D1&
2--
2N
E)
Foto
(D
1&
2--
NE
)
Sket
ch (
D1&
2--
NE
)
Ess
ay (
D2)
Car
ta (
D2)
Wro
k S
hee
t (D
2)
[Rea
der
s d
idn’t
in
clu
de
eval
uat
ion
fo
r ea
ch d
imen
sio
n]
31274
Ess
ay (
D1&
2)
No
ne
(D2
--N
E)
Work
Shee
t (D
1&
2)
Cover
&E
ssay
(D
1&
2)
Work
Shee
t (D
1&
2)
Ess
ay (
D1&
2)
Ess
ay
(D2)
Ess
ay
(D2)
Work
Shee
t (D
1)
Ess
ay (
D1&
2)
Ess
ay (
D1&
2)
Work
Shee
t &
Cov
er S
hee
t (D
1&
2)
Work
Shee
t &
Cov
er S
hee
t (D
1&
2)
Dim
ensi
on
1:
Ad
van
ced
Dim
ensi
on 2
: A
ccom
pli
shed
Note
: co
nsi
der
edas
pec
ts n
ot
note
d b
y s
tuden
t
65
App
endi
x 5
- C
ontin
uatio
nPO
RT
FOL
IOS
RA
TE
D B
Y O
NE
RE
AD
ER
Stud
ent #
Agr
eem
ent
Art
ifac
tsC
omm
ents
3049
8T
ape
(D2)
Tap
e (D
2)
Tap
e (D
2)
Wor
k Sh
eet (
D1)
Ess
ay (
D2)
Dra
win
g (D
2)
Post
er (
D1&
2)
Ess
ay (
D2)
Lis
t (D
1&2)
Ess
ay (
D1&
2)
Soup
(D
2)
Dim
ensi
on 1
: Not
eno
ugh
evid
ence
Dim
ensi
on 2
: Pro
mis
ing
66
Appendix 5 - Continuation
Correspondence of Student # and Portfolio #
Student # Portfolio #
31938 1
30129 2
30831 3
31929 4
31296 5
30351 6
30358 7
30144 8
30460 9
30816 10
30435 11
30124 12
30476 13
30457 14
30394 15
30396 16
30390 17
31948 18
30843 19
30342 20
30814 21
30131 22
30149 23
30470 24
30498 25
31274 26
31942 27
30819 28
30393 29
30357 30
Phase II Appendix
The Participants: Roles and Responsibilities
69
Appendix A
Preparation for Portfolio Assessment Scoring SessionsBoth the Spanish and English Pacesetter courses use portfolios as a mea-sure of student achievement. In each subject, portfolio collections of regu-lar classroom work are developed by students throughout the school yearas a way to demonstrate their achievement in relation to the coursedimensions. In both Pacesetter English and Spanish, the scoring guidesused to evaluate the portfolios at the end of the semester or year are usedthroughout the course to clarify course expectations and serve as a guideto help students build strong collections of work.
Portfolios provide a way of looking at student achievement that is quitedifferent from the other types of Pacesetter assessments. Whereas com-mon tasks and even the culminating assessment look at student achieve-ment on a particular task at a given time, the portfolio provides a view of astudent’s achievement across a body of various kinds of class work.Teachers who have participated as Readers at portfolio scorings oftencomment that looking at work from other Pacesetter classes providesmodels of instructional activities and strategies they can adapt for theirown teaching.
Pacesetter English
As with the culminating assessment, portfolio assessment focuses on eval-uating student performance in the two course dimensions of MakingMeaning from Texts and Creating and Presenting Texts. Because of thedesign of the portfolio scoring guides, it is possible to gather performance-level information about four aspects of each of the two course dimensionsas well as the two dimensions themselves. Teachers - Readers indicate thatlooking at student performance on various aspects of the course dimen-sions is especially helpful in planning the course and reflecting on theirteaching.
Pacesetter Spanish
Like the English portfolio, the Spanish portfolio also focuses on evaluatingstudent performance in two course dimensions: DemonstratingKnowledge of Hispanic Cultures and Using Spanish to CommunicateEffectively. Portfolio scoring guides for Pacesetter Spanish also make itpossible to evaluate student performance on several aspects of thesedimensions through a single reading of a portfolio.
Scoring portfolios can provide information about student achievement andprovide ideas for staff development. Because of the complex nature ofportfolios and the time it takes to score them (about 25 minutes per read-ing), different purposes can make seemingly opposing demands on thetime and activities of a portfolio reading.
The Relationship BetweenScoring Purpose and
Scoring Design
Portfolio AssessmentOverview
70
Preparation for Portfolio Assessment Scoring Sessions
In order to monitor the technical quality of scores and to determine theirreliability, either a sample of or all of the portfolios need to be read twice.Having extended discussions about the scoring criteria and how they areevidenced in student work supports staff development purposes.
Decisions need to be made about the use of time and how that relates tothe purpose(s) of the reading. Should fewer portfolios be read so that timecan be allocated for extended discussions? Should a larger sample of port-folios be read and the discussions be limited to training issues? Shouldthere be a balance in the use of time to reflect both purposes? Are thereother local purposes that might have an impact on the portfolio scoringdesign?
The following are some examples of the connection between scoring pur-pose and design:
• If the purpose of scoring portfolios is to attain scores to monitor stu-dent achievement, then a representative sample of portfolios needs tobe read. Some or all of the sample needs to be read twice.
• In some cases the purpose of a portfolio reading might be to providefeedback to teachers about how they are applying the scoring criteriain their individual classrooms. In this design, teachers might be askedto prescore a representative sample of portfolios from theirPacesetter classes and submit those portfolios for rescoring at thedistrictwide reading. Scores from the centralized reading would bereturned to teachers.
• If the primary purpose of the reading is to provide staff development,maximizing the time for teacher discussion about student perfor-mances in relation to the scoring criteria is important. Scoring asmaller representative sample of portfolios would be appropriate.
Depending on the purpose for the portfolio scoring, Readers might beasked to score each portfolio for the course dimensions and the aspects ofthe dimensions through a single reading of a portfolio. Some believe thatthis design models how teachers evaluate student performances in theirclassrooms where teachers consider several aspects of performance at thesame time.
It is also reasonable to organize a portfolio scoring so that Readers scoreonly one dimension and its aspects at a time. If a district is especially con-cerned about a particular dimension of student achievement, this designallows Readers to focus more attention on one dimension of performance.If the intent is to gain scores for all dimensions of performance within asubject area, this design requires more time because each portfolio mustbe read separately for each dimension.
Your district may have other purposes for holding a portfolio reading thatmay necessitate other designs. Determining what information is needed bywhom and for what purposes should guide your district in designing aportfolio scoring to meet your needs. No matter the purpose of the read-ing, coming together to examine student work in portfolios is a powerfulway to help Pacesetter teachers clarify and internalize the Pacesetter con-tent and performance standards.
Preparation for Portfolio Assessment Scoring Sessions
71
For more information about building and assessing portfolios, please referto the introduction in the Teachers’ Guides for Pacesetter English andSpanish.
Scoring Coordinator
As with the culminating assessment, a portfolio scoring needs a coordina-tor within the district who can champion the decision making related tothe scoring design, take responsibility for planning the scoring, involveother personnel as needed, and make necessary arrangements. In somedistricts, one person may serve as both the Scoring Coordinator and ChiefReader.
Chief Reader(s)
Because portfolio assessment is a relatively new practice there are fewexperienced Readers. An experienced Pacesetter teacher or district admin-istrator who knows the Pacesetter course well, understands the coursecontent and performance standards fully, and who can talk about how tofind evidence of the standards in student performances, can make a fineChief Reader. Responsibilities of the Chief Reader include selecting bench-mark portfolios to use during the training, conducting the training, andfacilitating discussions about evidence of achievement in student work.
Because training for portfolio scoring requires Chief Readers to know thecourse dimensions, scoring criteria, and benchmark portfolios well, somedistricts, or Chief Readers, may feel most comfortable with more than oneChief Reader. Having multiple Chief Readers may be especially helpful in dis-tricts that are scoring portfolios for the first time, since it allows ChiefReaders to combine their expertise and knowledge. Chief Readers may bemost comfortable focusing on only one course dimension and its aspects. Insome cases, the design of a portfolio scoring may require more than oneChief Reader, especially if Readers will be scoring each dimension separately.
Table Leaders
Whether or not Table Leaders are needed at a portfolio reading is deter-mined by the size and purpose of the reading. Because of the time it takesto read portfolios, Table Leaders do not generally monitor the scoring ofReaders at their table in the same way they might at a culminating assess-ment reading. Instead, their primary responsibilities are to assist with thetraining and to answer questions that arise during the reading. Each 8 to10 readers will need a Table Leader (or Chief Reader) they can turn to withquestions.
The purpose of the reading will determine the number and role of TableLeaders.
• If gathering student achievement information is the primary purpose,Table Leaders might be expected to turn their attention first to read-ing portfolios and second to answering questions, because reading alarger sample of portfolios is necessary. They might be used as anadditional Reader for selected portfolios or as a discussion facilitatorto help Readers agree on a score.
• If staff development is the focus, Table Leaders might lead additionaltable -level discussions about particular issues, facilitate discussions
The Roles andResponsibilities of the
Participants
Preparation for Portfolio Assessment Scoring Sessions
72
about portfolios between scorers with differing scores, bring ques-tions and concerns to the attention of the Chief Reader, and onlyscore portfolios as time permits.
Readers
Readers are usually current or future Pacesetter teachers or administra-tors familiar with the Pacesetter course. In some cases, teachers who donot teach Pacesetter, but who are interested in portfolio assessment orstandards-based assessment, might be interested in the experience ofscoring Pacesetter portfolios as a professional development opportunity.Some districts plan to involve local teacher training institute staff or com-munity members in readings. As with the culminating assessment, com-pensation and other incentives or conditions are governed by local condi-tions.
Aides
Although Table Leaders and Chief Readers can, if necessary, distribute,collect, and organize portfolios and scoring guide forms, Aides can be atremendous help and allow more time for Table Leaders and Chief Readersto answer scorer questions and lead discussions. Generally, one Aide isneeded for each 12 Readers.
Number-Person
If scores from the portfolio reading are to be summarized at the districtlevel, it may be helpful to involve a staff member who can set up and main-tain a data base for portfolio scores and create score reports, includingscore distributions, as needed. There is no calculation needed to deter-mine a proficiency score for portfolios because the “raw” score is the pro-ficiency level as stated in the portfolio scoring guide for that subject.
73
Preparation for Portfolio Assessment Scoring Sessions
Sample Size
As with other decisions, the purpose of the reading is important in deter-mining the number of portfolios that can be read. If more time is used fortraining and discussion, a smaller sample of portfolios can be read. Thefollowing calculation can help determine the size of the sample that can beread by the number of available Readers in the allotted time.
Worksheet for Determining the Sample Size of Portfolios
Sample Selection
If your district determines that not all of the Pacesetter portfolios can bescored at a centralized reading, then a representative sample needs to beidentified for scoring. If a sample rather than all of the portfolios is beingread, the sample should be as random and representative as possible.Some possible scenarios for sampling are listed below.
• In many districts, selecting the sample may fall to the Pacesetterteachers themselves. If this is the case, teachers should be givenguidelines about how to select a sample that is representative andbalanced for gender or other demographic features.
• In some districts, a district-level staff member may identify a randomsample of students from whom portfolios would be gathered and read.
Narrative Explanation Algorithm Calculation
A. Total hours available Days x 5.0 =A= Number of days for scoring x 5.0 hours(total hours a day minus lunch and breaks)
B. Scoring hours available A - 6 = B= A minus 6 hours for training(training time may differ by district)
C. Number of Readers available Readers = C
D. Total possible readings B x C x 2.5=D= Multiply B by C by 2.5 (portfolios read per hour)
E. Number of portfolios Portfolios to to be scored twice be double-
scored = E
F. Number of portfolios that can be D - E = Fread including those double - scored= D minus E
G. Number of classes Classes = G
H. Number of portfolios that can be F ÷ G = Hread from each class= F divided by G
Identifying the PortfolioSample
74
Preparation for Portfolio Assessment Scoring Sessions
• In other districts, teachers might send all of their portfolios to a cen-tral location where district-level staff select a random sample, balanc-ing for gender or other demographic features.
If a large pool of scorers is available or if a decision has been made toscore all of the portfolios, then the calculation below can be used to deter-mine the number of Readers needed to score the portfolios.
Worksheet for Calculating the Number of Scorers Needed
Because portfolios are built throughout the school year, the timing of aportfolio reading is dependent upon the school calendar and teacher avail-ability. Holding a scoring a few weeks before the end of the school yearallows teachers to use the training they receive and any portfolio scoresthat are available to help them determine student grades. However, dis-tricts may choose to have teachers collect a representative sample of port-folios for use at a summer portfolio reading.
As with the culminating assessment, a comfortable, well-lit, quiet locationwith reading tables is needed for a portfolio scoring. Generally, two portfo-lio Readers can comfortably read at a cafeteria table. The Chief Reader(s)is likely to need flip charts, tape, an overhead projector, and marking pens.Depending on the scoring design, number of portfolios, and security of thescoring room, a preparation room for organizing and storing portfoliosmay also be necessary.
Determining theNumber of Scorers
Facilities: Planning aSpace for the Scoring
Session(s)
Scheduling ofPortfolio Scorings
Narrative Explanation Algorithm Calculation
A. Total number of portfolios Portfolios = Ato be scored
B. Number of portfolios that will Double be double -scored Readings = B(times each portfolio will be read)
C. Number of total readings A + B = C= Add A to B
D. Number of scorer hours needed C ÷ 2.5 = D= C divided by 2.5 (portfolios per hour)
E. Total hours available Days x 5.0 = E= Number of days for scoring x 5 hours(total hours a day minus lunch and breaks)
F. Scoring hours available E - 6 = F= E minus 6 hours for training(training time may differ by district)
G. Number of scorers needed D ÷ F = G= D divided by F
Preparation for Portfolio Assessment Scoring Sessions
75
Coding portfolios can ease the collection of scores as well as the return ofportfolios. Your district may already have a system of student numbers orsome other unique code that may prove useful in portfolio identification. Ifnot, you may want to create a simple coding system and enlist teachersand students to help with coding portfolios prior to the reading.
If a portfolio code is used, a unique number for each student is needed. Acode that includes a number for the teacher, class period, and studentmay be the easiest. Precoded labels that students affix to their portfoliosprior to the reading are helpful. With the portfolios submitted for the read-ing, teachers should include a list of students and portfolio code numbers.
In order to prepare portfolios for scoring, some districts may decide tohave portfolios sent to a central location prior to the reading. Organizingthe portfolios might include selecting a random sample to be read. It mayinclude checking or affixing portfolio codes or creating a data base for col-lecting scores. It might also include creating batches of portfolios fromvarious classes to be read at particular tables. In some cases, it may bepossible to have Aides organize portfolios on the morning of the reading, ifthey are not involved in the training.
The scoring guide forms (pages 53 - 60) are meant to be copied and used atthe reading. Readers mark their scores directly on these scoring guides. IfAides are available before the reading, they can precode the forms andplace them in the portfolios. If Aide time is more limited, on the day of thereading place copies of the scoring guides on the tables for Readers. Fororganizational purposes, it is easier to copy each dimension of a subjectarea on a different color paper.
Assigning each Reader a unique code that can be used as an identifier onthe scoring guide forms may make score and reliability analyses easier.Determining whether to give Reader codes depends on local circum-stances.
Each district holding a portfolio scoring will need to select benchmarkportfolios to use during the training. Because portfolios are complicateddocuments and require extended time to read and discuss, training is usu-ally conducted using a small number (three to four) of benchmark portfo-lios. As with the culminating assessment, the Chief Reader(s) should workwith two to three other Pacesetter teachers to select benchmark portfolios.
Benchmark portfolios should represent different classrooms and include avariety of types of performances. Suggested selection criteria for bench-marks include:
• one portfolio that demonstrates strong, but not necessarily exem-plary, performance;
• another that demonstrates uneven performance across dimensions oraspects; and
• one or two other portfolios that are likely to bring about discussion onimportant issues such as evaluating English Language Learner perfor-mances, group versus individual performances, or other local concerns.
In all cases, benchmark portfolios must include work that can be easilyread and copied.
Preparing AssessmentPortfolios for Scoring
Choosing BenchmarkPortfolios to Guide
the Training
Preparation for Portfolio Assessment Scoring Sessions
76
Making copies of benchmark portfolios can be a time-consuming processthat includes:
• making a “master copy” of each benchmark portfolio (may take up toan hour to make the first copy);
• blocking out student, teacher, and school names as well as other per-sonal information that would identify the student;
• tracing writing and other marks that are too light to read;
• numbering each page in the “master copy”;
• adding a title page (for example: Benchmark A) to the “master copy”; and
• making multiple copies for use at the training.
While some Readers find it difficult to read a portfolio with another per-son, it is not uncommon to have Readers share a copy of benchmark port-folios during training. Reading a portfolio in pairs tends to promote discus-sion by the pair about how specific evidence is viewed and the judgmentsthat are being made. These discussions are important in helping Readerscome to a common understanding of what constitutes evidence of thedimensions and how to apply the scoring criteria.
At portfolio readings where Readers are scoring all dimensions of perfor-mance at the same time, most of the training is conducted by the ChiefReader. Table Leaders work with Readers at their table to help them inter-nalize the scoring criteria and use the scoring guide forms correctly. TableLeaders answer questions and refer problematic portfolios to the ChiefReader, as appropriate.
If your design allows, Table Leaders can read and score portfolios as theother Readers do. Because it takes an average of 25 minutes to read andscore a portfolio, rereading scored portfolios is not practical unless it ispart of a double -scoring design or unless a Table Leader or Chief Readerhas concerns about a particular Reader.
Prior to the reading, Table Leaders should be given copies of the bench-mark portfolios so they can familiarize themselves with the evidence. Anynotes that might have been created by the Chief Reader(s) about theassigned score for each benchmark and the evidence that supports thescoring judgment will be helpful to Table Leaders as they answer Readerquestions. It is also helpful for Table Leaders to meet with the ChiefReader(s) prior to the reading to discuss the benchmarks and possiblequestions and issues that might need to be addressed at the reading.
Portfolio-Reading Assumptions
A centralized portfolio reading, as described in this Handbook, is based onseveral underlying assumptions, including:
• scorers need to come to a commonunderstanding of the scoring criteria;
• the criteria in the scoring guides need to be applied consistently;
• scorers need to focus on the evidence of achievement in the work;
Preparing TableLeaders
Training the Readers
Preparation for Portfolio Assessment Scoring Sessions
77
• holistic judgments are based on the quality not the quantity of evi-dence; and
• scores reflect holistic judgments across the body of work in a portfolio.
Suggested Strategies for Reading Portfolios
Because portfolios often contain a large and varied collection of studentwork, they can seem unwieldy to read at first. Most Readers will have littleor no experience looking across a body of work to make judgments. Someof the following suggestions might be useful for Readers.
• Skim the entire portfolio to see the types of performances that areincluded.
• If multiple drafts are included, read the “final” draft closely, then skimthe other drafts.
• If several performances reflect the same type of work, such as a readinglog or several poems, read some selections closely and skim the others.
• Keep notes about evidence as the portfolio is read (sample evidencecharts for English and Spanish are included at the end of this docu-ment). Some Readers prefer to use “yellow stickies” to mark particu-lar pages as they read.
• Ignore teacher grades and comments.
Assigning Scores
Often the quality of performances varies within a portfolio, making it diffi-cult for Readers to assign a score. Readers are generally asked to assign ascore that reflects the preponderance of the evidence.
Sample Training Design
This section describes training for a portfolio reading in which eachReader is scoring all dimensions and aspects of performance within a sub-ject area from a single reading of a portfolio. A training process reflectingthis design is summarized in the following chart. A narrative explanationof the activities follows the chart. Approximate times for each activityhave been included for planning purposes.
78
Sample Training Activity Approximate Time
Training on the first dimension with first benchmark portfolio
• Chief Reader introduces the scoring guide 25 min.for one dimension
• Readers work in pairs reading, discussing, 40 min.and scoring the first benchmark portfolio
• Scores and issues are discussed in the 25 min.whole group
Training for additional dimension with first benchmark portfolio
• Chief Reader introduces the scoring guide 25 min.for an additional dimension
• Readers work in the same pairs to review, 20 min.discuss, and score the first benchmark portfolio for the other dimension
• Scores and issues are discussed in the 25 min.whole group
Training on second benchmark portfolio
• Chief Reader reviews the scoring guide(s) 15 min.as needed
• Readers work either in pairs or individually 30 min.to read and score a second benchmark portfolio for all aspects of performance they will be scoring during the remainder of the reading
• Scores and issues are discussed at each table 15 min.
• Issues and questions are discussed as a 30 min.whole group
Additional Training
• Discussion about issues and questions related to the scoring guides takes place after each major break
• Training on additional benchmark portfolios is done as needed
• Revisiting previously read benchmark portfolios to discuss issues is done as needed
• Individual help is provided as appropriate
Preparation for Portfolio Assessment Scoring Sessions
79
Preparation for Portfolio Assessment Scoring Sessions
As with the culminating assessment reading, the Chief Reader will want toset a professional tone by welcoming Readers, outlining the scoring task,and introducing Table Leaders and any guests. It is important to acknowl-edge that Readers come to the reading with a sense of what strong studentperformance looks like. However, it is essential that Readers develop ashared understanding of the scoring criteria and learn to apply the criteriaconsistently.
Because portfolio scoring guides are complex, training is conducted onedimension at a time. The Chief Reader begins by reviewing the scoringguide, focusing on the definitions of the aspects and the criteria for the“accomplished” and “promising” levels of performance. Involving Readersin actively looking at and discussing the scoring criteria is important totheir understanding of the differences between various aspects and perfor-mance levels. Chief Readers should then discuss the “not enough evidenceto judge” category, and other issues that the Chief Reader and TableLeaders identified prior to the reading.
Readers then work in pairs, reading together and discussing the firstbenchmark portfolio. After a pair has reviewed the entire benchmark port-folio, they should discuss and come to agreement on scores. The ChiefReader then collects, usually by a show of hands, all of the scores for thatbenchmark portfolio. After the scores are tallied, the Chief Reader involvesReaders in a discussion of each aspect of performance. Part of helpingReaders reach a common understanding is sharing the consensus scorethat had been assigned before the reading and citing specific evidence thatwas used to arrive at that score. This process of reading the first bench-mark portfolio and training on one dimension will take about 90 minutes,depending on the number of Readers and issues that arise.
After discussion is completed for the first scoring dimension, repeat theprocess for each additional dimension. The Chief Reader should begin byreviewing the scoring guide and involving Readers in identifying importantfeatures for the “accomplished” and “promising” performance levels.Readers should continue to work in the same pairs, reviewing and scoringthe same benchmark portfolio, this time for the additional dimension.After Readers have rescored the portfolio for this dimension, the ChiefReader again collects the scores, and leads a discussion about the evi-dence and consensus scores for each aspect. Because Readers are assess-ing a portfolio they have read before, this process of scoring and dis-cussing the second dimension will likely take about 70 minutes.
Training on the second benchmark portfolio is slightly different from train-ing on the first. The scoring guides may need only a brief review beforeReaders start reading the portfolio. Depending on the purpose of the port-folio reading and the number of copies of benchmark portfolios that areavailable, Readers may read the second benchmark portfolio either in apair or individually. They should, however, be reading and scoring as manydimensions and aspects as they will be expected to score during the restof the reading. After a reasonable time for reading and scoring the bench-mark (40–45 minutes), Table Leaders should collect scores and lead a dis-cussion at their table, taking note of issues and questions. The ChiefReader can then conduct a room- wide discussion about the scoring of thebenchmark, addressing specific questions on the scoring guides and com-menting on issues and questions that were raised at the tables. The train-ing on the second benchmark is likely to take about 90 minutes.
80
At this point, if it seems that most Readers are scoring at or within onepoint of the consensus score, the Chief Reader and Table Leaders maydecide to move ahead with scoring rather than training on another bench-mark. It is also possible to train on the next benchmark at the table level,having Readers read, score, and discuss aspects on which they disagree. Ifmore than one reading room is available, it is possible to have someReaders who seem confident start scoring portfolios while others receiveadditional training.
After every significant break, such as for lunch or overnight, the ChiefReader or Table Leaders will need to answer questions that Readers haveabout the scoring guide. If necessary, the Chief Reader may decide to re-train Readers using a new benchmark portfolio on the morning of the sec-ond day. It is also possible to reuse one of the benchmark portfolios thathas already been discussed as a way to refocus Readers on particularaspects or issues.
Limiting the movement of people and portfolios will make a reading moreproductive. One design for distributing portfolios is to organize portfoliosfrom different classrooms in batches (10 –12) and distribute a batch toeach table at the beginning of the day, after lunch, and as needed. Readersselect and read a portfolio from the batch, avoiding any from their ownstudents. Portfolios that have been read are placed in a “read once” stack.If the reading design calls for portfolios to be read a second time, Readersshould select portfolios from the “read once” stack and when finished,place them in a “read twice/completed” stack. Completed portfoliosshould be removed from the table periodically. When the Readers havecompleted a batch, then a new batch should be brought to the table. Thisdesign both reduces the movement of portfolios in the room and easilyallows for double readings of portfolios, if the design calls for that.
The easiest system for collecting scores is to photocopy the completedscoring guide forms, making sure that scores and identifying informationfor the portfolio and Reader are clearly visible. Some districts may wish todevelop their own system for collecting portfolio and Reader informationand scores.
Portfolio scores are reported as performance levels (using level descrip-tors from the scoring guide such as “Promising,” “Accomplished,” etc.).However, it may be important to discuss which scores should be reportedto which audiences. While teachers may find the scores for aspects of thedimensions useful in planning instruction, school board members may notneed this level of detail. Deciding which scores to report is an importantdecision each district will need to make.
As with the culminating assessment, your district may want to produce anumber of different reports using the portfolio results, including individualscore reports. The easiest form of an individual report is a copy of theactual score form that includes performance - level descriptions. In addi-tion, many teachers and administrators may find distributions to be use-ful. Caution should be used in creating distributions at the classroom orschool level because small samples may not be representative of the stu-dent population.
Preparation for Portfolio Assessment Scoring Sessions
Collecting the Readers’Scores
Reporting the Results
Controlling the Flow ofPortfolios
81
APPENDIX B
COVER SHEET (MUST ACCOMPANY EACH ARTIFACT)
PACESETTER SPANISH PORTFOLIO ASSESSMENT
NAME:
Selection: Date work was done:
Why did you select this piece of work?
What does it show or tell about you?
What did you learn from doing this assignment?
This piece of work shows evidence of the following dimension(s) or aspect(s):
• Dimension 1: Demonstrating knowledge of Hispanic cultures
Showing awareness of the diversity of Hispanic cultures
Identifying contributions of Hispanic figures
Making connections with other disciplines and own culture(s)
• Dimension 2: Using Spanish to communicate effectively
Deriving meaning from printed materials and oral discourse
Expressing meaning in oral and written form
COVER SHEET (MUST ACCOMPANY EACH ARTIFACT)
PACESETTTER SPANISH PORTFOLIO ASSESSMENT
NOMBRE:
Título: Fecha de ejecución:
¿Por qué seleccionaste esta obra?
¿Qué muestra o dice esta obra sobre ti?
¿Qué aprendiste de esta tarea?
Esta obra indica evidencia en los siguientes aspectos y dimensiones:
• Dimensión 1: Demostrar conocimiento de las culturas hispanas
Demostrar conciencia de la diversidad de las culturas hispanas
Identificar contribuciones de figuras hispanas
Hacer conexiones con otras disciplinas y cultura(s)
• Dimensión 2: Usar el español para comunicaciones eficientes
Comprensión de textos y comprensión auditiva
Expresión oral y escrita
82
83
Please list all artifacts included in your portfolio. Indicate the aspect(s) anddimension(s) for which you are showing evidence with each artifact you selected.
DIMENSION 1: Demonstrating knowledge of Hispanic cultures
• Showing awareness of the diversity of Hispanic cultures
• Identifying contributions of Hispanic figures
• Making connections with other disciplines and own culture(s)
DIMENSION 2: Using Spanish to communicate effectively
• Deriving meaning from printed materials and oral discourse
• Expressing meaning in oral and written form
App
end
ix C
PA
CE
SE
TT
ER
SP
AN
ISH
¿N
OS
CO
NO
CE
MO
S?
DIM
EN
SIO
N 1
: D
EM
ON
ST
RA
TIN
G K
NO
WL
ED
GE
OF
HIS
PA
NIC
CU
LT
UR
ES
Po
rtfo
lio
Ass
essm
ent:
Ev
alu
ati
ng
asp
ects
of
the
dim
ensi
on
Sh
ow
ing
aw
are
nes
s o
f th
e d
iver
sity
of
His
pa
nic
cult
ure
s
Beg
inn
ing
�
Dev
elo
pin
g
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
iden
ce t
o
jud
ge
�
• re
flec
tio
n o
n h
ow
div
erse
th
e H
isp
anic
wo
rld
is
• ex
amp
les
of
sev
eral
cu
ltu
res
of
the
His
pan
ic w
orl
d
(mo
re t
han
on
e)
• re
flec
tio
n o
n i
ssu
es l
ike
curr
ent
even
ts,
his
tori
cal
hig
hli
gh
ts,
geo
gra
ph
y,
lite
ratu
re,
etc.
Stu
den
t p
rese
nts
sup
erfi
cial,
va
gu
e a
nd
fra
gm
en
tary
ev
iden
ce
of
his
/her
aw
are
ness
of
the
div
ersi
ty o
f H
isp
anic
cult
ure
s.
Stu
den
t p
rese
nts
lim
ited
,
inco
mp
lete
an
d u
nev
en
evid
ence
of
his
/her
aw
are
ness
of
the
div
ersi
ty o
f H
isp
anic
cult
ure
s. S
tud
ent
pre
sen
ts
few
sp
ecif
ic e
xa
mp
les.
Stu
den
t p
rese
nts
ev
iden
ce o
f a
gen
era
l
aw
are
nes
s o
f th
e
div
ersi
ty o
f H
isp
anic
cult
ure
s. S
tud
ent
pre
sen
ts s
om
e g
en
era
l
insi
gh
ts a
nd
exam
ple
s.
Stu
den
t p
rese
nts
am
ple
evid
ence
of
his
/her
aw
are
ness
of
the
div
ersi
ty o
f H
isp
anic
cult
ure
s. S
tud
ent
pre
sen
ts
a w
ide
ran
ge
of
ex
am
ple
s, w
ith
so
me
clea
r in
sig
hts
.
Stu
den
t p
rese
nts
con
sist
entl
y i
nsi
gh
tfu
l
evid
ence
of
his
/her
aw
are
ness
of
the
div
ersi
ty o
f H
isp
anic
cult
ure
s. S
tud
ent
pre
sen
ts
va
ried
an
d e
xte
nd
ed
exa
mp
les.
Co
nta
ins
too
litt
le e
vid
en
ce
of
aw
are
ness
of
the
div
ersi
ty
of
His
pan
ic
cult
ure
s to
just
ify
a s
core
.
Iden
tify
ing c
on
trib
uti
on
s of
His
pan
ic f
igu
res
Beg
inn
ing
�
Dev
elo
pin
g
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
iden
ce t
o
jud
ge
�
• ex
am
ple
s o
f se
vera
l H
isp
an
ic f
igu
res
• co
ntr
ibu
tio
ns
of
con
tem
po
rary
as
wel
l as
his
tori
cal
fig
ure
s o
f th
e H
isp
anic
wo
rld
in
dif
fere
nt
fiel
ds
• co
ntr
ibu
tio
ns
of
eth
nic
gro
up
s
Stu
den
t p
rese
nts
sup
erfi
cial,
va
gu
e a
nd
fra
gm
en
tary
ev
iden
ce
of
con
trib
uti
on
s o
f
His
pan
ic f
igu
res.
Stu
den
t p
rese
nts
lis
ts
wit
h a
few
na
mes
an
d
des
crip
tio
ns.
Stu
den
t p
rese
nts
lim
ited
an
d u
nev
en e
vid
ence
of
con
trib
uti
on
s o
f H
isp
anic
fig
ure
s. S
tud
ent
pre
sen
ts
few
sp
ecif
ic n
oti
on
s a
nd
des
crip
tio
ns
of
con
trib
uti
on
s.
Stu
den
t sh
ow
s ev
iden
ce
of
som
e g
en
era
l
con
trib
uti
on
s o
f H
isp
anic
fig
ure
s. S
tud
ent
pre
sen
ts
som
e i
nfe
ren
ces
ab
ou
t
pla
ce a
nd
im
pa
ct i
n
his
tory
.
Stu
den
t p
rese
nts
am
ple
evid
ence
of
con
trib
uti
on
s
of
His
pan
ic f
igu
res.
Stu
den
t p
rese
nts
am
ple
infe
ren
ces
ab
ou
t im
pa
ct
an
d r
ole
in
his
tory
.
Stu
den
t p
rese
nts
con
sist
entl
y t
ho
rou
gh
evid
ence
of
con
trib
uti
on
s
of
His
pan
ic f
igu
res.
Stu
den
t p
rese
nts
insi
gh
tfu
l in
fere
nce
s an
d
an
aly
sis
of
con
trib
uti
on
s,
as
wel
l a
s im
pa
ct a
nd
role
in
his
tory
.
Co
nta
ins
too
litt
le e
vid
en
ce
of
con
trib
uti
on
s
of
His
pan
ic
fig
ure
s to
just
ify
a s
core
.
Mak
ing c
on
nec
tion
s w
ith
oth
er d
isci
pli
nes
an
d o
wn
cu
ltu
re(s
)
Beg
inn
ing
�
Dev
elo
pin
g
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
iden
ce t
o
jud
ge
�
• li
nk
s to
ow
n c
ult
ure
(s)
and
per
son
al e
xp
erie
nce
s
• co
mp
aris
on
an
d c
on
tras
t o
f H
isp
anic
cu
ltu
res
wit
h
stu
den
t's
ow
n c
ult
ure
(s)
• co
nn
ecti
on
of
his
tori
cal,
eco
no
mic
, li
tera
ry,
cult
ura
l,
po
liti
cal,
geo
gra
ph
ic,
and
en
vir
on
men
tal
info
rmat
ion
on
Sp
anis
h -
spea
kin
g c
ou
ntr
ies
to s
tud
ent's
wo
rk i
n s
oci
al
stu
die
s, E
ng
lish
, an
d o
ther
are
as o
f th
e cu
rric
ulu
m
Stu
den
t p
rese
nts
sup
erfi
cial
an
d i
sola
ted
evid
ence
of
con
nec
tio
ns
wit
h o
ther
dis
cip
lin
es
an
d o
wn
cu
ltu
re(s
).
Stu
den
t p
rese
nts
irrele
va
nt
ex
am
ple
s.
Stu
den
t p
rese
nts
lim
ited
,
part
ial
an
d u
nev
en
evid
ence
of
con
nec
tio
ns
wit
h o
ther
dis
cip
lin
es a
nd
ow
n c
ult
ure
(s).
Stu
den
t
pre
sen
ts e
xa
mp
les
tha
t
are
no
t a
lwa
ys
rele
va
nt.
Stu
den
t p
rese
nts
ev
iden
ce o
f so
me
gen
era
l co
nn
ecti
on
s
wit
h o
ther
dis
cip
lin
es
an
d o
wn
cu
ltu
re(s
).
Stu
den
t p
rese
nts
exa
mp
les
wit
ho
ut
mu
ch
dep
th o
r in
sigh
t.
Stu
den
t p
rese
nts
am
ple
evid
ence
of
con
nec
tio
ns
wit
h o
ther
dis
cip
lin
es a
nd
ow
n c
ult
ure
(s).
Stu
den
t
pre
sen
ts s
om
e
infe
ren
ces
an
d a
wid
e
ran
ge
of
insi
gh
tfu
l
exa
mp
les.
Stu
den
t p
rese
nts
con
sist
entl
y t
ho
rou
gh
evid
ence
of
con
nec
tio
ns
wit
h o
ther
dis
cip
lin
es a
nd
ow
n c
ult
ure
(s).
Stu
den
t
pre
sen
ts n
ew
, crea
tiv
e,
an
d o
rigin
al
exam
ple
s.
Co
nta
ins
too
litt
le e
vid
en
ce
of
con
nec
tio
ns
wit
h o
ther
dis
cip
lin
es a
nd
ow
n c
ult
ure
(s)
to j
ust
ify
a
sco
re.
84
85
PA
CE
SE
TT
ER
SP
AN
ISH
¿N
OS
CO
NO
CE
MO
S?
Po
rtfo
lio
Ass
essm
ent:
Ev
alu
ati
ng
Dim
ensi
on
1
DE
MO
NS
TR
AT
ING
KN
OW
LE
DG
E
OF
HIS
PA
NIC
CU
LT
UR
ES
Beg
inn
ing
�
Dev
elo
pin
g
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
va
nce
d
�
No
t en
ou
gh
ev
iden
ce
to j
ud
ge
�
• C
on
sid
er p
erfo
rman
ce o
n a
ll
thre
e as
pec
ts i
ncl
ud
ed i
n t
his
dim
ensi
on
.
Th
e co
llec
tio
n p
rese
nts
sup
erfi
cia
l, v
ag
ue,
an
d
fra
gm
en
tary
ev
iden
ce
of
kn
ow
led
ge o
f
His
pan
ic c
ult
ure
s.
Stu
den
t p
rese
nts
few
an
d i
rrele
van
t
exam
ple
s.
Th
e co
llec
tio
n p
rese
nts
lim
ited
an
d u
nev
en
evid
ence
of
kn
ow
led
ge
of
His
pan
ic c
ult
ure
s,
wit
h f
ew
sp
ecif
ic
exam
ple
s, w
hic
h m
ay
no
t al
way
s b
e re
lev
ant.
Th
e co
llec
tio
n p
rese
nts
ev
iden
ce o
f a
gen
era
l
kn
ow
led
ge
of
His
pan
ic
cult
ure
s, w
ith
so
me
gen
era
l ex
am
ple
s an
d
insi
gh
ts.
Th
e co
llec
tio
n p
rese
nts
am
ple
ev
iden
ce o
f
kn
ow
led
ge
of
His
pan
ic
cult
ure
s, w
ith
a w
ide
ran
ge
of
insi
gh
tfu
l
exam
ple
s.
Th
e co
llec
tio
n p
rese
nts
con
sist
entl
y i
nsi
gh
tfu
l
an
d t
ho
rou
gh
ev
iden
ce
of
kn
ow
led
ge o
f
His
pan
ic c
ult
ure
s, w
ith
var
ied
an
d o
rig
inal
anal
ysi
s an
d e
xam
ple
s.
Th
e co
llec
tio
n
con
tain
s to
o l
ittl
e
evid
ence
of
kn
ow
led
ge o
f
His
pan
ic c
ult
ure
s to
just
ify
a s
core
.
PA
CE
SE
TT
ER
SP
AN
ISH
¿N
OS
CO
NO
CE
MO
S?
DIM
EN
SIO
N 2
: U
SIN
G S
PA
NIS
H T
O C
OM
MU
NIC
AT
E E
FF
EC
TIV
EL
Y
Port
foli
o A
sses
smen
t: E
valu
ati
ng a
spec
ts o
f th
e d
imen
sion
Der
ivin
g m
ean
ing
fro
m p
rin
ted
ma
teri
als
an
d o
ral
dis
cou
rse
Beg
inn
ing
�
Dev
elop
ing
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
van
ced
�
Not
enou
gh
ev
iden
ce t
o
jud
ge
�
• sk
ills
in
clu
ded
: li
sten
ing
and
rea
din
g
• co
mp
reh
ensi
on
an
d i
nte
rpre
tati
on
of
pri
nte
d m
ater
ials
on
div
erse
to
pic
s o
f
var
yin
g l
evel
s o
f co
mp
lex
ity
an
d l
eng
th
• co
mp
reh
ensi
on
an
d i
nte
rpre
tati
on
of
ora
l d
isco
urs
e o
n d
iver
se t
op
ics
of
var
yin
g l
evel
s o
f co
mp
lex
ity
an
d l
eng
th
Stu
den
t p
rese
nts
lim
ited
evid
ence
of
der
ivin
g
mea
nin
g f
rom
pri
nte
d
mat
eria
ls a
nd
ora
l
dis
cou
rse.
Stu
den
t
un
der
sta
nd
s a
few
gen
era
l id
eas,
wit
h
con
tin
uou
s
mis
un
der
stan
din
gs.
Stu
den
t pre
sents
un
even
evid
ence
of
der
ivin
g
mea
nin
g f
rom
pri
nte
d
mat
eria
ls a
nd
ora
l d
isco
urs
e.
Stu
den
t u
nd
erst
an
ds
som
e
main
id
eas
on
fam
ilia
r
top
ics,
wit
h f
req
uen
t
mis
un
der
stan
din
gs.
Stu
den
t pre
sents
som
e
evid
ence
of
der
ivin
g
mea
nin
g f
rom
pri
nte
d
mat
eria
ls a
nd
ora
l d
isco
urs
e.
Stu
den
t u
nd
erst
an
ds
main
idea
s an
d s
om
e d
etail
s on
som
ew
ha
t fa
mil
iar t
op
ics
an
d m
ay m
ak
e so
me
infe
ren
ces
dep
end
ing o
n
co
mp
lex
ity
of
tex
t.
Stu
den
t pre
sents
ap
pro
pri
ate
an
d m
ean
ingfu
l ev
iden
ce o
f
der
ivin
g m
ean
ing
fro
m
pri
nte
d m
ater
ials
an
d o
ral
dis
cours
e. S
tuden
t
un
der
stan
ds
main
id
eas
an
d
most
det
ail
s ev
en o
n
un
fam
ilia
r to
pic
s, a
nd
ca
n m
ak
e i
nfe
ren
ces
ap
pro
pri
ate
to t
he
task
.
Stu
den
t pre
sents
th
oro
ugh
evid
ence
of
der
ivin
g
mea
nin
g f
rom
pri
nte
d
mat
eria
ls a
nd
ora
l
dis
cours
e. S
tuden
t
ex
tra
po
late
s in
form
ati
on
in n
ew w
ays,
in
clu
din
g
som
e so
cial
an
d c
ult
ura
l
refe
ren
ces
an
d a
ffecti
ve
ov
erto
nes.
Conta
ins
too
litt
le
evid
ence
of
der
ivin
g
mea
nin
g
from
pri
nte
d
mat
eria
ls a
nd
ora
l
dis
cours
e to
just
ify a
sco
re.
Exp
ress
ing m
ean
ing i
n o
ral
an
d w
ritt
en
form
Beg
inn
ing
�
Dev
elop
ing
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
van
ced
�
Not
enou
gh
ev
iden
ce t
o
jud
ge
�
• sk
ills
in
clu
ded
: sp
eak
ing a
nd
wri
tin
g •
dev
elo
p a
nd
pre
sen
t te
xts
ora
lly
• d
evel
op
an
d p
rese
nt
wri
tten
tex
ts i
n
fin
al d
raft
s
• d
emo
nst
rate
tec
hn
ical
co
mm
and
of
the
Sp
anis
h l
ang
uag
e (l
ang
uag
e u
sag
e,
stru
ctu
res
and
co
nv
enti
on
s)
Stu
den
t p
rese
nts
lim
ited
an
d f
ra
gm
en
tary
evid
ence
of
exp
ress
ing
mea
nin
g i
n o
ral
and
wri
tten
fo
rm. S
tud
ent
pre
sen
ts a
n i
na
deq
ua
te
tech
nic
al
com
ma
nd
of
the
Sp
an
ish
lan
gu
age
(e.g
., l
ack
of
flu
en
cy
,
iso
late
d s
ente
nce
s,
inco
mp
lete
utt
era
nces)
.
Co
mm
un
ica
tio
n i
mp
eded
by
in
terfe
ren
ce f
ro
m
an
oth
er l
an
gu
ag
e.
Stu
den
t pre
sents
un
even
evid
ence
of
exp
ress
ing
mea
nin
g i
n o
ral
and
wri
tten
form
. S
tuden
t pre
sents
lim
ited
tec
hn
ical
com
man
d
of
the
Sp
an
ish
lan
gu
age
(e.g
., l
imit
ed f
luen
cy,
un
co
nn
ecte
d d
isco
urs
e o
r
sen
ten
ces,
lim
ited
ran
ge
of
vo
cab
ula
ry).
Som
e
inte
rfe
ren
ce f
ro
m a
no
ther
lan
gu
age
may h
am
per
com
mu
nic
ati
on
.
Stu
den
t pre
sents
som
e
evid
ence
of
exp
ress
ing
mea
nin
g i
n o
ral
and
wri
tten
form
. S
tuden
t pre
sents
som
e
un
der
lyin
g
tech
nic
al
co
mm
an
d o
f th
e
Sp
an
ish
lan
gu
age
(e.g
.,
emer
gin
g n
oti
on
of
adeq
uat
e
reg
iste
r, a
deq
uate
ran
ge o
f
vo
cab
ula
ry,
self
-
co
rrecti
on
s).
Min
imal
inte
rfe
ren
ce f
ro
m a
no
ther
lan
gu
age
does
not
ham
per
com
mu
nic
ati
on
.
Stu
den
t pre
sents
ap
pro
pri
ate
an
d m
ean
ingfu
l ev
iden
ce o
f
exp
ress
ing
mea
nin
g i
n o
ral
and w
ritt
en f
orm
. S
tuden
t
pre
sents
pro
per
tec
hn
ical
com
man
d o
f th
e S
pan
ish
lan
gu
age
(e.g
., o
rgan
ized
and
art
icu
late
d m
essa
ges
,
pro
per
use
of
circ
um
locu
tio
n,
at t
imes
cre
ativ
e id
eas,
co
here
nt
use
of
vo
cab
ula
ry
and
mo
stly
ad
equ
ate
use
of
reg
iste
r).
Alm
ost
no
inte
rfe
ren
ce w
ith
com
mu
nic
ati
on
.
Stu
den
t pre
sents
hig
hly
crea
tive,
th
oro
ugh
, an
d
mea
nin
gfu
l ev
iden
ce o
f
exp
ress
ing
mea
nin
g i
n
ora
l an
d w
ritt
en f
orm
.
Stu
den
t pre
sents
cle
ar
tech
nic
al
com
man
d o
f
the
Sp
an
ish
lan
gu
age
(e.g
., w
ell
art
icu
late
d a
nd
org
an
ized
mess
ag
es,
ap
pro
pri
ate
use
of
reg
iste
r, w
ide r
an
ge o
f
vo
cab
ula
ry,
and
a h
igh
lev
el
of
flu
en
cy
).
Occ
asi
on
al
erro
rs d
o n
ot
inte
rfe
re w
ith
com
mu
nic
ati
on
.
Conta
ins
too
litt
le
evid
ence
of
expre
ssin
g
mea
nin
g i
n
ora
l an
d
wri
tten
form
to j
ust
ify a
sco
re.
86
87
PA
CE
SE
TT
ER
SP
AN
ISH
¿N
OS
CO
NO
CE
MO
S?
Po
rtfo
lio
Ass
essm
ent:
Ev
alu
ati
ng
Dim
ensi
on
2
US
ING
SP
AN
ISH
TO
CO
MM
UN
ICA
TE
EF
FE
CT
IVE
LY
Beg
inn
ing
�
Dev
elo
pin
g
�
Pro
mis
ing
�
Acc
om
pli
shed
�
Ad
van
ced
�
Not
enou
gh
evid
ence
to j
ud
ge
�
• C
on
sid
er p
erfo
rman
ce i
n t
he
two
aspec
ts i
ncl
uded
in t
his
dim
ensi
on
.
Th
e co
llec
tio
n p
rese
nts
lim
ited
ev
iden
ce o
f
usi
ng
Sp
anis
h t
o
com
mu
nic
ate
eff
ecti
vely
an
d a
n
inad
eq
uate
tech
nic
al
com
man
d o
f th
e
Sp
anis
h l
ang
uag
e.
Th
e co
llec
tio
n p
rese
nts
un
even
ev
iden
ce o
f
usi
ng
Sp
anis
h t
o
com
mu
nic
ate
eff
ecti
vely
an
d a
lim
ited
tec
hn
ical
com
man
d o
f th
e
Sp
anis
h l
ang
uag
e.
Th
e co
llec
tio
n p
rese
nts
som
e e
vid
en
ce o
f
usi
ng
Sp
anis
h t
o
com
mu
nic
ate
eff
ecti
vely
an
d s
om
e
un
der
lyin
g t
ech
nic
al
com
man
d o
f th
e
Sp
anis
h l
ang
uag
e.
Th
e co
llec
tio
n p
rese
nts
ap
pro
pri
ate
an
d
mea
nin
gfu
l ev
iden
ce
of
usi
ng
Sp
anis
h t
o
com
mu
nic
ate
eff
ecti
vely
an
d p
rop
er
tech
nic
al c
om
man
d o
f
the
Sp
anis
h l
ang
uag
e.
Th
e co
llec
tio
n p
rese
nts
crea
tiv
e,
tho
ro
ug
h
an
d m
ean
ingfu
l
evid
ence
of
usi
ng
Span
ish t
o
com
munic
ate
eff
ecti
vely
an
d c
lear
tech
nic
al c
om
man
d o
f
the
Sp
anis
h l
ang
uag
e.
Th
e co
llec
tio
n
conta
ins
too l
ittl
e
evid
ence
of
usi
ng
Span
ish t
o
com
munic
ate
effe
ctiv
ely
to
ju
stif
y a
sco
re.
APPENDIX D
Dimension 1 Dimension 2
Comments on Artifacts Comments on ArtifactsDiversity of Hispanic Cultures: 1 Deriving Meaning from Printed/Oral Materials: 1
-Would be useful evidence for aspect 3-Only underlined info.; needs to use cover sheet toexplain?-Only a list of foods-No ample inferences or impact-No insights included-Does not mention (Popo)-Was very well done but could have had moreexamples than just one.-Too much evidence; it was overwhelming
-Hard to judge; don't have a copy of the reading (H.Cortés)-Hard to tell what really knew; a lot was copied fromthe text-No evidence for reading-Cassette was marked (Los Hispanos en USA) forlistening but it was really for speaking.-This student must have done more projectsinvolving art/drawing that would have been betterevidence of his abilities/knowledge-Not the best example-Just by underlining hard to know if really reads well(noticias)-Hard to judge this artifact because not part of coursework (Leyenda)* Aspect I should refer to printed language instead ofprinted material
Contributions of Hispanic Figures: 2 Expressing Meaning in Oral & Written Form: 2
-No personal connection made (also applies toaspect 3)-Only mentioned one person-2 Non-Hispanics-Not in Pacesetter; outside source.-Seems to be a contradiction at the end-Same information in both artifacts-Does not show contributions-Not a good choice for aspect II (Popo e Ixtla)-Barely refers to the pictures-General awareness; El Greco-Several students conflate “Hispanic figures” andancient/pre-Colombian civilizations-Only mentioned one person-Piece is of only 1 Hispanic, not several (Blasón)
-Very bad quality of recording-It is not clear if the video included was supposed toaccompany this or not. This info would be helpful.-There wasn’t a prueba in the cassette-Could not find Martin Luther King-Not a reading activity and only had 2 shortquestions to listen to (Cinta-Ballena)-Video missing-Regarding aspect 2: difference between minimalinterference (accomplished) isn’t clear-No oral; good writing-Not enough evidence to judge speaking-Can’t hear tape-Student read his Maya essay and presented it asevidence of speaking; no evidence for listening
Connections W/Other Disciplines/Cultures: 3 -Not a good writing sample
-Fails to make connection-It is not clear what this is or how it is relevant-Linking to own culture lacking as well ascomparing and contrasting-No connection to own culture made-Does not demonstrate knowledge of Hispaniccultures; no personal connection (La Biografia)-Interview shows him interviewing applicant-No connection (Los Hispánicos)-Not a good example (Yo)-Only one sample, not enough evidence-Aspect III has at least two very distinctcomponents; should connection to self /own cultureand to other disciplines both be evident?-Unable to understand recording (Tape w/partner)-No mention of personal connection (Blasón)-No contribution to Hispanic figures (Dos leyendas)-Student is not clear on what connections w/otherdisciplines meanNo personal connections
-I’m finding hamper to be a key word in interpretingaspect 2; first language interference/influence rarelyimpedes communication, nor even hampercommunication, but rather only hamper the ease ofcommunication-An assignment on grammar is not good evidence ofwriting-Question of multiple evidence
88
89
Comments/Suggestions Gen.
-No cover sheet for more than half of the artifactspresented-A lot of the artifacts were irrelevant-Heroes essay is in English-No examples for Aspect-Well organized-In general, this student did not presentworksheets, but only cover sheets with reflections-Student should have chosen only the piecesstrongest in each aspect
-Shows a very positive attitude toward his learningand performance-No speaking samples-In many of the artifacts included as evidence ofreading comp, it was impossible to judge what theinformation included; the same happened with thehistory documents-Need more reading sample and also more listeningsamples-No speaking-Not clear what printed/oral materials meaning
-Cover sheet gave wrong dimension was supposedly derived from for 3 autobio--In many instances, the writing on the coversheet incopied from the writing in the artifact at length andthen is read on tape-This collection contains a lot. 13 artifacts for Dim1 to be evaluated in 27 aspects.-Portfolio is enormous; did not use but at most 8artifacts-Without clear info on coversheet, I would have notknown what the artifacts were proving (The coversheets are very important and should be part of theevaluation)-Student attached several artifacts to one coversheet* Well organized; We should use this sample tosuggest to other teachers and students how to
graphical pieces-No cover sheet-Very well rounded portfolio (02-B-13)-Students indicates 3 artifacts but only 2 found.-Not a listening or reading activity-In analyzing the time used it should be consideredthat this simultaneous processing of the activity,i.e., writing these notes, certainly takes some time.With a reasonable number of submissionscomprising a collection and disregarding technicaldifficulties, it doesn’t take that long to read/score-Do not know which artifact to use for speaking;side A had one speaking and side B had theassessment
send in their artifacts, cover sheets, etc.. (05-E- -No speaking or listening tasks were included07)-This portfolio is clearly organized- Some students have deliberately chosen workfrom throughout the year in order to demonstratetheir progress over time, which isn’t treated in therubrics. As it is weaker works submitted (maybefrom early in the year) for an aspect for whichstronger works also exist may lead to theimpression of limited or uneven overall-Many students either count Ixtla y Popo as“figures” or count the Nahna creators of the legendas “figures” for aspect 2
-Hard to tell if speaking is spoken or read-Maybe there should be a maximum # ofsubmissionsIt is very difficult to cue up tapes for eachrecording and identify which recordingcorresponds to which cover sheet.-Hard to know what to listen to without title orspeaking part.-No true writing sample-Needs to be more specific about what learned-Video with presentation was used to give score forspeaking-Too many artifacts were submitted; limit entries tofive.
200272