PowerPoint Presentation
Overreach in High-Stakes Testing: The case of Chiles National University Admission Test (PSU)
9th ITC Conference, San Sebastian, Spain
July 5, 2014
Mladen Koljatic
Monica Silva
Richard P. Phelps
2
Topics of the presentation
Types of admission tests
The PSU context and problems
Lessons to be learned
The role of international testing associations
Types of admission tests
Aptitude Tests or Achievement Tests in the Context of Admission?
Where school inputs vary widelyacademic achievement tests are likely to measure the opportunity to learn more than the ability to learn. [Developing] nations concerned about picking their future talent must consider the possibility that an aptitude testmay be more able to overcome the local differences in school quality (Heyneman, 1987, p. 253)
SOURCE: Phelps, Standardized Testing Primer, 2007
Aptitude Tests or Achievement Tests in the Context of Admission?
Achievement tests are fairer to students because they measure accomplishment rather than ill-defined notions of aptitude; they can be used to improve performance; they are less vulnerable to charges of socioeconomic bias; they are more appropriate for schools, because they set clear curricular guidelines and clarify what is important for students to learnthe movement away from aptitude tests is an appropriate step for U.S. students, schools and universities (Atkinson, 2001)
Purpose of Admission Tests
To predict college success
without providing an unfair or unjustified advantage to any particular group of test-takers
SOURCE: Phelps, Standardized Testing Primer, 2007
2. The PSU context and problems
School Context: 2 curriculums
Curricular BranchType of SchoolScientificHumanistic (SH)Technical-Professional (TP)MunicipalPrivate-SubsidizedPrivate- PaidThe context of Chiles PSU
PSU focused on measuring the SH curriculum
Students who attend the TP tracks
the poorest in the nation
over 90% wish to pursue tertiary education
social mobility highly related to university education in Chile
* Source: Mineduc, 2009
The context of Chiles PSU
PSU test scores used
Admission
Financial aid
Government subsidies to universities
very high-stakes associated to test scores
Students who attend TP often have no choice
Universities offer academic degrees and also technical and professional careers (such as occupational therapy, lab technician, enology)
All applicants are required to take the admission tests (PSU) for programs offered by public universities
Test scores used to assign financial aid and scholarships
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
11
The context of Chiles PSU
Centralized admission system to all public funded universities (+ some private universities)
PSU is run by a department within the largest public university (U. de Chile)
Admission system requires:
PSU test scores
High-school GPA (HSGP)
Class ranking
No other criterion is considered for application to higher education institutions
History of Chiles PSU
From 1966-2003 : aptitude-type test (similar to SAT)
From 2004- the present: PSU
Achievement tests: measure 100% of SH curriculum
Mathematics
Language
Science (Chemistry, Physics and Biology)
Social Sciences (History, Geography and Economics)
Change of tests aimed at evaluating outcomes of an educational reform
From 1966-2003
aptitude-type test modelled after SAT 1 and complemented by Subject Tests (like SAT 2 or AP) for a small number of careers that required knowledge of advanced contents
From 2004-
a set of mandatory achievement tests (Prueba de Seleccin Universitaria, PSU) that attempted to measure 100% of S&H curricular contents in Mathematics, Language, Science (Chemistry, Physics and Biology) and Social Sciences (History, Geography and Economics)
Change of tests was aimed at evaluating the outcomes of a recently implemented educational reform of secondary education (as well as selecting students for university education) a sort of SAT and NAEP combined
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
13
Project to Develop the PSU TESTS
Supported and promoted by the Ministry of Education and council of rectors of public universities (CRUCh)
Developed using public funds
Project led by an economist and a social psychologist with no training or expertise in the development of high-stakes tests
Multiplicity of Purposes for the PSU
Measure the implementation of a new curriculum
Fairly measure mastery of one of the two national curricula (SH)
Incentivize high schools to implement the new curriculum
Incentivize high school students to study more
Predict overall success of students in universities
Predict success across very different types of university programs
Reduce socio-economic disparities in performance
No single test can possibly do all of this.
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
15
Evidence of the tests technical quality should be provided for each purpose (St.13.2)
as well as adequate monitoring of potential negative consequences (St.13.1)
Multiplicity of Purposes for the PSU
How the PSU Runs
Source: adapted from the Pearson Report (2013)
Council of Rectors of Chilean Universities
"owners" of the PSU
designated by CRUCh as supervisors and official evaluators of the PSU PRESIDED BY CREATORS OF THE TEST
agency responsible for developing test items, test assembly, tests administration, test scoring, for CRUCh and associated universities
Main funder of the PSU since 2007.
It holds the purse for CRUCh universities, since they are funded through public monies
CRUCH
CRUCHS COMMITTEE OF TECHNICAL
ADVISORS FOR PSU
U. de Chile
Ministry of Education
Official Chilean Evaluations of the PSU
Eight evaluative reports from CRUCh between 2004-2010:
Pseudo-evaluations: overly optimistic reports aimed to construct a positive image of the test
Do not report any analyses for the TP group
When reporting gaps, omit the TP data from the analyses (without saying so)
Deny the existence of a growing score gap between private paid and municipal school groups, especially those attending the TP track
CRUCh denies access to data bases to independent researchers
19
International Evaluations of the PSU
ETS (2005)
Pearson (2013)
20
Efforts to suppress information
2005: ETS Report not released, according to U. Chile because of secrecy clause imposed by ETS
2012: New audit. CRUCh intended:
Auditors to report to expert team chosen by CRUCh
to curtail analyses for TP students
... to control and edit final report
21
2007: U. de Chile sued to release ETS report
A group of faculty and students represented by a pro-transparency nonprofit organization go to court
judge rules in favor of U. de Chile
Efforts to reveal information
22
Efforts to reveal information
ETS report is finally released in 2012 thanks to TRANSPARENCY LAWS passed in the country
From 2007: 5 years of pressure by university student leaders, faculty and leaders of NGOs for new audit
2012: Funded by the Ministry of Education (maximum amount suggested by CRUChs CTA: US$400,000)
Pearsons Final Report of the PSU (2013): Recommendations
PSU needs to be revised to advance equity for TP groups
reduction in contents
or study alternatives for the equitable assessment of CH and TP students
PSU is below predictive standards of other international tests
belying claims from CRUChs experts as to the predictive capacity of the PSU
24
Pearsons Final Report of the PSU (2013): SERIOUS DEFICITS
Inadequate use of PSU cut scores to assign financial help and scholarships
Inadequate use of scores to pass judgment on the quality of schools
25
Pearsons Final Report of the PSU (2013): SERIOUS DEFICITS
Faulty piloting of test items
DIF analyses does not include factors as socioeconomic status, high school curricular branch (S&H and TP)
Inadequate difficulty level of the PSU Math Test (detected in ETSs 2005 audit)
Actual reporting of PSU Science score deemed untenable: Science PSU requires not one, but three scores
26
Pearsons Final Report of the PSU (2013): SERIOUS DEFICITS
Inadequate level of analysis expected when equating a high-stakes assessment: misleading information regarding year-to-year equating and pilot calibrations. THESE ACTIVITIES ARE NOT TAKING PLACE EVEN THOUGH THE DOCUMENTATION PROVIDED SEEMS TO INDICATE THEY ARE. (p.43)
PSU equating is below international standards
ETC
27
Pearsons Evaluation of the PSU (2013)
Comprehensive
Pearson met with stakeholders other than CRUCh
Incorporated some analyses for the TP group (at no extra cost)
Specified new studies that needed (e.g.,, to guide reduction of contents)
However
Recommendations are not prioritized
Highly technical report, not aimed at policy makers
Did not look at effects on enrollment
Pearsons Final Report of the PSU (2013):
Revealed how easy it is to mislead the public when it comes to technical matters
CRUCh deceived the public as to the quality of the tests, in terms of prediction and fairness
CRUCH does NOT reduce contents in the PSU, against Pearson's reccomendation
CRUCh's new commitee of technical advisors (CTA) is controlled by an associate of the PSU test-makers
29
Aftermath of Pearsons Report...
Past Problems = Present Problems
Vested interests
No real changes in the governance system
Lack of transparency
No accountability
Main problems remain unsolved
30
Aftermath of Pearsons Report
Students demand an end to the PSU elimination of the test
Only HSGPA and class ranking as selection criteria
Moratorium not only to the PSU but to other educational tests as well
31
3. Lessons to be learned
32
Changes in tests
Some nations that are considering moving towards achievement tests might benefit from looking at the Chilean experience to avoid a proliferation of purposes, test corruption and abuse
Changes entail risks
-Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test
-There is a need for a governance scheme that guarantees independent evaluations in order to protect applicants rights to be tested by good and fair tests
-Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test
- Independent faculty that dispute pseudo-evaluations
-There is a need for a governance scheme that guarantees independent evaluations and applicants rights to be tested by good and fair tests
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
34
Data access is not only a problem in developing nations
The SAT item-level information is not frequently available to independent researchers who are not affiliated with ETS or the College Board
Source: Santelices & Wilson, 2010.
Governance Issues in Order to Prevent Test Abuse
Test-developing agency should be professional and independent
Account for funds received and quality of work
Report to external boards of experts that can demonstrate independence from the agency
Periodically audit tests for quality and fairness
Audit reports & data bases should be public
--> TRANSPARENCY AND ACCOUNTABILITY
4. The role of international testing associations
Changes in tests
Question to ask:
How can international organizations such as ITC play a more active role in promoting good consulting practices by test experts in developing nations?
Is there a larger role for testing associations such as ITC in the promotion of good practices worldwide?
The ITC Guidelines on Adapting Tests
The ITC Guidelines on Test Use
The ITC Guidelines on Computer-Based and Internet-delivered Testing
The ITC Guidelines on Quality Control in Scoring, Test Analysis and Reporting of Test Scores
The ITC Guidelines for Test Review?
The ITC Guidelines for International Consulting?
Overreach in High-Stakes Testing: The case of Chiles National University Admission Test (PSU)
Mladen Koljatic
Monica Silva
Richard P. Phelps
9th ITC Conference
San Sebastian, Spain
July 5, 2014
Sorry, but this presentation will be in English.
However, these slides will be made available later in both English and Spanish.
Click to edit Master text styles
Second level
Third level
Fourth level
Fifth level
41
Predictive validities of the PSU
(CRUCh vs Pearson estimates)
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013; CTA
Predictive validities: ACT and PSU
SOURCE: ACT, Research Summary Services, 1997_1998; Pearson, Final Report Evaluation of the Chile PSU, January 2013
Predictive validities: SAT and PSU
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013
44
Predictive validities: SAT and PSU
(BUSINESS MAJORS)
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013
Predictive validities: SAT and PSU
(Education)
SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013
46
Percentage-point drop in enrollment of municipal school graduates attending two top universities
UC U. de Chile
PAA (adm. 2003)1732
__________________________________
PSU (adm.2004)17 29
PSU (adm.2005)16 25
PSU (adm.2006)14 20
PSU (adm.2007)13 --
PSU (adm.2008)11 --
PSU (adm.2009)11 --
PSU (adm.2010) 12--
Source: official institutional info
Drop in enrolled students from Municipal Schools, other CRUCh Universities
Fuente: E. Simonsen, 2009
Estimated average 10% drop in enrollment from students from municipal schools at top CRUCh Universities in three years NO measures taken
porcentaje matrculados municipal
0
10
20
30
40
50
200320042005200620072008
PUC
U CONCE
USACH
PUCV
Top Related