Regulating for quality in assessment AAIA Conference 2010 Isabel Nisbet 15 September 2010.
Regulating for quality in assessment AAIA Conference 2010
description
Transcript of Regulating for quality in assessment AAIA Conference 2010
Regulating for quality in assessment AAIA Conference 2010
Isabel Nisbet15 September 2010
Summary
Ofqual Responsibilities for assessments Reliability Standards
Our approach in action
- GCSEs and A levels
- Checking that standards are maintained
Innovation and E-assessment and on-demand testing
Overview
Ofqual’s role in relation to qualifications and [National Curriculum] assessments:
encourage and support the development and implementation of [qualifications] [regulated assessments] that give a reliable indication of achievement, and that indicate a consistent level of attainment (including over time) between comparable [qualifications] [assessments]
promote public confidence in [regulated qualifications] [assessment arrangements]
Ofqual’s Reliability Programme
“..As the regulator of qualifications in England I believe that it is essential for us to understand better the reliability of assessments in our national systems… Ofqual will undertake an in-depth programme of work – call it a health check – on the reliability of tests, examinations and teacher assessments in this country.”
Kathleen Tattersall, May 2008
A perfect assessment?
"We have really got to begin to engage with the public in a dialogue about the nature of the examination system.
"There's a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on.
"Would a student have received the same result if she happened to have taken a different version of the examination, on a different day, with a different examiner marking her work?”
Kathleen Tattersall, 2008
Reliability
Reliability is an indicator of the quality of an assessment
Reliability work in England has generally been
- Isolated
- Partial
- Under-theorised
- Under-reported
- Misunderstood
Ofqual’s reliability programme will help to improve the situation
To gather evidence for Ofqual to develop regulatory policy on reliability of results from national tests, examinations and qualifications
Aims
Strand 1: Generating evidence of reliability
Strand 2: Interpreting and communicating evidence of reliability
Strand 3: Developing reliability policy
Strand 3a: Exploring public understanding of reliability
Strand 3b: Developing Ofqual policy on reliability
Programme structure
Work in progress
AQA: The reliabilities of GCSE and GCE components
Cambridge Assessment: The reliabilities of GCSE and GCE components and qualifications
City & Guilds: The reliabilities of vocational qualifications:
Assessment Europe: Quantifying and interpreting component reliability using G-theory analysis
Assessment Europe: Literature review of reliability studies of teacher assessments
TAG Report: Summary, further study and advice
Ofqual: Quantitative investigation of public perceptions using online questionnaire
Reliability – lessons
Concept of “error” misunderstood - Variability doesn’t mean “mistakes”
Public concerns but understanding about use of judgement
Issues around implications for progression and accountability of small differences in marks
Limits to reliability of other methods of assessment too - Pre-university selection tests- Interviews/vivas
Standards – some provocative remarks
Tired old arguments about standards over time
Important new arguments about whether qualifications (academic, professional or vocational) are FIT FOR PURPOSE
Time to challenge:– Unevidenced prejudices
» against some qualifications» against some awarding organisations
– Urban myths – National inferiority complexes – Unattainable ideals about reliability
Maintaining standards in practice – GCSEs and A levels
Ofqual sets conditions awarding organisations must meet to be recognised – governance; expertise; quality assurance
Ofqual recognises awarding organisations that show they can meet the conditions
Ofqual sets criteria for specific groups of qualifications (eg “GCSE history”)
Ofqual accredits qualifications that meet these criteria
Ofqual makes sure that different versions of the same qualification are comparable
Ofqual issues codes of practice - principles, processes and practices to be followed by organisations developing and delivering GCSEs and A levels
Ofqual makes sure that action is taken in real time to ensure that each set of awards is consistent and fair and that standards are maintained over time
Checking standards in all regulated qualifications
Monitoring the questions set and the marking of examiners
Checking that awarding organisations carry out their roles correctly, through
- scrutiny programme
- code of practice monitoring programme
- audits of awarding organisations
Comparability studies (reviewing qualification standards over time, across organisations and between subjects)
Planned international comparison study ALL qualifications, selected by risk
Innovation
Ofqual aims to facilitate innovation – e-assessment? On-demand?
Need for valid assessments of 21st Century learning
What might drive innovation?
- Efficiency
- Demand
- Quality
- Changes in teaching and learning
- Technological change
Ofqual has published principles to govern E-assessment to support innovation while protecting standards and integrity
Regulatory principles for e-assessment
Validity and reliability
Security
Data integrity
Access to e-assessment
Business continuity and disaster recovery
Use of e-portfolios
The extent of e-marking across Awarding Bodies: 2007
36% of GCSE and A level scripts electronically marked (wide variation across awarding bodies)
Awarding Body trials found: - Reliable- Timely- Scanning and supporting technologies did not fail- Accurate assembly of results and archiving - (Senior) Examiner satisfaction- No extra admin burden on centres
No link to concerns over quality of marking
The big step
From E-marking to E-testing
Over to you
How we can help you
- Providing essential information on assessments
- Providing information on what we require from those involved in delivering assessments
How you can help us
- Give us your view on what assessments should deliver
- Help us to make assessments as fair and accurate as possible
Contact us: www.ofqual.gov.uk
Summary
Ofqual Responsibilities for assessments Reliability Standards
Our approach in action
- GCSEs and A levels
- Checking that standards are maintained
Innovation and E-assessment and on-demand testing