Regulating for quality in assessment AAIA Conference 2010

20
Regulating for quality in assessment AAIA Conference 2010 Isabel Nisbet 15 September 2010

description

Regulating for quality in assessment AAIA Conference 2010. Isabel Nisbet 15 September 2010. Summary. Ofqual Responsibilities for assessments Reliability Standards Our approach in action - GCSEs and A levels - Checking that standards are maintained - PowerPoint PPT Presentation

Transcript of Regulating for quality in assessment AAIA Conference 2010

Page 1: Regulating for quality in assessment  AAIA Conference 2010

Regulating for quality in assessment AAIA Conference 2010

Isabel Nisbet15 September 2010

Page 2: Regulating for quality in assessment  AAIA Conference 2010

Summary

Ofqual Responsibilities for assessments Reliability Standards

Our approach in action

- GCSEs and A levels

- Checking that standards are maintained

Innovation and E-assessment and on-demand testing

Page 3: Regulating for quality in assessment  AAIA Conference 2010

Overview

Ofqual’s role in relation to qualifications and [National Curriculum] assessments:

encourage and support the development and implementation of [qualifications] [regulated assessments] that give a reliable indication of achievement, and that indicate a consistent level of attainment (including over time) between comparable [qualifications] [assessments]

promote public confidence in [regulated qualifications] [assessment arrangements]

Page 4: Regulating for quality in assessment  AAIA Conference 2010

Ofqual’s Reliability Programme

“..As the regulator of qualifications in England I believe that it is essential for us to understand better the reliability of assessments in our national systems… Ofqual will undertake an in-depth programme of work – call it a health check – on the reliability of tests, examinations and teacher assessments in this country.”

Kathleen Tattersall, May 2008

Page 5: Regulating for quality in assessment  AAIA Conference 2010

A perfect assessment?

"We have really got to begin to engage with the public in a dialogue about the nature of the examination system.

"There's a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on.

"Would a student have received the same result if she happened to have taken a different version of the examination, on a different day, with a different examiner marking her work?”

Kathleen Tattersall, 2008

Page 6: Regulating for quality in assessment  AAIA Conference 2010

Reliability

Reliability is an indicator of the quality of an assessment

Reliability work in England has generally been

- Isolated

- Partial

- Under-theorised

- Under-reported

- Misunderstood

Ofqual’s reliability programme will help to improve the situation

Page 7: Regulating for quality in assessment  AAIA Conference 2010

To gather evidence for Ofqual to develop regulatory policy on reliability of results from national tests, examinations and qualifications

Aims

Page 8: Regulating for quality in assessment  AAIA Conference 2010

Strand 1: Generating evidence of reliability

Strand 2: Interpreting and communicating evidence of reliability

Strand 3: Developing reliability policy

Strand 3a: Exploring public understanding of reliability

Strand 3b: Developing Ofqual policy on reliability

Programme structure

Page 9: Regulating for quality in assessment  AAIA Conference 2010

Work in progress

AQA: The reliabilities of GCSE and GCE components

Cambridge Assessment: The reliabilities of GCSE and GCE components and qualifications

City & Guilds: The reliabilities of vocational qualifications:

Assessment Europe: Quantifying and interpreting component reliability using G-theory analysis

Assessment Europe: Literature review of reliability studies of teacher assessments

TAG Report: Summary, further study and advice

Ofqual: Quantitative investigation of public perceptions using online questionnaire

Page 10: Regulating for quality in assessment  AAIA Conference 2010

Reliability – lessons

Concept of “error” misunderstood - Variability doesn’t mean “mistakes”

Public concerns but understanding about use of judgement

Issues around implications for progression and accountability of small differences in marks

Limits to reliability of other methods of assessment too - Pre-university selection tests- Interviews/vivas

Page 11: Regulating for quality in assessment  AAIA Conference 2010

Standards – some provocative remarks

Tired old arguments about standards over time

Important new arguments about whether qualifications (academic, professional or vocational) are FIT FOR PURPOSE

Time to challenge:– Unevidenced prejudices

» against some qualifications» against some awarding organisations

– Urban myths – National inferiority complexes – Unattainable ideals about reliability

Page 12: Regulating for quality in assessment  AAIA Conference 2010

Maintaining standards in practice – GCSEs and A levels

Ofqual sets conditions awarding organisations must meet to be recognised – governance; expertise; quality assurance

Ofqual recognises awarding organisations that show they can meet the conditions

Ofqual sets criteria for specific groups of qualifications (eg “GCSE history”)

Ofqual accredits qualifications that meet these criteria

Ofqual makes sure that different versions of the same qualification are comparable

Ofqual issues codes of practice - principles, processes and practices to be followed by organisations developing and delivering GCSEs and A levels

Ofqual makes sure that action is taken in real time to ensure that each set of awards is consistent and fair and that standards are maintained over time

Page 13: Regulating for quality in assessment  AAIA Conference 2010

Checking standards in all regulated qualifications

Monitoring the questions set and the marking of examiners

Checking that awarding organisations carry out their roles correctly, through

- scrutiny programme

- code of practice monitoring programme

- audits of awarding organisations

Comparability studies (reviewing qualification standards over time, across organisations and between subjects)

Planned international comparison study ALL qualifications, selected by risk

Page 14: Regulating for quality in assessment  AAIA Conference 2010

Innovation

Ofqual aims to facilitate innovation – e-assessment? On-demand?

Need for valid assessments of 21st Century learning

What might drive innovation?

- Efficiency

- Demand

- Quality

- Changes in teaching and learning

- Technological change

Ofqual has published principles to govern E-assessment to support innovation while protecting standards and integrity

Page 15: Regulating for quality in assessment  AAIA Conference 2010

Regulatory principles for e-assessment

Validity and reliability

Security

Data integrity

Access to e-assessment

Business continuity and disaster recovery

Use of e-portfolios

Page 16: Regulating for quality in assessment  AAIA Conference 2010

The extent of e-marking across Awarding Bodies: 2007

36% of GCSE and A level scripts electronically marked (wide variation across awarding bodies)

Awarding Body trials found: - Reliable- Timely- Scanning and supporting technologies did not fail- Accurate assembly of results and archiving - (Senior) Examiner satisfaction- No extra admin burden on centres

No link to concerns over quality of marking

Page 17: Regulating for quality in assessment  AAIA Conference 2010

The big step

From E-marking to E-testing

Page 18: Regulating for quality in assessment  AAIA Conference 2010

Over to you

How we can help you

- Providing essential information on assessments

- Providing information on what we require from those involved in delivering assessments

How you can help us

- Give us your view on what assessments should deliver

- Help us to make assessments as fair and accurate as possible

Contact us: www.ofqual.gov.uk

Page 19: Regulating for quality in assessment  AAIA Conference 2010

Summary

Ofqual Responsibilities for assessments Reliability Standards

Our approach in action

- GCSEs and A levels

- Checking that standards are maintained

Innovation and E-assessment and on-demand testing