Using Quality Management (QM) to Improve Language Testing
-
Upload
the-international-research-foundation-for-english-language-education-tirf -
Category
Business
-
view
1.392 -
download
0
description
Transcript of Using Quality Management (QM) to Improve Language Testing
![Page 1: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/1.jpg)
Using Quality Management (QM) to improve language testing
Nick Saville and Michael MilanovicLTRC 2011
![Page 2: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/2.jpg)
Outline – see handout
• Introduction - QMS
• Historical perspectives and definition of QM
• Key points in a QMS approach:
• People
• Processes
• Quality Control (QC) and Quality Assurance (QA) in
practice
• Illustrations of QC and QA in operational systems
• Conclusion - linking quality and validation
![Page 3: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/3.jpg)
Quality management system
“Getting it right every time”
– Test developers need to adopt managerial practices which
enable their organisation to implement error-free processes
QMS provides a basis for this
– processes continually improved and standards raised
– in keeping with validation as an ongoing activity
Key point - take home message• convergence between the twin concepts of quality and validity
![Page 4: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/4.jpg)
Definition of QMS
Distinction is made between
a) quality management (QM) - overarching concept
b) quality control and quality assurance (QC and QA) -
procedures for checking and assuring quality
QM concerned with the management of
processes which lead to improvements being
implemented
![Page 5: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/5.jpg)
Definition of QMS – meeting needsIn QM, meeting client needs is an important consideration
In language testing, clients are the test users:
– Primary users: test takers
– Secondary users: sponsors of test takers or users of
results to make decisions
See the Code of Fair Testing Practices (1988)
Testing organisations need to guarantee "fitness for purpose”:
to meet client needs for assessment in specific contexts of use, such as the
workplace
![Page 6: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/6.jpg)
Historical perspective - origins
QM - origins in the manufacturing sector
• Now applied across many types of organisation
• Only recently within educational systems
– E.g. Wild and Ramaswamy (eds), 2008
Improving Testing: Applying Process Tools and Techniques to
Assure Quality
![Page 7: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/7.jpg)
Historical perspective - founders
Founders - Taylor (1911), Shewhart (1931, 1939)
Shift from maintaining standards to improving standards
• Continual improvement axiomatic within all QM systems
– processes defined in terms of inputs and outputs
– links managed effectively between different parts of an
organisation
– reduce defective items before the final stage of a process
– cut out high cost of post hoc inspections to check on quality
![Page 8: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/8.jpg)
Historical perspective - TQM
W. Edwards Deming (1900-1993)
• leading figure in the movement
• first coherent "quality management systems“
(QMS)
• Total Quality Management - TQM
• Deming, 1986: Out of the Crisis
![Page 9: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/9.jpg)
Historical perspective - Quality standards
A mechanism of accountability
• dates back over 100 years
• British Standards Institution (BSI), London, 1901
• original quality mark 1903
BSI “kite mark”
![Page 10: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/10.jpg)
Historical perspective – ISO standards
International Organization for Standardization (ISO)
Quality Standards - ISO 9001: 2008 accreditation
Based on auditing
Certifies that processes are being applied consistently and effectively to meet stated objectives of an organisation
![Page 11: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/11.jpg)
Two key points in a QM approach
People • the importance of leadership and staff development
In our field this includes specific training in assessment literacy
Processes
• the importance of defining processes and managing
them effectively
![Page 12: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/12.jpg)
Key points - People
A key asset in achieving error-free processes:
• Managers ensure that staff are empowered to carry out
their responsibilities
• QC & QA procedures ensure all involved contribute to
quality
• Importance of:
– Expertise
– Training
– Ongoing professional development
![Page 13: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/13.jpg)
Key points - Processes
Core processes - the assessment cycle
– Planning and design
– Development - e.g. of systems for test assembly
– Delivery - routine test assembly and administration
– Processing – marking, grading, issue of results
– Review and evaluation
![Page 14: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/14.jpg)
Key points - process interactions map
Figure 1 inputs and outputs
![Page 15: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/15.jpg)
Key points – Plan, Do, Check, Act
• Mistakes, inaccuracies, human error, malpractice, cheating, etc. – all threaten validity
• Iterative processes provide feedback
–to evaluate effectiveness
–to correct errors
–to make improvements in subsequent iterations
![Page 16: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/16.jpg)
Key points - Figure 2
time
Operational System
Development
developing the assessment
decision to provide an assessment
reporting results
assembling tests
administering tests
marking tests
grading tests
eevviiddeennccee
ee vv ii dd ee nn cc ee
aim: produce test specifications
aim: produce test materials
aim: provide test takers and other stakeholders with the results and any other information needed for an appropriate interpretation
aim: put the performance of each test taker into a meaningful category so the test result can be understood
aim: provide an accurate and reliable score for each test taker
aim: collect information about each test taker’s ability
Aims,
Inputs,
Outputs
![Page 17: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/17.jpg)
QC and QA in practice – check, act
• QC synonymous with checking• systematic checks ensure that all aspects of a “work flow” meet the
standards as defined and described in documentation for each stage of the overall process
• checks are carried out by the people who have the responsibility for doing the tasks (not by a separate "quality control department")
• QA carried out to monitor, evaluate and improve• to ensure that all processes are maintained to the required standard
• to monitor and adjust processes within the system
• to evaluate whether core processes are working adequately
![Page 18: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/18.jpg)
QC in practice (1)Illustration 1 - Assembling test materials
• production of test specifications at the development stage and instructions for item writers for assembling tests - essential prerequisite
• checking against specifications - at various stages of the
operational cycle before the assessment is finalised and ready to be administered
• system to collect, store and process test materials • particularly important if a large number of items and tasks are involved
e.g. may require an item banking system
![Page 19: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/19.jpg)
QC in practice (1)Illustration 1 - Assembling test materials
A QC system for managing test materials might include:
• an ID and description of each task or set of materials
• a checklist to track progress and to provide a complete record of the stages completed, changes made and information about decisions taken
• test materials which are stored and are readily accessible with relevant documentation and management information accrued during the editing process
• rules for confidentiality and security in handling materials
![Page 20: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/20.jpg)
QC in practice (2)
• Illustration 2 - Administering tests
• in large-scale operations the administration of the assessment will be delegated to personnel at the testing venue
• the test developer needs to be confident that assessments are administered in a standardised way
Consistency is important
Uncontrolled variation undermines validity
• administrative procedures need to be clearly and comprehensively described and produced in a format which can be used under operational conditions:
![Page 21: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/21.jpg)
QC in practice (2)QC should include:• physical setting
• storage and handling of secure and confidential materials/information
• type and number of personnel needed to oversee the administration• to ensure safety and guarantee test security
• procedures for recruitment, training and management of invigilators /proctors and examiners
• management of the interaction between test takers and administrators• the checking of their identities and eligibility, seating arrangements, provision of
accommodations to meet special needs or requirements
• management of the assessment procedures themselves • providing instructions, handing out and collecting test papers and answer sheets,
monitoring for malpractice (including all kinds of cheating), ensuring that timings are respected
• handling of unforeseen eventualities specific circumstances • illness, fire alarms, power cuts, minor accidents, disruption from outside the venue
![Page 22: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/22.jpg)
• Failure to maintain quality of any aspect of the system threatens validity
• E.g. Undetected problems may lead to construct-irrelevant variance
• This may affect all test takers, groups of test takers, or individuals and lead to unfair outcomes
see Kunnan on Fairness, (2000; 2004)
Linking quality and validation
![Page 23: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/23.jpg)
Linking quality and validation
• Validation is the process of accumulating evidence to
support inferences made using test results
• Synergy between test validation and QM concept of
continual improvement
• Accrual of evidence at all stages in the cycle
Figure 3
![Page 24: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/24.jpg)
Linking quality and validation
Operational system
Development
developing the assessment
decision to provide an assessment
reporting results
producing test materials
administering tests
marking tests
grading tests
periodic review
ee vv ii dd ee nn cc ee
time
eevviiddeennccee
Figure 3 Assessment cycle showing periodic review
Follow the dotted red line _ _ _
![Page 25: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/25.jpg)
Conclusion - putting principles into practice
QMS can help
to bring about change and improvement in well-managed and principled ways
to connect good practice in language assessment with operational procedures which are both transparent and accountable
ISO 9001 in conjunction with an international Code of Practice provides the basis for error-free processes and achievement of auditable professional standards
![Page 26: Using Quality Management (QM) to Improve Language Testing](https://reader034.fdocuments.in/reader034/viewer/2022051514/54b463464a7959c0418b45d8/html5/thumbnails/26.jpg)
References – on handoutCouncil of Europe/ALTE (2011) Manual for language test development and examining. For use with the CEFR, Strasbourg: Language
Policy Division, available online www.coe.int/t/dg4/linguistic/ManualtLangageTest-Alte2011_EN.pdf
Deming, W. E. (1986). Out of the Crisis. Cambridge: Cambridge University Press.
Downing, S.M. (2006).Twelve Steps for Effective Test Development. In Downing, S.M.; Haladyna, T.M. 2006.
Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment – an advanced resource book, Abingdon: Routledge.
Hatch, M. J. with Cunliffe, A. L. (2006). Organization Theory: modern, symbolic and post-modern perspectives. (2nd edition). Oxford: Oxford University Press.
Kane, M (2006). Validation. In R. L. Brennan (Ed). Educational measurement (4th edition). Washington, DC: American Council on Education/Praeger.
Kemp, S. (2006). Quality Management Demystified. New York, McGraw Hill.
Kuijper, H. (2003). QMS as a Continuous Process Of Self-Evaluation and Quality Improvement for Testing Bodies. www.alte.org/qa/index.php
Kunnan, A. J. (2004). Test Fairness, in M. Milanovic and C. Weir (Eds). Studies in Language Testing 18:
Rose (2010) Research Notes, 39 (2-7). Downloadable from: www.cambridgeesol.org/rs_notes/offprints
Saville, N. (2005).Setting and monitoring professional standards: A QMS approach. Research Notes 22, 2-5. Cambridge: Cambridge ESOL.
Saville, N. (2010). Auditing the quality profile: From code of practice to standards. Research Notes, 39, 24-28. Cambridge: Cambridge ESOL.
Shewhart, W.A. (1931). Economic control of quality of manufactured product. New York: D. Van Nostrand Company.
Shewhart, W.A. (1939). Statistical method from the viewpoint of quality control. Washington, The Graduate School, the Department of Agriculture.
Taylor F.W. (1911). The Principles of Scientific Management. New York: Harper and Brothers Publishers,.
van Avermaet, P. (2003). QMS and The Setting of Minimum Standards: Issues of Contextualisation Variation between The Testing Bodies. Retrieved from www.alte.org
van Avermaet, P., Kuijper, H; and Saville, N. (2004). A Code of Practice and Quality Management System for International Language Examinations, Language Assessment Quarterly 1(2&3),137-150.
Wild, C.L. and Ramaswamy, R. (2008). Improving testing. Applying Process Tools and Techniques to Assure Quality. London: Routledge.