Assessment & Reporting with the Australian Curriculum 2012 Reporting with the NT A-E Descriptors
Writing assessment descriptors Ensuring validity and reliability in assessment Olwyn...
-
Upload
perry-shepard -
Category
Documents
-
view
218 -
download
0
Transcript of Writing assessment descriptors Ensuring validity and reliability in assessment Olwyn...
Writing assessment descriptors
Ensuring validity and reliability in assessment
Olwyn Alexander February 2015
Outline
Principles of test/assessment usefulness
Focus on validity: assessment construct
Focus on reliability: assessment criteria
Principles of test usefulness
Validity Reliability Authenticity Interactiveness Impact/Washback Practicality
Bachman & Palmer,1996
Principles of test usefulness
Validity – the test actually measures the performance it claims to measure and this is appropriate for the test purpose
Reliability – measurement is consistent between different assessors and different test takers
Authenticity – the tasks in the test are representative of tasks in the target situation
Bachman & Palmer,1996, Alexander et al., 2008
Principles of test usefulness
Interactiveness – the tasks in the test engage the test takers’ communicative language ability, background knowledge and strategic competence
Impact – the test will affect classroom teaching and learning (and wider institutional policy – entry levels)
Practicality – resources will be required to develop the test and train assessors to administer it
Bachman & Palmer,1996, Alexander et al., 2008
Test specification
Who individual/one class/all classes?
Why achievement/proficiency?
What construct/knowledge/skills?
How format/rating scales?
Bachman & Palmer,1996
Test specification
Who students in10-week pre-sessional classesat or near CEFR B2
Why to measure achievement of targetperformance in subject-specific context
What construct/knowledge/skills
How 2,000 word literature review exploring asubject-specific research question
Bachman & Palmer,1996
Test specification – construct
What is a literature review?
1. Audience/Purpose/Structure
2. Level of target discipline knowledge
3. Level of critical engagement with sources
Bachman & Palmer,1996
Literature review construct
1) Involves more than linguistic knowledge and skills
2) Requires students to engage with texts in the disciplines & struggle to understand them
3) Requires evaluation and management of information
4) Raising awareness of graduate attributes
5) Target performance described at SQA level 10: exit level of UG and entry level of PG degrees
Graduate attributes
• a sense of ‘research-mindedness’ enabling a wider more analytical perspective on individual practice
• the ability to identify problems, formulate research questions & interpret complex data to seek answers
• the ability to derive meaning from complexity and make informed judgments on the basis of evidence
• an openness to learning and positive orientation to new opportunities, ideas and ways of thinking
• a tolerance for ambiguity and unfamiliarity.
SCQF level 10 knowledge
A critical understanding of the principal theories, concepts and principles in a subject/discipline.
Detailed knowledge and understanding in one or more specialisms some of which is informed by or at the forefront of a subject/discipline.
Knowledge and understanding of the ways in which the subject/discipline is developed, including a range of established techniques of enquiry or research methodologies.
SCQF level 10 application
Execute a defined project of research, development or investigation and identify and implement relevant outcomes.
Practise in a range of professional level contexts which include a degree of unpredictability and/or specialism.
SCQF level 10 cognitive skills
Critically identify, define, conceptualise, and analyse complex/professional level problems and issues.
Critically review and consolidate knowledge, skills, practices and thinking in a subject/discipline.
Make judgements where data/information is limited or comes from a range of sources.
Test specification – construct
What is a literature review?
1. Audience/Purpose
2. Structure overall & within sections/paragraphs
3. Level of target discipline knowledge
4. Level of critical engagement with sources
Task: Choose one of the areas above and brainstorm ideas for this aspect of the construct
Audience
peers, novices (non-specialists) in the community of practice
lecturers (specialists) who assess research projects, dissertations, theses
educated non-specialists, e.g. in funding councils, NGOs or government departments who award grants or use the findings
Purpose
in-depth exploration of a specific aspect of a research area to indicate the current state of the art and suggest
new research directions OR to define and limit the scope of and provide a
context and framework for a piece of research. In-depth = reference to wide range of sources Context = current state of the art Framework = theoretical basis for research design
Structure Thematic comparison and classification of sources,
based on clear criteria usually specified in advance. Relates key papers & ideas to each other, e.g. shows
which make similar/opposite/more developed claims; which ideas stimulated by which earlier contributions
Paragraphs develop from general to specific with claims supported by evidence from relevant sources.
Review (& sections/paragraphs) structured from familiar to new; structure made explicit for the reader by summarising at the end of each section what has been discussed and how this links to what follows.
Subject knowledge
Responsibility of student (or subject expert if joint marking opportunity).
Includes recognition of key figures/papers which have moved the field on and therefore should be cited.
EAP tutors do not have this knowledge Nevertheless have to be able to read a text they do
not fully understand to assess whether it achieves its purpose for the stated audience through its structure and the level of critical engagement it demonstrates.
Critical engagement
Not simply description/summarising (SCQF, 2012) identify, define, conceptualise, & analyse
complex/ professional level problems & issues. (Bruce, 2014) evaluative judgement made within any
field of human activity about some aspect, object or behaviour of that field.
(Dodd, 2014) the different usefulness of knowledge…, appreciating how knowledge in its various forms holds potentially different value for different people in different places at different times.
Critical engagement
Not simply description/summarising (Argent, 2013) showing awareness of different
perspectives/stances (Argent, 2014) arguing cogently why some work makes a more relevant, useful or powerful contribution to the field generally or in the context of specific research.
(Spencer & Alexander, 2014) relates not to finding fault (criticising) but to criteria; evaluation must be in terms of /based on specific criteria, stated explicitly.
Test specification
How 2,000 word literature review withsubject-specific research questionSubmitted to TurnitinAssessed using customised
holistic assessment descriptors
Bachman & Palmer,1996
Assessment descriptors
Analytic (e.g. IELTS) or holistic (e.g. TOEFL)
Holistic – quicker/easier to give an impression mark
A response at this level largely achieves all of the following
A response at this level is marked by one or more of the following
A response at this level may reveal one or more of the following weaknesses
Assessment descriptors
Task: develop a set of statements which capture the aspects outlined in the construct
A response at this level largely achieves all of the following
1. Content & task achievement (audience/purpose)
2. Structure overall & use of sources & stance
3. Paragraph structure & use of language
Assessment descriptors
Task: develop sets of statements which capture varying levels of achievement
A response at this level largely achieves all of the following
A response at this level is marked by one or more of the following
A response at this level may reveal one or more of the following weaknesses
Assessment descriptors
Example: task achievement and content
Fully addresses the specific area (concept, problem, method) to be explored, showing achievements/gaps.
Addresses the specific area in sufficient depth to cover the main points, showing some achievements/gaps
The main points are covered but with some redundant ideas. Achievements/gaps are not shown clearly.
Not all aspects of the area are covered and achievements/gaps in research are not shown.