Assessment re-capitulated and questioned

Assessment Recap…

… as ever with added complications!

James AthertonUnit 3: 8 April 2014

The blog page to which some

notes refer is at:

http://pce2013.blogspot.co.uk/2014/04/8-april-back-to-assessment.html

Why?

Requirements

Approaches

Assessment

08/04/2014 - v2

Feedback(Formative)

Accreditation(Summative)

Validity

Reliability

Fairness

(Security)

Norm-referenced

Criterion-referenced

Ipsative

•“Assessment is central because it frames student learning”

(Oxford Brookes U. Position Paper on Assessment, n.d.)

•“From our students’ point of view, assessment always defines the actual curriculum”

(Ramsden P,1992)




(Ramsden P,1992)

Not for nothing do students

invariably cry, “Are we being assessed on

this?”




(Ramsden P,1992)

Not for nothing do students

invariably cry, “Are we being assessed on

this?”

And if the answer is “No”, then the

learning is automatically

worthless

It is assessment which turns;

• presenting

• story-telling

• demonstrating

• …

into teaching

• Assessment of learning

• (summative)

• Assessment for learning

• (formative)

Which is privileged?


• (summative)


• (formative)


Accreditation and

qualifications—important but not the whole

story


• (summative)


• (formative)


Accreditation and


storyImportance

increasingly recognised in the

rhetoric of education, if not the practice…


• (summative)


• (formative)


Accreditation and


storyImportance

increasingly recognised in the

rhetoric of education, if not the practice…

See link in notes

Formative

Summative

but…

Formative

Summative

but…All assessment,

including most of what is designed for summative

purposes, can be used formatively

Formative

Summative

but…All assessment,

including most of what is designed for summative

purposes, can be used

formatively…

…but only if you can get the students to

take their work back and read the comments!

PresenterPresenter AudienceLearner

Information

Feedback

TeacherTeacher/

learner

Learner/

teacher

Experiential environment

Theoretical representation

Goal-oriented behaviour

Conceptual representation

Teacher Student

Articulation/Re-articulation

Reflection/Adaptation

Reflection/Adaptation

Action/Feedback

Based on Laurillard, 2001

This is Diana Laurillard’s elaboration of the

previous graphic—link in the notes

Effect-size 2

• An effect-size of 1.0 indicates an increase of one standard deviation, typically associated with advancing [students’] achievement by one year, improving the rate of learning by 50%, or a correlation between some variable (e.g., amount of homework) and achievement of approximately .50. When

implementing a new program, an effect-size of 1.0 would mean that approximately 95% of outcomes positively enhance achievement, or average students receiving that treatment would exceed 84% of students not receiving that treatment.

Hattie, 1999:4 (my emphasis)

Just a reminder of Hattie—now familiar territory, with a link

in the notes.

Interventions and Effect Size

0

50

100

150

200

250

300

350

400

Effect Size

No. of

stu

die

s

Above average

effectsize

Influence Effect Size

•Feedback 1.13• Students' prior cognitive ability

1.04• Instructional quality 1.00• Direct instruction .82• Remediation/feedback .65• Students' disposition to learn .61• Class environment .56• Challenge of Goals .52• Peer tutoring .50• Mastery learning .50• Homework .43• Teacher Style .42• Questioning .41

•Feed-back, or

•Feed-forward ?Phil Race is the

principal advocate for feed-forward. Link in notes

Why?

Requirements

Approaches

Assessment

08/04/2014 - v2

Feedback(Formative)


Validity

Reliability

Fairness

(Security)

Norm-referenced


Ipsative

Requirements…

•Validity: the assessment measures what it is supposed to.

•What mismatches have you come across?

• (i.e. between desired outcomes and methods of assessment)

The boundaryis constantly

moving

Common observation suggests the belief is unfounded; we seldom find hard evidence of such correlations. Instead, we rely on test results for want of anything better. In any event, the skills required to perform well on school examinations may not be the same skills required to perform adequately in the situations the school trains people for.

(Becker, 1972)

We seldom argue that conventional tests measure the actual skills students are supposed to have learned. Rather, we believe that the test, … is … highly correlated with the ability to exercise those skills, though the mechanism … is seldom investigated or demonstrated.

Link to full paper in the notes

Reference points for Assessment

•Competitive (“Norm-referenced”)

See link in notes

Norm-Referenced Assessment

0

5

10

15

20

25

0 35 45 55 65 75 85 95 105

115

125

135

145

155

165

% o

f c

oh

ort

Mean

This curve is based on a real reference curve used for the standardisation of IQ tests, which is why it is not quite as graceful as a classic “bell-curve”. See

link in notes.

•When might it be appropriate to use norm-referenced assessment in a PCE setting?

Reference points for Assessment (2)

•Competitive (“Norm-referenced”)

•Competence (Criterion-referenced)[and

•Personal best (Ipsative)]

Criterion-Referencing

Threshold of competence

Object is to ensure minimum level of competence attained

One problem is whether that threshold effectively sets a

cap on the level of achievement: we briefly

discussed the question of inability to deal with

exceptional genius in the arts.

A complication of “competence”

Threshold

of Competence

Competence is as much a matter ofconsistency as of skill — what level of consistency is “competent”?

Answer

Facts Stimulus

IdeaConvergent thinking

- science and technology(?)

Divergent thinking- arts and humanities(?)

See link in notes

•How might you define achievement of intended outcomes in a divergent subject area?

Requirements (2)…


•Reliability: it produces consistent and replicable results Between judges Between occasions Between cohorts

Why?

Requirements

Approaches

Assessment

08/04/2014 - v2

Feedback(Formative)


Validity

Reliability

Fairness

(Security)

Norm-referenced


Ipsative

• "Is an indistinct photograph a picture of a person at all? Is it even always an advantage to replace a blurred picture with a sharp one? Isn't the blurred one often exactly what we need?" Wittgenstein (1953) para. 71

• “OK, you’ve got your PhD; now don’t touch anything!” (source unknown)

• “I claimed at the start that I would use the term 'ambiguity' to mean anything I liked,” (Empson, 1947; viii)16

Importance of clear criteria?

Criteriaa. Understanding and familiarity with the subjectb. Breadth of readingc. Relevance of content to essay titled. Structure of the argumente. Independence of thought and critical evaluationf. Presentation, including fluency of writing, grammar

and spellingg. Citation and referencing

How would you weight these?

(in relation to an HE1/NQF 4 task in a “soft” discipline)

20.0%20.0%13.3%13.3%20.0%

So, proportion of marks for each criterion

33223

Weighting of marks allocated to each criterion

Without necessarily being highly original, the work shows consistent critical evaluation of sources and ideas, and the author's own considered judgments

The argument is pursued clearly and in a balanced manner, supported by evidence and informed judgement

Essay addresses title and its underlying themes and assumptions, clearly explicated.

Evidence of extensive reading beyond the recommended texts, including critical evaluation of sources

Ability to contextualise subject within wider frameworks and critically explore connections

Critical evaluation and independent thought are the norm, but are either not consistent or coherent throughout.

A clear and generally structured argument let down by some unsound inferences or assertions without evidence

Essay addresses title in a focused and organised way with some awareness of any more general issues raised by it.

Recommended reading used, together with a range of other sources

Familiarity with subject demonstrated, including complex connections between elements

Evidence of occasional independent and critical thought, limited by either inconsistent or incoherent reasoning

Continuity of argument evident throughout the essay supported by evidence

Content clearly addresses set title as given.

Effective use made of reading, largely confined to reading list

Subject matter adequately understood, with some connections made

Critical evaluation limited to expressions of opinion, using mainly second-hand ideas

Argument patchy, inconsistent, or proceeding by assertion rather than evidence.

Essay generally addresses title, but with lapses, digressions or padding.

Limited use made of texts on set reading list

Evidence of knowledge, but only basic connections made between items

Uncritical and unreflective work, relying on undigested sources.

Little clear structure or argument evident.

Little or tenuous connection between title and essay content

Little or no reading evident, even within the set texts

Subject matter treated as set of discrete bits of knowledge

Independence of thought and critical evaluation

Structure of the argument

Relevance of content to essay title

Breadth of readingUnderstanding and familiarity with the subject

Criterion ECriterion DCriterion CCriterion BCriterion A

1Academic levelEducation, Individual and SocietyEduc 1002

5

4

3

2

1

More (including download) via link in

notes.

Backgroundof topic

Backgroundof topic

Knowledgeof topic

Backgroundof topic

Knowledgeof topic

Researchskills

Backgroundof topic

Knowledgeof topic

Ability toconduct argument

Researchskills

Backgroundof topic

Knowledgeof topic

Ability toconduct argument

Researchskills

“Hard” and “soft” objectives are

discussed in the notes.

Too clear and specific?Task-2

An annotated diagram which illustrates how you contribute to the Quality Assurance and Quality Improvement Processes, procedures and initiatives in the area of work you are engaged in. A short report evaluating and reflecting upon this contribution. Extended diagram(s) showing how quality framework systems work within

own teaching area making links to validity and reliability of the data collected.

Comprehensive annotations including links to areas for improvements and research.

Excellent critical analysis of the current issues relating to quality assurance and quality improvement processes in own area of teaching.

Sound application of theory to practice in order to analyse and evaluate the success of the quality procedures (both positive and negative) Applied theory to practice to identify own CPD.

Critical evaluation of the long term impact of QA and QI systems and procedures.

Detailed bibliography with substantial use of journal research sources. Quotes are presented and fully engaged.

Sound structure. Few errors of presentation.










This is an actual rubric from a supposedly

Master’s level course.










If you get this detailed and

prescriptive, can you possibly hit the level

of critical scholarship

expected of a graduate level

course?










If you get this detailed and

prescriptive, can you possibly hit the level

of critical scholarship

expected of a graduate level

course?

Despite fatuous usage of the

currently fashionable phrases.

Requirements (4)…


•Reliability: it produces consistent and replicable results

•Fairness

•Security

Requirements (4)…


•Reliability: it produces consistent and replicable results

•Fairness

•Security

What does “inclusivity” mean in

the context of assessment?

Why?

Requirements

Approaches

Assessment

08/04/2014 - v2

Feedback(Formative)


Validity

Reliability

Fairness

(Security)

Norm-referenced


Ipsative

Assessment re-capitulated and questioned

Education

Transcript of Assessment re-capitulated and questioned