EUROPEAN STUDIES CENTREST. ANTONY’S COLLEGE, OXFORD
The Ethics of Accountability in Education Assessment
Professor Paola MatteiAssociate Professor in Comparative Social PolicyUniversity of [email protected]
Symposium: Teacher Ethics in AssessmentOxford University Centre for Educational Assessment (OUCEA) & Ofqual
26 March, 2015St Anne’s College, University of Oxford
The Wire—Louie and Jane
“ I don’t want to go to school…It’s not good…the teachers don’t know
anything, they’re mean and tired…like why is there even an
America?” (Jane)
Oxford 26 March, 2015
2
Public Accountability: why bother?
Why is accountability important?
Public Accountability and democratic theory (Waldron, 2014)
Accountability essential for effective public-private partnership
and for market-based collaboration and new tools of
government (John, 2011; Ranson, 2003; Christensen and
Laegreid, 2007)
Central elements of accountability (Mulgan, 2014; Finer, 1941;
Mattei, 2012)
Delegation of authority from accountor to accountee (agency)
Public process (transparency)
Consequences (responsiveness)
Oxford 26 March, 2015
3
Accountability as institutional mechanism
Five formally structured relationships that influence public
organization functions and performance
Political
Principal-agent delegation to elected officials
Administrative/managerial
Ministerial accountability (pre-NPM); financial auditing; top political executive
controls; accountability for performance and results (“managerial
accountability”)
Professional
Codes, standards, norms
Legal
Courts
Social
Clients, customers, interest groups through media, public panels
Oxford 26 March, 2015
4
Multiple and competing accountabilities
M
Oxford 26 March, 2015
5
Types of accountability relationships
Political Managerial Professional
Direction Clear democratic
accountability lines from
electorate to elected
politicians
Accountability to
owners/shareholders (private)
or autonomous agencies if
public.
Accountability primarily to
professional forums and peers
Logic Emphasis on broader public
good/interest
Emphasis on “value for
money”
Emphasis on medical/
educational evidence
Focus Process dimensions
(openness, involvement, due
process etc.) and politically
determined goals
Output dimensions: bottom
line, business strategy; market
based coordination
Clinical output/outcome
Source: Mattei et al, 2013; Mattei, 2012
Measured Performance Indicators:
New Accountability in Government
A new approach to public services governance in the 2000s: targets and measured performance indicators linked to negative feedback/rewards (Osborne and Gaebler, 1992; Chubb and Moe, 1990)
Was it a decisive breakthrough in governance – or a partial repeat of the some of the history of the Soviet Union (Bevan and Hood, 2006)?
Key question: how far is the world of cheating and output distortions “unethical”?
Oxford 26 March, 2015
6
Governance by league tables and targets:
a necessary evil?
Targets threshold standards that a person, organization, country is expected
to reach at a specific time
They have powerful incentive effects in organizations
They help organizations to focus on performance deficits
League TablesUse of indicators to compare the performance of different
organizations
They attract media attention
Can encourage good performers to continue
Should they be scrapped all together (as in Scotland and Wales in 2001-2002)?
Oxford 26 March, 2015
7
Performance management systems: does the
removal of league tables matter?
1979-1997 Quasi-
markets
Inspection League
Tables
Targets
England V V V X
Wales V V V X
Scotland X X X X
1997-2009 Quasi-
markets
Inspection League
Tables
Targets
England V V V V
Wales V V X( abolished
in 2001)
V
Scotland X V X (in 2002) V
England, Wales and Scotland: Pupils with 5+ A* to C
GCSEs and SCQF at Level 5 (Mattei, 2012)
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
1993 1995 1997 1999 2001 2003 2005 2007
Wales England Scotland
Assessment as an accountability tool
1960s: “secret garden of the curriculum” (David Eccles,
Conservative Secretary for Education, cited in Timmins, 1995)
Late 1980s onwards: accountability through measured
assessment and performance indicators (Baird, 2014)
2000s: output targets (based on exam results) and new
performance management systems (Lawn, 2014; Mattei, 2012) to
address underperforming schools
Oxford 26 March, 2015
10
Performativity accountability regimes and controversies
Pro-reform claims External public scrutiny of the teaching
profession through measurable outcomes
Information available to parents to bring
the sanction of ‘exit’ (market
accountability)
Learning and professional self corrective
measures
Minimize teacher assessments and
stereotyping against ethnic minorities
(Burgess and Greaves, 2009)
Anti-reform claims Narrowing of the curriculum
Teaching to the tests and negative
curriculum reallocation
Gaming and cheating by teachers
Cream skimming (entry selection)
Schools give up on low performing
students and focus on those on the
margins
Need for broader indicators (child
wellbeing)
Oxford 26 March, 2015
11
Some problems with targets and PIs
Threshold effect incentive to concentrate on meeting the minimum target
e.g. teachers concentrate on narrow band of students on the margins to achieve targets
Output distortion incentive to those subject by targets to concentrate on
achieving success at the expense of other factors which are not measured by the target
e.g. teaching to the test (at the expense of sports, arts)
Oxford 26 March, 2015
12
Output distortions and cultural conditions
Role of culture (Mary Douglas, 1966; Christopher Hood, 2000; Dan Kahan, 2006) in policy making and administrative process.
Design performance indicators sensitive to organisational culture. Why?
How individuals and groups respond to measures depends on four
types of control systems/cultures
Strength of distortions
Ofqual survey questionnaire on acceptability of cheating behaviour is a measure of “culture”
Oxford 26 March, 2015
13
Grid-Group cultural theory and four control systems
Oxford 26 March, 2015
14
Fatalism
Low group cohesion,
apathy, sense of disorder
and distrust
Hierarchy
Oversight through within a
hierarchy characterised by
strong regulation and rule-
bound institutions
Individualism
Privileges markets,
unbridled
entrepreneurialism
competition, and
deregulation
Egalitarianism
Mutuality, solidarity,
communal governance,
participative decision-
making
GRID
GROUP
Types of actors/motivation (Hood 2007)
1)‘Saints’: who may not share mainstream goals, but whose public
service ethos is so high that they voluntarily disclose shortcomings to
central authorities
2) ‘Honest triers’: who broadly share mainstream goals and do not
voluntarily draw attention to their failures, but do not attempt to spin or
fiddle data in their favour
3)‘Reactive gamers’: who broadly share mainstream goals, but aim to
spin or fiddle data if they have a motive or opportunity to do so.
4)‘Rational maniacs’: who do not share mainstream goals and aim to
manipulate data to conceal their operations (gross misconduct)
Oxford 26 March, 2015
15
Illustration from another field: gaming the transplant system in the United States
For a patient in need of an organ transplant, life is a waiting game!
Low Group Low Grid—Individualist control system and cultural conditions
Majority of hospitals: private non-profits
78% of all medical procedures performed on a fee-for-service basis (incentives
volume targets and rewards)
Most remunerated procedure (transplant), highest DRGs for hospitals
High transplant prices – average liver transplant now more than $577,000 and
bonuses for surgeons
Policy challenge: do market pressures combined with high prices
transplant surgeons command, and financial incentives and bonus
rewards encourage unethical cheating behaviour?
Oxford 26 March, 2015
16
Cases of cheating by surgeons: control
systems or motivation?
2002: University of Illinois teaching hospital
Federal lawsuits against three centres in Chicago ($2mil to settle the lawsuit)
Misleading medical information and inappropriate hospitalisation
Admission to Intensive Care Unit to jump the transplant waiting list in the
region (Department of Justice, 2003)
MELD introduced in 2002 (new medical standards)
sharp decrease in admissions to ICUs (Snyder, 2010)
Largest decrease in ICU admissions in markets with highest provider
competition (controlling for surgeons’ rewards system)
Findings: cheating associated with type of target (volume
expansion, 12 procedures per year) and type of reward (bonus)
Oxford 26 March, 2015
17
Public trust (O’Neill, 2011) and morality in public policy
Public policy discourse of efficiency/accountability is now
increasingly tied to morality and trust (Simpson and Baird, 2013)
Malpractices in public services may diminish public trust
In live organ transplant (scarce supply), this is detrimental-- drop in
organ donation has severe implications for patients
Strengthening “defensive medicine” and “defensive teaching”
Erosion of professional ethos and demoralised professions
Oxford 26 March, 2015
18
Future research and policy challenges:
“Ethics and Gaming in Education”
Is cheating in the education system caused by regulatory failure or individual human action?
What is the relationship between cultural conditions (Grid-Group theory) and educational assessment?
Do some control systems deliver a better game-proof design?
What are the potential effects of changing administrative values and cultures (e.g. increase relational distance between teachers and regulators) on the strength of output distortions?
Oxford 26 March, 2015
19
Why ethics and accountability?
Move beyond institutional mechanisms and formal arrangements in the field of accountability
Understanding accountability through human relationships and social
interactions (Bovens, 2002; Dubnick, 2006)
Multiple diverse conflicting expectations (MDCE) from policy makers who face dilemmas and make choices among opposing values (Dubnick and Romzek, 1993; Mattei, 2015)
Accountability is intrinsically an ethical question.
Prestige management increasingly significant with PIs governance
Achieve moral status in the eyes of others (not only about bonuses)
Avoid moral blame that might result from wrongdoing
Oxford 26 March, 2015
20
Nozick’s ethical theory and public policy:
moral pushes and pulls
Action as outcome of tensions between moral push and moral pull
Moral push—my own values
internal motivation, individual, that determines own moral conduct
based on self-worth
Internalisation of values
Moral pull—the others’ values
external, institutional values and demands, conduct based on the
others’ values (structures and procedures)
Moral pull of ‘A’ puts a moral constraint on ‘B’ and determines the
behaviour of ‘B’ in accordance to ‘A’ values
When does ethical action occur?
Moral push is equal or greater than the moral pull
Oxford 26 March, 2015
21
Conclusion
Oxford 26 March, 2015
22
If
production
and
economic
values
Then, conflict
between compliance and fidelity
Top Related