Making the best of

unanticipated impacts:

boundary work, statistical

entrepreneurship, and

‘opening up’ indicators

Markku Lehtonen

Ecole des Hautes Etudes en Sciences Sociales, Paris, France

University of Sussex, Brighton, UK

m.lehtonen@sussex.ac.uk

Milan, 22-23 March 2018

Need for disruption – and stability

Direct, intended use or indirect, unanticipated influence

of indicators?

Need for SDIs to move beyond BAU: disruptive

indicators?

But how to disrupt while maintaining/strengthening trust

in official statistics and indicators?

Post-truth, Big Data revolution

“Opening up” and statistical entrepreneurship – to

combine stability and flexibility?

Disruptive indicators? Need for sustainability transitions – need for new,

transformative SD indicators?

Disruptive innovation, destabilisation, lock-in &

unlocking, co-evolution and cumulative causation…

Transitions: destabilisation of the dominant socio-

technical regime

Beyond GDP indicators as an attempt to destabilise the

prevailing regime

Multi-level perspective of socio-

technical transitions

Source: Geels & Schot (2007).

Socio-tecnical

regime of SD

indicators?

Big Data

Beyond GDP

Post-truth

Beyond

GDP

The weak transformative capacity of

Beyond GDP indicators (Malay, forthcoming)

Many/most Beyond GDP indicators reproduce the same

picture and country rankings as GDP – and hence reinforce

the status quo

Possible explanations

data quality requirements – MRV value (measurable, reportable

and verifiable)

Institutional inertia and political interests

Epistemic communities: shared common values, beliefs,

frameworks of thought, policy endeavour, criteria for what is a

good indicator, professional identities – beliefs concerning the

role of indicators in policy

The “law” of inherent conservatism in official statistics (van

Tuinen 2009)

Role of indicators in

policy:

intendend use vs.

unanticipated influence

Indicators

Indicator

design

process

Other

policies Targeted

policy

Society at large;

democracy, etc.

Use

Influence

No use No influence

Theoretical pathway

But this is a simplification,

because it does not take into

account co-evolution between

policy, indicators, technologies,

cultures...

Therefore…

Indicators as boundary

objects

Boundary objects

Mediate between different

categories of policy actors, or

“social worlds”

Facilitate dialogue, learning and

consensus-building across

policy boundaries:

• experts and non-experts

• state and non-state actors

• various governance levels

• “science”, “policy”, and “society”

• indicator expert communities

Boundary • Boundary = a shared

space, where the

meanings of ‘here’ and

‘there’ are confounded

Boundary objects

Flexible, vague and

ambiguous enough to cater to

different worldviews,

perspectives, cognitive

assumptions and frameworks,

while retaining sufficient

stability to…

…allow groups to work

together without consensus,

in a process of tacking back

and forth between the vague

and the more tailor-made

forms of the object

From boundary objects to boundary

infrastructure

Regimes and networks of boundary objects

National and international statistical institutions seek to:

control the tacking back-and-forth

standardise and make equivalent the ill-structured and

well-structured aspects of indicators

appropriate both the calculus and publication of the

indicators

These processes are essential for trust in statistics,

indicators and statistics authorities

But they also generate inertia that prevents innovation and

disruption

How to combine stability and

disruption, consensus and

controversy?

Opening up and statistical

entrepreneurship to the

rescue?

Opening up

Opening up of appraisal and

indicators

Breadth of inputs

Disciplines

Perspectives

Stakeholders

Expert groups

Openness of outputs

Alternative indicators

indicating the range of

possible outcomes

Enable rigorous debate on

contrasting policy options

Indicators as boundary objects;

“opening up”

Multiple data sources

Unique indicator value

Multiple data sources

Range of possibly contrasting indicators

& indicator values

Single source of data

Unique indicator value

Single source of data

Range of possibly contrasting indicators

& indicator values

Inp

uts

Outputs

Underw

ay

Needed

Needed

“Statistical entrepreneurship” (van

Tuinen 2009)

Towards a double identity of statistical offices:

1. Provider of authoritative and incontestable facts

2. Innovative and respected research organisation: indicators

that highlight complexities and uncertainties behind statistics,

and explore their practical significance

Change of culture within statistical offices:

“put in contact the citizens with official statistics, to make data

accessible, to expand the understanding of their analysis, to

support individuals, business and institutions in the decision-

making process” (Baldacci & Pelagalli 2017)

Post-truth, Big Data, opening up

and statistical entrepreneurship

Multiplicity of data sources: competition!

Big Data and the threat of private-sector data oligopolies

The traditional model of authority is no longer operational: need to innovate, engage with data users and the data communication technologies

Foundations of trust in statistics/indicators: are they the same in all countries? Virtues of mistrust?

Liberal democracy founded on mistrust

Nordic democracies built on high trust in state institutions

New (?) responsibilities for statistical

offices and statisticians (I)

1. Assist stakeholder groups with issues such as data quality; limitations of simplification; unanticipated and feedback effects of indicators; uncertainty; trade-offs; ambiguities; and roles of indicators, assessments, and evaluation

2. Diversify the range of indicator products Explore overlaps between different indicator sets (and develop

new ones) representing competing normative and cognitive perspectives

Relax the conventional indicator quality criteria?

3. Explore and characterise uncertainties with the help of tools such as the NUSAP “pedigree” framework (Funtowicz and Ravetz 1989)

New (?) responsibilities for statistical

offices and statisticians (II)

4. Explore and explicitly address problems & opportunities of

“performative” and feedback effects

shortcomings of commensuration; principle of incommensurability

benefits of feedback and performative effects: alternative

measures of wellbeing

5. Speed up innovation to “open up”

available data => new indicators that satisfy current needs to a

reasonable degree

dedicated R&D funding to produce “opening up” indicators

6. Ensure support and commitment to innovation from the

highest governmental level (e.g. budget and PM’s offices)

Declining trust in statistical institutions

“in France and in the UK, only one third of citizens trust official figures, and these countries are not exceptions” Stiglitz et al. (2010)

“…[i]n the UK, people don’t trust any indicators” (UK energy-sector official)

(Ambiguous) identities of statisticians and statistical institutions

custodians of objective, independent, incontestable data in defence of democracy and the underprivileged

“feedback” and performative effects from statistics/indicators: quantified data “change the world through their very existence, their circulation and their rhetorical usage in science, politics or journalism”

Institutionalisation of quantification and codification procedures to protect against the “politicisation” of indicators

Sources of trust and authority: cultural

differences?

1. Anglo-American liberal democracies

Mistrust as the basis of democracy

2. Nordic high-trust societies

High institutional trust as the basis of democracy

3. Central-Western European corporatist democracies

Combination of mistrust and trust?

Mistrust of authority, but trust in public service (FRA)

Consequences for the work of statistical authorities, for

statistical entrepreneurship?

The “law” of inherent

conservatism of

statistical offices (van

Tuinen 2009)

• Stressing shortcomings

and uncertainties,

developing alternative

visions would be at odds

with the core mission of

statistical offices

• “Client” groups of

statistical offices ask for

improvement of their pet

indicators, not new

indicators

• Sunk costs of the

institutionalisation of

indicators

• Conservatism of the

scientific world: old

paradigms prevail

Opening up, Big Data,

and post-truth: three

scenarios for stats

offices

Business as Usual decentralised information markets with strong segmentation new information intermediaries emerging through social platforms, digital

opinion leaders progressive marginalisation of stats offices in the public debate, Competition from multiple information sources, including non-certified

empirical evidence, pressure on the official statistics position in the information ecosystem

drastic reduction of public resources invested in official statistics, as a result of the perceived lack of relevance

public image: official statisticians challenged by alternative facts and figures which contradict statistical information

traditional communication strategies focused on reactive interventions in public debates about the correct figures in public policy issues

main communication priorities: reaching out different users with a mix a general and more tailored statistical products and providing clarity on definitions and methodologies used

continuing sliding into marginality of official statistics in public opinion’s perception.

Worst-case scenario massive restructuring of the information markets – from

decentralisation to oligopoly

market concentration of information platforms thanks to the availability of massive sources of data and the powerful engines to process them

Big Data managed and manageable by only big players Algorithms-driven information market

Profiling of information-consumers, who do not participate in the production of data

Little competition, little societal control of data and information processes

Statistical institutions seen as outdated, “dinosaurs” from the past

The good scenario sound regulations maintain a competitive information market

by preventing the emergence of dominant positions within and between countries

Modernisation of official statistics: the production of smart statistics while maintaining and enhancing the quality of the data provided

Stats offices use new technologies to provide tailor-made information to key user groups

Proactive efforts to help citizens better understand strengths and limitations of statistics/indicators

Certification of other information producers

Scientific literacy work: ensure that information provision encourages curiosity, highlights uncertainty, and provides answers to questions

Boundary work

Collaborative: the object retains its

vague, shared identity and meaning

• e.g. cross-country comparability

Local, tailored, disciplinary: locally

tailored identities and meanings

• e.g. adaptation of an indicator to

local/national policy needs

Boundary objects allow groups to work together without

consensus, in a process of tacking back and forth between

the vague and the more tailor-made forms of the object

The dilemma of statistical

institutions Professional identities and authority of statistical offices

relies on their ability to provide reliable, incontestable high-quality data, but

Indicators and statistics unavoidably produce unanticipated effects, including notably feedback effects on the subjects of the indicators – people change their behaviour in response to the indicators

Should statisticians take the feedback effects into account?

Cf. statisticians vs. accountants – distinct professional identities and views on the feedback effects

Use ≠ influence

Process Indicators

Argumentation &

dialogue Persuasion, Legitimisation

Critique, Defence

INFLUENCE

Intended

policy

Other

policies

Non-policy

impacts

Decisions and actions

New shared understandings (learning, framing, agenda-set.)

Networking

Legitimacy (internal, external)

Instrumental/

Rational-positivist

Conceptual/

Discursive-

interpretative

Political/

Strategic

ROLE OF KNOWLEDGE/

CONCEPTION OF POLICYMAKING