Public participation through citizen science – A critical reflection on the challenges for evaluation and monitoring

New Challenges for the Evaluation of Science and Research: Political Goals, Theory and Methodological Challenges Vienna, 11th of June 2015 Dr Larissa Talmon-Gros, Dr Thomas Teichler – Technopolis Group

Citizen Science can be defined as...

2

..„scientific work undertaken by members of the general public, often in collaboration with or under the direction of professional scientists and scientific institutions“1

1 Oxford English Dictionary

Characteristics of CiSi projects

Various types of activities

Application in all stages of the research process possible

Diverse actor base Intensity of citizen

involvement

New technologies as enablers Different Objectives

Two empirical examples illustrate the range of of CiSi projects.

3 Quellen: www.igel-in-bayern-br.de http://sensebox.uni-muenster.de/

In practice, Citizen Science can take very several forms, with different qualities of interaction.

4

Type of project Intensity of interaction

Objective

Passive sensors

Computing capacity

Volunteer thinking

(Environmental) monitoring

Participative sensors

Community science

‚Extreme‘ CiSi

passive

active

trained

creative

Scientific

Science/environmental education

Engagement/dialogue

We suggest a theory-based approach to evaluation of CiSi.

5

Objectives Inputs Outputs

Needs Problems

Issues

Evaluation

Efficiency

Value Added

Impact and Sustainability

Other public interventions Outcomes

Impacts

Effectiveness Coherence &

Complementarity

Relevance Citizen Science projects/programmes

Society Research Other

Suggestions for evaluation questions (1/2)

6

Relevance •  To what extent do project objectives correspond to societal needs? •  To what extent does the project setup (scope, mechanisms etc) correspond to project

objectives? •  …

Effective-ness

•  To what extent do the projects contribute to major objectives such as science education/engagement of the public/the generation of scientifically relevant data?

•  To what extent does the project contribute to other objectives, such as the involvement of underprivileged/other societal groups etc?

•  To what extent is the implementation of the project effective? Which factors are hindering an effective implementation?

•  …

Efficiency

•  To what extent are monetary costs proportionate to benefits? •  To what extent are non-monetary costs (eg free-time of volunteers) proportionate to

benefits? •  Are costs and benefits in line with initial expectations? •  …

Value Added

Suggestions for evaluation questions (2/2)

7

Coherence & comple-mentarity

•  To what extent do other activities cover the objectives of citizen science projects (for instance science education, science communication, public participation)?

•  Which additional benefits can CiSi projects bring? •  …

Impact & Sustain-ability

•  To what extent did the project contribute to a change in behaviour/change of the agenda/RRI/choice of study etc?

•  How sustained is the project conducted? •  How sustainable is citizen involvement in CiSi activities? •  …

•  Value added compared to existing approaches in science education/engagement •  Contribution to reaching objectives •  To what extent is action/funding by public bodies required? •  Value added compared to current research practice •  …

Indicators: Inputs

8

Inputs

•  Expenditures for human resources (eg researchers) •  Expenditure equivalents for unpaid human resources

(volunteers) •  Other related expenditures

Indicators: Outputs

9

Outputs

•  # of people reached / participants •  # of projects •  # of sites studied/ animals etc recorded •  # of trainings for citizens •  # of trained citizens •  # of transdisciplinary projects •  # and type of educational resources produced •  # of academic publications generated •  intensity of engagement •  Type of project •  Involvement of other societal/community actors (schools, clubs, etc) •  Use of IT & type of IT use •  Input for research/policy agenda •  Research data generated •  High quality data •  ...

Indicators: Outcomes

10

Outcomes

•  Textbook and process knowledge („how science is done“) •  Confidence to contribute to further research •  Engagement in and with science & technology •  Political ownership •  Formation of an opinion •  Research results •  Increased relevance of research for everyday life •  ...

Indicators: Impacts

11

Impacts

•  Change in behaviour (eg environmental engagement with S&T)

•  Interest in STEM/environmental career •  Responsible Research & Innovation through adjusted policy

agendas •  Changed climate for science in the society •  High quality research •  ...

Summary & Discussion

12

•  In your view, is the general approach suitable?

•  Which other indicators and evaluation questions do you propose?

•  Which importance do participative elements play? CiSi evaluation as a CiSi project?

•  CiSi is receiving increasing attention, also by policy makers.

•  CiSi projects are characterised by a large variety in terms of project type, intensity of interaction and objectives.

•  When evaluating CiSi projects, we propose to handle this complexity by using a theory-based evaluation approach.

•  We suggest as a first set of evaluation questions and several indicators for inputs, outputs and outcomes as well.

13

Thank you!

Dr Larissa Talmon-Gros technopolis |group| Deutschland GmbH Große Seestraße 26 60486 Frankfurt am Main Deutschland Larissa.Talmon-Gros@technopolis-group.com www.technopolis-group.com technopolis |group| has offices in Amsterdam, Brighton, Brussels, Frankfurt/Main, Paris, Stockholm, Tallinn und Vienna