1

What we think is what matters: the role of impact assessment (IA)

in the CGIAR change management

Javier M. Ekboir

2

Overview of the presentation

•Review of evaluation and IA approaches

• How can philosophy of science contribute to evaluation and IA?

•The nature of organizational change

•Evaluation and IA in the history of the CGIAR

• Evaluation and IA in the on-going change process

3

Evaluation and IA can be categorized from two perspectives: goals and methods

Goals: • summative or judgment-oriented to determine

the value of a program• formative or improvement-oriented to identify

steps to improve a program• developmental to facilitate learning during

implementation

4

Evaluation and IA can be categorized from two perspectives (2)

Methods: • Qualitative• Quantitative – rates of return and cost-benefit analysis – other approaches, especially related to how

to measure changes in indicators in non-experimental settings

• Combination of qualitative and quantitative

5

Which approaches and methods are appropriate depends on what important

stakeholders think is appropriate!

6

The role of data in learningArgentina’s amazing discovery

Every data set (experimental or not) is open to an infinite number of hypotheses (underdetermination thesis)

Let me show you the picture of a bird

Or is it a rabbit?

Data are collected and have a meaning within a “theoretical” framework (“theory-ladenness” of data)

Data are important because they provide information to build and “confirm” theories

7

Hypotheses cannot be tested (T, H, A1, A2 … An)

•Then, what do researchers do?

•They “build explanations”

•But there is no consensus on what a valid explanation is

• Scientists build narratives that reflect how they see the processes they study

8

The social nature of knowledge

The acceptance or rejection of a narrative depends on the consensus among the majority of scientists and other stakeholders about what is a good explanation

This does not mean that all knowledge is relative

Just that knowledge is determined by the interaction of data and social conventions

Consensus about climate change

9

Organizational change depends, among other factors, on:

The nature of leadership

The organization’s culture

The organization’s learning routines

Governance

In NGOs, also external consensus about what needs to be done

10

The nature of change in organizations

Stakeholders influence organizational change according to their beliefs, resources and connections

Similar minded stakeholders form informal coalitions

The more effective a coalition, the greater its influence on the change process

11

The nature of change in organizations (2)

Influence can be gained in two ways:

•commanding resources

• convincing other stakeholders by presenting a compelling argument about what change is needed and how to measure it

Evaluations do not bring organizational change by themselves, but influence the perception of what changes are needed

12

Why was impact assessment implemented in the CGIAR?

In the 1990s, CGIAR centers had to justify their existence, i.e., accountability was the most important motive for IA

For decades, economists had calculated rates of return to research, which was considered by some professionals a rigorous method to measure impacts

Some of these professionals occupied important positions among donors, in the science council and in the centers

Over time, other approaches for IA were developed by CGIAR and external scientists, but they remained at the fringes of the system

13

Why was impact assessment implemented in the CGIAR? (2)

In recent years, some donors and scientists realized that an alternative to the rates of return was needed

New approaches for evaluation, IA and organizational learning are being adopted by donors, CGIAR scientists, other organizations working on development, and important segments of the evaluation community

The issue today is what type of IA meets scientific standards and is useful for the donors and the CGIAR

14

IA and change management in the CGIAR

Evaluations influence the change management process by helping to create a consensus about the new role of the CGIAR and the changes needed

IA, as designed by the SC, did not induce change in the system

How the new approaches for evaluation can influence change in the CGIAR depends on the system’s ability to use the new information

This depends, among other factors, on the coalitions that influence the process

15

How can new evaluation approaches help change in the CGIAR?

Generating new frameworks and information to help the Consortium, similarly-minded donors and DGs to provide effective leadership for change

Synthesizing the new approaches for evaluation and IA developed by CGIAR scientists and providing a unifying framework

16

How can new evaluation approaches help change in the CGIAR? (2)

Developing new approaches for IA based on novel frameworks for the analysis of social processes

Building a new, compelling argument for reassessing the role of science (including the CGIAR) and IA in development by bringing in new ideas and challenging orthodoxies

17

On its own, IA will have little impact on the change process

To have an impact, it is necessary to work simultaneously along three lines of action:

• Give more visibility to the new methods for evaluation and IA developed by CGIAR and/or external scientists

• Develop a conceptual framework that unifies the new methods for evaluation and IA and provides a compelling argument for using evaluation as a tool for organizational learning

• Build a coalition of similarly-minded stakeholders

18

Thank you!