WHERE BER STARTEDEvaluation of Oxfam

GB’s Global Climate Change Campaign

Solution to evaluation challenge of considering value for money of a complex, multi-unit, international campaign

1

EVALUATION CHALLENGES

Simplifying complex multi-unit programs

Many evaluations are about ROI, but ROI is difficult to assess in social contexts

Nothing is good or bad, except in comparison to something else

2

THE BER SOLUTION

Provide a simple framework for evaluating complex multi-component programs, campaigns, or activities

Build on the basic concepts of SROI to evaluate unit's impact compared to their resources

Offers a relative perspective on performance where units of analysis are judged in comparison to their peer units, operating under similar conditions

3

THEORETICAL FOUNDATIONS: MATRIX ANALYSIS FRAMEWORKS• Boston Consulting Group

• General Electric Grid

• Customer satisfaction quadrant analysis by Andreasen

• Bloc modeling techniques used by social network analysts

• Multi dimensional scaling

• SROI

4

BER VARIABLES

input constitutes a program’s resources which may be measured by its budget, number of staff, pool of talent, social capital, or any measure of capacity, concrete or abstract.

5

Input

High

Below Average Efficiency

Average Efficiency

Low Average Efficiency

Above Average Efficiency

Low High

Output

Output measures a program’s impact, and will vary according to a program's purpose – may include behaviour change; public awareness; policy change; reduced inequality; improved environmental health… or any other measure depending on a program's goals

CONDUCTING A BER ANALYSISThe example in following section is fictional and for illustrative purposes.

6

1. SELECTING UNITS OF ANALYSIS

7

For example: Coalition/partnership building; intra organisational coordination; lobbying and advocacy; media relations; online engagment; public mobilization; research...

Input Output Program budgets (perceived and

real) Number of staff Number and level of staff

How often a lobbying keyword appeared in policy

Number of widgets produced Number of people engaged Perceptions of impact

2. DATA TYPES

• Quantitative input data may include budgets, number of staff, or combined multi-dimensional resource measure

• Quantitative output data may include process evaluation measures such as the number of people engaged by a campaign or media hits

• Qualitative measures can include perceived program investments and perceived output achieved

8

3. MEASUREMENT TOOLS

9

Completely

Disagree1

2 3 4 5

Completely

Agree6

I can't say

Online engagementResearch papersLobbying and advocacy

1. Units of Analysis

3. Perceived Input (Budgets)

2. Perceived Output (impact)

Online engagement

•Most•Average•Least•I don't know

•Most•Average•Least•I don't know

Research papers •Most•Average•Least•I don't know

•Most•Average•Least•I don't know

Lobbying and advocacy

•Most•Average•Least•I don't know

•Most•Average•Least•I don't know

Organisation X has influenced policies or legislation because of the following activities:

4. VISUALIZATION APPROACHES

10

High

Low

Low High

4. VISUALIZATION APPROACHES

11

Input

High

Media relations Public relations

Lobbying and advocacy

Coalition/partnership building

Low

Intra organisational coordination

Online engagement

Research papersPublic mobilization

Low HighOutput

4. VISUALIZATION APPROACHES

12

CASE STUDY: OXFAM GB’S GLOBAL CLIMATE CHANGE CAMPAIGN

13

3.53.02.52.0

Investment

4.5

4.0

3.5

3.0

Impact

Visual stunts & media relations

Effective com at policy events

Using celebrities

Global Oxfam affiliates

Research papers

Rapid news dissemination

Public campaigns/mobilization

Policy analysis

Partnerships with others

Oxfam internal program links

Online campaigning

Media engagement

Lobbying and advocacy

Staff in UNFCCC delegations

Climate hearings

Adopt a negotiator

Perc

eive

d im

pact

Perceived resourcing

High

HighLow Low

5. INTERPRETATION

Use BER is as a starting point for deeper discussions into the performance of intervention units, their challenges, opportunities, and operating environment

Understand the units of analysis and the informants who shared their perceptions

Not all units within a program operate under the same conditions

Some units contribute indirect effects, by empowering other units

14

LIMITATIONS AND RISKS

It is easy to draw conclusions from the simple visualizations that would never stand in the face of a deeper understanding of the reality behind charts

Kotler et al. (2005) noted, reliance on matrix approaches prompted a number of companies to sell off strategic assets and plunge into businesses that they lacked the experience to manage

15