Designing towards an impact evaluation framework for a collaborative information supply chainKENNY MEESTERS, BARTEL VAN DE WALLEISCRAM BADEN-BADEN, MAY 2013

OutlineOutline

Domain Processes

Evaluation types

Systems evaluatio

n

Evaluationperspectives

Scope Measurement

Concept Supply Usage

Findings

Objectives

Indicators

Conclusion

Results

V&TC

Design

Approach

Research

Future work

V&TC

Data collection•Media•Geo-location•SMS

Data processing•Analysis•Verification

Dissemenation•Information products

•Maps, reports, etc

Information consumers•Decision making•Monitoring

Outline

Volunteer Training

Open*

SBTF

Transition

Impact Evaluation Decision Makers

Data Scramble

Data licensing

Preparedness

UN OCHAIM

ISCRAM

IMMAP, Google, Mapaction

Woodrow, UN,Harvard, OSM

Mapaction,GISCorp,Munster

ISCRAM, ICT4Peace,Mapaction

SBTF, UNV

ISCRAM, Harvard, UvT

TBC

UNV, Munster

“The challenge is to improve coordination between the structured humanitarian system and the relatively loosely organized volunteer and technical communities. ” - Valerie Amos, UN Under-Secretary-General 

V&TC

ISCRAM, Harvard, UvT

Impact Evaluation

Impact Evaluation

What do we need?What do we want to know?What do we know?

BusinessV&TC

Practice

Theory

Impact Evaluation

What do we need?In general…

BusinessV&TC

Practice

Theory(2) Measure

Status quo

New situation

(1) Define

Indicators

Situations

(3) Analyze

Comparison

Conclusion

In general…Applications

BusinessV&TC

Practice

Theory

Project A

1. Impact evaluation

2. Impact assessment

3. Program evaluation

Project B

BusinessV&TC

Practice

Theory

• Determine how well specific initiatives perform• Adjust and fine tune specific decisions/projects• Determine ‘best’ response• Manage provided solutions• Secure resources• Advocate V&TC

• Impact evaluation

• Impact assessment

• Program evaluation

Use for V&TCApplications

BusinessV&TC

Practice

Theory

Use for V&TC

• Design principles of frameworks• Types

• Measurements

• Indicators

• V&TC• Objective

• Scope and focus

• Indicators

• Evaluate the framework• Case studies

• Refine-able, usable tools

Next steps

Next stepsEvaluation types

Evaluation

Perspective

Systems

General Formative

Resource centered

Efficiency oriented

Goal centered

Effectiveness oriented

Summative

Evaluation typesSystem evaluation

Evaluation

Perspective

Efficiency oriented

Effectiveness oriented

Systems

Resource investment

Production capability

Resource consumption

Organization

Organizational performance

User performance

System performance

System evaluationEvaluation implementation

Organizational performance

Department A

(Sub)Project A

(Sub)Project B

(Sub)Project C

Department B

Eval

uatio

n fr

amew

ork

Project Management

Efficiency orientedEffectiveness oriented

Scope

Overall impact of the response to a crisis

Impact of the decision making process on crisis

Impact of information productson the decision making process

Effect of data processing on information products

Impact of data collectionon data processing

Soft- en hardware impacton the system performance

1.

2.

3.

4.

5.

6.

SUPPLIER

CON

SUM

ER

Evaluation implementation

level 0:Request / definition

level 1:Resource allocation

level 2:Team capability

level 3:Investments

Impactlevel 1:

Support & information

level 2:Decision making

level 3:Response effectiveness

ScopeMeasurements

Efficiency-oriented

perspective

Effectiveness-oriented

perspective

System implementatio

n

Product generation efficiency

Response effectiveness

V&TC deployment

MeasurementsIndicators

level 1:Resource allocation

level 2:Team capability

level 1:Support & information

level 2:Decision making

Impact

ObjectivePerformance measure

Applied to V&TC

System development

Facilities allocation Availability of required (tech.) facilities

Schedule compliance Time required to setup required systems

Requirements definition

The clarity of requested products

Operational resources

Data collection efforts Time/effort required to analyze data

System maintenance Time/effort required to maintain system

Training/support/comm

Efforts for user assistance.ObjectivePerformance measure

Applied to V&TC

Team capacityProductivity rate Level of V&TC body deploymentRequired man-hours The total amount of hours used

Operational capability

Throughput Products delivered/users servedUtilization rate Hours to product ratioResponse time Turn-around time on specific

requests

ObjectivePerformance measure

Applied to V&TC

System quality

Usability Ease of use of information productsSystem features Customization of information

productsAccess / Availability Ease to reach information products

Information quality

Understandability Presentation of gathered information

Consistency Provided information is consistentImportance / Relevance

Relevance of provided information

ObjectivePerformance measure

Applied to V&TC

Individual impact

Awareness / Recall Better situational awareness Decision effectiveness Enhanced effectiveness of jobIndividual productivity Increased personal productivity

Organizational impact

Cost-effective Information products save resources

Increased capacity Increased effectiveness of operations

Overall productivity Potentially improved outcomes

4 suppliers, 7 consumers7 suppliers, 12 consumers

IndicatorsCase Study

Developpers and entry team

Specific knowledgeTime critical

No budgetGeographically seperate

Users are ‘unkown’

Developpers vs data entryExpertise availableTime limitedLimited budgetLocated in 1 officeDirect contact w. users

==≈≈≠≠

NGODHN

Case StudyInformation SupplySystem NGO V&TC FA:SC:RD:

Facilities allocation

Schedule compliance

Requirements definitionData collection efforts

System maintenance

Training, support and communication

Operational resources System developmentNGO Development V&TC Deployement

Resource NGO V&TCDC:SM:TS:

Productivity

Required man-hours

Throughput

Utilization rate

Response time

Operational capability Team capacityNGO Development V&TC Deployement

System NGO V&TC PD:MH:

Resource NGO V&TCTP:UR:RT:

Level 2: Capabilities

Level 1: Resources

Information SupplyInformation useSystem NGO V&TC SF:AV:US:

Inform. NGO V&TCUS:CO:IM:

Individ. NGO V&TC US:AW:EF:PR:Organ. NGO V&TCOP:CI:CE:US:

Level 2: Processes

Level 1: Information

Availability

Usability

System featuresUnderstandability

Consistency

Importance

NGO Development V&TC DeployementSystem quality Information quality

Usage

Awareness

Effectiveness

ProductivityUsage

Cost-effective

Capacity increase

Overall productivity

NGO Development V&TC DeployementIndividual impact Organization impact

Information useFindings

• Agile vs. Waterfall• Organizational use• Strong integration• Requirement analysis• Sample selection• Identifying population• Also for other information supply chains

• Difference in system use

• Increasing impact

• Improving evaluation

Future work

V&TC

Feedback:

Increase deployment impact

Advocacy:

Secure resources

Manage:

Improve products

Findings

• Historical data• Feedback loops• Add/remove variables• Scope of evaluation

• Refinements

• Framework design

• Application

Apply framework

Select case

Impact EvaluationFramework

Select particiapants

Control group

Conducted InterviewApply framework

Select particiapants

Control group

Conducted Interview

Verify results Verify resultsRefine Framework

Statistical analysis

Data store

Model Refinement loop

Impact evaluation outcome

Evaluation approach

V&TC

Feedback:

Increase deployment impact

Advocacy:

Secure resources

Manage:

Improve products

Feedback:

Manage pool of

resources

Advocacy:

Common understan-diing of IS impact

Manage:

Identify gaps,

ensure good fit

Coordination

Feedback:

Manage pool of

resources

Advocacy:

Common understan-diing of IS impact

Manage:

Identify gaps,

ensure good fit

Coordination

Feedback:

Improve effectivenss by IS

use

Advocacy:

Articulate needs,

and require-ments

Manage:

Improve IS use in future

responses

Decision Makers

Feedback:

Improve effectivenss by IS

use

Advocacy:

Articulate needs,

and require-ments

Manage:

Improve IS use in future

responses

Decision Makers

Impact Evaluation for the V&TC:Communicate, Learn, Advocate