Building the evidential quality of evaluations

ALNAP March 5, 2013

Tony Redmond

Humanitarian vs Development

Hierarchy of Evidence

Most evidence in the humanitarian sphere is in the weakest categories or outside

the pyramid altogether

Expert Opinion

• Consensus Statements Regarding the Multidisciplinary Care of Limb Amputation Patients in Disasters or Humanitarian Emergencies: Report of the 2011 Humanitarian Action Summit Surgical Working Group on Amputations Following Disasters or Conflict

• Delphi Studiens

Gathering the Evidence

bias

• Incentive• Observer• Systematic (non random) error

– Confidence intervals

RCTSibbald B, BMJ 1998;316:201

• the most rigorous way of determining whether a cause-effect relation exists between intervention and outcome and

• for assessing the cost effectiveness of an intervention.

Dangerous Data

• Wrong inference• Randomisation• Open and transparent

HERR

• Medical teams are 200 times as cost effective as UKISAR

• That is they cost less, stay longer, treat more people and save more lives

sampling

Convenience sampling

• Using those who are willing to volunteer, or cases which are presented to you as a sample.

Judgement sampling

• Based on deliberate choice and excludes any random process.

• Normal application is for small samples from a population that is well understood and there is a clear method for picking the sample.

• Is used to provide illustrative examples or case studies.

Simple random sampling

• Ensures every member of the population has an equal chance of selection.

• Produces defensible estimates of the population and sampling error.

• Simple sample design and interpretation.

Cluster sampling

Units in the population can often be found in geographical groups or clusters. A random sample of clusters is taken, then all units within those clusters are examined.• Quicker, easier and cheaper than other forms of random sampling.• Does not require complete population information.• Useful for face-to-face interviews.• Works best when each cluster can be regarded as a microcosm of the

population.• Larger sampling error than other forms of random sampling.• If clusters are not small it can become expensive.• A larger sample size may beneeded to compensate forgreater sampling error.

Why is evidence ignored?

• Evidence influencing policy• Policy influencing evidence• We've always done it this way…• ……and it’s done no harm (regardless of

the evidence)• Maybe the evidence was always there….

Epidemiology/statistics

• Observation• Pattern Recognition• Cause and effect or coincidence• Change of practice

CONFIDENCE INTERVALS (CI)

MEAN VALUE

You have a 95% confidence that the TRUE VALUEshould fall between the CI if the study is repeated multiple times in a similar population

95% CONFIDENCE INTERVAL

assessments

• Early warning• Risk analysis• Trending• Needs assessment

Evidence based medicine

• Clinical trials in the field – in an emergency?

• Multiple disciplines

Evaluation

• What was done?• What was the reason for doing it?• What did they find?

• SO WHAT?

Benefit cost

What you have to evaluateadapted from DFID WASH Evaluation 2011

• Internal validity – was what they did good enough?

• External validity – relevance to you?• Can you implement it ?• Sustainability?• Wider impact?• COST

Is good enough, good enough?

Humanitarian

• Historically draws on concept of compassion, individuality and reform (transformation)

• It is evidence and evaluation that will help us reform and lead us to prevention