7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
1/26
ALNAP Working Paper
Harnessing the Power ofEvaluation in Humanitarian Action:An initiative to improve understanding and use of evaluation
Alistair Hallam
June 2011
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
2/26
1. Inroducion 3
Background 3Limiaions 4
Ouline o his paper 4
A ramework o improve undersanding and use o EHA 4
Acknowledgemens 6
1. Leadership, culure and srucure 7
Ensure leadership is supporive o evaluaion and monioring 7
Promoe an evaluaion culure 7
Increase he inernal demand or evaluaion inormaion 8
Creae organisaional srucures ha promoe evaluaion 9
2. Evaluaion purpose and policy 10
Clariy he purpose o evaluaion (accounabiliy, audi, learning) 10
Clearly ariculae evaluaion policy 11
Ensure evaluaion processes are imely and orm an
inegral par o he decision-making cycle 12
Emphasise qualiy no quaniy 12
3. Evaluaion processes and sysems 14 Develop a sraegic approach o selecing wha should be evaluaed 14
Involve key sakeholders hroughou he process 14
Use boh inernal and exernal personnel o encourage a culure o evaluaion 16
Improve he echnical qualiy o he evaluaion process 16
Assign high prioriy o eecive disseminaion o ndings, including
hrough new media (video, web) 17
Ensure here is a managemen response o evaluaions 18
Carry ou periodic mea-evaluaions and evaluaion synheses,
and review recommendaions 19
4. Supporing processes and mechanisms 20
Improve monioring hroughou he programme cycle 20
Provide he necessary human resources and incenive srucures 20
Secure adequae nancial resources 21
Undersand and ake advanage o he exernal environmen 22
Nex seps 24
Reerences24
A B L E O F C O N E N S
2
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
3/26
Background
ALNAP has done more work on evaluaions o humaniarian acion (EHAs) during he pas ew
years han a any oher ime in is hisory. Tis has included research on evaluaion uilisaion, join
evaluaions, a pilo guide on real-ime evaluaions, a meeing and sudy on impac assessmens, a
new look Evaluaion epors Daabase, and he new Sae o he Humaniarian Sysem repor, which
included an exensive evaluaion synhesis.
However, despie hese eors, and some real improvemens wihin agencies in how hey carry
ou evaluaions, here remains a eeling among many in he secor ha he ull poenial bene
o humaniarian evaluaions is no being realised. oo oen, humaniarian evaluaions exis as
a disconneced process, raher han becoming embedded as par o he culure and mindse o
humaniarian organisaions. Te secor has made real advances in he qualiy o evaluaions; i would
seem ha he challenge now is o ensure ha evaluaions are acually used o improve operaional
perormance.
ALNAP is no alone in coming o such conclusions. Te UN Evaluaion Group (UNEG)1 and he
wider OECD-DAC group2 have been carrying ou imporan work on improving approaches o
evaluaion or some years across he UN sysem, and also carrying ou peer-reviews o members
evaluaion sysems and srucures. Following a UNEG peer-review, he Oce o Evaluaion ohe World Food Programme (WFP) commissioned work o enhance he learning purpose o
he organisaions evaluaions (WFP 2009). Similarly, he UKs Deparmen or Inernaional
Developmen (DFID) has recenly commissioned a sudy rom he Overseas Developmen Insiue
on how o improve is learning rom boh evaluaion and research (Jones and Mendizabal 2010).
UNICEF has carried ou similar work (Soddard 2005).
Tere have been similar debaes on evaluaion eeciveness wihin he developmen communiy
or decades. Ta he humaniarian communiy is coming o hese issues relaively lae may refec
he simple ac ha i also came laer o evaluaion isel. However, here is perhaps also somehingdieren abou humaniarian aid in he scale and iner-connecedness o is ambiion, and he
subsequenly greaer diculies o undersanding and atribuing impac.
Agains his background, ALNAP has commissioned a process o acion research, o which his paper
is a par, ocusing on srenghening insiuional undersanding as well as capaciies and processes in
order o beter harness he power o EHA. Te iniial research used lieraure review and inerviews
across he ALNAP member organisaions and idenied areas or urher examinaion. A cenral
emerging lesson is ha improving humaniarian evaluaion capaciies and processes is a complex
underaking ha involves a wide range o acors and acors including allocaion o resources;
1 . I N O D U C I O N
1. htp://www.uneval.org/2. htp://www.oecd.org/deparmen/0,3355,en_2649_34435_1_1_1_1_1,00.hml
3
http://www.uneval.org/http://www.oecd.org/department/0,3355,en_2649_34435_1_1_1_1_1,00.htmlhttp://www.oecd.org/department/0,3355,en_2649_34435_1_1_1_1_1,00.htmlhttp://www.uneval.org/7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
4/26
engaging senior leaders; and realigning operaional procedures.
Tis paper sars he process o developing he ideas generaed hrough he lieraure review and
inerviews. I is hoped ha by illuminaing he range o signican acors involved, his paper will helpALNAP members ineresed in srenghening heir evaluaion eors by helping hem o ideniy
prioriy areas o concern, share ideas across he membership abou wha has and has no worked, and
develop new sraegies or ackling longsanding issues.
A dra o his paper was presened a a workshop in Sepember 2010 ha brough ogeher evaluaors
rom UN, NGO, ed Cross and donor organisaions. Te paper was subsequenly redraed o
ake ino accoun commens made a ha workshop. Te paper was also peer-reviewed by ALNAP
members and changes were made o he dra as a resul o he helpul and insighul commens
received. Publicaion o his paper represens he end o Phase I o his projec.
Limiaions
Te work discussed in his paper is preliminary, and remains a work in progress. Te inerviews
inorming his paper generally involved only one or wo people per organisaion usually evaluaion
specialiss. In uure, we would need more inpu rom programme, eld and managemen personnel.
Tereore, his paper is a sepping-sone o a broader undersanding o rends and issues in
srenghening organisaional capaciies, processes or, and undersanding o he evaluaion o
humaniarian aciviies. I is inended as a saging pos on a roue o opening discussion and asking hekinds o quesions which will help o advance he debae in pragmaic and pracically useul ways.
Ouline o his paper
Te remainder o his Inroducion presens a simple dra ramework or increasing he impac and
use o evaluaions o humaniarian acion. Tis ramework akes inspiraion rom general lieraure
on evaluaion uilisaion and capaciy building, bu is modied according o he specics o he
humaniarian conex.
Te ramework is hen populaed in Chapers 1 o 4 wih discussion on he specic issues ha
seem mos likely o increase impac and uilisaion as drawn rom he lieraure review and rom
inerviews wih a small sample o ALNAP members. Te concluding Nex seps secion oulines
plans or coninuing and developing he work presened in he preceding chapers.
A ramework or improving undersanding and use o EHA
A dra analyical ramework is presened here as an aid o urher analysis o wha infuences
undersanding and use o EHA. Te purpose o he ramework is o help shape discussions among
4
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
5/26
ALNAP members and ohers involved in evaluaing humaniarian acion on how o address he
problems o poor evaluaion impac. Te ramework draws on lieraure on capaciy developmen,
and was rened ollowing discussions wih humaniarian evaluaors in order o make i moreappropriae or use by hose involved in EHA.
Te ramework is a work in progress. Framework caegories ha do no add much will be removed,
while ohers will be added as needed. Te aim is o keep he ramework as ligh and as easy o
use as possible, wih a clear ocus on he key issues. Te aim is also no o be overly prescripive.
A imes, where here appears o be consensus rom inerviews and he lieraure, he discussions
wihin he ramework poin owards a soluion. However, elsewhere, he answer o a problem may
be very agency-specic, when an exploraion o he pros and cons o dieren approaches is more
appropriae.
Te bare bones o he ramework are presened here, wih discussion o each capaciy area in he
our chapers ha ollow. Overarching issues o leadership, culure and srucure are presened rs,
ollowed by issues o purpose and policy. Nex are more specic evaluaion processes and sysems.
Te ourh and nal capaciy area is supporing processes and mechanisms, some o which are o
wider relevance, and do no relae solely o evaluaions. Eecive capaciy building requires atenion
a all levels o he ramework, wih much iner-dependence beween hem.
Capaciy Area 1: Leadership, culure and srucure
Ensure leadership is supporive o evaluaion and monioring
Promoe an evaluaion culure
Increase he inernal demand or evaluaion inormaion
Creae organisaional srucures ha promoe evaluaion
Capaciy Area 2: Evaluaion purpose and policy
Clariy he purpose o evaluaion (accounabiliy, audi, learning)
Clearly ariculae evaluaion policy Ensure evaluaion processes are imely and orm an inegral par o he decision-making
cycle
Emphasise qualiy no quaniy
Capaciy Area 3: Evaluaion processes and sysems
Develop a sraegic approach o selecing wha should be evaluaed
Involve key sakeholders hroughou he process
Use boh inernal and exernal personnel o encourage a culure o evaluaion
5
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
6/26
Improve he echnical qualiy o he evaluaion process
Assign high prioriy o eecive disseminaion o ndings, including hrough new media
(video, web) Ensure here is a managemen response o evaluaions
Carry ou periodic mea-evaluaions and evaluaion synheses, and review
recommendaions
Capaciy Area 4: Supporing processes and mechanisms
Improve monioring hroughou he programme cycle
Provide he necessary human resources and incenive srucures
Secure adequae nancial resources
Undersand and ake advanage o he exernal environmen
Use peer neworks o encourage change
Engage wih media demands or inormaion
Engage wih donors on heir evaluaion needs
Acknowledgemens
Many hanks o everyone a he ALNAP Secrearia or heir suppor. John Michell, Ben
amalingam and Yuka Hasegawa all gave considerable ime o orienaing he work and commeningon dras. Josh Harris helped o se up he pre-workshop quesionnaire, and Colin Hadkiss helped
wih he organisaion o he workshop. Tanks o he many ALNAP and non-ALNAP members
who agreed o be inerviewed. Tanks, in paricular, o achel Bedouin, Aine Hearns and Karen
Proudlock, who gave deailed commens as ormal or inormal peer-reviewers.
6
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
7/26
Ensure leadership is supporive o evaluaion and monioring
A high proporion o he inerviews carried ou or his sudy idenied leadership as key in
improving he impac o EHA. Tis is refeced in he academic lieraure as well as in reviews o
he evaluaion uncions o various humaniarian organisaions. Inerviewees commened on
heir experience o a change in leadership having a proound and posiive impac on he value and
eeciveness o evaluaions. Where leaders are no ineresed in evaluaion, or are overly deensive
abou he perormance o heir organisaions and hence relucan o accep evaluaion ndings, a
culure develops agains learning rom experience. I daa and analysis are no valued a senior level,
his can permeae hroughou he organisaion and lead o relucance even o collec he necessary
inormaion in he rs place.
Even wihou supporive leadership, an evaluaion champion can show he wider organisaion
he benes ha can arise rom evaluaion. Good qualiy evaluaions ha highligh imporan
issues wihin programmes ha would oherwise have been missed can serve as a ool o convince
managers o he imporance o evaluaion. Te board or unders o an organisaion can seek o
ensure ha recruimen o senior managers emphasises he imporance o evaluaion. A recen sudy
on srenghening learning rom research and evaluaion in DFID (Jones and Mendizabal 2010)
suggesed ha a direcor be appoined as knowledge and learning champion.
Promoe an evaluaion culure
Proudlock (2009) noes ha he evaluaion lieraure (eg Paton 2008; Mayne 2008) suggess ha
culure, as well as leadership, are a key deerminan o evaluaive capaciy. Ideally, an organisaional
culure will acively seek inormaion on perormance in order o learn how o improve he
managemen and delivery o is programmes and services, and o improve is perormance. Tere
needs o be a commimen o using evaluaion daa and hen changing behaviour based on ha daa.
Enabling credible evaluaion ha is useul and ges used requires esablishing eecive sysems and
srucures, such as organisaional policies, sraegic planning, programme reporing and inormaion
sysems, budgeing, human resources, accounabiliy rameworks and learning processes. Boh Paton
(2008) and Mayne (2008) poin ou, however, ha his is only ever secondary o esablishing he
wo key elemens o culure and leadership. According o Mayne (2008), eors o creae evaluaion
sysems wihou addressing organisaional culure or leadership are likely o end up as burdensome
and could poenially work agains a culure o seeing evaluaion (or any resuls daa) as valuable and
worh pursuing.
1 . L E A D E S H I P, C U L U EA N D S U C U E
7
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
8/26
How do organisaions go abou culivaing a posiive inernal culure? Alhough, as above, evaluaion
expers sae ha culure mus come rs, and hen sysems, i also seems ha developing eecivesysems and srucures or credible evaluaions is an imporan par o he process o changing he
culure. Success breeds success, so ha high-qualiy evaluaions, ha address real inormaion needs,
improve he way in which evaluaions are regarded. Tis posiively aecs he evaluaion culure,
which, in urn, increases he demand or evaluaions, and makes i more likely ha evaluaion sysems
and srucures are improved in a susainable way. A viruous circle ensues. ackling issues a each sage
o he ramework idenied in his paper can help in creaing ha viruous circle.
While many o he building blocks or increasing evaluaion capaciy wil l be similar or any
organisaion, each organisaion already has is own unique culure and way o doing hings. Tereore,
each will need o analyse is own curren siuaion, is readiness or change, and he challenges o
building evaluaion capaciy. Dieren sraegies will hen be required or each organisaion. Tose
promoing change will need o be opporunisic, seizing he easy wins rs, so ha impac is quickly
seen and suppor is gahered or he challenge o changing he evaluaion culure.
Organisaions also sand o learn rom he atemps boh successul and unsuccessul ha similar
organisaions have made o improve evaluaion uilisaion. Furher developmen o he ramework
presened in his sudy will involve workshops, case sudies and join work by small groups o
ALNAP members, o help aciliae he process o peer learning.
Increase he inernal demand or evaluaion inormaion
Increasing demand or inormaion in order o make managemen decisions will increase he demand
or evaluaions. Bu how does one increase demand or inormaion? Paton, wriing on uilisaion-
ocused evaluaion (2008), alks abou he need, when considering poenial sakeholders and
possible uses, o move rom he general and absrac o he real and specic. Very early on in he
process, one needs o ideniy primary inended users and heir explici commimens o concree,
specic uses. In Patons view, he evaluaor aciliaes judgemen and decision-making by inended
users, raher han acing as a disan, independen judge.
Inended users are more likely o use evaluaions i hey undersand and eel ownership o he
evaluaion process and ndings. Tis is more likely i users have been acively involved in every sep o
he process. Paton suggess ha evaluaion should sar wih he generaion by end users o quesions
ha need o be answered. Such quesions should ideniy inormaion ha inended users would like,
and which would make a dierence o wha hey do.
Paton reers o research ideniying he personal acor as one o wo key acors ha emerged as
consisenly imporan in explaining evaluaion uilisaion.
8
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
9/26
Te personal acor is he presence o an ideniable individual or
group o people who personally care abou he evaluaion and hendings i generaes. Where such a person or group was presen,
evaluaions were used; where he personal acor was absen, here was
a correspondingly marked absence o evaluaion impac. (Paton 2008)
For Paton, he evaluaion process mus discover he key sakeholders, namely hose who really wan
o know somehing. Formal posiion and auhoriy may mater less han enhusiasm and ineres.
Evaluaors need o nurure and susain ineres, and his requires skills in building relaionships,
aciliaing groups, managing confics, and eecive inerpersonal skills as well as more echnical
evaluaion skills.
Tese ndings are echoed in a recen sudy on srenghening learning rom research and evaluaion
wihin DFID evaluaions:
he lack o ownership o large evaluaions or research programmes delivered exernally can help
explain heir lack o infuence. elaed o his, direc inerpersonal relaions beween sa and boh
researchers and evaluaors, as well as o he research and evaluaion processes, mater a grea deal.
(Jones and Mendizabal 2010)
Creae organisaional srucures ha promoe evaluaion
Tere are several ways o organising he managemen o evaluaions. In some organisaions, a separae
and/or cenral evaluaion uni is responsible or carrying ou evaluaions; in ohers, responsibiliy or
evaluaion is decenralised hroughou he organisaion (Hallam 1998). Te ype o managemen
srucure infuences, or good and bad, he impac o EHA. For example, having a cenral uni
dedicaed o evaluaions migh improve he qualiy o evaluaions bu could undermine broader
insiuional learning.
Tere is signican debae abou o whom an evaluaion deparmen should repor. Someorganisaions have an evaluaion deparmen ha repors direcly o he Board, or o he Chie
Execuive. Oher organisaions place he evaluaion deparmen lower down in he hierarchy. While
here is no righ or wrong approach, i is imporan or organisaions o recognise he impac ha
insiuional srucure has on learning and accounabiliy. Te IMF has an independen evaluaion
uni answerable o he Board. However, a recen evaluaion o he uni ound ha IMF managers were
very resisan o he Evaluaion Deparmen (personal communicaion).
9
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
10/26
Clariy he purpose o evaluaion (accounabiliy,audi, learning)
Wheher an evaluaion can simulaneously mee boh accounabiliy and learning needs has been
he mos conenious issue arising rom he inerviews held so ar. Te lieraure ends o sugges
ha hese wo aims are in confic, and many inerviewees agree. Indeed, a brainsorming session
among experienced humaniarian evaluaors a a recen raining course on evaluaion ound ha an
accounabiliy ocus in evaluaions was a acor in heir poor uilisaion (personal communicaion).
Paton noes ha an evaluaion required by a under oen becomes an end in isel o be done
because i is mandaed, no because i will be useul. Mandaed evaluaions can undercu uiliy by
making he reason or he evaluaion compliance wih a unding requiremen, raher han genuine
ineres in being more eecive (Paton 2008). Te recen ODI sudy on learning lessons rom
research and evaluaions wihin DFID (Jones and Mendizabal 2010) quoes a similar nding: here
is generally a ension beween he independence o evaluaion deparmens and heir success in
engaging users o evaluaion (Foresi 2007).
Ohers, however, are equally adaman ha any evaluaion mus address boh aims, and ha one canonly ruly learn by being held rigorously accounable. Te ruh will vary by organisaion and culure,
and one canno be prescripive abou such hings. Noneheless, wha does seem o be lacking in some
organisaions is clariy abou he purpose o evaluaions, and recogniion ha dieren approaches
may be required or dieren evaluaion aims. For example, he peer review o he evaluaion uncion
a WFP observes ha he ension and complemenariies beween evaluaion or accounabiliy and
or learning seem no o be acknowledged everywhere in WFP (Baker e al 2007).
Some argue ha real learning and perormance improvemen requires rank and open discussion
wihin an agency, an abiliy o share ailures as well as successes, and a non-judgemenal approach.When an evaluaion has been exernally commissioned, and is carried ou by eams o ousiders, such
an approach may be paricularly dicul o achieve. Some NGOs, ha have no been advocaes o he
independen, accounabiliy-syle approach o evaluaion, have, however, ully adoped more inernal
evaluaion approaches, which can sill involve a considerable degree o criicism. Such criicism is
more readily acceped when i comes rom insiders who undersand he organisaion well, and who
have hemselves been involved in managing similar programmes. Some organisaions have ound ha
hones and rank discussion is consrained where evaluaions are o be made available o he public.
2 . E VA L U A I O N P U P O S EA N D P O L I C Y
10
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
11/26
One way o miigae he ension beween he diering aims o evaluaion is o separae accounabiliy
evaluaions rom learning evaluaions, and no ry o mee all agendas wih one exercise. Arguably,
i migh even be beter o call he wo ypes o evaluaion by dieren names, and have he processesmanaged and run by dieren individuals and deparmens wihin he organisaion. Few people enjoy
or welcome nancial audis, or necessarily expec o learn rom hem, bu no one hinks hey should
no happen. In a similar way, perhaps, accounabiliy evaluaions should be carried ou by he audi
and accounabiliy deparmen, whereas eld-led, learning evaluaions should be carried ou by a
deparmen o knowledge managemen and learning.
Tis separaion may already be happening o some degree. Te UK Disasers Emergency Commitee
(DEC) now has an Accounabiliy and Audi Manager, as well as someone responsible or learning.
A recen DEC repor reers o separaing ou and srenghening learning. A SIDA, here is an aid
wachdog as well as an inernal evaluaion uni, and he inernal deparmen has a major ocus on
uilisaion-based learning evaluaions. DFID may be heading in a similar direcion. Once an explici
separaion has been made beween accounabiliy and learning evaluaions, i hen becomes easier o
develop a sraegy o improve boh approaches.
Clearly ariculae evaluaion policy
EHA policies and guidelines reer o he overall purposes and objecives o he evaluaion uncion or
a paricular agency and should be where all EHA originaes. However, alhough many organisaions
have evaluaion guidelines, no all disinguish beween developmen evaluaion and EHA. Ideally,organisaions should have a disinc se o deailed policy guidelines or EHA, as agencies wihou an
agreed policy someimes experience conusion regarding he evaluaion unis mandae and/or roles
(Foresi 2007).
Tose promoing change may need o sar wih developing he appropriae policy, bu hey will
hen need o ake a sraegic approach o rolling his ou hroughou he organisaion. When wriing
EHA policy guidelines, i is imporan o consider impac and uilisaion rom he ouse. Arguably,
uilisaion should be he rs aim o any evaluaion policy. Tis will increase he likelihood ha he
enire evaluaion process is designed around end users and heir needs.
Te organisaional challenges o ranslaing policy ino pracice are no limied o high urnover
o sa and poor knowledge managemen, and are oen underesimaed by evaluaors. For many
reasons, policies are no always ollowed, and i is naive o expec anyhing dieren. Where specic
EHA guidelines do exis, hey should be made available o evaluaion eams. Failure o do his can
resul in sub-sandard and even unusable work being produced (Soddard 2005).
11
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
12/26
Ensure evaluaion processes are imely and orm an inegral paro he decision-making cycle
A common complain rom poenial evaluaion users is ha evaluaions oen arrive oo lae o be o
any use in decision-making. A recen repor on he way DFID learns rom research and evaluaions
noes:
Te mos common criicism o evaluaions among he inerviewees was
iming: alhough hey mark imporan rhyhms or Counry Oces,
hey were seen o generally ake oo long o be relevan o policy eams,
and insucien atenion is paid o ying hem ino policy cycles and
windows o opporuniy. ( Jones and Mendizabal 2010)
Tis is even more he case or humaniarian evaluaions. I is no unusual or evaluaions o nish
(or even sar) only aer he peak o he emergency is over. No only is signican analysis made
more dicul by he passage o ime, as he memories o key inormans wane and inormaion is
los, bu long delays beween programmes and evaluaion conclusions means ha here is hen no
opporuniy o use he ndings o change he programme concerned.
As a resul o concerns such as hese, here has been a growh in he number o real-ime evaluaions
(Es) by a range o humaniarian organisaions. A recen E o he Haii Earhquake (Groupe
UD 2010) was able o underake a eld mission wihin one monh o he earhquake, and ocapure lessons abou he immediae response. I was also able o eed back o acors in he eld, eiher
bilaerally or o he cluser groups.
Some evaluaion deparmens work o heir own imeables, planning and commissioning evaluaions
wihou regard o he managemen processes ha acually infuence programmes. Evaluaions ha
appear aer major decisions have been made are acing an uphill sruggle i hey are o have any impac.
o have a beter chance o bringing abou change, evaluaion imeabling should sar wih an analysis o
programme planning cycles, and ensure ha evaluaion producs eed ino his.
Emphasise qualiy no quaniy
Tere is nie capaciy wihin any organisaion o commission, implemen and hen learn rom
evaluaion. Some organisaions commission evaluaions or a large proporion o heir programmes,
bu hen nd hemselves sruggling o ensure he qualiy o he process. Even where qualiy is
mainained, real refecion on he ndings o an evaluaion repor akes ime, and considering and
implemening recommendaions is even more demanding.
12
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
13/26
ICC used o reques all ineresed paries o ideniy he range o quesions and issues hey would
like included in an evaluaion. Te evaluaion deparmen hen reramed his ino evaluable
quesions. However, i was ound ha he scope o he evaluaion always grew, unil i became diculo manage he process. o miigae his, he evaluaion deparmen now ries o ocus on jus hree
key quesions or each evaluaion.
Inerviewees repored ha hey eel ha oo many (oen poor-qualiy) evaluaions are being carried
ou, and ha his works agains he developmen o a rue evaluaive culure wihin an organisaion.
Where repors are o insucien qualiy, hey lose credibiliy, and he evaluaion process becomes less
valued in he eyes o managers and implemeners. When he same recommendaions go unheeded
ime aer ime, because o a lack o capaciy o debae and ac on hem, no one akes hem seriously.
A percepion develops ha he evaluaions are done only or appearances sake. Someimes, a qualiy-
ocused approach is made more dicul i evaluaions are required by donors as a unding condiion,
orcing organisaions o commission more evaluaions han hey can absorb. Donors can be similarly
overwhelmed by he repors ha hey commission.
13
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
14/26
Develop a sraegic approach o selecing wha shouldbe evaluaed
Some evaluaion deparmens review heir organisaions annual expendiure and plan evaluaions on
he basis o ensuring ha all major programmes are evaluaed every ew years. Te guiding principle
o such approaches is ha accounabiliy demands coverage o he organisaions aciviies on a
cyclical basis. However, such a mechanisic approach does no necessarily lend isel o ensuring
eecive uilisaion and impac o EHA.
Tere may be litle ha is conroversial abou large programmes ha have been running or years,
and evaluaions are unlikely o lead o any radical changes. Tere may also be all sors o barriers o
change in some areas, where evaluaion ndings are likely o be ignored. For accounabiliy reasons,
evaluaions are oen required o large-budge programmes. A mehodology ha is ligher on he
learning aspec migh be more appropriae when such programmes are unconroversial. (Tis
underlines he imporance o he need o clariy he purpose o he evaluaion a he ouse, as noed
in Secion 2 above.)
A sraegic approach o selecing wha should be evaluaed is undamenally dieren rom he cyclicapproach described above, and sars by looking a how he evaluaion process can add value. As a
good example, SIDAs evaluaion planning cycle sars wih he evaluaion uni having conversaions
wih all operaional unis, o deermine heir knowledge needs, wha hey would like o know and
how evaluaion could help. A lis o around 100 ideas or evaluaions is iniially generaed, rom which
he evaluaion deparmen chooses he 15 hey will carry ou.
Evaluaions no chosen are subjec o decenralised evaluaions, on which he evaluaion deparmen
gives eedback and advice. Sa members who proposed any o he 15 seleced evaluaions orm
reerence groups or ha evaluaion. Tey mus lis he inended use and users. I here are no enoughusers, he evaluaion is dropped. Oherwise, hese inended users are hen involved in draing
erms o reerence, and work wih he reerence group hroughou (rom a presenaion by Joakim
Molander, SIDA, a DFID workshop on lesson learning, 22 June 2010).
Involve key sakeholders hroughou he process
Evaluaions are poliical and concern various ineres groups (Carlsson e al, 1994, in Sandison,
2006) and sakeholders, boh inernal and exernal. Te mos imporan exernal sakeholders are
he aeced populaions hemselves, who should arguably be a he cenre o no jus he evaluaion
3 . E VA L U A I O N P O C E S S E SA N D S Y S E M S
14
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
15/26
process, bu he whole humaniarian sysem. Alhough here is sill a long way o go or his o
be achieved, a large number o humaniarian organisaions are working o improve he curren
siuaion. I is now slowly becoming he norm or beneciaries o be involved a some sage o heevaluaion process, alhough his is usually only as inormans raher han in seting he evaluaion
agenda. Noneheless, his represens an improvemen on he siuaion o a decade ago, when ew
humaniarian evaluaions involved srucured discussions wih he aeced populaion a all.
Despie recen advances in downward accounabiliy, he Humaniarian Accounabiliy Programmes
recen annual repor3 nds ha considerably more needs o be done wih respec o evaluaion
pracice. However, given ha he evoluion o EHA o dae has been driven largely by greaer demand
or upward accounabiliy o donors, eors o reorien he evaluaion process owards he concerns
o disaser-aeced sakeholders may require enirely new mechanisms: i may no be possible simply
o use exising mechanisms o ull new and dieren uncions.
Tis paper does no seek o duplicae discussions on humaniarian accounabiliy being conduced
elsewhere. However, as a minimum, all evaluaions should ensure ha beneciaries are enabled o
conribue, wih heir ideas eeding ino analysis, learning and recommendaions. A large lieraure
exiss on a range o mehods or obaining a valid specrum o beneciary views, which is no o say
ha his is easy o achieve. Tere remain very considerable challenges in ensuring rue paricipaion
in siuaions where power srucures preven evaluaors rom accessing cerain voices, ypically hose
o women and minoriy groups. Tis is exacerbaed in confic siuaions where expressing opinions
migh be no only dicul bu also dangerous.
Oher exernal evaluaion sakeholders include he general public, he media, various governance
srucures, and donors. FAO sysemaically involves sakeholders rom he beginning o hemaic
or counry evaluaions, using consulaive groups including aeced people, major donors and
programme personnel. Te sakeholders orm an advisory group, and build rus and buy-in
considered essenial by he FAO Evaluaion Uni in achieving change hrough evaluaion. Te group
ensures ha he righ quesions are asked, and has a chance o commen on dra producs.
esearch on srenghening learning rom research and evaluaion in DFID (Jones and Mendizabal,2010) ound ha increasing he sense o ownership o evaluaions by DFID sa members was
essenial. Where inerviewees had been involved acively wih he DFID Evaluaion Deparmen
in one way or anoher, his improved heir undersanding o he deparmen, breaking down any
misconcepions abou being audied, and was el o be valuable. On he oher hand, concerns
were noed ha lessons rom evaluaions relevan o DFID more generally, a a sraegic level, had
no owner, and were hus less likely o be aced upon. UNHC has a seering commitee or each
evaluaion ha includes sa members rom he relevan unis.
3. htp://www.hapinernaional.org/pool/les/a-he-2009-humaniarian-accounabiliy-repor.pd
15
http://www.hapinternational.org/pool/files/a-the-2009-humanitarian-accountability-report.pdfhttp://www.hapinternational.org/pool/files/a-the-2009-humanitarian-accountability-report.pdf7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
16/26
Use boh inernal and exernal personnel o encourage a culureo evaluaion
A review o he lieraure and discussions wih eld sa reveals a common heme o eld and
operaional sa being unhappy wih curren evaluaion pracices. Many eld personnel eel
ha evaluaions ake up valuable ime, and ha hey end up eaching he evaluaors abou he
programme and he issues, and hen subsequenly learn litle rom he repors, i hey ever see
and read hem. Furhermore, hey requenly nd ha nohing seems o change as a resul o he
evaluaion. Tere is also concern ha ousiders need a lo o ime o learn abou he culure and
pracice o he organisaion being evaluaed. ecommendaions can be unrealisic where his learning
has no aken place, damaging he credibiliy o he evaluaion.
A number o inerviewees menioned ha hey have sared using more inernal evaluaion, or
mixed eams o insiders and ousiders. Te involvemen o insiders means ha ndings and
recommendaions are more likely o be appropriae. In addiion, insiders are more likely o have a
beter undersanding o he concerns o eld personnel, and o heir perspecive on key issues. Using
inernal evaluaors also brings he enormous bene o reaining he experience and knowledge
gained by hose carrying ou he evaluaion. I is expensive o hire exernal evaluaors o learn huge
amouns abou he organisaion and he emergency response, only or hem o ake away his
knowledge when heir work is done.
Despie he many benes o using insiders, some organisaions are implacably opposed o doingso, earing ha his will limi independence and resric any radical recommendaions ha migh
be required on resrucuring he organisaions response. As noed above, i is essenial o clariy he
reasons or carrying ou an evaluaion, and o recognise ha rade-os are ineviable.
Improve he echnical qualiy o he evaluaion process
Te raionale behind his sudy on improving evaluaion uilisaion is ha here are many acors
aecing he impac o EHA, and ha echnical qualiy is only one o hese. However, echnical
qualiy is clearly very imporan. High-qualiy evaluaions increase he credibiliy o he wholeevaluaion process, and creae he poenial or a viruous circle o develop: i evaluaions are valued
more highly, his creaes he righ condiions or more o hem o be o higher qualiy in uure.
ALNAP is commissioning a new Guide o he Evaluaion o Humaniarian Acion as a separae
exercise, and so his imporan issue will no be discussed urher here.
16
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
17/26
Assign high prioriy o efecive disseminaion o ndings,including hrough new media (video, web)
Evaluaions are o poenial ineres o agency managers, beneciar y represenaives, local
governmens, donor ocials, sudens o humaniarian issues, and he press, as well as he general
public. Ideally, each audience should receive a dieren produc ailored o is needs (Hallam 1998).
However, increasing impac is also abou consrucing pahways or he evaluaion ndings o make a
dierence wihin he organisaion:
perorming a good qualiy evaluaion is only he rs sep. Te lessons
hen have o be absorbed, aken orward, and implemened in pracice
beore organisaional learning can be said o have occurred. (Soddard
2005)
I is oen assumed ha evaluaions yield inormaion or provide lessons learned ha fow direcly
back ino he policy cycle and are hus incorporaed in he planning o uure programmes and
projecs. In his way, here is a consan learning process leading o ever-improving perormance
(Frerks and Hilhors 2002). Tis undersanding presupposes a raional, scienic planning model
which has never been adoped in daily developmen pracice. In a plural, complex and disorderly
sociey, decisions on goals and programmes are oen poliical compromises ha do no necessarily
correspond wih he oucomes o evaluaion (Frerks and Hilhors 2002).
Sraegic disseminaion o ndings is key o making evaluaions more eecive. In a personal
communicaion, a senior evaluaor commened on he dierence in he impac o evaluaions he
had noed when moving rom a large and well-resourced evaluaion deparmen o a much smaller
one in a dieren organisaion. Despie he dierences in resources available, he evaluaions carried
ou by he smaller organisaion had signicanly more impac. Tere were several reasons or his,
bu he mos imporan reason was he smaller organisaions commimen o disseminaing he
resuls and argeing ha disseminaion eecively. Indeed, he evaluaion process began wih he
communicaion sraegy, raher han disseminaion being hough abou only once here was a
(perhaps inappropriae) repor in hand. Planning ahead infuences he ype o inormaion collecedhroughou he evaluaion, and ensures i mees he needs o decision-makers.
A recen ODI paper on learning lessons rom research and evaluaion ( Jones and Mendizabal 2010)
echoed he above ndings, and ound ha how and when evidence is produced, presened and
communicaed maters. Opions include argeed seminars and presenaions, one-o-one briengs
or eam leaders, an evaluaion deparmen newsleter or brieng papers, shor email producs, and
he developmen o new producs such as documens ha presen lessons rom evaluaions along
hemaic, regional/naional and programmaic/policy lines. Focusing on personal inerrelaions
raher han inermediaries was also considered imporan. Te same sudy ound ha he orma and
17
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
18/26
presenaion o evaluaions could be improved: Some inerviewees el ha he ull repors were oo
long and echnical... i is likely ha his is a common problem wih he evaluaion proession (Jones
and Mendizabal 2010).
Some agencies have sopped puting recommendaions ino evaluaion repors, because hey eel ha
ocus on recommendaions can derac rom analysis and learning. Te mos imporan par o he
evaluaion or hem is he analysis, because his is wha encourages learning. I is beter ha hose who
know and undersand he organisaion develop and ake ownership o he recommendaions. I is
also imporan ha evaluaion repors and relevan ndings are easily accessible. Ways o enabling his
include: good caaloguing; improving inranes o make i easier o search and browse, and making
hem SS compaible o allow sa o link o hem wihou having o access he sie every ime hey
wan inormaion; and producing global lessons learned rom mea-analyses.
Inerviews conduced or his sudy revealed some innovaive disseminaion sraegies. A number
o organisaions, including UNHC, ICC and Groupe UD, have made documenaries around
evaluaions, some reporing powerully on he key issues ound in he evaluaion solely in he words
o beneciaries. ALNAP isel produced a Lessons Paper on humaniarian aciviies aer earhquakes,
by reading and disilling he lessons rom a number o evaluaions o earhquake response (Cosgrave
2008). Tis paper was downloaded over 3,500 imes wihin days o he January 2010 earhquake in
Haii.
Ensure here is a managemen response o evaluaions
Managemen response and ollow-up o evaluaions is a key area or improving he impac o
evaluaions. Tere are numerous anecdoal repors o evaluaion repors gahering dus, largely
unread, and wih litle ormal or inormal ollow-up on ndings and recommendaions. Tis may be
parly because o problems noed above paricularly lack o clariy o purpose o he evaluaion,
and lack o involvemen by key sakeholders. Noneheless, ormal sysems o managemen response
o evaluaions serve o reduce he chance ha an evaluaion process ends wih he producion o he
repor.
Dieren organisaions ake dieren approaches o his issue. UNDP has creaed an Evaluaion
esource Cenre. Tis is a public plaorm, designed o make UNDP more accounable, where
evaluaion plans are logged along wih a managemen response. esponses and ollow-up are racked
by he evaluaion deparmen and repored o he Execuive Board o UNDP. Some DFID counry
oces hold an in day o go over perormance rameworks and evaluaion resuls, o make sure ha
key lessons are no los.
In FAO, here is a process in which senior managemen commens on he qualiy o evaluaion
repors, as well as on wha ndings hey accep and wha acions are planned o address hese.
18
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
19/26
Tis eedback is presened o he governing body along wih he evaluaion. In addiion, or major
evaluaions, here is a urher sep in he process: wo years aer he evaluaion, managers are required
o repor o he governing body on acion aken on he recommendaions hey acceped a he ime.For FAO, hese have proven o be powerul managemen ools, and which include he opporuniy o
revisi evaluaion ndings, and o have a dialogue abou he managemen response o hem.
Carry ou periodic mea-evaluaions and evaluaion synheses,and review recommendaions
Te huge demand ollowing he 2010 Haii earhquake or ALNAPs Lessons Paper esponding o
Earhquakes: Learning rom earhquakes relie and recovery operaions (Cosgrave 2008) has been
menioned above. I seems here is a signican demand wihin he humaniarian secor or such
mea-analysis and hemaic reviews.
Oxam has recenly sared o use he same six benchmarks or is inernal repors.
Every ew years, a mea-evaluaion is carried ou agains hese benchmarks which is
incredibly valuable (personal communicaion).
CAE has carried ou a mea-review o evaluaions and aer-acion reviews rom
24 emergency responses, and drawn ogeher key learning rom his.
Every year, NORD produces a synhesis repor o lessons rom evaluaions.
Te ODI paper on learning lessons rom research and evaluaions (Jones and
Mendizabal 2010) ound glowing reerences rom inerviewees o research andevaluaion ha oered synhesis and comparison o work rom around DFID on
paricular hemes or secoral areas.
Such mea-approaches and synheses are imporan in exracing ull value rom expensive evaluaion
processes. Tey help o ensure ha ndings across many dieren evaluaions are validaed and are
no specic o jus one projec. Greaer consisency o ndings across programmes leads o more
condence in heir credibiliy, and so o greaer poenial impac o making changes on he basis o
such ndings.
19
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
20/26
Improve monioring hroughou he programme cycle
Evaluaors requenly nd ha a lack o monioring daa signicanly impedes heir abiliy o make
meaningul commens on he eeciveness o a humaniarian programme. A imes, managers seem
o reques evaluaions o compensae or he lack o monioring inormaion. Tis is an expensive way
o obaining inormaion ha should be rouinely colleced and analysed.
Te lack o monioring daa is a long-sanding problem, due parly o he naure and circumsances o
humaniarian work: a mulipliciy o acors, he fuidiy o he siuaion, he diculies o working in
he conex o war and insabiliy, and he requen absence o baseline daa on he previous condiion
o he aeced/arge populaions. However, here are also atiudinal consrains, where emergency
personnel do no consider i a prioriy o se up monioring sysems rom he ouse. Tere is
also a lack o agreemen on sandardised monioring procedures and proocols among agencies,
despie widespread adopion by many humaniarian agencies o Sphere sandards. Indeed, several
inerviewees suggesed ha all humaniarian programmes should repor agains Sphere sandards.
Sandardisaion o daa collecing and reporing would make i easier or he sysem as a whole o see
where gaps exis in humaniarian provision. I agencies and donors could agree on wha daa shouldbe colleced, his would signicanly increase he likelihood o i happening, meaning ha evaluaors
could be more conden ha hey will nd he daa hey need o carry ou heir work. A disadvanage
o such an approach would be ha here migh be a endency o ocus on he measurable and pay less
atenion o conexual analysis, already a weak poin in some humaniarian evaluaions. Conex-
specic monioring would always be required as well, o ensure ha a programme was meeing is
saed objecives, alhough his would no preven sandardisaion in many areas.
Provide he necessary human resources and incenive srucures
Te incenive srucures o... agencies do no necessarily reward
hose in he evaluaion deparmens, which, as a resul, are no able
o ofer clear career opporuniies or saf members (Foresi 2007).
A WFP repor made similar commens abou career incenives or
saf in evaluaion deparmens: [evaluaion oce] poss are highly
sressul regarding relaions beween saf and oher members o he
organisaion Some el ha a posing in [he evaluaion oce] was a
bad career move and migh afec he individuals uure career. (Baker
a al 2007)
4 . S U P P O I N G P O C E S S E SA N D M E C H A N I S M S
20
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
21/26
Given such incenive srucures or lack o hem, i may be ha evaluaion deparmens will no
perorm a opimal levels.
As well as ormal incenive srucures, here are also many inormal incenives ha can eiher promoe
good evaluaion or ac as an obsacle o i. Such incenives can be suble o ousiders, bu play an
imporan role in deermining wheher evaluaions are likely o be used eecively. For example,
where managers demand an evidence base beore implemening new policies and programmes,
evaluaions are likely o be more highly valued.
A recen sudy o lesson learning rom DFID research and evaluaions (Jones and Mendizabal 2010)
ound ha he demand or an evidence base increased in circumsances where a rapid response was
required o an unexpeced even. . Under hese circumsances, he abiliy o personnel o use evidence
and lessons rom research and evaluaions wihin heir work undoubedly buil heir credibiliy and
infuence wihin DFID. In paricular, evidence ha proved he value or money o a paricular policy
area was highly prized, as his was useul o DFID leaders in srenghening heir case or increased
budges wih he UK reasury (underlining he imporance o messages rom he op).
However, in general, he sudy ound ha original hough and new ideas were more likely o be
rewarded han he use o proven ideas or lessons learned elsewhere. Oriening incenives owards
new hinking, whils undersandable, is likely o miliae agains he eecive use o evaluaion. Te
same sudy also ound ha ime pressures encouraged sa members o rely on experience, raher
han evidence based on research and evaluaion, o suppor heir argumens.
Te inormal incenive srucures wihin an organisaion are par o is culure, and merely changing
ormal srucures migh no ackle he underlying and less easily measured consrains o eecive
evaluaion. Srong leadership, which suppors he use o evidence and evaluaion, is imporan in
bringing abou change in such circumsances. However, i is also imporan no o neglec he ormal
incenive srucures and processes. Evaluaion sa should be rained o an appropriae degree, and
have access o coninuing educaion on evaluaion. Sa roaion hrough he evaluaion deparmen
should be encouraged and regular.
New members o sa should learn abou he evaluaion sysem when hey are induced ino he
organisaion. As discussed above, using a mix o insiders and ousiders in evaluaion eams can help
sa members o learn abou evaluaion, improve heir analyical skills and promoe heir accepance
o evaluaion. Programme planning processes should be designed so ha here is ime and space o
consider and debae evaluaion ndings, and or recommendaions (where acceped) o be adoped.
Secure adequae nancial resources
Alhough many evaluaions are driven by issues o cos-eeciveness, he coss and benes o
21
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
22/26
evaluaion isel are rarely analysed. esearch in his area could lead o a beter undersanding o
he concree benes o evaluaions, proving ha Doing humaniarian evaluaions beter hrough
an ongoing process o refecion and learning is he mos cos-eecive way or ensuring hahumaniarian programmes improve (Aphorpe e al 2001), and hus improving he saus o
evaluaion sa and deparmens.
Tere is a widespread percepion ha join evaluaions are prohibiively cosly, ye, provided hey
are well-designed and managed, heir benes may ouweigh hese coss. Many eld workers and
managers complain abou he coss o evaluaions, ye do no always sop o consider he poenially
huge nancial and human coss o coninuing wih unsuccessul programmes (perhaps because
evaluaions only rarely succeed in infuencing such programmes).
In he pas, some organisaions allocaed a xed percenage o programme coss o evaluaion
budges. While his was iniially useul as a way o generaing resources or evaluaions in
organisaions no used o doing hem a all, i is unlikely o be he mos cos-eecive way o using
scarce resources. Beter o ake a sraegic approach, wih proporionaely more resources going
o complex programmes, or programmes ha are more conenious and where evidence or
eeciveness is less readily available.
Undersand and ake advanage o he exernal environmen
Use peer neworks o encourage change
Peer neworks are an essenial componen o bringing abou change. Tis paper was commissioned
by ALNAP, a key nework on humaniarian evaluaion, in order o promoe peer learning around
evaluaion impac. Peers, by deniion, work in similar conexs, and so heir experiences can be used
by similar organisaions o increase heir own learning. ALNAP promoes peer learning hrough
workshops and publicaions, and visis o member organisaions.
Te evaluaion deparmens o he UN organisaions also have a peer grouping known as he UN
Evaluaion Group (UNEG). UNEG has is own peer review process, which involves visiing one ohe organisaions and assessing is evaluaion uncion agains a se o norms and sandards developed
by he group as a whole. Members have ound he peer review process useul or raising he prole o
evaluaion wihin heir organisaions, and helping evaluaions become more embedded wihin he
organisaions processes.
Some organisaions have sough o develop heir own peer groups. For example, FAO has now
insiuionalised peer reviews, heir Charer o Evaluaions demanding and budgeing or a biannual
peer review o he organisaions evaluaion uncion.
22
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
23/26
Engage wih media demands or inormaion
Te media have a endency o caegorise humaniarian acors as eiher sains or sinners. odaysmajor humaniarian responses are characerised by huge media ineres, wih journaliss scrambling
o be he rs on he scene and hen complaining abou he slow speed o he response, ignoring
he many consrains and challenges involved. Tis can lead o a deensive response by agencies
and unwillingness o acknowledge any problems or ear o criicism. Tis works agains a culure o
learning and improvemen.
Te UK DEC changed is policy on commissioning and publishing evaluaion sudies ollowing
negaive media repors aer he response o he Souhern Arica drough in 2002/03. Ironically, he
evaluaion repor was generally very posiive, wih relaively ew criicisms. Unorunaely, i was hese
criicisms ha were seized on by he press. An arguably more eecive response over he longer erm
would be a greaer openness and a willingness o accep ha problems occur, and o debae hem.
Tis, aer all, happens wih many oher public inervenions (healh service delivery, or example)
wihou he public losing suppor or he inervenion.
Ineresingly, he response o he 2004 sunami appeal he one immediaely aer he appeal or
he Souhern Arica drough discussed above was srongly suppored by he UK public, wih no
evidence ha any donaions were held back because o earlier media criicisms o he same agencies.
Agencies should also engage direcly wih he media o educae hem abou humaniarian response,
o avoid he unbalanced and unhelpul journalism ha can accompany a large-scale emergency.
Engage with donors on their evaluation needs
According o Frerks and Hilhors (2002), EHAs are more oen han no donor iniiaives driven
by donor values and ineress. I is, hereore, vial ha donors suppor eors o improve he
impac o EHA. A signican barrier wihin organisaions o developing an evaluaion culure
is ha evaluaions are oen seen as compliance insrumens (as discussed in Secion 2 above).
Organisaions commited o evaluaions only as a condiion or accessing unding are unlikely
o value or have ownership o he resuling producs. Consequenly, agencies may under valueevaluaions, which can represen poor value or money in erms o he acual learning and
perormance change ha hey bring abou.
A repor on DFIDs lesson learning rom research and evaluaions ( Jones and Mendizabal 2010)
ound ha, in counries ha were higher HQ and UK poliical prioriies, he prole o evaluaion was
enhanced. Tis mean ha recommendaions were more likely o be acceped (again demonsraing
he powerul infuence o clear leadership demands on pracice). I donors and agencies can agree a
sraegic approach, hen boh could bene rom carrying ou ewer, more argeed evaluaions ha
ocus on key issues raher han a predeermined lis o projecs and programmes.
23
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
24/26
Nex seps
Te purpose o his paper is o simulae debae on wha is required o increase he impac o
humaniarian evaluaions. From his iniial analysis, and subsequen workshops wih praciioners, we
hope o develop plans o build on his work and bring abou meaningul change.
Wihou doub, he way orward will include building sronger links beween evaluaion and
programme/managemen sa. I is undeniable ha he majoriy o he characerisics and
impedimens o organisaional change are beyond he conrol o evaluaion (Sandison 2006).
Evaluaion mus be placed and judged wihin a larger managemen ramework, or i we coninue
o expec evaluaion o cover mos o he accounabiliy and learning needs o he secor, we will be
disappoined (Sandison 2006).
Te nex ask will be o develop a sel-assessmen quesionnaire, so ha organisaions can see how
hey are doing in erms o addressing he issues ha infuence evaluaion uilisaion. Tis paper, as
well as he quesionnaire, will provide a ramework or generaing more deailed case sudies o how
organisaions have improved he uilisaion o humaniarian evaluaions. Chie execuives o ALNAP
member organisaions will be inerviewed, o nd ou wha inormaion senior decision-makers wan,
where hey ge his inormaion rom, and how evaluaions could beter serve heir needs.
I is vial ha we check evaluaors prioriies or change agains hose o operaional managers.
Wihou dialogue beween he wo groups, he poenial impac o evaluaion as a managemen ool
will no be realised. Te case sudies and he inormaion generaed rom urher in-deph inerviews
should signicanly enrich he ideas discussed in his paper, and will lead o a revised version laer in2011.
E F E E N C E S
Cosgrave, J. (2008) esponding o Earhquakes: Learning rom earhquake relie and recovery
operaions. ALNAP Lessons Paper. London: ALNAP.
Aphorpe, R., Boron J. & Woods (eds.) (2001) Evaluaing Inernaional Humaniarian
Acion: efecions rom Praciioners. London: Zed Press.
Baker, J. e al (2007) Peer eview: Evaluaion Funcion a he World Food Programme. SIDA.
Carlsson, J. e al (1994) Te Poliical Economy o Evaluaion. London: Macmillan Press.
Groupe URD (2010) eal-ime Evaluaion o he esponse o he Haii Earhquake o 12 January
2010. (available a www.urd.org).
Hallam, A. (1998) Evaluaing Humaniarian Assisance Programmes in Complex Emergencies.
elie and ehabiliaion Nework, Good Pracice eview 7. London: ODI.
24
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
25/26
Foresi, M. (2007)A Comparaive Sudy o Evaluaion Policies and Pracices in Developmen
Agencies. London: ODI.
Frerks, G. & Hilhors, D. (2002) Evaluaion o Humaniarian Assisance in Emergency
Siuaions, UNHC/EPAU: New Issues in eugee esearch, Working Paper 56.
Jones, H. & Mendizabal, E. (2010) Srenghening learning rom research and evaluaion: going
wih he grain. London: ODI.
Mayne, J. (2008) Building an Evaluaive Culure or Eecive Evaluaion and esuls Managemen.
Insiuional Learning and Change Iniiaive (ILAC) Brie 20. (available a www.cgiar-ilac.org).
Paton, M. (2008) Uilizaion-Focused Evaluaion. London: SAGE Publicaions.
Proudlock, K. (2009) Srenghening Evaluaion Capaciy in he Humaniarian Secor: A
Summary o Key Issues. (auhors unpublished noes).
Sandison, P. (2006) Te Uilisaion o Evaluaions in ALNAP eview o Humaniarian Acion:
Evaluaion Uilisaion. London: ODI.
Soddard, A. (2005) eview o UNICEFs Evaluaion o Humaniarian Acion. New York: Unied
Naions Childrens Fund.
WFP (2009) Closing he Learning Loop Harvesing Lessons rom Evaluaions: epor o PhaseI. ome: World Food Programme.
25
7/31/2019 Harnessing the Power of Evaluation in Humanitarian Action: An initiative to improve understanding and use of eva
26/26
Overseas Developmen Insiue111 Wesminser Bridge oadLondon SE1 7JD, UK
el: + 44 (0)20 7922 0300Fax:+ 44 (0)20 7922 0399Email: [email protected]
ALNAP Working Paper
Top Related