Humanitarian Indicator Tool (Dr Vivien Walden, Oxfam)
description
Transcript of Humanitarian Indicator Tool (Dr Vivien Walden, Oxfam)
Improving quality of humanitarian programmes through the use of a scoring system: the Humanitarian Indicator Tool
Dr Vivien Margaret Walden and Nigel Timmins
Page 2
Acknowledgements
• We wish to acknowledge the consultants, Andy Featherstone, Peta Sandison, Marlise Turnbull and Sarah House who helped refine the tools and the 10 countries offices who have been through the process and have hopefully come out of it unscathed.
• Photographs used in this presentation are by Caroline Gluck, Oxfam
Page 3
Why a new tool?
• In 1995, the ODI evaluation of the response to the Rwanda crisis raised concerns around the quality of service delivery
• As a result of findings, the Sphere project was started
• In 1997 the People in Aid Code of Best Practice was published
• In 2000 the Humanitarian Ombudsman became HAP
• In 2001 Griekspoor and Sondorp wrote a paper highlighting the various quality assurance measures and posed the question “Do all of the developments in the field of accountability and performance actually improve overall performance?”
Page 4
Why a new tool?
• Oxfam’s hypothesis – by improving the quality of a humanitarian programme, the likelihood of positive impact is increased
• The tool does not try to prove the link
• Oxfam has introduced the Global Performance Framework as part of reporting to DFID
• The framework has global indictors of which the Humanitarian Indicator is one
• Whereas the other indicators measure impact, the humanitarian indictor looks at quality of the response
Page 5
Oxfam’s Global Performance Framework
Livelihood Enhancement Support:# of women and men directly supported to increase income via enhancing production and/or market access
Humanitarian Assistance:Total number of people provided with appropriate humanitarian assistance, disaggregated by sex
Campaigning and Advocacy:#of campaign actions directly undertaken or supported, e.g. contacts made with policy targets, online and offline actions taken, media coverage, publications, etc.
Citizen Mobilisation:# of a) citizens, CBO members and CSO staff supported to engage with state institutions/other relevant actors; and b) duty bearers benefiting from capacity support
Adaptation and Risk Reduction:# of people supported to understand current and likely future hazards, reduce risk, and/or adapt to climatic changes and uncertainty, disaggregated by sex
Women’s Empowerment:# of people reached to enable women to gain increased control over factors affecting their own priorities and interests
% of supported households demonstrating greater income, as measured by daily consumption and expenditure per capita
Improved quality of life for poor
women and men
Degree to which selected interventions have contributed to affecting outcome change, as generated from findings of rigorous qualitative evaluations
Degree to which selected interventions have contributed to affecting outcome change, as generated from findings of rigorous qualitative evaluations
% of supported households demonstrating greater ability to minimise risk from shocks and adapt to emerging trends & uncertainty
% of supported women demonstrating greater involvement in household decision-making and influencing affairs at the community level
Degree to which humanitarian responses meet recognised quality standards for humanitarian programming (e.g. Sphere guidelines)
Global Output Indicators Global Outcome Indicators
Degree to which selected interventions meet recognised standards for accountable programming
Extent to which selected project delivery good value for money
Page 6
Page 7
The indicator
• Output indicator – the total number of people provided with appropriate humanitarian assistance, disaggregated by sex
• Data collected annually through online system
• Outcome indicator – the degree to which humanitarian responses meet recognised quality standards for humanitarian programming
• Data collected through HIT
Page 8
The tool – for rapid onset emergencies
Global Humanitarian Indicator: Degree to which humanitarian responses meet recognised quality standards for humanitarian programming
RAPID ONSET EMERGNECY – EARTHQUAKE, SUDDEN FLOODS, TSUNAMI, CYCLONES, TYPHOONS, HURRICANES, SUDDEN CONFLICT WITH DISPLACEMENT, AWD OUTBREAKS Number Quality standard Met
(score6) Almost met (4)
Partially met (score 2)
Not met (score 0)
1 Timeliness - rapid appraisal/assessment enough to make decisions within 24 hours and initial implementation within three days
2 Coverage uses 25% of affected population as an planned figure (response should reflect the scale of the disaster) with clear justification for final count
3 Technical aspects of programme measured against Sphere standards
Number Quality standard Met
(score3) Almost met (score 2)
Partially met (score 1)
Not met (score 0)
4 MEAL strategy and plan in place and being implemented using appropriate indicators
5 Feedback/complaints system for affected population in place and functioning and documented evidence of information sharing, consultation and participation leading to a programme relevant to context and needs
6 Partner relationships defined, capacity assessed and partners fully engaged in all stages of programme cycle
7 Programme is considered a safe programme: action taken to avoid harm and programme considered conflict sensitive
8 Programme (including advocacy) addresses gender equity and specific concerns
Page 9
The benchmarks • Timeliness - rapid appraisal/assessment enough to make
decisions within 24 hours and initial implementation within three days
• Coverage uses 25% of affected population as an planned figure (response should reflect the scale of the disaster) with clear justification for final count
• Technical aspects of programme measured against Sphere standards
• MEAL strategy and plan in place and being implemented using appropriate indicators
• Feedback/complaints system for affected population in place and functioning and documented evidence of information sharing, consultation and participation leading to a programme relevant to context and needs
Page 10
The benchmarks • Partner relationships defined, capacity assessed and partners
fully engaged in all stages of programme cycle
• Programme is considered a safe programme: action taken to avoid harm and programme considered conflict sensitive
• Programme (including advocacy) addresses gender equity and specific concerns and needs of women, girls, men and boys and vulnerable groups
• Evidence that preparedness measures were in place and effectively actioned
• Programme has an advocacy/campaigns strategy and has incorporated advocacy into programme plans based on evidence from the field
Page 11
The benchmarks
• Country programme has an integrated approach including reducing and managing risk though existing longer-term development programmes and building resilience for the future
• Evidence of appropriate staff capacity to ensure quality programming
Page 12
The methodology
• Done by an external consultant – although preferably one who has a knowledge of Oxfam
• Done as a desk study using documentation and some telephone/Skype interviews
• Follows a pre-determined scoring system and list of documents
• Has to be commented on and accepted by the country
• The country writes a management response
• All reports and a summary for each are published on the Oxfam website – www. Oxfam.org
Page 13
The scoring
Quality standard Evidence needed Met (score 6) Almost met (4) Partially met (score 2) Not met (score 0)
3 Proposals MEAL strategy and plans PH and EFSL strategies Technical adviser visits Training agendas and presentations LogFrames and monitoring frameworks donor reports RTE and other evaluation reports learning event or review reports
Sphere standards proposed and put in place with adjusted standards for context Training in standards carried out for staff and partners Indicators use standards and monitoring against standards takes place regularly Standards evaluated
NA Sphere standards proposed and adjusted to context Standards mentioned in proposals and LogFrames but not monitored against Some evidence of training but not widespread (staff but not partners or only in one area)
Standards only mentioned in proposals but not replicated in plans Or No mention of Sphere in any document
Page 14
Instructions for use
Benchmark Evidence Quality check Benchmark 3 Technical aspects of
programme measured against Sphere standards
Proposals MEAL strategy and plans PH and EFSL strategies Technical adviser visits Training agendas and presentations LogFrames and monitoring frameworks donor reports RTE and other evaluation reports learning event or review reports
Check proposals and strategies to see if standards are mentioned not just as a possibility but that they are considered in the context of the response – this might mean that Sphere has been adapted to suit the context The indicators on the LogFrame for technical areas should reflect Sphere standards The MEAL strategy should have Sphere as indicators and for data collection methods Check adviser reports for mention of standards and how these were implemented Check the RTE report for mention of Sphere standards Check WASH and EFSL strategies and adviser reports to see if any training was carried out for staff and partners Check review and evaluation reports for mention of standards
Page 15
Final score
• First year total score could be 30
• Adjusted for non-applicable benchmarks
• First year scoring
• Somalia – 17/28
• Kenya – 24/30
• Ethiopia – 9/28
• Pakistan – 19/30
• Colombia – the test case for the HIT – 18/26
• Second year – only South Sudan is complete – 21/30
Page 16
The findings – Somalia
• Benchmark 5 on accountability
• Score Met – 2/2
• “Oxfam’s partners appeared to differ in the level of beneficiary participation in design and delivery. Some documented highly participatory process, with qualitative and quantitative data. As well as gathering information, rapid assessments were done to establish VRCs to improve participation (criteria, entitlements, payment points, registration, complaints and feedback). Mobile phone hotlines were set up where possible, with feedback protocol to guide staff on how to register and follow-up complaints”
Page 17
Pakistan
• Benchmark 7 – protection and gender
• Partially met – ½
• Some protection concerns were identified relating to security for staff and women and girls using WASH facilities. Some actions were taken responding to dignity and protection including involving women in different activities. Post-distribution monitoring investigated some security concerns related to cashing cheques and distribution points, including analysis of responses from women. No protection problems observed were communicated to agencies or authorities responsible for, or specialising in, protection
Page 18
Ethiopia
• Benchmark 11 – advocacy
• Not met – 0/2
• Advocacy activities were clearly part of the intended response and there is a regional advocacy action plan with Ethiopia objectives and a media, advocacy and campaign strategy which includes a number of plans for Ethiopia. However, no Ethiopia country advocacy strategy was provided for the evaluation. The sitreps do mention Oxfam’s participation in influential meetings, but are not tied to an explicit strategy. There is no record of the impact of Oxfam’s advocacy activities
Page 19
Lessons learnt
• There are limitations to doing a desk study – it relies heavily on documentation and scores do not reflect absence of documentation or actual absence of good programming
• Several country teams objected to being judged solely on “little pieces of paper”
• There is no opportunity to get the views of the affected population unless this is already documented
• Telephone/Skype interviews can be biased – it is sometimes difficult to triangulate
• The process needs the goodwill and buy-in from the country team
Page 20
Advantages
• The process is fairly inexpensive – under £6000
• The country does not have to host a consultant
• The methodology can be used to track progress in subsequent responses in one country
• The scores are comparable across programmes (although context should be considered)
Page 21
She deserves the best we can give – thank you!