WORLD HEALTH ORGANIZATION Draft ReportWHO/HQ Geneva – Dr. Sasha Goubarev

WHO/SEARO & WHO/Nepal 

Presented by Karen GladbachContributions by Arie Rotem & Amitai Rotem

U.N. Senior Fellowship Officers Meeting 2-4 November 2010 – Paris

Impact Evaluation of WHO FellowshipsNepal Pilot Study

The Journey begins . . .

Why Nepal? Supportive team: WHO/SEARO & WHO/Nepal, home & host

institutes• Quantity sample: 55 long-term Fellows (3 months+) • Supporting FEL documents: readily available• Core institutes: less diversity

The pilot study aimed to:

1) Evaluate the impact of WHO fellowships in Nepal: 2004-2008

2) Derive lessons from Contribution Analysis to evaluate impact

3) Assess availability + usefulness of existing FEL information from national & international fellowship authorities

4) Develop and/or refine tools and protocols to obtain

evidence

The Study Sample

Step 1 – AnalysisReviewed all 55 long term fellows--three cohorts (2004-

2008)

Analysis of existing information available in WHO files

Step 2 – InterviewsFace-to-Face (F2F) interviews with 26 FellowsFocus groups with fellows, peers and home institutes Kathmandu , Dharan, and districts in Chitwan and Pokhara provinces

Step 3 – Final Analysis and draft report

Milestones: FONTS

F – Future: What is Fellow expected to do with training post-Fellowship?

O – Objectives: Are objectives SMART? Will they enable Fellow to meet post- Fellowship expectations?

N – National Priorities: Are national priorities clearly defined and known by Fellow, training & home institute? Was the Fellow the right choice?

T – Training: Is the training institute aware of and able to address the training objectives aligned with national priorities? Will they conduct their own post- training evaluation? Are they aware of post-Fellowship expectations?

S – Sharing: How will the Fellow share the new knowledge and skills? How will this sharing lead to impact?

Levels of Impact

Sustainable Impact: Created a new unit, established new services, applied new procedures , major transfer to others, sustainable capacity building, likely to be sustained beyond the fellow’s direct involvement

Considerable Impact: Significant improvement of competence leading to better performance and better outcomes, sharing with others

Some Impact: Gained skills, some opportunity to apply or share with others

No Immediate Impact: No significant gain in capabilities

and/or inability to apply in Nepal context

Assessment of Impact

Based on this analysis it has been concluded that:

One in five fellows interviewed (19%) made sustainable impact as a result of their fellowship training

Two in five (42%) demonstrated a considerable impact

One third (31%) had at least some impact

In two cases (8%) no impact was evident due to inappropriate placement that led to incomplete program

Impact rating # %

Clinical specialists

Senior managers

District level officers

Clinical academics

Non clinician academics

Sustainable 5 19 3 1 - 1 -

Considerable 11 42 1 2 3 5 -

Some 8 31 1 2 1 1 3

No immediate 2 8 2 - - - -

Total 26 100 7 5 4 7 3

Overall Impact Ratings Interviewed Nepal Fellows 2004-2008

Impact

Sustainable Impact: Established new Intensive Care unit in a major hospital Contributed to development of new medical education model

Considerable Impact: Introduced new techniques for testing and preparing antigens Improved the use of phototherapy in treatment and research Taught new methodologies to colleagues and/or students

Some Impact: Managed investigation and surveillance of local cholera

outbreak, applying new surveillance techniques

Impact

No Impact: Senior neurologist placed at an institution that could not

provide advanced training -- Fellow returned home early with no gained capacity

Surgeon not able to complete study in urology due to budget restriction -- Fellow not recognised as urologist so could not to apply new knowledge and skills

Equally important to learn from no impact cases

WHO Findings

1) Current information collected by WHO has the potential to address data needs to assess fellowship impact

Fellowship Application Form: Fellow’s details, study objectives, endorsements

Letter of Award: relevant information on training program & fundingTermination of Studies Report: Fellows & host institutes’ assessment of

trainingUtilization of Studies Report : Fellow & supervisor’ assessment one-year

laterFellows in-depth written reports: Required by some home institutes

BUT forms are forms – the quality and depth of responses vary

WHO Findings

2) Face-to-Face (F2F) Interviews are crucial Better understand complexity of individual cases

Elicits potentially sensitive information Offers richer, in-depth ‘st ories’ of the journey

3) Debriefing important at various intervals Post-Fellowship (1-3 years)

4) Significant contributions were made in strengthening institutional capacity and provision of services

Recommendations

1) Provide briefing of national needs to training institutions

2) Require progress and post-training reports from training institutions

3) Require fellows to prepare detailed and insightful termination and follow up reports

4) F2F debriefings on utilization of training with a sample of fellows 1 -3 years post- fellowship

5) Review barriers to utilization and provide further support, when feasible and appropriate

6) Develop Fellowship alumni network

What Next?

What can we do now to improve our FEL program?

• Gain Fellows’ commitment to follow-up evaluations (1-5 years)At program onset, include post-training evaluation commitment in application

• Incorporate interviews into FEL program, not as ‘add-on’ activity but integral componentSelect cohort of 20 each year

• Share information – ‘get the message out’ to stakeholdersPackage’ according to audience: Ministry / government, Home institutes, Host

institutes

KISS Keep It Simple Sweetheart

The Journey Continues . . .

What’s your next step?