CONTRIBUTION ANALYSIS AS AN APPROACH TO IMPACT EVALUATION OF AN AFTERSCHOOL CARE PROGRAM IN LOW-INCOME COMMUNITY SETTING

BY ROY PONCE

Overview

This paper argues that contribution analysis can be used as a framework for impact evaluation of an afterschool care program. It provides a rationale as to the connection of the problem being addressed by the program to the development of its program theory. This paper also examines the realist approach of identifying the causal mechanisms in the program’s context that may establish practical generalizability and transferability of the program. It then sets out to provide the steps for implementation of contribution analysis in the chosen setting.

INTRODUCTION

Objectives and Structure of the Paper

Impact evaluation determines the evidences of causal links between an intervention and its outcomes. However, the concept of causation draws much philosophical debates among evaluation research circles. To an extent, these debates are beneficial because they lead to studies of diverse methodological approaches that are available for evaluation practice. One of these methodologies, called Contribution Analysis, is looking at the contribution of an intervention program to outcomes as a way of establishing causation when direct attribution approaches are not feasible.

This paper argues that contribution analysis can be used as a framework for impact evaluation of an intervention program. It shows that determining the program theory and the causal mechanisms of an intervention program within its context could help establish the practical generalizability and transferability of the program. This paper analyses the case of a pilot afterschool care program in a low income community to demonstrate these aims.

The discussion begins with a description of the program under study detailing its design, problem and context. The rationale of contribution analysis as a framework for impact evaluation study then follows. Within the contribution analysis framework, the program theory and causal mechanisms are discussed. The paper concludes by providing the implementation steps to conduct the impact evaluation plan using contribution analysis.

1

THE EVALUAND

Background of the Evaluand

The Happy Fish Kids (HFK) is a pilot afterschool care program that aims to keep children in school in a fishing community in Mindanao, Philippines. It provides afterschool care intervention as a strategy to prevent kids from poor families drop out of school. It is a social intervention that builds on human capital among disadvantaged children by assisting them to fully utilise available formal school services provided by the government.

Afterschool care programs are popular and have been practiced for decades in developed countries like Australia and the United States. Several studies demonstrate the impact of the intervention on disadvantaged children in these countries. Howes, Olenick and Der-Kiuregian (1987) have shown the complementary effects of afterschool care program to formal school in the social development of kids. The time spent by low-income children in afterschool care programs was positively correlated with academic and conduct grades, peer relations and emotional adjustments (Posner & Vandell, 1994). In the United States, the federal government has shown growing interest and strong support for afterschool care program provision for low-income children (Halpern, 1999).

The funds for the establishment of the HFK were donated an Australian organisation. The afterschool care support provided to children in Australian afterschool care centres is a model for the provision of similar services to poor children in a low-income community in the Philippines. However, there is a marked difference in terms of purposes and context of implementation. In particular, the HFK is designed to operate in a context of limited resources.

In developed countries, afterschool care programs are established primarily as an after school hours engagement of children while waiting for parents or guardians to pick them up from school. They provide various skills development activities for kids in children this 2-3 hour waiting time. These activities could be in the areas of sports, arts, music or academic follow-up such as tutorial and remedial activities.

The HFK was established as a preventative program for children dropping out of school particularly children from low-income families. This intervention implements motivation strategies for children to stay in school, children who would otherwise drop out of school due to lack of social support or to pursue opportunities to earn a living. These motivation strategies include program activities such as daily adult supervision for assignment follow-up; tutorials and remedial lessons; provision of basic school materials; school attendance and participation monitoring; and weekend mentoring and fun activities in the afterschool care centre. The centre is equipped with learning materials such as textbooks, encyclopaedia, dictionaries and children’s story books, toys and game boards and audio-visual equipment. It opens daily after school hours from 4 – 7 in the afternoon. The centre is managed by two full time female staff selected from the community and who have at least a college level qualification. Youth volunteers from tertiary schools in the city also assist during weekends and organise special activities as part of their school leadership programs.

The number of children participating in HFK increased from 37 at the start of its operations in 2008 to 147 children at present. This is approximately 68% of the elementary school enrolment in the community. Twenty (20) HFK children who completed the program (‘finishers’) have progressed to

2

high school and have 100% attendance at school. The program design is then expanded to continue to engage these HFK finishers. For example, school fare subsidies are provided conditional on their participation in mentoring and monitoring activities.

The Problem: focus and scope of the intervention

The problem that the program intends to address is the prevalence of children not attending school on a regular basis leading to low school completion rate. The Philippines still has a national average drop-out rate of 9% and only 73% completion rate in elementary schools. What are the reasons for these school outcomes? An Asian Development Bank report on education outcomes in the Philippines ranks ‘lack of personal interest’ and ‘high cost of education’ as the main reported reasons for children aged 7 - 12 not being in school (Maligalig, Caoli-Rodriguez, Martinez, & Cuevas, 2010). This means that the problem of school drop-outs is tied to the children’s lack of schooling motivation and poverty.

Literature supports the link between low schooling motivation and poverty (Haberman, 1991; McInerney, 2001). A further situation analysis could reveal more reasons and could be very useful in developing the program theory (Funnell & Rogers, 2011). The lack of financial and material resources needed by a child to confidently participate in school affects motivation. Worse can be expected when basic family needs are not met such as food, shelter and clothing.

The children’s motivation to attend school is also influenced by parental attitudes towards schooling. Most parents of families experiencing poverty have low education which could be a reason that the parents themselves do not motivate their children to attend school. Often parents in fishing communities prefer children to be helping by fishing for daily subsistence rather than spending time in school. Another possible cause is the opportunity cost of going to school for children who can work. Child labour is found to be associated with low school enrolment and these children are in the bottom 30% of the country’s income groups (Maligalig, et al., 2010). In fishing villages, children are recruited as members of a fishing boat crew since they can work at low wages or smaller share of fish catch. For a child who works, daily earnings could mean food for himself or the family while attending school will seem like a loss of earning opportunity.

The supply side of education could also be a source of lack of motivation for children to go to school. Most schools in remote villages only have the basic classroom equipment consisting of chalkboards and chairs. Textbooks are in short supply and there are no functional libraries. Student services and facilities are inadequate which makes schools unattractive. Teacher motivational competencies and commitment could also be considered as possible contributing factors to this problem.

The climatic conditions, weather cycles and livelihood patterns are also possible barriers to motivation for children to attend school. There are months of heavy rains and typhoons that prevent children from going to school. In addition to these natural forces are unsafe roads and creeks and lack of suitable transport and weather protective clothes. The livelihood patterns in fishing villages are closely aligned with the monsoon seasons. There are seasons of good fish catch which may revive interests in school. However, declining marine resources are observed due to poor coastal resource management so more and more fishing effort is needed to meet subsistence catch. This in

3

turn requires more resource with children providing the additional labour. Seasonal droughts also affect the copra industry along coastal communities. Poor coconut farm yield means less income from parent labour. This needs to be supplemented by children working.

Thus, the problem of school drop outs among children is tied to intrinsic motivation among children, lack of parental support, poor school learning conditions and erratic livelihood patterns in the community. Why is there a need to find a solution to this problem of schooling motivation? Poor school motivation leads to lack of education which simply perpetuates the problem of poverty, passing on poverty from one generation to the next.

Looking at the problem, its possible causes and consequences, the HFK intervention is only a small solution compared to the problem’s magnitude. It addresses only one aspect that of motivating children to attend school with a focus on providing the basic school needs for children along with adult contact to communicate with them and highlight the need for schooling. However, the HFK’s program scope includes the involvement of the community leaders, parents and teachers as their participation in solving the problem is a parallel condition for effective program delivery.

The HFK program outcome of 100% HFK finishers continuing to secondary education cannot be totally attributed to the HFK intervention. However, it can be argued that the program has contributed to this outcome. Hence, there is a need to provide a case for contribution, a contribution story of the HFK.

The HFK Stakeholders

A diagram of the relationships between the HFK stakeholders is shown in Figure 1. The main steering players are the Social Network Group, the HFK Administration and the HFK Centre Team. The Social Network Group called the Global Filipino Social Innovators is an internet based group of mostly overseas Filipino workers and migrants who support the program in terms of advocacy and funding. A core team of this network group composes the HFK Administration that directs the activities of the program being implemented by the HFK Centre Team. This means that the HFK Centre Team has direct contact with the program participants. These are the pre-schoolers, elementary and high school children. The HFK Centre Team is also in contact with the community stakeholders such as the parents, the formal school teachers and community leaders. Other organisations and individuals such as academic institutions, professionals and volunteers provide various services to the program and directly engage with the HFK Administration and the HFK Centre Team.

The diagram helps identify the interplay among the stakeholders and how they directly or indirectly affect all possible expected and unexpected outcomes of the intervention. This relationship map is an important tool in conjunction with the program theory to understand and provide a contextual framework of the underlying program mechanisms.

The Replication Questions

There is a perceived need to replicate the program in the neighbouring communities. The news of provision of school materials, follow-up activities and access of children to the HFK centre spreads throughout the nearby communities. Some neighbouring community leaders have explicitly

4

requested that the intervention be implemented in their communities. In support, the communities were willing to offer some form of counterpart and community volunteers. This implies that the provision of HFK in the pilot community is perceived as a ‘social disparity’ of services and benefits provided. The HFK Administration and Social Network Group considered a replication plan and posed the following: ‘Is the program replicable?’; ‘Will the program work in a different context?; and most particularly, ‘Is the HFK making a difference in the children’s schooling participation?’.

The questions raised are indeed impact evaluation questions. The questions not only point to causal issues but also to issues about validity and generalizability of the causal claims of intervention. This has become complex as the HFK started as out as a pilot program without counterfactual considerations in its design. This is due to financial constraints and lack of evaluation foresight.

To add to the complexity, the existing intervention has changed. The delivery of program activities is continuously adapting to the demands of growing participation of children. For example, with only a small number of starting children in 2008, the services were confined to daily assignment follow-up of children. The learning resources were few and other related activities were not yet available. At present, there are more than a hundred children participating. This has reduced the number of contact days with children because of scheduling due to lack of centre space. The benefits, however, are that the learning resources like the library are improving and the number of volunteers are increasing with more frequent additional activities like children’s camps, health services and mentoring activities. Hence, impact evaluation methods such as randomised controlled designs are not suitable.

5

Figure 1 The Happy Fish Kids After-school Care Program Stakeholders

6

Social Network Group

Academic Institutions and

Other organisations

Professionals

HFK Administration

HFK Centre Team

Community Leaders

Parents and Guardians

Pre-school and Elementary Kids

High school Kids

Formal SchoolAdministration and

Teachers

Manages networking, marketing, sourcing of funds

and other resources.

Provide learning and other resources inputs.

Provide professional services such as educational

and health services.

Sets program direction and administrative control.

Makes available the learning resources and delivery of

services. Monitors and reports activities. Provides the demand-

side support to government educational and development

programs.

Volunteers

Provide specialized services such as camps, music and

arts workshops.

Provide community clearance, coordination and

feedback.

Supports program participation of children and

other counterparts.

Participate in the program to avail resources and

services available.

Participate in the program to avail resources and services available.

Mentoring elementary kids.

Provide the government supply-side responsibility for

educational development.

Visual sample: Boxes refer to the various stakeholders and players of the program The green boxes show the primary program beneficiaries Dashed text boxes refer to targeted and ideal functions Red boxes and red directional lines refer to the programs major players, intended interventions and direct control Thin blue lines refer to interactions which may include expected and unexpected outcomes

CONTRIBUTION ANALYSIS

The HFK replication questions lead to consideration of alternative approaches that can explore causation and at the same time address issues of generalisability to provide information to the HFK replication plans. Contribution analysis is one of these alternatives.

What is Contribution Analysis?

Contribution analysis, advocated by John Mayne (2001), explores attribution through assessing the contribution a program is making to observed outcomes. The approach of contribution analysis is to verify the theory of change behind the program while taking into consideration other factors. Contribution analysis infers causality by identifying if: (1) the program is based on a reasoned theory of change; (2) there is program implementation; (3) there is evidence of the occurrence of the chain of expected results in the theory of change; and (4) the consideration and recognition of all other factors that may contribute to the observed outcomes.

The aim of contribution analysis is not to provide a proof of one-on-one linear causal linkage between an intervention and its outcomes, nor to produce the exact magnitude of the contribution. Its aim is to determine the logic-based ‘contribution story’ of the intervention that provides evidence ‘beyond reasonable doubt’ (Lemire, 2010). This approach is gaining acceptance among funding agencies. For example, contribution analysis was introduced in 2005 to the AusAID funded Fiji Education Sector Program (Kotvojs, 2006). In the context of the HFK, if the children stay in school and move from one grade level to the next, then what is the contribution story of the HFK to this success? Contribution analysis thus provides an opportunity to examine the impact of HFK as an alternative to the rigorous requirements of randomised controlled design, which was not considered anyway when the program was implemented.

Internal Validity and Generalizability

Will contribution analysis address the issues of internal and external validity? That is, to what extent does contribution analysis account for the influence of external factors and to what extent can the conclusions of a contribution analysis can be generalised.

The internal validity issue can be viewed from two different perspectives. One is when contribution analysis is referred as a method. In this view, contribution analysis makes valid conclusions on the basis of logical arguments. This is analogous to defending or advancing a case in court by providing evidences through reasoned statements. When conclusion follows from a sound logic of linking arguments to conclusion, the evidence and the methods of evidence accumulation could come from any suitable approach whether it is of qualitative or quantitative nature, experimental or non-experimental.

The other way of looking into validity issue in contribution analysis is the way causal claims are made by the intervention to the outcomes. This internal validity issue can be related to the requirement of contribution analysis to provide evidence of ‘recognition of all other factors’. This essentially refers to the consideration of ‘other alternative explanations’ that may have caused the outcomes rather than the intervention. Then it can be argued that the Campbellian ‘threats to internal validity’ list

7

may be useful and the method of ruling out alternative explanations until outcomes are linked to plausible causes (Shadish, Cook, & Campbell, 2002). The contribution analysis framework does not prescribe specific approaches to this matter. It could even push further to employ quasi-experimental approaches when possible to build up the evidences of the contribution story.

Furthermore, since the development of program theory is a primary task in contribution analysis, the program theory also requires a form of internal validity interpreted by Funnel and Rogers (2011)as demonstrations of the following: the clarity of description; the outcomes chain as the organizing principle; the demonstration of how the desired outcomes relate to addressing the problem; plausibility of the logical argument; and the articulation of mechanisms for change that underpin the outcomes.

The external validity or generalisability of contribution analysis is also embedded within the program theory. Funnel and Rogers (2011) argue that external validation is about how a program can stand up in relation to external sources of evidence, the context within which it operates and its intended uses. This means that a causal claim has external validity if the program is consistent with accepted theories and whether external conditions and mechanisms have been addressed in the program. This argument is in line with the realist thinking that generalizability can be achieved by understanding the mechanisms why the program worked and in what contexts. A program theory with realist view is recognised by Funnel and Rogers (2011) in the form of ‘realist program theory matrix’. How is this possible? Pawson and Tilley (1997) argue that transferability, that is program replication, can only be generalised once program outcomes as regularities are understood from the causal mechanisms that are at work on certain conditions or contexts.

Hence, this version of contribution analysis as an application to examining the impact of the HFK afterschool care program is an example of combining causal strategies. With the contribution analysis as the overarching methodology, it integrates the concepts of program theory, the Campbellian methods of controlling for threats to internal validity and the realist approach of identifying generative causal mechanisms within the intervention context.

Application of Contribution Analysis to HFK Afterschool Care Program

The implementation of contribution analysis is straightforward as defined by the following steps outlined by Mayne (2001):

Step 1: Set out the attribution problem to be addressed;Step 2: Develop a theory of change and risks to it;Step 3: Gather existing evidence on the theory of change;Step 4: Assemble and assess the contribution story and challenges to it;Step 5: Seek out additional evidence;Step 6: Revise and strengthen the contribution story.

The steps are iterative in nature so as to refine and strengthen the contribution story and the approach can be undertaken in a participatory mode. The succeeding sections will go through the steps of implementation in relation to the context of the HFK afterschool care program.

8

The Attribution Problem

This requires the explicit articulation of the attribution problem to be addressed. It aims to give an overview of what kinds of decisions will be based on the findings; what would show that the program made an important contribution; the identification of other key influencing factors; and the plausibility of the program in relation to the size of the program. As described by Mayne (1999), this requires an honest look of the intervention and acknowledging that other factors are at play.

The main attribution problem in the case of the HFK can be captured by the questions, ‘Is it reasonable to conclude that the HFK program has made a difference in keeping poor children in school?’ and ‘To what extent does the HFK help children stay in school?’. These questions acknowledge the cause-effect nature of the problem to be addressed. The answers to these questions will determine the impact of the intervention program and will have implications in the decisions of the administrative team and funders. Proof of the program’s contribution will be based on the gathered evidences along the line of the program theory - how the HFK intervention works. Other factors may include the level of responsiveness of the children, their socio-economic support system and influences of the formal school.

Is there a match between the intervention program and size of the problem being addressed? This question can be addressed by critically reviewing the problem being addressed. Although the problem is already considerably defined, more refinement is needed to answer this ‘matching’ between the problem and the intervention. This can be done by doing focus group discussions among stakeholders and gathering baseline evidence of the extent of the problem and to assess whether the HFK intervention is able to address it.

In this instance, a good situation analysis of the program is useful. The earlier discussion is a good start by being able to identify that the focus of the HFK intervention is how to motivate children to stay in school given their social and environmental context. It also considers the broader scope of the need to connect with community leaders, parents and teachers as contributory players of the intervention’s success.

The Theory of Change

This step requires the explicit presentation of the theory of change of the intervention and identifying the assumptions and risks attached to it. Theory of change is defined as the central processes or drivers of an intervention by which change comes about. Its main features are the situation of the intervention, the focusing and scoping and the determination of the outcomes chain (Funnell & Rogers, 2011).

For the HFK, the earlier discussion of the problem lends itself to understanding the intervention situation and at the same time defining the focus and scope of the program. It remains to develop the outcomes chain. When the outcomes chain is coupled with a theory of action – that is, the ways in which the intervention is constructed to activate the theory of change, it completes the picture of the intervention, the program theory.

9

The Program Theory

The initial program theory of the HFK afterschool care program, shown in Table 1, is a simple input-process-outcomes model. It shows the intervention inputs, the expected processes, outputs, short-term outcomes and outcomes that will determine how the intervention program might work to bring about the desired change in the community. However, pipeline models lack coherent links between the chains of outcomes needed for the contribution analysis, hence, new program logic for the HFK intervention is produced subject to validation by stakeholders (Figure 2).

The HFK program logic model in Figure 2 shows the outcomes chain of the program theory. It starts with the inputs provided by the program at the bottom and going up to the sequence of outputs, short term outcomes, long term outcomes and ultimate outcomes. The logic model is adapted to specify the program’s focus and scope in terms of participants and implementation. Closely tied to the problem being addressed, the program focuses on the motivation of children to stay in school. This focusing helps in identifying the expected outcomes in the results chain. The logic model recognises the fact that key factors such as home , school and community environments are significant contributory factors to the motivation (or demotivation) of children in school. Hence these are included in the model as the conditions that influence the outcomes among the children. For example, the mindset of parents needs to change if they are to encourage children to stay in school. Teachers can implement child-friendly strategies to motivate children to stay in school and the community leaders can facilitate good access and safety in the school and community environments. Thus, the program theory directly addresses the identified problem of ‘lack of motivation’. Finally the model shows the assumptions that the program makes in achieving its objectives. The logic model can help the HFK administration and operations team identify the measures that need to be documented along the outcomes chain. These are detailed in the gathering of evidences for the contribution story.

The Realist Program Theory Matrix

The program theory does not explicitly present the causal mechanisms that need to be triggered that may lead to the expected outcomes. Hence, ‘realist program theory matrix’ is developed to identify the generative causal mechanisms that are at work in the context of the program. Once identified, these mechanisms can be verified by gathering actual evidence from the field. The analysis of these mechanisms, contexts and outcomes shall be the basis for inferring the generalizability and transferability of the program. This addresses and provides insight into the feasibility of replication of the program.

Realist matrices represent program theory in a tabular form that shows the particular context, like the implementation environment and participant characteristics, in which causal mechanisms operate to generate outcomes of interest. However, Funnel and Rogers (2011) insist that realist matrices are not an alternative to outcomes chain logic models but a complementary approach that focuses on one or more causal mechanisms in a program theory and explores what it is about the program that makes this causal mechanism work. Table 2 shows an initial sketch of a realist matrix logic model in one of the possible mechanisms that may be at identified at HFK.

10

Table 1 Program Logic of the HFK after-school care program (Pipeline Model)

Inputs Processes Output Short-term Outcomes Outcomes

The HFK learning centre equipped with tables, chairs, electric light, learning materials, books and toys.

Kids accessed availability of the centre resources 5/days a week at 4-7pm and whole day on Saturdays.

Kids made use of the learning resources in the HFK learning centre.

Consistent attendance at HFK; Improved formal school attendance.

Kids stay in school

Move to the next grade level.

Finish basic education

Provision of basic school supplies (paper, pencils, bags, slippers, etc)

Provided to kids upon need and regular provision on school openings.

Basic school material needs of kids are met.

Built confidence to go to school; Improved formal school attendance.

Constant contact with children, school attendance monitoring and follow-up activities such as assignment support, remedial tutoring and fun activities.

Kids engaged in adult (centre supervisor, high school mentor or college volunteer) one-on-one follow-up and motivation activities during weekdays and weekends;Kids get key message of the importance of schooling.

Kids are motivated, reminded and get a sense of support for schooling.

Improved social interaction; Improved classroom interaction and performance; Improved formal school attendance.

Contact and follow-up with parents.

Parents engaged in one-on-one or group motivation to support their kids in school.

Parents expressed a sense of responsibility to support their kids in school.

Parents demonstrated a sense of partnership with the program by providing feedback about kids’ progress or difficulties.

Contact and follow-up with teachers.

Teachers are made aware of the program and helped monitor kids’ school progress.

Teachers expressed awareness of kids who are in the program.

Teachers demonstrated a sense of partnership with the program by providing feedback on kids’ progress or difficulties.

11

Figure 2 Program Logic of the HFK Afterschool Care Program (Outcomes Chain Model) with focus and scope of program delivery

Outcomes Chain

Program Focus Program Scope Participants Program Delivery

School Age ChildrenParents and Guardians

Teachers and School

Administration

Community Leaders

Assumptions and Risks

Ultimate Outcomes

Productive citizens and poverty alleviation

Sense of fulfilment and satisfaction

Professional satisfaction

Leadership achievement

Program contribution to community development

Proceed to further skills training or higher education

Full support to skills training or higher education

Maintain tracer contact with kids as school alumni

Link with higher government agencies to support basic education finishers

Constant contact

Long Term Outcomes

Finish basic education

Encourage kids further skills training or higher education; plan for financial support

Provide career guidance

Find opportunities for community scholarships

Program success stories; Plan for higher education support (sourcing of scholarships and placements)

Grade level progressExpress long term plan for children’s education

Monitors and coordinates with HFK Centre Team

Create family or individual incentives

School coordination on monitoring (participation and learning)

12

Long Term Outcomes Improved school attendance

and participation

Show support to children’s schooling

Improved commitment

Infrastructure improvements (road access and safety)

Recognition of performers; address special needs

Short Term Outcomes

Children get a sense of support for education and change in mindset

towards education

Change of mindset towards education

Practice child friendly school environment

Campaign importance of education

Constant contact and program monitoring

OutputsChildren’s voluntary engagements

with the program activities and provision

Support the children to join centre activities

Encourage kids to avail of centre resources

Community recognition of the program

Learning resources and materials are utilized; services correctly delivered

InputsVarious HFK Centre activities and

provisionsAwareness and engagement Program objectives

clearly communicated

13

Table 2 A Realist Matrix Logic Model of the Happy Fish Kids Afterschool Care Program

Context Mechanisms Outcomes

HFK participants

Children of low-income families in the fishing community who show interest in schooling by enrolling in the HFK afterschool care program and voluntarily visiting the HFK centre for school materials and adult support.

Social support Positive peer pressure Reinforcement Attraction to the centre

resources (books, toys, food and fun activities)

Voluntary engagement with program activities and provisions

Sense of support Change in mindset

towards education Improved school

attendance and participation

Grade level progress Finish basic education

Parent participants

Parents who are voluntarily engaged in the HFK afterschool care program and expressed commitment to support their children in formal school and attending the HFK activities.

Help us help our kids Valuing education

Awareness and engagement

Change of mindset Support children Long-term plan Financial support Sense of parental

fulfilment

Volunteers and Staff

Volunteers and staff who are adequately briefed of the program design, have time commitment and provided with resources to render the services needed by the program.

Altruism Idealism Voluntarism Role modelling

Clear understanding of program objectives

Services correctly delivered

Sense of fulfilment

Formal School Teachers

Teachers in the nearby formal school who are aware of the existence of the program and who are committed to support the monitoring of children’s participation in school.

Teaching help Performance

monitoring anxiety

Awareness and engagement

Practice child-friendly approach

Monitors and coordinates with HFK

Professional fulfilment

14

Continuation of Table 2

Organisation

The administration team and the organisation of funders who are committed to developing the program and provide resources for the program to operate.

Altruism National pride Expatriate’s ‘guilt’ of

leaving home country

Program theory clearly understood and communicated

Mobilise for further funding needs

Achievement

Local community leaders

The local community governance that provides legitimacy of the program activities that contributes to the acceptability of the program in the community.

Personal political agenda

Leadership principles

Awareness and engagement

Mobilise school and community improvement projects

Leadership fulfilment

Other agencies

The academic, professional and non-government organisations who linked with the program and are committed to render specialised services to the HFK.

Corporate responsibility

Clear understanding of the program

Services fit-in Sense of

organisational /professional accomplishment

Evidences of the Theory of Change

This step will first review the strengths and weaknesses of the program logic and the plausibility of the various assumptions in the theory. The evidence gathered the looks to validate the theory of change. Evidence is needed in the following areas:

1. Evidence on results and activities;2. Evidence on assumptions; and3. Evidence on other influencing factors.

The evidence on results and activities is the documentation needed to support whether key results occur or not. These look into the outputs, short term and long term outcomes of the program focus and scope. It is important to include evidence that the program is implemented as planned. This evidence should answer the questions: (1) How do the children engage in the program?; (2)Are the services and materials that the program provides used?; (3) Do school participation and attendance improve for the children participants in HFK?; (4) Are children progressing to the next year level?

15

The evidence on assumptions used in the program will show if the assumptions are valid. This may include research findings that support these assumptions. This is particularly useful in finding possible existing theories and research evidence on the underlying mechanisms that are hypothesised to work in the program. This can be linked to the searching of theories that may support the underlying mechanisms in the realist program theory matrix.

The evidence on other factors influencing the program could be the identification of alternative explanations to program outcomes. In this area, Campbell’s list to threats to internal validity is appropriate.

As an example of evidence use, the data on formal school attendance of children attending the HFK is monitored and compared at different levels of children’s participation at HFK (Figure 3). The statistical comparison then can lead to the investigation of possible causal mechanisms. For instance, Figure 1 report data shows that not all Grade 1 children attending the HFK have afterschool attendance and yet low attendees appear to have better formal school attendance. Possibly these are the children whose parents place high value on education and do not entirely depend on the HFK centre to provide their basic school materials. Why does August attendance drop despite the fact that children improved their HFK attendance? Further investigation revealed two possible reasons. The month of August is the Ramadan celebration of Moslems (48% of the children) where school attendance is not compulsory or the month of August had three successive typhoons during which the creeks and shores were highly dangerous and attending school was not advised during heavy rains. These were possible historical threats to internal validity in the case of claiming that the program does not work.

Figure 3 HFK Formal school attendance monitoring report

16

Source: http://www.facebook.com/photo.php?fbid=10150343987344381&set=o.147108592017820&type=3&theater

Assembly and Revision of the Contribution Story

The last three steps consist of the assembly and revision of the contribution story which can be done iteratively. These steps require a participatory approach that may include the program beneficiaries, key stakeholders and even program design experts to assess and validate the contribution story.

Assemble and Assess the Contribution Story and its Challenges

The information gathered from previous steps is assembled to create the contribution story. The presentation will be through a case that starts with the program theory, the realist matrix of causal mechanisms and stories of documentary evidences. This step will also identify the strengths and weaknesses in the causal links. Kotvojs (2006) refer to this as the ‘performance story’ in the contribution analysis of the Fiji Education Sector Program. The story is presented in a matrix format that summarises the results chain, the expectations, achievements and alternative explanations of the program. Mayne (2001) did not specify any format on how to assemble the contribution story but certainly providing evidence on the ‘causal mechanisms’ at work in the context of the program will add a strong element of credibility to the story.

Seek Out Additional Evidence

This step will seek to identify the challenges to the credibility of the contribution story. This is where new evidence may be required to possibly adjust and strengthen the theory of change. For example in HFK, when the initial ‘program theory’ was uploaded in the social network group of the stakeholders, some funders suggested that data be included on report cards rather than only on attendance to improve the program design. There is also a suggestion on case studies on identified children who are in extreme low and high school participation.

Revise and Strengthen the Contribution Story

This final step simply points to the iterative nature of the contribution analysis as a framework of impact evaluation. The iteration is done by reassessing the credibility of the contribution story and seeking further data and evidences that will strengthen it.

Discussions: Observations, replication questions and possible biases

The close scrutiny of the application of contribution analysis as an approach to impact evaluation of the HFK intervention leads to some observations.

Firstly, contribution analysis has an emphasis on evidence accumulation over the time of program accumulation. This is an advantage compared to experimental designs that worry on strict temporal element as well as implementation contamination between the intervention and control groups. Secondly, the methodology is more consistent with observing and documenting outcomes as the program implementation unfolds in its natural social setting, no controls and tends to welcome all possible interactions of the intervention to everything around it. This is consistent with the understanding that social intervention programs function in open systems. Thirdly, there is openness to the approach to evidence accumulation. The program managers have the option to choose methods that work for them and they are familiar with as long as it is logically supporting the

17

contribution story. Lastly, the iterative nature of the methodology allows program theory revision and strengthening which is a key to innovation and adaptation to changes in the social environment. These revisions ensure the program’s success in a constantly changing context.

Examination of the community where the new program is to be implemented; and looking for possible causal similarities can provide the HFK administration and funders with answers to the replication questions. Then the program theory of change can then be examined if outcomes chain and the focus and scope are similarly applicable. It is also important to see whether the assumptions of the program delivery at HFK are valid in the new replication community. If there is substantial information as to these inner workings of the HFK implementation and possible matching and adjusting to the new replication site, the replication plan can be expected to work.

Finally, this paper identifies some possible biases as to the methodological choice and application. Specifically, on putting together the ideas of the program theory, the realist program theory matrix and the Campbellian methods of examining alternative explanations are based on the writer’s interpretation of these approaches. There is a high risk of conflict in the theoretical underpinnings of such methodologies. The paper also takes advantage of the ‘open’ prescription as to the specific approaches within the contribution analysis framework. Furthermore, this paper is written from the perspective of a person very familiar with the HFK program. This familiarity could lead to some aspects of the program being omitted and which could impact on another person fully understanding the HFK environment.

CONCLUSION

This paper has presented how contribution analysis can be used as a framework for impact evaluation of the HFK afterschool care as a social intervention program. It has shown how the problem being addressed connects with the development of the program theory with an examination of the causal mechanisms within the program’s context that could be a basis for replication decisions. Finally, the steps of contribution analysis for the impact evaluation are presented as guides for its actual field implementation. The evaluation findings could possibly lead to future theoretical and empirical studies in the area of afterschool care intervention in developing countries.

18

REFERENCES

Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory : effective use of theories of change and logic models / Sue C. Funnell and Patricia J. Rogers: San Francisco, CA : Jossey-Bass, 2011.

Haberman, M. (1991). The Pedagogy of Poverty versus Good Teaching. The Phi Delta Kappan, 73(4), 290-294.

Halpern, R. (1999). After-School Programs for Low-Income Children: Promise and Challenges. The Future of Children, 9(2), 81-95.

Howes, C., Olenick, M., & Der-Kiureghian, T. (1987). After-School Child Care in an Elementary School: Social Development and Continuity and Complementarity of Programs. The Elementary School Journal, 88(1), 93-103.

Kotvojs, F. (2006). Contribution Analysis: A new approach to Evaluation in International Development. Paper presented at the Australasian Evaluation Society International Conference 2006, Darwin, Australia.

Lemire, S. (2010). Contribution analysis: the promising new approach to causal claims. Retrieved from http://www.europeanevaluation.org/images/file/Conference/Past_Conference/2010_Prague/FullPapers/3_Lemire_Sebastian.pdf

Maligalig, D., Caoli-Rodriguez, R., Martinez, A., & Cuevas, S. (2010). Education outcomes in the Philippines ADB Economics Working Paper Series No. 199: Asian Development Bank.

Mayne, J. (2001). Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly. Canadian Journal of Program Evaluation, 16(1), 1-24.

McInerney, D. M. (2001). Research on Sociocultural Influences on Motivation and Learning Vol. 1.

Pawson, R., & Tilley, N. (1997). Realistic evaluation / Ray Pawson and Nick Tilley: London ; Thousand Oaks, Calif. : Sage, 1997.

Posner, J. K., & Vandell, D. L. (1994). Low-Income Children's After-School Care: Are There Beneficial Effects of After-School Programs? Child Development, 65(2), 440-456. doi: 10.1111/j.1467-8624.1994.tb00762.x

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference: Boston:Houghton Mifflin.

19