Independence in development evaluations – Getting the balance...

Inde

pend

ence

in d

evel

opm

ent e

valu

atio

ns –

Get

ting

the

bala

nce

right

with

sta

keho

lder

eng

agem

ent

Considering the importance of stakeholder engagement in ensuring credible and useful evaluations, maintaining independence while achieving this has become a recurring chal-lenge in development evaluations over the years. This paper focuses in particular on the challenge of possible tensions among key actors in both ensuring independence during the eval-uation and applying adequate stakeholder engagement efforts1 in the evaluation. It builds on discussions from literature, experiences and observations of evaluation practices among the multilateral development institutions, includ-ing the African Development Bank, to shed light on possible ways to deal with the challenge.

The paper suggests that there is not one single approach that fits in all circumstances in deal-ing with this challenge. It argues that there is always the need to adopt and explore the organ-isational context in our evaluation approaches. This implies an approach that combines the use of an appropriate evaluation system arrange-ment supported by evaluators’ experience, judgement and beliefs around the context and expectations of the evaluation.

Independence in development evaluations – Getting the balance right with stakeholder engagement 21

eVALUation Matters First Quarter 2017

Introduction

There is widespread recog-nition in the development community of the impor-tance of independence in development evaluations.

This emanates from the crucial role that evaluations play in strategic orientation of evidence-based policy and program decision making. Evaluation has become an essential component of good govern-ance and has been a good support for reflection and periodic adaptations in the project, program and policy cycle (Organisation for Economic Cooperation and Development – Development Assis-tance Committee [OECD-DAC]). Mayne (2012) describes the principle of inde-pendence as a function in evaluation systems. In his opinion and similarly described by OeCD, this is a principle that instils trust in the evaluation function as related to governance/oversight. To the DAC community, independence of eval-uations is the degree of autonomy and separation from policy and operations. This allows evaluators to be able to report truthfully and without fear of repercus-sions or compromising to any demands of operations managers or entities being evaluated. There is a common argument about this practice (independence) that if the evaluation body is too remote from the everyday operations of the institu-tion or concerned parties, then the eval-uations may not be closely aligned with required information.

Following from the above, independ-ent evaluations2 are considered part of sound management practices and they are reflected on three dimensions3: a) organisational independence where the evaluation function should be separated from those responsible for the design and implementation of the entities being evaluated. It also ensures that evaluation staff have full access to the information needed; b) behavioural independence, an expectation of ethics based conduct that seeks to prevent or manage the potential for conflict of interest; c) and functional independence requiring mechanisms in place to ensure freedom from influence in the planning, funding and reporting of evaluations. Literature indicates the need for a balance to be struck between independence and isolation. The OeCD-DAC review report (2016) on evaluation systems cautions on the danger of isolation when seeking evaluation independence: a) lack of connection with operational units hence problems to access data and information, b) a reduced sense of ownership of the evalu-ation results specifically by management/policy makers hence lower uptake; c) risk of being perceived as disloyal and untrust-worthy or bringing in unwelcome criti-cisms considered as potentially damaging to uphold confidence.

In practice, these issues consistently contend with good practice standards of stakeholder engagement that are impor-tant and usually required for ensuring credible and useful evaluations.

Akua Arthur-Kissi, African Development Bank

Independence in development evaluations – Getting the balance right with stakeholder engagement22


Experience shows that involving deci-sion makers and stakeholders in the evalua-tion process enhances the prospects of the evaluation being more salient, legitimate and credible to ensure use and influence. In the same realm within the develop-ment evaluation profession and from diverse evaluation literature, stakeholder participation in evaluation processes is considered important to attaining sustain-ability, ownership and mutual accountabil-ity. However, a recent survey by OeCD DAC4

among its member networks showed that only a small number of agencies confirm that frequent or close stakeholder partic-ipation is encouraged during evaluation planning and design phases. While each stakeholder engagement effort will require a tailored approach based on the scope, scale and desired results, the expectation is that evaluation approaches should be consistent with the guiding principles of stakeholder engagement5.

Morra & Rist (2009) affirm the importance of engaging stakeholders early enough in the evaluation process, from planning through to completion of the evaluation. In practice, it is not evident finding the right balance between remaining inde-pendent and engaging meaningfully with clients, management, program owners and others. One may ask how do we remain independent if we have to engage that much with stakeholders? Where do we place the boundary to keep the poten-tial consequences at bay?

Different literature and observed prac-tices in multilateral development banks (MDBs), including the AfDB, point to a widespread recognition among the devel-opment community on the importance of stakeholder engagement in evalua-tion processes.

At the AfDB, the approach to stakeholder engagement is embedded in the planning and review stages of all evaluations. The

experiences show the powerful role that the different stakeholders play in the success of an evaluation. This stems from clients in the regional member countries and staff in the Bank who have interest in the subject of evaluation.

Considering the criticality of these key principles and the effect they have on the ability of development evaluations to achieve usefulness and influence, this paper focuses on bringing to light the challenges that evaluators have to endure to ensure a good balance of applying these two principles in a mutually connected way. It also seeks to explore possible ways of approach-ing the challenges through an analy-sis of the different scenarios and the possible context where they manifest. It concludes by proposing possible options to explore by evaluators in the search for the appropriate approach to deal with the potential challenges.

The issue of staying independent in development evaluation

As mentioned earlier, the principle of independence in evaluation has seen a growing prominence in both evaluation discussions and in literature. Although there is contention about its clarity and usage by some, it is mostly considered as a key principle to ensuring the credibil-ity and quality of an evaluation. There is no clear definition of evaluation inde-pendence (Kaiser & Brass, 2010). A number of development evaluation agencies adopt the OeCD guide, DAC Quality Stand-ards for Development Evaluation (OECD, 2010) which considers independence as a key overarching principle. It defines independence as when “the evaluation process is transparent and independ-ent from policy makers and program management”.



Loud and Mayne 2014 agree on this principle and further allude that the cred-ibility of an evaluation will mostly be at least dependent on the level of independ-ence accorded to the evaluation function. This notion places evaluation close to the audit function and proposes to ensure independence in the implementation and management of evaluation processes. There is however the need for the right policy and practical guidelines for deci-sion-making and follow up to be in place. In addition, experienced evaluators highlight the criticality of independence to ensur-ing evaluations credibility. For example, Robert Picciotto, former head of the World Bank's independent evaluation group (IeG) argues that independence remains an issue that affects an evaluation’s credibil-ity despite all efforts made in an evaluation for ensuring better quality of work (2003). Equally, Scriven (1975) in his discussion on preventive measures to reduce risks of bias in evaluations proposed independence in

Evaluation Commissioner

Evaluand

Evaluation

Evaluator

Information

Guidance

AccountabilityAsse

ssm

ent

TOR

, Rem

uner

atio

n

Enquiry

Figure 1: Framework of the tricky triangle by Bastiaan de Laat

Source: Adapted from Loud & Mayne (2014) to illustrate the relationships among the three roles in the evaluation process.

evaluations as one of the ways to reduce the tendency of bias.

Experiences and literature show that credibility matters a lot to internal and external audiences of evaluations. It is particularly a key factor that affects the usefulness and use of evaluation recom-mendations (Penny Hawkins in Loud and Mayne 2014). For example, a lack of perceived independence is most often used as reason to dismiss or contend eval-uation findings even when there could be other types of possible issues around information type or a strong opinion by an influential entity as reasons. It is obvious that strengthening independence goes along with the recognition of credible and useful evaluation work.

The ability to sustain full independence without becoming too remote from real-ities of practice and policy remain a chal-lenge here but experience shows that



this is to an extent achievable when the evaluation policy and manuals clearly guide the evaluation processes to attain-ing this as a goal.

How does stakeholder engagement matter in achieving credible evaluations?

Engaging stakeholders early on in the evaluation process gives the evaluator a better understanding of the interven-tion or issue of evaluation, what it was intended for and the issues and challenges they encountered in the process. This helps the evaluation team to be better informed about the issues to be covered in the evaluation as well as specific informa-tion needed, when the information will be required and who can provide it. Engaging with key stakeholders helps to ensure that the evaluation does not miss any major crucial issue. Stakeholders´ engagement is expected to help increase their support of and interest in the evaluation which forms the potential for buy-in and effec-tive use of the evaluation. Although there are many benefits claimed for participa-tion, there is a growing cynicism amongst evaluators and stakeholders who feel let down that sometimes these claims yields suboptimal outcome in terms of efficiency (longer evaluation lifespan) and subsequently the usefulness of the evaluation. This also relates to the caution described by Loud and Mayne (2014) in the analysis of the “tricky triangle” concept by Bastiaan de Laat , where they illustrate the importance of the relationship and at the same time the dangers with the different configurations of the relationship with respect to independence, bias and control in evaluations.

The question now is what constitutes a “good” balance between independence and stakeholder engagement to achieve the design and completion of credible

and useful evaluations? This is a question that incites a contentious uptake among the evaluation and development commu-nity and it has not reached consensus to date among evaluation bodies. From an evaluation user’s point of view, main-taining this balance depends on several factors. This includes the existence of an approach and set up that seeks to capture political and technical perspectives of the different stakeholders, and the ability to consider them objectively in the evalua-tion questions. Bryson et al 2010 provides interesting guidance on appropriate tools to achieve these. This similarly touches on the potential tension between the differ-ent roles in evaluation and the influence of their relationship configuration on evalu-ation use described by Bastiaan de Laat in the tricky triangle concept (figure 1).

Understanding the concepts of evaluation actors’ relationship configurations

In the tricky triangle concept by De Laat in Loud and Mayne (2014) three different roles exist in evaluation settings. They are defined as follows; a) the “evaluand” referred to here as the one being evalu-ated; b) the evaluator, the one performing the evaluation; c) the evaluation commis-sioner, the body or entity that calls for the evaluation and finances the evaluation service that is to be delivered. The analysis argues that maintaining a certain level of balance to achieve credibility and ability to minimise bias and control depends on the type of configuration and relation-ship between these three roles. From the discussions, what makes the triangle tricky is that there is an anticipation from the different actors with respect to the evaluation outcome which has to be considered for an effective evaluation.

The scenarios illustrated in figure 1 point to the fact that the three roles (actors) involved in the evaluation process



do seek at any point to have a level of control or influence which relates to final judgements and outcome of the evaluation. This is critical to its credibility and usefulness. Previous mention of the dangers of the tricky triangle illustrate that maintaining an ideal set up or config-uration does not necessarily work particu-larly in dealing with the challenges of potential influence of the different actors and its detrimental effect to sustaining credibility. Rather, having agreed policies and evaluation procedures that manage the relational factor is key to managing the risks.

Understanding how these roles are configured in an institutional setting matters a lot when trying to plan for the ideal balance between independence and stakeholders’ involvement. No matter how neutral each actor may be there is the temptation to adapt the evaluation objective to one’s own agenda or specific expectation which could influence the evaluation questions or the approach. In the same way, the evaluand (one being evaluated) in his engagement may seek to push out certain topics from the eval-uation or direct the focus to only the bright spots.

A closer look at practical evaluation settings and set ups in some MDBs

The usual practice in most MDBs, including AfDB, presents two modes of evaluation setups. There is an internal evaluation where all roles are within the same organi-sation (particularly undertaking self evalu-ations). There is also a hybrid confi guration where the role of the evaluation commis-sioner and the evaluator coincide and the evaluand is separate through independent evaluations. The practice is a combination of the two modes. With the hybrid config-uration, the evaluation unit may be the commissioner of external evaluations or studies but the evaluation unit’s portfolio

of programs is commissioned indirectly by the Board of Directors (BoD) who have to endorse the workplan. This endorsement makes the BoD the defacto commissioner of the evaluations. The endorsed workplan becomes a contract between the evalu-ation unit and the BoD to carry out the planned evaluations within a given time and resources. With the former mode, the practice is often within operations units of MDBs in undertaking self evaluations in the form of self-assessment notes, monitoring reports and project completion reports. Generally, there is no discrete evaluation commissioner but rather these self eval-uations are directed by required internal or external organisational procedures thereby acting by default as commissioners and guidance for the implementation of the evaluation though there is a clear line of authority. Consequently this mode seems to place a stain on the validity and use of the evaluation depending on the extent to which evaluators are honest and objective in these self evaluations. It also relates to the perception of the use of self evaluations as being only formative or developmental by its nature (Patton, 2011) rather than for accountability as there is a percieved higher risk of bias and therefore often only used as data sources in broader independent evalu-ations as practiced in most MDBs.

With the latter mode of practice, the three roles are within the same organisation but they are generally separated through a sepa-rate dedicated independent evaluation unit (evaluators) who only report to executive board members (the commissioner) and not to senior management who manages the financing and implementation of the organsation’s programs (the evaluand). The hybrid configuration often involves a combination of two modes in practice; both fully fledged internal evaluations, with the support of external expert evaluators or sometimes externalised evaluations through the evaluation unit who plays a supervisory role. Within the AfDB, both



are practiced but the experiences are characterised by issues around the tensions of the tricky triangle and its possible risks to independence and the evaluation outcome and use. Some literature criticises this mode of operation as having the potential to cause role pressures within the config-uration and a high tendency towards posi-tive bias (Mathison, 2005, Wenar 2006, Loud & Mayne 2014). It is however argued that these challenges could be addressed with the adoption and application of appropri-ate evaluation policies and measures that ensure credibility of evaluations and guar-antees independence. The development evaluation community have done well to promote the adoption and institutionali-sation of this practice among its members. The Evaluation Cooperation Group6 (eCG), proposes a set of “good practice standards” to guide members to establish independ-ence of the internal evaluation units. This standard practice guideline maintains that the independence of evaluation units is not a given, but rather, it should be nurtured and protected regularly in all activities to achieve success.

Reaching the balance of independence and engaging actors for a useful evaluation

Scriven (1975) reiterated that seeking to be objective and independent in evaluations often leads to the use of “external” evalua-tors. Though externality could be relative, nevertheless, the possibility for the exter-nal evaluator to give in to pressures from a client does not disappear if the chances of getting another contract is at stake. This is also affects independent judgment in eval-uations (Mathison, 1999).

In the same realm, building long-term relationships between evaluators and program managers is perceived potentially harmful either as it would lead to the eval-uator increasingly identifying with the

organisation she or he works for, therewith, increasingly behaving in a “biased” way as an internal evaluator. Following from these points, it is evident that as much as evalu-ation practices encourage effective engage-ment of all actors, how the evaluation will be used and for which purpose should be the key yardstick in determining a frame-work for their collaboration in dealing with the relationships. This also serves to test the extent to which the policies guiding the evaluation practices in our institutions support efforts to mitigate the risks.

Let us look at two typical instances in the AfDB's independent evaluation unit (IDeV) concerning such risks. IDeV prac-tices are guided by its established policies and principles on its independence (in behaviour, set up and functions) which were approved by the Board of Directors. The first instance was an independent country program evaluation which had to be cancelled halfway through its imple-mentation. This was a result of limited collaboration among key actors and other stakeholders on the purpose, timing and usefulness of the evaluation. In this case, the practice of independence over shadowed the required level of engage-ment among key actors particularly the evaluands and their relationship thereof. A contrary situation however was the independent evaluation of the operations procurement policy of the AfDB. This eval-uation achieved the buy – in and effective use of outcome by the key stakeholders i.e. the Board of Directors, operations staff and procurement officers of the AfDB, as well as clients in the regional member countries. The evaluation received full cooperation and access to information following a good engagement process during and throughout the evaluation activities (from planning to completion). The engagement process was driven by the common agreement on the purpose and use of the evaluation by the key



actors. With a clear sense of objec-tivity for the evaluation, the potential diverse interest and influence of the key actors (i.e. the evaluation commissioner, the evaluators, the external evaluators and the evaluand) were considered and curbed to the evaluation focus and use to achieve independence in the planning and implementation process. An exam-ple of the actions was the fact that the evaluand was engaged to follow up and ensure that the key issues emerging from self-assessment reports were considered within the broader questions of the eval-uation. These elements facilitated effec-tive engagement and a common sense of purpose and use for the evaluation among the stakeholders. Basically, the underlying factors in both cases points to issues around the following but not limited to: a) key actors’ collaboration on the purpose and usefulness of the eval-uation; b) management of the interest and influence of key actors by the evalu-ators in the planning and implementation process; and c) existence and application of appropriate policies and frameworks

guiding practices for independence in the institutional setting.

Managing extent of independence and engagement in evaluations

In practice, knowing the importance of being inclusive, it is also similarly important not to be so at the expense of independence. By this, it implies the important trade-offs to be made. One has to be cautious of the likely danger of tension between all stakeholders’7 inter-est about certain information so that the scope remains relevant to the objective and purpose of evaluation use. There have not been any clear standards or guidance on how to manage this tension. However evaluation experiences point to enabling factors for consideration, some of which are identified and described in the AfDB experiences above. To conclude, the emphasis remains that in evaluation planning and implementation, recogniz-ing the factors that have positive



effects on managing stakeholder interest and influence goes a long way to deal with the challenge. The concept of stakeholder management also calls on evaluators to be wary of the elements of different types of stakeholders’ power and interest so as to take appropriate actions to manage their weighting of influence towards the outcome. In all these, acceptable standard practices of evaluators' independence should not be overlooked.

A take away for evaluators or practitioners

In summary, three morals emerge in the above discussions for consideration by evaluators or evaluation practitioners, as they seek to manage tensions and maintain a balance between the extent of independ-ence and engagement of stakeholders for credible and useful evaluations:

A. Firstly, stakeholders have to be engaged to the extent possible throughout the evaluation process for credibility and use. However, an appropriate organisational setting for evaluations through its policies and strategies will be the cata-pult for the evaluator’s ability to preserve a balance of engagement and independence;

b. Secondly, an understanding of the evaluation configuration and aware-ness of common pitfalls in applying good standard practices presents options to explore for effective inde-pendent evaluations.

c. Finally, practising independence in evaluations is as important as managing relational issues and common purpose among key actors of the evaluations. Ultimately, the main focus should be on how credible and useful the final product will be to the users, not only commission-ers but those whose lives may be affected by decisions informed by the evaluations.

Endnotes

1 Stakeholder engagement: Defined here as a management

practice aimed at capturing knowledge, increasing owner-

ship of the project by users, reducing conflict, encouraging

innovation and facilitating spin‐off partnerships among

actors. The practices involves identifying the key actors

through an analysis of their interest and influence and

involving them meaning fully to enhance inclusive deci-

sion making, promoting equity. From a social perspective

it is a process where diverse stakeholders share a common

forum, learn about each other’s values, reflect on these

values and create a shared vision and shared objective.

The practice is useful in increasing awareness, changing

attitudes and affecting behaviors (Bryson, 2004).

2 Independent evaluation: An evaluation carried out by

people or entities free of the control of those responsible

for the design and implementation of the development

intervention. (OECD 2002, P.25).

3 According to good practice standards of evaluation by

Evaluation Cooperation Group (ECG), 2008.

4 Evaluation Systems in Development Cooperation: 2016

Review, OECD 2016.

5 Summarized in “Seven guiding principles of stakeholder

engagement”. (Gautrey, 2003).

6 The peer group of MDB’s and MFI’s Evaluation Units.

7 All actors with interest and influence on the evaluation

Including users of evaluation outcome.



Aut

hor’s

pro

file

Akua Arthur-Kissi is an evaluation officer in the Inde-pendent Development Evaluation (IDeV) of the African Development Bank. She has been working on corporate, country program and thematic evaluations for the past 8 years. She co-task managed a recently completed country strategy and program evaluation of South Africa (2004–2015), and also takes credit for her contributions to previous influential evaluations by IDeV: The Paris Declaration Review (2010), Fragile States (2011–12) and Policy Based Operations evaluations (2010) of the Bank. Akua previously worked in sector and country opera-tions departments and the knowledge management unit of the Bank for 8 years. Before joining the Bank, she worked momentar-ily with GIZ – Regional Aids Program for Africa in Ghana. Akua holds an MSC degree in Project and Programmes management from Warwick University, UK, and certification in monitoring, development evaluation and capacity building.

References & bibliography

AfDB (2005), Report on the independence of the department of

Operations Evaluation (IDEV).

Bernstein, D.J., Whitsett, M. D.& Rakesh M. (2002), Addressing

Sponsor and Stakeholder Needs in the Evaluation Authorizing

Environment: Trends and Implications (New Directors for Eval-

uation, No. 95, Fall 2002, Wiley Periodicals).

Bryson, J.M. (2004). What to do when stakeholders matter: stake-

holder identification and analysis techniques. Public manage-

ment review, 6(1), pp.21-53.

Bryson J.M, Patton M.Q & Bowman R. A. (2010), working with

evaluation stakeholders: A ration, step-wise approach & tool kit,

Journal of Evaluation and Program Planning – Elsevier 2010.

Evaluation Cooperation Group – ECG (2010). Good practice

standards on independence of international financial institu-

tions’ central evaluation departments.

Gautrey, C. (2003), ‘Seven guiding principles of stakeholder engage-

ment’, Blogs at ATD, blog post, viewed 15 March 2017, «https://

www.td.org/Publications/Blogs/Management-Blog/2013/06/

Seven-Guiding-Principles-Stakeholder-Engagement».

Kaiser, F.M. & Brass, C. T. (2010). Independent evaluators of

federal programs: Approaches, devices and examples. Wash-

ington D.C.

Loud M. & Mayne J. (2014), Enhancing Evaluation Use, Insights

from Internal Evaluation Units.

Mathison, S (1999) Rights, Responsibilities and duties: A

comparison of ethics for internal and external evaluation

(Special Issue). New Directions for Evaluation.

Mayne, J. (2012). Independence in Evaluation and the Role of

Culture. Evaluation cultures: Sense-making in complex times,

105-138.

Morra – Imas ,L & Rist R. (2009), The Road to Results: Designing

and Conducting Effective Development Evaluations.

OECD (1991), DAC Principles for the Evaluation of Develop-

ment Assistance.

OECD (2010), Quality Standards for Development Evaluation

OECD (2016), Review of Evaluation Systems in develop-

ment cooperation.

Patton, M.Q. (2011). Developmental evaluation: Applying

complexity concepts to enhance innovation and use. New York,

Guilford Press.

Piciotto, R. (2003). International trends and development evalu-

ation: The need for ideas. American Journal of Evaluation.

Scriven, M (1975); Evaluation bias and its control. Berkeley:

University of California.

Wenar, L. (2006) Accountability International Development Aid.

Ethics and International Affairs.

Independence in development evaluations – Getting the balance...

Documents

Transcript of Independence in development evaluations – Getting the balance...