Evaluation of the

Independent Science and Partnership Council (ISPC)

Final Report

Mary O’Kane and Eija Pehu

October 2017

1

10 October 2017

Ms Rachel Sauvinet-Bedouin Head Independent Evaluation Arrangement CGIAR System

Dear Ms Sauvinet-Bedouin,

Final Report – Evaluation of the Independent Science and Partnership Council (ISPC)

Attached, on behalf of Dr Pehu and myself, is the final report of the evaluation of the ISPC which the IEA commissioned in March 2017 in line with its 2017 work plan.

In presenting this report we would like to thank all those who contributed their time to the evaluation – those we interviewed, those who participated in the survey, and the many members of the System Council and its SIMEC, the ISPC and others with roles in CGIAR who assisted us.

I would also like to thank the ISPC Secretariat and your own IEA for their assistance.

Yours sincerely,

Mary O’Kane

2

Contents

Executive Summary .............................................................................................................................. 4 1. Introduction ................................................................................................................................. 8

1.1 Structure of the evaluation .................................................................................................. 8 1.2 Evaluation terms of reference ............................................................................................. 8 1.3 The approach of this evaluation ......................................................................................... 9

1.3.1 Information ......................................................................................................... 10 1.3.2 Analyses and what can be derived from them ................................................... 10

1.4 Limitations of the evaluation ............................................................................................. 11 2. The ISPC: its structure, remit, history and context ............................................................... 12

2.1 Structure ........................................................................................................................... 12 2.1.1 Composition ....................................................................................................... 12 2.1.2 Mode of operation .............................................................................................. 14 2.1.3 Budget and work plan ........................................................................................ 15 2.1.4 Secretariat .......................................................................................................... 17

2.2 Context in which the ISPC operates ................................................................................. 17 2.2.1 Societal grand challenges and CGIAR .............................................................. 17 2.2.2 Strength of the System amidst reform and financial uncertainty ....................... 18 2.2.3 Evolution of science advice at System level ...................................................... 19

2.3 ISPC – its unsettled role ................................................................................................... 21 2.3.1 Mid Term Review ............................................................................................... 21 2.3.2 ISPC Task Force................................................................................................ 21 2.3.3 SIMEC and System Council .............................................................................. 22 2.3.4 ISPC terms of reference: an ongoing issue ....................................................... 23

3. What the ISPC does and how it is perceived ......................................................................... 24 3.1 The ISPC’s role in the CGIAR System ............................................................................. 24 3.2 Mobilizing Science and Partnerships................................................................................ 25

3.2.1 Output ................................................................................................................ 25 3.2.2 Analysis .............................................................................................................. 26

3.3 Impact assessment ........................................................................................................... 27 3.3.1 Output ................................................................................................................ 27 3.3.2 Analysis .............................................................................................................. 28

3.4 Independent program review ............................................................................................ 29 3.4.1 Output ................................................................................................................ 29 3.4.2 Analysis .............................................................................................................. 30

3.5 Strategy and trends .......................................................................................................... 32 3.5.1 Output ................................................................................................................ 32 3.5.2 Analysis .............................................................................................................. 33

4. What type of science/research advice does CGIAR need? .................................................. 36 4.1 A mismatch ....................................................................................................................... 36 4.2 There is a lot of science advice available ......................................................................... 37 4.3 Is a standing science advisory body needed at all? ......................................................... 37 4.4 Should the science advice function be re-calibrated or changed in its framing? ............. 38 4.5 Foresight analysis ............................................................................................................. 39 4.6 What characterizes good advice from a science/research advisory body to a complex

research organization like CGIAR? .................................................................................. 39 4.6.1 Top research centers make good use of eminent science advisory bodies ...... 39 4.6.2 CSIRO has restructured its advice recently ....................................................... 40 4.6.3 Science advice needs to be trustworthy and useful to the body commissioning it

........................................................................................................................... 41 4.6.4 Lessons from other organizations on good science advice arrangements ....... 42

4.7 What science and science-impact advice does CGIAR need? ........................................ 42 4.8 Advice must be able to be absorbed and guide decision-making .................................... 44 4.9 How will CGIAR know if the advisory mechanism is being successful? .......................... 44

5. Findings and significant observations ................................................................................... 45 6. Conclusions and Recommendations ...................................................................................... 49

6.1 A note on the structure of the recommendations ............................................................. 50 6.2 Recommendations ............................................................................................................ 50 6.3 Comparison of current ISPC and possible new body ....................................................... 52

3

6.4 Option 2 Recommendations – ad hoc advice ................................................................... 53 6.5 Option 3 Recommendations – ISPC modifies its operations ........................................... 53

Annexes ............................................................................................................................................... 54 Annex 1: Members of the evaluation team ................................................................................. 54 Annex 2: Formal CGIAR meetings and events addressed/participated in by evaluation team

members ........................................................................................................................... 56 Annex 3: List of interviewees ...................................................................................................... 57 Annex 4: Summary analysis of survey responses ...................................................................... 59

Tables and figures

Table 1: Survey responses ................................................................................................................... 10 Table 2: ISPC expense items................................................................................................................ 16 Table 3: ISPC Technical Activities / Work Stream budget .................................................................... 16 Table 4: ISPC outcome-based budget .................................................................................................. 16 Table 5: Evolution of the Scientific Advisory body in CGIAR ................................................................ 20 Table 6: Citations of ISPC Science Fora Articles as of September 2017 ............................................. 25 Table 7: Characteristics of current ISPC and possible new body ......................................................... 52 Figure 1: ISPC Theory of Change ......................................................................................................... 25

4

Executive Summary

The Independent Science and Partnership Council (ISPC), a “standing panel of experts appointed by the System Council to serve as an independent advisor to the System Council on science and research matters”1, was established in 2011.

This evaluation has been commissioned by the Independent Evaluation Arrangement of CGIAR (IEA) and forms part of its 2017 work plan2. The evaluation ran from March to October and was primarily desk-based.

The formal terms of reference of the evaluation require it to assess the ISPC against four criteria – value, relevance, functional performance and operational performance – to achieve two main objectives:

i. to provide accountability to the System Council and CGIAR as a whole on the relevance, value-added and overall performance of the ISPC with respect to all dimensions of the ISPC’s functions and work;

ii. to draw lessons and make recommendations for the future, with a view for the ISPC to best serve the System Council and CGIAR as a whole in the context of the governance reform and the implementation of the Strategy and Results Framework (SRF) 2016-30.

As signaled in its Inception Report, the evaluation team also took into account two strategic issues throughout the evaluation:

• Have ISPC contributions led to and are further contributions likely to lead to improvements in the overall delivery of CGIAR’s vision, mission and goals (reducing poverty, improving food and nutrition security, and improving natural resources and ecosystems)?

• The counterfactual proposition – what would be the effect on the System if the ISPC didn’t exist?

To reach its findings, the evaluation triangulated evidence from three main sources: extensive documentation from and about the ISPC and CGIAR more generally, particularly the changes and reforms over the past ten years; a large number of interviews with stakeholders, including System Council and ISPC members; and a survey of center board chairs, directors-general and deputy directors-general and CRP independent steering committee chairs, directors and flagship leaders. We also reviewed literature and drew on our own experience about what is needed for effective science advice in complex science research systems.

In summary, the evaluation found that the ISPC has professionally delivered significant output. Major achievements across its work streams have included three well attended science fora, a strategic study in agriculture and food systems partnerships (2015), several well received reports and expert events on research impact, including a recently held workshop organized jointly with the CRP on Policy, Institutions and Market (PIM) in Nairobi (2017) on rigorous evidence for policy formulation. ISPC also undertook two rounds of reviews of the CRP proposals and platforms between 2010-16. In its strategy and trends work stream it managed several studies such as Biotechnology in CGIAR (2014) and Data, Metrics and Monitoring (2014). ISPC was also deeply involved in the development of the SRF. In the past two years, it has stepped up its engagement in foresight analysis. Since the Mid-Term Review (MTR) in 2014, ISPC has worked hard during a challenging period when its terms of reference have

1 As defined in both the CGIAR System Framework and the Charter of the CGIAR System Organization. 2 http://www.cgiar.org/wp-content/uploads/2016/11/Exhibit-2_IEA-2017_PWB_Oct2016.pdf

5

been the subject of ongoing discussion for almost three years. As its Chair noted, it has tried in this period to work out what form of advice on science the System needs, and delivered on that.

Nevertheless, despite the significant output produced by ISPC, there is considerable debate at the System Council, its standing committee, SIMEC, the System Management Board and among donors about what exactly the ISPC should do in the future. There is general agreement that there needs to be a strong capability to identify the research challenges to delivering on CGIAR’s goals as well the ability to identify which new research developments should be harnessed to address these challenges (foresighting). There is also agreement that measuring and evaluating the impact of research in the System is vital.

Debate is ongoing about the utility of other roles currently carried out by the ISPC and whether some of the ISPC outputs would be better delivered through other mechanisms. Another part of the debate is whether the ISPC is value for money.

In the light of these issues and despite the considerable achievements of the ISPC, the evaluation makes the following recommendations which, in summary, suggest that the nature of the ISPC work today (a mix of a think tank and an assessor) should change, and be replaced by a more closely connected advisory body for the System:

Recommendation 1 – establishing what kind of advice and advisory structures are needed That the System Council, including through its SIMEC, continues to move its focus from trying to finalize terms of reference for advisory bodies to re-addressing first the questions of exactly what types of advice the System needs, secondly how to measure the quality of this advice, and after these matters are settled, how that advice is best commissioned and delivered, be it via standing committees or specially-commissioned ad hoc arrangements, or a mixture of the two. Recommendation 2 – a possible new advisory body That, if the System Council decides to proceed with a standing committee, it avoids minor adjustments to the current ISPC given the widespread unease with the ISPC, and considers more radical change by establishing a new, high-level and eminent science/research/ innovation/development body with a new name and a new mission, with the characteristics set out below.

This body would be a formal but independently constituted and operated sub-committee of the System Council and would have strong links to the System Management Board. It would receive formal references from both bodies, requesting advice and guidance both on major science/research/innovation/development issues and on processes. In turn, the new body would provide the advice needed back to these bodies within agreed timeframes. It would also have the ability to send advice it initiated itself to the two bodies for consideration. The governing bodies would maintain the discipline of formally responding to the advice. Operationally, the new body would have strong working links with sub-committees of both bodies (such as their SIMECs) and would be served by a high-quality secretariat operating to support the independent thinking of the new body but maintaining strong links with the System Office. Options for enhancing these links and achieving greater efficiency and economies of scale should be carefully considered by the System Council in consultation with the new body and System Office.

The chair needs to be:

• an ex officio, non-voting member of the System Council reinforcing the centrality of research in CGIAR. Whether the chair could also be a full member of the System Management Board, or at least an Active Observer, should be considered;

6

• a globally renowned individual with a deep knowledge of science and development issues who is a proven, effective chair; a talented leader; and an outstanding communicator;

• a person with substantial availability, at least about a quarter of their time; • a person who can and will work closely with the chairs of the System Council and the

SMB and their SIMECs, as well as the centers and CRPs, to ensure the System research agenda is appropriately brought forward, debated and acted on in a way that allows CGIAR to tackle really big challenges effectively.

The membership of the new advisory would be relatively small with up to six members in addition to the chair, all of whom would be eminent as leaders in making complex research systems work well. Some might be world-famous researchers heading major research laboratories, and others might be senior figures providing effective advice on research priorities and change through guiding national and transnational research, innovation and development systems. For example, members might be winners of the major global prizes; others might be equally eminent as leaders in fields such as energy, sustainability and research systems. Their eminence would likely limit their availability to serve more than, say, 15 days a year. The members would be drawn from diverse backgrounds. Gender balance and appointment of people from developing countries would be important. Terms would be 3-5 years with a rolling appointment structure so the whole body does not turn over at once.

Given the limited time availability of members, it will be important for the body to be supported by strong secretariat that can work with others to assist the System Organization to follow through on decisions about science, research, innovation and development impact.

It is important to recognize what the new body will not do. It will be a body providing excellent and appropriate advice but it will not be operationalizing this advice although it might make suggestions on how to carry out the operation and it would comment on evaluations.

The System Charter would change to state that the Science, Innovation and Development Committee (or whatever name is chosen) is a “standing panel of experts appointed by the System Council to serve as an independent advisor to the System Council on science, research, innovation and development matters”. Recommendation 3 – a simple measure of success That the System Council adopts a relatively simple metric for assessing success of the new body, such as that the System Council and System Management Board find their interactions with the new body deliver significant insights and help to the System Council and SMB in major areas of concern. The success or otherwise of the body would be assessed at least once annually. If it is not meeting expectations, it should be disbanded quickly and another mechanism such as commissioning ad hoc advice substituted. As well the new body would also need to:

• be seen to add significant value to the System in terms of reaching the System goals more effectively and sooner;

• provide advice the System absorbs and uses; • be perceived as good value for money.

Recommendation 4 – some existing structures will be needed to support the new body That, while the new advisory body would offer high-level commentary on issues such as mechanisms for research assessment, evaluation and metrics, specialist bodies such as SPIA and specially constituted assessment panels would still be needed to feed into the new advisory body and to carry out the detailed work involved in evaluations, assessments and measuring research quality.

7

Recommendation 5 – transition arrangements That, given the multi-year hiatus on the formal arrangements for high-level science advice to the System, any new body should be planned to commence by end of 2018 at the latest and the current ISPC would finish up by the same time. Two other options for Recommendation 2 The evaluation notes that there is no absolutely right way to deliver science advice centrally in a complex research system and therefore offers two other options to Recommendations 2, 3, etc. The first is the option of not having a central science advisory body but rather relying on ad hoc advice commissioned when needed, and the second is for the System Council through its SIMEC to work with the ISPC to achieve any needed change in ISPC’s focus, mode of operation and membership to deliver the type of advice the System most needs and will be able to absorb and respond to.

8

1. Introduction

According to the System Charter and the System Framework, the Independent Science and Partnership Council (ISPC) is intended to be a “standing panel of experts appointed by the System Council to serve as an independent advisor to the System Council on science and research matters”3. It was established in 2011 (an interim ISPC operated in 2010) following major reforms to the CGIAR System. It replaced the Science Council.

1.1 Structure of the evaluation This evaluation of the ISPC was commissioned by the Independent Evaluation Arrangement of CGIAR (IEA) and forms part of its approved 2017 work plan.

Mary O’Kane & Associates Pty Ltd (nominated person Mary O’Kane) and Dr Eija Pehu were engaged in March 2017 to undertake the evaluation.

Since its establishment in 2001, Mary O’Kane & Associates has a long history of international reviews and evaluations in science and research. Professor O’Kane is a senior figure in the Australian science and innovation system and a former university president. She also has experience in development, having been a long-term member of the Australian Aid Advisory Council, and chair for several years of the board of Development Gateway, an international development not-for-profit entity based in Washington.

Dr Pehu has a strong background in agriculture and development. She was a Professor of Agronomy and Head of the Department of Plant Production at the University of Helsinki before holding the position of Science Advisor in the Agriculture Global Practice of the World Bank from 2000 to 2016. In that capacity she led the Department’s program on agricultural research, extension and innovation interacting with external partners, including CGIAR and academia, as well as national and regional research organizations.

Further details on Professor O’Kane and Dr Pehu are available at Annex 1.

1.2 Evaluation terms of reference The formal terms of reference for this evaluation require the evaluation to assess the ISPC in terms of the following four criteria:

• the relevance and scope of the ISPC’s leadership and advisory functions as well as its work, in relation to past and evolving System needs and expectations. distinguishing various stakeholder groups, and in a context of emerging challenges in agriculture research for development

• the value ISPC adds to the System overall (including the ISPC’s contributions to CGIAR in the development of the latest SRF), to other actors in the System that have similar functions at different levels, and to the general environment in which scientific direction is provided to centers and CRPs

• the functional performance of the ISPC as a whole and in its areas of activity, including its credibility, which depends on the independence and quality of the ISPC’s advice, and on the utility and influence of its products and services for the System Council and for the whole CGIAR scientific community

• the operational performance of the ISPC as a whole and in its areas of activity, including the extent to which the governance, management and capacity of the ISPC (including SPIA and the ISPC Secretariat), optimally support the ISPC in delivering on

3 As defined in both the CGIAR System Framework and the Charter of the CGIAR System Organization.

9

its mandate; and the extent to which the recent changes in CGIAR governance have impacted on the ability of the ISPC to deliver.

The evaluation has two main objectives:

i. to provide accountability to the System Council and CGIAR as a whole on the relevance, value-added and overall performance of the ISPC with respect to all dimensions of the ISPC’s functions and work;

ii. to draw lessons and make recommendations for the future, with a view for the ISPC to best serve the System Council and CGIAR as a whole in the context of the governance reform and the implementation of the Strategy and Results Framework (SRF) 2016-30.

As stated in its Inception Report, and following interviews and investigations for that report, the evaluation team quickly came to the view that two strategic overarching questions were needed to frame the evaluation. They are:

• Have ISPC contributions led to and are further contributions likely to lead to improvements in the overall delivery of CGIAR’s vision, mission and goals (reducing poverty, improving food and nutrition security, and improving natural resources and ecosystems)?

• The counterfactual proposition – what would be the effect on the System if the ISPC didn’t exist?

To answer these questions, the evaluation team framed the following core questions (relevant evaluation criteria noted in bold) to help it respond to the two framing questions:

• Does everyone involved in the System have a clear concept of the ISPC’s formal role and functions, what it actually does, who it reports to and how it communicates with different stakeholders, and how it assesses the impact of CGIAR research? (relevance, functional & operational performance)

• Where is the ISPC very effective and where is it less effective? (value) • Which parts of the System particularly value the ISPC and why, and which parts have

reservations and concerns and why? (value, functional performance) • How do we know the ISPC is working, and are the current key performance indicators

(KPIs) for measuring effectiveness correct? (relevance, value, functional & operational performance)

• What changes could be made to the ISPC and inter-connecting bodies to make it highly effective, who would need to act and over what time frame, and which changes would need to be done quickly and which could be implemented at a longer time-frame?

The evaluation terms of reference contained a specific range of sub-questions related to the four evaluation criteria. All of these are addressed throughout the report, as part of the analysis and findings. In terms of relative weights, the analysis and recommendations focus on relevance and value of the main science advisory body in the reformed CGIAR, and less on ISPC’s functional and operational performance. This forward-looking approach was discussed and agreed with IEA at the onset of the evaluation.

1.3 The approach of this evaluation The evaluation triangulated information from a range of different sources, notably extensive CGIAR documentation, interviews, results of a survey, and literature and experience on what is needed for effective advice regarding science in complex research systems. The triangulation highlighted where views are consistent and where they are divergent.

10

1.3.1 Information Literature We studied the extensive literature on the ISPC, reading meeting summaries and papers, budgets, work plans, activity reports and annual reports of CGIAR entities, including the ISPC, the System Council and the SMB and their predecessor bodies, as well as a variety of other current and historical documents available from the CGIAR website and elsewhere or provided by the IEA.

ISPC outputs We studied the outputs of the ISPC, including its various reports, summaries of ISPC-organized fora, etc. Where available, we studied evaluations of these outputs.

Interviews We conducted over 70 interviews, including 12 current System Council or SIMEC members, the current and former ISPC Chairs, 11 past and present ISPC members, and a range of other participants in the System, such as System Management Board members, center board chairs, CRP directors, directors-general and deputy directors-general for research, System Office and ISPC secretariat staff, donor representatives, and others who had perspectives of relevance to this evaluation.

A list of interviews is at Annex 3.

Survey A qualitative survey consisting of six open-ended questions was sent to stakeholders, and the response rates are listed below. A summary of the analysis of the survey is at Annex 4.

Table 1: Survey responses Stakeholder categories Total sent No of responses Response rate Center Board Chairs 17 7 41% Directors-General 15 8 53% Deputy Directors-General 15 4 27% CRP ISC Chairs 9 5 56% CRP Directors 16 9 56% Flagship Leaders 64 19 30% Total 136 52 38%

Understanding effective advice on science We reviewed a range of literature on incorporating science advice into complex research systems, and on the way governments and international organizations draw on science advice to deal with major policy problems. We also drew on our own experience, including Professor O’Kane’s service as a government chief scientist, as a member of the board of Australia’s CSIRO, and her experience as a lead generalist evaluator of research programs in various countries as well as Dr Pehu’s experience as a science advisor in the World Bank.

1.3.2 Analyses and what can be derived from them From this material, we have been able to do:

• systems analysis – how ISPC functions as part of the CGIAR System, and to what extent its advice as an independent scientific advisory body enables CGIAR to be more relevant and effective as a whole in reaching the System goals;

• organizational analysis of ISPC (different pillars of work, effectiveness, relative weights, gaps) and examination of ISPC operating processes;

• historical analysis – especially since the 2008 CGIAR reforms; • financial analyses – changes to CGIAR funding and how that affects ISPC’s work; and

budget;

11

• assessment of the relevance of the ISPC science advice to the System. From the analyses, we have been able to identify major themes, concerns and suggestions for change, including the alignment and harmonizing of current ISPC activities with CGIAR’s key purposes and its messaging and, ultimately, with its delivery on CGIAR’s vision, mission and goals.

1.4 Limitations of the evaluation The evaluation was set up by the IEA to be primarily desk-based. Only Dr Pehu travelled for the evaluation, attending the annual CRP leaders/DDGs Research meeting in Montpellier and meeting ISPC secretariat staff in Rome in June. However, Professor O’Kane along with Dr Pehu participated in some meetings by teleconference (see Annex 3), and both attached some face-to-face meetings to travel for other purposes. The majority of the interviews therefore had to be conducted remotely, which did not allow the interviewers to pick up on body language or some of the subtleties of a discussion.

It became clear particularly through the interviews that there are quite differing views of the contributions of ISPC to the System. As we have only been able to interview mainly senior people in the System, and not many researchers, this evaluation does not reflect researchers’ views unless they also have a leadership or management role.

There are many sources of science advice in the System. It is clear that the principle of subsidiarity is (appropriately) quite strong across CGIAR with groups drawing their advice primarily from the advice structure most immediate to them. Understanding all the complexities of this advice and how they interact, if at all, is beyond the scope of this evaluation.

Finally, the evaluation team did not have an opportunity to observe the main System level meetings, e.g., of the System Council, the System Management Board and the ISPC itself. However, the team has conducted interviews with most System Council and ISPC members, and a wide range of other System participants.

12

2. The ISPC: its structure, remit, history and context

2.1 Structure 2.1.1 Composition The ISPC comprises 9 members including the chair.

The composition of the ISPC was established by the Fund Council, on the recommendation of the Fund Office, in February 2010, following the 2009 decision to establish the ISPC. An interim ISPC operated for the first 12 months, until appointments were made to the new ISPC which held its first formal meeting in March 2011. At that time the council consisted of seven members including the chair, as well as the chair of the standing panel on impact assessment (SPIA) ex officio.

The ISPC chair was to be appointed for an initial period of three years, with the possibility of extension for up to a total of five years, with the position close to half-time. Regular members were to be appointed initially for two years, with a possibility of renewal up to four years without further extension.

a. Chair The ISPC has had just 2 chairs. Ken Cassman was the inaugural chair, from 2011 to 2014. Maggie Gill has been a member of the ISPC since its first meeting, and became chair in March 2014, making a total of 6½ years on the council. The ISPC Chair was described in the MTR Report as “essentially the chief scientist of CGIAR”4.

As well as chairing ISPC meetings, Professor Gill is an Active Observer on the System Council and the System Management Board5. She also attends, and participates in, many other meetings across the System.

Professor Gill is contracted on a part-time basis. Since 2014, she has variously worked between 50% and 70% of a full-time load, and is currently working at 60%. Such a high-level involvement of the chair matches and indicates the nature of the work and the breadth of the agenda of ISPC today. The chair and the council members get quite deeply involved with the work of the council (the program reviews, managing content for reports, etc.).

The evaluation team noted widespread respect for Professor Gill and her commitment to CGIAR, evidenced by the range of activities and meetings she participates in.

b. Members Currently, the ISPC has 7 appointed members in addition to the chair and the SPIA chair who has always been an ex officio member.

During its early years the ISPC operated with 5 appointed members. In 2015 this increased to 6 “in anticipation of the final ISPC Task Force report” 6 , and in 2016 to 7. The 2016 appointments were intended to “strengthen the ISPC’s coverage of natural resource management, foresight, priority setting, and partnership issues”7.

Members are appointed for fixed terms in their individual capacities (this is one of the core elements that goes to keeping the ISPC independent). Between them they have a wide range

4 Final report from the Mid-Term Review Panel of the CGIAR Reform, October 2014, p.36. 5 Professor Gill was a Standing Observer on the SC’s predecessor, the Fund Council, but not on the Consortium Board. 6 End of Meeting Report, ISPC 11th meeting, 30 March – 1 April 2015, p.30. 7 ISPC Work Plan and Budget 2017, October 2016, p.15.

13

of biological sciences, economics and development expertise and experience. More on their profiles is available on the ISPC website8.

A recent self-evaluation by the ISPC indicated that half the council members think the council has the appropriate balance and mix of skills, while the other half think it could be improved9. They noted that, without a clear mandate for the ISPC, it was difficult to say what skill mix would be optimal. Several interviewees echoed similar sentiments. The evaluation team’s view is that, overall, the balance of disciplines and skills is reasonably appropriate for the ISPC’s current remit. However, thought needs to be given to major recent trends in science – thus at some point a member who is a key researcher in using data analytics in agriculture would be a useful addition to the skill mix. The evaluation team commends the System Council for recently strengthening the ISPC’s capacity for foresighting.

Of the 13 members who have served on the council since 2014, one (chair of SPIA) has worked 50-60 days per year; 3 members 40-50 days per year; 4 members 30-40 days per year; and 4 members less than 30 days per year10. Contributions have varied based on the particular work program demands and years on the council (more familiarity, more work input) and relevance of the council member skillset.

Council members receive a daily fee of $76011. Most work virtually from their home base attending two to three face-to face-business meetings of the council per year plus special meetings such as forums. Again, the relatively high time allocation reflects the nature of the ISPC today, the depth of involvement by council members in planning and supervising the work, and the somewhat academic working culture. This suggests that the ISPC is, in practice, operating more as a ‘think-tank’ than simply being an advisory body.

In the recent self-evaluation of the ISPC many council members called for greater empowerment of the ISPC, and active use of the advice provided. Many also felt that their expertise is under-utilized by the System12. The evaluation team agrees.

The evaluation team commends council members for their excellent work, commitment and time allocation to ISPC. The evaluation team, however, agrees with the comment at the end of its recent self-evaluation that:

The major challenge for the ISPC in the next year is to clarify better its own channels for impact. We need to ask some tough questions about our Theories of Change for each of the five work streams. We need to ascertain that there is really demand for the ISPC's intellectual input on the topics of the work streams13.

c. Appointment process Since the ISPC’s establishment in 2010, the two chairs and all members have been appointed on the advice of an ISPC Search and Nomination Committee (SNC) which was itself established in 2010. The chair throughout this period has been Peter McPherson (President, US Association of Public and Land-Grant Universities). The Fund Office has supported the process for selection of new ISPC members until 2016 – when ISPC Secretariat took over this role – including obtaining approval of the Fund Council for new appointments by circulation14, on a no-objection basis.

8 http://ispc.cgiar.org/about/people . 9 ISPC Council Self-Evaluation. Summary (undated). 10 Information provided by ISPC. 11 Information provided by ISPC. 12 Ibid. 13 ISPC Council Self-Evaluation. Summary (undated). 14 Based on CGIAR Fund Office Report to the 14th meeting of the Fund Council, October 2015, p.4.

14

Factors taken into consideration in selection of ISPC members include “level of capability and ability to contribute to the work of the ISPC, the range of areas of expertise needed for the work of the ISPC as well as the gender and geographical makeup of the ISPC”15.

The revised draft Terms of Reference for the ISPC circulated in March 2017 (and discussed further below) propose that future ISPC members be appointed by the System Council, either on the nomination of a System Council-appointed SNC, or on the nomination of the System Council’s SIMEC. The evaluation has been advised that the SNC has been disbanded and nominations will be proposed by the SIMEC16.

The evaluation team commends this change, noting that the ISPC input on what was needed did not seem to have been fully factored into the final recommendations for appointments.

2.1.2 Mode of operation The ISPC holds two meetings a year with open and closed sessions, and with a variety of other meetings attached. The subjects of key technical presentations are selected so that the discussions and resulting outputs can contribute to advice for the next System Council meeting – the ISPC meetings are typically timed to occur not long before System Council meetings. As at October 2017 the ISPC has held 16 meetings. These are attended by the ISPC members and Secretariat staff and 30-50 additional attendees including scientists, donors and hosting center staff. The number of additional attendees typically reflects where the ISPC meeting is held; the open meetings providing a good opportunity for staff from local CGIAR research entities to familiarize themselves with ISPC matters.

In the recent self-evaluation, the council members indicated their satisfaction with the frequency and length of these meetings, but called for a stronger focus on System-wide strategic issues. Some also noted that early availability of background documents would improve effectiveness of the discussions. ISPC also has short and focused monthly teleconferences. Many interviewed council members indicated their satisfaction with these. These teleconferences are well organized and chaired, and provide a good briefing on current issues in the System.

The evaluation commends the council’s commitment to, and implementation of, self-evaluation.

The chair assigns particular tasks, such as forum planning, foresighting, prioritization, communication to various council members. Each council member then has a Secretariat staff member to support the thematic work.

The ISPC also has its own website, http://ispc.cgiar.org. The site does not provide a direct link back to the CGIAR’s own homepage, but does contain comprehensive information on the ISPC’s work streams and outputs, as well as events, news and meeting details, including end of meeting reports. The website was updated and modernized in September 2017. The evaluation team considers it a major improvement on the previous one, and recognizes that more effort is being made to disseminate the ISPC’s role and outputs.

With the number of days contributed by council members, the large Secretariat and the range of activities it engages in, the ISPC’s role appears to be much more than an advisory one. When it is engaged in the review and assessment of CRP proposals, it acts more like the assessment panel of a major research funding organization. But it also operates like a think-tank in respect of the Science Fora and the commissioning of strategic reports. And, just like

15 Email from Peter McPherson to Interim Executive Director, CGIAR System Organization, 20 July 2016. 16 Email from ISPC Executive Director to evaluation team, 1 August 2017.

15

a think-tank, its members and its Secretariat like to operate at arm’s length rather than up close with the System’s governance. That said, the ISPC noted, in responding to a draft of this report, that the degree of collaboration/consultation with the System’s governance bodies varies across work streams and for specific activities. It pointed out that there was close collaboration with the Fund Office and Consortium Office in the development of the Strategy and Results Framework and the guidelines for CRP Phase 217. It also cited other examples, including the Science Fora, the development of a frame of reference for Quality of Research for Development (QoR4D)18, and the impact assessment work stream.

2.1.3 Budget and work plan The ISPC submits an annual work plan and budget for approval by the System Council, which is followed up by an annual activity report prepared by the ISPC Secretariat. Past activity reports were sent to the Fund Council’s Peer Review Team (PRT) which was responsible for reviewing CGIAR Entities’ System Costs and Efficiencies. The report (which includes both technical and financial sections) is submitted after the first quarter of the following work plan & budget year. It includes a summary table of achievements against planned and approved activities. The PRT no longer functions, and the System Office19 will now be responsible for sending the report to the System Council. The PRT mainly focused on the budgetary aspects of ISPC in its reporting to the System Council. The evaluation team believes that in taking over the role, the System Office should include a deeper discussion about what the System Council needs that year from the ISPC (or any replacement body); and suggestions from the ISPC about how it can enrich the CGIAR science agenda.

The most recent work plan, for 2017, was approved by the System Council in November 2016 without, according to interviews, much discussion on its content20.

Until 2015 the ISPC was funded partly through FAO, which still hosts the ISPC Secretariat and manages the associated finances under an agreement that goes to 2022. For example, in 2014, 36% of the ISPC’s budget was from FAO21; in 2015, it was 42%22 .From 2016 the ISPC has been funded totally from CGIAR funds23 but FAO still provides in-kind support for accommodation and related overheads such as IT support. The budget is intended to cover the cost of: the ISPC Chair and Council members; the ISPC secretariat (professional and support staff and consultants); carrying out ISPC technical activities; travel; and meetings, communications (web, publications etc.) and miscellaneous expenses24.

The indicative budget for 2017 for ISPC is $3.52M25, which represents 22.7% of the CGIAR System budget, and 0.39% of the total CGIAR Portfolio26. Expense details from the 2017 ISPC work plan are as set out in the table below. The work plan does not provide detail of any external sources of funding to the ISPC, e.g. SIAC project grants to the SPIA, or in-kind accommodation support from FAO. It operates as a funding request to the System Council, for a specific amount of CGIAR Funds rather than a budget. Likewise, the breakdown by work stream does not indicate what additional funding from other sources might contribute to these

17 As evidenced, for example, by the Final Meeting Summary, 13th CGIAR Fund Council meeting, Bogor, 28-30 April 2015, when the Fund Council chair “highlighted” the engagement of the ISPC in the SRF’s development. 18 ISPC, Quality of Research for Development Workshop: Inputs and Way Forward. Brief number 52, March 2017 19 The evaluation team has used System Office, not System Management Office, throughout this report, noting the reference in the ISPC 15th End of Meeting Report that System Office is the preferred name. 20 See SC3-04 Funding System Actions and Entities Budgets– Exhibit 1 ISPC Work Plan and Budget 2017, 3rd CGIAR System Council meeting. 21 ISPC Work Plan and Budget 2016, October 2015. 22 ISPC Work Plan and Budget 2017, October 2016. 23 Ibid. 24 Ibid. 25 $ in this report refers to USD. 26 SC2-05 2017 CGIAR system entity budget envelopes, Item 8, 2nd CGIAR System Council meeting.

16

work streams. A full budget including external sources and in-kind contributions would provide a more complete picture of ISPC’s operations.

Table 2: ISPC expense items27

Expense items 2015 Actual $000

2016 Actual $000

2017 Budget $000

Council honoraria (chair & office; Council & panel members)

345 382 385

Work streams 993 1,352 1,120 Personnel Costs 1,726 1,635 1,795 Travel 218 1990 155 Operating expenses 210 74 65 Additional funds to support 2015

Task Force 59 83

GRAND TOTAL 3,551 3,446 3,520 Total from CGIAR 2,561 3,725 3,520 Total from FAO 1,830

The 2017 budget also includes two breakdowns by work stream, one for technical activities, and one encompassing, for the first time, the entire CGIAR allocation:

Table 3: ISPC Technical Activities / Work Stream budget28

2015 actual $000

2016 actual $000 2017 budget $000

Foresight and prioritization* 56 159 185 Independent Program Review

86 165 105

Science Dialogue** 147 525

125 Agri-food system Innovation and Partnership**

22 175

Impact Assessment 682 503 530 * Part of Strategy & Trends in 2015 and 2016 ** Part of Mobilizing Science/Partnerships in 2015 and 2016

Table 4: ISPC outcome-based budget29

Works streams / Outcomes $000 % of total Foresight and Prioritization 742 21 Independent Program Review 477 14 Science Dialogue 458 13 Agri-food system Innovation and Partnership 597 17 Impact Assessment 1,246 35 Total CGIAR requested budget 3,520 100

Clearly, the best funded work stream is Impact Assessment (and this does not reflect any funding for the SIAC program from the Gates Foundation and DFID). There is also increased funding for foresight and prioritization, “in order to respond to greater demands for ISPC work and guidance in these areas”30. The greater funding for Agri-food System Innovation and Partnership reflects the “increased importance of bridging research to impact in the new

27 ISPC Work Plan and Budget 2017, October 2016 for 2015 actual and 2017 budget. 2016 actuals provided by IEA. 28 Ibid. 29 ISPC Work Plan and Budget 2017, October 2016. 30 Ibid., p.3

17

CGIAR system and the need for a system-wide approach”31. The breakdown presented here is a snapshot of one year’s activities indicating the dominant share of SPIA from ISPC’s internal resources. This remains fairly constant over the years as does the level of the total annual budget, while the relative shares of the work streams fluctuate quite a bit annually.

2.1.4 Secretariat A full time Secretariat, hosted by FAO in Rome, supports and manages the work of the ISPC.

The Secretariat comprises an Executive Director, 9 other staff and 2 consultants. Secretariat staff are employed by FAO although the costs are paid by CGIAR. Each work stream is allocated secretariat staff, who work with the council members assigned to the work streams by the chair. The new Executive Director has a participatory and inclusive style of management and has re-organized the workflows and staff allocations across the work streams. She has also organized the support staff team. Being familiar with FAO, she has started to build linkages between the ISPC and FAO research and extension units, and with IFAD, the World Bank and the FAO Strategic Program on Rural Poverty Reduction and Sustainable Food and Agriculture. The staff interviewed appreciated these changes and the open communication style of the Executive Director. The impression of the evaluation team is that the Secretariat staff, both professional and supporting staff, make an effective and dedicated team led by a competent Executive Director. This impression was widely shared by the people interviewed across the stakeholder groups, and expressed by the council members in the ISPC self-evaluation.

But as discussed below there were several who questioned the overall cost of the ISPC Secretariat. The evaluation team considers the size appropriate for the current work plan and methodologies used by the ISPC but, if the remit of the ISPC changes, the size of the Secretariat should be adjusted accordingly.

Complex and often slow bureaucratic FAO processes was frequently referred to in interviews, by different classes of stakeholders32. Also, the professional staff appointment processes of FAO limit the independence of ISPC to make decisions on hiring staff. In the view of the evaluation team, it is essential for an independent science council like the ISPC to be able to recruit the best professionals to its team in a timely fashion, so top candidates can be engaged.

2.2 Context in which the ISPC operates In this evaluation, the complex context in which the ISPC operates was a dominant and recurring feature of the interviews carried out. This context is not only complex but has also been changing significantly over several years. While most interviewees talked primarily about the changing CGIAR and donor context, ISPC is also affected by major changes in science and research and in the way science advice is managed.

2.2.1 Societal grand challenges and CGIAR The great challenge of the 21st century is to end poverty and hunger so that the food system can feed everyone, raise real incomes of the poorest, and provide safe and nutritious food while stewarding the finite natural resource base33. Agriculture with its reliance on land, water and biodiversity is a significant part of this challenge. But it is, and increasingly can be, a part

31 Ibid., p.4. 32 The ISPC evaluation team also noted that the 2016 SIAC evaluation report stated that “the administrative systems of FAO have been a major constraint for the project, leading to delays and limitations on contracting partners, and the closure of the incipient SIAC small grant scheme”: CGIAR-IEA (2016), Evaluation of the “Strengthening Impact Assessment in CGIAR” (SIAC), Project Phase 1, 2013-16. Rome, Italy: Independent Evaluation Arrangement (IEA) of CGIAR. 33 2015. Ending Poverty and Hunger by 2030. An Agenda for the Global Food System.

18

of the solution. Science, basic and applied, done creatively in a disciplinary or inter- and trans-disciplinary context, and framed around the issues of sustainable and climate smart food systems set within a broader bioeconomy, is possible and essential. CGIAR with its research competence in a range of agriculture sub-sectors and disciplines, wide partnerships, problem-oriented mission and sharpened governance structure is uniquely placed to lead this effort.

But the pace of change is fast; erratic weather events and prolonged droughts are increasing in frequency; the macro change of rural-urban transformation is advancing rapidly in many countries; the demographic pressure of young people reaching working age is daunting; and the science itself, increasingly enabled by big data and data analytics, is changing quickly. Also the scientific organizations in developing countries and their innovation systems at large are developing, which raises the need for a fresh look at partnering models. Effective partnering between the CGIAR on the one hand and development agencies and the private sector on the other is a long-persisting dilemma. CGIAR needs to be at the pulse of these changes, at the table when these issues are discussed, and constantly asking questions on how to use agricultural science to respond most effectively. There is little time for poor choices, or poor science, or science disconnected from partners in research and development.

2.2.2 Strength of the System amidst reform and financial uncertainty CGIAR is a successful organization that is unique in the scale and scope of its activities. In its 45 years of existence it has achieved a great deal. As Bill Gates said in 2012: “If the CGIAR system didn’t already exist, we’d need to invent it”34.

The System has also been through considerable organizational change, and financial crises, during its long history. Most recently, the great reforms over 2008-10 have led to the operationalizing of its Strategic Results Framework – built in part to address the Millennium Development Goals – through multi-disciplinary, cross-institutional CGIAR Research Programs (CRPs), and to new, more centralized, governance structures in 2016. Coinciding with these changes have been increasing financial pressures in donor countries, and changes in donor priorities about how they spend their aid budget generally.

Other major influences on CGIAR over the past 15 years include:

• the growth in CGIAR funding portfolio from under $500M at 2007 to over $900M a decade later;

• the creation of the CGIAR Fund, with its three funding Windows (W1: co-mingled, untied funds largely allocated to CRPs by System Council decision; W2: funds to donor-selected CRPs; W3: funds to donor-selected specific CGIAR centers); and the rapid growth in total annual funding that followed the establishment of the Fund over the five years to 2015, though this was accompanied by a marked decline in the proportion associated with W1 and W2. The combined share of W1 and W2 dropped from 84% of the Fund to 44% with a corresponding increase in W3 from 16% to 55% from 2011 to 201535;

• the influx of big philanthropic donations (e.g. from the Gates Foundation), alongside Official Development Assistance from donor countries;

• with the introduction of three platforms (in addition to CRPs), an acknowledgement that the System is paying greater attention to research infrastructure issues;

34 Comment during dialogue with 7th CGIAR Fund Council meeting at Bill and Melinda Gates Foundation, March 2012: http://www.cgiar.org/cgiar-fund-council-holds-meeting-at-gates-foundation-bill-gates-highlights-the-value-of-the-cgiar-investing-in-agriculture-march-2012/ 35 Wadsworth, J. 2016. Discussion Paper – April 2016 Brief history of the CGIAR Fund.

19

• significant changes in corporate governance worldwide, especially post the 2008 Financial Crisis, requiring the members of structures like CGIAR center boards to adhere to much greater fiduciary responsibilities and higher governance standards;

• the support for the System that is provided through an established System Office in Montpellier (though the ISPC secretariat and IEA are hosted by FAO in Rome).

CGIAR now has a powerful structure to take on the big contemporary agri-food systems for development challenges, including climate change. Whether it has sufficient mechanisms to ensure that the CGIAR’s overall goals and activities are continuously linked to the grand societal challenges is the subject of comment later in this report.

2.2.3 Evolution of science advice at System level Since its establishment in 1971, CGIAR has always included an entity tasked with providing independent scientific advice at the system level (Table 3). The actual role and the name has changed as the System has evolved.

a. Technical Advisory Committee (TAC) In 1971, the founders of CGIAR established a Technical Advisory Committee (TAC) of international experts to advise them on research priorities, investment opportunities and the quality of science. The TAC played a prominent role in the evolution of the system in the 1970s and early 1980s. The environment was conducive to receiving its advice. Donors wanted to build a strong global research network for agriculture, the mandate for food security was unquestioned, and the financial climate was positive (7-fold increase in funding over 8 years). By the late 1980s and early 1990s donors were beginning to exert greater influence, deciding, for example, without TAC’s recommendation, to expand the mission of CGIAR to include forestry and to have two separate forestry institutions (CIFOR and ICRAF). This marked the beginning of a power shift from the TAC to donors and the chair of the System. The World Bank’s decision to change its funding modality from a balancing donor to a matching donor further weakened the means to enforce TAC-recommended System priorities. The most popular activities began to receive the largest contributions, sometimes counter to the TAC-recommended CGIAR research agenda36.

b. Science Council In 2001, with the backdrop of the Millennium Development Goals, CGIAR initiated a reform program, to focus “a major part of its efforts on large multi-institutional research programs which address specific problem areas”. These were called the Challenge Programs. The vision of the CGIAR chair, Ian Johnson, was to elevate the game of CGIAR and to open it up to cooperation with other partners. In this reform, several governance changes were initiated, among them the transformation of the TAC into a Science Council. An interim Science Council took over from the TAC while a Working Group on the Establishment of CGIAR Science Council prepared a detailed proposal for the conceptual and operational aspects. The Science Council officially began its operations in 2004 and, as recommended by the Working Group, included the additional responsibility of “helping to mobilize the best global scientific expertise for addressing the goals of the international agricultural research community”37.

A major achievement of the Science Council was the development of a new set of system priorities for a period of 10 years. While the priorities had a strong rationale and the endorsement by System management, they alone could not effect change. The Science Council did not have a ‘carrot’ (power over resource allocation), and CGIAR did not have a ‘stick’, being a consultative body, with no legal authority over centers. One way to endorse the

36 CGIAR, 2011. The CGIAR at 40: Institutional Evolution of the World’s Premier Agricultural Research Network, Selçuk Özgediz. 37 2002. Report of the Executive Council's Working Group on the Establishment of a CGIAR Science Council.

20

priorities would have been the ‘power of the purse’, but while some donors did pay attention to Science Council recommendations, many had to go by their own national priorities. In short, this period saw the start of several needed initiatives, but the governance structure called for a deeper reform.

c. ISPC In 2008, CGIAR engaged in another reform process seeking to find better alignment with funding and System priorities, and moving CGIAR more towards centralized decision making. This led to the establishment of the current ISPC, with new “roles and responsibilities” as follows:

The Independent Science and Partnership Council (ISPC) will be a standing panel of world-class scientific experts. The Council’s overarching purpose is to provide independent advice and expertise to the funders of CGIAR through services to the Fund Council and the Funders Forum. It will also serve as an intellectual bridge between the funders and the Consortium of CGIAR Centers.

The ISPC plays a vital role for CGIAR to strengthen science, to improve productivity and quality of science, to catalyze the partnering of CGIAR science with other institutions of international agricultural research and to support the important role of CGIAR as honest broker in various global debates38.

A comparison of the roles of the three science bodies, as provided in the evaluation terms of reference, is set out below.

Table 5: Evolution of the Scientific Advisory body in CGIAR39

Technical Advisory Committee (1971 – 2001)

Science Council (200240- 2010)

ISPC (201041-present)

Main Functions (+) indicates functions that have been added

• Provide independent advice and judgements on strategic issues and on the quality of the scientific programs supported by CGIAR

• Recommend research priorities and strategies to CGIAR

• Ensure the quality of research supported by the Group and its relevance to the CGIAR’s goals and objectives

• Recommend the allocation of resources among Centers in the context of CGIAR-approved priorities and strategies

• Ensuring the relevance of science

• Enhancing the quality of science

• Assessing the impact of CGIAR research

• Mobilizing the global scientific community (+)

• Contribute to the system strategy and priorities

• Promote the quality and relevance of science

• Assessing the impact of CGIAR research

• Promoting effective partnerships (+)

Membership Up to 14 (and Secretariat) 6 plus Chair 6 (8 from 2016) plus Chair

Reporting lines CGIAR as a whole (through International Centers Week and Mid-term meeting)

Both CGIAR as a whole (through Annual General Meeting) and Executive Committee

Fund Council/System Council

38 Roles and Responsibilities of the Independent Science and Partnership Council (Annex 1 of the document ISPC: Search and Selection Process, 1st Fund Council meeting, Brussels, 22-23 February 2010). 39 Table provided by IEA in the evaluation’s terms of reference. 40 Interim Science Council (2002-2004). 41 Interim ISPC 2010.

21

2.3 ISPC – its unsettled role The ISPC in its current format has been active since 2011, but for the past three years its role and terms of reference have been unsettled. Concern about this has been a consistent theme in CGIAR documentation and in evaluation interviews. The unsettling period begins with the 2014 Mid-Term Review.

2.3.1 Mid Term Review The Mid-Term Review (MTR) recommended further change to CGIAR, including the replacement of the Fund Council and Funders Forum with different bodies, namely the System Council and the System Management Board.

In its wide-ranging report, the MTR also made comments on the ISPC, and concurred with a view expressed to it that

foresight studies, exploratory initiatives, novel discoveries are not now being sufficiently addressed. Research oversight is essentially focused on the CRPs; and yet research quality, new research capacities and design of research programs are essentially managed by the Centers. Centers are of the view that some of the current modalities of executing the CGIAR’s agenda erode their capacity for global scientific leadership”42.

It recommended 43 , under the heading “Optimizing Knowledge Impact”, that the ISPC’s responsibilities be “elevated” so it can be proactive in “providing strategic guidance, foresight analyses, and assessing and reporting on quality of research results across the System”. It called for a detailed proposal for the new functions of the ISPC “or its replacement” to be prepared immediately by a Fund Council task force. It called for “independent research panels comprising world-class scientific leaders to advise on particular issues as required under the overall guidance of the ISPC Chair” and called for the establishment of a “partnership forum” to bring “partners together to share lessons and knowledge”.

By 2016, the new CGIAR System Framework44 and the Charter of the CGIAR System Organization45 were in place. Both define the ISPC as

a standing panel of experts appointed by the System Council to serve as an independent advisor to the System Council on science and research matters, including strategies for effective partnerships along the research for development continuum. ISPC is functionally independent from the System Organization and the organization hosting the ISPC Secretariat.

However, other work to address the role of the ISPC was still in progress.

2.3.2 ISPC Task Force To follow up on the 2014 MTR recommendation, a Task Force on “Strengthening the ISPC” was constituted in 2015, chaired by ISPC chair Maggie Gill. The Task Force made a number of recommendations in a September 2015 draft report. During this time, the decision to replace the Fund Council with a System Council was made. The Task Force noted that it “found it difficult to identify key actions in the vacuum of not knowing the detailed remits of the proposed System Office and System Council”46.

42 Final report from the Mid-Term Review Panel of the CGIAR Reform, October 2014, p.31. 43 Ibid., p.39. 44 See definition (n). 45 See definition 2(p). 46 Draft of ISPC Task Force Report, 9 September 2015, p.1.

22

The report included a request for a 41% increase in funding, from $3.245M a year to $4.59M, to allow ISPC to undertake the additional activities proposed in the MTR. Over 50% of the increase was to be allocated to foresighting. The report was discussed at the 14th Fund Council meeting in November 2015, but not approved. Instead, the Fund Council Chair requested that the ISPC reconsider options, in collaboration with the Peer Review Team, and provide a reduced budget proposal. The Chair also noted the different views expressed about the highest priorities to be addressed.

Key points from the discussion included the following:

... b) ... independent strategic analysis and peer reviews are essential for scientific quality control, and short- and long-term managerial decisions. ...

c) [there is a] need for greater ISPC foresight, science quality, partnerships and impact. ...

j) [there is a] delicate balance between the executive versus advisory roles and responsibilities and decision-making, and ... a strong preference for maintaining the ISPC’s advisory role without decision-making authority47.

2.3.3 SIMEC and System Council The proposed revised functions of the ISPC suggested by the Task Force were then articulated into a draft set of terms of reference by the Science Working Group of the CGIAR Transition Team in June 2016.

While the ISPC Terms of Reference are still under discussion, the System Council’s 4th meeting in May 2017 approved the terms of reference for its Strategic Impact, Monitoring and Evaluation Committee (SIMEC), establishing it as a standing committee whose purpose is to:

assist the System Council in: (1) reviewing research program evaluations; (2) overseeing the strategic direction and efficiency of the System Organization; and (3) monitoring efficiency, effectiveness and impact of CGIAR Research48.

In interviews SIMEC members stressed SIMEC’s important role in assessing the impact of CGIAR research and value for money (effectiveness and efficiency), even though it was acknowledged that this ground was partially covered by the ISPC e.g. through SPIA.

SIMEC’s role also includes advising and making recommendations related to the System Council’s:

a. Approval of the Terms of Reference for the Independent Science and Partnership Council (“ISPC”) ... that set forth their purposes and functions related to strategic impact, monitoring, or evaluation;

b. Role in the selection of the ISPC Chair and members ...49

In addition, at the 4th System Council meeting, under the Agenda item “Getting the best possible advice” there was a considerable discussion of the role of the ISPC. The action outcome was that:

47 Item 6, Meeting Summary, 14th CGIAR Fund Council meeting, 4-5 November 2015, Washington DC, p.15. 48 Terms of Reference for System Council’s Strategic Impact, Monitoring and Evaluation Committee, as set out in Appendix 1 to document SC4-AOB-Revised TOR-System Council-SIMEC_10May2017, 4th System Council Meeting, May 2017. 49 Ibid.

23

SIMEC will provide a concept note, seeking inputs from across the Council, on the way forward for ... independent science and research advice ... to be initially discussed in the coming months, in a virtual meeting, in advance of the 5th System Council meeting in November 201750.

The May 2017 version of the ISPC’s proposed terms of reference integrate a number of MTR and Task Force recommendations.

2.3.4 ISPC terms of reference: an ongoing issue This evaluation’s own terms of reference state that the evaluation is “expected to provide inputs for the finalization of the ISPC’s terms of reference”51.

At the time of starting the evaluation, the evaluation team had the impression that these were, indeed, close to finalization. However, as it progressed, it became clear from interviews with stakeholders that finalization of the current draft ISPC terms of reference was far from a certainty, and that the role of the ISPC was a continuing matter of debate at the System Council, its SIMEC and the SMB.

The evaluation team accepted an invitation from SIMEC to discuss “reflections and preliminary thoughts from the evaluation”52 by video and teleconference at the SIMEC meeting on 18 August. This was a particularly productive discussion for the evaluation and complemented useful individual interviews with most SIMEC members.

It is very clear from talking to senior stakeholders that there is ongoing unease about the cost of the ISPC, its value for money, level of foresighting and contribution to strategic direction, as well as the level of connection of the ISPC as a whole with the rest of the System. Many stakeholders are ambivalent about the profile and roles of the ISPC and suggest it is disconnected from the new governance set-up53.

Apart from the ISPC itself, none of the other groups interviewed were satisfied with the currently-proposed terms of reference.

In light of all the above, this evaluation has focused on providing an analysis that will help the System move forward to resolve the question of what science advice it needs, and how it should be provided. In doing this, the evaluation’s views have been greatly informed by the many assessments, both positive and negative, garnered from interviews, the survey and CGIAR documents.

50 See SC4-07A-Revision1, item 7, and Meeting Summary, 4th System Council meeting, 10-11 May 2017, The Netherlands. 51 Terms of Reference, Evaluation of the Independent Science and Partnership Council (ISPC), May 2017, p.5. 52 Email from System Office to evaluation team leader, 2 August, 2017. 53 The ISPC has informed the Review that it has strived to keep costs low, but its ability to return funds back to the CGIAR was reduced when FAO ceased funding the ISPC in 2015.

24

3. What the ISPC does and how it is perceived

3.1 The ISPC’s role in the CGIAR System In the ISPC Work Plan and Budget 2017, the ISPC refers to the ongoing difficulties of having its mandate and functions under review since 2014. Nevertheless, drawing on the definition of the ISPC in the Charter of CGIAR System Organization, the ISPC states that:

It is our current understanding of the ISPC’s role that has informed the recently renewed ISPC Vision, Mission, Goals and Theory of Change statements as follows:

• Vision: Our vision is that of CGIAR—A world free of poverty, hunger, and environmental degradation.

• Mission: To help strengthen the quality, relevance, and impact of CGIAR research to provide solutions to poverty, hunger, and environmental degradation, as articulated in the CGIAR System-Level Outcomes.

• Goals: To provide independent scientific, research, and partnership advice to the CGIAR System Council, and leadership to the CGIAR scientific community to develop, support, and implement research that contributes to the System-Level Outcomes and to the achievement of the Sustainable Development Goals (SDGs)54.

It has also developed a Theory of Change (see Figure 1), described as follows:

The ISPC provides the CGIAR System Council with independent advice on science, research, and partnership strategies, to enhance the Council’s capacity to make evidence-based decisions in support of effective agricultural research programs for development. It is the System Management Board, supported by the System Council, that implements the independent scientific advice from the ISPC. The ISPC provides the CGIAR scientific community with leadership on science, research, and partnership strategies to develop, and implement research that contributes to CGIAR System-Level Outcomes and to the achievement of the SDGs. It does this by drawing upon expertise across the CGIAR System, but conducts its own analysis of the information obtained to maintain the independence of its advice.55.

The evaluation team notes that the ISPC acknowledges that “it is the SMB, supported by the SC, that implements the scientific advice from the ISPC”. However, the evidence of the SMB or the System Council, or their predecessors receiving advice from the ISPC in a form that lends itself to implementation is scarce, other than the advice on the CRPs and SRF.

Until the end of 2016, the ISPC operated in four main areas of activity: Strategy and Trends, Independent Program Review, Mobilizing Science and Partnerships, and Impact Assessment. In 2017, as illustrated in the diagram below, these areas became five, mainly because the work on Mobilizing Science and Partnerships was divided into two separate work streams (Science Dialogue and Agri-Food Innovation and Partnerships)56. For the purposes of this chapter, the evaluation will look at the four streams in place until this year.

Each year the ISPC prepares a Work Plan and Budget, which is formally approved by the System Council (or its predecessor). This document also includes a summary of activities agreed in the previous year, and progress made.

54 ISPC Work Plan and Budget 2017, October 2016, pp.4-5. 55 Ibid. 56 Terms of Reference, Evaluation of the ISPC, May 2017, p.4

25

Figure 1: ISPC Theory of Change

3.2 Mobilizing Science and Partnerships 3.2.1 Output The range of output under the Mobilizing Science component of this work stream predominantly relates to the Science Fora. The ISPC has organized three of these since 2009. They have been well attended, with about 200 at the one organized in 2016. Following each forum, the ISPC has produced a “special issue journal publication to inform CGIAR research and the field at large”57. A formal evaluation of the 2016 forum has been conducted and published58. It concluded that SF 2016 was “successful in meeting its main objectives of bringing people together to discuss an important agricultural theme”. However, it “fell short on a few issues, particularly regarding maximizing diversity and stakeholder representation”. It commented that many participants were not aware of the large amount of information provided on the ISPC website prior to and during the forum.

The evaluation also includes a citation analysis for the previous three fora59. The analysis, updated by the IEA for this evaluation (see Table 6 below), indicates that a small number of papers from these fora have attracted considerable interest. This illustrates the global public goods component of the fora, and enhances the external visibility of CGIAR. However, at this juncture, the evaluation team views the internal impact and relevance of such fora for reaching System goals as more important than the public goods value.

Table 6: Citations of ISPC Science Fora Articles as of September 2017 No. of Citations

(Google Scholar) SF 2009 articles published in Crop Science, March-April,

2010, Volume 50, Supplement 1 (journal Impact Factor 1.513)

SF 2011 articles published in PNAS May

21st 2013 (journal Impact Factor 9.737)

SF 2013 articles published in Food Security, Volume 7, Issue

3, June 2015 (journal impact factor 1.517)

0 0 0 0 1 to 29 4 0 7 30 to 49 3 2 4 50 to 99 3 3 0 100 to 199 3 3 0 More than 300 2 1 0

57 See http://ispc.cgiar.org/publication/science-forum-2016-special-issue-inception-workshop. 58 ISPC, An evaluation of the ISPC Science Forum 2016, February 2017. 59 Ibid.

26

The ISPC particularly references the fora as highlights of its activities in the three years 2014-16:

• A Science Forum 2016 was held in April in Addis Ababa on the topic of Agricultural research for rural prosperity: Rethinking the pathways with a high level of participation by CGIAR scientists

• As a follow-up to the 2013 Science Forum, ISPC Briefs on Insights and recommendations from the Science Forum as well as an evaluation of the ISPC Science Fora were published in 201460.

The range of output under the Partnerships component is more difficult to identify. Nevertheless, some work has been done. In 2015 the ISPC listed as a highlight:

• ISPC responded to a request to help broker engagement between the donor community and CGIAR scientists in the further development of the SRF61.

More recently it identified to the evaluation team the following: • joint CSIRO Agriculture & ISPC outputs, namely a discussion paper on agri-food

innovation and impact 62 , a fact sheet on Agriculture Research, Multi-stakeholder Partnerships and the SDGs (undated)63, and a workshop and associated report on Resetting the conversation on agri-food system innovation (2016)64.

A major partnership study was the Strategic study of good practice in AR4D partnership (2015)65. It examined two distinct partnership domains: Multi-stakeholder partnerships (MSPs) in Agricultural Research for Development and Global MSP approaches; and developed a framework of Partnership and Innovation Modes to assist CGIAR in “embedding its work within the wider architecture of partnerships, platforms, and networks that will be required to tackle global scale challenges”.

The ISPC also sees the Science Fora as a means of “fostering partnerships that best complement the expertise of CGIAR and its partners on research initiatives and for development impacts”66.

3.2.2 Analysis In respect of the Science Fora, only three of the 52 survey respondents specifically mentioned them, two positively and one partly positively.

A small number of those interviewed had attended the fora, and they gave consistent messages: they were generally good events, enjoyed and valued by participants for being inspirational and providing networking opportunities, but did not have a direct spillover effect to the rest of the System. Some questioned the overall value-added, in a world of many conferences. Others questioned how well the outcomes of the fora were followed up. That being said, follow-up workshops were held after both SF13 and SF16 to feed the results into the Science Forum special issue publications, and for SF13 with an explicit objective to “inform further the 2nd round of CRP proposals in terms of enhancing the design of agricultural

60 ISPC Summary of the Work Plan and Budget from 2014 to 2016, July 2016. 61 Ibid. 62 Discussion Paper #1, 13 July 2016, Synopsis: Towards a framework for unlocking transformative agricultural innovation. 63 Fact Sheet (undated). 64 Workshop Report (2016). 65 ISPC, 2015. Strategic study of good practice in AR4D partnership. Rome, Italy. CGIAR Independent Science and Partnership Council (ISPC), xiii + 60pp + annex 50pp. 66 See http://ispc.cgiar.org/science-dialogue.

27

research to contribute to the System-Level Outcome on “Improving Nutrition and Health””67. Indeed, the nutrition forum was acknowledged as helping with the design of the A4NH CRP.

The evaluation team suggests that, rather than scheduling such events routinely in the work program, these be organized as needed to tackle critical issues for the CGIAR system.

In respect of Partnerships, many interviewees commented on the ISPC’s role. Several were of the view that this activity does not fit well with the role of the ISPC. They found it hard to understand what the agenda would be for ISPC. The evaluation team understands that the ‘P’ was somewhat of an add-on in the early reform stages to stress the importance of inclusiveness in the reforming system, but that the comparative advantage of making this ISPC’s responsibility was not clearly thought through. The ISPC’s intellectual agenda for partnerships has been slow to start, beyond the study on good practice in AR4D partnership referred to above. The evaluation team has seen only limited evidence up to now that this study has informed practice or decision-making at the System management level. It considers the report to be a good review of principles in innovation partnerships in agri-food systems, but too abstract to be helpful at the research management level.

However, the team notes that the study’s findings have already been incorporated in the CO’s Final Guidance for Full Proposals for the CRPs 2017-2022 CGIAR Research Program, Portfolio (CRP2) and that it was listed as a useful background document for the IEA Evaluation of partnerships in CGIAR.

Many interviewees from across the stakeholder categories have identified the need for partnering, both upstream with advanced science institutions and downstream with development partners. Many have also raised the need for CGIAR to be more intentional in its partnering with NARS and the agricultural innovation systems more broadly in partner countries. But some see this as being place-based, and best done by the centers and CRP teams closer to the research for development ‘frontline’. The evaluation team suggests this topic needs considerably more discussion at the governance levels of the System.

3.3 Impact assessment 3.3.1 Output The Standing Panel on Impact Assessment (SPIA) is a sub-group of the ISPC, and its chair is an ex officio member of the ISPC. Its Strengthening Impact Assessment in CGIAR (SIAC) program is clearly the most funded work stream in the ISPC. The SIAC project started in November 2012 with funding from the Bill and Melinda Gates Foundation (to late 2016 totaling $5.2M), channeled through the CGIAR Consortium; and additional funding from DFID (to late 2016 totaling $4.5M) started in 2014, channeled through the Fund Council (as a ‘Window1 special project’) and managed through FAO68. It also receives most funding of all the work streams from the ISPC’s own budget (approximately $500,000) compared to the other three work streams allocated between USD 100-200,000 annually.

Highlights of SPIA in recent years include:

• the SIAC program organized a major workshop on poverty reduction (University of Minnesota, Minneapolis, July 2014);

67 https://www.slideshare.net/ISPC-CGIAR/a4-nh-ispcworkshopagenda22sept2014 68 CGIAR-IEA (2016), Evaluation of the “Strengthening Impact Assessment in CGIAR” (SIAC), Project Phase 1, 2013-16. Rome, Italy: Independent Evaluation Arrangement (IEA) of CGIAR.

28

• the SIAC program held a workshop to which it invited donors and other major stakeholders to discuss how impact assessment can best be met at System level, based on SIAC results (Washington DC, March 2016)69;

• a book published in 2015 on Crop Improvement, Adoption and Impact of Improved Varieties in Food Crops in Sub-Saharan Africa. This SPIA-generated study greatly advanced knowledge of varietal adoption and diffusion in SSA;

• the SPIA and the Policy, Institutions and Markets CRP co-organized a well-attended conference on the Impact of International Agriculture Research – Rigorous Evidence for Policy (Nairobi, July 2017).

Other ‘events’ that showcase collaboration with/outreach to the CGIAR scientific community, specifically mentioned by the ISPC, include:

• presentations to / discussions with the CGIAR monitoring, evaluation and learning (MEL) communities of practices (COP) as well as evaluation COP (ECOP) meetings organized by IEA and SMB each year, starting 2013;

• discussions with the Science Leaders in Montpellier on a potential SIAC Phase 2; • a workshop on lessons from adoption of data collection (large-scale, and innovative

pilots) in Boston in July/August 2016 with the CGIAR Impact Assessment Focal Points as well as other relevant researchers within/outside CGIAR.

3.3.2 Analysis Phase I of the SIAC project was recently evaluated70. The evaluation was in general positive and noted the boost SIAC had provided to CGIAR on Impact Assessment. FAO administrative constraints were highlighted in the report, which have also been observed by the present evaluation team regarding ISPC Secretariat operations (see section 2.1.5). Similarly, the Fund Council’s oversight of SPIA/SIAC was noted in the SIAC evaluation not to have been very effective. A similar observation has been made for the entire ISPC in the present evaluation71. The oversight has been more administrative and managerial, rather than on content and direction.

In the present evaluation SPIA/SIAC was rated consistently well with interviewees across stakeholder groups. High quality of ex post impact assessment was praised, and its helpfulness to show donors and the development community the impact of CGIAR work was noted. There were, however, also a few critical voices among the interviewees, referring to the narrow scope of work (mainly on commodities, and technology option studies), and calling for assessments at the CGIAR system level, and also on policy and NRM work as well as socio-economic impacts of CGIAR research. These issues were also raised in the recent SPIA/PIM organized meeting in Nairobi. The evaluation team endorses these points and recognizes the drive to meet these challenges expressed by SPIA in the Nairobi meeting.

SPIA works with individual centers’ social science teams on particular topics. A critical issue here is the variability and sometimes low capacity for impact assessment work at the centers, which was also noted in the IEA Synthesis Review72 and by the SIAC evaluation. SPIA has engaged in capacity building to address the issue. An additional complication for CGIAR impact assessment is that, in addition to SPIA impact studies, the centers engage in a range of impact studies of their bilaterally funded programs. These are often quite small, carried out under time pressure and not always rigorous in their methodology. This was noted in a few

69 ISPC Summary of the Work Plan and Budget from 2014 to 2016, July 2016. 70 CGIAR-IEA (2016), Evaluation of the “Strengthening Impact Assessment in CGIAR” (SIAC), Project Phase 1, 2013-16. Rome, Italy: Independent Evaluation Arrangement (IEA) of CGIAR. 71 Recommendation 1 addresses this issue. 72 Birner, R. and Byerlee D. (2016): Synthesis and Lessons Learned from 15 CRP Evaluations. Rome, Italy: Independent Evaluation Arrangement (IEA) of CGIAR.

29

interviews. This has resulted in different parts of the system coming out with a range of impact figures on their research. Further, in the current funding climate, there is also a tendency for reporting ‘good news’. CGIAR needs reliable and rigorous impact assessment to guide its strategic choices and resource allocation decisions. However, the reality is that resource allocations (which are done on an annual basis and/or every 3 to 5 years for projects) can be informed by but cannot depend directly on the results of impact assessments that relate to research started 15-20 years ago. The evaluation team recognizes that SPIA and ISPC are well aware of these challenges, which were discussed in a recent meeting in Nairobi73. The evaluation team urges SPIA to find ways of providing guidelines, models and capacity for the center and CRP teams to harmonize their methodologies as much as possible.

On the use of the SPIA products, the evaluation team noted that, in its 2015 background report II for the ISPC Task Force, the ISPC reported that, for the two SPIA studies cited in the report, relevance was assessed high, but uptake by centers not very significant. The Task Force report also included statistics on the use of SPIA IMPACT briefs among 17 donors, which indicated that 70% had read at least one, but the influence of the briefs on their decision making was highly variable.

3.4 Independent program review 3.4.1 Output In line with the move facilitated by the 2009 reform, to “integrate the work of the research centers, enhance collaboration with partners, ensure effective governance and improve efficiency in providing and using resources”74, the CGIAR Research Programs (CRPs) were established as the major vehicle for strategic, longer term, integrated research.

Between 2010 and 2016, the ISPC was involved in the review of both the first and the second phase of CRP proposals and platforms:

• the ISPC reviewed the initial set of 15 CRPs established in the SRF of 2011. These CRPs started in 2011, and by 2013 all 15 were operational;

• in 2013, the ISPC conducted reviews/assessments of Gender Action Plan (2013), CGIAR Genebanks Options Paper (2015), AATP Virtual Information Plan, and Open Data/Open Access plans;

• in May-June 2014, 6-month extension proposals for all 16 CRPs were reviewed and a written perspective on the CRP portfolio was provided;

• in 2015, evaluation and analysis of the CGIAR research portfolio presented in the CRP pre-proposals was provided and guidance given on developing the full proposals (challenging the proposers to develop proposals which had coherence both within each CRP and with the System-wide portfolio);

• in 2016 each of the 12 CRP full-proposals and 3 platform proposals were reviewed, using a similar methodology to 2011, involving the ISPC Chair, council members and secretariat managing a process of engaging over 50 external experts commissioned by the ISPC. Commentaries on each CRP and platform and a commentary on the portfolio as a whole were presented to Science Leaders in Montpellier on 16 June 201775.

73 SPIA and PIM organized meeting on: Impacts of international agricultural research: Rigorous evidence for policy. Nairobi, 6-8 July, 2017. 74 Final Report of the Mid-Term Review Panel of CGIAR Reform, p.5. 75 ISPC Summary of the Work Plan and Budget from 2014 to 2016, July 2016.

30

The preparation of CRP proposals, as well as the different iterations of the ISPC reviews, absorbed a considerable amount of time and resources of ISPC members and Secretariat, as well as centers and their staff.

However, an issue came up with the CRP phase 2 review methodology that led to a range of concerns amongst donors sometime after the assessment process had started. This took the public form of a lengthy discussion on the new CRP Portfolio at the Fund Council’s last meeting in May 2016, where the Fund Council discussed an ISPC request for “input and agreement on a proposed rating system so that evaluations provide data and information needed by donors to make decisions” 76. The ISPC chair stated that it was her intent “to provoke a discussion and agreement on what information and data decision-makers in donor agencies need, and in what form, in order to take decisions on the CRP proposals and stimulate funding” 77 . In the end, the meeting decided to establish a working group “to develop complementary criteria to be used to assess the CRP proposals, particularly with a view to guiding donors’ investment decisions” 78 . A set of “Donor-perspective review criteria for flagships” was attached to the Meeting Summary.

At the 1st System Council meeting in July 2016, the chair of the working group, known as the Fund Effectiveness Working Group (FEWG), made a presentation and stated that its primary purpose was to “consider how to reverse the trend of donors leaving the pooled elements of CGIAR funding modalities, and not to disrupt the existing processes or create additional work for the Centers themselves” 79 . She reported that an ad hoc review process had been established, with 3 reviewers for each CRP proposal, with the review being focused at flagship level, not program level.

Immediately before the 2nd System Council meeting in November 2016, a System Council Workshop on Prioritizing CRPs was held, at which the differences between the evaluation under the FEWG’s auspices, and the evaluation by ISPC were discussed. By the end of the SC meeting, with “the guidance of the ISPC and input from donor reviews of the CRPs”, the SC had agreed a “uniformly high-quality portfolio for investment via Windows 1 and 2 (‘W1/W2’) of the CGIAR Trust Fund”80.

It is clear to the evaluation, from both interviews and the survey, that this two-part process, which was not instigated by the ISPC, was not always well understood by those preparing CRP proposals.

Nevertheless, it was clear from discussions with interviewees that a major concern was that the process was not leading to a clear picture of which proposals were of high quality and should be funded, and which were of poor quality and should be abandoned.

3.4.2 Analysis In the past two years the ISPC’s work program has been dominated by CRP proposal reviews. Not surprisingly, the ISPC’s work in guiding, reviewing and analyzing CRP proposals was commentated on more extensively by interviewees and survey respondents than any other aspect of the ISPC’s program. Over 80% of survey respondents referred to CRP proposal reviews as the main CGIAR activity in which the ISPC has an impact. It was an area that also received a significant number of interview comments. Many of the comments were positive,

76 Meeting Summary, 15th Fund Council meeting, Rome, 5-6 May 2016, p.5. 77 Ibid., p.6. 78 Ibid., p.16. 79 Meeting Summary, 1st System Council meeting, Paris, 12 July 2016, p.12. 80 Meeting Summary, 2nd System Council meeting, Mexico City, 25-26 September 2016, p.26.

31

however, the spread of views was quite wide. These comments can be grouped into the following themes, focusing on the second phase of CRP proposals81.

1. Many respondents and interviewees felt that ISPC did a good job, recognizing that it was a challenging and hard task performed under considerable time pressure. The secretariat was commended for its effective coordination of the reviews.

2. Several, especially CRP and Flagship leaders, said they found the review comments helpful for improving proposals.

3. Others thought that the ISPC operated at too detailed a level. The ISPC was especially criticized for being too involved in many cases undertaking a coaching role which had potential to lead to a conflict of interest at the proposal assessment stage.

4. Some System Council members, during CRP Phase 2, had expected, but felt they did not receive, clear guidance on which CRPs were of high quality and should be funded and which ones were of poor quality and should be abandoned. They were disappointed that so many of the ones that seemed weak had been rewritten with ISPC’s help to make them notionally fundable. This was criticized on two points – lack of clear process that could be trusted across the System and a lingering concern that funding these revamped proposals would be reputationally damaging to the whole System.

5. Another set of donor concerns was that the ISPC CRP assessments were too focused on the science and not enough on anticipated development impact.

6. The introduction of a separate assessment process, managed by FEWG, added further complexity and confusion for those submitting proposals, especially when the assessments differed significantly.

7. There were criticisms, from those directly assessed but also from center leaders, of inappropriate assessors being chosen with no quality control mechanisms to weed these out.

8. Many asked whether CGIAR needs a standing committee to do work that only happens once every four years or so.

9. In some interviews, it was claimed by some that donors, especially smaller donors, rely on ISPC’s assessments for assurance of science quality, but interviews with various donors indicate that this not true for all and at least some of them rely on advice from other donors especially from the same geographic area.

In this regard, the evaluation team notes the following matters:

• CRP proposal review is a central task for CGIAR, and one where it is fundamental that the assessment process is clear to all, and well and transparently communicated and managed, so that review recommendations are seen as fair and able to be trusted across the System (especially in the proponent centers) and by other key stakeholders such as donors. This is particularly important at a time when donors are increasingly considering moving funding from W1/W2 to W3 or bilateral funding. If W1/W2 reduces much more, the whole CRP process comes into question (and thus the whole current CGIAR structure).

• It is important in the assessment process to get an adequate number of expert opinions on the quality of the proposals; relevance of the research being proposed; fit to the overall portfolio; and anticipated development and innovation impact. In most contemporary research grant assessment processes, proponents are often given a chance to comment in a formal way on the expert opinions and rebut particular points

81 The number of respondents across the themes was so spread and the total pool so small that indicating specific respondent numbers is not very meaningful

32

with this comment/rebuttal material feeding into the funding recommendation process. In the view of the evaluation team, the review process managed by the ISPC of the science in the CRP proposals – engaging over 50 external reviewers – was sound and followed a reasonable methodology. However, in some cases different reviewers were used for the first and second iteration of the proposals, on occasion resulting in conflicting review comments (even if the commentary came from ISPC, proponents were able to infer from the inconsistency that the comments came from different reviewers). Going forward, there would also need to be tighter guidance on the length of the proposals. In the last round, the proposals were several hundred pages in length. The guidance for the proposal preparation came from the Consortium Office, and the ISPC had indicated it would strongly favour a shorter length. This shows a need for the System Office and the ISPC to work closely together on this type of issue.

• Even though the criteria included impact, several of those interviewed indicated that the ISPC’s process had not provided enough evidence of this or of the value for money. These were particularly important aspects of the ‘donor-led review’.

• Lastly, the comments raise the question of the suitability and fitness for purpose of ISPC to carry out the review process. Does the ISPC have deep research program assessment expertise among its members and secretariat? How deeply involved should the ISPC be in the actual process? Should it, for example, be finding assessment experts itself? Is a standing committee needed for a task occurring only periodically?

The evaluation team notes the very extensive comments on this matter made by everyone interviewed, with almost all raising concerns, some at great length. In terms of analyzing what happened, it is hard to work out what went wrong, but it is very clear that something was seriously amiss. While the ISPC members expressed frustration and concern about the parallel ‘donor-led review’, a great many System Council and SIMEC members and leaders in the System associated with centers and the central organization were extremely critical of the ISPC’s process. It has been clear that the CRP assessment process has been divisive and that any future assessment rounds need to build in a good process for hearing and responding to feedback as it arises.

Nevertheless, despite all the controversy about the recent CRP round, the CRPs over time appear to be producing good outcomes. A recent report by Birner and Byerlee (2016) 82 concludes that CRPs overall have produced research outputs of international quality even if this has come often from heterogenous CRP teams where some groups are very strong and others less so. The ISPC has been an important contributor to this success.

3.5 Strategy and trends 3.5.1 Output One of the four work streams of ISPC up until 2017 was called Strategy and Trends. Major outputs from this work stream have been work on a qualitative prioritization framework to underpin the SRF and strategic studies addressing current issues, either internal or external, affecting CGIAR or addressing the impact of agricultural research (SPIA’s work). These are commissioned by the ISPC often with a council member and a secretariat staff member supervising the study. The topics are identified by the council or secretariat members, or requested by CGIAR community.

In its highlight83 for the period 2014-16, the ISPC lists the following reports:

82 Birner R. and Byerlee, D. 2016. Synthesis and lessons learnt from 15 CRP evaluations. Summary, pp-4-6. 83 ISPC Summary of the Work Plan and Budget from 2014 to 2016, July 2016.

33

• a strategic study of Biotechnology in the CGIAR was completed and a review workshop was held in Washington, DC in March 2014 and the report published. This study addresses an area of research where the system has been active for the past 20 years. It presents a research pipeline on biotechnology in CGIAR, and examines partnerships in this area and assesses potential constraints to bringing biotech innovations to large scale production;

• a strategic study on “Data, Metrics and Monitoring” was completed and published in 2014. This was a timely study to help in times of intense work on the SRF and associated indicators.

The ISPC specifically pointed out to the evaluation team the 2013 Farm Size and Urbanization study that was cited by multiple CRPs in their proposals for assessment by the ISPC, as was the Agricultural Growth Corridors (2016) work.

In 2017, the work stream on Strategy and Trends was re-defined as Foresight and Prioritization. The need for additional work in foresight and prioritization, as identified by the MTR, was referred to under the Strategy and Trends heading in the 2016 ISPC work plan.84 The 2017 work plan noted the inclusion of foresight and prioritization in the (then) latest version of the draft ISPC terms of reference, and incorporated the previous types of activity under the former Strategy and Trends heading into the new work stream85.

In respect of prioritization, “ISPC 2017 work stream on prioritization is built as the first of a two year program that involves a series of studies, workshops and consultations in order to develop a recommendation to the System Council in 2018” 86.

In respect of foresight analysis, since it was raised as an issue by the MTR, the ISPC has made a few attempts to extend its work in this area, with more systematic work beginning last year when an ISPC member with skills in this area joined the council. Foresight is clearly a topic of interest across the System and the need for it was commented on frequently by interviewees and survey respondents.

In April 2017, the ISPC held a Foresight Workshop on Threats and Opportunities to Agri-Food Systems in 2050. Eighteen papers were solicited and presented to a group of 32 international experts, with the ISPC intending to publish the papers in a book in 2018.

The ISPC’s website also states that, as a next step in the ISPC foresight work, “CGIAR members will be invited to further develop the key outcomes of the workshop and provide inputs on the foresight process to help guide future strategies and priority setting of CGIAR research”87.

The evaluation team understands that the foresight work has been largely put on hold pending clarity on the roles of system entities in this area.

3.5.2 Analysis The self-initiated detailed Strategic Studies commissioned by the ISPC are strong bodies of work. However, the evaluation team noted, for example, that the Strategic Study of Biotechnology in CGIAR, commissioned by the ISPC, was an extensive report of 55 pages

84 ISPC Work Plan and Budget 2016, October 2015. 85 ISPC Work Plan and Budget 2017, October 2016: see reference on p.6 to activities/outputs including “initiating at least one Strategy and Trends study” amongst other foresighting and prioritization activities. 86 ISPC Work Plan and Budget 2017, October 2016. 87 http://ispc.cgiar.org/news/ispc-foresight-workshop-threats-and-opportunities-agri-food-systems-2050.

34

which made some very concrete recommendations for action, with an implementation plan.88 The report as published included a four-page commentary from the ISPC, in which it indicated that it did not agree with all the recommendations, but did support a number, including the need for an urgent review of biotechnology activities across the System, and the temporary establishment of a GM advisory board and a biosafety network. The ISPC also commented that the (then) Consortium Office “should take the lead in coordinating the efforts for improving the coherence, targeting and management of biotechnology across the system”. As mentioned, the report was the topic of a workshop in 2014. There is then no evidence that the recommendations in the report were forwarded to any decision-making body in CGIAR with power to progress them to implementation. This does raise questions about the utility, relevance and value of the study, and on the capacity of the System management and governance (Consortium and Fund Council) to follow through the recommendations at that time.

Another angle to achieving impact with the strategic reports relates to the extent they have reached a receptive audience, i.e. their dissemination and uptake. Several interviewees commented on the limited communication and dissemination of ISPC products internally (and externally), which leads to questions about the value of the investment in those products, and the likelihood of any significant impact from them actually being realized.

A report on the impact and utility of 11 such products was prepared by the ISPC Secretariat for the 2015 Task Force (discussed above)89. Seventy percent of the studies were initiated by the ISPC (council members or Secretariat) and 30% were requested by the Fund Council or Consortium Office.

Two thirds of the products were assessed to be relevant (cited by others); and about one third was assessed to have utility for CGIAR (used by FC, or in CRPs, or CO followed up). For the two SPIA studies relevance was high, but uptake by centers not very significant. The review also included statistics on SPIA IMPACT briefs among 17 donors: 70% had read at least one, but the impact across donors was highly variable.

These assessments are well aligned with the responses the evaluation team got in stakeholder interviews. The quality of the work is viewed positively, but the utility of it remains fairly low. The evaluation team concurs with this observation, but notes that there is a range in the utility; a few have gained a high public goods merit.

The ISPC’s 2017 Work Plan indicates that the ISPC is intending to do a survey on the current use of ISPC products by ISPC audiences and clients, to identify unmet needs and future opportunities90. This suggests that the ISPC is searching for meaning in the absence of an already articulated need.

Feedback received from many interviewees and survey respondents indicated that foresight is an important area for CGIAR closely linked with research prioritization, but that the ISPC is quite weak in this space.

The ISPC itself has recognized this, by making it a work stream in its own right in its 2017 work plan, under the leadership of ISPC member Prabhu Pingali and supported by a staff member in the Secretariat. The earlier lack of activity was clearly, in part, a resource issue,

88 ISPC. 2014. Strategic Study of Biotechnology Research in CGIAR. Rome, Italy, CGIAR Independent Science and Partnership Council (ISPC). xvi + 68 pp. 89 ISPC Secretariat, June 2015. Background Material II for the ISPC Task Force. Impressions of Success of Previous ISPC Studies and Other Outputs. 90 ISPC Work Plan and Budget 2017, October 2016, items 1.4 and 1.5.

35

thus the request for an additional $700,000 per year for foresighting in the draft ISPC Task Force report mentioned in section 2.3.2 above.

The ISPC foresight activities have also been criticized because they seem to not adequately take on board the expertise in the centers, especially IFPRI, and for lack of partnering with already on-going foresight work internationally91. The process initiated by Prabhu Pingali aims to include both a review of external foresighting work (on-going), then bringing it to CGIAR and working with the scientists in the centers, and linking the process with external foresight studies like the John Ingram-led work in Oxford.

The evaluation team holds the view that, while the ISPC commissioned strategic studies are largely good quality work, the topics chosen are not always ones that rank highly with current System Council members. The evaluation does note that the ISPC prior to the MTR prepared a set of possible strategic studies and sought feedback from various CGIAR stakeholders including donors. Nevertheless, the strategic studies are not being clearly owned by the System governance or the research community of CGIAR. This has reduced their uptake and influence to guide the System. Communication and dissemination of the studies has also been lacking both internally and especially externally. Ongoing efforts by the ISPC to improve its communication are acknowledged.

Going forward, given especially the resource-constrained environment, the evaluation team recommends that such studies are conducted when a clear demand has been expressed by the System Council, or when the ISPC is given a clear go-ahead by the System Council, i.e. they would not be generated entirely independently within the ISPC and wholesale approved with the work plan and budget, without the System Council endorsing the rationale and substance of such studies explicitly.

The evaluation team supports increased emphasis on foresight and prioritization – again, especially when requested by the System Council and done in partnership with international efforts on foresight and future scenario planning.

91 However, ISPC notes that “CGIAR Centers/CRPs that engage in foresight activities such as IFPRI, and more recently CCAFS, as well as GFAR have been involved from the very beginning in discussions related to foresight during the ISPC open meetings (and beyond) – i.e., 4 consecutive meetings starting ISPC-14 (Peru)” (Comment from ISPC on this report.)

36

4. What type of science/research advice does CGIAR need?

4.1 A mismatch It has become evident to the evaluation team that there is considerable unease within the System’s governance about ISPC’s role, and a general, though not always clearly articulated, feeling that there is a mismatch between what the System most needs and what the ISPC does.

In ISPC there is a well-qualified group producing a number of high quality outputs which are intended to have an impact on how agricultural research in the developing world moves forward and which have led to the production of some important global public goods. However, it is clear to the evaluation team that the System Council is not getting what it feels is appropriate advice on research priorities and assurance on likely science impact on development (hence the parallel FEWG review during CRP phase 2), or appropriate foresight-based advice on future scenarios and how to harness science breakthroughs to accelerate progress on realizing the System goals. Despite its significant outputs from all its work streams, the ISPC’s main influence on System decisions in recent years has been through its CRP assessments. Its other work does not seem to be having a significant impact on System decisions at governance level. This limit to its influence was also commented on in earlier years: the 2014 Mid-Term Review noted that ISPC’s authority was limited in terms of Fund Council acting on its recommendations. There clearly is a mismatch between the advice needed and advice provided. However, from discussions with the System Council’s SIMEC and from interviews with System Council’s current and former members, secretariat staff, and the ISPC Chair, it is also apparent that the System Council does not quite know what it wants from the ISPC despite a lot of recent discussion on this matter.

The mismatch could be, if anything, due to different parts of the System evolving on different tracks. As discussed in Section 2.2, the leadership role played by the TAC and its successor, the Science Council, in the first two decades of the System was not the same during the second two decades. The first decade featured a building stage; agriculture was viewed as important to development work; donors were keen to build a science network for agriculture, and funding to agriculture was not so constrained. The TAC was looked to as an authority for science priorities. It also gave advice on investments. Towards the end of the second decade, the system started to expand. But then the World Bank changed its funding modality from a “balancing donor” to a “matching donor” by using a formula for its allocations rather than following the TAC’s priorities. This, compounded by a trend in donor funding from unrestricted to restricted, marked a shift in power decoupling funding from the TAC’s priority setting92. The third decade started with continuing financial crises in the system, and a shift to open up the CGIAR towards external partners. During that period the TAC lost quite a bit of its influence and the new Science Council was not able to mobilize external science as expected93.

At the System level the reforms of the last decade have moved CGIAR more towards centralized decision making on research programs, and the profile and roles of the ISPC remain somewhat disconnected from the new governance set-up. In some sense, the ISPC is ‘hanging’ between the Old (more consultative) and New (more centralized) CGIAR and between providing detailed science review service (CRPs) while trying to elevate its strategic role (foresighting, strategic studies). As stated in CGIAR at 40:

It could be argued that a strong technical advisory body was needed more during the formative years of CGIAR than in later years when the need for technical advice may

92 The CGIAR at 31: Meta-evaluation of the CGIAR, 2004, p.130 93 The CGIAR at 40: Institutional Evolution of the World's Premier Agricultural Research Network, Selcuk Ozgediz, http://hdl.handle.net/10947/2761, August 2012.

37

have ebbed as the system matured and Centers began to work better collectively toward addressing their common strategic concerns. Nevertheless, it is clear from CGIAR experience that technical advice is heeded more when it is coupled with a commensurate incentive mechanism94.

A pivotal change in the reform was to shift the responsibility for leading the strategy development (SRF) to the Consortium (to the ‘doers’), having ISPC as a participant and a contributor (albeit a very significant contributor), rather than a leader in this process.

4.2 There is a lot of science advice available There is an abundance of science advice available across the System at present, particularly at the center and CRP level, a fact noted by many interviewees. The center boards have many eminent scientists on them and these boards determine center strategy which is primarily concerned with research activities in each center’s particular field and how that research is translated into appropriate outputs. This discussion is often led by program committees of the center boards. Similarly, each CRP is required to have an Independent Steering Committee which provides advice on the CRP’s research program, with membership that covers the specialized expertise of that CRP. Formal CRP evaluations have also provided guidance on the quality and impact of each CRP’s research.

Not surprisingly, most center and CRP leaders told us that this more immediate advice from center Boards and CRP Independent Steering Committees was what they turned to first, as it is more relevant to their needs and they have easy communication channels with it.

The report of the ISPC Task Force (see 2.3 above) concluded that attention to science quality and strategic thinking at the System level has suffered from the disconnectedness between the different System entities created as part of the reform process, coupled with the number of boards, panels, committees etc. which are involved in assuring science quality and developing strategies at different levels. The Task Force lamented the lack of a clear mechanism for engagement between these bodies or co-ordination of their work and the consequent situation whereby ISPC is perhaps not currently recognized as the preeminent source of advice to the Fund Council (now System Council). It is understandable and appropriate that science/research advice is provided at different levels of such an extensive research system. We suggest the key to success is to ensure the science advice provided is appropriate in content, timeliness and communication channel to the need at whatever point in the System it is commissioned and is not just science advice provided for the sake of that part of the System being seen to have science advice available.

Some donors indicated they considered ISPC’s advice when making investment decisions, but donors representing the majority of the funding to CGIAR pointed out that they also have their own sources of high quality advice both on research and on its likely development impact (a few have it in-house, but many donor agencies seek advice from their national research institutions and development professionals). Thus, many donors feel more comfortable drawing on this advice directly (rather than seeking advice from a System advisory body such as the ISPC) and are moving their investments increasingly to Window 3 or bilateral investments.

4.3 Is a standing science advisory body needed at all? The evaluation explored the issue of whether a body like the ISPC was needed at all. Several of those interviewed thought it was not. Some of those stating this consider that the System has matured and that there is sufficient science expertise in the System, and within the research networks to which donors have access, to provide specialist advice on an as-needed,

94 Ibid. p.118.

38

ad hoc basis. Some indicate that ISPC’s activities today do not provide good value for money and that the money supporting the ISPC would be better devoted to research.

Others have suggested that the exercise of thinking about the consequences of abolishing the ISPC is worthwhile to tease out what really is needed. And others have recommended scaling back the functions of the ISPC and/or distributing some of the functions to other bodies. Yet others noted that science advice is not needed as much as development impact and innovation advice.

The evaluation team believes that this ‘counterfactual question of no ISPC’ is an important matter to discuss, both because it was raised several times by interviewees, and because one of the best ways to sharpen understanding of what is needed is to think of what happens when it is taken away. Some issues to ponder in this regard include the following:

• Where does the quality assurance for science in the CGIAR come from? What are the success factors?

• Is the System’s science advice afraid to say ‘no’ to poor quality? • Which of the current ISPC functions are pivotal, which ones are nice to have, and which

ones can be done without? • Is there a need for System-level science advice? • Is ISPC needed for assessments, or could those be done by other means?

4.4 Should the science advice function be re-calibrated or changed in its framing? As discussed in in 2.3 above, the 2014 Mid-Term Review (the most recent review of the System as a whole) emphasized an elevated, strategic role for the ISPC, and its closer connection with the reformed governance of the System. More specifically, on optimizing knowledge impact, the MTR recommended that the responsibilities of the ISPC “should be elevated to empower it to be proactive in terms of providing strategic guidance, foresight analyses, and assessing and reporting on quality of research results across the system.”95

It also recommended that the ISPC chair be an ex officio member of the Board and that the ISPC remit be extended. It stated, among other things, that the review and reporting functions should be “at least as rigorous as was previously provided by the Science Council”; that it was “critically important to ensure that high-quality research review and advice is consistently provided by qualified researchers”; and that there should be “independent research panels comprising world-class research leaders to advise on particular issues, as required, under the overall guidance of the ISPC Chair”96.

Support for these recommendations has been expressed by some of those interviewed.

As noted above, the ISPC chair led a Task Force which worked on the means of implementing these recommendations. In its 2015 draft report, in addition to proposing a set of principles for the ISPC, the Task Force proposed an increased budget to cover the extra duties. The Task Force recommendations were not accepted.

Another structural suggestion, raised a few times in interviews by stakeholders from different parts of the System, is that the main science advisory body should report to the SMB, not the System Council. Currently, the ISPC chair is an Active Observer at the SMB meetings (though she was not at the Consortium Board). According to several of those interviewed, the

95 Final report from the Mid-Term Review Panel of the CGIAR Reform, October 2014, p.39. 96 Ibid.

39

relationship between the ISPC and SMB is contentious. ISPC makes it clear that it is accountable to the System Council, not the SMB, and goes to great lengths to be independent.

As noted in the previous section, at the level of the System Council, in addition to science advice, there is also a need for sound advice on development impact and innovation.

The present evaluation team agrees with the essence of the MTR recommendations on elevating the role of the ISPC and finding ways to ensure that research, science, innovation and development impact issues become key themes in System Council meetings. The team further suggests that while the ISPC should remain answerable to the System Council (particularly providing strong strategic guidance on research and science), the working relationship with SMB be closer (providing advice on operationalizing research).

4.5 Foresight analysis As in the MTR, a common theme with many of the stakeholders interviewed and surveyed has been the importance of improving the quality of foresighting. Comments such as “the ISPC should do much more foresight guiding CGIAR where to go in the next decades” (from a survey respondent) were not uncommon.

As noted in earlier chapters, the ISPC has acknowledged strategy and foresighting as one of its responsibilities, but its output has not been adequate for current System needs. Several of those interviewed noted the need for ISPC to work with the centers already engaged with foresight work, most notably IFPRI; and others referred to the many foresight programs ongoing internationally. They suggested that the ISPC could partner with these to leverage good advice for CGIAR.

Again, as noted several times above, recently the ISPC has intensified its foresight effort. However, the respective roles of different parts of the System Organization and ISPC in contributing to this theme are not quite clear, which has slowed down the momentum somewhat.

In times such as we are experiencing now – fast change, many uncertainties and daunting challenges in the external environment – the role of science council-like units in large organizations often shifts to strategic assessments, foresight, and scenario planning.

The evaluation team is of the view that a forward-looking dimension of science advice for CGIAR should be strengthened (see also 3.5.1). But the approach and effort level needs to be assessed carefully, including partnering with players already active in this area. Good foresight analysis can inform the discussion on priorities of the System.

4.6 What characterizes good advice from a science/research advisory body to a complex research organization like CGIAR?

The dichotomy between the large amount of good quality analyses and studies produced by ISPC and the limited uptake and influence of them, especially internally in strategic guidance and decision making, led the evaluation team to address the question of what characterizes good advice from a science/research advisory body in a complex research organization. Quite an extensive body of literature (largely grey) is available in this area, which is also a specialization of the evaluation team.

4.6.1 Top research centers make good use of eminent science advisory bodies Science advisory board structures are a common feature of most major research centers funded by highly competitive government programs in many countries. These boards are often largely drawn from the most eminent scientists around the world in the research discipline of the center. They play the role of a ‘critical friend’ and broker of useful global connections, and

40

often produce reports which funding bodies’ evaluation teams take into account for mid-term evaluations with funding decisions attached. The eminence of the members of the science advisory board adds to the luster of the center and helps it attract top-level staff and postgraduate students. Not all centers and especially their governing boards and management, make optimal use of their science advisory boards but it is noticeable that the top performing centers at both board and management levels make heavy use of them with good results97.

4.6.2 CSIRO has restructured its advice recently Large science agencies like CSIRO98 in Australia have over time established a range of advisory bodies. For many years these advised on science but after introducing its Flagship structure in 2002 to address major national challenges and opportunities, CSIRO largely moved away from science advisory committees and instead highlighted its Flagship Advisory Committees and worked hard to attract reasonably senior figures from business and public-sector bodies on to them. These bodies still covered some science matters but through the lens of the challenges each particular Flagship was addressing.

The Flagship Advisory Committees, established for each Flagship, focus on how to maximise the effectiveness of the Flagship portfolio to achieve its goals. The Committees comprise representatives from industry, government, non-government organisations and other stakeholders99.

In mid-2016 and following an organizational restructure and a difficult time in which it commissioned a report from Ernst & Young on its governance, process, capability, and organizational culture, CSIRO introduced a new advisory structure focused much more on ensuring business impact of its science than on the quality or focus of the science itself – CSIRO Business Advisory Committees:

The purpose of a CSIRO Business Advisory Committee (CBAC) is to provide independent, external advice to CSIRO on how to maximise the effectiveness of a Business, the appropriateness of its pathways to impact and on how to achieve its goals. It also assists CSIRO on broader strategic issues relevant to the sectors in which the Business operates100.

In CGIAR terms, this is the equivalent of an advisory committee advising more on development impact than on science quality.

And then around the beginning of 2017, the CSIRO Board created a Science Excellence Committee as a board subcommittee comprising only board members, in this case board members who are highly knowledgeable about research systems. This Committee takes ‘deep dives’ into selected CSIRO science activities. The Board is particularly pleased with how this Committee is operating; its report is currently the most popular item on the Board agenda according to the CSIRO Chair101.

In summary, CSIRO has evolved its advice about science and science-impact matters to meet what it believes are the best advisory structures for its present needs. There is no suggestion

97 http://www.publector.org/Third_Evaluation_of_VINN_Excellence_Centres/Title_Section. See Section 2 and then comments in evaluations of individual centers. 98 It is not suggested that CSIRO be used a close comparator for CGIAR. Rather, its experiences with multiple shifts in advisory structures are presented to illustrate that advisory structures need to be changed to respond to changing needs in an organization. 99 https://www.csiro.au/en/About/Our-impact/Reporting-our-impact/Annual-reports/11-12-annual-report/Part3/Management-and-accountability . 100 https://www.csiro.au/en/About/Strategy-structure/Business-Advisory-Committee-Charter. 101 Personal communication: David Thodey to Mary O’Kane, September 2017.

41

that the former advisory committees did not provide good advice or work well but in the view of the CSIRO board and management of the time102103, they were not fit for current purpose.

4.6.3 Science advice needs to be trustworthy and useful to the body commissioning it Another illuminating take on science advice comes through the practice and literature on the processes and mechanisms for providing science advice to governments or international bodies addressing contentious issues, so-called ‘wicked problems’. This advice is often commissioned through and moderated by a government Chief Scientist (also often termed Chief Science Advisor). Sir Peter Gluckman, the New Zealand Prime Minister’s Chief Science Adviser, says that there are 10 principles for guiding “all those providing advice to senior levels of government”104:

• Maintain the trust of many • Protect the independence of advice • Report to the top • Distinguish science for policy from policy for science • Expect to inform policy, not make it • Give science privilege as input to policy • Recognize the limits of science • Act as a broker not an advocate • Engage the scientific community • Engage the policy community

Mary O’Kane (team leader of this evaluation, who is also Chief Scientist & Engineer for the state of New South Wales in Australia), added an 11th principle in a recent talk105:

• Governments need to be able to absorb and act on advice.

In practice, this can be as straightforward as finding the time to allow for government ministers to attend briefings on the issue at hand and making sure the Cabinet has the right papers and time for its discussion of the issue.

In a 2015 publication the OECD presented pithy guidance on what should characterize good science advice for developing effective policy. It notes that there are five key phases in the science advisory process:

1. Framing the question 2. Selecting the advisors 3. Producing the advice 4. Communicating and using the advice 5. Assessing impact106.

The UK Government Chief Scientific Adviser’s Guidelines on the Use of Scientific and Engineering Advice in Policy Making makes similar points107.

102 Ibid. 103 Personal communication: Geoff Garrett to Mary O’Kane, September 2017. 104 “Policy: The Art of Science Advice to Government”, Nature, vol 507, issue 7491, 12 March 2014. 105 “What government needs from university researchers”, talk at Macquarie University, 7 July 2017, unpublished. 106 OECD (2015), “Scientific Advice for Policy Making: The Role and Responsibility of Expert Bodies and Individual Scientists”, OECD Science, Technology and Industry Policy Papers,No. 21, OECD Publishing, Paris. http://dx.doi.org/10.1787/5js33l1jcpwb-en. 107 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/293037/10-669-gcsa-guidelines-scientific-engineering-advice-policy-making.pdf.

42

The focus on the issues of communicating and using the advice and assessing impact is relatively recent but gaining increasing attention. The importance of the body receiving the advice having the right to reject it is also increasingly highlighted. For example, in describing the purpose of its Business Advisory Committees (CBAC), CSIRO notes:

CSIRO is not bound by advice received from a CBAC but will consider it appropriately in the context of the overall strategy and direction of the Organisation. It will advise the CBAC on the position or decisions taken by CSIRO in respect to the advice received from the CBAC108.

4.6.4 Lessons from other organizations on good science advice arrangements The main lessons from this analysis of what characterizes good quality science advice to a complex research system can be summarized as follows:

• the boards and management of the best impact-focused research centers make heavy use of their science advisory committees and attract leading world talent onto them;

• science and business impact advisory structures need to be changed as necessary to meet a research organization’s present needs;

• a relatively simple success measure of an advisory body can be very useful such as the CSIRO Board Chair’s noting that all board members want to spend more time on the matters raised by the Board’s Science Excellence Committee;

• the body commissioning science advice needs to be able to absorb and use the advice and then assess its impact;

• the committee offering the advice is aware that it is informing decisions/policy not making them, and that the commissioning organization is free to reject the advice offered.

4.7 What science and science-impact advice does CGIAR need? Although the proposed terms of reference for the ISPC touch on much of the advice that the System needs, the evaluation team suggests that it is important to review what categories of advice CGIAR needs at the top levels of the System. This list is derived from CGIAR documents, interviews, the survey and through our own observations of other major research systems. Not all the advice needs to be provided by a single entity. Indeed, it is probably neither wise nor cost effective to provide it through a single science advisory council, though such a body might have some role in commenting on how most of the tasks are carried out.

a. Advice on the big issues There is widespread but not universal agreement that there should be a body that provides very high level, high quality strategic advice on science and research foresighting in the System. But the advice needs to be provided in a way that the System Council (or other System bodies) can take it on board and, when required, implement it. In more detail, the System Council and CGIAR as a whole need advice on the core science and development challenges inherent in delivering on System goals, how these should be tackled in the System’s research and innovation agendas, and how science breakthroughs and major emerging science developments might be harnessed to assist with this. In this context, particular advice would be needed on issues such as the following:

i. What are the core difficult, intransigent and wicked problems within those challenges? ii. What are the pieces of work that have to be done to get there and over what time

frame and what budget?

108 https://www.csiro.au/en/About/Strategy-structure/Business-Advisory-Committee-Charter

43

iii. What should be solved internally within the System, and what should be done by leveraging the work of partners outside?

iv. How should all this fit in with the NARS and innovation Systems of partner countries?

b. Advice on process for operationalizing these Advice is needed on how to go about operationalizing needed changes to the CGIAR’s research agenda. Particular advice would be needed on:

i. structures for delivering high impact research to solve the big challenges; ii. processes for calling for and assessing proposals; iii. processes for finding right assessors and research partners; iv. the high-level outcomes of a call; v. processes for linking the research agenda with the innovation agendas of System

partners.

c. Advice on assessment processes for research quality and impact Advice is needed on how to go about evaluating the quality of the System’s research and its impact. Particular advice would be needed on:

i. appropriate assessment metrics; ii. what should be in an evaluation program, the evaluations themselves and what

recommendations from them should be actioned ; iii. additional uses to which evaluations could be put (e.g. leveraging additional funding

from other sources in beneficiary countries).

d. Advice on maximizing and connecting the research advice available across the System

Advice is needed on how to maximize the value of the extensive research advice available across the System, for example, where appropriate leveraging the expertise that sits in advisory bodies to the centers and the CRPs.

e. Advice on research infrastructure and platforms Advice is needed on research infrastructure and platforms. The evaluation team notes with concern that in the current work of the ISPC and in the proposed new terms of reference for it there is little reference to research infrastructure which is odd for a body such as CGIAR whose research is infrastructure-intense. Most large research organizations and national science systems now have policies relating to major facilities and policies and approaches regarding shared and remote equipment access. CGIAR would benefit from such policies too.

In this regard, however, the evaluation team commends CGIAR for its Platform for Big Data in Agriculture, a fundamental initiative for contemporary large-scale science.

f. Advice on research training and links to the NARS One of the great strengths of the System is the opportunities it offers to invest in, train, nurture and develop early career scientists across the System. The evaluation team again notes with concern that this matter was also missing from the proposed ISPC terms of reference (except incidentally through policies on gender). The team suggests that advice on how to maximize the opportunities for these researchers to develop effectively through being given opportunities across the System is likely to be some of the most high-impact advice that can be offered by the System’s main science advisory body.

Policies that help early career scientists develop their research, innovation and leadership skills are vital to the issue of how CGIAR interfaces with NARS in developing counties, a matter that was raised several times in interviews.

44

Even though capacity development is an important responsibility of centers and CRPs, the evaluation team believes that high level policies on this matter are best driven centrally to ensure opportunities are maximized across a global system such as CGIAR.

g. Advice to strengthen the CRPs Given the CRPs are the main research structure in the System, specific attention needs to be given to ensuring they are operating as effectively as possible. Birner and Byerlee (2016)109 refer to variations across the CRPs and to address weaknesses in some CRPs they call for harmonizing research quality control, active partnering with ARIs, mentoring of junior staff, incentives for performance, adding more social scientists, and publishing in peer reviewed journals. In short, achieving and maintaining an institutional culture of high quality science is influenced by many factors. While some of these factors are best addressed at center and CRP levels, others need to be addressed through System-wide policies and processes as noted above with regard to capacity development.

4.8 Advice must be able to be absorbed and guide decision-making As noted in Section 4.6.4, for advice covering the categories above to be useful, the System has to have mechanisms/structures (especially at the senior governance levels) to hear, absorb, react to and act on the advice in a timely manner. There need to be effective links and communication channels between the body(s) providing the advice and the bodies commissioning it. Conversely, the advice needs to be presented in formats that can be readily absorbed by the parts of the System which need it most and can act on it.

As noted above, the top-level science advisory body might provide high quality advice at some level on the types of issues listed above. Generally, this advisory body would not be the body(s) that implemented the advice however.

4.9 How will CGIAR know if the advisory mechanism is being successful? A test of whether the research/science advice structure is working would be that all parts of the System (System Council, SMB, centers, CRPs, funders, users) think the research/science advisory body(s) is an entity that adds significant and timely, tangible value to the System, even if they don’t agree with everything it says. One simple measure of success would be that System Council members would look forward to the agenda items from the science/research advisory body, even if they know there is going to be considerable debate.

In terms of measuring this, if the advisory structures are very successful, it will be obvious from discussion within the System especially and initially within the System Council and the SMB. If there is uncertainty as to the effectiveness of the advisory structures, then more formal evaluation techniques will be needed ranging from pulse surveys through scorecard reporting (already used in the System) to formal evaluations.

109 Birner R. and Byerlee, D. 2016. Synthesis and lessons learnt from 15 CRP evaluations. Summary, pp-4-6.

45

5. Findings and significant observations

This chapter brings together the main findings and significant observations of the evaluation drawing on the analysis presented in previous chapters.

The ISPC and Secretariat deliver significant output professionally • The ISPC and its Secretariat work very professionally, producing significant outputs

each year in line with the annual work plan and delivering within budget.

• The ISPC worked hard to deliver assessments on the two CRP rounds to date.

• The ISPC produces significant global public goods in the form of commissioned quality reports on challenging topics and through its Science Fora, each dedicated to a particular theme. Several interviewees have highlighted this contribution to global public goods, if not so much to providing high-level System advice.

• The work done by SPIA is valued highly by many in the System. The ISPC is effectively the CGIAR think tank (but also has funding body characteristics)

• The ISPC is effectively the CGIAR think tank especially in its work streams on foresight and prioritization, impact assessment, science dialogue, and agri-food innovation and partnerships. As noted above, it holds well-constructed fora and commissions and produces high quality think pieces and makes these available openly so they can be of value not just to CGIAR.

• In its program review work stream, ISPC operates more like a research grants body although the evaluation team notes that it doesn’t have the luxury of fully-developed grants polices and structures such as quality-controlled assessor databases that major national research grants bodies (such as the NSF) or philanthropic organizations (such as the BMGF) have.

The ISPC has been fortunate in its chairs • The two chairs of the ISPC (Ken Cassman 2010-2014; Maggie Gill 2014-present) have

been able and dedicated leaders of the Council.

• Professor Gill is a committed and vibrant contributor to the System generally, contributing significantly at System Council and System Management Board meetings in her role as an Active Observer. She took a particularly active personal involvement in the recent CRP assessment, travelling extensively to deliver ISPC feedback on proposals personally. Her manifold contributions are much appreciated across the System.

The ISPC is known across the System for its research quality and assessment roles • In the survey, there was a high level of consistency across all categories of

respondents that the current role of the ISPC was some form of advising/ ensuring/ strengthening/ maintaining/ oversighting/ guiding/ critiquing/ evaluating the quality of science/research within the System.

• Almost half the survey respondents thought the ISPC’s main impact on the System was through its assistance/assessment during the pre-proposal and assessment stages of the Phase II CRPs.

What ISPC’s exact role should be is not clear to itself or many others in the System • The ISPC is frustrated that its terms of reference following the 2014 Mid-Term Review

have not been finalized. According to the ISPC Chair, in this vacuum the ISPC has tried to work out what the System needs and then deliver that. This approach is

46

expressed in its annual work plan and budget, the last being approved seemingly without explicit discussion by the System Council at its November 2016 meeting110.

• The ISPC range of activities and its personnel are much less well known in the System than its chair. Individual ISPC members do not have much visibility within and of the System Council and SMB. ISPC members would like more opportunity to interact with the System Council and SMB, which would help clarify what is needed and make it easier to deliver appropriate outcomes.

• Interviews and the survey (see Annex 4) indicate that across the System there is a wide range of views of what the ISPC’s role should be.

The System Council is still uncertain about what it needs from the ISPC • The future role of ISPC has been under consideration since the MTR in 2014 and

decisions to finalize the draft ISPC terms of reference, currently before the System Council, have been postponed several times despite being on the System Council agenda at all its meetings to date.

• In summarizing the discussion at the most recent System Council meeting, its chair “highlighted that the context and the needs have now changed for the independent advisory functions, in large part due to the growing level of confidence among the Council members that the science is now being applied within the System in the centers to a higher level. With a stronger System, the observation is that there is now perhaps less need for external handholding and a lot can now be done in-house”111.

Many in the System suggest that ISPC’s role needs to change significantly • As noted in the record of the 4th meeting of the System Council (SC4), “There was still

a gap in terms of future scanning of what big issues are likely to come up that should be known about as investment is being decided.” This was also raised in many interviews and the survey. While it was acknowledged that the ISPC had commissioned foresight advice, it was often felt not to be in a form to help the System with current issues.

• SC4 noted (again at Item 7), that “looking to the future, the observation was made that ISPC should focus more of its analysis on upstream work to help funders to prioritize and make funding allocation decisions using a robust evidence base.”

• ‘Partnership’ is in the ISPC name, but there is little evidence of ISPC’s activity fostering new partnerships, and little expectation that it can undertake this role effectively, a view held even among some ISPC members. This matter was raised several times in interviews (the phrase ‘Drop the P’ was often used) and was a matter also discussed at SC4, Item 7.

• ISPC’s work with CRP assessments receives mixed comment, much negative (especially at senior levels) but with a significant group appreciating the help it provided. (See survey results at Annex 4.)

• A recurring comment, particularly, but not exclusively, related to ISPC’s role in CRP assessments, was that the ISPC spends too much time ‘down in the weeds’, e.g. with CRP proposal assessments the ISPC’s role was characterized by some of those interviewed as coaching, followed by judging, with inherent conflict of interest issues.

• The role ISPC has in assuring donors of CGIAR research quality was raised several times in interviews. This is important to some donors but not to others who indicate

110 Meeting Summary, 3rd System Council meeting, virtual, 23 November 2016, p.13. 111 Item 7 ‘Getting the best possible independent advice’ SC4 in May 2017 (see https://www.cgiar.org/wp-content/uploads/2017/08/SC4-11_MeetingSummary_APPROVED.pdf).

47

that they assess science quality themselves, using a variety of internal and external techniques, or get reassurance from other sources.

• Several suggested that the ISPC should have a role in evaluations, at a minimum commenting on the science related recommendations.

Interpretation of independence possibly inhibits impact • The ISPC is protective of its independence, particularly from the governance structures

of the System, which is reinforced by the fact it is set up in FAO at arm’s length from other System bodies. But, possibly because of the emphasis on independence, many of its activities are not then ‘owned’ anywhere else in the System. The problem could be the lack of a formal reporting structure. Although its work plan and budget are approved by the System Council annually, System Council members were often unclear about the details of ISPC work apart from CRP assessments. Or the System Council/Fund Council simply could be disinterested in what is going on – formal advice from the ISPC does not appear on the Fund Council/System Council agenda nor is there a standing item on science/research directions.

• The ISPC’s Theory of Change goes some way to addressing how it expects to make an impact on various parts of the System, but while this was mentioned in some interviews and survey responses, most were unaware of it or did not find it particularly helpful.

• Other major System entities (SMB, centers, etc.) rarely seem to refer matters to the ISPC for advice.

• The way independence has been interpreted has led to a disconnect between what the ISPC does and what the governing bodies discern as their needs. Much of the ISPC’s considerable body of work is rarely looked at or valued by the governing bodies and therefore does not seem to feed into strategy or forward thinking as much as it might.

Whether the ISPC is value for money is questioned by many • A significant number of those interviewed indicated that the cost of the ISPC was too

high in a time of fiscal constraint for the value it provided.

• At SC4 “A statement on behalf of the Centers with respect to ISPC and IEA pointed to the collective view of CGIAR’s 15 Centers that the budget, scope and performance of ISPC and IEA should be subjected to the rigor applied to all System entities, with the request that the System Council take steps to ensure that this is so given the reporting lines of the ISPC and IEA”112.

• It is important to note that while a small number of those interviewed indicated that most of what the ISPC does was not very useful, most questioning the funding going to ISPC felt that (to the extent they were familiar with it), ISPC’s work and outputs were good. But recent changes in the CGIAR Fund, especially the decline of funding to Windows 1 and 2 meant that the System could not afford many of the services offered by ISPC anymore.

There are some areas of science advice that should be discussed more at a System level

• As noted in Section 4.7.e and f, the evaluation team is surprised there is not more ISPC focus on research infrastructure and platforms and research training and links to the NARS. We suggest this be rectified.

112 https://www.cgiar.org/wp-content/uploads/2017/08/SC4-11_MeetingSummary_APPROVED.pdf

48

Advice structures in other complex research organizations offer guidance • As noted in Section 4.6.4 there are useful lessons to learn from the way other complex

research systems structure their advice arrangements. In summary, there is considerable unease in the System about the overall value and impact of the ISPC in its present form.

In terms of the evaluation’s formal terms of reference:

• it would seem that the relevance and scope of the ISPC’s leadership and advisory functions work well, although its activities are increasingly mismatched with evolving System needs and expectations in a context of significant and fast-emerging challenges and also new opportunities in agriculture research for development

• the value ISPC adds to the System overall is mixed though the value in terms of global public goods is quite considerable. On balance and in the absence of any other process until the FEWG review commenced, the CRP assessment activities were seen as adding considerable value to the research program of the System as a whole. There is considerable value in the work of SPIA

• the functional performance of the ISPC as a whole and in its areas of activity is good, especially given the limitations associated with the ongoing change in the System and the lack of formal ISPC terms of reference. Note that the rating here was good rather than very good because the utility and influence of several its products and services for the System Council and for the whole CGIAR scientific community were not clear to many interviewed and surveyed

• the operational performance of the ISPC as a whole and in its areas of activity is very good. The ISPC, its Secretariat, and especially its chair are to be commended for such a strong operational performance in the face of a considerable workload on CRP proposal reviews and the recent changes in CGIAR governance and funding arrangements.

With regard to the following two strategic issues pursued throughout the evaluation:

• Have ISPC contributions led to and are further contributions likely to lead to improvements in the overall delivery of CGIAR’s vision, mission and goals (reducing poverty, improving food and nutrition security, and improving natural resources and ecosystems)? It seems likely that ISPC contributions have led to improvements in the overall delivery of CGIAR’s vision, mission and goals, but exactly how is not clear. The ISPC’s Theory of Change does not engage directly with delivery of System goals so this impact is difficult to map. It would seem that the issue of how the ISPC might be best seen to lead to improvements in the overall delivery of CGIAR’s vision, mission and goals is troubling for those on the System Council attempting to finalize the advisory structures.

• The counterfactual proposition – what would be the effect on the System if the ISPC didn’t exist? If ISPC did not exist, another part of the System would have to pick up tasks such as foresight, CRP proposal reviews and SPIA. However, it is not clear that any other activities of the current ISPC are so essential to the System’s ongoing operations that they would need to be re-housed. A caveat – it is important to note that the various problems identified have their roots in a variety of historical causes and differing perceptions of what is needed and what can be afforded. We note that the ISPC has worked vigorously to deliver what it perceives as needed at a time when its mandate is not clear.

49

6. Conclusions and Recommendations

The evaluation concludes that the ISPC is functioning well according to its remit as expressed through its work plan and budget and in the absence of formal terms of reference. However, the evaluation notes that current arrangements for high-level science advice in CGIAR are unsatisfactory. They need re-envisioning and a new arrangement needs to be put into place. The 2014 Mid-Term Review was on the right path, but it did not sufficiently deal with engagement and connectivity between the science advisory body and the rest of the System.

It is now almost three years since the MTR Report was finalized and new terms of reference for the ISPC are still unresolved, with the obvious implication that the current science advice structure in the system and what is proposed in the ISPC terms of reference under consideration are not what is wanted going forward.

Achieving resolution of this matter is an important part of finalizing the most recent phase of the reform process in CGIAR and is rightly a high priority with the System Council and the SMB.

In the recommendations given below, the evaluation team suggests that, as a first exercise, the System Council through its SIMEC and in consultation with the SMB should agree what types of science advice are needed, preferably in priority order (using as a starting point the checklist in section 4.7); then working out how it would measure success; and only then moving to mechanisms and structure for delivery of this advice.

While full agreement on the ranking on the categories of science and science-related advice that should be delivered centrally in the System might be hard to achieve, two needs were identified by a significant majority of key stakeholders interviewed for this evaluation. These provide a good starting point for any new advisory body dealing with science and science-related matters:

1) what was often termed foresighting and prioritization, and, when unpacked, seems to refer to the ‘big questions’ in Section 4.7a 2) the need for science advice to be offered along with or in the context of advice on the impact of that science in terms of development, innovation and translation.

As to structure, we suggest a vital characteristic of any new science advisory body is that it be tightly integrated with, fully cognizant of and responsive to the thinking to the System Council. In practice, this would mean it would enjoy the same level of closeness to the System Council that SIMEC has while still operating independently in coming to its own conclusions and any advice it offers.

We further suggest that the System use a new title for this body, as it is likely to be quite different in scope of activities and operation to the current ISPC.

With regard to membership, CGIAR is an outstanding entity and continues to be in a position to invite the very best, most eminent and appropriate people to serve it with good advice. There are many leading researchers and leaders of research, innovation and development who would see it as an honour to serve CGIAR in such a capacity and the System can also draw on the eminent members of the current ISPC.

The current ISPC has a well-led and skilled secretariat. The evaluation team suggests that this secretariat should continue to serve any new body. Its capacity, experience and skills will be particularly needed as it is likely that a new body would focus at a high level and therefore would be particularly reliant on the expertise of its secretariat to source or prepare high quality papers and background material.

50

On independence, what is important is that advice on science, research, innovation and development impact is offered by independent thinkers working collectively. The secretariat would have to have the dual role of, on the one hand, supporting the functions of this body and transmitting its views, even if they are unpopular, and, on the other, working very closely with the secretariat functions of the System Council and SMB to ensure tight integration of agendas and effective dialogue. In effect, the best analogy is probably the system audit function which is a vital part of the System but also operates in terms of its work program independently from the rest of the System.

6.1 A note on the structure of the recommendations Recommendation 1 is the pivotal recommendation of this evaluation. The other recommendations make no sense until it is addressed.

With regard to Recommendations 2-5, we note that in terms of the best structures through which to source science and related advice, there is generally no single right answer. So, whatever structure is chosen for the central science body, the new body must meet the following baseline criteria:

• be seen to add significant value to the System in terms of reaching the System goals more effectively and sooner;

• provide advice the System absorbs and uses; • be perceived as good value for money.

Conscious that there is no absolutely right answer on structure, the evaluation team offers two lower-ranked alternatives to Recommendations 2-4. The first is an option of not having a central science advisory body but rather relying on ad hoc advice commissioned when needed. The second option is to retain the current ISPC but with a negotiated change to its focus, mode of operation and membership so that it delivers the type of advice the System most needs. The process of change would be led by the SIMEC working with the ISPC and its secretariat on behalf of the System Council.

The recommendation on transition arrangements (Recommendation 5) would be the same for all three options.

6.2 Recommendations The primary recommendations of the evaluation are presented below.

Recommendation 1 – establishing what kind of advice and advisory structures are needed That the System Council, including through its SIMEC, continues to move its focus from trying to finalize terms of reference for advisory bodies to re-addressing first the questions of exactly what types of advice the System needs, secondly how to measure the quality of this advice, and after these matters are settled, how that advice is best commissioned and delivered, be it via standing committees or specially-commissioned ad hoc arrangements, or a mixture of the two. Specific questions to consider in this exercise include:

• What are the most important science/research/innovation/development impact questions where advice is needed? Is it around the future scanning of what big issues are likely to come up that should be known about as investment is being decided? If so, how should this be investigated? By a standing committee of very eminent experts? Through scenario planning workshops involving scientists from across the System maybe with a small number of external experts called in for each workshop?

51

• Should any standing science advisory committee just advise on science or be a mixture of science, innovation and development experts? If the latter, presumably they then advise on innovation and development as well as science?

Recommendation 2 – a possible new advisory body That, if the System Council decides to proceed with a standing committee, it avoids minor adjustments to the current ISPC given the widespread unease with the ISPC, and considers more radical change by establishing a new, high-level and eminent science/research/ innovation/development body with a new name and a new mission, with the characteristics set out below.

This body would be a formal but independently constituted and operated sub-committee of the System Council and would have strong links to the System Management Board. It would receive formal references from both bodies, requesting advice and guidance both on major science/research/innovation/development issues and on processes. In turn, the new body would provide the advice needed back to these bodies within agreed timeframes. It would also have the ability to send advice it initiated itself to the two bodies for consideration. The governing bodies would maintain the discipline of formally responding to the advice. Operationally, the new body would have strong working links with sub-committees of both bodies (such as their SIMECs) and would be served by a high-quality secretariat operating to support the independent thinking of the new body but maintaining strong links with the System Office. Options for enhancing these links and achieving greater efficiency and economies of scale should be carefully considered by the System Council in consultation with the new body and System Office.

The chair needs to be:

• an ex officio, non-voting member of the System Council reinforcing the centrality of research in CGIAR. Whether the chair could also be a full member of the System Management Board, or at least an Active Observer, should be considered

• a globally renowned individual with a deep knowledge of science and development issues who is a proven, effective chair; a talented leader; and an outstanding communicator

• a person with substantial availability, at least about a quarter of their time • a person who can and will work closely with the chairs of the System Council and the

SMB and their SIMECs, as well as the centers and CRPs, to ensure the System research agenda is appropriately brought forward, debated and acted on in a way that allows CGIAR to tackle really big challenges effectively.

The membership of the new advisory would be relatively small with up to six members in addition to the chair, all of whom would be eminent as leaders in making complex research systems work well. Some might be world-famous researchers heading major research laboratories, and others might be senior figures providing effective advice on research priorities and change through guiding national and transnational research, innovation and development systems. For example, members might be winners of the major global prizes; others might be equally eminent as leaders in fields such as energy, sustainability and research systems. Their eminence would likely limit their availability to serve more than, say, 15 days a year. The members would be drawn from diverse backgrounds. Gender balance and appointment of people from developing countries would be important. Terms would be 3-5 years with a rolling appointment structure so the whole body does not turn over at once.

Given the limited time availability of members, it will be important for the body to be supported by strong secretariat that can work with others to assist the System Organization to follow through on decisions about science, research, innovation and development impact.

52

It is important to recognize what the new body will not do. It will be a body providing excellent and appropriate advice but it will not be operationalizing this advice although it might make suggestions on how to carry out the operation and it would comment on evaluations.

The System Charter would change to state that the Science, Innovation and Development Committee (or whatever name is chosen) is a “standing panel of experts appointed by the System Council to serve as an independent advisor to the System Council on science, research, innovation and development matters”. A comment has been raised about this recommendation that, in providing advice to the SMB as well as the System Council, the new body could be put in a position of conflict of interest, e.g. if it were asked for advice by the System Council about SMB decisions on research matters. In such cases, the needs of the System Council would always prevail, as the new body is a sub-committee of the System Council. Recommendation 3 – a simple measure of success That the System Council adopts a relatively simple metric for assessing success of the new body, such as that the System Council and System Management Board find their interactions with the new body deliver significant insights and help to the System Council and SMB in major areas of concern. The success or otherwise of the body would be assessed at least once annually. If it is not meeting expectations, it should be disbanded quickly and another mechanism such as commissioning ad hoc advice substituted. As well the new body would also need to:

• be seen to add significant value to the System in terms of reaching the System goals more effectively and sooner

• provide advice the System absorbs and uses • be perceived as good value for money.

Recommendation 4 – some existing structures will be needed to support the new body That, while the new advisory body would offer high-level commentary on issues such as mechanisms for research assessment, evaluation and metrics, specialist bodies such as SPIA and specially constituted assessment panels would still be needed to feed into the new advisory body and to carry out the detailed work involved in evaluations, assessments and measuring research quality. Recommendation 5 – transition arrangements That, given the multi-year hiatus on the formal arrangements for high-level science advice to the System, any new body should be planned to commence by end of 2018 at the latest and the current ISPC would finish up by the same time.

6.3 Comparison of current ISPC and possible new body The differences between the recommended new body and the current ISPC are illustrated in the table below. Table 7: Characteristics of current ISPC and possible new body

From To ISPC Science, Innovation and Development Committee of the System

Council Working level High level Chair and Members outstanding sector scientists

Chair and Members high profile science/development/innovation thinkers from private, public and NGO sectors

Independent agenda setting SC requests/independent Five established work streams Strategic/foresight/ agenda dynamic Secretariat distant from the rest of the system entities

Secretariat in closer interaction with system entities and management

Members involved 30-50 days a year

Members involved about 15 days a year max

53

6.4 Option 2 Recommendations – ad hoc advice Recommendation 2 – System Council commissions ad hoc advice That the System Council commissions ad hoc advice on science/research/innovation/development as needed through a high-quality secretariat which:

• also coordinates activities of specialist standing bodies such as SPIA and manages processes such as specially constituted assessment panels

• includes a research quality metrics and assessment function.

Recommendation 3 – Is there anything missing? After two years, the System Council holds special session at its November meeting assessing if this advice process is adequate for its needs.

6.5 Option 3 Recommendations – ISPC modifies its operations This version of the Recommendations 2 and 3 is included after the ISPC commented on a presentation from the evaluation team on findings and recommendations, suggesting since the evaluation acknowledges that ISPC has performed well, it is reasonable to think that the best course for settling science advice in the System at central level is for the System Council to determine what advice it wants and how it wants that advice delivered and then to negotiate with ISPC about how ISPC might change to meet the System Council requirements.

Recommendation 2 – System Council and ISPC work together on a new version of ISPC That the System Council after deciding the type of science/research/innovation/development advice it needs and in what format it wants it delivered, asks its SIMEC to work with the ISPC and secretariat to change the ISPC’s focus, mode of operation and membership so it delivers the type of advice the System Council most needs and will be able to absorb and respond to well.

Recommendation 3 – a simple measure of success would be the same Recommendation 3 as in Option 1 above.

54

7. Annexes

Annex 1: Members of the evaluation team Team Leader - Emeritus Professor Mary O’Kane AC

Mary is a consultant and company director. She is Executive Chairman of O’Kane Associates, advising governments and the private sector on innovation, research, education and development. She is also Chief Scientist & Engineer (3 days/week) for the Government of New South Wales. Mary was Vice-Chancellor and President of the University of Adelaide (one of Australia’s most distinguished performers in agricultural and biological sciences research) from 1996- 2001 and Deputy Vice-Chancellor (Research) from 1994-1996.

Before that she was Dean of the Faculty of Information Sciences and Engineering at the University of Canberra. Her research field was automatic speech recognition. Mary has served on several boards and committees in the public, private and community sectors, especially related to development, research, engineering, ICT, energy, and science. From 2009-15 she was Chair of the Development Gateway, ‘an international nonprofit delivering technology and information solutions for international development’ (http://www.developmentgateway.org/about/ ).

She is currently chair of the Boards of the Cooperative Research Centre (CRC) for Spatial Information, the Space Environment Management CRC, and of the Institute of Marine and Antarctic Studies at the University of Tasmania. She serves on the boards of Cross River Rail Delivery Authority, Business Events Sydney Ltd, Queensland University of Technology, the New Zealand Antarctic Research Institute, the Development Gateway, the Capital Markets CRC and the Innovative Manufacturing CRC. She was Chair of the Australian Centre for Renewable Energy, a Director of FH Faulding & Co Ltd and was a Member of the Australian Research Council, the Cooperative Research Centres Committee and the Boards of the CSIRO and NICTA. Mary is a Fellow of the Australian Academy of Technological Sciences and Engineering and an Honorary Fellow of Engineers Australia.

Team Member – Dr Eija Pehu

Dr Eija Pehu joined the Agriculture Global Practice of the World Bank in 2000 as the Science Advisor. She led the Department’s program on agricultural research, extension and innovation interacting with both external partners including CGIAR and academia as well as national and regional research organizations involved in the field operations supported by the World Bank.

Prior to joining the World Bank, Dr Pehu was a Professor of Agronomy and Head of the Department of Plant Production at the University of Helsinki and the founder and science director of two start-up companies in the Helsinki Science Park. She earned her Ph.D. in Horticulture from Virginia Polytechnic Institute and State University and her M.Sc. degree from the Department of Crop Production, University of Helsinki with her field work conducted in Tanzania and India.

Dr Pehu has published extensively in cellular biology, physiology and biotechnology of crops, and also in tropical agriculture and international development. Her major interests in development are institutional designs of organizing science and innovation, including public-

55

private partnerships and uptake pathways for new technologies. In all her development work she has an interest to understand and integrate gender.

56

Annex 2: Formal CGIAR meetings and events addressed/participated in by evaluation team members

Meeting/event Date Attendee Mode ISPC – item on the evaluation

5 April Professor O’Kane Dr Pehu

Skype

Annual CRP leaders/DDGs Research meeting

June Dr Pehu In person

SPIA-PIM meeting on Technology options and impact, Nairobi

6 July Dr Pehu Webstreaming

SIMEC – item on finalizing ISPC ToR

18 August Professor O’Kane Dr Pehu

Bluejeans

ISPC 16 meeting 17 September Professor O’Kane Dr Pehu

Skype

57

Annex 3: List of interviewees Victoria Alemany Martín

Meeting Coordinator, ISPC Secretariat

Kym Anderson Center Board Chair, IFPRI, with other CGIAR Australian Leadership Group members

Nick Austin Gates Foundation, and former Interim Director of CGIAR System Organization & Former Director General, ACIAR

Marianne Banziger DDG Research, CIMMYT Michel Bernhardt SIMEC chair also spoke on behalf of Stefan Schmitz, SC Member Nicole Birrell Center Board Chair, CIMMYT, System Council Non-Voting

member, Convener of the Center Board Chairs, interviewed with other CGIAR Australian Leadership Group members

Bas Bouman CRP Director, Rice, IRRI Hans Braun CRP Director, Wheat, CIMMYT Karen Brooks CRP Director, PIM, IFPRI Derek Byerlee Former SPIA chair and ISPC member Marck Cackler System Council member, Sector Manager, World Bank Andrew Campbell System Council member, Director General ACIAR Ken Cassman Former ISPC Chair Tony Cavalieri System Council member, Gates Foundation Philip Chiverton System Council Member, SIDA Vhiba Dhawan Former ISPC member Jeroen Dijkman Senior Agricultural Research Officer, ISPC Secretariat Etienne Duveiller DDG Research, AfricaRice Michel Evequoz System Council Member, Switzerland Lindsay Falvey Center Board Chair, ILRI Samy Gaiji Chief of FAO Research & Extension Unit Peter Gardiner System Organization Geoff Garrett former CSIRO CEO & former Qld Chief Scientist Nighisthy Ghezae ISPC Council Member Maggie Gill ISPC Chair Vincent Gitz CRP Director, FTA, ICRAF Douglas Gollin ISPC Council Member and SPIA chair Elwyn Grainger-Jones

System Organization

Stephen Hall Former Center DG Jacqueline Hughes DDG Research, IRRI Anne-Marie Izac Former Chief Science Officer of Consortium Office (until 2014).

Now Chair of CRP Forest Trees Agroforestry Independent Steering Committee

Dougou Keita SIMEC Member interviewed with other SIMEC members Lakshmi Krishnan Agricultural Research Officer, ISPC Secretariat

Nancy Johnson Senior Agricultural Research Officer, ISPC Secretariat

Victor Kommerell CRP Manager, Wheat, CIMMYT, and Active Observer on SMB Peter Langridge University of Adelaide Melle Leenstra System Council Member, Netherlands Preet Lidder Agricultural Research Officer, ISPC Secretariat Leslie Lipper ISPC Secretariat Executive Director John McDermott CRP Director Holger Meinke ISPC Council Member

58

Tergut Orman representing Nevzat Birisik

System Council member, Turkey

Oscar Ortiz DDG Research, CIP Rodomiro Ortiz ISPC Council Member Selcuk Ozgediz formerly World Bank; wrote CGIAR at 40 Raj Paroda SIMEC member Gabrielle Persley FEWG appointed CRP reviewer Prabhu Pingali ISPC Council Member Muriel Pougheon Administrative Coordinator, ISPC Secretariat Ravi Prabhu DDG Research, ICRAF Tom Randolph CRP Director, L&F, ILRI Bernard Rey System Council and SIMEC member, European Commission Jeff Sayer Former member of ISPC Rachid Serraj Senior Agricultural Research Officer, ISPC Secretariat Jimmy Smith SMB member & Director-General, IPFRI James Stevenson Agricultural Research Officer, ISPC Secretariat David Thodey CSIRO Board Chair Jennifer Thomson ISPC Council Member Alan Tollervey System Council Member, DFID Thomas Tomich ISPC Council Member Daniel van Gilst System Council Member, Norway Stan Vandersyp MFAT, NZ Ira Vater Programme Officer, ISPC Secretariat Juergen Voegele System Council Chair, World Bank Jonathan Wadsworth Former Executive Secretary of Fund Council Ren Wang System Council Member (non-voting) ADG-AG,FAO Patrick Webb ISPC Council Member Stephan Weise DDG Research, Bioversity Eric Witte & Vern Long

SIMEC member, USAID; SC alternate

Mellissa Wood SIMEC Member, ACIAR, SC alternate Beth Woods Former Center Board Chair, WorldFish Gong Xifing System Council Member, CAAS

59

Annex 4: Summary analysis of survey responses A survey of six questions was sent to the following CGIAR stakeholders:

Stakeholder categories Total sent No of responses Response rate Center Board Chairs 17 7 41% Directors-General 15 8 53% Deputy Directors-General 15 4 27% CRP ISC Chairs 9 5 56% CRP Directors 16 9 56% Flagship Leaders 64 19 30% Total 136 52 38%

The questions were:

1. What is your understanding of the role the ISPC? 2. What is the major impact of ISPC on CGIAR? 3. How does its work affect your part of CGIAR? 4. What does ISPC do really well? 5. Are there any changes/modifications to the ISPC’s role and operations that should

be made to improve the effectiveness of CGIAR and its major components?

6. If you could change just one thing about ISPC and how it operates, what would that be? And why?

Caveat The analysis of the survey responses presented here is a summary analysis in which the responses by categories are suppressed as comments on an earlier draft of this report indicated that it might be possible to identify respondents if the category analysis was included.

Also, this high-level analysis here does not reflect the rich set of comments in the responses. The evaluation team thanks those who took part for the detailed commentary many provided. It has informed the overall analysis even if it is not fully reflected below.

Summary of responses to questions

Q1 What is your understanding of the role the ISPC? There was a high level of consistency across all categories of respondent that the role of the ISPC was some form of advising/ ensuring/ strengthening/ maintaining/ oversighting/ guiding/ critiquing/ evaluating the quality of science/research within the System.

Q2 What is the major impact of ISPC on CGIAR? Nearly half the respondents (24/52) thought the ISPC’s main impact on the System was through its assistance/assessment during the pre-proposal and assessment stages of the Phase II CRPs. The next most frequent comments were: lifting science/research quality in a general way (9) or not able to say (5).

As noted, the majority of respondents feel that ISPC has had a major impact on CGIAR through the reviews of the CRP proposals. Some other issues mentioned are included in the table below. Not all aspects mentioned were positive with a few respondents feeling that the ISPC had too much influence on which CRPs were funded but most identified some aspects that were positive. Five respondents felt that ISPC has only had a negative impact.

Most respondents (50) answered this question; 2 respondents did not consider themselves informed enough to answer.

60

Area/keyword mentioned Number (%) of respondents who mentioned this area

Shaping the portfolio and ensuring consistency across system

28 (56%)

Scientific guidance and ensuring science quality and rigour in proposals and overall

20 (40%)

Providing scientific credibility to funders and others (with repercussions on funding decisions)

12 (24%) + 5 (10%) who mentioned only impact on funding decisions (2 out of these 5 mentioned it negatively)

ISPC failed in strengthening system approaches in CGIAR/limited breadth of views in terms of trans-disciplinarity

5 (10%)

Impact assessment 4 (8%) Increase overheads/transaction costs in proposal preparation

3 (6%)

ISPC has had no impact in helping to understand role of research in development

3 (6%)

Note: denominator used is 50

Q3 How does its work affect your part of CGIAR? Area/keyword mentioned Number (%) of respondents who mentioned

this area CRPs 43 (83%) Impact assessment 3 (6%) negative impact on resources or budget 2 (4%) Can’t say/don’t know 2 (4%) Fora 1 (2%) Other 1 (2%)

Note: all respondents answered; denominator used is 52

Q4 What does ISPC do really well? 46 out of the 52 respondents answered this question, six of those not responding to this question were flagship leaders.

Most respondents answered the question directly, but 2 respondents indicated that ISPC does not do anything really well.

Area/keyword mentioned Number (%) of respondents who mentioned this area

Program review/science quality review/review 19 (41%) Independence/neutrality 12 (26%) SPIA/Impact assessment 9 (20%) Guidance 8 (17%) Look across CRP portfolio/system wide perspective/enable collaboration of centers

7 (15%)

Studies/thought pieces 4 (9%) Notes: denominator used is 46; results total >100% as some respondents mentioned more than one topic.

61

Q5 Are there any changes/modifications to the ISPC’s role and operations that should be made to improve the effectiveness of CGIAR and its major components? 48 out of the 52 respondents answered this question.

Area/keyword mentioned Number (%) of respondents who mentioned this area

Increase clarity and transparency in CRP assessments

8 (17%)

More engagement/collaboration with centers and researchers (this was mentioned in two variations – engage more with centers rather than only with CRP management; and engage more with researchers and not only with center/CRP senior management)

8 (17%)

Partnerships - more guidance needed from ISPC: 6 & drop partnerships 1

7 (15%)

Strengthen foresight 7 (15%) all from center governance/management

Reduce size and cost of ISPC 6 (12%) Define ISPC role more clearly 4 (8%) More engagement/dialogue with donors 3 (6%) Ensure that the needs of the system to have science quality considered in the context of the research for development impact imperative are fully embraced

3 (6%)

Note: denominator used is 48.

The responses on what should be changed were very varied. Several other things were mentioned, not captured here.

Q6 If you could change just one thing about ISPC and how it operates, what would that be? And why? 43 out of the 52 respondents answered this question. Some respondents indicated that they did not answer this question because they would have repeated points written in reply to question 5.

Area/keyword mentioned Number (%) of respondents who mentioned this area

CRP review process:

• No more reviews • Keep reviews but focus on assessment of science

quality only • More transparency • Other

8 (19%):

• 2 (5%) • 2 (5%)

. • 2 (5%) • 2 (5%)

Increase engagement/interaction with centers and researchers

8 (19%)

Be more of a think tank and innovation broker 7 (16%) Hold fewer meetings 3 (7%) Better representation of disciplines amongst Council members

3 (7%)

Note: denominator used is 43.

The ‘just one thing to change’ range of responses on what should be changed was also very varied. Several other things were mentioned, not captured here.

The ‘why’ responses are not included as they might lead to respondents being identified.

62

Other comments Taking the responses as a whole: ISPC works well: 7 yes; 13 no; 31 could be better. ISPC work with CRPs positive or negative: 23 positive; 21 negative; neutral 7.