Measuring and Improving the Effectiveness

of R&D Systems in Sub-Saharan Africa

Leonard Oruko

IAAE Symposium on Improving Returns to Agricultural Research in Sub-Saharan AfricaFoz do Iguaçu | 20 August 2012

R&D Results Measurement Challenges

• What is an effective R&D system?– Outputs impact on poverty, food security and income growth– Generates relevant products and services in a timely fashion– Has adequate human capacity and financial resources

• Can we demonstrate Results of Ag. R&D Investments?– Human capital, infrastructure and operational funding constraints – Long time lags from the point of investment to the manifestation of

returns– “ A fishing expedition or a shooting range” – Well functioning support services and institutions must be in place for

research outputs to have an impact on development outcomes

R&D Results Measurement Challenges..

• Research evaluation has supported the case for R&D– The tools and information developed for evidence-based policy were

linked to the development imperatives of the day

• Research evaluation responded to questions being asked – Economic returns , welfare analysis, priority setting, funding of

research issues– Concerns with poverty (well beyond producer and consumer surplus),

NRM and sustainability (beyond production systems) , and later climate change at increasing scale

• Impact assessment has had to balance the needs for accountability to funders versus learning and change by actors– Economic return, experiments and quasi experiments (quantitative)– Utilization-focused evaluation, qualitative

Status of Results Measurement in SSA Ag. R&D Institutions

• Ex-ante impact evaluation*CAADP Framework– Evidence of some NARS adopting objective criteria, with support from

the CG Centres– ASARECA, CORAF and CCARDESA focus on estimating spillover

potential

• Managing research implementation process– Accountability focus pushing NARS towards RBM-PABRA– FARA and SRO gravitating around RBM derived CPMF– CG-CRP

• Ex post impact evaluation– Adoption– Precise measurement of impact with RCT emerging as “ the Gold

Standard”

Impact of Ag. R&D: Common practice

• Primary focus has been that of establishing the impact on development outcomes

Source: Block, 2010

Outcomes of Ag. R&D

• Estimates of adoption primarily case specific – Targeted studies estimate adoption levels and determinants of

adoption; guide research planning and priority setting • Important lessons for R&D results measurement systems

– LSMS-ISA– DIVA Initiative

Crop % cropped area

Maize (in west and central Africa) 67

Cassava 39

Beans 32

Sorghum 14

Source: Alene, et al, 2011

Adoption of improved varieties

Inputs and outputs

Commodity 2010 1998Rice 10.9 6Maize (west and central Africa) 9.2 10Cassava 1.2 3Sorghum 1.7 5*Beans 33.7 21

Source: Alene, et al,2011

• The DIVA Initiative• Research expenditure• Full Time Equivalent Scientists(FTEs)• Research Intensity• Mean Incidence of varietal Output

Changes in researcher Intensity ratio over time

Improving the Results measurement of R&D systems

“A major knowledge gap in understanding and strengthening R&D systems stems from the lack of empirical application of

framework, metrics, and benchmarks to measure organizational performance and institutional impact in the context of

agricultural research”Ragasa, 2011

• Organization design theory in the context of innovation system– Coordination mechanisms that provide incentives for innovation– Demand responsiveness and connectivity to other actors in the

innovation system

Empirics from Ghana and Nigeria : Perception Ratings)

• Output, outcome and impact indicators are standard-Technologies generated-Publications– Adoption of technologies (most researchers unaware of the adoption

rates)– Limited complementarity and consistency across the indicators

• Connectivity– Linkage with other researchers exist, limited in the case of extension,

farmers and other innovation actors

• Organization culture and job satisfaction– Satisfaction with outputs– Staff morale – Perception on effectiveness of the organization

Way Forwards for results Measurement

• There is no substitute for valid and credible data– Real time data for operational management not available in the majority of

cases – Measurement error arising from reported area and output data (LSMS-ISA) – Data is a valuable resource\treasure often kept in “armory” – Challenges with data sharing protocols hence despite the noble intentions

espoused in; CAADP, CRP, SRO

• Getting adequate data for results measurement is costly – Owing to scarcity of resources , collection of performance data and information

is rarely given priority-donors are pushing for a reversal!– Greater chances of getting resources when framed as a research endeavor– Operational management data is often treated as confidential

Way Forwards for Results Measurement

• Strategic Focus in SSA– ASTI initiative to support the NARS in the institutionalization of

data collection and expand to include output indicators– Work with SRO and RECS

– FARA and SRO’s to focus on quantifying the externalities/spillovers and, support the NARS in developing measurement approaches for effective coordination and management

– NARS to plug into the broader innovation system and NIMES in order to demonstrate contribution to broader development agenda

Way Forwards for Results Measurement

• Rigorous impact evaluation approaches recommended– Selective use of RCT and other quasi experimental methods given the

associated costs– Develop rapid and robust approaches for measuring the impact of

R&D on development outcomes• Operational management support

– Great research opportunity in the area of agricultural innovation systems employing management science tools

• Effective results measurement systems respond to information needs in a timely fashion-an art– Proactive strategic analyses– Consistent data collection effort– Focus on generating evidence and catalyzing use*Duplication of efforts arising from fragmented approach to data collection

THANK YOU