Serbia - Innovation Serbia Project - Implementation Completion and Results Report(TF071742)
(US$11.9 MILLION EQUIVALENT)
CURRENCY EQUIVALENTS
Currency Unit = Euro (€)
EU IPA
FINS
FM
ICT
IMGGE
IMPR
IP
MOSTD
MOESTD
MPI
TA
Senior Global Practice Director:
E. Bank Staff………………………………………………………………………….ii
G. Ratings of Project Performance in ISRs………………………………………......vi
H. Restructuring ……………………………………………………………………..vi
3. Assessment of Outcomes 16
4. Assessment of Risk to Development Outcome 27
5. Assessment of Bank and Recipient Performance 28
6. Lessons Learned 30
Annex 1. Project Costs and Financing 32
Annex 2. Outputs by Component 33
Annex 3. Economic and Financial Analysis 40
Annex 4. Bank Lending and Implementation Support/Supervision Processes 41
Annex 5. Beneficiary Survey Results 43
Annex 6. Stakeholder Workshop Report and Results 47
Annex 7. Summary of Recipient's ICR and/or Comments on Draft ICR 48
Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders 50
Annex 9. List of Supporting Documents 51
MAP …………………………………………………………………………………..52
US$9.52 million
Disbursed Amount:
US$8.35 million[footnoteRef:2] [2: This system generated datasheet only reflects Recipient Executed disbursement amounts and does not reflect Bank Executed components or fees associated with this hybrid trust fund. This table would suggest significant undisbursed and cancelled resources at the end. However, only Euro 655,000 were cancelled at project closing. Differences between the Revised amount and the Disbursed Amount are most likely explained by the Euro vs. US$ exchange rate differential between appraisal and closing.]
Revised Amount:
US$7.15 million
Implementing Agencies: Ministry of Education, Science and Technological Development (MOESTD); Serbia Innovation Fund (IF)
Co-financiers and Other External Partners: The Donor is the EU Delegation in Serbia
B. Key Dates
Outcomes:
Satisfactory
C.2 Detailed Ratings of Bank and Recipient Performance (by ICR)
Bank
Ratings
Recipient
Ratings
Implementation Performance
No
No
Satisfactory
Original
Actual
Public administration - Financial Sector
10
10
Technology diffusion
Project Development Objectives (from Project Appraisal Document)
The development objective of the project is to assist in building the institutional capacity to stimulate innovative activities in the enterprise sector by:
(a) Supporting the operationalization of the Serbia Innovation Fund (IF);
(b) Piloting financial instruments for technological development and innovation in enterprises; and
(c) Encouraging selected research and development institutes (RDIs) to engage in technology transfer and commercialization, and assisting in formulating RDI sector reform policy.
Revised Project Development Objectives (No changes, as approved by original approving authority
(b) Intermediate Outcome Indicator(s)
Formally Revised Target Values
Indicator 1 :
Value
(quantitative
Number of IF managers trained and applying newly learnt skills
Value
(quantitative
Value
(quantitative
Number of networking and educational events organized for enterprises, academia and research
Value
(quantitative
Number of RDIs identified for technology transfer and commercialization support
Value
(quantitative
Number of RDIs assessed jointly by international experts & RDI management
Value
(quantitative
Indicator
Formally Revised Target Values
Indicator 1 :
Amount of innovation financing mobilized by IF in addition to this EU IPA project
Value
Date achieved
Value
Number of new products and processes launched by beneficiary enterprises
Value
Value
Value
Reason for Restructuring & Key Changes Made
DO
IP
No
S
S
6.36
Closing date extended from November 30, 2014 to January 10, 2016. The extension allowed IF to complete core program activities, including contracting of the new awardees from the fourth call for proposals as well as disbursing earlier approved funds to the existing awardees.
I. Disbursement Profile[footnoteRef:3] [3: This table would suggest significant undisbursed and cancelled resources at the end. In fact, there was a small cancellation of remaining funds at Closing, equaling Euro 655,000. This chart is explained by the exchange rate differential at appraisal and closing of the Euro vs. US$.]
3
1. Project Context, Development Objectives and Design[footnoteRef:4] [4: This section is based on the Project Appraisal Document (PAD).]
1.1 Context at Appraisal
1. The national innovation system and composition of research and development (R&D) funding in 2012 did not support enterprise innovation at any significant level, and the research sector modernization agenda was in its infancy. The Serbian R&D sector was dominated by the public sector, largely inefficient, and most importantly delinked from industry needs, particularly those of local small and medium enterprises (SMEs). The Serbian industry’s capacity, both in terms of human capital and financial resources for R&D and innovation, had been severely weakened post-transition and exacerbated by the lingering effects of the financial crisis. Brain drain was a major concern, in the private sector and even more so in the scientific community.
2. Serbia spent roughly €100 million (0.3 percent of gross domestic product [GDP] in 2010) on R&D, with a goal to reach 1 percent by 2014. While R&D spending had increased from €28 million in 2001, Serbia still lagged significantly behind its neighbors. Slovenia, the Czech Republic, and Croatia spend more than 1 percent of GDP, and the European average was 1.8 percent. In Serbia, the majority of R&D spending went to basic research, which accounted for 50 percent of all R&D funding. A significant shift from basic to applied research was needed for research commercialization and development of an innovative SME sector. This, along with developing stronger connections between science and industry, was to be key in raising R&D expenditures to 2 percent of GDP by 2020, with a target of leveraging half of this from the private sector.
3. Outputs from the R&D sector were not commensurate with the public resources being invested and did not support modernization of the Serbian economy. While the number of scientific publications had increased during the previous few years, quality remained poor. Intellectual property (IP) was either not being created or not being protected. Merely 21 patents were registered by public research and development institutes (RDIs) from 2003 to 2008, with only 36 patent applications, mostly in Serbia. Figures for the private sector were similar.
4. In an attempt to modernize the national innovation system, the authorities had implemented a reform program over the previous two years to revamp the regulatory and institutional framework governing science, technology, and innovation. It had enacted a number of laws to promote science and innovation and formulated the Serbian Scientific and Technological Development Strategy 2009–2014. In addition, the Ministry of Education and Science was created as the central body responsible for science and innovation in Serbia.[footnoteRef:5] [5: The March 2011 restructuring of the Government of Serbia (GoS) led to a merger of the Ministry of Science and Technological Development (MOSTD) with the Ministry of Education, thus creating a new entity, the Ministry of Education, Science and Technological Development (MOESTD). The innovation and technology development priorities have remained unchanged as a result of restructuring. The review of programs described herein was prepared by MOSTD.]
5. Science infrastructure improvement had been the main focus to that point, while stimulation of private sector-led R&D and innovation had been negligible. The European Investment Bank provided a loan of €200 million to upgrade infrastructure at universities and RDIs and to set up science parks as well as Centers of Excellence in priority areas including biotechnology, nanotechnology, and advanced computing. In 2010, the Government disbursed approximately €70 million for basic research, technological development, integral and interdisciplinary research, and innovation activities. However, almost 80 percent of the funds intended for science projects actually went to salaries for researchers.
6. RDIs were the primary recipients of financing, with private firms receiving funding only under the technological development component. The 471 technological development projects supported by the MOSTD were led by RDIs and not by industry. Public RDIs depended on these projects as they did not receive regular institutional budget allocations. Projects generally had a 90 percent approval rate, creating disincentives for researchers to innovate. Except in a few cases, technology transfer and diffusion from RDIs was quite low with only a few spin-offs and real technology-based start-ups, indicating the need to place greater emphasis on supporting the various stages of technology development and innovation within enterprises—for example, technology transfer, commercialization, diffusion, absorption, adaption, and application.
7. Few mechanisms to incentivize private sector R&D and innovation had been implemented to that point in time. The Ministry of Economy and Regional Development (MOERD) administered the Program to Strengthen Innovation in SMEs, that is, matching grants, to support SME investment in innovation through co-financing. However, the total budget for 2010 was RSD 40 million (€380,000). All legal entities were eligible for co-financing of up to 50 percent of justified expenses for innovation activity. Matching funds had to be provided from the SME’s own resources and could not include any other public support. The awarded amounts could not exceed RSD 800,000 (€7,500) for the first group of eligible activities, and RSD 1.5 million (€14,000) for the second group. In addition, under the Instrument for Pre-Accession (IPA) 2010 program, €3 million was allocated for the Integrated Innovation Support Program with the objective to ‘increase the competitiveness and economic growth in Serbia, through strengthening of innovation in SMEs in accordance to National Strategy for Development of Competitive and Innovative SMEs 2008–2013.’ However, implementation of this program had not yet started.
8. In the post-crisis environment, the Government was expected to play a critical role in stimulating economic recovery through policies that target firm-based R&D and innovation. While the Government and other international donors had demonstrated efforts to support R&D and innovation, these were focused on renewing infrastructure investments in public RDIs with negligible support for private sector innovation. Given the World Bank’s experience garnered in supporting private sector development in post-transition economies, including in neighboring Croatia, the GoS was keen on a World Bank engagement on the innovation agenda. Hence, the World Bank’s technical assistance (TA) to the GoS and to the future Innovation Fund (IF), being established under the project as the implementing agency, focused on filling these gaps and developing mechanisms to stimulate enterprise-led technology development and innovation.
9. The objective of the FY12-15 CPS was to support Serbia’s EU accession and help the Government strengthen competitiveness and improve the efficiency and outcomes of social spending. Under the Strengthening Competitiveness pillar Improved innovation capacity was identified as one of objectives. The CPS noted that the system and composition of R&D funding did not support Serbia’s agenda to modernize and enhance its competitiveness. The CPS sought to change this situation by supporting the establishment of institutional capacity to stimulate activities in the enterprise sector. The capacity would arise from creating a Serbia Innovation Fund, using financial instruments for innovation and technological development in enterprises, and having R&D institutes engaged in transferring and commercializing technology and in helping formulate RDI reform policy.
1.2 Original Project Development Objectives (PDO) and Key Indicators
10. The project development objective (PDO) was to assist in building the institutional capacity to stimulate innovative activities in the enterprise sector by:
(a) Supporting the operationalization of the Serbia Innovation Fund (IF);
(b) Piloting financial instruments for technological development and innovation in enterprises; and
(c) Encouraging selected research and development institutes (RDIs) to engage in technology transfer and commercialization, and assisting in formulating RDI sector reform policy.
11. The key performance indicators for this project included:
(a) Amount of innovation financing mobilized in addition to this European Union Instrument for Pre-Accession (EU IPA) project;
(b) Number of active technology start-ups funded;
(c) Number of new products and processes launched by beneficiary enterprises;
(d) Number of technologies transferred by RDIs; and
(e) Dissemination of policy recommendations for RDI sector reform.
1.3 Revised PDO and Key Indicators
12. The PDO was not revised. PDO-level indicators 1 and 2 were refined for clarity’s sake during negotiations with the donor and recipient, and are fundamentally the same as in the PAD.
1.4 Main Beneficiaries
13. The primary beneficiaries of this pilot project were the staff of the IF whose capacity was built under Component 1; entrepreneurs, start-ups, and firms that received financing and mentoring/training under Component 2; researchers and leadership of select RDIs that benefited from exposure to modern practices in IP management, technology transfer, and commercialization, as well as the Ministry of Education, Science and Technological Development (MOESTD) that also benefited from the policy recommendations developed based on the RDI TA program under Component 3. Important secondary beneficiaries included stakeholders engaged during project launch and implementation through numerous open houses/workshops/conferences on such topics as entrepreneurship, research commercialization, or innovation finance. These stakeholders ranged from across Serbia’s national innovation system, including leading universities, RDIs, incubators, technology transfer offices, nascent early stage investors, and private sector actors.
1.5 Original Components
14. In order to build institutional capacity to stimulate innovative activities in the enterprise sector, the project design intended to leverage three complementary components that would enable piloting of institutional, financial, and commercialization support mechanisms absent from the national innovation ecosystem with the view to support and ideally demonstrate Serbia’s enterprise innovation potential.
Component 1: Capacity Building of the Serbia Innovation Fund - Recipient-Executed (€1.1 million)
15. The capacity of the IF was being established to encourage entrepreneurship, including by financing enterprise innovation, as well as participating in long-term programming efforts by national authorities, international organizations, financial institutions, and the private sector. In this context, this component provided support for:
(a) operations of the Investment Committee and due diligence of grant applications;
(b) engagement of strategic and operations advisors to assist the IF’s current and future programs, strategy, operations, and procedures;
(c) operational project management in relation to the European Union (EU) and World Bank;
(d) IF staff training;
(g) IF operational infrastructure, including information and communication technology (ICT) and financial management (FM) development.
Component 2: Implementation of Financial Instruments Supporting Enterprise Innovation - Recipient-Executed (€6.0 million)
16. This component financed (a) Mini Grants targeting the proof of concept and prototyping stages, including but not limited to IP protection and business plan preparation for initial capital mobilization; and (b) Matching Grants targeting R&D in technology development projects, for new or improved technologies, products and processes. Incorporated entrepreneurs, innovative start-ups, spin-offs, micro and small enterprises, with majority Serbian private sector ownership were eligible for Mini and Matching grants. The Mini and Matching Grants applications were evaluated by the IF’s independent Investment Committee (IC), with input from international peer reviewers.
17. The resources under this component were earmarked notionally per instrument with the objective of flexible reallocation among instruments based on demand. The terms and conditions of the grant programs were designed with the IF advisors with sufficient flexibility to accommodate changing market conditions and the quality and volume of the project pipeline. A guiding principle was to decrease the matching contribution from the IF over time, particularly for the Matching Grants instrument.
Table 1. Design Features of the Mini Grant Programs at Appraisal
Mini Grants
Baby Seed
€1,500,000–€3,000,000
Proof of concept, prototyping stage, IP protection, business plan preparation for mobilization of initial capital
Recipient
Incorporated entrepreneurs, innovative startups, spin offs, micro and SMEs, all with majority Serbian private sector ownership
Grant size
Up to €80,000 for projects that will be completed within 12 months
Features
Up to 85% of approved project costs
Table 2. Design Features of the Mini Grant Programs at Appraisal
Matching Grants
€3,000,000–€4,500,000
Objective and stage
R&D (technology development) and commercialization projects for new or improved technologies, products, and processes
Recipient
Incorporated entrepreneurs, innovative startups, spin offs, micro and SMEs, all with majority Serbian private sector ownership
Grant size
Up to €300,000 for projects that will be completed within 2 years.
Features
Up to 75% Matching Grant with an optional 3–6% royalty component based on sales revenue, up to 120% of the original grant within a predetermined time frame
Component 3: Provision of Technical Assistance to Research and Development Institutes (RDIs) - Bank-Executed (€0.7 million)
18. Component 3 was to provide (a) customized TA to up to two RDIs based on a detailed needs assessment; and (b) limited TA to up to four RDIs based on a general needs assessment; and (c) technical input to the Government’s future RDI sector reform program based on lessons learned from the aforementioned TA program.
19. A customized TA program for up to two select RDIs entailed conducting a detailed assessment to be carried out by international experts in partnership with the RDIs’ management. The assessment was to cover issues, including but not limited to, the organization’s management structure, human and budgetary resources, researcher promotion and incentive system, infrastructure and laboratory facilities, research capabilities and outputs, IP protection, challenges and potential for applied R&D, and technology transfer and commercialization. Based on this diagnosis, technical support was to be provided for the design and implementation of RDI-specific upgrading programs. Upgrading of labs and other infrastructure would not be supported. The RDIs were identified jointly by MOESTD and the World Bank based on several key factors, including the level of commitment to reform from the RDI management team, potential for improvement, likelihood of success, sectoral coverage, and availability of resources. The lessons derived from this program were expected to inform future strategic planning and reforms in the RDI sector. Specifically, the recommendations were expected to be focused and actionable, while identifying quick wins. The World Bank team was to organize a workshop with the ministry in charge to disseminate the experience, lessons learned, and recommendations for a future RDI system reform program.
1.6 Revised Components
1.7 Other Significant Changes
21. There were no significant changes. A simple project restructuring was conducted with approval from the management of the Europe and Central Asia Region and the donor to extend the closing date from November 30, 2014 to January 10, 2016. The extension allowed the IF to complete core activities, such as contracting of awardees from the fourth call and disbursing previously approved funds to awardees.
22. The US$ appreciated almost 30 percent against the EURO during project implementation. The US dollar amount at project approval was $11.9 million. The final disbursement amount – both Bank and Recipient Executed - was approximately US$ 9.7 million, with disbursement via the Recipient amounting to US$8.35 million. Of the total EURO 8.4 million, EURO 655,000 was cancelled. Fortunately, the exchange rate differential in dollars did not affect implementation as the grant and expenditures were in Euros.
2. Key Factors Affecting Implementation and Outcomes
2.1 Project Preparation, Design and Quality at Entry
(a) Soundness of the Background Analysis
23. The background analysis for this project was sound, given its pilot nature and little government experience in supporting enterprise innovation or RDI sector reforms. The project was aligned with the FY08–FY11 Country Partnership Strategy, wherein a key priority included encouraging dynamic private sector-led growth to promote convergence with European levels. In FY09 and FY10, the World Bank team delivered a TA program, supporting MOESTD to update legislative frameworks that encouraged firm uptake of innovation support. Specifically, the World Bank team worked on identifying challenges with existing innovation programs, reviewing the legal framework governing R&D and innovation, and a new innovation strategy. Further, at entry, the focus was on (a) appraising institutional arrangements supporting research and innovation; (b) surveying and assessing firm perception of existing innovation and technology absorption support programs and appetite for proposed instruments; and (c) discerning level of opposition for support to firms and for research sector reform. The analysis and consultations were instrumental in sharing regional and global experience, building stakeholder confidence on the need to deploy a pilot project that would initiate a steady transition to an enterprise-led model.
(b) Assessment of the Project Design
24. Pilot. The project was designed carefully as a modest pilot initiative that aimed at building the missing institutional capacity in the Serbian innovation system, testing financial instruments and mentoring mechanisms to reveal the emerging innovative entrepreneurial sector, and identifying viable approaches to the challenges of Serbia’s ailing research sector. Given high expectations from this pilot, the risky nature of both demand- and supply-side interventions, accompanied by significant latent opposition from researchers in the public RDIs, the split between World Bank and recipient-executed components was prudent. Equally judicious was the attraction of top global expertise to build credibility for this pilot, and demonstration of key principles in innovation policy management, including horizontality, independence, good governance, efficiency, transparency, monitoring and evaluation (M&E), and open communication with stakeholders.
25. Selection Procedures. Noteworthy design features included the IF’s efficient and independent grant selection processes that engaged international peer reviewers and an IC featuring international and diaspora experts with a background in scientific research, entrepreneurship, technology commercialization, market trends, and venture investments.
26. Visibility. The donor placed significant emphasis on organizing project visibility activities and these ended up being instrumental given the risky and novel nature of the pilot. Most visibility activities were high level in nature and co-hosted by the recipient, the donor, and the World Bank, which helped reinforce the Government’s commitment in spite of multiple ministerial reshufflings within the lifetime of the project.
27. Ownership. The World Bank-executed TA component was designed to support knowledge transfer and research commercialization within the lifetime of the project. Only those RDIs where management demonstrated (a) willingness to be assessed by international expert teams convened for their institution and (b) commitment to act on key recommendations from such assessments, were included in the RDI TA program. This allowed the World Bank team to narrow down the list of RDIs it chose to provide TA support. In retrospect, this important design feature afforded flexibility, as both time and financial resources were limited.
(c) Adequacy of Government’s Commitment
28. The Government’s commitment toward the project—and especially for the enterprise innovation objectives—was high at entry. The Government demonstrated this by pursuing the Delegation of the European Union (EUD) to earmark EU IPA funding for enterprise innovation for the very first time, approving a notional operational budget for the IF from 2011 to 2015, and attracting a senior diaspora member to lead the IF. The World Bank was nominated by the GoS to enter into a trust fund arrangement and execute Component 3, which was controversial at the time.
(d) Assessment of Risks
29. Several risks were identified during project preparation and adequate risk mitigation measures were incorporated into the project design as follows:
· A pioneer on several fronts, there were important coordination and reputational risks stemming from this pilot. This was the first ever EU-financed World Bank-administered trust fund in Serbia, addressing unchartered policy territory, facing skepticism by entrenched interests and significant political instability. Through meticulous supervision and dialogue with the EUD, MOESTD, MOERD, IF, and other stakeholders during missions and steering committee meetings, the World Bank fostered shared awareness on project progress and ownership of challenges as they arose.
· Significant resistance from entrenched interests to RDI sector reforms. The World Bank team managed the RDI TA program closely, expanding the scope of assistance only to volunteer RDIs where management demonstrated firm commitment to optimization and knowledge transfer and research commercialization outcomes.
· The IF had no demonstrable capacity to transparently select projects, manage FM, procurement, environmental screenings, or support the private sector. The World Bank’s no objection to the selection of IC members, and to procedures followed during the first call, supported the credibility of the selection processes. The international advisors provided continuous capacity building for IF staff on program design couched in global practice, whereas the IC safeguarded the IF’s independence, together tempering undue influence from the research community or political interference.
2.2 Implementation
30. The following factors had an impact on project implementation:
· Factors having a positive impact
· The establishment of the IF as an efficient independent institution with strong and transparent governance practices was instrumental in smooth project implementation.
· The independent and professional manner in which beneficiaries were selected and monitored during implementation ensured that the selected subprojects were successfully implemented.
· Continuous access to the IF Advisory team, oversight from the World Bank, and access to MOESTD, EUD, and the IF’s Board, was important in allowing the IF to overcome teething and political challenges over the course of the pilot, focus on creating a positive environment for its beneficiaries, and introduce new innovation and technology transfer programs for Serbia.
· In line with the concept of the pilot, both recipient- and World Bank-executed components were monitored closely and pivoted incrementally to enhance program outcomes and inform longer-term instrument design. A few examples include design of the mentoring program to accommodate mandatory and customized training elements for IF staff and beneficiary entrepreneurs; deploying the Mini and Matching Grant programs sequentially so the IF could incorporate lessons from the first call of the Mini Grant Program; increasing the private sector contribution (from 25 percent to 30 percent ) under the Matching Grant program; converging on design details for the royalty scheme; and engaging commercialization brokers to work with RDIs to identify commercialization wins within the lifetime of the project.
· Under Component 3, the growing acceptance of knowledge transfer and research commercialization was an important cultural shift among researchers and managers in the RDI community, enabling the recipient to consider sector wide reforms in the future.
· Continuous access to the IF Advisory team, oversight from the World Bank, and access to MOESTD, EUD, and the IF’s Board, was important in allowing the IF to overcome teething and political challenges over the course of the pilot, focus on creating a positive environment for its beneficiaries, and introduce new innovation and technology transfer programs for Serbia.
· Factors having a negative impact
· 2012–2013. During the early stages of project implementation, disbursements were lower than planned at entry. This was due to GoS insistence at entry to plan as many as two Mini and two Matching Grant Calls for Proposals (CFPs) in 2012. The IF had just been established and did not have the capacity at the time to handle the proposed workload. However, based on guidance the IF received from its advisors, it was eventually agreed to phase the rollout of the programs and to pilot the first CFP for the Mini Grant Program separately, and incorporate any lessons from this call into future ones. In retrospect, this was an important adjustment, enabling IF staff to be better prepared for subsequent calls.
· 2014. Serbia faced a turbulent year in 2014 and so did the IF. This included catastrophic flooding, multiple changes in the Government, delayed funding, and management restructuring within the IF. Implementation progress slowed due to the lack of budget funds for the IF’s operations and given the uncertainty, the IF experienced significant staff departure. Additionally, MOESTD’s interest in engaging the World Bank on the public research sector reform agenda diminished in the absence of a larger World Bank operation. The project was downgraded from Satisfactory to Moderately Satisfactory. Nonetheless, with guidance from its advisory team, the IF was able to rebuild staff capacity by conducting additional trainings to bring the new cohort to speed. During this period, project implementation continued, albeit at a slower speed. By July 2014, the IF was fully staffed, the budget fully funded, and operations commenced satisfactorily. After the mid-term review in August 2014, the project was again rated Satisfactory, and this status was maintained through closing in January 10, 2016.
2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization
Rating: Substantial
31. M&E design. There was no systematic effort to capture firm-level innovation or research commercialization data in Serbia at entry. Publicly available indicators were limited to typical inputs (for example, public R&D spend) or research outputs (for example, publications and patents). Given minimal baseline information, skepticism around the pilot in Serbia, the growing debate on effectiveness of Matching Grants in the World Bank, and high donor and recipient expectations, the M&E activities envisaged under this project were deliberate. Beyond the set of input, output, and outcome indicators in the results matrix, the project focused on building the IF’s long-term capacity to identify additional outcome indicators, build its internal systems to capture and analyze firm-level innovation data to support future evidence-based policy making in Serbia beyond this modest pilot. At project inception, M&E activities were managed and supported by two IF staff members, a program manager, and a senior associate, whose tasks entailed assisting in data collection, overseeing deliverables, and payment schedules. However, as implementation progressed, the IF realized that it lacked the M&E expertise to establish and operate an in–house system. As a result, a reputable external think tank with a track record in conducting innovation program impact evaluations was engaged by the IF from May 2013 to November 2015 to ensure the quality of methodology and implementation, and to design the most relevant approach for the small sample size of beneficiaries expected under the pilot.
32. M&E implementation. The consulting firm developed a framework and procedures for M&E activities under the project which was incorporated in an M&E manual for the IF, and in addition, trained the relevant staff. Staff training provided by the external consultant focused on identifying relevant indicators, monitoring processes, specifically deploying IF data collection systems. As part of their assignment, the firm and IF gathered extensive data on both beneficiaries and applicants. M&E functions were to monitor and analyze the performance of the Mini and Matching Grant programs’ subprojects. IF staff were also trained to monitor financing programs to be managed by the IF in the future. The IF produced semiannual M&E reports. These reports were shared with the Government, World Bank, and EUD, and served as a useful tool for project monitoring and progress reporting. The independent evaluation was led by an independent firm. For the World Bank-executed component, M&E data was collected and analysed by the World Bank team.
33. M&E utilization. The IF and the World Bank monitored the interim outputs of the project systematically to assess progress toward end-of-project target outcomes, identifying issues and risks, and reporting on time-bound action plans to mitigate them. In August 2015, with participation of IF staff, the consulting firm presented its findings to key stakeholders, including senior representatives from MOESTD, MOERD, EUD, Ministry of Finance, and IP office, among others. The focus of the presentation was on the selection process, demonstrating the additionality of IF programs and recommendations for scaling of future programing. The evaluations of the Mini and Matching Grant programs justified the EUD’s decision to provide additional funding for these programs under its future programming and also justified inclusion of these programs under the World Bank-funded Competitiveness and Jobs Project (C&JP), a results-based operation (2015). On Component 3, the monitoring indicators proved useful in concentrating efforts of RDI management and researchers on knowledge transfer and commercialization activities. Importantly, it allowed the World Bank team to estimate the financing gap for knowledge transfer and research commercialization in Serbia, enabling the GoS to advocate for the sequel EU IPA-financed program with the EUD, which supported the establishment of the national Technology Transfer Facility (TTF) and the Collaborative Grant Scheme under the ongoing Serbia Research, Innovation and Technology Transfer Project (SRITTP).
2.4 Safeguard and Fiduciary Compliance
34. The project did not entail any major environmental risk, include construction or land acquisition. The project was rated category ‘B’. There were no issues in compliance with environmental safeguards. Due to the limited size and scope of the grants, there was no significant environmental impact associated with the project’s activities. The overall innovation process under this component did not support environmentally unfriendly technologies and practices. The Environmental Management Framework was implemented successfully. The project did not entail any social risk or risk of adverse social impact.
35. FM was assessed as Satisfactory. There were no outstanding audit reports for the project and all audit reports were found satisfactory to the World Bank. Quarterly interim unaudited financial reports were submitted to the World Bank on time. The project audits received ‘unqualified’ opinions of the audited financial statements and no major systems and control issues were identified by the auditors.
36. Procurement was rated Satisfactory, based on review of prior review documents and post review missions. The IF and beneficiaries followed the World Bank's procurement guidelines and no major issues were encountered. For procurement under the grants, World Bank-approved simplified commercial practices were used.
2.5 Post-completion Operation/Next Phase
37. In 2014, the Government was unable to finance post-pilot calls for the IF grant programs as budget resources were diverted to post-flood remediation efforts. However, key elements for a post-pilot transition are now in place with the approval of the EU IPA SRITTP and the C&JP operations in 2015. SRITTP provides support for national technology transfer activities, collaborative industry/academia research activities, and support for development of the new strategy, the infrastructure road map, and their related action plans. C&JP supports among other activities, the integration of the IF’s operational budget into the GoS’s annual budget, continuation of the Mini and Matching Grant Programs, scaling of technology transfer initiatives, and RDI sector reforms through a results-based approach.
3. Assessment of Outcomes
38. There were no major shortcomings in achievement of project objectives, efficiency, or relevance.
3.1 Relevance of Objectives, Design, and Implementation
(a) Relevance of Objectives
39. The relevance of the objective is rated High. The objectives and related outcomes of this project are as relevant at the time of the Implementation Completion and Results Report (ICR) as they were at entry and remain strongly aligned with the World Bank’s engagement in Serbia and the Government’s development agenda. This project remains highly relevant for the overarching objectives laid out in the FY16-20 CPF which is to support Serbia in creating a competitive and inclusive economy and, through this, to promote integration into the EU. One of two broad CPF focus areas - Private sector growth and economic inclusion should address significant constraints to private sector development and economic inclusion: financing, investment, connectivity, and labor market constraints. This Project was both timely and relevant as it supported the creation of institutional capacity and programming for private sector innovation that can be scaled with future IPA and eventually EU funding.
40. The GoS is undertaking measures to rebalance the economy, enhance the quality and efficiency of public expenditures and service delivery (including state-owned enterprise reform), and stimulate private sector initiatives to enhance Serbia’s competitiveness. In March 2016, the Government adopted the Strategy for Science and Technology Development: Research for Innovation 2016–2020, specifically calling for reforms in the public RDI sector and reinforcing the importance of enterprise innovation and technology transfer for the economy, including support for the activities carried out by the IF. This pilot furnishes evidence to engage on innovation policy and is demonstrated by GoS engagement in the SRITTP and C&JP.
(b) Relevance of Design
41. The relevance of the design is rated Substantial. Project design was essential in supporting the delivery of PDO outcomes and enabled successful piloting of missing elements of the Serbian innovation system. The core design features remain relevant and are therefore being continued or expanded under SRITTP and C&JP. They would warrant being supplemented by increased research infrastructure investments, coupled with reforms in the public research sector, and direct support instruments for enterprise innovation and market access in the future. Under Component 1, the IF leveraged funding and partnerships with international donor organizations to strategically build entrepreneurship and innovation programming from traditional early-stage matching grant funding (SRITTP, C&JP) to early stage equity funding (that is, Western Balkans Enterprise Development and Innovation Facility [WB EDIF]). Through Component 2, the IF demonstrated long-term demand for early-stage enterprise innovation funding and the ability of Serbia’s private sector to lead innovation projects (often in partnership with the research sector) and co-finance these with own matching resources. Under Component 3, the World Bank managed to provide in-depth TA to four RDIs and deliver technology transfer targets by being selective and staying focused. This built the case to establish a national TTF to stimulate collaborative research under the SRITTP and support RDI sector reform activities under the C&JP. At entry, there were no visible private ecosystem players present in Serbia; hence, the pilot could not be designed to engage them. In future iterations of innovation programming and financial instrument design (including of the Matching Grant programs), the IF could explore leveraging local private ecosystem actors—for example, business angels, other early stage investors, accelerators, and so on—in its selection, co-funding, and/or mentoring processes.
(c) Relevance of Implementation
42. Relevance of implementation is rated Substantial. Implementation outcomes remain relevant, although implementation arrangements can be further strengthened in the future. The emphasis placed on good international practices, efficiency, horizontality, independence, and transparency is deemed relevant for the implementation outcomes of the pilot. So much so, that the donors and the World Bank continue to leverage the IF and MOESTD for additional innovation activities under the SRITTP and C&JP. Although currently manageable, if the IF and MOESTD become solely responsible for innovation, without commensurate staffing and capacity building in the new areas of responsibility (for example, technology transfer), there could be trade-offs in terms of quality, delivery, and overall direction of the agenda. Further expansion in the IF’s implementation responsibilities ought to be accompanied by wider ownership and endorsement by key line ministries of the cross-cutting research and innovation agenda, such as ministries in charge of the finance, economy, agriculture, and so on.
3.2 Achievement of Project Development Objectives
43. Achievement of PDO is rated High. Overall, the project achieved or exceeded its stated objectives of building institutional capacity to stimulate innovative activities in the enterprise sector as explained below. This was achieved through the operationalization of the IF, piloting of financial instruments for technological development and innovation in enterprises; and encouraging selected RDIs to engage in technology transfer and commercialization, and assisting in formulating RDI sector reform policy.
44. Operationalization of the IF. As the main implementing agency, the IF was responsible for (a) successfully designing, piloting, and managing enterprise innovation programs; (b) supporting proof of concept and prototyping activities and demonstrating a strong pipeline of entrepreneurs and innovative firms in Serbia; and (c) participating in co-financing of programs and leveraging other activities organized by international organizations, financial institutions, and the private sector, all to further the innovation agenda on behalf of the GoS. The IF engaged its international strategic, operations, and regional advisors to develop its own capacity and operational procedures and provide guidance to enterprises and the GoS on enterprise innovation policy and programs.
45. Building Institutional Capacity within the IF and Beyond. An essential service provided by the IF was the mentoring provided through Enterprise Training, in addition to the specialized training programs that entrepreneurs would elect themselves. This was conducted by international advisors covering key innovation management issues, such as business formation and expansion, IP protection and commercialization, investment readiness, and early stage technology and business development. Entrepreneurs reported these trainings to be very useful, valuable, and otherwise inaccessible. This training also prepared IF staff for the advanced capacity building led by the advisory team, and especially for the international trainings in Israel, Finland, and Croatia that focused on ecosystem development, institutional, policy, early-stage entrepreneurship and innovation program design, sequencing, and evaluation. The IF today is well-regarded by its peers, and newer innovation agencies, including Macedonia and Georgia, have requested the IF to conduct training programs for their staff.
46. Supporting Evolution of the Ecosystem. The IF played a vital awareness-building role for the nascent innovation ecosystem by conducting around 22 educational and networking events for its beneficiaries and a broad set of institutional stakeholders. These activities included project launch activities across six cities in Serbia, successive launches of the Mini and Matching Grants Programs, IP and Technology Transfer Workshop, WB EDIF workshop and four info-days for the awardees (addressing project implementation and training held by the Serbian IP office), four Enterprise Training Sessions for awardees, presentation of the IF Annual Report, Finland-Israel Knowledge Economy Workshop, consultations with four major technology transfer offices on the design of the centralized TTF, industry-RDI consultations on the design of the Collaborative Grant Scheme, and project closing.
47. Leveraging of Resources by the IF. During design, the intention was for the IF be established as an entity that is both experienced at managing and capable of raising additional funds beyond this pilot. The aforementioned factors and IF’s strong delivery positioned it as an institution with significant credibility to manage the EU and other donor funds in a professional independent and efficient manner, far beyond what was originally planned at the onset of the pilot, including for the following activities:
(a) WB EDIF (€140 million). The IF led the preparation of this regional program and mobilized €28 million from the GoS.
(b) SRITTP (€6.9 million). A sequel to the current pilot, the SRITTP is financed through the 2013 EU IPA.
(c) Serbia C&JP (€89.5 million). Under C&JP, the IF can access €12 million for its operational budget, financing of new calls for the Mini and Matching Grant Programs, and technology transfer activities.
44. Piloting financial instruments. The responsible governance mechanisms established by the IF—through its 60-plus independent international peer reviewers, 5-member professional diaspora-led independent IC, efficient under-90-day selection procedures codified in its Mini and Matching Grant Manuals (GMs)—underscored the transparent merit-based selection processes it developed. The IF completed four CFPs for the Mini and Matching Grants programs by October 2013, announcing the last batch of awardees on February 14, 2014. Funding was awarded for 41 Mini Grants and 11 Matching Grants to start-ups and firms with innovative projects and high potential for commercialization. Nineteen new products and processes were launched by the beneficiary companies, over 300 high-value jobs were supported, and 51 companies were mentored through Enterprise Training.[footnoteRef:6] By closing, €5.4 million had been disbursed to awardees who expressed their gratitude to the IF and authorities for designing and managing the grant facility in a transparent manner. Several of these early-stage projects are beginning to demonstrate successes, with some receiving noteworthy coverage of their achievements in the local and international press, including in the Financial Times, the BBC, and the New York Times. At the time of ICR completion, 36 of the 38 startups funded under the Mini Grant program and all 10 of the small and medium size firms financed under the Matching Grant program were still active, demonstrating a 95 percent survival rate. In line with international good practice, this pilot project was subject to a rigorous independent evaluation and annex 5 captures both key findings on additionality of programs as well as critical recommendations for future scale up by GoS. [6: Detailed description of each sub-grant and the amounts disbursed can be found in project files. Annex 2 provides a few examples of Mini and Matching Grant Projects. ]
Box 1. Case Study of a Successful Mini Grant Program: Start-up Strawberry Energy
Strawberry Energy was a Mini Grant recipient. This Serbian startup made renewable energy sources more accessible by developing, designing, and implementing clean technologies. It developed a solar charger and then applied the expertise gained to create the ‘Strawberry Tree Mini’—a flexible and transportable mini solar charger for cell phones—with IF financing. Besides improved characteristics of power production, consumption management, and functionality of solar panels and batteries, the mini charger includes an interactive component that shares renewable energy information with end users. A finalist of the 2013 World Technology Award, Strawberry Energy has been recognized as an innovation leader in environmental and energy efficiency. Strawberry Tree was featured on Mashable among 25 top technologies every smart city should have. Following completion of the IF project, Strawberry Energy has continued development of its technology and has created the ‘Strawberry Mini Rural’, a solar charger suitable for areas with low access to electrical energy. The Accelerator Venture Fund Eleven from Bulgaria made a €100,000 investment in Strawberry Energy in 2014. In October 2014, Strawberry Energy was the only participating company from outside the United States to win the international start-up contest organized by the GreenBiz Group, held in San Francisco, California.
Table 3. Additional Achievements of Mini and Matching Grant Awardees
Activity
19
2
13
4
5
Source: IF.
45. Technology transfer and commercialization. Another activity that this pilot tackled was to encourage RDIs to engage in institutional assessments and knowledge transfer and research commercialization activities to glean insights for sector wide policy reforms. The World Bank team conducted detailed diagnostic assessments of four RDIs (classified as Category I); that is, two more RDIs than planned at entry: Institute of Physics, Belgrade (IPB), Institute of Molecular Genetics and Genetic Engineering (IMGGE), Institute of Food Technology (FINS), Novi Sad, and Institute of Medicinal Plants Research (IMPR). These assessments provided detailed recommendations (and specific action plans) for improved institutional capacity, performance management, and knowledge transfer and research commercialization practices that were the basis for customized technical support provided to these RDIs. As part of the TA, the World Bank organized trainings for a broader group of 10 RDIs (classified as Category II) to identify opportunities for institutional improvements. These institutions participated in discussions to identify cross-cutting topics of interest to RDIs, including: Program/Project Management, IP Management, R&D Marketing and Sales, Performance Evaluation and Career Planning/Development, and Managing/Developing R&D Capabilities. Based on expressed interest, the World Bank delivered three high-level training workshops on topics of interest to the RDI sector community.
46. Research Sector Reform. Given its modest means, the TA program made important contributions to the voluntary institutional adjustments and technology transfer efforts at IPB and IMGGE. These two RDIs were responsible for three of the early-stage technology transfer targets met under this project. However, supporting technology transfer and commercialization was very challenging owing to basic research orientation of most RDIs and the weak institutional capabilities, low level of technological and even lower market readiness of the proposed R&D projects. With the establishment of an Innovation Center at the IPB, technology transfer activities progress picked up during the project. IPB leadership as well as many department heads and researchers demonstrated strong interest and willingness to undertake some difficult institutional and mind set changes. The IMGGE transformed itself from a university-like organization to a research institute comparable with its international counterparts and at closing was on the verge of exporting a new product supported under the TA (see annex 3 for details). Both RDIs reported a significant change in attitudes among its management teams and researchers in engaging the private sector on potential knowledge transfer and commercialization projects. Both RDIs reported the culture as more favorable toward conducting applied research with commercialization potential by partnering with the private sector, internal reforms in performance and institutional management, and the need to look for non-budgetary sources of revenues for the future. Both RDIs were able to obtain several knowledge transfer contracts from the private sector firms and European organizations, produce patent filings and high quality publications. 
47. The management of the FINS and the IMPR—the two other RDIs supported under the detailed TA—appreciated the in-depth assessments, but found it difficult to motivate staff to proceed with the recommended institutional reforms without an explicit mandate push or support from MOESTD.
48. Overall, this World Bank-led activity provided extensive insight into the market relevance of the RDI projects and revealed the financing gap to support commercialization and conduct RDI sector reforms nationally. In October 2013, the World Bank delivered a Policy Note on RDI Sector Reform to MOESTD. In 2015, subsequent MOESTD leadership took up recommendations from the Policy Note to initiate systematic reforms in the R&D sector. The World Bank and IF teams also provided specific inputs to the ministry officials for the design of two projects: (a) the EU IPA-funded SRITTP and (b) World Bank’s C&JP. For instance, under the C&JP, the GoS is adopting a strategy that commits to undertaking reforms in the public research sector by conducting RDI-level assessments with global experts and establishing market-aligned key performance indicators that are for both individual researchers and RDIs. Both projects were approved and currently are under implementation.
3.3 Efficiency
Rating: Substantial
49. Overall, this pilot achieved its objectives efficiently and is rated Substantial. It is not useful to conduct a cost-benefit analysis for a project (a) that was attempting to pilot multiple missing elements in an innovation ecosystem; (b) that was going to trigger systemic changes; (c) that required significant scaling of financing to demonstrate impact; and (d) where the full effect of the outcomes would be seen only several years post pilot. Comparing costs of similar programs elsewhere can lead to erroneous inferences as initial circumstances vary by country. The PDO was successfully achieved with the operationalization of the IF (€1.1 million) and the project achieved better outcomes than originally envisaged: (a) leverage of at least €35 million through three donor projects versus €20 million target; (b) grant decision making in 60–90 days versus regional average of 150 days; (c) design of four financial instruments during the project versus target of three; (d) Funding of 38 active startup that launched 19 new products versus 10 and 8, respectively; (e) mentoring of 51 startups versus initial target of 10; (f) rebuilding staff capacity in spite of two waves of staff departures; and (g) being recognized by regional peers.
50. The independent quantitative evaluation (see annex 3 for analysis) delivered by the external consulting firm in October 2015, titled ‘Estimation of the Effect of Participating in the IF Financing Programs on Firm Performance,’ concluded that (a) the IF was a credible and transparent platform, and accessible to many Serbian businesses, not just a small group of entrepreneurs; (b) Mini and Matching Grant Programs, developed and piloted within the project, were found to be clear, accessible, and effective, with the application process being similar to other international programs and the evaluation and decision process being transparent and trustworthy; (c) the IF succeeded in expanding the opportunities for tech-based entrepreneurs by creating a number of innovative start-ups/spin-offs. Almost half (156 out of 326) of the firms that submitted the application for IF financing were created for them to apply for the IF operation. In addition, the evaluation found a high survival rate among supported firms. The IF succeeded in expanding the network of tech-based entrepreneurs by creating a relatively high amount of collaborations with academia and industry (also internationally); and (d) under the Mini Grants Program (€2.9 million), 40 firms were approved, of which 37 were active at the end of the project period (January 2016). Simple differences in sales changes between firms that received a Mini Grant and firms that applied but were rejected revealed that the former increased their sales by an additional €27,000 and their employment by an additional five employees, on average. Based on these numbers, on average, the additional sales generated as a result of the Mini Grants was approximately €1 million. Based on which, the investment in this component would be recouped in a period of three years.
51. The RDI TA (€0.7 million), conducted four instead of two in-depth RDI assessments planned, identified 10 RDIs for potential institutional improvements support instead of five, delivered on three technologies transferred, and brought a new commercialization culture to these RDIs. Most importantly, perhaps, learnings under this component informed the design of the SRITTP and C&JP.
3.4 Justification of Overall Outcome Rating
Rating: Satisfactory
Relevance of
Objective
Design
Implementation
Objective
High
Substantial
Substantial
High
Substantial
Satisfactory
52. The overall outcome rating is based on ratings for Relevance, Efficacy, and Efficiency as broken down in Table 4. Project Objective remains relevant and aligned with Serbia’s overall development priorities and is rated as High. Project Design and Implementation are both deemed to be relevant in the EU pre-accession context and are therefore rated as Substantial. Some delays were experienced during early stages of implementation and there was also a need to extend the closing date of the project. The delays were justified (and agreed to by the World Bank, donor, and recipient) given strategic guidance provided to the IF by its advisory team to stagger the launch of the Mini and Matching Grant Programs early during implementation to allow for learning to be incorporated in subsequent rounds. The extension facilitated the completion of all awardees’ projects, achievement of technology transfer targets with RDIs, and, most importantly, rebuilt the ministry’s commitment on innovation policy and RDI reforms. In this context, given that the Project mostly exceeded the PDO outcome targets, its Efficacy is rated as High and Efficiency was rated as Substantial. All of the aforementioned ratings lead to the Overall Outcome Rating of Satisfactory.
3.5 Overarching Themes, Other Outcomes and Impacts
(a) Poverty Impacts, Gender Aspects, and Social Development
53. This was a small pilot project aimed at stimulating innovative activities in the enterprise sector. The 51 projects financed through the pilot in the total amount of €6 million over four years were too small to have an impact on poverty, gender aspects, or social development. However, the increased income and employment generated by the pilot program have given an indication that, if scaled up substantially, the program could have a positive impact on poverty reduction and economic development (see annex 3).
54. Women entrepreneurs. The IF programs supported a few remarkable women researcher become entrepreneurs who were even featured in the international media. They have become an inspiration for younger women entrepreneurs. It would be important to strengthen gender emphasis in future programming given the very specific challenges Serbian women entrepreneurs face in entering the market, accessing both debt and equity finance, and extensive harassment while running a business.
(b) Institutional Change/Strengthening
55. The project has had a significant impact with respect to institutional changes, especially with respect to the IF, but also the management teams and researchers at RDIs supported under the TA program, as well as MOESTD, as discussed above.
(c) Other Unintended Outcomes and Impacts (positive or negative)
56. At entry, there were few players of any significance in Serbia’s innovation and entrepreneurship ecosystem, with the exception of incubators in Belgrade and Novi Sad. By project closing multiple public and private actors were active in Serbia, including business angels, accelerators, incubators, technology transfer offices, science parks, and so on. As the project did not actively finance ecosystem development, causality cannot be inferred between the project and its evolution. However, stakeholders and beneficiaries have credited the IF and its programming for shining a spotlight on Serbia’s early-stage entrepreneurial pipeline, attracting regional investors, highlighting its policy efforts to promote research and enterprise innovation.
3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops
57. Based on the interviews carried out with the beneficiaries as part of the M&E process, the M&E consultants and World Bank staff concluded that (a) overall project objectives were adequately specified; (b) the Mini and Matching Grants Programs were clear, accessible, and effective; (c) program evaluation and decision process was transparent and trustworthy; (d) application process was similar to other programs (internationally); (e) the IF staff ranked high on professionalism and willingness to help, and were flexible; (f) implementation was effective—most of the projects achieved their objectives; (g) RDI management valued the changes in researcher behavior toward technology transfer; and (h) the Government takes ownership of R&D sector reform agenda.
4. Assessment of Risk to Development Outcome
Rating: Negligible
58. The risk to development outcome is considered Negligible. All activities are either completed or sustainable under follow-on projects. Although there was a change of Government during project implementation, the risk that the program would be derailed did not materialize. Sustainability of the pilot’s achievements has been further enhanced by the Government and the World Bank’s commitment to further support the PDO through the SRITTP and C&JP.
5. Assessment of Bank and Recipient Performance
5.1 Bank Performance
Rating: Satisfactory
59. The World Bank’s performance for quality at entry was rated Satisfactory. The World Bank heavily leveraged top-notch global expertise and lessons learned from multiple post-transition economies to recast the innovation policy discourse to focus on the role of the Government and entrepreneurship in building that national innovation system. Additionally, it emphasized the importance of piloting and stronger than usual M&E practices for this institutional capacity-building project.
(b) Quality of Supervision
Rating: Highly Satisfactory
60. Quality of supervision was far more intense than a regular World Bank project and very meticulous given the pilot nature of the project and first time relationship with the donor. Supervision missions were carried out at least twice a year and up to five times in the early stages of the project owing to the flexibility afforded by the World Bank-executed TA component. The World Bank team ensured continuity in project management and a strong thematic and operational skills mix with strong support from staff in the local office. The World Bank-executed component enabled mobilization of leading global practitioners in their respective fields, thus earning the trust and respect of the research community. The team was proactive and responsive to the multiple GoS requests for change during project implementation and demonstrated flexibility and understanding in the use of project resources. The World Bank led project restructuring in consultation with the donor and recipient to maximize achievement of the PDO. On fiduciary requirements, the performance was Highly Satisfactory with FM, procurement, and environment activities under the project implemented effectively with significant capacity building and no major issues.
(c) Justification of Rating for Overall Bank Performance
Rating: Satisfactory
61. Overall, the World Bank was very proactive and responsive to the client’s needs from the design stage through to project closing, with close supervision and excellent support from the country office. The World Bank team also played a very important role in establishing the IF’s capacity by offering its own experience and guidance. Based on the above, the overall World Bank performance in ensuring quality at entry and quality of supervision is rated Satisfactory.
5.2 Recipient Performance
(a) Government Performance
Rating: Moderately Satisfactory
62. The recipient worked closely with the World Bank during project preparation in a satisfactory manner. During implementation, the recipient could have been more proactive in providing support to the IF (especially its operating budget), and introducing some quick win policy recommendations resulting from the RDI TA program. Although the Recipient complied with all covenants under the Grant Agreement and development outcomes were met by the project, the shortcomings in budget support to the IF slowed down implementation progress in early 2014 and destabilized the IF’s operations. Therefore, the overall performance of the Recipient is rated as Moderately Satisfactory.
(b) Implementing Agency or Agencies Performance
Rating: Highly Satisfactory
63. The IF as the implementing arm of the Recipient was very effective in providing efficient implementation, monitoring, and reporting, allowing for better management and implementation of the project and facilitating the World Bank’s supervision activities. The IF was staffed with qualified experts, selected through a competitive process and adequately compensated for their work. Although there were teething problems such as the absence of a permanent managing director and lack of timely GoS budget support on multiple occasions, the IF staff have done a remarkable job in implementing the project and achieving the PDOs. The IF staff should be commended for the professionalism exhibited in the discharge of their duties and the performance is rated Highly Satisfactory.
(c) Justification of Rating for Overall Recipient Performance
Rating: Satisfactory
64. The overall Recipient performance is rated Satisfactory. The success of the project was to a large extent due to the efficiency and professionalism demonstrated by IF staff in the implementation of the program and they should be highly commended. The Recipient, although committed, could have been more proactive in providing budget and policy support to the IF, RDIs, and the sector overall.
6. Lessons Learned
65. Governance arrangements are key for institutional integrity and program delivery. Governance mechanisms established at the IF were underscored by the transparent selection process that placed a premium on merit and commercialization, and integrated feedback from international peer reviewers, decision making by an independent IC and continuous guidance from a professional global advisory team. This positioned the IF as an institution with significant credibility and capacity to absorb and distribute EU funds in a thorough and independent manner. This aspect was confirmed by beneficiaries, who commended the transparent and independent process and the mentoring received from peer reviewers.
66. Institutional capacity matters for delivery of innovation programming. A well-functioning national innovation agency, staffed with competent and committed people and fairly remunerated is essential for implementation of a complex project, such as this, and is an important element of the national innovation system. The IF is an institution that is well regarded by its beneficiaries and its peers, and should be nurtured to continue to serve the country and remain agile in an ever changing global innovation landscape. This will mean additional rounds of capacity building and staff renewal to meet new client and market demands. This should remain part of the IF’s strategic planning efforts.
67. Horizontality of innovation policy and programming matters in nascent ecosystems. One of the principles of design considered important in nascent post-transition innovation ecosystems—dominated by entrenched state enterprise and RDIs—is horizontality of the innovation policy and interventions, that is, sector agnosticism. At entry, many assumed that the deal flow would be limited to innovative projects in ICT and digital technology from Novi Sad and Belgrade. The pilot—by adhering to horizontal selection criteria that emphasized merit and commercialization potential—revealed a diverse pipeline in terms of sectors, cities, and gender.
68. Size of innovation effort matters and so do rhythm and scale. The 51 projects the IF financed over four years was too small a number to have an economy-wide impact, but manageable for capacity building. However, to move the needle at an economy-wide level, the independent evaluation estimates that the IF would need to provide funding to 200–300 projects annually. Having established the IF and piloted instruments that demonstrate additionality for enterprise innovation, the GoS should ensure continuous uninterrupted support to retain IF capacity, generate innovative entrepreneurs, rebalance the enterprise innovation system, and continue to position Serbia as a budding knowledge and IP creator in the global innovation landscape. Losing the momentum generated would be ill advised and create significant loss of trust in public institutions and could undermine future efforts.
69. Visibility of innovation activities really matters. World Bank operations could take a serious lesson from the EU’s approach and benefit from a more systematic approach to providing visibility to project activities with stakeholders, sharing lessons learned, and celebrating successes. The Project Opening Ceremony engaged the IF’s advisory and management teams, and its IC to showcase the capacity engaged and discuss intentions transparently, thus building credibility among the sceptical research and entrepreneurial communities. The Closing Ceremony—a celebration of the journey the Serbian ecosystem had made over the course of the project—displayed the institutional capacity of the IF and RDIs, the remarkable 50 plus entrepreneurs who demonstrated their firms’ prototypes and viable products, as well as the investor groups and entrepreneurship support institutions that did not exist at the onset of the project. The unrehearsed testimonials provided by these ecosystem actors during visibility events were picked up by policy makers and donors and are informing future decisions on financing.
70. Support for the evolution of an innovation ecosystem is critical. In the early days of the project, the IF was a lonely advocate for the ecosystem and would often come under undue scrutiny and criticism. Eventually the strong entrepreneurial pipeline attracted regional partners and global investors, making it easier to advocate for support to other actors and promote long-term symbiotic relationships in the ecosystem.
71. The policy world is changing fast. The innovation arena is witnessing an increase in well-networked private sector investors and support institutions globally, including in incubation, acceleration, and financing. Policy makers are beginning to review policies and adopt legislation to enable crowd funding, which could further democratize early-stage innovation finance. While not an immediate concern for any of the post-pilot operations, the GoS and IF should familiarize themselves with these developments and policy implications to ensure relevance of future design and implementation.
72. Flexibility during design and implementation are key for pilots. During design and especially during implementation of this pilot, the World Bank team worked closely with the IF to monitor project progress. When issues were encountered, the teams worked quickly to find solutions. This flexibility allowed for smoother and efficient implementation. A few examples include (a) design of the mentoring program to accommodate mandatory and customized training elements for IF staff and beneficiary entrepreneurs to be more useful; (b) deploying the Mini and Matching Grant programs sequentially so the IF could incorporate lessons from the first call of the Mini Grant Program for subsequent programs; (c) increasing the private sector contribution (from 25 percent to 30 percent) under the Matching Grant program; (d) converging on design details for the royalty scheme; and (e) engaging commercialization brokers to work with RDIs to identify commercialization wins within the lifetime of the project.
73. Incentives Matters: Entrepreneurs must have “Skin in the Game”. The requirement for the beneficiary entrepreneurs to provide a match in case only (not in kind) and the success royalty component weeded out several serial ‘grant writers’ during the project. This scenario is known from international practice and hence influenced project design and played out in Serbia.
74. Building relationships with development partners takes times but is worthwhile. This was the first time in Serbia that a Trust Fund Agreement was forged between the Bank and the EU Delegation. As both organizations were unfamiliar with policies and procedures of their counterparts, this led to some delays during project preparation. However, once both organizations were comfortable with each other’s policies and procedures, project implementation progressed smoothly. It would be prudent that for future joint collaborations to allow some lead time for project preparation so that any procedural issues could be sorted out and not delay project preparation. Additionally, the Bank team ensured that it met with the EUD counterparts during each mission to allow maximum information sharing on project progress and any challenges that could be jointly addressed in bilateral conversations with the GoS.
7. Comments on Issues Raised by Recipient/Implementing Agencies/Partners
(a) Recipient/implementing agencies: please see Annex 7 for report and statement from IF.
(b) Co-financiers: please see Annex 8 for statement from EUD.
(c) Other partners and stakeholders: Please see Annex 8 from key partner: Belgrade Incubator
Annex 1. Project Costs and Financing
(a) Project Costs by Component (in Euro)
Appraisal Estimate
3.21
1.11
2.10
35
6.00
6.00
0.0
100
Component 3. Provision of Technical Assistance to Research and Development Institutes (RDIs)
0.70
0.70
0.0
100
0.59
3.82
1.06
2.76
28
6.51
5.41
1.1**
83
Component 3. Provision of Technical Assistance to Research and Development Institutes (RDIs)
0.80
0.80
0
100
0.59
7.86***
Note: *Government financing covering period from 2011–2015 (funds are expressed in € using middle exchange rate of National Bank of Serbia on the last day of each year). Initially, project closing date was set on November 30, 2014. However, it was extended to January 10, 2016, which caused increased Government financing (Government financing in 2015 equals around €534,000).
**Contributions from the Mini and Matching Grant private sector enterprises were at least €1.1 million.
*** EURO 0.655 million was cancelled.
(b) Financing
Annex 2. Outputs by Component
Component 1: Capacity Building of the Serbia Innovation Fund - Recipient Executed
1. The IF was to be established with the objective of encouraging entrepreneurship and managing financing for innovation, as well as participating in co-financing of programs, projects, and other activities organized by international organizations, financial institutions, and the private sector. This has been fully achieved. Under this component, the IF developed the requisite capacity for designing and managing grant instruments geared toward technology-based firms. Importantly, the IF has built and retained an impressive international team, including strategic, operations, and regional advisors, in addition to strong governance practices espoused by its international diaspora-based independent IC.
2. The major networking and educational events held by the IF were project launch and launch of the Mini Grants Program, followed by the launch of the Matching Grants Program, the IP workshop (for RDIs and key stakeholders), WB EDIF workshop (for RDIs and private sector and key government stakeholders), and four info-days for the awardees (addressing project implementation and training held by Serbian IP office), presentation of the IF Annual Report, knowledge economy workshop, industry and RDI consultation regarding the new technology transfer project design, consultations with local Technology Transfer Offices on the cooperation and functioning of centralized TTF.
3. The IF also provided several training programs for its staff and beneficiary enterprises. One of the essential services the IF provided to its grant awardees was Enterprise Training, which was conducted by international advisors covering topics of innovation, business formation and expansion, IP protection and commercialization, investment readiness, and other valuable themes associated with early-stage technology and business development. As part of its program, the IF also conducted several project visibility events and workshops. Overall, this component is considered highly satisfactory for the establishment of the IF, with its remarkable governance and the institutional capacity.
4. Staff training for the IF. The following training courses were attended by IF staff.
· Preparation and management of the World Bank-funded projects (World Bank office, Belgrade). IF staff received training on all aspects on the preparation and implementation of World Bank-funded projects.
· Management of innovation and technology support programs (at BICRO, Croatia). Training provided at BICRO was expected to introduce IF staff to BICRO programs and organization, management of calls for financing, due diligence, M&E. Introduction to programs for supporting operation of incubators and technology transfer programs was also provided.
· FM (Turin, Italy). Training was provided to the financial manager to design and operate the FM systems and execute corresponding payments/disbursements in accordance with sound professional standards and the harmonized requirements of donors and their respective governments.
· Venture capital (VC, Tel Aviv, Israel). Training was provided to acquire critical skills in five key areas: (a) interpreting governance diagnostics and political economy analyses, (b) creating multi-stakeholder coalition building strategies and tactics to support reform, (c) providing communication skills that support the implementation on governance reforms, (d) leveraging social/digital media tools and analytics effectively, and (e) developing communication metrics and applying M&E frameworks. Training was attended by the IF’s interim managing director.
· VC (Tel Aviv, Israel - two times for staff). Training consisted of introduction to organization, operations, and principles of private sector VC funds, transaction, and due diligence process for early-stage financing. Training was useful for the IF managing director and program managers to structure and implement privately managed/publically funded equity instruments and to efficiently connect Serbian companies with private sector VC funds.
· World Bank procurement and FM procedures (Sarajevo, Bosnia and Herzegovina). Introduction to the World Bank procurement and FM procedures and guidelines was provided. Training was attended by the procurement associate and financial manager.
· Innovation funding programs (Helsinki, Finland). Tailor-made training for the IF managing director and program managers at TEKES and SITRA in Finland was provided to get an overview of financing mechanisms for private sector companies and enterprise R&D as well as fund management in Finland; applications evaluation; monitoring, funds management; and programs for large companies.
· Program management (Turin, Italy). Two-week training on project and program cycle management based on the logical framework approach, including project identification, stakeholder analysis, project design, and the development of M&E systems was provided. Training was attended by the IF’s senior associate.
· Represent. The IF’s public relations (PR) person and a person in charge of managing the social network pages (LinkedIn and Facebook) have received training to provide knowledge on the most important aspects of presenting the institution through social network profiles/pages.
· Training on EU PRAG procedures. IF employees have attended the tailor-made training designed to provide better understanding of the PRAG procedures that was required in the implementation of the projects financed by IPA funds.
· PR training. IF employees received PR training from the World Bank representative (Belgrade office) in Belgrade.
5. A critical ingredient for the success of the project was that a responsible governance mechanism was established at the IF, underscored by the transparent selection process that placed a premium on merit and commercialization potential of projects, and included inputs from international peer reviewers and decision making by an independent IC. This has positioned the IF as an institution with significant credibility and capacity to absorb and distribute EU funds in a professional and independent manner.
Component 2: Implementation of Financial Instruments Supporting Enterprise Innovation - Recipient Executed
6. The IF started its operation on December 5, 2011, as Serbia’s main agency for innovation support through two main programs: the Mini Grants Program and the Matching Grants Program.
7. The purpose of the Mini Grants Program was to expand opportunities for tech-based entrepreneurs by stimulating commercialization of R&D and the creation of innovative enterprises based on knowledge through start-ups and/or spin-offs. The Matching Grants Program aimed to expand collaboration opportunities for innovative micro and small companies with strategic partners (for example, private sector industry, R&D organizations, and VC/private equity funds) with the goal of increasing private sector investment in R&D and the commercialization of projects promoting new and improved products and services.
8. The financing offered by the Mini Grants Program covered a maximum of 85 percent of the total approved project budget and up to €80,000 for a one-year project. The reminder of the total approved project budget was to be secured by the applicant from other, preferably private sector sources, independent of the IF.
9. The financing offered by the Matching Grants Program covered a maximum of 70 percent of the total approved project budget and up to €300,000 for a two-year project. The remainder of the total approved project Budget was to be secured by the applicant from other sources independent of the IF such as private sector industry, private investors/VC/private equity funds or the applicant’s own internal resources.[footnoteRef:7] [7: Upon successful technology commercialization, the revenue (and not just the profits) derived from the sales of the product/service and any or subsequent products/service based on the technology developed in the Matching project financed by the IF, become the basis for a royalty payment in compensation for the IF financing, at a rate of 5 percent of annual revenue derived from sales of product or service or 15 percent of licensing revenue derived from such product/service up to 120 percent of financing received from the IF for a period of up to five years after project completion, whichever is achieved first.]
10. Financing decisions were made on a competitive basis by the IF’s independent IC. The number of awards was determined by the quality of the proposals and subject to the total funds available (allocated) to the programs. In addition, these programs provided support for FM, IP protection, and business development training, as well as mentoring of the awardees to enable them to prepare for the next stage of development.
11. Between January 31, 2012 and October 28, 2013, the IF managed four financing calls during which 326 Serbian enterprises submitted applications for either the Mini or Matching Grants Programs. The IF held about 20 visibility events per call to promote the CFPs and to explain the Mini and Matching Grants Programs. Some 91 firms applied to the Matching Grants Program, while 247 firms applied to the Mini Grants Program (12 firms applied to both the Matching and Mini Grant Programs). Overall, 471 projects were submitted by the 326 firms. A total of 55 projects were approved for IF support, out of which 52 projects were financed (out of the 471 projects); a 12 percent approval rate. Details of the selected