Post on 06-Jan-2016
description
1
The Use of Surveys in Determining Outcomes of Research, Technology and Development ProgramsPresented at: Joint CES / AEA
Conference
Presented by: Suzanne Lafortune, PartnerPerformance Management Network Inc.
Date: October 26, 2005
Tel: (613) 236-2320Fax: (613) 236-8644Email:
suzanne.lafortune@pmn.net
Results & Resources versus Reach
Source: Steve Montague, The Three Rs of Performance: Core concepts for planning, measurement, and management, Performance Management Network Inc., 1997, p 120.
The ‘clustering’ of services or activities by results, resources
per user, and reach can provide new perspective.Intensive Problem
Solving / R&D
Technical Specialist Support
Education
Information / Advice
Awareness Building
Reach
Results+
Resources per user
education and communication(workshops, speaking, trade shows)
routine advisorydissemination of voluntary codes
mass media
Information products
policy developmentprogram fundingenforcement and prosecution
specialized advisoryinvestigationsinspections and monitoring
www.pmn.net
Frequency and Type of Measurement
Reach
Results & Resources versus Reach
Results
+
Resources
per user
Intensive Problem Solving / R&D
Technical Specialist Support
Education
Information / Advice
Awareness Building
Source: Steve Montague, The Three Rs of Performance: Core concepts for planning measurement and management, Performance Management Network Inc., 1997, p.20
Methods
Ongoing Form. Eval Sum. Eval
Documents
Databases
Interviews
Surveys**
Case studies
All
Documents
Databases
Surveys*
Interviews
Surveys**
Case studies
All
Documents
Databases
Surveys*
Interviews
Surveys**
Documents
Databases
Surveys*
Interviews
Surveys**
Documents
Databases
Surveys*
Interviews
3
* Project related surveys
** Periodic surveys related to more in-depth impacts
4
Why Use Surveys for Formative / Early Evaluations of R&D
Programs?
Relevance of program design to R&D needs Satisfaction with ways in which program is
designed and delivered Eligibility criteria Application process Use of advisory boards Deadlines Time of year (academic community versus private
sector) Speed of decision making Feedback / advice / staff communications and support
before project / in early phases of project Technical competence / expertise of staff
5
Why Use Surveys for Formative / Early Evaluations of R&D
Programs?
Early outcomes – before project Awareness building Strategic alliances with partners Leveraging
Early outcomes – during project Active participation of partners Knowledge gained Job creation / maintenance within organization during project
Early outcomes – at the very end of the project Improved technical capability New / improved product / process / technology / service
Project incrementality Important to measure early on, particularly for longer term R&D
projects
6
Why Use Surveys for Summative Evaluations of R&D
Programs?
Relevance of program implementation to R&D needs
Satisfaction with ways in which program was implemented Feedback / advice / staff communications and support Technical competence / expertise of staff Reporting requirements Payment process
7
Why Use Surveys for Summative Evaluations of R&D
Programs?
Intermediate outcomes Impact on recipient organizations
Revenues / sales Productivity Sustained / long term jobs Competitiveness / increased market share Increased profitability Cost savings Exports / new client base Diversification Expansion Etc.
Impact on industry Spin offs Technology transfer
8
Why Use Surveys for Summative Evaluations of R&D
Programs?
Long term outcomes Impact on the economy / on society
Socio-economic benefits– Reduced consumer costs
– Employment
– Improved quality of life
– Etc.
Environmental benefits Improved energy efficiency Health and safety benefits Etc.
9
Types of Target Groups to Survey for R&D Evaluations
Non-recipients Formative evaluation – relevance to needs, satisfaction,
incrementality Summative evaluation – relevance to needs, incrementality
Recipients Project partners
Formative evaluation – impact of program on participation, early project outcomes / results on partner organizations
Summative evaluation – extent of participation, long-term collaborations, other outcomes / results on partner organizations
Beneficiaries Summative evaluation – impacts of projects on beneficiaries