European Construction Technology Platform (ECTP) Presentation
Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... ·...
Transcript of Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... ·...
![Page 1: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/1.jpg)
Impact Assessment E2B
Govert Gijsbers -TNO
![Page 2: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/2.jpg)
2
Overview Workshop
Programme
• 10.00-10.45 Introductory presentation
• 10.45-11.15 Discussion
• 11.15-11.30 Break
• 11.30-12.30 Towards indicators: Generating
ideas and indicators on long-term impact,
outcomes and outputs
• 12.30-12.45 Wrap-up
![Page 3: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/3.jpg)
3
Introductory Presentation
• An impact case
• Some methodological issues
• The impact assessment process
![Page 4: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/4.jpg)
I. Mini case study
Dr. Pasveer’s Ditch
![Page 5: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/5.jpg)
5
The “Pasveer sloot”
1950s: Sewage treatment was expensive and only available for larger
cities, no cost-effective solution for smaller towns existed.
Dr. Pasveer at TNO set out to develop a low cost solution, mimicking
natural processes in rivers, in a closed system:
• An „oval‟ ditch with an inlet for sewage and an outlet for treated water
• A rotor and brushes to circulate the water and to add oxygen
• Storage and treatment for sludge
• First pilot at TNO in Delft
![Page 6: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/6.jpg)
6
1954: The cutting edge of environmental technology!
- Fully operational plant (capacity 600 user equivalents)
- Excellent technical performance: much better than existing systems
- Cheap: 50% lower cost than existing technology
![Page 7: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/7.jpg)
7
Scaling up and scaling out
- 400,000 user equivalents!
- World-wide adoption
![Page 8: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/8.jpg)
8
1990s: Disadoption -Now an industrial monument
- Stricter norms for emissions (fully aerobic process, energy-
intensive, CO2 emissions)
- Huge surface area required for large scale treatment facilities
- But not the end of the story…
![Page 9: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/9.jpg)
9
Technical improvements: Carousel
model
• Underlying technology is the same
• Used globally today
![Page 10: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/10.jpg)
10
Huge impact – with hindsight (1)
• Initially hard to recognize as a breakthrough technology
• Long term perspective: 1954-1991 (NL)
– still in use in other countries, and the basis for current
technology
• TNO didn‟t commercialize it – never saw a penny in
return
• Dutch engineering companies built systems all over the
world and profited handsomely
• Is that a problem? Public money, public goods?
![Page 11: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/11.jpg)
11
Huge impact – with hindsight (2)
• Huge benefits; different types: financial, environmental,
public health
• How to measure?
– Worldwide spillovers
– Impact on subsequent technology (carousel model)
• Finally: Who cares?
– Then was then and now is now!
– Past performance not a criterion for future funding
– Key issue: IA can only be done after the fact, but then
stakeholders (policy makers) have lost interest.
![Page 12: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/12.jpg)
12
A few lessons
• Impact may take a long time to emerge
– but it may also persist for a long time
• Different types benefits/impacts
– financial, environmental, public health
• Technology adoption can be world-wide
• More actors involved: R&D, companies:
– attribution
• Subsequent technical improvements
![Page 13: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/13.jpg)
II. Some methodological
issues
![Page 14: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/14.jpg)
14
Why Impact Assessment?
• R&D / innovation programmes challenged to
demonstrate their impact on clients, stakeholders and
society
• Two main reasons for impact assessment (IA)
– Accountability (external)
– Learning (internal)
• Much talk about IA, but not much practical work
– Because IA is a difficult issue
• Challenge to develop a workable system
![Page 15: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/15.jpg)
15
Eight issues in IA
1. Level of analysis– Impact of what?
2. Level of analysis– Impact on what?
3. Attribution
4. Time lags
5. Selection bias
6. Non-linearity
7. The problem of the counterfactual
8. Measurement issues
![Page 16: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/16.jpg)
16
1. Level of analysis – Impact of
what?
– Projects, Programs, Technologies, Business Units,
Departments, Organisations?
– Is the impact of the whole greater than the sum of the
parts?
![Page 17: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/17.jpg)
17
2. Level of analysis – Impact on
what?
– Direct outcomes for clients
– Indirect outcomes for clients and stakeholders
(intended and unintended)
– Impact on society at large
![Page 18: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/18.jpg)
18
3. Attribution• Attribution (of impact) a key problem in IA
• With every step away from the direct outputs produced by
an organization or program, the question of attribution
becomes more pertinent
![Page 19: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/19.jpg)
19
The Attribution Gap
![Page 20: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/20.jpg)
20
“Contribution analysis”(to deal with attribution problems)
“Contribution analysis focuses explicitly on the issues of
attribution and contribution and aims to present a
credible story of how a program or project contributes
to solving a problem.” (John Mayne, Canadian Auditor
General Office )
• “Plausible association”: whether “a reasonable person,
knowing what has occurred in the program and that the
intended outcomes actually occurred, agrees that the
program contributed to those outcomes?”
![Page 21: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/21.jpg)
21
4. Time lags
• New technologies may take decades to impact on
society
• Looking back 25 years at past achievements is no longer
of much policy interest
![Page 22: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/22.jpg)
22
5. Selection bias
• “Unsuccessful” projects not well documented
• Needs an estimate of the distribution of project success
rates (representative portfolio)
• Alternatively, focus on a few “blockbusters”
– R&D inherently an uncertain business, where many
project must fail
– a few mega successes may repay the cost of R&D
many times over
![Page 23: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/23.jpg)
23
6. Non-linearity
• The impact chain presents a linear model
– a bit like the linear model of innovation
• But we know that impact is often achieved in roundabout
and unexpected ways
![Page 24: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/24.jpg)
24
7. The problem of the
counterfactual
• We don‟t know what would have happened
anyway (without the project / program in
place)
• With / without
• Control groups?
• (Behavioural) additionality
![Page 25: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/25.jpg)
25
8. Measurement issues
• Quantitative and qualitative approaches– from econometric models to story telling
• Quantification limited by conceptual and data problems
• R&D produces “knowledge”, which is hard to measure, but is more than the number of patents.
• Need to develop indicators that can be measured
• Start with Identification of all outputs (not a trivial task for a major programme such as E2B)
• Numbers supported by qualitative stories
![Page 26: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/26.jpg)
III. Developing an IA process
![Page 27: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/27.jpg)
27
Key steps
1. Define IA requirements
• with key stakeholders
2. Build an E2B “logic model”
– inputs, outputs, outcomes, impact
3. Measurement:
– ID indicators for outcomes
4. Implementation
• responsibilities, etc.
![Page 28: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/28.jpg)
28
1. Impact assessment system
requirements E2B
• Provides relevant information on E2B‟s performance (outputs, outcomes and impact) to different users
• Is accepted by E2B funding agencies as a credible set of instruments
– this is the political aspect
• Is cost-effective in information collection and maintenance
• Strong links to E2B mission and strategic objectives
• Builds on best practice from public and private sector, nationally and internationally.
![Page 29: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/29.jpg)
29
2. Build a “logic model”
• A logic model connects the inputs and outputs of
a project or programme to intermediate and final
outcomes (impact)
• A.k.a.
– program logic
– intervention logic
– outcomes model
– theory of action
– impact pathway
– mental model of the program
![Page 30: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/30.jpg)
30
InputsActivities/Process
Intermediate
Outcomes
Products /Outputs
• Finance
• People
• Infrastructure
• R&D and
innovation
projects
• Technologies
• Publications
• Business
models
• Prototypes
• Services
• Technologies
/ materials
adopted
• Houses
insulated /
energy neutral
• Construction
processes
adopted
• Reduced GHG
emission
Program logic - general
Final
outcomes /Impact
• Incomes
• Employment
• Environment
• Climate
change
• Public health
• Cultural
heritage
![Page 31: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/31.jpg)
31
Example: Overview model
![Page 32: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/32.jpg)
32
Sub model - training
![Page 33: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/33.jpg)
33
Logic model
![Page 34: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/34.jpg)
34
3. Develop indicators
• Start with outputs
– basically counting, but…
– many R&D organisations/ programs do record all
outputs
• Add indicators to outcomes
– (to be) collected by program, or
– collected by others (statistics offices)
• ID “contributions” to aggregate societal/economic impact
• Key point: indicator development comes at the end, not
the beginning
![Page 35: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/35.jpg)
35
4. Implementation
• Assign responsibilities to project partners
• Set up (annual) monitoring, data collection
– primary, secondary
• Complement quantitative indicator-based
information with in-depth case studies on
specific topics
• Communication of impact to funders,
clients, stakeholders
![Page 36: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/36.jpg)
36
Overview Workshop
Programme
• 10.00-10.45 Introductory presentation
• 10.45-11.15 Discussion
• 11.15-11.30 Break
• 11.30-12.30 Towards indicators: Generating
ideas and indicators on long-term impact,
outcomes and outputs
• 12.30-12.45 Wrap-up
![Page 37: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/37.jpg)
E2B expected impacts
![Page 38: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/38.jpg)
38
Impacts: Energy
• New zero energy, energy+ buildings
• Renovated energy efficient buildings
• Increased RES installments
• Reduced energy dependence
– reduced energy consumption
![Page 39: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/39.jpg)
39
Outcomes: environmental
• Reduce CO2 emissions
– 65 Mt /yr
• Low energy content materials
• Reduction of waste, 75% recycling
• Municipal solid wast reduction
– 26 Mtoe/yr
![Page 40: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/40.jpg)
40
Outcomes: social
• 90-150k jobs
• Improved indoor climate:
– CO2 content 400 ppm, humidity 30-50%
• Sound climate (noise)
• Lighting comfort & reduced energy
consumption (using ICT)
• Accessibility
![Page 41: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/41.jpg)
41
Outcomes: Economic
• Investment in efficient & clean technologie
of 12-17,000 M €/yr
• Cost reduction of 10-15% on above
investment
• Shorten time to market
• Increased labor productivity
• Cost savings: lighting, heating, water
– 126-150,000 M€
![Page 42: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/42.jpg)
42
Outputs, outcomes and impact
![Page 43: Impact Assessment E2B - ECTPectp.ectp.org/cws/params/ectp/download_files/37D1464v1_ExecBoard6... · project must fail –a few mega successes may repay the cost of R&D many times](https://reader033.fdocuments.in/reader033/viewer/2022041703/5e42dfc948c45127ea7e34fe/html5/thumbnails/43.jpg)
43
“Contribution analysis”
Contribution Analysis: Addressing Attribution with
Performance Measures
• Acknowledge the attribution problem
• Present the logic of the program – “theory of
action”, “impact pathways”
• Identify and document behavioral changes (clients
or beneficiaries)
• Use discriminating indicators.
• Track performance over time.
• Discuss, and test alternative explanations.
• Gather additional relevant evidence.
• Gather multiple lines of evidence.