S T P I

Where are We:“The Science of Science Policy”

Bhavya Lal, Core Staff Member

Metrics, Performance Measurement, and Evaluation

Science and Technology Policy Institute (STPI)1899 Pennsylvania Avenue NW

Washington DC 20006

blal@ida.org

American Evaluation Association

Portland Oregon

November 4, 2006

This presentation reflects the author’s views and not those of the Science and Technology Policy Institute nor any other group or organization with whom the author works

2

S T P I

Overview

• What are we trying to do?

• Why is it important?

• Why is it hard?

• What do we know already?

• What needs to be done next?

S T P I

What are we trying to do?

4

S T P I

Background

• Science policy discussions are often dominated by advocates for particular scientific fields or missions

• Policy decisions are frequently based upon past practice, beliefs or data trends that may be out of date or have limited relevance to the current situation

• We do not have the capacity to predict how best to make and manage future investments so as to exploit the most promising and important opportunities

5

S T P I

An Intellectual Call to Arms“Are we funding all the R&D we need to defend ourselves, improve and sustain our quality of life, and compete with other nations in a globalized high-technology economy?...

How much should a nation spend on science? What kind of science? How much from private versus public sectors? Does demand for funding by potential science performers imply a shortage of funding or a surfeit of performers?...

…We need econometric models that encompass enough variables in a sufficient number of countries to

produce reasonable simulations of the effect of specific policy choices.”

John Marburger, Director Office of Science and Technology Policy

Executive Office of the PresidentApril – May 2005

S T P I

Why is it important?

7

S T P I

Not Just Rhetorical Questions• Justify public R&D investments – not taken for granted

– S&T investments are proposed as the solution to all national problems – from Sputnik to Chinindia

• Help create frameworks for discussion and prioritization for future R&D investments in times of scarce resources and increasing global competition– Fear that other countries are doing better

than we are

• Overall, help understand linkages between inputs, outputs and enabling conditions – Identify levers other than more

funding for interventions

But what, apart from the roads, the sewers, the

medicine, the Forum, the theater, education,

public order, irrigation, the fresh-water system and public baths... what have the Romans ever

done for us?(and the wine, don’t forget the wine…)

Monty Python film “Life of Brian”

S T P I

Why is it hard?

9

S T P I

Lack of Theory and Frameworks

• Modeling the innovation system is a challenge– Federal investment in research only partial component of and

not always the first step in a non-sequential, complex, tightly interconnected innovation system

• few metrics to characterize the innovation system

• it might underestimate the impact of publicly funded research

– Unless other conditions in place and measured appropriately, cannot attribute outcomes to R&D funding alone

• role of regulatory and legal systems (e.g. R&D tax credits, Bayh-Dole) needs to be better understood

– International dimension – especially for modeling workforce - needs to be much more explicitly considered

Insights from evaluation, but piecemeal

10

S T P I

Data Limitations• No reliable way to define and incorporate R&D that

contributes to growth– Studies tend to use only one source of R&D, but likely some

fraction of other R&D (e.g. industry, national laboratories, venture capital) may also contribute

• No reliable mechanism for disaggregating funding of research by field– Current Federal databases are inadequate– Agencies’ own funding databases are not complete, or cross-

comparable

• No reliable mechanism for attributing ideas (whether publications or papers) to funding source or agency– Limitations of attribution have been identified

11

S T P I

Loss of Voice of the Science Policy Analyst

• Tenuous link between policy makers and policy analysts– Insights, even when they exist, may not find

their way into the decision-making process

S T P I

What do we know (or have we done) already?

13

S T P I

Summary of Efforts To-Date

• Various versions and pieces of the questions have been addressed since the 1950s primarily through– (not discussed – retrospective evaluations)– Econometric/large scale studies – Case studies

14

S T P I

Efforts To-Date: Econometric Studies

• Both cross-sectional and time-series and at levels of aggregation ranging from the firm to the nation

• Focus on calculating returns to investment in publicly funded R&D

• Methodological and data constraints limit the applicability of current econometric techniques– times-series models don’t mesh with cross-sectional models– spillover benefits typically not captured– most measure the contribution of private sector R&D and

especially in the manufacturing rather than the service sector– all assume a sequential, primarily uni-directional flows from

research to commercialization– more recent studies do not give returns but rather marginal

returns, or stop short of a numerical value

15

S T P I

Example: Generalized Econometric Model

Inputs to science(R&D, educationalFunding; capital

infrastructure)

Aggregated/disaggregated by sector, field, performer

Total factor productivity (Griliches school)

Final outcomes (TFP growth, R&D

ROI)

Intermediateoutcomes

(workforce, Scientific discovery,

Scientific Infrastructure)

Endogenous growth models -workforce (Jones)

Comin step one (growth rate of R&D-driven technology)

Comin step two

Porter/Stern step one (sources of patenting)

Workforce (Freeman)

Porter/Stern step two (relationship between patents and TFP growth)

Inputs to science(R&D, educationalFunding; capital

infrastructure)

Aggregated/disaggregated by sector, field, performer

Total factor productivity (Griliches school)

Final outcomes (TFP growth, R&D

ROI)

Intermediateoutcomes

(workforce, Scientific discovery,

Scientific Infrastructure)

Endogenous growth models -workforce (Jones)

Comin step one (growth rate of R&D-driven technology)

Comin step two

Porter/Stern step one (sources of patenting)

Workforce (Freeman)

Porter/Stern step two (relationship between patents and TFP growth)

Credit: Brian Zuckerman, STPI

16

S T P I

Efforts To-Date: Case Studies

• Studies range from large-scale agency-wide analyses to project-level cost-benefit studies

• Methodologies limit the ability of the case studies to address larger questions – disparate goals and methods make them

difficult to generalize or aggregate up to a national level (what is needed)

17

S T P I

Example: Nanobank Study

18

S T P I

Summary – What We Know

• No methodological winners or losers yet - each method brings its own strengths and weaknesses– Some simplify the structure and feedback

loops and rely more on data (econometric models)

– Others focus more on structures and relationships (system dynamics)

– Some focus on telling a coherent story (case studies)

– All reveal need for better data and many lead to theoretical and methodological insights

19

S T P I

http://www.cs.unibo.it/schools/AC2005/docs/Bertinoro.ppt#266,11,The Blind Men and the Elephant

Parts are studied and understood better than the whole!

Summary – What We Know

S T P I

What needs to be done next?

21

S T P I

Eleven Questions About the Universe

A panel of US physicists and astronomers has identified a list of eleven fundamental questions about the nature of the universe that will require the combined skills of particle physicists and astrophysicists to answer.

1. What is dark matter?

2. What are the masses of the neutrinos, and how have they shaped the evolution of the universe?

3. Are there additional spacetime dimensions?

4. What is the nature of the dark energy?

5. Are protons unstable?

6. How did the Universe begin?

7. Did Einstein have the last word on gravity?

8. How do cosmic accelerators work and what are they accelerating?

9. Are there new states of matter at exceedingly high density and temperature?

10. Is a new theory of matter and light needed at the highest energies?

11. How were the elements from iron to uranium made?

"From quarks to the cosmos", the first report from the committee on the physics of the universe set up by the National Academy of Sciences, 2001

22

S T P I

1. Identify a Set of Questions

• Dr. Marburger identified some– “Are we funding all the R&D we need to defend

ourselves, improve and sustain our quality of life, and compete with other nations in a globalized high-technology economy?”

(Talk about a Big Hairy Audacious Question)

• Are there others? What are they? Can/should the S&T policy community agree on a set? How?– A Grand Challenge meeting?

23

S T P I

2. Develop Framework

Prospective

Making future investments

- in what

- how much

- why

Retrospective

Role of past investments

- impact

- how

• No broadly accepted frameworks exist

24

S T P I

Retrospective Assessments

• The Good– A thriving evaluation community

– New methods being incorporated continually

– Issues around data availability, timeliness, quality etc. being recognized and addressed

– International cooperation (especially around data issues)

• The Bad– Focus at the program level

• Less emphasis on portfolio, agency or system levels

– Data collection is based on an outdated innovation system

• The Ugly– Efforts uneven

– Long way before there is enough data of the right kind that is of high enough quality that is linked causally

– Unclear how retrospective assessments are being used (or if)

25

S T P I

Prospective Assessments

• The Good– Need has been articulated– Tools exist in many other disciples and communities of practice – possible to

borrow?

• The Bad– Predictive tools are challenging to develop– Need system level understanding – difficult to model

• The Ugly– Somewhat discredited

• (quantitative tools) Remember systems analysis and the whiz kids?

• (predictions) NSF predictions about a shortfall of scientists and engineers in the 1990s

– Decisionmaking may continue to be (has always been) driven by equity and ideological criteria rather than scientific criteria

– Little structured dialogue between policymakers and science policy experts

• A role for evaluators to address the conceptual, methodological and computational challenges

26

S T P I

3. Review State-of-the-Art

• Best practices in frameworks, tools and techniques from other communities and domains – Private sector

• R&D investment decisionmaking in firms• Venture capital community• Supply chain management

– Math and operations research• Algorithms and models

– Hard sciences • Modeling tools and techniques

– In-SPIRE

27

S T P I

Emerging Frameworks: System Dynamics Modeling

• Tom Fiddaman’s work with the Office of Science

• Work at DOE labs to model the aging workforce in sensitive fields of research

ScientistsScientist Hiring Scientist Turnover

IndicatedScientists

Desired HiringRate Scientist Turnover

RateScientist

Adjustment Time

Scientist Salary

Funding Required

Funding Adequacy

Effect of Funding onScientists

Initial Scientists

Desired ResearchBudget

<Lab Research $>

28

S T P I

Emerging Tools: Text Mining to Visualize Clusters of Research

Data Source: PubMed publications affiliated with institutions in South Africa, 2003-2005

University of Cape TownStellenbosch UniversityUniversity of Witwatersrand

29

S T P I

Emerging Tools: Visualization of the Growth of Research Networks

• Katy Borner’s work in visualization

• Caroline Wagner’s work in network analyses

• Gretchen Jordan - frameworks

Source: http://iv.slis.indiana.edu/ref/iv04contest/Ke-Borner-Viswanath.gif

30

S T P I

4. Promote Policy Effort

• Government-wide effort– Data issues

• volume, quality, connectivity– NSF SRS working with the domestic and international

community

– Theories, frameworks and tools for modeling and analysis

• NSF SBE launching new program to fund research• Best practices from other domains

– Coordination• Interagency task group established to develop roadmap

31

S T P I

Discussion• What are the issues/questions in the space called “science of

science policy”? • How can these questions be explored (framework(s))?• What tools are techniques can be applied to address questions?

How can best practices – for frameworks, models, and tools – from other domains be adapted to this space?

• Other questions/comments/concerns

But Mousie, thou art no thy-lane

In proving foresight may be vain;

The best-laid schemes o' mice an' men

Gang aft agley,

An lea'e us nought but grief an' pain,

For promis'd joy!

Robert Burns, "To a Mouse”

32

S T P I

Thank You For Staying

“Some bargains are Faustian, and some horses are Trojan. Dance carefully with the porcupine, and know in advance the price of intimacy”

Source: http://nanoandsociety.com/ourlibrary/documents/bsts-nano.pdf

FeedbackBhavya Lal 617 331 0567blal@ida.org