Science Measures as Relevant and Accurate Forms of Evidence

Irwin FellerSenior Visiting Scientist, American Association

for the Advancement of Science

Workshop on Measuring the Impacts of Science

Montreal, Canada June 17-18, 2004

Rip’s Triangle

22

2Scientists

Scientometricians

12

2

32

2Science Administrators/

Managers

Worldview

3rd Corner Perspective

2nd Corner Findings

1st Corner Audience

Problem-driven, Policy ContextsNational Science Foundation—

Government Performance and Results Act;

National Institutes of Health—Criteria for Establishing Centers

Department of Energy-Office of Science—Science Indicators

Office of Management and Budget—R&D Investment Criteria;Performance Assessment Rating Tool

Research University-Strategic Planning, Performance Measures---------------------

NIST: Advanced Technology Program NIST: Manufacturing Extension Partnership Program

State Technology Programs

Turn of a Kaleidoscope

Long standing issues/Familiar Material

Shift in Perspective/New patterns and colors

Analytical Context

“…the usefulness of economic data …cannot be gauged without relating them to the uses (theorems) to which they can be fitted” (Morgenstern, On the Accuracy of Economic Observations, 1963, p. 95)

“…to achieve quality requires an assessment of situations and processes which is essentially qualitative (and often sociological),” Rip (1997, p. 18)

Policy Framework

• Decision Settings are Contested

• Competing Strategies for the Support and Conduct of Science

• Science Measures are Evidence used to Test (and Formulate) Hypotheses

Contested Environment: Example

“Interdisciplinarity” is the mantra of contemporary science policy discourse. Yet “everyone knows” (at least in the U.S. scientific community) that major scientific discoveries arise from single investigator-led, disciplinary based research.

Trade-offs and internecine conflicts arise for science managers (and within scientific communities) in accommodating both perspectives in period of static budgets.

Decision-making Stages

1. Prospective Decisions-Fields of Science

2. Modes of Research Support

3. Retrospective Assessments

The SCIENCE MANAGER’SDECISION CYCLE

Outyear CycleInitial Conditions Annual Planning, Budgeting, & Assessment Cycle (New Conditions)

Existing Stateof Science

MissionObjectives

NationalPriorities

OrganizationalCapabilities

Budget

ProspectiveScience

Decisions

ProspectiveScience

Decisions

OrganizationalAssignments

OrganizationalActivities

Outcomes

RetrospectiveAssessment

T0 T1 T1...n Time

Prospective Decisions: I

“If you can look into the seeds of time

And say which grain will grow and which will not,

Speak then to me…

Banquo (Macbeth, Act 1, Scene 3)

Prospective Decisions: II

Senior U.S. Science Official:

Choice of problem is the most important decision that a researcher makes.

Can We Scale up from principal investigator choices to program level/agency level decisions?

When do you stop funding an area of research? (deadends; loss of vitality)

Prospective Decisions: IIIModes of Research Support

Design/choice of allocation mechanisms (e.g, competitive merit review; formula funding)

Choice of funding mechanism

Choice of performer

Retrospective Assessments

GPRA and all that

Relevance for Science Policy?

Critique of Current Discourse on Science Measurement

1) Underlying theory is too heavily weighted towards communication flows and prestige races and inadequately grounded in historical treatments of processes of scientific discovery

2) Distinctions among priority setting, project selection, and performance assessment are blurred

3) Rooted in partial assessments of outputs and outcomes, muting importance of multiple interconnections

Positive Program

Systematically ground science measures in terms of

• sets of decisions confronted by science managers;

• processes of scientific discovery

• connect (1) and (2)

Science Measures and Processes of Scientific Discovery: I

Heightened importance of longstanding recognition that:

“not all highly cited papers contain important “ideas” in the narrow sense of the word”, and

“important should not be confused with correct, for an idea need not be correct to be important” (Garfield, Malin and Small, 1978)

Science Measures and Processes of Scientific Discovery: II

By definition, high level of agreement on core knowledge;

Consensus at the frontiers of science is relatively low

(Cole, Making Science)

“Last month a team of neuroscientists announced a long-awaited breakthrough—an exquisitely detailed portrait of a type of protein that is crucial for the generation of nerve impulses…(U)nfortunately, however the model contradicts the widely accepted view of how the channel works. And although everyone agrees that deriving the new structure was a tremendous technical feat, many…say the final product must be flawed” (Science, 27 June 2002)

Cole’s Cone

CoreKnowledge

Frontiers of Science

Cole’s Cone in aDecision Making Context

D1 “Vision/Foresight”

D2 PART

D3 GPRA

Degree ofuncertaintyaboutoutcomes/(consensusaboutfindings)

Cole Cone as viewed byScience managers

Accuracy ofScience measures

TimeD1 D2 D3

Science Measures of Formsof Evidence

Measures “inform” decision-makers.They improve their abilities to discriminate between

1) Revolutionary, important, correct, and relevant outcomes from “high” performing/leading organizations and

2) Mundane, unimportant, incorrect and irrelevant outcomes from “low” performing /lagging organizations

Science Measures as Evidence:Types of Errors

Type 1 — False Negatives

Type 2 — False PositivesType 3 — Right Answers,

Unimportant Questions

Type 4 — Premature Discoveries?

Type 5 — Important Questions,

No Answers

Type 1 – False Negatives

Ocogenes“It is difficult to think of another case of scientific advance where almost every one of the key pioneers encountered political resistance from his community of peers. Unlike earlier episodes in the history of science, the resistance originated in neither religious or ideological prejudice. It derived from the skepticism of a professional community of biomedical scientists whose beliefs were grounded in available laboratory science”

Kelves “Pursuing the Unpopular” in Hidden Stories of Science, p. 106

Planck’s Thesis Redux

“A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it”

Type 2-False Positives

“The rise of Polywater shows how men and women actually behave in the laboratory, under the stress of today’s extremely competitive science system: their behavior does not always conform to the ideal search for abstract truth that has long been the classic image. The desire to be an active part of a revolutionary movement is often in conflict with the natural reluctance to jettison the set of accepted ideas on which he has made his way in the profession” (I. B. Cohen, Revolution in Science, p. 35)

Type 3 - Right Answers,Wrong Questions

Self-referential, institutionalized fields of science (established study sections) but questionable scientific “vitality” or relevance

Type 4: Premature Discoveries

Hook thesis on prematurity in scientific discovery

van Raan’s “Sleeping Beauties”

Multiple Impacts

National Science Foundation - GPRA Strategic Goals

People, Ideas, Tools

Similar classifications found in other science agency plans

Well recognized interactions among outputs, outcomes, impacts, but few measures

Knowledge Power Ball

New Applications

New Facilities/Instruments

New HumanCapital

New Discoveries