Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis...
-
Upload
makaila-piercey -
Category
Documents
-
view
212 -
download
0
Transcript of Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis...
![Page 1: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/1.jpg)
Copyright © 2012
American Institutes
for Research.
All rights reserved.
Building a Scientific Basis for Research Evaluation
Rebecca F. Rosen, PhDSenior Researcher
Research Trends Seminar
October 17, 2012
![Page 2: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/2.jpg)
2
Outline
• Science of science policy• A proposed conceptual framework• Empirical approaches:
NSF Engineering Dashboard ASTRA – Australia HELIOS – France
• Final thoughts
![Page 3: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/3.jpg)
3
Outline
• Science of science policy• A proposed conceptual framework• Empirical approaches:
NSF Engineering Dashboard ASTRA – Australia HELIOS – France
• Final thoughts
![Page 4: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/4.jpg)
4
The emergence of a science of science policy
• Jack Marburger’s challenge (2005)
• Science of Science & Innovation Policy Program at the National Science Foundation (2007) An emerging, highly interdisciplinary research field
• Science of Science Policy Interagency Task Group publishes a “Federal Research Roadmap” (2008): The data infrastructure is inadequate for decision-
making
• STAR METRICS (2010)
![Page 5: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/5.jpg)
5
Why a science of science policy?
• Evidence-based investments• Good metrics = good incentives• Science is networked and global
• Build a bridge between researchers and policymakers• Researchers ask the right questions
• The adjacent possible: leverage existing and new research and expertise • New tools to describe & measure
communication
![Page 7: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/7.jpg)
7
A conceptual framework for a science of science policy
![Page 8: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/8.jpg)
8
Getting the right framework matters
• What you measure is what you get Poor incentives Falsification
• Usefulness• Effectiveness
![Page 9: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/9.jpg)
9
A proposed conceptual framework
Adapted from Ian Foster, University of Chicago
![Page 10: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/10.jpg)
10
A framework to drive person-centric data collection
WHO is doing the research
WHAT is the topic of their research
HOW are the researchers funded
WHERE do they work
With WHOM do they work
What are their PRODUCTS
![Page 11: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/11.jpg)
11
Challenge – The data infrastructure didn’t exist
However, some of the data do exist
![Page 12: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/12.jpg)
12
Empirical Approaches
Leveraging existing data to begin describing results of the scientific enterprise
![Page 13: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/13.jpg)
13
An empirical approach
• Enhance the utility of enterprise data• Identify authoritative “core” data elements• Develop an Application Programming
Interface (API) Data platform that provides programmatic
access to public (or private) agency information
• Develop a tool to demonstrate value of API
![Page 14: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/14.jpg)
14
Topic modeling: Enhancing the value of existing data
David Newman - UC Irvine
NSF proposals
NSF proposals Topic Model:
- Use words from(all) text
- Learn T topics
t49t18t114t305
t49t18t114t305
Topic tags for each and every proposal
Automatically learned topics (e.g.):
…
t6. conflict violence war international military …
t7. model method data estimation variables …
t8. parameter method point local estimates …
t9. optimization uncertainty optimal stochastic …
t10. surface surfaces interfaces interface …
t11. speech sound acoustic recognition human …
t12. museum public exhibit center informal outreach
t13. particles particle colloidal granular material …
t14. ocean marine scientist oceanography …
…
![Page 15: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/15.jpg)
15
Stepwise empirical approach
• Enhance the utility of enterprise data• Identify authoritative “core” data elements• Develop an Application Programming
Interface (API) Data platform that provides flexible,
programmatic access to public (or private) agency information
• Develop a tool to demonstrate value of API
![Page 16: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/16.jpg)
16
![Page 17: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/17.jpg)
17
Stepwise empirical approach
• Enhance the utility of enterprise data• Identify authoritative “core” data elements• Develop an Application Programming
Interface (API) Data platform that provides programmatic
access to public (or private) agency information
• Develop a tool to demonstrate value of API
![Page 18: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/18.jpg)
18
Outline
• Science of science policy• A proposed conceptual framework• Empirical approaches:
NSF Engineering Dashboard ASTRA – Australia HELIOS – France
• Final thoughts
![Page 19: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/19.jpg)
19
![Page 20: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/20.jpg)
20
![Page 21: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/21.jpg)
21
![Page 22: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/22.jpg)
22
![Page 23: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/23.jpg)
23
![Page 24: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/24.jpg)
24
Outline
• Science of science policy• A proposed conceptual framework• Empirical approaches:
NSF Engineering Dashboard ASTRA – Australia HELIOS – France
• Final thoughts
![Page 25: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/25.jpg)
25
Linking administrative and grant funding data in Australia
![Page 26: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/26.jpg)
26
Outline
• Science of science policy• A proposed conceptual framework• Empirical approaches:
NSF Engineering Dashboard ASTRA – Australia HELIOS – France
• Final thoughts
![Page 27: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/27.jpg)
27
People People
Describing public-private partnerships in France
![Page 28: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/28.jpg)
28
What does getting it right mean?
• A community driven empirical data framework should be: Timely Generalizable and replicable Low cost, high quality
• The utility of “Big Data”: Disambiguated data on individuals
- Comparison groups
New text mining approaches to describe and measure communication
??
![Page 29: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/29.jpg)
29
Final thoughts
![Page 30: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/30.jpg)
30
Policy makers can engage SciSIP communities:
• Patent Network Dataverse; Fleming at Harvard and Berkeley
• Medline-Patent Disambiguation; Torvik & Smalheiser at U Illinois)
• COMETS (Connecting Outcome Measures in Entrepreneurship Technology and Science); Zucker & Darby at UCLA
![Page 31: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/31.jpg)
31
The power of open research communities
• Internet and data technology can transform effectiveness of science: Informing policy Communicating science to the public Enabling scientific collaborations
• Interoperability is key
• Publishers are an important part of the community
![Page 32: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/32.jpg)
32
Rebecca F. Rosen, PhD
E-Mail: [email protected]
1000 Thomas Jefferson Street NWWashington, DC 20007
General Information: 202-403-5000TTY: 887-334-3499
Website: www.air.org
THANK YOU!
![Page 33: Copyright © 2012 American Institutes for Research. All rights reserved. Building a Scientific Basis for Research Evaluation Rebecca F. Rosen, PhD Senior.](https://reader038.fdocuments.in/reader038/viewer/2022110303/551636c455034694308b627b/html5/thumbnails/33.jpg)
33