Impact of Funding on Scientific Output and … › wp-content › uploads › 2013 › 12 ›...
Transcript of Impact of Funding on Scientific Output and … › wp-content › uploads › 2013 › 12 ›...
11/2/13
1
Impact of Funding on Scientific Output and Collaboration
Ashkan Ebadi Hadi Shahidi Nejad
Andrea Schiffauerova
2 November 2013
1 / 16
Outline
q Funding q Collaboration q Impact of funding on collaboration q Impact of funding on scientific output q Research gaps and conclusion
11/2/13
2
2 / 16
Funding
One of the most crucial factors for improving the research performance • Huge annual investment on R&D
• Importance of the link between funding and scientific development
• Evaluation is needed! (King, 1987)
(Martin, 2003)
3 / 16
Role of Funding
Affects size, efficiency and productivity of R&D sector
(Jacob & Lefgren, 2007)
Benefits of the funded research: Ø Increasing the level of available knowledge Ø Motivating collaboration networks Ø Training skillful graduates Ø Creating new jobs and companies Ø Augmenting the problem-solving ability of the researchers
(Martin et al., 1996)
11/2/13
3
4 / 16
Why researchers collaborate?
(Lee & Bozeman, 2005)
Collaboration!
Higher produc-vity
Be2er and faster access to exper-se
Idea exchange among scien-fic disciplines
Be2er access to funding resources
(Beaver, 2001)
(Katz, et al., 1997)
(Heinze, et al., 2008)
(Beaver, 2001)
How to measure it
Ø Co-authorship Advantages
Prac9cal
Invariant
Verifiable
Inexpensive
Quan9fiable
Disadvantages
Collabora9on not necessarily results in a joint ar9cle
Par9cipa9on share
(Subramanyam, 1983; Katz & Martin, 1997)
Ø Sub-authorship • Coordina9on cost • Finding right partners • Delays
(He, Geng, & Campbell-Hunt, 2009)
5 / 16
Funding Impact on Collaboration
Author(s) Year Methodology Data Target area Result(s) Beaver and Rosen
1979 Indicators (no of authors per article)
24 scientific fields Positive relation Heffner 1981 Statistics · 395 articles, 28 journals
· 1974-1975 4 scientific fields Positive relation found in 2 (out of
4) disciplines Lewison and Dawson
1998 Statistical descriptive analysis
Research Outputs Database (ROD) 12,925 UK papers 1988-1994
Biomedical (gastroenterology)
Collaboration and number of funding bodies increase paper quality
Bozeman and Corley
2004 Questionnaire Regression analysis
451 scientists and engineers in the US
Scientists’ collaboration choices
Significantly positive
Adams et al. 2005 Regression analysis
2.4 million papers of 110 US universities 1981-1999
Top US universities
Positive effect of funding on team size
Gulbrandsen and Smeby
2005 Questionnaires Logistic regression analysis
1,967 records Tenured university professors in Norway
A positive relation observed
Lundberg et al. 2006 Indicators Industrial funding to a medical university 1993-2003
Co-authorship between university and industry
Incomplete results Some signs of positive collaboration
11/2/13
4
6 / 16
Funding Impact on Collaboration
Author(s) Year Methodology Data Target area Result(s)
Thune 2007 Qualitative analysis
Interviews with 29 researchers and R&D managers
Collaborative R&D projects in two academic fields
Important, but other factors are also involved
Rosenzweig et al. 2008 Logistic regression 5,728 articles published in 4 American journals 1994-2003
US Emergency Medicine (EM)
No significant relation
Defazio et al. 2009 Regression analysis
· Panel of 294 scientists · 39 EU research networks
Researchers in the EU funding program framework
Funding may affect the formation of more effective collaboration networks
7 / 16
Funding Impact on Collaboration
Literature Review
Article Ctrl.
Grp
Independent Variables
Type Various
sources of funding
Different funding periods?
Have
network structure variables?
Fund Gender Prestige/career Scientific
fields Regional
share Past prod.
H e f f n e r (1981) ü ü Desc.
analysis No No No
Bozeman and Corley (2004)
ü ü ü ü OLS
Linear regression
No No No
Adams et al. (2005) ü ü ü Linear
regression No No No
Gulbrandsen and Smeby (2005) ü ü ü ü ü ü Logistic
regression Yes No No
Rosenzweig et al. (2008)
ü ü Logistic regression Yes No No
Defazio et al. (2009) ü ü ü Linear
regression No Yes No
11/2/13
5
8 / 16
Funding Impact on Collaboration
Highlights: • Relatively new • From simple indicators to statistical analysis • No study done in Canada! • Relatively limited data sets • Lack of using a control group • Vague net impact of funding on collaboration
9 / 16
Scientific Output
Number of publica-ons as a measure of output quan-ty • Very reliable measure for large-‐scale data • Quan9fiable
Number of cita-ons as an indicator of quality • A good index of the mean impact at the aggregate level • Easy to calculate • Drawbacks: • Papers of famous scien9sts are more likely to be cited • Self cita9on • Etc.
(Gingras, 1996)
(Okubo, 1997)
11/2/13
6
10 / 16
Funding Impact on Scientific Output
Author(s) Year Methodology Data Target area Result(s)
McAllister and Narin
1983 Bibliometric indicators
NIH funded researchers
Strong positive relationship High quality schools are more productive
Peritz 1990 Statistical analysis Articles published in two British journals in 1978 and 1979
Economics Funded researchers are more being cited
Gingras 1996 Bibliometric indicators
NSERC funded research 1984-1993
Feasibility study of using bibliometrics
Feasible to apply at disciplines or specialties level
Arora et al. 1998 Econometrics OLS regression
Research funded by CNR 1989-1993
Biotechnology, bio-instrumentation
Italy
More unequal distribution of funds may increase the output in the short term
Lewison and Dawson
1998 Statistical analysis Research Outputs Database (ROD)
12,925 UK papers 1988-1994
Biomedical (gastroenterology)
Positive relation between the number of funding bodies and research output impact
Boyack and Borner 2002 VxInsight 3D map 33,448 grants 4,549 outputs
1975-2001
BSR program output NIA grants
Positive relation between funding and output in most of their cases
Godin 2003 Bibliometrics Science Citation Index (SCI) database 1990-1999
NSERC funded research
Positive relation between funding and productivity, but no impact on quality
Payne and Siow 2003 Regression analysis Federal funding 1972-1998
74 research universities
Positive effect on output No impact on research quality
11 / 16
Funding Impact on Scientific Output
Author(s) Year Methodology Data Target area Result(s) H u f f m a n and Evenson
2005 Econometrics USDA 1970-1999
Agricultural research productivity
Negative impact
Carayol and Matt
2006 Statistical analysis OLS regression
Louis Pasteur University (ULP) 1993-2000
Faculty members No significant effect of contractual funding
Jacob and Lefgren
2007 OLS regression 1980-2000 NIH Positive impact on the output
Crespi and Geuna
2008 Econometric Thomson ISI database 1981-2002
Higher education in 14 OECD countries
A significant impact on the time lag structure of the output
Albrecht 2009 Bibliometric indicators PubMed database, 1994-2003 CANSA No conclusion
Leydesdorff and Wagner
2009 Main S&T indicators of OECD (2008)
Thomson’s Web of Science M a c r o - l e v e l comparisons
Different patterns of the link between inves tment and wor ld-share of publication
Campbell et al.
2010 Bibliometrics Thomson Reuters’ Web of Science (WoS) database
NCIC Positive relation between funds and output quality
Shapira and Wang
2010 Bibliometric indicators Thomson ISI database 2008-2009
C r o s s - c o u n t r y evaluation
Positive impact of China’s investment on output quantity, but no major effect on the quality
B e a u d r y and Clerk-lamalice
2010 Regression analysis 1985-2005 3,678 articles
C a n a d i a n b i o t e c h n o l o g y academics
Positive effect of strong network position and individual funding on scientific output
B e a u d r y and Allaoui
2012 Regression analysis Articles and patents 1985-2005
3,724 articles, 566 patents
C a n a d i a n n a n o t e c h n o l o g y researchers
Positive effect of public funding, no impact of private funding
11/2/13
7
Godin (2003) studied the impact of NSERC funding on the productivity and papers’ quality of the supported researchers:
Ø Bibliometric evaluation Ø 1990-1999 Ø Science Citation Index (SCI) database Ø Positive relation between funding and productivity Ø No impact of funding on papers’ quality
12 / 16
Funding Impact on Output - Canada
Literature Review
Gingras (1996) discussed the feasibility of bibliometric evaluation of funded research:
Ø 1984-1993 Ø Two grant selection committees of NSERC (ME and EE) Ø Bibliometric indicators are good measures for investigating the
relation among funds and productivity but not at individual level
1996
Gingras
2003 Feasibility study
Godin
• Positive impact on productivity • No impact on quality
2009-2010
Campbell et al.
Few case studies done by Campbell et al. in 2009-2010
Ø Evaluated the impact of funded research Ø National Cancer Institute of Canada (NCIC) Ø Thomson Reuters’ Web of Science (WoS) database Ø Bibliometrics Ø Positive relation between NCIC funds and scientific performance
• Case studies • Positive impact on performance • No impact on quality
Ø Evaluated the impact of funded research Ø Canadian Forest Service (CFS) Ø Bibliometrics Ø Assessed CFS internal, national and international position
Ø Evaluated specifically the selection procedure of Genome Canada Ø Thomson Reuters’ Web of Science (WoS) database Ø Bibliometrics Ø Peer-review process was successful in researchers selection Ø Higher scientific impact of the funded researchers’ papers
2012 • Network variables • Positive impact of public funding • No impact of private funding
Beaudry and Allaoui
Beaudry and Allaoui (2012) studied the impact of funding on the productivity of Canadian nanotechnology researchers:
Ø Negative binomial regression Ø 1985-2005 Ø 3,724 articles and 566 patents Ø Positive relation between public funding and productivity Ø No impact of private funding
13 / 16
Funding Impact on Scientific Output
Article Ctrl.
Grp.
Independent Variables
Type
Various sources
of funding
Different funding periods?
Network structure variables? Fund
Other productiv-ity shocks
Prestige/ career/
Age
Content/ scientific
fields
Regional share
Group size
Paper quality
Peritz (1990) ü ü ü Descriptive analysis No No No Arora et al. (1998) ü ü ü ü ü OLS regression No No No
Lewison and Dawson (1998) ü ü ü ü Statistical analysis No No No
Godin (2003) ü ü ü ü Descriptive analysis No No No
Payne and Siow (2003) ü ü ü Linear regression No No No
Huffman and Evenson (2005) ü ü Econometrics Yes No No
Carayol and Matt (2006) ü ü ü ü OLS regression No No No
Jacob and Lefgren (2007) ü ü ü ü OLS regression No Yes No
Crespi and Geuna (2008) ü ü ü Econometrics No Yes No
Campbell et al. (2010) ü ü ü ü Descriptive analysis No No No Beaudry and Clerk-lamalice (2010)
ü ü Negative binomial
regression Yes No Yes
Beaudry and Allaoui (2012) ü ü ü
Negative binomial regression Yes No Yes
11/2/13
8
14 / 16
Funding Impact on Scientific Output
Highlights: • Recently attracted more attention • Simple indicators and statistical analysis • Several studies in Canada • Relatively limited data sets • Lack of using a control group • Vague net impact of funding on output • Less studies done at the individual level
15/ 16
Research Gaps
Ø Lack of a comprehensive study Ø The importance of the impact of network structure Ø Bibliometrics as the main methodology Ø Consider different periods of funding Ø Relatively old results Ø Adjust funding!
Ø Vague relation among funding, collaboration and scientific production
Ø Lots of contradictory results
Ø Very limited knowledge about the effects of funding on collaboration
Ø Limited scope of the studies (collaboration among the university
professors or the cooperation between universities and industry)
Ø No test group
Ø Simplified linear regressions
Ø Score collaborations! (Linking to the same institute gets lower score)
11/2/13
9
Ø The need for developing a clear link between the funding,
multidisciplinary collaboration and knowledge production
Ø Most of the studies have used bibliometrics or statistical
methods for performing the analysis
Ø The need for a comprehensive study that covers several
scientific fields and aspects
17 16/16
Research Summary
18 / 22
Suggested Solution
Bibliometrics
Sta9s9cal Analysis
Visualiza9on Techniques
Social Network Analysis
Data & Text Mining
Survey
11/2/13
10
Bibliometric Analysis Ø Scientific output and research quality Ø Funding trend and funding efficiency Ø Universities (regional) performance Ø Collaboration trend and research group structure Ø Measuring the complexity of research
19 / 22
Methodologies
Data Mining Ø Dimensionality and size reduction Ø Clusters of researchers Ø Pattern and rules detection
Text Mining Ø Pre-processing and data cleaning Ø Keyword extraction Ø Co-authorship trend (probabilities of a connection)
Collaboration Network Analysis Ø Network structure variables Ø Trend of collaboration Ø Collaboration patterns
Statistical Analysis Ø Statistical validation of the results
Survey Ø Qualitative investigation to check the quantitative results
20 / 22
Methodologies
Why several methods Ø A multi-dimensional phenomenon! Ø “Triangulation” (cross examination), using multiple methods are
used in a study to empower the validation of results Ø Enabling comparisons of different explanations (Fulop et al., 2001)
11/2/13
11
22 / 22
Evaluation Methods
Scientometrics
Bibliometrics Peer review
Econometric Cost-‐benefit analysis
Surveys Case studies
Data and text mining
Visualiza9on techniques Social network analysis
Peer review
Ø Fast and low cost Ø High quality works are most of the time detected Ø Highly depends on the experts and criteria Ø More fame will result in geZng higher funds Ø High administra9ve cost
(King, 1987)
Ø Simple and easy to use Ø Not integrated and simplified (Ruegg, 2007)
Sciento-‐metrics
Ø Good tool for testing the findings Ø Simplified assumptions for creating the model
(Salter & Martin, 2001) Surveys Case studies
Ø Narrative, easy to understand Ø Less convincing Ø Could be inconsistent (Ruegg, 2007)
Econometric
Ø Modern technique Ø Relatively expensive (Ruegg, 2007)
Data mining
Ø Give a big picture of the subject Ø Limited flexibility
Visualiza9on
Ø Useful tool for studying collaboration Ø Useful tool for studying impact of a program Ø The generated network can be time limited (Ruegg, 2007)
11/2/13
12
23
Funding Impact on Collaboration
Literature Review
Ø Higher financial investment can change the structure of research Ø The efficient collaboration network will not be stable due to huge
coordination cost Ø Funding enables researchers to cover the collaboration costs
Ø Co-authorship as a measure of scientific collaboration Ø Studying the Impact of governmental funding on scientific collaboration
and formation of scientific networks is relatively new (Katz & Martin, 1997; Lee & Bozeman, 2005)
(Ubfal & Maffioli, 2011)
Ø Funding may help the central actor(s) to make a balance between the new knowledge creation and the management of the existing collaborative relationships in the network
(Porac et al., 2004)
24
Funding Impact on Collaboration
Literature Review
Heffner (1981) analyzed the relation between funding and multiple authorship:
Ø 500 articles published in 28 journals in four scientific fields Ø 1974-1975 Ø Statistical analysis Ø Positive relation between financial support and size of the research teams Ø Statistically significant just in chemistry and biology
Researchers of top US universities with larger amounts of funding tend to work in larger scientific groups (Adams et al., 2005)
Defazio et al. (2009) studied the impacts of funding on collaborative behavior and productivity of the researchers:
Ø Used a panel of 294 scientists in 39 EU research networks Ø 15-year time period Ø Funding may play an important role in forming more effective collaboration
networks
11/2/13
13
25
Funding Impact on Collaboration
Literature Review
Gulbrandsen and Smeby (2005) studied the effect of industrial funding on the performance of university professors:
Ø Norway Ø Used questionnaires to collect data from all tenured professors Ø Statistical analysis Ø A Positive relation observed Ø Funded professors tend to collaborate more with other researchers from
both universities and industries
A significant positive effect of funding on the collaboration of the US university researchers (Bozeman & Corley, 2004; Lee & Bozeman, 2005)
26
Funding Impact on Output
Literature Review
Ø Number of publications as a measure of output quantity Ø Number of citations as an indicator of quality
Peritz (1990) found that even if both funded and unfunded researches are published in a high-impact journal, the funded research will be more cited.
Lewison and Dawson (1998) used journal impact as a quality measure. They concluded that the number of authors per article and the number of funding bodies have a great effect on the impact of research output.
McAllister and Narin (1983) found a positive relation between NIH’s funding and number of publications of the U.S. medical schools.
11/2/13
14
27
Funding Impact on Output – Cross Country
Literature Review
Leydesdorff and Wagner (2009) analyzed the relation between research macro-level investment and world share of publication:
Ø Different efficiencies observed among examined countries Ø Different schemes of funding
Crespi and Geuna (2008) analyzed impact of investment on scientific productivity of 14 OECD countries:
Ø Used Thomson ISI database Ø 1981-2002 Ø Focused on the time lag Ø Found a significant impact of investment
Shapira and Wang (2010) investigated impact of nanotechnology funding:
Ø Used Thomson Reuters database Ø August 2008 to July 2009 Ø Very basic bibliometric indicators Ø China is getting closer to the U.S. in terms of the number of
publications but still lower quality
28
Publications
Conferences: 1. Ebadi, A. & Schiffauerova, A., 2011. Investigating Climate Change Patterns in Montreal: A
Data Mining Approach, Third International Conference on Climate Change, Brezil. 2. Ebadi, A., Moazami, A. & Schiffauerova, A., 2012. Analyzing the Collaboration Trend of
Canadian Researchers in Natural Sciences and Engineering: The Impact of Geographical Proximity and Research Funding, Proximity Days Conference - 7th Edition, Montreal, Canada.
3. Ebadi, A. & Schiffauerova, A., 2012. Impact of Funding on Scientific Output and Collaboration Patterns of Canadian Researchers: A Bibliometric and Text Mining Approach”, 13th Collnet Conference, Seoul, South Korea.
Journal: 1. Eslami, H., Ebadi, A. & Schiffauerova, A., 2012. Effect of collaboration network structure on
knowledge and innovation productivity: The case of biotechnology in Canada.
11/2/13
15
29
Data Gathering Procedure
144,156 articles 184,770 authors
1996-2010
381,197 grantees