Towards a bibliometric database for the social sciences and

humanities

Carmen Lopez Illescas, Felix de Moya Anegon, Janus Linmans, Anton Nederhof and Henk F. Moed

SCIMago Research Group, Univ Granada, SpainCWTS, Leiden Univ, the Netherlands

This presentation …..

• Highlights wider context and its dynamics

• Makes analytical distinctions

• Describes actual developments

• Discusses options

Structure of the presentation

• Introduction; background

• Analysis of existing SSH databases

• Actual use of bibliometric indicators in research assessment

• Actual practices in dealing with non-publication outputs

• Options for creating a comprehensive database of SSH outputs

General scheme

Research AssessmentObjectives; criteria

Assessment methodologies;indicators

Databases

1.

Introduction; background

About SSH

General trends

Indicators

Science vs. Social Sci & Humanities (SSH)

Science Soc Sci + Humanities

Source/ document types

Journal research article is main

type

Great diversity; books important

(e.g. monographs, edited works)

Journal system Concentration in limited number of international

journals

Less concentration;

national journals also important

Bibliographical system

Concentration in limited number

of large internat. databases

Great diversity of bibliographies and catalogs (national,

specialized)

SSH outputs and impacts

Impacts Publication/text Non-publication / non-text

Scientific-scholarly

Research paper; monograph; book

chapter

Research data file; video of experiment

Educational Teaching course book;

PhDs

Economic Patent Product; process; device; design; image

Cultural Newspaper article; TV interviews; Performances;

exhibits; events

Trends-1 : Increasing importance of research assessment

• Performance-based funding across scientific institutions

• Institutions operate in a global market

• Public rankings of institutions

• European Research Area and Research Council

• Management tools within institutions

• Need for adequate tools for research assessment in SSH fields

Trends - 2 : Electronic publishing; more databases

• Publishers make their content electronically available on-line

• More and more documents freely available via the Web (Open access, self archiving)

• Standardisation of meta-data and meta-data infastructures

• More repositories (institutional/disciplinary)

• More institutional research management systems

• Web of Science (WoS), Scopus and Google Scholar are genuine competitors

Trends – 3 : Better coverage of SSH; more indicators

• WoS and Scopus expand their source coverage of SSH fields

• Bibliographical databases implement bibliometric features

• Calculation of bibliometric indicators not merely done by bibliometric experts

Two types of bibliometric activity

Type of activity Description Type of database used

Desk-top or poor man’s bibliometrics

Collects simple indicator data directly from

database

Bibliographic database;

indicator sets

Advanced bibliometrics

Data collection protocols;

verification; sophisticated

indicators

Bibliometric database

Bibliometric database: examples

• Cited references parsed and linked to corresponding targets (citation index)

• De-duplicated institutional affiliations

• Author names linked to unique researchers

• Abstracts parsed into noun phrases

• Dates expressed as numbers

• Policy relevant subfield delimitations

• Acknowledgements parsed; funders de-duplicated

Principal bibliometric indicators and database requirements – 1

Concept Definition Minimal database requirements

Production Nr. written documents published

Publ. meta-data; publ. types

Importance of publ. source

Impact factor; expert ratings

Source categorizations

Impact Citations Citation index

Collaboration (Institutional) co-authorship

Multiple authors/ addresses

Principal bibliometric indicators and database requirements – 2

Concept Definition Minimal database requirements

Cognitive structures

e.g. co-word maps Titles, abstracts

Qualitative citation analysis

Citation context analysis

Full texts

Semantics-based detection of links

e.g., scientific instruments

mentioned in full texts

Full texts

Bibliometric impact indicators

Indicator Comment

Citations What do citations in SSH fields measure?

Full text downloads Limited availability; What do downloads measure?

Hyperlinks Webometric assessment; combine with other indicators

Book purchases Normally not publicly available (but see amazon.com)

Library holdings Exploratory studies published recently

Library loans Not used in res. perf. studies

2. Analysis of existing SSH

databases

Sources of publication meta dataNational bibliographies

Library catalogues (OPAC) US Library of Congress; Acad libr; OCLC Worldcat

Short title catalogues ESTC; national bibliogr. for older books

Publisher or vendor catalogues

Amazon.com; Springer ebook catalogue

Special bibliographies and abstracts

FRANCIS, Sociol Abstr, PsychInfo, ECONLIT, …

Citation indexes Web of Science, Scopus, Google Scholar

Repositories In principle open access

Institutional research management systems

Output registration systems annual research reports

Google Book Search See later slide

Sources of citation data

Citation Indexes Web of Science (SSCI, A&HCI), Scopus

Special bibliographies PsychInfo, Sociol Abstr, World Polit Abstr,… ; most do not have citations

Repositories CiteseerX; most repositories do not have citations

Google Scholar Harzing’s publish-or-perish software allows citation counting

CrossRef Cross-publisher linking system provided by publishers

ERIH Journal classification for 11 Humanities research fields (6,000 journals)

Class Description

A High ranking international journals with very strong reputation

B Standard international journals with good reputation

C Important domestic research journals

Analysis of databases

Producers

Special bibliographies

20 mainly European bibliographies

Big citation indexes

Thomson, Elsevier, Google

Publishers University Presses

Scientific institutions

Institutional repositories

Scientific institutions

Research management systems

European special bibliographies:Aspects considered – 1

• Database producer• Dates of coverage• (Sub)disciplines covered• Number of records• Selection criteria of sources• Countries or languages covered• Type of sources covered • Nr authors of a source publication included• Nr institutional affiliations included• Standardisation of author names/affiliations

European special bibliographies:Aspects considered – 2

• Categorization of documents• Type of content classification system• Does it contain cited references in source

publications?• Has the database ever been used in

bibliometric studies?

Database Author affiliations Cited references

Francis Soc Sci + Hum Y (All, as from ‘97) N

CSA Ling & Lang Y N

Econlit Y N

IBSS Social Sci Y N

LISA Libr & Inf Sci Y (as from 2006) N

Psychinfo Y (all authors) Y

Sociolog Abstr Y (1st author only) Y (as from 2002)

World Polit Abstr Y (1st author only) Y (as from 2001)

Historical Abstr N Y

America: Hist & Life N Y

ERIC: Educat Res Y (1st author only) N (Y in full texts)

SOLIS Social Sci Y (in Author field) N

Special bibliographies: Conclusions

• Some interesting databases for field-specific studies

• In most databases no author affiliations and no cited references

• Mainly journal articles covered

Citation Indexes: Major recent developments

Database Journals Books

Web of Science

Adds in 2009 1,500 “regional” journals (many SSH, from

ERIH lists)

?

Scopus Adds 3,500 titles (1,500 in April 2009); 2,250 from ERIH lists (all A, 1,000 B, 250 C)

Adds meta-data on highly cited book

titles

Google Further enhancements of Google Scholar

Expands Google Book Search; Link

with Google Scholar?

Google Book Search – 1

Aims: • Enhance the user’s ability to access and read

books• Opportunity for authors and publishers to

make their books available.

Two sources: • Partner program: Publishers and authors • Library project: partner libraries

Google Book Search – 2

• For books protected by copyright, search results are limited to meta-data and selected (random) text passages

• Books out-of-copyright may be read online in full length or downloaded

• Cited references not a part of meta-data• GBS books as targets in Google Scholar

OAPEN - Open Access Publishing in European Networks

• OAPEN: European scholarly university presses active in SSH and in book publishing

• EU funded project aims to improve quantity, visibility and usability of high-quality OA content.

• Uses Driver infrastructure

European Institutional repositories

• EU funded project DRIVER aims to establish a cohesive, pan-European infrastructure of Digital Repositories

• Driver Inventory study (2006): 230 European institutions have a repository (=10-20%?)

• These cover 37 % of institutions’ recent publication output (based on questionnaire)

• 18 % of materials is books /chapters

• 30 % of materials is SSH

Driver Inventory study (2006)

Type %

Textual research materials

Meta-data only 61 %

Full texts 29 %

Non textual materials

Images, videos, music, primary data sets

5 %

Other Learning materials, student papers

5 %

Institutions need research management systems for ….

• National research assessment exercises (e.g., UK REF)

• Funding parameters at a national level

• International rankings and benchmarking

• Positioning in European Research Area

• Information for general public and clients

• Internal research management at central and departmental level

3.Actual use of bibliometric

indicators in research assessment

In progress

Spain:

ANEP, ANECA

Norway:

Norwegian modelObjective Evaluation /

promotion of individual scholars

Distribution of funds across institutions

Unit of assessment

Individuals Institutions

Indicators Nr papers weighted with impact factor

of JCR/WoS covered journals

Nr publications weighted with peer ratings of sources

and their publishers

Flanders: Distribution of funds across institutions

• Current funding parameters based on Web of Science

• Creation database of scientific-scholarly SSH publications per institution

• Institutions establish selection criteria for sources and document types

• Operational in 2011

4.Actual practices in dealing with

non-publication outputs

The UK Research RAE 2008

• Research: original investigation undertaken in order to gain knowledge and understanding

• Scholarship: creation, development and maintenance of the intellectual infrastructure of subjects and disciplines

The UK Research RAE 2008Panel O

Research outputs must be verifiable and relate to research, and may include:

• New materials• Devices• Images• Products and buildings• Intellectual property (patents or other forms)• Performances, exhibits or events• Work published in non-print media

RAE 2008 submissions: criteria

• Submitted research outputs must be verifiable and relate to research

• Non-text output: evidence of their dissemination in the public domain and of their research contents are required

Australia: ERA non-publication outputs

Applied research• Registered designs (if clear link with

research)

Esteem:• Editorial roles and contribution to

prestigious works of reference• Curatorial role of a prestigious event

Yardsticks

• Formal yardsticks for scholarly non-publication output are missing in RAE and ERA

• Prestigious events, performance outlets , media might be ranked according to prestige

5.Options for creating a

comprehensive database of SSH outputs

Options

1. Combine European special SSH bibliographies

2. Further enhance SSH coverage of Web of Science and / or Scopus (books!)

3. Stimulate creation and standardization of institutional research management systems

4. Further stimulate institutional repositories

Four options: discussion

Feasibility Problems

Combine special European SSH bibliographies

Costly; current databases to be

restructured

Expand SSH coverage of WoS and Scopus

Feasible as long as there is a market for it

Sufficiently comprehensive?

(books!)

Stimulate institut. res. management systems EU initiative fits

into concept of Europ Res Area

Pre-selection of public. types necessary?Stimulate

institutional repositories

Further research

• Exploration of Google Scholar / Book Search

• Further information-scientific and sociological studies of SSH fields

• How to discriminate between scholarly and non-scholarly sources (pre-selection criteria)

END