www.esf.org/erih

Evaluation, Quality and Impact of

Open Access Publications

• Chair: Julianne Nyhan (European Science Foundation)

• Susan Murray (Director, African Journals online)

• Christiane Fritze (Berlin-Brandenburg Academy of Sciences and Humanities)

www.esf.org/erih2

European Reference Index for the Humanities – ERIH

Better access to and assessment of Humanities research in Europe and beyond

Dr Julianne Nyhan

erih@esf.org

www.esf.org/erih3

European Humanities research– the challenges

• Difficulty to identify and compare Humanities excellence- vis-à-vis other sciences- across all languages at a supra-national (European) level

• Poor international visibility of research output in national languages(considered as necessary vehicles for Humanities research)

• Multiplicity of formats for research output (monographs, edited volumes, journals [less than 1/3 !], conference proceedings, web-based content and data, outreach)

www.esf.org/erih4

ERIH – the origins

SCH Workshop on evaluation of scientific production in Humanities (Budapest 2001):

• Existing citation indices (AHCI, SSCI) have unsatisfactory coverage of European Humanities research

• Need for a European citation index for the Humanities as additional tool for research assessment

• ESF requested to compile lists of reference journals in the first instance

www.esf.org/erih5

ERIH – objectives / methods

• Benchmarking tool for comparisons at aggregate level

• Focus on format used in other sciences (journals)> achieve a degree of initial comparabilityBUT: methodology needed for other formats

• Encourage ’best practice’ in the publication of journals in the HumanitesPeer-review: discursive identification of excellence> across all fields of the Humanities> across Europe < national consultations

• European Research Area:Improve access to European Humanities research across all languages (basis for VLE?)

ERIH as a tool for the strengthening of Humanities scholarship in Europe created by scholars for scholars

www.esf.org/erih6

ERIH – process

Set up Steering Committee• Define the first 15 disciplines to be included• Define categories of journals: A, B, C• Compile guidelines for inter/national panels • Approves Expert Panels (members suggested by

MO’s, SCH, StComm, in the future: ENCoPs)• Validates work of expert panels

Produce and process material• Input from ESF MO’s (national panels / reference

tools) according to guidelines • Initially 15 international panels who compile

categorised lists of good peer-reviewed research journals in 15 disciplines

www.esf.org/erih7

ERIH – current coverage

Index Coverage (publication criteria):Peer-reviewed journals; ISSN; bibliographic datafor cited references; author data

Index coverage (discipline criteria):15 subject groups defined, some covering areasbetween Humanities and Social Sciences(anthropology; education; psychology)

Index coverage (internationality criteria):Cat. A (<25%)/B: international journals (defined by status:

reputation, readership, authors, citations),Cat. C: important European journals with more regional

circulation (ESF MO Europe: not Russia)

All journals included – whether A-B-C - to be considered good research journals

www.esf.org/erih8

ERIH – categories

Precision of criteria for international journals• a genuine, varied and regular international cohort of

contributors and readership• a consistently high-quality scholarly content• a broad consensus within the field concerning international

status and visibility, insofar as possible• a quality control mechanism, normally through peer-review

Plus some, not necessarily all, of the following characteristics:• Active international advisory board• Openness to unsolicited contributions• Highly discriminating and selective in the choice of articles

published• Publication on time and to an agreed schedule

Degree to which these characteristics are applied could be added to the criteria distinguishing journals in categories ‘A’ and ‘B’

www.esf.org/erih9

ERIH – current disciplines/panels

• ANTHROPOLOGY• Archaeology• Art and Art History• Classical Studies• Gender Studies• History• History & Philosophy

of Science• Linguistics

• Literary Studies• Musicology• Oriental & African Studies• Pedagogical & Educational

Research• Philosophy• Psychology• Religious Studies

& Theology

Possible Additions for updates:

•Archives, Library & Museum Studies•Film, Media & Cultural Studies•General Humanities

www.esf.org/erih10

ERIH – towards the « initial lists »

• 2003/04: MO’s provide input based on previous national consultation (panels; reference tools)

• 2005/II: Expert Panels work- define field / remit (“scope notes”),- analyse and assess input received,- suggest circles of consultation,- consult where gaps are identified (eliminate / add),- discuss categories (multiple listing; single listing),- produce draft list, explain methods and problems.

• 2006 (mid): Wide consultation of- ESF MO’s,- (European) subject associations,- specialist research centres

• 2007 / 2008: Publication of the lists in three batches • 2008: Update / feedback

Online form (quantitative information): contact publishers, editors, European subject associations, national subject associations (through ENCoPs)

www.esf.org/erih11

Peer review at work…

Layers:• Input: National panels / scientific communities• Selection: Expert Panels define scope, analyse and

assess input, produce lists• Consultation: MOs, subject associations (European

level and some national), specialist research centres• Calibrate/harmonise: ERIH Steering Committee• Feedback: open process leading to updates in 2008Challenges:• Wide differences in quality of lists received from MOs• Domain-specific (e.g.: cult./soc./evol. anthropology;

philosophy and ethics)• Language (e.g.: “forum languages” in art history;

“international languages” in Oriental & African studies

• Some panels more reluctant to overrule authority of (own) national panels

• Outside peer pressure during consultation phases

www.esf.org/erih12

ERIH insights – quality through peer review?

• Peer review recognised as the only practicable method in basic research (standard method used in scientific communications themselves)

• Peer review can overrule weight of numbers for better (detect originality) or for worse (defend conservatism)

• Peer review introduces comparability into discussions of national discourses in Humanities scholarship

www.esf.org/erih13

Evolution of ERIH Lists

1st batch of published initial lists are highlighted in yellow        

DisciplineInitialsubmission

1st draft lists Consultation 1a Consultation 1b Initial lists

    Mar-Apr 2006 Spring 2006 Summer 2006 2007

Anthropology (social and evolutionary) 752 154 133 17 242

Archaeology 1310 524 290 25 419

Art and Art History 938 445 292 16 472

Classical Studies 619 256 321 11 251

Gender Studies 155 181 37 6 119

History 1419 874 508 71 907

History and Philosophy of Science 806 145 64 4 166

Linguistics 1093 680 391 34 586

Literature 1453 1481 706 10 802

Music and Musicology 204 n/a 187 4 166

Oriental and African Studies 196 588 386 14 505

Pedagogical and Educational Research 666 404 271 92 470

Philosophy 658 320 153 22 305

Psychology 1198 1201 159 4 634

Religious Studies and Theology 745 n/a 580 10 371

www.esf.org/erih14

ERIH – Update in 2009

• Recomposition of Expert panels: panel rotation mechanism and inclusion of new experts, document setting out methodology of recomposition

• Summary and Process of Methodology that underpinned work on ERIH “initial Lists”

• Summary and Process of Methodology that underpinned work of ERIH Steering Committee

• Panel meetings to take place from October 2008 – February 2009. ERIH Lists to be published incrementally in 2008 and 2009

www.esf.org/erih15

ERIH – Update in 2009

Feedback on ERIH “initial List” Anthropology (2007/08)

• Of crucial importance in the updating of the “initial Lists”

• ERIH office has been contacting Editors and Publishers of relevant journals to ask that they submit quantitative data about their journals

• ERIH Feedback form: https://www2.esf.org/feedback.asp

• Difficult process because no central database of contact details exists (info@email.org etc)

• Only 26% of journals on “initial List” have been commented upon by Editors / Publishers

www.esf.org/erih16

ERIH – futures?

Open Source

• - ERIH is creating consensus on excellence

•             (> priority list for inclusions)•             - ERIH includes publications of

ESF MO’s, incl. learned societies •             (> institutional incentive?)•             - link of access / assess may

create incentive to use OA•             [perhaps further stimulated by

universities through support for IR’s]

www.esf.org/erih17

ERIH – futures?

Open source futures?

• - ERIH aims to create a consensus on excellence

            (> priority list for inclusions)•             - ERIH includes publications of

ESF MO’s, incl. learned societies             (> institutional incentive?)            - link of access / assess may

create incentive to use OA            [perhaps further stimulated by

universities through support for IR’s]

www.esf.org/erih18

ERIH – futures?

• Open Questions:•             - Measuring influence of OA journals when

not cited: downloads and citation refer to different phases in research process

•             - OA portal appropriate for a highly diverse field?

            - How to link OA to IR’s [additional institutional stakeholders]?

•             - OA as linked to budget model of learned societies?

•             - How to integrate current knowledge about delayed impact in most Humanities [6-month-rule not applicable?]?