1

Galway 1st Feb 2012

Where next for the ABS Guide?

Michael Rowlinson (Queen Mary University of London)

Based on Charles Harvey Aidan Kelly HuwBased on Charles Harvey, Aidan Kelly, Huw Morris, & Michael Rowlinson eds.

The Association of Business Schools Academic Journal Quality Guide Version 4. March 2010.

A brief history of the ABS Guide

• Originated as the Harvey Morris guide

• Adopted by the ABS

• Produced annually up to 2010, now in version 4

• Revamped version to appear biannually from 2013from 2013

• 90,000 downloads from 100 countries in 2010

2

Purposes of academic journals

• A stamp of quality. Publication in a peer reviewed journal indicates an article is worth jreading.

• To disseminate research findings and stimulate debate.

• An official record of knowledge and information. A definitive version of an article that can be catalogued, indexed, accessed and cited.

• Managing intellectual property rights and permissions. Ensuring rights are properly assigned between authors and publishers so that permission for reproduction can be granted.

Functions of Journal Quality Lists

• Help authors decide what to read and where to publish especially for earlywhere to publish, especially for early career researchers, and inter-disciplinary research.

• Inform staffing decisions for appointment, promotion, and reward.I f lib h i d i i• Inform library purchasing decisions.

• To aid internal and external reviews of research activity and evaluation of outputs

3

Types of Journal Quality List

• Institutional lists• Derived lists extrapolated from ratings awardedDerived lists extrapolated from ratings awarded

in assessment and audit e.g. the UK RAE (Geary, Marriott & Rowlinson, 2004; Mingers, Watson & Scaparra 2009).

• Peer surveys made on the basis of assessment of peers in a field or sub-field.

• Citation studies based on the number of times an average article in a journal is cited.

• Hybrid lists (Harvey, Kelly, Morris & Rowlinson, 2010; Harzing, 2011).

Commonly perceived problems of Journal Quality Lists

• Wheat and chaff. Sometimes highly rated journals publish poor quality work and low rated j p p q yjournals publish high quality work. – But this is not commonplace and journals can move

up and down ratings• Special issues dilute editorial standards

– No evidence for 3 and 4 ABS rated journals• New journals get lower ratings

– Need for a track record• Citation impact factors are sufficient to measure

journal quality– Impact factors are problematic as a standalone proxy

4

Limitations of Citation Impact Factors as a measure of journal quality

• Incomplete coverage. 459/821 journals in ABS List 2010 have an ISI citation impact factor for 2008.

• Differences in epistemological traditions need to be• Differences in epistemological traditions need to be recognized. High citation impact results from repeated inclusion of articles in referential chains

• Herding means that academics associate their work with highly cited authors and highly cited journals, reinforcing perceived superiority of highly cited journals

• Content bias means that journals featuring survey articles lit reviews methodology and conceptualarticles, lit reviews, methodology and conceptual development have higher citation impact that journals publishing original research

• Game playing and differences in maturity between fields– Established fields such as economics have higher citation rates

than newer fields such as tourism and hospitality– The rules of the game in mature fields require citation of journals

in referential networks

Main features of the ABS List

• Provides an expert assessment of the quality of empirical and theoretical research articles typically

bli h d i d i j lpublished in an academic journal• A consensus list that rests on general acceptance of the

fair and balanced view taken by the editors and panel members

• Is not comprehensive but includes all core journals with 2 or more submissions for bus & mgt in UK RAE2008 plus limited othersRatings not fixed and will be reviewed biennially from• Ratings not fixed and will be reviewed biennially from 2010 (the next version is due 2013)

• The rating scale is not the same as the UK RAE and makes no reference to ‘national’ and ‘international’ standards

5

ABS List specification of Journal Quality Standards

QualityRating

Meaning of Quality Rating No. and (%)

4* World Elite Journals - a small number of grade four journals recognized worldwide as exemplars of excellence. Their high status acknowledged by inclusion well regarded international journal quality lists.

22(2.7%)

4 publish the most original and best executed research. Typically have highsubmission and low acceptance rates. Papers heavily refereed. Generally havethe highest citation impact factors within their field.

72(8.7%)

3 publish original and well executed research papers and are highly regarded.Typically have good submission rates and are very selective. Papers are heavilyrefereed. Have fair to good citation impact factors relative to others in their field,

230(27.9%)

but not all journals in this category carry a citation Impact factor.

2 publish original research of an acceptable standard. Well regarded journal in itsfield, papers are fully refereed according to accepted standards and conventions.Have modest citation impact factors or do not have one at all.

295(35.8%)

1 publish research of a recognized standard and are modest standard journalsWithin their field. Papers refereed relatively lightly according to accepted conventions. Few journals in this category carry a citation impact factor.

204(24.8%)

Proportion of journals at each ABS grade

Grade ABS 2008 ABS 2009 ABS 2010

4 10.3 10.5 11.4

3 24.9 24.4 27.9

2 37.8 37.4 35.9

1 27.0 27.8 24.8

Total 1017 1033 825

6

The ABS List has a highly consistent pattern of correlation with other ranking schemes, RAE 2008

outcomes and citation impact factors

Table 5: Mean scores on RAE 2008 data by ABS 2010

Grade Mean cites in

BMS Mean outputs

GPA Mean Overall

GPA

4 24.5 3.5 3.8

3 19.9 2.9 3.2

2 8.7 2.5 2.7

1 5.2 2.2 2.3

Total 12.7 2.6 2.9

Should the REF use The ABS List?

Taylor’s (fc) results from his analysis of the RAE 2008: ‘support the use of journal quality indicators in the pp j q yresearch assessment process. Requiring the panels to take bibliometric indicators such as journal quality scores into account should help not only to reduce their workload but also to mitigate the implicit bias indicated by the statistical analyses reported in this paper. This process would also be more consistent with the widespread use of journal quality indices such as the p j q yABS Journal Quality Guide by business schools in assessing the quality of their own research, as they did in preparation for the 2008 RAE.’

7

Should the REF use The ABS List?

According to Taylor’s (fc) analysis, the estimated bias in favour of the Russell Group was muchbias in favour of the Russell Group was much greater for the esteem score than for research outputs:

‘This may be a consequence of the fact that the existence of the ABS Guide could have helped to moderate the panel’s judgements of theto moderate the panel s judgements of the quality of a department’s research output, whereas it was less constrained in the case of judgements about a department’s esteem.’

Should the REF use The ABS List? (we was robbed in the RAE 2008!)

8

ABSv4 RAE08 outputs

233

21

120

140

160

3730

66

13 12

37

27

10

2 8

10

4

6

18

7

60

80

100

120

no

ou

tpu

ts

nonABS

ABS1

ABS2

ABS3

ABS4

3239

2436

23

3745

39

0

20

40

QM08 King's08 Leicester08 Royal Holl08 St Andrews08

RAE08 output profiles

35

3631 33 33

94

8 8 8

0 0 0 0 0

75

100

4247

33

42 41

35

50

U/C

1

2

3

4

14 13

28

17 18

0

25

Leicester 08 StAndrews 08 King's 08 QM 08 Royal Holl08

9

ABS scores

350

400

450

150

200

250

300

cum

AB

S s

core

s QM08

Kings08

Leicester08

RoyalHoll08

StAndrews08

0

50

100

0 5 10 15 20 25 30 35 40 45 50

staff nos

Issues to consider if the Guide is to become more European/International

• Separation from the UK Research Excellence Framework (the REF)– The REF provides a valuable publicly available data setThe REF provides a valuable publicly available data set

• Weighting of Web of Science citation impact factors– Difficulty of using Thomson Reuters proprietary data

• Distinctiveness from other rankings and ratings– e.g. Publish or Perish

• Identify or generate alternative sources of accessible non proprietary data

• Avoid being bound by an algorithm while not• Avoid being bound by an algorithm, while not succumbing to lobbying from special interests

• Facilitate validation of a wide variety of European and international business school missions

• Retain the principle of sub-field normalization, in recognition that measurements such as impact factors vary between fields

10

References

Geary, J., Marriott, L., & Rowlinson, M. 2004. Journal rankings in business and management and the 2001 Research Assessment Exercise in the UK. B iti h J l f M t 15(2) 95 141British Journal of Management, 15(2): 95-141.

Harvey, C., Kelly, A., Morris, H., & Rowlinson, M. 2010. The Association of Business Schools Academic Journal Quality Guide Version 4.

Harzing, A-W. 2011. Journal Quality List Thirty-eight Edition.

Mingers, J., Watson, K., & Scaparra, P. 2009. Estimating Business and Management Journal Quality from the 2008 Research Assessment Exercise in the UK, Kent Business School Working Paper Series.

Taylor, J. forthcoming. The Assessment of Research Quality in UK Universities: y , g yPeer Review or Metrics? British Journal of Management