109

THE RANKING OF AGRICULTURAL ECONOMICS JOURNALS

Michael Burton and Euan Phimister*

This note reports a ranking for agricultural economics journals, based on citation data for the years 1986 and 1991. The rankings are generated using data envelopment techniques, and form part of a wider evaluation of economics journals. The results indicate that the American Journal of Agricultural Economics and Journal of Agricultural Economics and Land Economics are consistently ranked highly, irrespective of time period or citation measure used. They also indicate that the performance of the agricultural economics journals considered is broadly comparable with other applied economics journals.

1. Introduction There is a growing literature on techniques for ranking academic journals based on citation data (Doreian, 1987; Diamond, 1989; Hodgson, 1993; Bush et al., 1974; Liebowitz and Palmer, 1984; Todorov and Glanzel, 1988; Laband and Piette, 1994). In part this work is driven by an interest in the process by which knowledge is produced and distributed but such rankings are increasingly being used to evaluate both individual researchers and institutions. AsTijseenetal. (1987) note,

“...extreme opinions are to be found on the use of citations in constructing quantitative measures of science, extending from those who would employ citation-based measures as a reliable indication of the peer recognition of scientific work to those who only advocate the use of citations in literature search.”

Concern with the use of citations as an indicator of quality are largely related to issues such as the existence of negative citations, irrelevant citations and citation-database limitations. While citation data obviously provide an imperfect measure of journal quality, they are one of the few objective pieces of information on which journals may be judged. Further, if rankings based on citations are to be generated we feel that it is important that the techniques used to rank publications should be transparent, and issues such as combining different characteristics of the citation data systematically treated. * The authors are members of the Centre for Agricultural, Food and Resource Economics, School

of Economic Studies, University of Manchester, UK. The permission of the Institute for Scientific Information to access data from the Social Science Citations Index@ is gratefully acknowledged.

Joiirnal o f Agricultural Economics 17(1)(1996)109-114

110 M. BURTON AND E. PHIMISTER

Burton and Phimister (1995) present a methodology for ranking journals based on multiple measures of journal performance, using the techniques of Data Envelopment Analysis. That ranking was for all economics journals, and details of only the top fifth (27 journals) were reported. The purpose of the current note is to report the rankings of agricultural economics journals within that overall ranking, and compare them with other applied economics journals.

2. Data and Methodology The standard source for citations data is the IS1 Social Science Citations Index@ (SSCIB) and its associated reports. One can differentiate between three basic types of information: the total number of citations made to a journal during a year, the number of self-citations made by a journal during a year, and the temporal distribution of the articles that are cited. The first of these would normally be considered a positive measure, but less so if the second is high. For the third category, it is usually considered advantageous if there are high levels of citations to articles that have been recently published, as this indicates that the journal’s output is quickly taken up and incorporated into the work of others: the journal is at the cutting edge of the discipline. This measure is usually reported as an impact factor, C,t-n/Ai-n,. where the numerator is the number of citations made in year t to articles published by journal i in year t-n, and the denominator is the total number of articles published by journal i in year t-n. The analysis that follows is based on these three measures (or variations on them), based on the list of economics journals reported in SSCIB for the years 1986 and 1991. The number of journals available differs over time (there are 128 in 1986 and 133 in 1991) and this highlights the problem that the IS1 database is incomplete, both with respect to economics journals in general, and agricultural economics journals in particular. For example, the ex-regional US journals are not included, nor the European Review of Agriculfural Economics.

Data envelopment analysis can be used as a method of generating a ranking of journals based on multiple measures of’the journals output. Given M measures of output and N journals the D E A problem for the kth journal can be formally stated as

max E, = !iwixik

subject to

i Wi

1 2 !iwixii 3 0 j = l...N i

where xik is the ith output measure for the kth journal and wi is the weight attached to that output. Once the definitions of output to be used are selected, each journal can be interpreted as choosing to weight the outputs such that the value of their index Ek (which one could interpret as a form of efficiency index) is maximised. The constraints imply that, given the weights chosen by journal k, the index value E for all journals has to lie between zero and unity. The

THE RANKING OF AGRICULTURAL ECONOMICS JOURNALS 111

repetition of this procedure for all journals generates a ranking based on the value of El, achieved by each journal (see Seiford (1990) for an extensive bibliography of DEA applications).

Alternatively, an integer programming approach can be used, which allows each journal to select the weights that explicitly maximise the ranking rather than the index, but for the data sets considered this ha5 little effect on the rankings (Burton and Phimister, 1995), and the results are not reported here.

Choosing the definition of the output measures is clearly important. Here we report the results from using two definitions. In the first, all data are reported in levels: i.e. for the 1986 data set the three measures used are total self-citations, total citations by other journals and citations by other journals to articles published in 1984 and 1985. The latter is the numerator of an impact factor based on the previous two years publications. In order to avoid the criticism that a journal can have a high ranking solely by self-citation, the weight attached to self-citation is constrained to be less than or equal to the weight attached to total other citations. Using this data one would expect that journals that have been published for a long time, have a large number of issues and a large number of articles per issue will be able to achieve high rankings as compared to smaller, more recently established journals. Thus a ranking based on this data set will be a ranking of the journal itself, rather than the ranking of the average article in that journal. In order to accommodate this size effect an alternative data set has been constructed. This is defined as total other citations and total self-citations made in 1986 to articles published in the period 1977- 1986 (to control for longevity) and citations to articles published in 1984 and 1985, but then all three measures normalised by the number of articles published in 1984 and 1985 (to control for volume size). This second data set is expected to generate a ranking based on the quality of the average article published in it (although it will still penalise those very recently established journals). Two further data sets have also been constructed based on analogous definitions, but for citations made in 1991.

There are seven agricultural economics journals identified in the IS1 list: The American Journal of Agricultural Economics (AJA E), Land Economics, Journal of Agricultural Economics (JA E), Canadian Journal of Agricultural Economics, Food Policy, Agricultural Economics Research and The Australian Journal of Agricultural Economics. The choice of journals to be included is proscribed entirely by their presence in the IS1 data base. A number of citation measures for 1991 are reported in Table 1 , for these seven journals.

Table 1 Selected Citations Data for Agricultural Economics Journals, 1991 Total Self- Impact Source

Citations Citations Factor Items' Ag Econ Research 41 1 0.085 35 Am J Ag Econ 1318 435 0.647 283 Aus J Ag Econ 60 10 0.194 31 Can J Ag Econ 76 30 0.114 176 Food Policy 94 23 0.554 65 J Ag Econ 144 40 0.591 66 Land Econ 443 60 0.573 82 * Number of articles published. 1989-90. Source: Garfield (1992).

112 M. BURTON A N D E. PHIMISTER

Table 2 DEA Results Using Levels Citation Data: Agricultural Economics Journals I986 1991

Ek Rank Ek Rank Am J Agr Econ 0.166 14 0.208 13 Land Econ 0.098 32 0.073 40 J Ag Econ 0.037 57 0.042 62 Can J Ag Econ 0.019 76 0.017 85 Food Policy 0.017 83 0.041 65 Ag Econ Research 0.009 102 0.006 114 Aus J AE Econ 0.007 108 0.009 107

~~ ~~ ~

Note: Rank is the overall position within the list of 128 economics journals in 1986, and to 133 economics journals in 1991, based on the value of Ek.

Table 3 DEA Results Usine Normalised Citation Data: Aericultural Economics Journals 1986 1991

Ek Rank Ek Rank Land Econ 0.148 36 0.096 40 J Ag Econ 0.092 56 0.069 58 Am J Ag Econ 0.086 58 0.092 49 Ag Econ Research 0.058 74 0.019 105 Can J Ag Econ 0,050 80 0.013 112 Food Policy 0.048 81 0.071 55 Aus J Ag Econ 0.045 84 0.025 98

See note to Table 2.

3. Results The results from applying DEA to the four alternative data sets outlined in the previous section are summarised in Tables 2 and 3. These give the ‘efficiency’ scores of the journals and their position in the overall ranking of the economics journals. As one may expect, the AJA E scores highly using the levels data, and in fact is on the margin of the top ten per cent of the entire economics list. The ordering of the journals is similar over the two time periods, with only Food Policy moving up the list. It is notable, however, that its efficiency score has increased markedly over the period, bringing it very close to the JAE. Using the normalised data set the AJAE no longer benefits from its relatively large size (it has 283 source items in 90-91, compared to 82 for Land Economics and 66 for the JAE), and hence moves from the top of the list of atricultural economics journals, but still achieves a high position. The dispersion of the ‘efficiency’ indices is reduced with the normalised data, as one would expect. Table 4 re orts the weights generated by the DEA procedure for each journal,

three ‘output’ components, which supports the use of a technique which can synthesise a variety of output measures. The difference in scale of the weights between the levels and normalised data merely reflects the differing scale of the data in each case.

In comparing the agricultural economics journals with other applied journals one has a problem in that there is no other single area of study which is so well represented in the IS1 data base. We have selected seven applied journals (Journal of Labour Economics, Journal of Development Economics, Journal of Industrial Economics, Journal of Environmental Economics and Management (JEEM), Journal of Accounting and Economics, Applied Economics and Journal of Heath Economics), and Tables 5 and 6 report the

for the 19 !Y 1 data set. It is notable that most journals attach some weight to all

THE RANKING OF AGRICULTURAL ECONOMICS JOURNALS 113

Table 4 Weights Attached to Each Outout Measure. 1991 Data

w l Ag Econ Research 1.61E-4 Am J Ag Econ 4.39E-5 Aus J Ag Econ 1 S7E-4 Can J Ag Econ 4.39E-5 Food Policy 4.39E-5 J Ag Econ 4.39E-5 Land Econ 4.39E-5

Levels Data w2

1.86E-5 4.398-5 1.578-4 4.39E-5 4.39E-5 4.39E-5 4.39E-5

Normalised Data w3 w l w2 w3

0.009 0.009 0.145 0.001 0.009 0.009 0.145

0.009 0.009 0.145 0.001 0.009 0.009 0.145 0.001 0.007 0.007 0.156 0.001 0.007 0.007 0.156 0.001 0.009 0.009 0.145

~~~~

Note: The columns wl, w2 and w3 for the Levels Data are the weights attached to total other citations, total self citations and citations to 89-90 articles respectively. The Normalised Data weightsare related to analogous, adjusted data, asdetailed in the text.

Table 5 DEA Results Using Levels Citation Data: Applied Economics Journals 1986 1991

E k Rank Ek Rank J Labour Econ 0.133 19 0.096 33 J Development Econ 0.098 31 0.071 42 J Industrial Econ 0.068 40 0.052 53 J Env Econ Man 0.066 41 0.084 35 J Account Econ 0.061 44 0.082 37 Applied Econ 0.052 49 0.054 51 J Health Econ 0.038 56 0.058 50

See note to Table 2.

ranking and efficiency score for the normalised and levels data set, for both 1986 and 1991. On the whole, the agricultural economics journals perform relatively well, especially as the other applied journals may be considered, individually, the leading journals in their respective areas, with Land Economics and the AJAE on a par with JEEM, and Food Policy and JAE comparable with the Journal of Development Economics and the Journal of Industrial Economics.

In terms of changes over time, the ranking of the applied journals is relatively stable in both Tables 5 and 6 . While this is generally true far the agricultural economics journals when ranked using the levels data (Table 2) there is more significant movement in the normalised data, with a divergence in the ranking. Both the AJAE and Food Policy improve their position, while the Canadian Journal of Agricultural Economics, Australian Journal of

Table 6 DEA Results Using Normalised Citation Data: Applied Economics Journals I986 I991

Ek Rank Ek Rank J Account Econ 0.360 10 0.186 19

J Env Econ Man 0.164 30 0.098 37 J Health Econ 0.159 33 0.154 22 J Industrial Econ 0.135 39 0.074 51 J Development Econ 0.124 42 0.077 50 Applied Econ 0.052 78 0.025 92

J Labour Econ 0.299 13 0.201 15

See note toTable2.

114 M. BURTON AND E. PHIMISTER

Agricultural Economics and Agricultural Economic Research fall dramatically. This may suggest an increasing polarity within the discipline.

4. Conclusions This note has reported information on the ranking of agricultural economics journals within the context of a ranking of all economics journals. If one uses gross measures of citations then the AJA E appears high in the rankings relative to all economics journals. If one corrects output by a measure of the size of the journal then the ordering changes, with Land Economics moving to the top of the list, and the AJAE moving down the overall rankings. Both the JAE and Food Policy also score relatively highly within the group, with a notable increase in the latters’ performance over the period 1986 to 1991. Overall, the performance of the agricultural economics journals is broadly comparable to the applied economics journals considered, although there is some tendency for increased polarisation in the former. If one is prepared to accept the value of citations as an indicator of quality, the methodology used here provides a transparent way to rank journals. However, an important caveat has to be borne in mind: if a journal is not included in the IS1 data base, then it cannot be ranked, irrespective of its quality.

References

Burton, M. P., Phimister, E. (1995). Core Journals: A Reappraisal of the Diamond List. Economic

Bush, W. C . , Hamelman, P. W. and Staaf. R. J. (1973). A Quality Index for Economic Journals.

Diamond, A. M. (1989). The Core Journals in Economics. Current Contents, 21,4-12. Doreian, P. (1987). A Revised Measure of Standing of Journals in Stratified Networks.

Scientometrics, 11,71-80. Garfield, E. (1987), (1992). Social Science Citation Index Journal Citation Reports, Philadelphia:

Institute for Scientific Information. Hodgson, G. (1993). The Ranking of Heterodox Economics Journals in the UK Research Funding

Excercise. European Association for Evolutionary Political Economy, Mimeo. Laband. D. N. and Piette, M. J. (1994). The Relative Impacts of Economics Journals: 1979-1990.

Journal of Economic Literature, 32,630-666. Liebowitz, S . J. and Palmer, J. P.(1984). Assessing the Relative Impacts of Journals. Journal of

Economic Literature, 22,77-88, Seiford, L. M. (1990). A Bibliography of Data Envelopment Analysis (1978-90). Version 5.0,

Technical Report. Department of Industrial Engineering, University of Massachusetts, Amherst.

Tijseen, R. J. W., Leeuw, J. De and Van Raan, A. F. L. (1987). Quasi-correspondence Analysis on ScientometricTransaction Matrices, Scientometrics, 11,351-366.

Todorov, R. and Glanzel, W. (1988). Journal Citation Measures: A Concise Review. Journal of Information Science, 14,47-56.

Journal, 105,361-373.

Review of Economics and Statistics, 56, 123-5.