Research Quality, Bibliometrics and the Republic of Science

Michael A Peters

Presentation at the SymposiumHigher Education: Quality Assurance Widens

Democracy or Not?University of StellenboschThursday 10 August, 2006

Outline

1. Ideologies of Knowledge & Knowledge Cultures(i) English Renaissance: Forbidden Knowledge – Marlow’s Dr Faustus(ii) French Enlightenment – Encyclopedic Knowledge – Diderot’s

L'Encyclopédie(iii) Postmodern Knowledge Economy - Thomson’s ‘total information

solution’

2. Byblos, Bibliographies & Bibliometrics(i) Journals, Journology & The Origins of Peer Review(ii) Bibliometrics and the Architecture of Global Science(iii) Research Quality and the Development of National Systems

3. Peer Review, Bibliometrics & the Governance of Science

(i) Quality Assurance Replaces ‘Truth’ as Core Commitment of Post-normal Science & the Case for ‘Extended Peer Review’

(ii) The Centrality of Peer Review to the Republic of Science and the Shift to Bibliometrics

(iii) The Limitations of Citation Analysis

1. Ideologies of Knowledge & Knowledge Cultures

(i) English Renaissance: Forbidden knowledgeO lente, lente, currite noctis equi:The stars move still, time runs, the clock will strike.The devil will come, and Faustus must be damned.O, I'll leap up to my God: who pulls me down?See, see where Christ's blood streames in the firmament;One drop would save my soule, half a drop, ah, my Christ! I'll burn my books! Ah, Mephistophilis. --FaustusChristopher Marlowe, The Tragicall History of D. Faustus, 1589

The Tragicall History of D. Faustus

The play trades on the theme of knowledge in its relation to sin, redemption and damnation at the heart of Christianity’s understanding of the world and also the conflict between medieval and renaissance values, knowledge and power as corrupting forces, magic and the supernatural, and the divided nature of man. The story of Faustus then exemplifies the new spirit of inquiry that characterizes Renaissance science and the intellectual ambition for ultimate knowledge and the power it brings against a specifically Christian ideology that warns about the dangers of seeking forbidden knowledge.

Forbidden Knowledge• Marlowe was a child of the English Renaissance and the Reformation, which

was also that troubled period called by the great scholar Dame Frances Yates, "the false dawn of the Enlightenment," which was doomed to suppression and delay. He shared his birth year,1564, with Galileo (and with Shakespeare). It was a dangerous time in which to express an eager interest in the new scientific discoveries that were exciting the minds of intellectuals all over Europe.

• In England Sir Walter Raleigh and the young Earl of Northumberland, Henry Percy (also born in 1564), led a group of intellectuals, a select band of advanced thinking noblemen, courtiers and educated commoners, including mathematicians, astronomers, voyagers who had explored the New World, geographers, philosophers and poets. They formed an esoteric club nicknamed "The School of Night"   which met secretly to discuss this forbidden knowledge, always "behind closed doors." Marlowe became a member of this close circle, who were called Free-Thinkers and were all stigmatised as "Atheists" in order to blacken them in the eyes of the ignorant.

• Francis Bacon’s (1561-1626) utopian inductive philosophy of science (The Advancement of Learning, Novum Organum, The New Atlantis) strongly influenced by alchemy & magick but also based on observation and experiment, became the guiding ethos of the Royal Society established in 1660 as ‘The Empire of Learning’ led by Robert Boyle, Sir Christopher Wren, Sir Robert Moray. The Philosophical Transactions of the Royal Society became the first academic journal, its first edition edited by Henry Oldenburg in 1665. Peer review was introduced as a practice of the journal thereafter.

(ii) Encyclopedic Knowledge

Nothing is more contrary to the progress of knowledge than mystery.... If it happens that an invention favorable to the progress of the arts and sciences comes to my knowledge, I burn to divulge it; that is my mania. Born communicative as much as it is possible for a man to be, it is too bad that I was not born more inventive; I would have told my ideas to the first comer. Had I but one secret for all my stock in trade, it seems to me that if the general good should require the publication of it, I should prefer to die honestly on a street corner, my back against a post, than let my fellow men suffer.- Denis Diderot

Diderot’s L’Encyclopédie

Denis Diderot with Jean Le Rond d'Alembert was the chief editor of L'Encyclopédie which was published between 1751 and 1772 in 17 volumes of text and 11 volumes of engravings, representing perhaps one of the greatest achievements of learning of the Enlightenment. The Encyclopédie outlined the then present state of knowledge about the sciences, arts, and crafts, and was explicitly designed to make the knowledge possessed by the few accessible to the many. Most of the 71, 818 articles in the Encyclopédie were written by Diderot and d'Alembert but also included prominent thinkers of the day including Rousseau, Voltaire, Montesquieu, Baron d'Holbach, Necker, Turgot, and Buffon.

Encyclopedic Knowledge Diversity

The notion of the encyclopedia, originally from a Greek word meaning literally ‘a general education,’ represents the attempt to collect all of the world’s knowledge in a single system and arguably dates from the 37 volumed work of Pliny’s Naturalis Historia in the first century CE. Many other attempts at this total knowledge system followed: Cassiodorus’ Institutiones (560 CE), the first catholic encyclopedia that inspired St Isidore of Seville’s Etymologiae (636); Bartholomeus de Glanvilla’s De proprietatibus rebus (1240), probably the most widely read encyclopedia in the Middle Ages; Abu Bakr al-Razi’s encyclopedia of science, Al-Kindi’s (801-873) corpus of over 250 works in music, philosophy, astronomy, medicine, mathematics and geography; the Yongle encyclopedia of the Ming dynasty consisting in 11,000 handwritten volumes completed in 1408.

History of the Encyclopedia

The history of the encyclopedia indicates something about the organization of knowledge, the history of science and technology, the development of library science and information, the place of reference works per se, and imbued with a conscious ideology or philosophy, invariably was tied to new approaches to knowledge involving its democratization. Diderot’s aim, for instance, was to ‘change the common way of thinking’ through the expansion of knowledge and the development of critical modes of thought.

Encyclopaideia, Philosopedia, or

Macropedagogy? Of all pedagogical systems and reference works -- such as dictionaries, almanacs, gazetteers, atlases and directories -- the encyclopedia was the only one to aim at a total comprehensive and self-contained system, although there was no one set of principles guiding the method for classification of entries or arrangement of contents. Classifications varied considerably in the period before the alphabetical arrangement of entries was introduced through a standardization that came with printing, reflecting the working epistemologies of the age and the evolution of disciplines and the formation of knowledges.

Ideology of the Encyclopedia

Diderot and D’Alembert developed their underlying epistemology governing their Encyclopédie under the influence of Locke and Condillac: a rational and positivistic classification of existing knowledge based on the assumption of the unity of theory and praxis. Diderot planned to provide complete alphabetical treatment of the whole field of human knowledge from the standpoint of the "Enlightenment" and the contributors included Montesquieu, Voltaire, Rousseau, Turgot and Condorcet. The 28 volumes of the Encyclopédie completed between 1751-72 were not simply a repository of human knowledge but also a polemic from the viewploint of the French Enlightenment and as such included a strong emphasis on democracy and equality, and a kind of tolerance of philosophical views that were essentially subversive of the established order.

Peters & Ghirdaldelli (1999) http://www.vusst.hr/ENCYCLOPAEDIA/main.htm

Internet as EncyclopediaIn a few short years, the Internet has gone from being a specialist site for a few scientists and engineers to a place -- a soft and almost infinitely flexible architecture -- which incorporates a staggering variety of spaces: not just information exchange or reference banks or dictionaries but a vast conglomeration of different spaces, teeming with activity.The Internet, perhaps like the encyclopedia of old, now speaks to implicit knowledge formations, hybrid discourses, personal homepages that approve the existential conditions for aesthetically transforming oneself --chatgroups, bargain basements, advertising, new businesses and the whole vista of bookish elements now chaotically "shelved" alongside video clips, images and other non-textual items. For the first time, the desire of the medieval or Enlightenment encyclopedists of bringing together the entire scope and corpus of human knowledge and information looks more than a possibility rather than simple an historical reflection of the vanity of a bygone age

2. Byblos, bibliographies, bibliometrics

Byblos - 37 kilometres north of Beirut – was the commercial center of the papyrus trade; the Greek name for the Phoenician city Gebal (earlier Gubla) so called because it was through this port that Egyptian papyrus was imported into Greece. Byblos is also directly associated with the history and diffusion of the Phoenician alphabet.Bibliography (1) Publication in either printed or electronic format, usually arranged by author, date or subject, which offers a listing of works produced either by one author, in one subject, in one place, or during one period. (2) A list of works consulted in the composition of a book, article, or assignment.www.library.nuigalway.ie/help/jargon/ Bibliometrics is a type of research method used in library and information science. It utilizes quantitative analysis and statistics to describe patterns of publication within a given field or body of literature. Researchers may use bibliometric methods of evaluation to determine the influence of a single writer, for example, or to describe the relationship between two or more writers or works.

Bibliometrics and the Changing Architecture of

World Science • It is only since the 1960s with the development of research

evaluation and increasing sophistication of bibliometrics that it has been possible to map the emerging economy of global science, at least on a comparative national and continental basis. The Science Citation Index provides bibliographic and citational information from 3,700 of the world’s scientific and technical journals covering over one hundred disciplines.[1] The expanded index available in an online version covers more than 5,800 journals. Comparable ‘products’ in the social sciences (SSCI) and humanities (A&HCI) cover, respectively, bibliographic information from 1,700 journals in fifty disciplines and 1,130 journals. Additionally, the Web of Science covers more than 5,800 scientific journals.

• Thomson’s ‘scientific products’ provide ‘a total information solution is one that enables users to effortlessly navigate between essential research information sources including …Web of Knowledge.., full-text documents hosted by primary publishers, a growing list of key databases, and other options such as online public access catalogs’ (my emphasis).

Thomson Scientific – An Information Utility

The SCI is owned by Thomson Scientific and together with SSCI, A&HCI, and the Web of Science http://scientific.thomson.com/products/sci/. Thomson Scientific is part of the Thomson Corporation which advertises itself as ‘a leading global provider of integrated information-based solutions to business and professional customers.’ It is one of the new leading information utility corporations with some $8.5 billion in revenues from legal & regulatory, learning, financial, and scientific and healthcare global market groups. Beginning in newspapers in Canada in the 1930s Thomson expanded rapidly into TV in the 1950s, magazines and books in the 1960s, as well as travel, oil & gas, specialized information services, textbooks and technical books, legal publishing, and market data systems in subsequent decades, acquiring the Institute for Scientific Information (which was established by Eugene Garfield in 1958 in Philadelphia) in 1992. During the 1990s and 2000s Thomson acquired 30 major information-related companies.

Source: http://scientific.thomson.com/products/sci/.

Growth of PeriodicalsUlrich’s Directory of International Periodicals lists nearly 165,000 journals from 78,000 publishers. However, most of the significant information produced each year appears in only a small portion of them. This principle is referred to as Bradford’s Law. It states, in essence, that an essential core of journals forms the literature basis for all disciplines and that most of the important papers are published in relatively few journals. Recent citation analyses have shown that as few as 150 journals account for half of what is cited and about one quarter of what is published. A core of approximately 2,000 journals now account for about 85 percent of the world’s published articles and 95 percent of the world’s cited articles. Only 40 percent of all published papers receive more than one citation. About 25 percent of all published papers will never be cited and less than 17 percent are cited 17 or more times.

Hattendorf Westney (1998)

World Scientific Production(1981-2000)

In 2000 the SCI included a total of 584,982 papers, representing a 57.5% increase from 1981, when 371,346 papers were published worldwide. Authors with addresses in developed countries wrote 87.9% of the papers in 2000, a decrease from 93.6% in 1981. Developing countries, on the other hand, saw a steady increase in their share of scientific production: from 7.5% of world papers in 1981 to 17.1% in 2000… Since 1981 the world map of publications changed significantly. North America lost the lead it had in 1996, and in 2000 produced 36.8% of the world total, a decrease from 41.4% in 1981. The opposite trend can be found in the European Union, which in 2000 published 40.2% of the world total, up from 32.8% in 1981. Japan went up from 6.9% to 10.7% in 2000. Collectively this ‘triad’ has therefore maintained its dominance, accounting for 81% of the world total of scientific publications in 2000, up from 72% in 1981.

UIS Bulletin on Science and Technology Statistics (UNESCO 2005)

The Evolution of Science Indicators

The evolution of science indicators only merged relatively recently with research evaluation in the 1960s with the Frascati Manual developed by the OECD in 1963 for standardizing the collection of statistics on R & D, revised in 2002 to take account of the impact of globalization and the importance of innovation as a key element in the knowledge-based economy (OECD, 1963; 2002). As Leydesdorff (2005) comments mission-driven research was largely a result of WWII and it was ‘Vannevar Bush’s 1945 report to the US President entitled The Endless Frontier contained a plea for a return to a liberal organization of science’ based upon traditional peer review mechanisms (p. 1510). Most national science agencies followed the model of the US National Science foundation established in 1947 determining the eventual role that the federal government would play in supporting fundamental research at universities. Of course these developments were deeply implicated in Cold War politics and the Sputnik shock of 1957, when the Soviet science model seemed to produce better results.

The Emerging Political Economy of Global Science

The countries occupying the top eight places in the science citation rank order … produced about 84.5% of the top 1% most cited publications between 1993 and 2001. The next nine countries produced 13%, and the final group share 2.5%. There is a stark disparity between the first and second divisions in the scientific impact of nations. Moreover, although my analysis includes only 31 of the world’s 193 countries, these produce 97.5% of the world’s most cited papers (p. 314).The political implications of this last comparison are difficult to exaggerate. South Africa, at 29th place in my rank ordering, is the only African country on the list. The Islamic countries are only represented by Iran at 30th, despite the high GDP of many of them and the prominence of some individuals, such as Nobel prizewinners Abdus Salam (physics, 1979) and Ahmed Zewail (chemistry,1999) (p. 314).

Development of National Systems of Research Quality

• UK’s RAE• Australia’s RQF• NZ’s PBRF

Peer review versus bibliometrics

3. Peer Review, Bibliometrics & the Governance of Science

The old ideal of science as autonomous and independent of social conditions first gave way to‘Kuhn's disenchanted picture of 'normal' science (Kuhn 1970). Finally, post-Feyerabend studies of science re-examined the whole institution without presupposing any privileged status in relation to either virtue or natural knowledge’

Quality Assurance Replaces ‘Truth’ as Core Commitment of Post-normal Science & the Case

for ‘Extended Peer Review’

Quality assurance can thus be seen as a core commitment of post-normal science, replacing ‘truth’ as science’s ultimate regulative principle (Funtowicz and Ravetz 1992). Defined in terms of uncertainties and decision-stakes, quality assurance encompasses ‘public interest’, ‘citizen’, and ‘vernacular’ sciences. In a period of domination by globalized corporate science (Gibbons et al 1994), this effort to make scientists accountable to interested groups presents a coherent conceptual alternative for the survival of the ‘public knowledge’ tradition of science. Collegial peer review is thereby transformed into review by an ‘extended peer community’. This new form of quality-assurance will be given its formal structure and routines by those who put it into practice.Funtowicz (2001)

The Republican Model of Science Governance

• ‘republicanism is ultimately a theory of liberty’, an ‘explicit product of a legal system’, that requires ‘significant civic participation’ (Fuller,2002: 198)

• The challenge from Mode 2 knowledge and a form of knowledge managerialism now at the center of knowledge capitalism

Mode 2 Knowledge• in Mode 1 problems are set and solved in a context governed by

the (largely academic) interests of a specific community. By contrast, in Mode 2 knowledge is produced in a context of application;

• Mode 1 is disciplinary while Mode 2 is transdisciplinary;• Mode 1 is characterised by relative homogeneity of skills, Mode 2

by their heterogeneity;• in organisational terms, Mode 1 is hierarchical and, in academic

life at least, has tended to preserve its form, while in Mode 2 the preference is for flatter hierarchies using organisational structures which are transient;

• in comparison with Mode 1, Mode 2 is more socially accountable and reflexive.

• in comparison with Mode 1, Mode 2 involves a much expanded system of quality control. Peer review still exists to be sure, but in Mode 2 it includes a wider, more temporary and heterogeneous set of practitioners, collaborating on a problem defined in a specific and localised context.

Gibbons et al (1994)

Research Quality in Mode 1Quality in Mode 1 is determined essentially through the peer review judgements about the contributions made by individuals. Control is maintained by careful selection of those judged competent to act as peers, which is in part determined by their previous contributions to their discipline. In this way, the peer review process is one in which quality and control mutually re-enforce one another. It has both cognitive and social dimensions, in that there is professional control over what problems and techniques are deemed important to work on as well as who is qualified to pursue their solution. In disciplinary science, peer review operates to channel individuals towards work on problems judged to be central to the advance of the discipline. These problems are defined largely in terms of criteria which reflect the intellectual interests and pre-occupations of the discipline and its gatekeepers

Research Quality in Mode 2In Mode 2 additional criteria are added through the context of application which now incorporates a diverse range of intellectual interests as well as other social, economic or political ones. To the criterion of intellectual interest and its interaction, further questions are posed, ‘Will the solution, if found, be competitive in the market? Will it be cost effective? Will it be socially acceptable?’ Mode 2 quality is determined by a wider set of criteria that reflects the broadening social composition of the review system. This implies that ‘good science’ becomes more difficult to determine. Since it is no longer limited strictly to the judgements of disciplinary peers, the fear is that control will be weaker and result in lower quality work. Although the quality control process in Mode 2 is more broadly based, it does not follow that because a wider range of expertise is brought to bear on a problem that it will necessarily be of lower quality. It is of a more composite, multi-dimensional kind (Gibbons, 1998: 9-10).

Accountability & Democracy

First, core concepts of democratic theory, such as citizenship, deliberation, and accountability, cannot be understood satisfactorily without taking on board the politics of science and technology. Second, in all three countries, policies for the life sciences have been incorporated into "nation-building" projects that seek to reimagine what the nation stands for. Third, political culture influences democratic politics, and it works through the institutionalized ways in which citizens understand and evaluate public knowledge (http://www.pupress.princeton.edu/titles/7958.html)

Jasanoff: The Cognitive Front

On the cognitive front, the shift is from a realist to a constructivist view of knowledge. Years of work on the social construction of science and technology, and the contingency of similarity and difference judgments, have taught us to be skeptical of absolutist claims concerning objectivity and progress. Scientific knowledge, it is now widely accepted, does not simply accumulate, nor does technology invariably advance benign human interests...The methods with which policymakers carry on their business similarly cannot be taken as neutral, but must be seen as the result of political compromise and careful boundary maintenance, favoring some voices and viewpoints at others’ expense. The criteria by which one measures policy success or failure are likewise products of negotiation; in applying them, one implicitly adopts contingent, locally specific standards of reliability and validity. The special authority of scientific claims is in competition with other representations of reality diffused through the global media, and scientific expertise is subject to appropriation by multiple, diffracted social identities and interests. Any attempt to compare the performance of national policy systems today must take these complexities into account (p. 13).

Jasanoff: The Political Front• On the political front, the shift is toward a fracturing of the

authority of nation-states, with consequent pressures to rethink the forms of democratic governance. State sovereignty is eroding under the onslaught of environmental change, financial and labor mobility, increased communication, the global transfer of technical skills and scientific knowledge, and the rise of transnational organizations, multinational corporations, and social movements. Supranational concerns, such as the demand for free trade or globally sustainable development, are gaining political salience, but they are at the same time encountering resistance from tendencies toward greater local autonomy based on particularities of culture and place. As a result, the “old” politics of modernity—with its core values of rationality, objectivity, universalism, centralization, and efficiency—is confronting, and possibly yielding to, a “new” politics of pluralism, localism, irreducible ambiguity, and aestheticism in matters of lifestyle and taste (p.14).

The Governance of Science There are strong theoretical grounds for approaching this

complex question in a way that recognizes: • the changing nature and status of science – including the

history and philosophy of science, and the relationship of science to politics and technology;

• the changing architecture of world science and its place within an emerging global knowledge economy and international knowledge system;

• the changing relationship between natural and social science – including the significance of the ‘science wars’;

• the disciplinary hegemony of neoclassical economics and associated concepts in business and management sciences;

• the development of new information and communication technologies that are changing the social ecology of academic journals and academic publishing including the emergence of bibliometrics and webometrics.

Limitations of Citation Analysis

Cultural and Language bias• American and English-language bias• Draws only on a small subset of the total number of

journals world wide• Marginalizes local, indigenous and cultural

knowledges – all those that are not journalized and nominated by ISI

Discipline & Knowledge bias between Natural and Social Sciences/Humanities

• Distributed knowledge system favors natural sciences over social sciences and humanities (350,000 new science citations weekly, 50,000 in social sciences and 15,000 in humanities – Goodall, 2006)

Limitations (cont’)Discipline & Knowledge bias (cont’)• Ignores differences between knowledge production models and

cycles of natural sciences on the one hand and social sciences and humanities on the other

• Differences in publication rhythms and lead times for article publication with weekly journals in medicine and quarterlies, three or even two journals publications in some humanities disciplines

• Differences in form of publication where natural science favors journal articles and social sciences and humanities favor the monograph/book (which is indicative of differences in research organization)

• Citation analysis does NOT take into account book citation or reviews, or wider difference between research and scholarship

• Differences between natural and social sciences in co-authorship – up to 3 or 4 in the social sciences versus 10s and 20s in natural and life sciences

• Problems of comparisons across disciplines (lifetime citation scores differ radically for top scientists in natural (15-20K) and social science (3-4K) (Goodall, 2006)

Limitations (cont’)Methodological Problems • Authors receive citations for ‘bad’ as well as ‘good’ work• Problems with accuracy, consistency and reliability in citation data collection• Problems of self-citationPolitical Problems• Its original use designed to examine impact of scientific work not to measure

quality; the relationship between citation and quality is still not clear or well understood (Albach, 2006)

• Misuse of citations may encourage distortions in the international knowledge system favoring developed over developing countries and influence the uneven distribution of research funds that further consolidates western dominance

• Misuse of citations may encourage distortions in knowledge cultures at institutional and departmental levels preventing openness, sharing, collegiality and trust and through excessive competition promoting knowledge hoarding, secrecy and academic individualism.

Ownership and Governance problems• Who owns and governs science? What is the relationship between ownership and

governance? How do existing arrangements square with the republic of science and with the enhancement of public knowledge cultures? Who is the republic of science meant to serve? (These are political questions that citation analysis as a set of techniques cannot answer but yet presuppose).

New Developments• New information and communication technologies and the rapid changes

that are taking place in this area generates a range of new questions concerning the republic of science, including not only those about the governance of open source, open access, open course ware and free science (and the central question of what ‘freedom’ means here in relation to electronic academic publishing), but also a raft of new characteristics of the Web 2.0 platform that is altering the digital learning and research environments

• Web 2.0 as platform is said to operate differently from Web 1.0 and to be distinguished by tagging not taxonomy, rich user experiences, user as contributor, customer self-service, participation not publishing, radical trust (e.g. Wikipedia), and decentralization.[1] Its oft cited ‘democratic’ feature where amateurs can surpass professionals, ‘when they have the right kind of system to channel their efforts’, as Paul Graham notes,[2] raises all sorts of issues and problems for universities, for the republic of science, and for notions of research quality.

• These developments have the potential not only to change the communication of science but also to radically alter its traditional institutions, its organization, its value and the historical underlying ideologies of forbidden knowledge (Christian), of the encyclopedic knowledge system (Englightenment secularism), and of the knowledge economy (capitalist technoscience).