Impact of Research

John Peters Joint CEO Emerald Global, Bingley UK President Emerald Group Publishing Inc, Cambridge Mass.March 2010

Aim of this presentation

• Share experience and thoughts on research impact

• Provoke and stimulate debate about impact

• Present a discussion framework by which impact can be measured and assessed

Defining impact

“making a demonstrable difference in a non-academic context” Jonathan Wolff, Professor of Philosophy, University College London, 2010

“[understanding] the nature and purposes of business school research and [demonstrating] its overall value and visibility” AACSB Impact of Research Report, 2008

“the beneficial application of research to achieve social, economic, environmental and/or cultural outcomes. This is not to be confused with impact in the academic domain, which is seen more as an indicator of the intrinsic quality of the research on scholarly or academic measures” Australian Research Quality Framework, 2006

Business and management matters

How our businesses, governments and other institutions are managed, influences our world.

Our research institutions should produce more useful, relevant, readable and rigorous research which can influence management practice.

And a better-managed world will be a better world – fairer, wealthier, more educated, more careful of our environment.

Why do we care about impact?

• In 2009 the University of Michigan spent more than $1bn on research (U of M is one of the ‘big 4’ in terms of US university research spend) (Detroit Examiner, February 2010)

• [in 2005] U.S.-based AACSB-accredited business schools reported spending a total of $320 million annually to support faculty research (AACSB Impact of Research report)

• The proposed UK Research Excellence Foundation (REF) will distribute £2bn of government research funding (UK HEFCE)

Governments care about impact – but no common framework has emerged

• UK RAE – withdrawn after 20 years – to be replaced by REF

• US PART – program assessment rating tool - 800 Federal programs

• Australian RQF – proposed but never used – to be replaced by Excellence in Research Australia ERA

• EU Framework Program

• ERIC Netherlands – Evaluating Research in Context

• CNRS France – currently under review

• Several others

…and not everyone thinks we should measure impact

“We believe that it is counterproductive to make funding for the best research conditional on its perceived economic and social benefits…” Cambridge University statement to the UK Higher Education Funding Council, signed by six Nobel prizewinners in chemistry

“I came to this country [UK] to devote a large part of my life to basic research in astrophysics .... considering the management of the new council…I see no option than to leave this country as many of my colleagues have already done” comment, Times Higher Education Supplement, October 2009

UK Research Excellence Framework – an example

• Current view is that 25% of assessment will be judged on research impact

• REF defines ‘impact’ in terms of the economy and on society more broadly… but explicitly excludes academic impact

• Not every paper is expected to have impact – but should state a strategy for impact.

• Immediate impact is not always expected – it can sometimes only be assessed in retrospect

• Debate raging as to whether ‘impact’ is expected in all disciplines

A proposal – for consideration

• Scholarly research impact measures should not exclude contributions to scholarship and the body of knowledge

• “research that has an impact on teaching and learning; that contributes to the body of knowledge; that helps companies to be better managed; that influences public bodies and policy makers; that benefits society and/or the environment; that contributes to economic development”

Emerald working paper, 2010

Peer assessment in published work

• Self-statement of impact in a peer-reviewed publication can be accepted as valid. Any author assertion will have been tested for validity by a good peer review system. This is the equivalent of a third-party quality audit.

• Self-statement can assist with the issues of ‘potential to have impact’

Six Impact Zones

• Knowledge – scholarship which contributes to the body of knowledge and generates further research. Assessed by: citations, usage statistics, peer recognition, self-stated research conclusions

• Teaching and learning – students and faculty are direct consumers of research. Assessed by: clarity of conclusions to aid learning, provision of case studies and teaching examples, usage statistics, course adoption/curricula change.

• Practice – business leaders, practitioners and consultants in both private and public sector organizations are all affected by the outcomes of research. Assessed by: university-business collaboration, consultancy application, implications for practice self-stated

• Public policy – State officials, politicians, decision makers in public bodies, institutions and charities draw on research to shape their policies. Assessed by: self-stated potential implications, subsequent policy revisions.

• Society and environment –includes impact on the environment, the ability to influence social responsibility in industry, business and public policy and the incorporation of social and environmental values in research outputs. Assessed by: informing social policy, industry adoption, implications for society self-stated

• Economy – research which contributes to organization-level or macro-level wealth creation and business advancement. Assessed by: future economic savings, revenue increase, self-assessed business/economic impact.

Variables

Using any single variable for impact assessment is a blunt instrument.

Not all research is designed to make an impact in all the zones

Not all the zones will be relevant – may vary by institution, need, time and place

This allows weightings to be given to the significance of research

A model

Impact Zone Audience Scope A. Zone importance

B. Relevance & significance

Impact score

1. Knowledge Researchers Scholars

Discipline based research

Conceptual research

0 - 5 0 - 5 A x B

2. Practice Managers Professionals

Applied research 0 - 5 0 - 5 A x B

3. Teaching & Learning

TeachersStudents

Pedagogical utility 0 - 5 0 - 5 A x B

4. Public Policy Policy makers Government

Applied ResearchCross-discipline

research

0 - 5 0 - 5 A x B

5. Society andenvironment

GovernmentPolicy makersWider public

Applied ResearchCross-discipline

research

0 - 5 0 - 5 A x B

6. Economic Managers ProfessionalsGovernmentResearchersWider public

Applied ResearchCross-discipline

researchAction research

0 - 5 0 - 5 A x B

Impact score 1 + 2 + 3 + 4 + 5 + 6

Zone Importance – expected to be pre-set by an evaluating or host body

EXAMPLE: Orientation to ‘pure’ research and scholarship:

Knowledge 5

Practice 1

Teaching & Learning 3

Policy 2

Society & Environment 2

Economic 0

EXAMPLE: Applied/action research into business innovation

Knowledge 2

Practice 5

Teaching & Learning 3

Policy 4

Society & Environment 4

Economic 4

Zone relevance and significance - how to judge a piece of work

• A published paper, or other research output can be highly relevant to a zone, but not necessarily advance it significantly.

• In theoretical or discipline-based study, both relevance and significance would normally be expected

Result of zone importance – the same ‘relevance’ scores will yield different total scores

Impact Zone Audience Scope A. Zone importance

B. Relevance & significance

Impact score

1. Knowledge Researchers Scholars

Discipline based research

Conceptual research

5

2

4 20

8

2. Practice Managers Professionals

Applied research 1

5

3 3

15

3. Teaching & Learning

TeachersStudents

Pedagogical utility 3

3

3 9

9

4. Public Policy Policy makers Government

Applied ResearchCross-discipline

research

2

4

2 4

8

5. Society andenvironment

GovernmentPolicy makersWider public

Applied ResearchCross-discipline

research

2

4

3 6

12

6. Economic Managers ProfessionalsGovernmentResearchersWider public

Applied ResearchCross-discipline

researchAction research

0

4

2 0

8

Impact score 42 Example 1

60 example 2

Publisher role

• Peer review – validates self-stated claims (authors should be guided to be explicit about impact

• Structured abstract – requests research implications, practical implications, social implications (new 2010) as well as originality and contribution

• Explicit guidelines in Notes for Authors and for editors and review boards – what impact areas is a journal/collection looking for?

• Retrospectives – which papers seem to have been major contributors in impact zones.

• Online data gathering – what are you intending to do with this paper?

Discussion

• This work is presented as a discussion framework

• We are gathering comments and ideas from our stakeholders – editors, reviewers, research directors, deans, funding bodies, assessment bodies

• We hope to report on it at conferences and as published ‘white papers’

• Emerald are pleased to be working with ABS, AACSB, EFMD, and others

Debate makes the outcome stronger

• jpeters@emeraldinsight.com

• impact@emeraldinsight.com

• Ideas and thoughts are welcome

Final thought…

“HEC's aim is to become a School of Managers rather than a School of Management. We want to train students that are able to contemplate both political and social implications. Indeed, we believe that whilst the ‘technical’ side of things is important, it should nevertheless not overshadow the ‘human’ aspect in the strictest sense of the word.”

HEC School of Management, Paris, France