1

Some reflections on quality in research applicationsSome reflections on quality in research applications

Sharon WitherspoonDeputy DirectorNuffield Foundation

2

3

The Nuffield FoundationThe Nuffield FoundationThe Nuffield FoundationThe Nuffield Foundation

Endowed charitable trust

Endowed in Morris Motors shares, 1943

General objective:

‘to advance social well-being, particularly by means of scientific research’

4

Main Areas of ActivityMain Areas of ActivityMain Areas of ActivityMain Areas of Activity

Social research, social science and social innovation (welfare)

Education

Science(mainly new capacity and some areas of science policy)

5

We fundWe fundWe fundWe fund

Critical ‘synthetic’ reviews, drawing together and evaluating evidence from a range of studies

Evaluation of particular interventions – both quantitative and qualitative

Self-contained research projects:

oQuantitative and qualitativeoPrimary data-collection and secondary analysisoDescriptive and experimental

6

Key Criteria for Project GrantsKey Criteria for Project GrantsKey Criteria for Project GrantsKey Criteria for Project Grants

Is the topic of interest to the Foundation?

Does project have implications for policy or practice?

Does the plan for dissemination back this up?Will it be disseminated to practitioners or policy makers?

Is the research design rigorous and appropriate?Methodology of highest standard, precisely described?Is methodology apt for the questions being asked?

Are the results truly of interest, or is it ‘fancy that’ research?

Is the right sort of team assembled?

7

A ‘judgement’ frameworkA ‘judgement’ frameworkA ‘judgement’ frameworkA ‘judgement’ framework

Explicitly based on judgements

o By ‘scientific’ refereeso By referees judging importance, relevance, timelinesso By Foundation Trustees (zero-sum)

But not arbitrary:

o Procedural quality does not guarantee perfect judgements, but is correlated with it

8

Indicators of quality (1)Indicators of quality (1)Indicators of quality (1)Indicators of quality (1)

Precision of concepts – operationalisations

Precision of hypothesis or of exploration of meaning, taking meaning as problematic (not just what people say).

Sceptical scrutiny: consideration of alternatives

Alternative hypothesis (disproving hypotheses, rather than proving in classic mode), or consideration of alternative meanings and mechanisms. Some detachment from preferred explanations.

9

Indicators of quality (2)Indicators of quality (2)Indicators of quality (2)Indicators of quality (2)

Clarity about epistemological aims:

Descriptive:

What being described, why description matters (lack of previous knowledge, refuting commonly held views)

May be useful to think of contrast groups (is the phenomenon really specially important to group being studied?)

Explanatory

Here contrast or comparison groups, or control groups, or longitudinal study almost essential

10

Indicators of quality (3)Indicators of quality (3)Indicators of quality (3)Indicators of quality (3)

Clarity about how methods illuminate the underlying questions:

Are questions of incidence, incidence, prevalence, distribution, spread, correlations crucial to your argument? …..Qualitative alone unlikely to suffice

Are questions of meaning, understanding, motivation, affect crucial to your argument? …..Quantitative alone unlikely to suffice

Do you need both? In what way: preliminary; joint; elucidation of mechanism

….all this before consideration of more technical questions about research methods, analysis techniques (though in fact iterate back and forth)

11

Some examples (1)Some examples (1)Some examples (1)Some examples (1)

Use of qualitative to illuminate concepts, how people understand them, language used:

o Hazel Genn, et al: Paths to Justice

o Wellings, Field, Johnson, Wadsworth. Sexual behaviour in Britain

Use of qualitative to get “beneath” superficial justifications

o Successive projects by Carol Smart and others on cohabitation andcommitment

12

Some examples (2)Some examples (2)Some examples (2)Some examples (2)

Use of quantitative descriptive study to put this into context

o "Just a piece of paper? Marriage and cohabitation" by Anne Barlow, Simon Duncan, Grace James and Alison Park in British Social Attitudes: the 18th Report

Use of qualitative to adduce evidence about a mechanism – why a correlation ‘causes’ something:

o John H. Laub, Robert J. Sampson . Shared Beginnings, Divergent Lives: Delinquent Boys to Age 70

13

Judgements of ‘import’Judgements of ‘import’Judgements of ‘import’Judgements of ‘import’Clearly less objective, more room for discussion

Policy/practical import:

o NOT tied to what government or powerful feel is important

o Rational consideration of balancing setting long-term agenda with short-term effect

o Power to stop – consider the discussion of why government doesn’t do more pilots in the report ‘Trying it out’ ….versus influence…. versus agenda-setting

Scientific importance:

o Takes the field forward

o New concept, new operationalisation, new site of study

o Innovation ….. extension ….. replication

14

Some personal predictions and Some personal predictions and predilections…..predilections…..

Some personal predictions and Some personal predictions and predilections…..predilections…..

Will (and should) see more invested in larger, quantitative or longitudinal studies

Will (and should) see more concern with outcomes

Will (and should not) see diminution in government enthusiasm for much research

15

And even more controversially …And even more controversially …And even more controversially …And even more controversially …

Will (and should) see that researchers who get good quantitative training early on have a head-start in thinking through analytic framework and posing big questions for future research programmes (as opposed to projects)

Refuse to predict though how this will work structurally with research concentration, RAE, funding changes ahead …..

16

www.nuffieldfoundation.orgwww.nuffieldfoundation.org