Mixed Methods

30
MIXED METHODS RESEARCH: A discussion paper JULIA BRANNEN ESRC National Centre for Research Methods NCRM Methods Review Papers NCRM/005 1

Transcript of Mixed Methods

Page 1: Mixed Methods

MIXED METHODS RESEARCH: A discussion paper

JULIA BRANNEN

ESRC National Centre for Research Methods

NCRM Methods Review Papers

NCRM/005

1

Page 2: Mixed Methods

MIXED METHODS RESEARCH: A discussion paper

JULIA BRANNEN Institute of Education, University of London

ABSTRACT This paper proposes to • Define mixed method research • Set out some of the reasons why mixed methods may currently be in the

ascendancy and identify opportunities and risks attached to these for researchers • Consider some of the main rationales for choosing a mixed method research

strategy – the three Ps of paradigms, pragmatics and politics • Explicate how a mixed method strategy plays out during the research process: the

research design phase, the fieldwork phase, the analysis phase and contextualisation

• Address particular issues: Quality criteria, teaching mixed methods, writing up mixed methods research

2

Page 3: Mixed Methods

CONTENTS PAGE

1. What is mixed methods research?

2. Mixed methods in the ascendancy: opportunities and risks

3. Rationales for choice of different methods: the three ‘P’s’

Paradigms and philosophical assumptions Pragmatics Politics

4. Combining methods during the research process

Making sense of different data: different meanings or different forms of triangulation Research design phase Fieldwork phase Analysis phase Contextualisation

5. Other issues in mixed methods research

Quality criteria in assessing mixed method research Teaching and learning mixed methods Writing up mixed methods research Resources

References

3

Page 4: Mixed Methods

1. What is mixed method research?

In order to address a research question or set of research questions, researchers must devise a strategy or, as Bryman suggests, ‘a general orientation to the conduct of social research’ (Bryman 2001: 20). Mixed methods research means adopting a research strategy employing more than one type of research method. The methods may be a mix or qualitative and quantitative methods, a mix of quantitative methods or a mix of qualitative methods. If mixed methods research is a research strategy does it represent a particular type of research design? The answer is both yes and no. Adopting a mixed method strategy may constitute a strategy in its own right or it may be subsumed within another research strategy as in the case of adopting a case study design in which a number of different methods are embedded. Ethnography and action research are also research strategies that may also employ more than one method. Mixed methods research also means working with different types of data. It may also involve using different investigators – sometimes different research teams working in different research paradigms. For these reasons mixed method research is often referred to as multi-strategy research (Bryman 2001) implying the application of a number of different research strategies related to a complex range of research questions and a complex research design. On the other hand, mixed methods may form part of a long term strategy (several years) as in the case of a research programme that is pursued over time by a group of researchers applying different methods and approaches consecutively (see Kelle 2005 for an example).

2. Mixed methods in the ascendancy: opportunities and risks

Currently it seems that mixed methods research strategies are being increasingly employed. As someone who co-edits a methods journal (The International Journal of Social Research Methodology) and who wrote about mixed methods and edited a text on the subject in the 1990s (Brannen 1992), I have noted a recent surge of interest in the last two years. In 2003 a Sage Handbook of Mixed Methods Research was published (Tashakorri and Teddlie 2003a). A number of UK and international seminars and workshops have been held in the past year devoted to the discussion of mixed methods research. For example in the UK a one day conference on mixed methods research was organised for health researchers (Sheffield, November 2004), an event on mixed methods was put on by the Royal Statistical Society (London, March 2005) and a two day workshop on the topic by the ESRC Research Methods Programme (Manchester, October 2005). A journal of mixed methods research is planned by Sage. On the other hand, mixed methods research may be more popular now because it is named and reflected upon. We may ask why are mixed methods, in particular research strategies that combine qualitative and quantitative approaches, coming to the fore? The reasons are several and while they represent an opportunity for advancements in methodology they also present possible risks for researchers.

4

Page 5: Mixed Methods

First, mixed methods research presents an opportunity for skills enhancement. In western society there has been a growth in formal training and in credentialism. The knowledge society is principally achieved through an emphasis upon skill. In the social sciences, skills are increasing acquired through training courses across a range of different methods rather than as part of a vocational apprenticeship in a particular type of research or in a particular discipline. A person’s repertoire of methodological expertise is displayed on their curricula vitae in terms of course attendance and certification as well as through the practical application of research expertise achieved through experience doing research. Having a PhD is increasingly a requirement to enter a research occupation while training across the spectrum of research methods is now a mandatory part of the doctoral experience. Many students take masters courses in social research methods before they enter doctoral programmes. But training is not the same as learning by practice; knowledge fades if it is not put into practice fast. Second, and relatedly, mixed method research training and experience are an opportunity for lifelong learning. Broadening one’s methodological repertoire mitigates against ‘trained incapacities’, as Reiss (1968) termed them - the entrenchment of researchers in particular methods or types of research. Today methodology has a higher profile than it used to be; formerly theory had a higher status attached to it. However, in putting more emphasis on methodology, we need also to be mindful of Lewis Coser’s admonition to the American Sociological Association made in 1975 against producing new generations of researchers ‘with superior research skills but with a trained incapacity to think in theoretically innovative ways’ (Coser 1975). Third, mixed methods research is an opportunity that deflects attention away from theoretical work that is often specific to particular disciplines. Thus it may encourage thinking ‘outside the box’, a practice to be welcomed. On the other hand, we are seeing a growth in importance in the UK social sciences of substantive fields bringing together researchers across disciplinary boundaries. Increased funding has been allocated by ESRC to programmes of research that are defined in considerable part by particular substantive fields: for example, programmes on work, childhood, youth, migration, social exclusion. While there are undoubted benefits for the stakeholders and researchers in learning about and integrating research evidence within a field and bringing together researchers across disciplines or irrespective of disciplines, there may be some disadvantages. Researchers may escape exposure to the traditions of a particular discipline and may fail to acquire a secure identity within a discipline. In so far as the choice of a mixed method research strategy is determined by practical rather than disciplinary influences, then approaches to theory becomes more eclectic. There is a danger that researchers who are not sufficiently theoretically grounded before they do their research will import theory when they write it up in order to strengthen or support a particular set of findings. Theory should also inform the research questions one poses at the start of a project. Yet there are competing influences here. In Britain, and increasingly elsewhere, academic and researchers are required to publish in scholarly journals. It remains the case that many of the most prestigious journals are discipline based and have leanings to particular types of methodology. Thus writing up mixed methods research as such may present a problem for the researcher as to where to publish.

5

Page 6: Mixed Methods

Fourth, developing a mixed method strategy fits with the political currency accorded to ‘practical enquiry’ that speaks to policy and policymakers and that informs practice (Hammersley 2000), while scientific research may require closer attention to and justification of the methods used and the types of data generated in reaching conclusions. Researchers are required to address the needs of research stakeholders and users, with funders framing our research questions and sometimes even our methods. Research has always been subject to political climates and persuasions (Finch 1986). In Britain we have seen a whole industry of mixed method research created around evidence based policy and over a longer time frame in the evaluation of policy (Ritchie 2003; Tashakorri and Teddlie 2003a). However the downside to this is that researchers have less and less lee way to define their own research questions and to follow their own ideas. Fifthly, with the growth of strategic and practically oriented research which meets the needs of users, there is increased emphasis upon dissemination. Researchers must speak at least two languages – the technical language of research but also the language which makes research results simple to communicate and its messages easy to understand (Duncan and Harrop in press). Thus in writing up research, words become as important as numbers. Mixed methods research that uses both quantitative and qualitative approaches has the advantage of allowing for both. However the different presentational modes may sit awkwardly together on the page. Or more commonly the latter issue may be addressed by reserving the different analyses for different publications especially when in the case of academic journals, convention and editorial policy do not appear to welcome mixed method research. Sixth, opportunities for mixed methods research are increasing with the rise in cross-national research in the context of the growth in European Union funding. Many EU projects typically comprise context mapping exercises, involving secondary analysis of macro-level data and the collection of national statistical data. These are often adjuncts to the use of more intensive research approaches that address the micro level (for a discussion of recent EU funded research projects in family and welfare see Hantrais in press). This contextualising work is an essential part of cross-national research. However, there is a danger that such contextualising data are collected but do not sufficiently inform the analysis of primary data. Mixed methods research offers therefore both opportunities and risks. It may offer creative possibilities for addressing research questions in terms of a range of methods. However these possibilities should not take the place of creative thinking. Mixed methods research may come into its own at a time in which social science research is first and foremost required to be practically relevant and applicable to policy. Practical relevance should not substitute for theoretical relevance. Finally, the opportunity to learn new research skills is to be welcomed and is particularly facilitative of cross-disciplinary collaboration but should not undermine disciplines and the importance of theory. 3. Rationales for choice of different methods: the three ‘Ps’ Notwithstanding the renewed interest in mixed method research strategies, dialogue between researchers describing themselves as either qualitative or quantitative researchers has often been fraught. It has been marked by misconceptions about the

6

Page 7: Mixed Methods

‘other’, making understanding of the other difficult. Quantitative researchers have seen qualitative researchers as too context specific, their samples as unrepresentative and their claims about their work as unwarranted – that is judged from the vantage point of statistical generalisation. For their part qualitative researchers view quantitative research as overly simplistic, decontextualised, reductionist in terms of its generalisations, and failing to capture the meanings that actors attach to their lives and circumstances. What then are the kinds of rationales that underlie the choice of method, mixed or otherwise? I will refer to these as the ‘three Ps’: paradigms, pragmatics and politics. All or any one of these may shape a researcher’s choice of method. Paradigms and philosophical assumptions Here the researcher’s choice of methods is said to be chiefly driven by the philosophical assumptions - ontological and epistemological - which frame the research or the researcher’s frame of reference. Researchers writing on these issues typically refer to the ‘paradigm wars’. The perception that qualitative and quantitative research are distinct is that they are said to be based on different philosophical principles. To the extent that these principles are seen to be competing, they are said to belong to different ‘paradigms’. According to Kuhn (1970) paradigms are incommensurable. In short, according to the paradigmatic position, qualitative and quantitative research are seen to be intrinsically different beasts underpinned by different philosophical assumptions. Two philosophical traditions have dominated the discussion of mixed method research strategies: positivism and interpretevism. Qualitative researchers typically locate themselves within an interpretevist tradition, albeit they also often hold realist assumptions about the world and the contextual conditions that shape and embed the perspectives of those they seek to study. Quantitative research is by contrast associated with positivism, often by those defining themselves as qualitative researchers. As Bryman observed in 1984 qualitative researchers have spent more time defining quantitative methods than quantitative researchers have themselves (Bryman 1984). How quantitative researchers actually identify themselves is less evident since those writing about quantitative research typically give much less attention to epistemological and ontological assumptions in discussing their research. Such a dichotomous portrayal of qualitative and quantitative research proves to be more complex under closer scrutiny, as methodologists such as Bryman and Hammersley have noted on many occasions. For example, surveys are not necessarily conducted on the basis of positivist assumptions, as Bryman warned us in an early article on the subject, quoting Cathy Marsh on the subject in 1979. Similarly, qualitative researchers such as Whyte, Gans or Skolnick working in the tradition of participant observation work in a realist tradition (Bryman 1984: 89). The paradigmatic position assumes working from the principle that choice of method is not made in a philosophical void: research questions should be thought about in relation to epistemological assumptions (see for example Barbour 1999). Thus in terms of best practice, researchers may be well advised to consider what kind of knowledge they seek to generate.

7

Page 8: Mixed Methods

There is another dimension too which relates to the transcendence of paradigms. Some social scientists are concerned with generating understandings at the micro level while others are concerned with the macro level. Thus those in the former group emphasise the agency of those they study through an emphasis upon studying subjective interpretations and perspectives. Those working at the macro level are concerned with larger scale patterns and trends and seek to pose structural explanations. However all researchers aim to understand individuals in society. If one is to transcend conceptually the micro and the macro levels then methods must be developed to reflect this transcendence (Kelle 2001). Whether those who apply a paradigm rationality will apply both qualitative and quantitative methods will depend upon the extent to which they seek to produce different levels and types of explanation. However if research paradigms are all important in shaping the choice of methods then the researcher is likely to rule out particular methods from the start and not be governed by the research process and the context as it unfolds. Pragmatics As Bryman suggested first in1984, in practice much research is driven by pragmatic assumptions, or what Bryman terms ‘technical’ issues as much as it is driven by philosophical assumptions. Most textbooks argue that sound methodological practice is to choose a method appropriate to the research question (Blaikie 2000; de Vaus 2001; Mason 2002; Cresswell 2003). The framing of research questions is in part shaped by epistemological assumptions but is also influenced by the need to find theory that ‘fits’ a specific set of cases or contexts. Researchers in the grounded theory tradition following Glaser and Strauss (1967) expect to reformulate their research questions during the course of an investigation. However it is rare for a researcher working on a project to pose only one research question. Indeed any piece of research is likely to comprise a complex of research questions. While the key research question or questions in a piece of research may be underpinned by realist assumptions, some research questions may be underpinned by interpretevist assumptions, for example concerning how people make sense of their actions. A quantitative researcher may be more concerned with the actions and behaviour of informants while they may also have an interest in informants’ meanings, framed in terms of attitudes. Moreover a focus on meaning within quantitative research is often inescapable since researchers typically study people’s behaviour via self reports of behaviour. Researchers of both quantitative and qualitative persuasions may assume that reports of behaviour have some close semblance to actual behaviour. For even if researchers choose to treat such reports with caution, it is likely that those who commission the research or those who read the results will construe such results as ‘real’. The framing of research questions may therefore be underpinned by both philosophical and pragmatic issues. Some researchers set out to do mixed method research for both pragmatic and philosophical reasons, as in the case of a study of

8

Page 9: Mixed Methods

educational effectiveness in early childhood institutions (for example Sammons et al 2005, discussed in the following section). Moreover even if researchers set out to choose their methods according to a particular framing of a research question and its associated philosophical assumptions, in practice their accounts of research practice may belie their intentions. As Bryman (in press b) has identified in relation to the choice of a mixed methods strategy, while researchers may initially justify their approach in terms of tailoring methods to research questions, in practice they may make reference to the outcomes of the research. So that while the choice of methods may start off in relation to the former in terms of research design, the practicalities of the research process may divert the original intention. In many accounts of their research, as Bryman (in press a) has shown, researchers appeal to what he terms a universalistic discourse concerning the advantages of mixed methods - a belief that mixed methods research produces better outcomes than single method research. Thus, Bryman suggests, such rationales eschew the stance that research questions should determine the research methods, the stance taken in most methodology texts. This position is demonstrated in an article concerning a mixed methods research project conducted by Hammond (2005) (see also next section) in a programme of research set up to explore the wider benefits of adult learning. This author concludes that it would have been ‘absurd to attempt to map out this area using only one method’ and goes on to suggest that it was too premature in the investigation to frame questions too specifically at such an early stage of the research (p253). Similarly, Sammons et al (2005) discussing school effectiveness justify using mixed methods in situations where ‘complex and pluralistic social contexts demand analysis that is informed by multiple and diverse perspectives’ (p221), thereby suggesting that the inferences they can make from their research are in general strengthened by the use of a mix of methods. It is noteworthy that this comment is made at the end of their paper in which methods were justified post hoc in terms of the ways in which different data were integrated in drawing overall conclusions from the study. Thus some of the advantages of mixed method research may not emerge until the end of the research process. Indeed the generation of new perspectives is seen as one of the important possible advantages of their use (Green, Caracelli and Graham, 1989). However this may not always be anticipated at the outset of a research project. Such post hoc justification contrasts with rationales generated in the research design phase of the research process. Indeed innovative insights may arise irrespective of the original research questions posed in the investigation and may indeed lead to the replacement of the original questions with new questions. A pragmatic rationale for mixed methods research may also relate to the resources available to researchers, even dictating which questions we ask and the way we frame them. Thus a researcher working in the quantitative tradition may seek to observe actual behaviour but not have the resources to do so. Instead he or she may have to be content with reports of behaviour as collected in a self-completion questionnaire survey. Criteria that may govern the kind of methods used include the skills and strengths of a research team and the research cultures they inhabit (Brannen 1992).

9

Page 10: Mixed Methods

The issue is in part at least determined by the feasibility of particular methods within a particular project. One feasibility issue concerns the nature of the research population; whether the population is difficult to access or not may affect choice of method. A survey is unlikely to work with invisible population for example illegal immigrants. Particular methods of collecting data may be chosen because they make for better cooperation with research informants; for example semi- or unstructured interviews are likely to be used with those in powerful positions in organisation since their perspectives are likely to be (or be believed to be) unique within an organisation. In addition, some methods more than others are deemed to have a better ‘fit’ because they are more sensitive to complex social phenomena; for example the first funded research project I carried out concerned the study of marital problems, problems that some had difficulty owning up to. Thus I decided to use a semi-structured interview and to approach the issue in a circuitous, non-confrontational way (Brannen and Collard 1984). What then distinguishes the pragmatic researcher from the paradigm-oriented one? In the paradigmatic vision of the world the former is more interested in ideas and their origins, in the ideas which drive the research and the ideals upon which research should be founded. The concern of the pragmatist is more to open up the world to social enquiry and hence to be less purist in terms of methods and preconceptions (about theory and method). Such researchers are oriented to the production of research results that they seek to link to practical and policy ends (Hammersley 2000). Thus a pragmatic rationality will more readily embrace a mix of methods if the research questions and practicalities of the research context suggest it. Politics A third rationality relates to the politics of the research and the researcher. The political researcher is concerned about forms of knowledge and ways of knowing – research for whom and for what? Many feminists in the 1980s chose qualitative methods for particular political purposes: to make the voices of silenced women heard. In identifying with their informants their research was a project in which they sought to address the cause of women in general. In this regard many preferred in depth interviewing to elicit women’s own views of the world while some continue to adopt such methods for this reason. However, as Ribbens and Edwards (1998) argue citing a paper by Cain (1993) on Foucault, the epistemological base of women’s everyday lives and knowledge is not easily accessible as in concepts such as ‘views’ and ‘attitudes’. Moreover in listening to women’s own voices about their private worlds researchers are busy transforming these into public knowledge. Thus while women’s own knowledge is often subjugated to other more powerful knowledges in their everyday lives, often internalising and reproducing them, so researchers are at risk of compounding this situation by turning these voices into research findings. However in order to examine and address (politically) the conditions of women in the general population – for example women’s lower access to managerial positions and equal pay, it is necessary to draw upon large scale data as well as qualitative data if they wish to understand and expose gendered inequalities in the population, as feminists have increasingly done (see Graham 1983; Walby 1997). Thus political rationales for using mixed methods may have more in common with pragmatic

10

Page 11: Mixed Methods

rationalities than paradigmatic ones (see also Oakley 1999 writing about her own research trajectory). Another example of researchers using mixed methods for political purposes concerns research into social justice, in particular with respect to minority groups who are often invisible or hard to reach via large scale surveys (Mertens 1998). Such researchers who adopt a political, humanist bias in their research often have to employ qualitative approaches to find such groups. However the impetus is also to make explicit the understandings of oppressed groups while making the case effectively for changing the life situations of the oppressed. Hence they draw increasingly upon a range of methods including quantitative methods in order to study oppressed groups extensively while representing in depth the perspectives of individuals. Other examples one can cite here are researchers who study disability and researchers in the new childhood studies. Indeed these researchers often refer to their methods as participatory or transformative and hence as ‘new’ or different from traditional methods. However it is not the methods per se that distinguish their approaches but what the methods are used for. Thus the aim is to get the best handle on social inequality and power differentials both in society and also, very importantly, in research relations. For example Alderson (2001) argues for the use of children as researchers in the study of childhood as children are the primary source of knowledge about their own views and experiences. 4. Combining methods during the research process There are two contexts in the research process in which methodological considerations concerning the application of a mixed methods research strategy come to the fore (Brannen 2004, 2005). First is the context of enquiry or the research design phase. At this phase of the research process we wrestle with framing ‘researchable questions.’ Do we want to know the frequency of a defined social phenomenon according to some pre-defined variables? Do we want to generalise those frequencies and their associations to a parent population? Do we want to explore what people think about a particular social phenomenon and how those perceptions link to other perspectives and informant characteristics? Or, more mundanely and practically, in terms of the choice of method, do we want to use one field method to find a particular group and to use another field method to study a subset of that group? We are likely in many research projects to ask more than one such question. The kind of questions we pose leads to the choice of method and, increasingly commonly, to a complex of methods. Thus choice of method is in part linked to the nature of the research question(s) and needs to take account of their epistemological bases. However a research strategy is devised as best suited to a particular purpose rather than being only tied to a philosophical position. Our methods and their assumptions are revisited in a second context - what is known as the context of justification where the data are analysed and interpreted. As some would argue, in the context of justification the resulting data sets cannot be linked together unproblematically (Smith and Heshusius 1986). For it is at this phase that ontological, epistemological and theoretical issues do raise their heads in the researcher’s encounter with data. In the cold light of data analysis we are forced to reflect on different kinds of ‘truth’ and ‘validity’ and to take account of the fact that

11

Page 12: Mixed Methods

our different types of data are constituted by the assumptions and methods which elicit them. Making sense of different data: different meanings or different forms of triangulation? This is where the term triangulation enters. The purpose for which mixed methods are used is often, misleadingly in my view, referred to as triangulation a term taken from navigation and surveying in which different bearings are taken in order to arrive at a precise physical location. As Hammersley (2005) citing Erzberger and Kelle (2003) points out, the second bearing is not used to check or verify the first bearing; rather each complements the other in order to identify a particular location. However in social science the early use of the term triangulation was taken to mean something very different, that is to ascertain how different methods check, validate or corroborate one another. The idea was to enable an understanding of a social phenomenon from different vantage points (methods, investigators): how in effect different data analyses come to the same conclusion. (Denzin, 1970). As Hammersley (2005) reminds us, this does not necessarily mean combining different methods as such; it may mean combining for example the same observations but in different settings. My own view and that of others (cf Moran – Ellis et al in press) is that data collected from different methods cannot simply be added together to produce a unitary or rounded reality or truth. As Hammersley points out, if we move away from assuming that we are trying to arrive at a single reality we need to understand how different accounts are arrived at and the purposes these accounts serve (Hammersley 2005). The use of triangulation in the sense that it is taken to mean the corroboration of research results is only one of at least four possibilities (Bryman, 2001 citing Morgan 1998; Hammersley 1996; Rossman and Wilson (1994)). There are other ways of combining the results from different data analyses in addition to corroboration. These include: (1) Elaboration or expansion – for example qualitative data analysis may exemplify how patterns based on quantitative data analysis apply in particular cases. Here the use of one type of data analysis adds to the understanding being gained by another. (2) Initiation: the use of a first method sparks new hypotheses or research questions that can be pursued using a different method. (3) Complementarity – qualitative and quantitative results are treated as different beasts. Each type of data analysis enhances the other . Together the data analyses from the two methods are juxtaposed and generate complementary insights that together create a bigger picture. (4) Contradictions - where qualitative data and quantitative findings conflict. Exploring contradictions between different types of data assumed to reflect the same phenomenon may lead to an interrogation of the methods and to discounting of one method in favour of another (in terms of assessments of validity or reliability).

12

Page 13: Mixed Methods

Alternatively the researcher may simply juxtapose the contradictions for others to explore in further research. More commonly one type of data may be presented and assumed to be ‘better’ rather than seeking to explain the contradictions n relation to some ontological reality (Hammerley 2005 quoting Denzin and Lincoln 2005). As Hammersley (2005) points out, these different ways of combining different data analyses all to some extent make assumptions that there is some reality out there to be captured, despite the caveats expressed about each method constructing data differently. Thus just as seeking to corroborate data may not lead us down the path of ‘validation’, so too the complementarity rationale for mixing methods may not complete the picture either. There may be no meeting point between epistemological positions but, as Hammersley suggests, there is a need for a dialogue between them. What is clear is that mixed methods research if carried out in a technicist way obviates the need for reflection about methods. As Hammersley sensibly points out ‘these (different forms of triangulation) are investigative strategies that offer evidence to inform judgements , not techniques that provide guaranteed truth or completeness.’ (p12 2005). Peter Halfpenny (2005) cuts to the heart of some of the problems researchers seem to get into when they justify a combination of methods in terms of philosophical positions. He too challenged the assumption of a simple correspondence between philosophical position and research techniques. Rather there are a number of logics at play in devising research questions, creating a research design and analysing data. These logics do not map on to one another neatly. What seems to get lost in the focus upon triangulation is the relation between theory, methods and data. For example a recent debate about apparently conflicting findings from two studies, one using qualitative and one quantitative methods turns not so much upon the methods used per se. The studies explored young people’s ways of thinking about, and their plans for, the future. The theoretical propositions and conceptualisations that the researchers employed in these studies were very different. However these conceptualisations informed the kinds of questions they framed to young people and produced very different results (see Brannen and Nilsen 2002; Anderson et al 2005). These studies used different methods – one a large scale survey and the other a cross national qualitative study using focus groups and interviews. However this is not to say that it might have been possible to formulate similar though not exactly the same questions to young people had the theoretical and conceptual formulations concerning planning (to be distinguished from aspirations, hopes and dreams) been similarly sensitive. Both sought to explore variation in ways young people thought about the future and differences in meanings (Brannen and Nilsen forthcoming).

Research design phase Commentators have categorised mixed methods designs on a number of key dimensions (see Morse 2003 for example). These considerations should be born in mind at the outset of creating a research strategy, albeit such plans may change in the course of a study. First, there is logic of enquiry that drives the study. Is the study primarily to be inductive aimed at discovery? Or is it to be deductive aimed at testing

13

Page 14: Mixed Methods

hypotheses? Many projects seek to combine inductive and deductive logics of enquiry. As noted about, no one type of method need be associated with a particular logic of enquiry. Surveys may involve inductive and deductive logics while qualitative methods so often associated with grounded theory often test ideas as well as generate them. Second, if the logic of enquiry and the nature of the research question suggest the usefulness of a mixed method approach, researchers need to consider the ordering of their methods. Are the methods sequential or simultaneous? Are particular methods preferable at the start of an enquiry, for example aimed to help generate representative samples from which sub groups may be selected for further intensive study using other methods? Are particular methods selected for purposes of piloting particular methods to be used later as in the case of qualitative interviews that precede the development of a survey? Are different methods – qualitative and quantitative – to be used in tandem to study the same or different phenomena? Is qualitative research to follow statistical enquiry? This latter possibility is said to be one of the least utilised combinations (Ritchie 2003; Bryman in press b). Third, researchers must consider how dominant a particular method is going to be in planning the deployment of what are usually scarce resources. Is one data set being treated as secondary or supplementary in terms of having lesser resources of time being devoted to them in terms of data collection and also in the analysis phase and the writing up? Of course it is possible and likely that the relative importance of different data sets may not emerge until these later phases. In some studies equal weight may be intended to be given to both but in writing up the research they may appear in separate reports. It is also important to add that research design is not only the plans made at the start of a study but includes changes made in the course of the study. Some qualitative studies are designed to allow such flexibility. In particular this is the case where analytic induction is adopted, as in case study research designs in which the second and subsequent cases are selected in relation to the analysis of those that precede them (see Hammersley 1989). Drawing upon Morse (2003), the possible permutations of research designs may be presented in terms of both the sequencing and dominance of qualitative and quantitative methods. As in Morse (2003), the arrows below indicate sequencing of methods and the plus signs indicate simultaneity. Dominance of a method is indicated in CAPITAL letters. Simultaneous designs 1. QUAL + quan or 2. QUAL + QUAN 3. QUAN + quan or 4. QUAN + QUAN 5. QUAL + qual or 6. QUAL + QUAL Sequential designs 1. QUAL > qual or 2. qual > QUAL or 3. QUAL> QUAL 4. QUAN > quan or 5. quan > QUAN or 6. QUAN > QUAN 7. QUAL > quan or 8. qual > QUAN or 9. QUAL > QUAN 10. QUAN > qual or 11. quan > QUAL or 12. QUAN > QUAL

14

Page 15: Mixed Methods

In discussing mixed methods research it is important not to neglect the fact that methods may be combined within either the quantitative or qualitative paradigm. It is commonplace for qualitative studies to combine different qualitative methods. A study of the sleep of women aged 60 and over used ‘a multi-method approach’ of a sequential variety (Hislop and Arber 2003) and collected qualitative data in three ways through: focus groups, interviews, and audio sleep diaries. The focus groups sought to generate discussions about attitudes to sleep, patterns of sleep, sleeping as a shared experience, ageing and sleep, the effects of poor sleep, and strategies for overcoming sleep problems. In-depth interviews with a further group of women offered opportunities for women to discuss changes in their sleep patterns in relation to major life events and transitions. Audio-sleep diaries were written by the women each morning for a 7-day period in which they assessed their sleep for the night before. Similarly a quantitative study may apply a mix of quantitative methods, for example diary data and survey data based on pre-coded questions. It is not always easy to assess on the basis of publications the relative resources devoted to different methods nor the importance of different components (qualitative and quantitative) without perusing the research proposals and the costings of the research. However it is clear that some designs are more common than others. In terms of the designs that combine qualitative and quantitative research: design number 8 of the sequential designs - where qualitative pilot work is likely to precede and be subservient to a large survey - is typically how large scale survey research precedes. Less common is design number 10 where more highly resourced quantitative research is followed by lower resourced qualitative research. In general it seems to be less common for qualitative research to be done as a follow- up to quantitative study (Sieber 1973; Ritchie 2003). For example a panel or cohort study may be designed so that after one of its waves a subset of its members are exposed to qualitative methods in order to explore their understandings of the outcomes detected in the longitudinal quantitative analysis. This is rarely done (see Thompson 2004 for an example) although there may be important reasons for doing this, for example where a sub group is too small for statistical analysis, being under represented in a random sample of the population. As noted in Ritchie (2003), a subgroup was identified in a survey of people registered as disabled. These were the severely disabled who were working in open, as opposed to, sheltered employment. This group was followed up using qualitative methods to see how it was that this group managed to gain, sustain, and retain open employment (Thomas 1992 cited in Ritchie 2003). More common is design number 11 where more highly resources qualitative research follows lesser resourced quantitative work, as in this next example (see also Becker and Bryman 2004).

15

Page 16: Mixed Methods

BOX 1 Sequential design number 11: quan > QUAL: Children’s concepts of care and family life (Brannen et al 2000; Brannen 2004) First (secondary) method: a survey of 10-12 years olds attending mixed sex state schools in two local authorities in London Purpose: main purpose to identify children for interview who were growing up in different types of households via a self-completion questionnaire survey; secondary purpose to provide extensive data and to provide local data for schools as a recompense for their cooperation. Second (main) method: to explore children’s own perspectives on care and their experiences of family life via semi-structured interviews. Other qualitative methods were included: network maps, vignettes and genealogical diagrams. Purpose: the mix of qualitative methods took account of children’s interests and competencies, while being attentive to the sensitive topic of family life and family change. Children’s mothers were also interviewed as key figures in children’s family lives and because they provided important contextual material about family change and children’s relationships, especially about children’s non-resident fathers about whom many children were reluctant to talk.

Research designs may be sequential but studies may also employ separate research teams. From the perspective of each team each part of the design may be considered equally important. However in the case of a qualitative component to a large scale cohort study, the cohort may have the upper hand as it continues beyond the life of the qualitative study. Such designs employing qualitative components within a large-scale cohort study may benefit quantitative researchers through achieving a better handle upon the meanings of underlying statistical associations, while it gives qualitative researchers the chance to select cases based on knowledge of the wider sample and to test hypotheses on large, statistically representative samples (Thompson 2004). Finally, there are other designs involving several sequences of methods. Drawing again on my own current research, this design covers a sequence of four data collection methods and phases (see Box 2)

16

Page 17: Mixed Methods

BOX 2 Multi- sequenced design: The work-family careers of childcare workers caring for vulnerable children (Brannen et al forthcoming) Method 1: In an ongoing study of the work-family careers of childcare workers caring for vulnerable children (Brannen et al forthcoming), a survey was carried out in several social services mainly to identify a pool of four types of childcare workers (residential care workers, family support workers, foster carers and sponsored childminders). These workers like the care workforce in general are increasingly thought to be in short supply. Method 2: The survey was followed by semi-structured interviews with managers in order to examine the context in which childcare workers’ careers develop. Method 3: Next a purposive sample of childcare workers was selected to represent a range of workers with their own caring responsibilities, while biographical methods were used to capture these workers’ experiences of care and care work over the life course. Method 4: A follow-up telephone survey was carried out to ascertain changes in the work-family careers of those not accessed in the biographical interviews.

Fieldwork phase As I have already hinted despite the best laid plans, research projects change. Sometimes the design changes with new methods introduced or others modified. In terms of the benefits of a particular research strategy, they are not necessarily apparent until the analysis phase of the project. Indeed the rationales that researchers give for their design and methods choices are normally post hoc - written after they have done the analysis. A comparison of these accounts with their original research proposals would no doubt be enlightening. The exemplar study I give here also drawn from my own research experience is a study where major alterations were made not to the study’s research design (longitudinal) but to the interviewing method. The study focussed on first time mothers and their return to employment following maternity leave carried out during the early 1980s (Brannen and Moss 1991; see Brannen 2004). The paradigm in which the original proposal was written was positivistic and quantitative methods presumed. The rationale for the methodological changes that were made in the course of the study were pragmatic, paradigmatic and political. The study stretched over a six year period allowing considerable scope in time for its development. The research was part of a programme carried out in the 1980s, a period when funding was more generous. It was supported by the UK’s Department of Health who provided considerable support to researchers for methodological development. The project also allowed for the introduction of a different paradigmatic position namely to take on board the preferences of the recruited research personnel, namely to study mothers’ own experiences and perspectives. An important conceptual shift took place, away from a focus mainly on behavioural outcomes (mothers’ physical and mental health) to a

17

Page 18: Mixed Methods

focus also upon meanings: how mothers made sense of their situations and responsibilities and the ways in which they and their households (the children’s fathers) organised and construed employment and parenthood. This change in theoretical / epistemological focus also had an impetus of a political nature. At that time the debate about gender inequity in the labour market and in the home was at its height and the desire to find out from women about how they experienced these inequities was considered paramount. These rationales of paradigm, pragmatics and politics translated into a change in the study’s method of interviewing, with a new set of aims that underpinned the collection of qualitative as well as quantitative data. The result was an interview schedule which combined structured questions (the responses to which were categorised according to predefined codes) with open-ended questions which gave scope for probing (responses were transcribed and analysed qualitatively). The researchers remained committed to collecting the structured data originally promised but required the interviewers to collect such data while seeming to adopt a flexible, in depth mode of interviewing. Indeed this combined interviewing approach was so successful that, in one of the later waves of the longitudinal study when, for resource reasons, we decided to collect only quantitative data, we found the interviewees reluctant to comply; they continued to respond in the way they had done in the earlier semi-structured interviews. These changes were well made in that their benefits became apparent in the analysis of the different data. For the different types of responses generated by using interviewing method to generate structured and unstructured material represented the experiences of the mothers in all their complexity and ambiguity. The return to full-time employment in children’s early years was unusual in Britain in the early 1980s with the dominant ideology still favouring full-time motherhood (Brannen and Moss 1991). Many mothers therefore experienced ambivalent feelings about returning to work in that context as well as being subject to conflicting practical demands of home and work. The development of a methodology which allowed for the expression of contradictory views and feelings was therefore an important development in this study: the responses women gave to single closed questions differed from the narratives in which they embedded their experiences. These different types of data illuminated moreover broader theoretical concerns and served to confront the contradictions in, and to highlight, the fragmented and multi-faceted nature of human consciousness and also drew out the interpenetration of dominant ideologies of the times with personal concerns and practicalities of everyday life (Brannen and Moss 1991: 7; Brannen 2004). Analysis phase In exploring how researchers apply mixed methods in the analysis of data, my exemplar studies are in the field of education and are taken from a recent (2005) Special Issue of the International Journal of Social Research Methodology: Theory and Practice (IJSRM) devoted to mixed methods research. This is a small sample of mixed methods studies and inevitably biased. However the studies have the advantage of straddling the continuum of different types of mixed methods designs. The mixed method studies are of three types:

18

Page 19: Mixed Methods

(a) Mixed methods studies where the quantitative component of the study is the more dominant and the researchers are known for their quantitative work; in these cases this preceded the qualitative component. I refer to these as QUAN > qual studies (4 studies)

(b) Mixed methods studies where the qualitative component has priority and the researchers identify themselves primarily as qualitative researchers; the qualitative component similarly followed on from the quantitative component - Quan > QUAL studies (2 studies)

(a) Lastly a study in which the quantitative component came last and where neither approach appears to dominate – QUAL>QUANT (1 study)

(a) QUAN > qual studies In this group of studies, quantitative researchers writing about using mixed methods in this Special Issue have as their main rationale the need for a longitudinal research design for analysing change (Plewis and Mason 2005: 188; Sammons et al 2005: 213) rather than mixed methods per se. Some argue the need for particular statistical techniques such as regression analysis, clustered samples and multi level modelling. The use of qualitative methods to study particular groups or sites are likewise justified in terms of another research design – case study – rather than focussing upon the benefits of a mixed methods strategy. The qualitative case studies were selected on the basis of the quantitative findings collected from the longitudinal study in order to assist in the clarification of the latter. Thus Hoyles et al in their study of children’s mathematical reasoning took as their main method longitudinal quantitative methods in order to track children’s progress in mathematics - in terms of attainment and reasoning. They sampled children attending randomly selected schools within nine geographically diverse English regions. They also strategically selected samples - particular groups of pupils from the quantitative study for qualitative investigation, notably those whose progress in mathematics reasoning decreased over time (lower at Time 2 than at Time1) and interestingly where the children’s ability to calculate mathematically increased. They employed interviews with students to explore these findings further. Similarly, Sammons et al (2005) employed a longitudinal quantitative design to explore the effects of pre-school education on children’s attainment and development at entry to school (the sample was drawn from six English local authorities and six types of pre-school provision). They selected a small number of the early education centres from their original sample on the basis of their contrasting profiles. A feature of these studies is that a relatively early stage of the research process there appears to be a transformation of the data: data from the qualitative case study are transformed into a quantitative form during data processing. Thus Hoyles et al in their study of pupils’ mathematical reasoning turn their qualitative data into four types of reasoning signalled by mathematical symbols ‘c1 to c4’ (p. 229). Similarly, Sammons et al (2005) who carried out qualitative case studies of early education centres coded the qualitative data so that the ‘reduced data’ (p219) were used to provide statistical explanations for the outcome data produced in the quantitative longitudinal study. Thus a key concept that was derived from the qualitative data analysis appears to have been transformed into a quantitative variable and correlated with outcome variables.

19

Page 20: Mixed Methods

In this latter case, it seems that qualitative data were used to explain the quantitative results. Indeed in this latter study the considerable extent to which the quantitative component dominated this study is evident in the strategy whereby the fieldworkers doing the qualitative case studies were ‘protected’ from knowledge of the quantitative results, and so were not biased by them. This suggests that the study in question while using different methods analysed the data within a similar set of epistemological assumptions deriving from a quantitative paradigm. Yet, in justifying their research strategy, it is interesting that the rationale given for different methods is in terms of offering ‘complementary strengths’ and minimising weaknesses ‘associated with reliance on only one paradigm’ (p221). That is the researchers in reflecting upon their use of mixed methods seek to identify themselves less in terms of a single paradigm while their practice suggests the methods are applied very much from within a particular paradigm. In a third example, Blatchford (2005) justifies using a mixed method strategy and also a longitudinal design to investigate the relationship between classroom size and pupils’ educational achievement. (The quantitative sample consisted of 10,000 children from a random selection of schools.) Blatchford’s rationale is framed in terms of the power of mixed methods to ‘reconcile inconsistencies in previous research’. Quantitative information was required to examine associations or relationships statistically - class size, adult-pupil ratios, teacher time and pupil behaviour etc. He also noted that qualitative methods were needed to assess such relationships in particular case studies. However he suggests that in the analysis phase ‘priorities had to be set and some areas of investigation received more attention than others.’ (2005: 204). It seems that in this study the analysis was restricted to exploring causal relations – the statistical links between class size and educational achievement. This latter type of quantitative analysis dominated despite the researchers having collected ‘fine grained data on classroom processes’ that lend themselves to other kinds of analysis, for example to do with understanding process of learning in different classroom environments. Thus there is limited use here of the qualitative data. As Blatchford himself notes, the qualitative data could have been analysed both qualitatively and quantitatively. Rather it seems that some transformation of the qualitative data into quantitative data took place during the data processing phase. On the other hand, the key to deciding whether qualitative data should be treated as such or transformed into quantitative data lies in their depth (see O’Cathain and Thomas 2004). Of course the depth of qualitative data is likely to be affected by the interview approach or other method used to prompt the respondent. On the other hand, the quantitative researcher who applies a qualitative method may be wary of analysing qualitative data as such since not all respondents answer questions in equal depth. Thus a qualitative analysis may be unrepresentative of the whole quantitative sample. (b) Quan > QUAL studies The ‘cases’ of dominant qualitative methods with a subsidiary quantitative component are taken again from my own research. In discussing research design above I referred to the more highly resourced qualitative study (in terms of researcher time) being preceded by a questionnaire survey. I have done several such studies. The study

20

Page 21: Mixed Methods

quoted above concerned children’s concepts of care and their contribution to family life (Brannen et al 2000). The data analysis stuck mainly to the aims: with the survey providing contextual information for the study and sample and the interviews providing understandings of children’s experiences of living in different types of families and their own concepts of care. An examination of the book (Brannen et al 2000) shows that the survey was used for particular purposes, for example to provide extensive data on reported behaviour on particular items that were listed in the questionnaire such as children’s contribution to family work and their views and reports of parental practices carried out by mothers and fathers, issues that were not systematically explored in the interviews. However in reflecting upon the methodology of the study the authors focused upon the picture provided by the study as a complex whole and concluded that focusing upon children’s perspectives, even accessed via qualitative methods, do not enable children to emerge as authors of their own stories of family life. Rather their understandings have to be pieced together by researchers drawing upon multiple sources of data, including the reports of their mothers, and also a number of different qualitative methods that were linked to the interviews. In the book no play is made of the fact of the mixed methods design and there is no transformation of data. A second study in which I was engaged, Young people’s health and family life (Brannen et al 1994), adopted a similar mixed method strategy: a self completion questionnaire survey conducted with young people in schools in a multi-ethnic area of London and a subset of young people and their mothers and fathers who were interviewed in depth. Broadly the two methods were conceived as addressing complementary aims. The questionnaire study sought to provide descriptive and contextual data while the interview study was intended to understand process, focusing upon the processes of negotiation of responsibility for health between adolescents and their parents. The survey also included a sampling aim – to find young people and their parents for the interview study. In addition the survey sought to provide some of the type of data generated in large scale surveys of young people’s health. Thus the study’s survey data could be compared with these data and thus address the generalisability of our results generated in a metropolitan context. An inspection of the book (Brannen et al 1994) shows that the balance of the analysis was qualitative. Several of the chapters do however interweave evidence presented quantitatively with the qualitative evidence; some of the former is presented in tabular form while most is presented thematically or as case analysis via typologies. Qualitative and quantitative data appear side by side in some chapters, for example patterns of young people’s ill health and satisfaction with services derived from the survey are set alongside parents’ and young people’s interview accounts of the persons to whom young people first reported illness, who identified the particular health problem and the kinds of lay solutions proffered. In a number of instances qualitative and quantitative data collected on the same issues (using similar questions in the survey and interviews) are compared and found to be contradictory. For example, reported drug use by young people is higher in the survey than in face to face interviews. This is explained in terms of the method: the survey method of self completion questionnaire (not anonymous) making it easier for young people to admit to drug use than being questioned in an interview by an adult. Discrepant results emerged on young people’s reports about visiting the GP unaccompanied with the interview study showing more young people doing this than in the survey. This is

21

Page 22: Mixed Methods

interpreted in terms of the lapse of time between methods, with the interview having taken place when the young people were older. On the other hand, there are instances in which qualitative and quantitative data address similar issues and broadly support one another, although the authors are careful to note that the questions were not identical. For example a survey question about young people’ relationships with their parents is supported by an analysis not only of what the young people said when interviewed but also according to their parents’ accounts. The interviews – conducted with all three parties (young people, mothers and fathers) - supplement the survey picture revealing the closest relationships during young people’s adolescence to exist between mothers and daughters and least close between fathers and daughters. The data here are transposed in the sense that the concepts are assumed to have some correspondence across the two methods of data collection. (c ) QUAL> QUAN Examples of qualitative methods preceding quantitative methods are less easy to find. An exception is the use of ‘pilot ‘questioning to develop coded questions for use in a survey. Hammond’s mixed methods study reported in IJSRM (Hammond 2005) is justified as part of a new programme of research on the wider benefits of adult learning (referred to above). It claims to use qualitative research as a way of establishing significant variables for isolation and examination in an existing national longitudinal data set. The rationale for the first qualitative phase (biographical interviews) is that the research area was under-researched and the research questions relatively unformulated (p241). Thus the qualitative component is conceived as a ‘mapping ‘ exercise carried out to inform the research design and implementation of the quantitative part of the study: the identification of variables for the quantitative analysis (p243). This has parallels with qualitative pilot work as a prologue to a survey although in this case the qualitative part of the study was to be analysed in its own right. The quantitative data set in question - a national cohort study which was recruited in 1958 called the National Child Development Study - was already in existence. This fact justifies, in part at least, doing the mixed methods study. However at the start of the study the rationale for using the cohort study is rather weak – being thought to be ‘potentially useful’. Interestingly this article contrasts with other studies moulded in the quantitative tradition in this Special Issue. Despite using a quantitative longitudinal data, this author is insistent that causal outcomes should not be inferred from the quantitative evidence. While suggesting that the qualitative data is used to identify appropriate analysis for the QUAN part of the programme she also insists that that these data should not be used to explain quantitatively arrived at outcomes but to interrogate them further (p 244). Unusually for studies in educational research this author goes on to cite results found in the quantitative analysis that apparently contradict the analysis from the biographical interviews. Hammond reports that the effect of adult learning on life satisfaction found in the cohort analysis was greater for men than for women while women reported themselves in the interviews to be positive about the courses they had taken. On this issue the biographical interviews were regarded as being ‘more sensitive’ than the quantitative measure. The interview data showed that improved sense of well being in the present was not necessarily incompatible with a negative

22

Page 23: Mixed Methods

view of the future. The quantitative analysis was found wanting in having conflated satisfaction with ‘life so far’ and with ‘life in the future’. Similarly, another finding from the NCDS study suggesting marginal benefits to individuals of taking several adult education courses was modified by the earlier qualitative evidence that taking courses may act as a replacement activity for those who lacked informal support networks, thus balancing out the additional beneficial effects of courses. A further contradiction is explained in term of the lack of representativeness of the qualitative study. One explanation for why this researcher took issue with the different results produced by the two methods is perhaps the priority given to the biographical interviews in the first place and perhaps the identification of the researcher in question with a qualitative approach (although this is not clear). In any event the biographical interviews were conducted before the secondary analysis of the pre-existing quantitative data and were used to shape the latter. Hence the qualitative data threw up hypotheses while the quantitative data set could be used to reject or confirm the qualitative evidence. Another factor perhaps is that the researcher in question had no stake in creating or shaping the quantitative data set (since these data had already been collected). Indeed while the research design was shaped by the existence of this cohort study, there was no protocol suggesting the ways in which it might be useful in pursuing this programme of research (on life long learning). However what is interesting about the rationale given for this iterative use of qualitative work to test quantitative evidence (that is in turn refuted by the qualitative evidence) is the rationale given: namely the way mixed method strategy can pose new lines of questioning (Green, Caracelli and Graham, 1989) – a result not necessarily anticipated at the outset of the research project. Contextualisation In many studies, it is common to draw upon or refer to data analyses that are not part of the primary data collected. In national studies it has become commonplace also to carry out secondary analysis of large scale data as the archiving of data has become more common. Such contextualisation is relevant at all phases of the research process, for example shaping a line of enquiry during the analysis of data as well as at the later stage when the conclusions are drawn and the overall analysis interpreted. Bringing contextual data into the frame can inform the conceptualisation of new research. For example, the new sociology of sleep has arisen in the context of an awareness that research on sleep has been dominated by hard science in which researchers have been concerned with the physiological aspects of sleep as related to ageing and sleep disruption. The realisation of this limited focus prompted Hislop and Arber (2003) to turn their attention to the psycho-social factors of sleep patterns in relation to later life. Bringing in contextual data from other sources than the empirical research project can be much more than a literature review. Contextualisation is particularly important in cross-national research. Indeed as Linda Hantrais (2005) shows for the welfare family cluster of EU projects under Framework 5, multi-method research is typical.

23

Page 24: Mixed Methods

Empirical studies of a qualitative kind are supplemented by mapping exercises, literature reviews and secondary analysis of large-scale data sets. In a current EU-funded study concerning the transition to parenthood we have done case studies in two types of organisations in seven countries (www.workliferesearch.org/transitions). The main fieldwork method involves interviews and focus groups in organisations (to capture the discourses in the workplace about being a working parent) followed by biographical interviews with a sub group of parents and their partners (with a focus on managing their work-family lives and the transition to parenthood). In this project, it is crucial to contextualise our empirical data in broader terms not only the organisations but also the national contexts and characteristics. The sociologists in the team were particularly mindful of the tendency in some qualitative enquiry to place undue emphasis upon the perspectives of informants. This is a particular problem in qualitative cross-national research in which the context cannot be taken for granted. In an earlier cross-national study of young people’s views of work-family life, we found that young people made little reference to their structural context and the constraints upon their lives. For example, young Norwegians university students displayed what we termed a ‘confident planning mentality’ about their future lives as parents and workers but failed to suggest how such feelings of mastery and independence were premised upon the support of a strong welfare state in Norway (Brannen and Nilsen 2002). It was therefore important to reveal the link between the individual’s sense of agency within the structural context and inject this into the interpretation of the data analysis. Thus in this current project we are using the mapping exercise involving secondary analysis and commentary from a demographer in putting our empirical material into context. We are also making use of our national research teams. For example, in writing up the individual biographies we have added two phases to our analysis: (a) an exercise involving noting separately the contextual features in each interview summary as well as a summary of individuals’ perspectives ; and then (b) to exchange our case summaries with partners in other countries who will question what is unclear or omitted in the descriptions of the context. 5. Other issues in mixed method research In conclusion, brief mention will be made of three other important issues that need to be addressed in mixed methods research. As yet there is rather little guidance available on these issues. Quality criteria for assessing mixed methods research How can we assess the use of and the claims made for mixed methods? This is a difficult issue since, as has been suggested, the rationales for methodological decisions are often justified in the light of the way data have been analysed and the questions addressed in writing up the research. These are not necessarily the same rationales given at the outset of the research when the research proposals were written. Even in accounts of methodological practice, as in the case of the Special Issue of The International Journal of Social Research Methodology (2005) discussed

24

Page 25: Mixed Methods

above, these issues are far from transparent and in some cases involve a slippage between the general claims researchers may make for their use of mixed methods and their particular practices. Universal agreement seems to have been reached that quality concepts developed for quantitative research such as generalisability, validity, reliability and replicability cannot nor ought not to be applied to qualitative research (Spencer et al 2003). Rather, drawing upon Lincoln and Guba (1985), broadly equivalent concepts can be found that apply to qualitative research. For example:

• Credibility/ trustworthiness : internal validity • fittingness : external validity • auditability : reliability

In doing mixed methods research how far do we work with these separate criteria or do we develop new specific or convergent criteria for mixed method research? As Bryman (in press b) suggests, the criteria we use is likely to depend upon the dominance of the qualitative or quantitative method and type of data analysis used within the project. Thus if the qualitative component is dominant, then it may be more appropriate to use the criteria by which such research is judged and similarly when the quantitative component dominates, although a further consideration is how far the different results are integrated in the overall analysis. However currently the solution is less obvious or satisfactory where both qualitative and quantitative components are equally significant. Bespoke or convergent criteria may be required here. As Tashakorri and Teddlie (2003b) suggest, a new nomenclature could be created; they suggest the term ‘inference quality’ as a substitute for validity/ trustworthiness in order to convey the quality of the conclusions that can be drawn from a study. Teaching and learning mixed methods How should mixed methods research be taught? The organisation of research methods teaching tends to separate qualitative and quantitative methods. Typically the process of learning is also sequential so that a student is introduced to qualitative methods and then to quantitative methods or vice versa. Tashakorri and Teddlie (2003) argue that students should be exposed from the start of a course to mixed methods research strategies. However this may prevent the student from developing a firm grounding in either. Moreover since many students seem to find the hardest part of the research process to be the formulation of research questions, it is important that mixed methods courses are taught with this in mind so that students are introduced to exemplars of studies where different research questions clearly suggest different research methods. That the issues may be more deep seated must also be taken into account namely students’ prior allegiances to particular philosophical assumptions and paradigms (that may be assumed to require particular methodological approaches). One suggestion is to ensure that mixed methods research is taught in a way that links methods and data more closely to theoretical concerns as well as to research questions. A mixed methods training course is to be distinguished from a multi methods course in that the former will directly address issues arising from combination of methods

25

Page 26: Mixed Methods

within a single study, rather than cover a number of separate methods (Bazeley 2003). Thus students may be exposed to possibilities in method choice as well as being expected to achieve proficiency in different methods. Given the rather different exigencies of quantitative and qualitative methods, attention needs to be given to how best to train researchers in mixed methods research. Some qualitative methods require skills that have to be learned and practiced over extended time periods in order that the researchers becomes sufficiently expert. Qualitative methods typically require considerable reflexivity on the part of the individuals practicing them. Thus courses that introduce students to new methods should not constitute substitutes for proper apprenticeships in the relevant method and approach. Mixed methods courses should not be short cuts to training researchers fully in particular methods and should allow for extended training and apprenticeship. Writing up mixed methods research What models are there for writing up mixed methods research? The answer is that there is a lack of exemplary studies that demonstrate different ways of writing up evidence based on different methods. This is unsurprising since, as we have noted, this is not straightforward. For one thing academic journals tend to be organised around disciplines and may favour particular types of research. Moreover different types of data analyses may sit awkwardly together on the published page and may require rather a lot of space to justify their validity and credibility. Some researchers using mixed methods may for such reasons report their qualitative and quantitative results separately. Researchers presenting evidence based on both qualitative and quantitative methods but drawing upon one set of evidence and under reporting the other may risk criticism for not fully exploiting the possibilities for the analysis of both data sets. Resources There are a number of resources which explore some of the issues outlined in this paper. The following links to the ESRC Research Methods Programme may be useful: http://www.ccsr.ac.uk/methods/events/Mixed/programme.htmhttp://www.ccsr.ac.uk/methods/projects/posters/bryman.shtml http://www.tandf.co.uk/journals/titles/13645579.asp References Alderson P (2001) Research by children, The International Journal of Social

Research: Theory and Practice, 4, 2 139-155 Anderson, M, F. Bechhofer, D. McCrone, L. Jamieson, Y. Li, R. Stewart (2005)

‘Timespans and Plans Among Young Adults’ Sociology 39 (1) pp. 139-155 Barbour, R.S. (1999) The case of combining qualitative and quantitative approaches

in health services research. Journal of Health Services Research Policy, 4, (1) 39-43

Bazeley, P. (2003). Teaching mixed methods. Qualitative Research Journal, Special Issue, 4, 117-126.

26

Page 27: Mixed Methods

Becker, S and Bryman A (2004) Understanding Research for Social Policy and Practice: Themes, methods and approaches Bristol: The Policy Press

Blaikie, N. (2000). Designing Social Research. Oxford: Blackwell. Blatchford, P. (2005). A multi-method approach to the study of school class size

differences. International Journal of Social Research Methodology, 8(3), 195-205.

Brannen, J and Collard J (1984) Marriages in Trouble: The process of seeking help London: Unwin Hyman Brannen, J., & Moss, P. (1991). Managing Mothers and Earner Households after

Maternity Leave. London: Unwin Hymen. Brannen, J. (1992). Mixing Methods: Qualitative and Quantitative Research.

Aldershop: Ashgate Publishing Limited. Brannen, J., Dodd, K., Oakley, A., & Storey, P. (1994). Young People, Health and

Family Life. Buckingham: Open University Press. Brannen, J., Hepinstall, E., & Bhopal, K. (2000). Connecting Children: Core and

Family Life in Later Childhood. London: Routledge. Brannen, J., & Nilsen, A. (2002). Young people’s time perspectives: From youth to

adulthood. Sociology: 36(3), 513-537. Brannen, J. (2004). Working qualitatively and quantitatively. In C. Seale, G. Gobo,

J.F. Gubrium, & D. Silverman (eds.), Qualitative Research Practice. London: Sage.

Brannen, J. (2005). Mixing methods: The entry of qualitative and quantitative approaches into the research process. The International Journal of Social Research Methodology, Special Issue, 8(3), 173-185.

Brannen and Nilsen (forthcoming) Young People, Time Horizons and Planning: a response to Anderson et al

Brannen, J., Brockmann, M., Mooney, A., & Statham, J. (forthcoming). Coming to Care: Childcare Workers’ Careers and Experiences. Bristol: Policy Press.

Bryman, A (1984) The debate about quantitative and qualitative research: a question of method or epistemology, The British Journal of Sociology, 35, 1 76-92

Bryman, A (1988) Quality and Quantity in Social Research, London: Routledge Bryman, A (2001) Social Research Methods, Oxford: Oxford University Press Bryman, A. (in press a) The research question in social research: What is its role? The

International Journal of Social Research Methodology: Theory and Practice Bryman, A (in press b) Paradigm peace and implications for quality, International

Journal of Social Research Methodology: Theory and Practice Cain, M (1993) Foucault, feminism and feeling: what Foucault can and cannot

contribute to feminist epistemology in C Ramazanoglu (ed.) Up against Foucault: Explorations of some tensions between Foucault and feminism. London: Routledge

Coser, L. (1975). Presidential address: Two methods in search of a substance. American Sociological Review, 40(6), 691-701.

Cresswell, J.W. (2003). Research Design: Qualitative, Quantitative and Mixed Method Approaches. Thousand Oaks, CA: Sage.

Denzin, N.K. (1970). The Research Act: A Theoretical Introduction to Sociological Methods. Englewood Cliffs, NJ: Prentice-Hall.

De Vaus, D. (2001). Research Design in Social Research. London: Sage. Duncan, S and Harrop, A (in press) A user perspective on research quality

International Journal of Social Research Methodology: Theory and Practice

27

Page 28: Mixed Methods

Ertzberger C and Kelle U (2003) Making inferences in mixed methods: the rules of integration. In Tashakorri and C Teddlie (eds) Handbook of Mixed Methods in Social and Behavioural research, Thousand Oaks CA: Sage

Fielding N and Schreier M Introduction: On the compatibility between qualitative and quantitative methods FQS (Forum: Qualitative Social Research) 2, (1) e-paper http://www.qualitative-research.net/fqs-texte/1-01/1-01hrsg-e.htm

Finch, J. (1986). Research and Policy: The Uses of Qualitative Methods in Social and Educational Research. Lewis: Falmer Press.

Foucault, M., & Cain, M. (1993). Foucault, feminism and feeling: What Foucault can and cannot contribute to feminist epistemology. In C. Ramazanoglu (ed.), Up Against Foucault: Explorations of Some Tensions Between Foucault and Feminism. London: Routledge.

Glaser, B.G., & Strauss, A.L. (1967). The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago: Aldine.

Green, J., Caracelli, V.J., & Graham, W.F. (1989). Towards a conceptual framework for mixed-method evaluation designs. Education, Evaluation and Policy Analysis, 11(3), 255-274.

Graham, H (1983) Do her answers fit his questions? Women and the survey method. In E Gamarnikow, D Morgan, J Purvis and D Taylorson The Public and the Private. London: Heinemann

Halfpenny, P. (2005). The reflection between quantitative and qualitative research, paper presented at Mixed Methods Workshop, ESRC, Research Methods Programme, Manchester, October.

Hammersley, M The Dilemma of Qualitative Method: Herbert Blumer and the Chicago Tradition, London: Routledge

Hammersley, M. (1996). The relationship between qualitative and quantitative research: Paradigm loyalty versus methodological eclecticism. In J.T.E. Richardson (ed.), Handbook of Research in Psychology and the Social Sciences. Leicester UK: BPS Books

Hammersley, M. (2000). Varieties of social research: A typology. The International Journal of Social research Methodology: Theory and Practice, 3(3), 221-231.

Hammersley, M. (2005). Troubles with triangulation. Paper presented at Mixed Methods Workshop, ESRC, Research Methods Programme, Manchester, October.

Hammond, C. (2005). The wider benefits of adult learning: An illustration of the advantages of multi-method research. International Journal of Social Research Methodology, 8(3), 239-255.

Hantrais, L. (2005). Combining methods: A key to understanding complexity European Societies, 7 (3) 399-421.

Hislop, J., & Arber, S. (2003). Sleepers awake! The gendered nature of sleep disruption among mid-life women. Sociology, 37(4), 695-711.

Hoyles, C., Küchemann, D., Healy, L., & Yang, M. (2005). Students’ developing knowledge in a subject discipline: Insights from combining quantitative and qualitative methods. International Journal of Social Research Methods, 8(3), 225-238.

Kelle, U. (2001) Sociological explanations between micro and macro and the integration of qualitative and quantitative methods, FQS (Forum: Qualitative Research), 2, 1: e-paper. http://www.qualitative-research.net/fqs-eng.htm

Kuhn, T. (1970). The Structure of Scientific Revolutions (2nd ed.). Chicago: University of Chicago Press.

28

Page 29: Mixed Methods

Lincoln, Y.S. and Guba, G.E. (1985) Naturalistic Enquiry, Beverley Hills: CA: Sage Mason, J. (2002). Qualitative Researching (2nd ed.). London: Sage. Mertens, D. (1998). Research Methods in Education and Psychology: Integrating

Diversity with Quantitative and Qualitative Approaches. Thousand Oaks, CA: Sage.

Mertens, D (2003) Mixed methods and the politics of social research: the transformative-emancipatory perspective in A Tashakorri and C Teddlie Handbook of Mixed Methods in Social and Behavioural Research. London: Sage.

Moran-Ellis, J., Alexander, V.D., Cronin, A., Dickenson, M., Fielding, J., Sleney, J., and Thomas, H. (in press). Triangulation and integration: Process, claims and implications. Qualitative Research.

Morgan, D.L. (1996) The relationship between qualitative and quantitative research: Paradigm loyalty versus methodological eclecticism. In J.T.E. Richardson (ed.) Handbook of Research in Psychology and the Social Sciences, Leicester UK: BPS Books

Morse, J.M. (2003). Principles of mixed method and multi-method research design. In C. Teddlie and A. Tashakkori, Handbook of Mixed Methods in Social and Behavioural Research. London: Sage.

Oakley, A (1999) Paradigm wars: some thoughts on a person and public trajectory, The International Journal of Social Research methodology: Theory and Practice, 2, (3) 247-255

O’Cathain, A and Thomas, K. J. (2004) Any other comments? Open questions on questionnaires – a bane or a bonus to research? BMC Medical Research Methodology, 4:25

Pawson, R (1995) Quality and quantity, agency and structure, mechanism and context, dons and cons, Bulletin de Methodologie Sociologique, 47, 4-48

Plewis, I., & Mason, P. (2005). What works and why: Combining quantitative and qualitative approaches in large-scale evaluations. International Journal of Social Research Methodology, 8(3), 185-194.

Reiss, A.L. (1968). Stuff and nonsense about social surveys and participant observation. In H.L. Becker, B. Geer, D. Riesman, and R.S. Weiss (eds.), Institutions and the Person: Papers in Memory of Everett C. Hughes. Chicago: Aldine.

Ribbens, J., & Edwards, R. (1998). Feminist Dilemmas in Qualitative Research: Public Knowledge and Private Lives. London: Sage.

Ritchie, J (2003) The application of qualitative research methods in Ritchie, J., & Lewis, J. (eds). Qualitative Research in Practice: Public Knowledge and Private Lives. London: Sage.

Rossman, G.B., & Wilson, B.L. (1994). Numbers and words revisited: Being ‘shamelessly eclectic’. Quality and Quantity, 28, 315-327.

Sammons, P., Siraj-Blatchford, I., Sylva, K., Melhuish, E., Taggard, B., & Elliot, K. (2005). Investigating the effects of pre-school provision: Using mixed methods in the EPPE research. International Journal of Social Research Methods, 8(3), 207-224.

Sieber, S.D. (1973). The integration of fieldwork and survey methods. American Journal of Sociology, 78(6), 1335-1359.

Smith, J., & Hershusius, L. (1986). Closing down the conversation: The end of the quantitative-qualitative debate among educational enquiries. Educational Research, 15, 4-12.

29

Page 30: Mixed Methods

Spencer, L Ritchie, J, Lewis, J and Dillon, L (2003) Quality in Qualitative Evaluation: A framework for assessing research evidence, A Quality Framework, London: Cabinet Office.

Tashakkori, A. and Teddlie, C (2003a). Handbook of Mixed Methods in Social and Behavioural Research. London: Sage.

Tashakorri, A and Teddlie, A (2003b) Issues and dilemmas in teaching research methods courses in social and behavioural sciences: a US perspective, The International Journal of Social Research Methodology: Theory and Practice, 6, 1, 61-79

Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in the social and behavioural sciences. In C. Teddlie and A. Tashakkori, Handbook of Mixed Methods in Social and Behavioural Research. London: Sage.

Teddlie, C., & Tashakkori, A. (2003). The past and future of mixed method research: From data triangulation to mixed model designs. In A Tashakorri and C. Teddlie (eds), Handbook of Mixed Methods in Social and Behavioural Research. London: Sage.

Thomas, A (1992) Working with a disability: Barriers and facilitators, London SCPR Thompson, P. (2004). Researching family and social mobility with two eyes: Some

experiences of the interaction between qualitative and quantitative data. The International Journal of Social Research Methodology, 7(3), 237-259.

Walby, S (1997) Gender Transformations, London: Routledge

30