Qualitative Data Analysis

‘Student L’ Lee James Cox

B0049872

MA in Technical Communication

1 Module: 14-7104-OOY-B-20123 – Portfolio of Research Skills (A-2012/3)

This presentation aims to show:

How I would go about analysing and presenting interview data

based on example of:

Transcripts of 5 text chat interviews with ‘Student L’ in 2006/7

for a research question of:

How well a University module supported them in developing research skills

2

Considerations for Research Analysis

3

Collate Analyse Present

Reliability Investigation that the source data is

complete and reliable.

Coding Assigning codes to simplify and standardize data for analytical

purposes.

Agree purpose and format Clear scope and question. Ensure format is suitable for audience.

Reshaping Ordering of chaotic data into a

standard usable format.

Annotating Drawing attention to what you consider to be most significant.

Validity Describing potential bias, methods

followed and any subjective conclusions.

Reducing Removing ‘noise’ not relevant to the

research.

Labelling Pigeon-holing data so it can be easily

grouped together.

Interpretation Explaining what the analysis may

mean and themes.

Transforming Moving data into a format that is

easier to manipulate.

Selection Choosing of interesting, significant or representative items to illustrate an

argument.

Conclusions Answering the research question and

making recommendations

Summarising Choosing one or more examples as a

synopsis of whole data set.

Note that the research question, sampling, methodology and actual collection have already been performed and are out of scope for this presentation.

(Adapted from check-lists in ‘How to Research’, 2010 [1])

4

Collate How would I the findings?

1. Check source data reliability – Screen grabs rather than open documents that can be manipulated – Researcher and subject confirmation that comments are complete and not edited – Filter biased or leading questions

2. Remove personal details by ‘blacking out’ things we shouldn’t know. E.g.

– Student is called L. Goldwin, has 2 children – one born in 2006, the other in 2008. – Was recently living in Israel (08/09) working in Tech Publishing for a software company

3. Remove ‘chit-chat’ dialogue that helped to build a relationship even if they may have had some influence on answers

4. Transfer the data into a format that reduces it, and makes it easier to manipulate, e.g. using Excel with columns headings: 5. Add in context to comments – to make them understandable and to make it easy to draw attention to important comments later

5

Analyse How would I the findings?

1. Categorise the data – to create new data points that can be extracted from the free form comments. This helps the proving or disproving of theories in any presentation of conclusions.

E.g. If the subject area was difficult (according to questionnaire feedback) did it get most negative statements or the most

positive suggestions? (See Attitude System [2] for more detailed examples)

2. Care with coding – categories must be

clearly defined allowing any 3rd party to

independently conclude the same choice.

3. Counting - statistically counting and

correlating data sets can reveal interesting

trends and relationships between data. E.g. was the subject simply in a good mood on 8/12/6

4. Map it – social network analysis tools may describe potential relationship between comments. Does a positive on one statement affect the next?

“I see the interactions as the greatest asset of this form of studying. ”

Present How would I the findings?

1. Agree a suitable format for the audience. Report with technical data or presentation with hand-

outs?

2. Establish why the research is representative – findings should be valid & reliable – Bias and leading questions from researcher declared

– Without breaking privacy or other confidentialities, establish why listen to this students’ comments

3. Directly answer the research question as an executive summary: – E.g. ‘Our research indicated that it is likely to be the most difficult topics

such as Paradigms that helps students to develop better research skills.’

– Give weight to the summary by establishing a baseline (even if only assumed) around

predefined measures of what ‘good’ looks like, e.g. Mijksenaar’s gauge [4]

4. Share the key learnings (both positive and negative) from the data analysis – Extract direct quotes from subject to give weight to conclusions

– Use of graphs and tables to summarise ‘workings’ and support conclusions

5. Validate my conclusions and ask for feedback from the

original researcher – Florence Dujardin

0123456789 Realis affect

Irrealis affect

Judgements

Appreciations

Weaknesses & Strengths of Qualitative Data Analysis Weaknesses Strengths

Bias of researcher and their positional power over subject: means data may have inherent reliability issues

Provides depth and detail. A single insight can be elevated to prominence in the presentation rather than ‘lost’ in the data.

Content analysis is complex: associating occurrences of specific words in open dialog of what is relevant

Even a small data set makes data analysis possible.

Mistakes can occur in categorisation of data. Detailed knowledge of the subject is sometimes required to follow the flow of a conversation.

Describes people's individual experiences: a picture can be built up about why people act in certain ways and their feelings about these actions

Privacy more difficult to protect when full disclosure of transcripts are required to show reliability of source data.

More open: data can lead to expansion of subjects responses to new topic areas not initially considered

Key data points can be lost in the transfer the source text in to format that can be analysed.

Likely to avoids pre-judgements

7

Conclusions The sample contained a mix of methods – open interview questions as well as a specific questionnaire that used a fixed rating system. The richness of the content is likely to reveal interesting points for further cross-tabulation and investigation which will make interesting reading. However the analysis phase will take some considerable time because of the intertwined nature of natural dialog and the requirement to understand the subject matter to ‘code’ responses. Meyer & Avery [2] succinctly describe the challenge of Qualitative Analysis: ‘Researchers are faced with an overwhelming amount of data’..they are often interested in precisely the connections and nuances that are frequently lost when filtering and synthesizing.’ Unfortunately there is also likely to be some question over the validity of any findings because:

i. There is only one research subject. A different subject might have completely opposite views* ii. The interviewer has positional influence over the interviewee

*Nielsen [6] suggests a minimum for 3-5 is needed for heuristic evaluation whereas Baker & Edwards [7] between 15 and 40.

8

References

[1] Blaxter, L., Hughes C. and Tight M. (2010). How to Research. McGraw Hill. 211-277.

[2] Meyer, D., Avery, L. (2009) Excel as a Qualitative Data Analysis Tool, Field Methods, 21 (1): 91-112

[3] Lipovsky, C. (2008) Attitude System in 'Constructing affiliation and solidarity in job interviews', Discourse & Communication, 2(4): 413-416

[4] Mijksenaar P. (1997) Visual Function: an Introduction to Information Design, 010 Publishers

[5] Nielsen, J. (2005) How to conduct a heuristic evaluation. Nielsen Norman Group. Online. Last accessed 11/12/13 at http://www.useit.com/papers/heuristic/heuristic_evaluation.html

[6] Baker, S. and Edwards, R. (2013). How many qualitative interviews is enough? National Centre for Research Methods Review Paper. Online. Last accessed 14/12/13 at http://eprints.ncrm.ac.uk/2273/4/how_many_interviews.pdf

9