Interviews & Focus Groups

KSE966/986 Seminar - Fall 2012/Spring 2013 May 10, 2013Uichin Lee

Contents

1. Pros and cons of interviews2. Applications of interview research3. Who to interview4. Interview strategies5. Interviews vs focus groups6. Types of questions7. Conducting an interview8. Electronically mediated interviews 9. Analyzing interview data

What is interview?Gathering direct feedback from

interested individualsSurvey: reaching out a large number

of people, but often restrictive ◦Open-ended questions receive short (or

no) answersAlternative: direct discussions with

concerned participations◦ Interviews with individuals◦Focus groups involving multiple users at

one time

1. Pros and cons of interviews

Pros◦Go deep◦Flexible

Cons◦Skill to manage (unbounded discussion?

Setting up meetings, listening/taking notes)

◦Time and resource intensive (both interviewing and analyzing content)

◦Recall problems (similar to surveys)

2. Applications of interview researchInitial exploration

Requirements elicitation

Evaluation and Subjective Reactions

2. Applications of interview researchInitial exploration

Starting from clean slate, try to understand users’ goals/needs, to probe new possibilities, etc. by asking broader questions

Requirements elicitation

Evaluation and Subjective Reactions

Breaking the Disposable Technology Paradigm: Opportunities for Sustainable Interaction Design for Mobile Phones (CHI 2008)Goals:

◦ How people understand the lifespan of their phones

◦ What factors, such as style, service contracts, and functionality, affect how they attribute value to their phones, and their awareness and actions regarding mobile phone sustainability

Methods: ◦ a 34-question web survey (n=79) ◦ a series of in-depth, semi-structured phone

interviews with participants selected based on survey responses (who suggested an interesting history or experience) (n=10)

How do people organize their desks? Implications for the design of office information systems (ACM TOIS 1983)

Goals:◦ In the early 1980s, desktop info sys is relatively new; understanding how

people organize info was incomplete◦ Goal is to understand how people organize info

Methods: Interviewed 10 workers in their offices

◦ Workers are asked to describe the layout of the office (indicating where info is stored and why) Only clarification questions are asked Some are asked to find some documents..

◦ Also asked about information practices: How well organize (Likert scale) Biggest problems? Do you keep lists of things to do? Do you keep a calendar of appointments? How often are you unable to do…

Results: files (structured) vs. piles (unorganized, but also acting as reminders)

2. Applications of interview researchInitial exploration

Requirements elicitation◦Try to understand needs/concerns/goals

during the early stage of design

Evaluation and subjective reactions◦Try to get summative evaluations of

completed products (e.g., usability surveys) (say for planning design revision)

3. Who to interview Beyond users – Stakeholders

◦ Anyone who may be affected by the use of a system◦ There could be multiple types of users (or related people)

E.g., hospital info systems; patients and their family members

Interview representatives of different groups from different perspectives

Key informants who are knowledgeable about how work is done◦ Particularly useful and informative individuals◦ Must be selected carefully to avoid bias

When selecting, respect the participants (e.g., reaching out homeless people on the street or people with disabilities)

4. Interview strategies:Types of Interviews

Fully Structured

Semi-Structured

Unstructured

Less structure: greater difficulty in conducting and interpreting interview

But

More opportunity for insight

4. Interview strategies:Types of InterviewsFully structured – Orally-presented survey

◦Stick with the script.◦Good for comparison across individuals

Semi-structured – pre-specified questions serve as starting point for discussion◦Digression is okay

Unstructured – initial question, possible list of topics, but no pre-defined script ◦Less structure good for open ended

exploration

4. Interview strategies:Focused & Contextual InterviewsGo beyond asking questionsAsk participant to

◦ Demonstrate use of technology (instead of explaining how to do)

◦ Rely on artifacts (papers, photos, etc.) and context ◦ Use external aids to elicit feedback or reaction (known

as “probes” e.g., props or prototypes)

Example:◦ Asking a user to show the office and tell how to store

things◦ Asking a user to show how tech is used at home in

detail◦ Asking to use a functional prototype and complete

some tasks

5. Interviews vs focus groups

Interviews take time◦1 hour or more/response◦Several hours for analyzing notes

Focus groups◦More people in less time ◦Up to 8-12 people at once.

5. Interviews vs focus groupsFocus Groups: Pros & Cons

Pros◦Broad range of viewpoints and insights◦Each group will likely have at least one

person who will stimulate others to talk

Cons◦Hard to manage group dynamics◦Generally can't be fully structured◦May need to ask fewer questions◦Selection can be challenging

6. Types of questions Closed-ended Questions

◦ “On a scale of 1-10, 10 being best, how did you like the web page?”

◦ Easy to analyze, but may not be informative

Yes/no questions◦ Remember, the goal is to get interviewees to give you useful

insight◦ Simple questions get simple answers◦ “Did you like the home page?” You'll get a one-word answer

Open-ended questions◦ Invite elaboration, discussion: e.g., What did you think about the

web page?”◦ Ask users to complete a sentence: “My favorite web browser

feature is...”◦ Conceptual mapping: draw pictures or layouts to describe

understanding of a situation or problem

6. Types of questionsSimple questions – no jargon

Avoid compound questions with multiple parts◦Not "“What were the strengths and weaknesses

of the menu layout and the toolbar?”◦Ask two separate questions instead

Avoid judgmental phrasing or tone◦Not “Don’t you think this is hard to use?”◦Possible bias, instead ask “What did you think?”

6. Types of questionsQuestions in un- or semi-

structured interviews◦Often, questions are generated in

response to participant comments◦Can be hard to do this well◦Start with structured interviews

Get a few under your belt before moving to unstructured

7. Conducting an interview

PreparationRecording responsesDuring the interview

◦Rapport◦Introduction◦Getting down to business◦Promoting discussion◦Debriefing

7. Conducting an interviewPreparationPilot test – w/ colleagues and

participants◦Good for logistics and for confidence

Write an interview guide listing what to do and when◦Particularly good if multiple

researchers are involvedLogistical backups

◦Batteries for laptop, audio recorder, extra paper, etc.

7. Conducting an interviewRecording responses: notes

• Audio and video recordings are fine, but

• Paper notes are still important– Record insights, non-verbal responses,

etc.– Try to record what you can, but– Don't do so at the expense of listening

• Summarize written notes as soon as possible after the interview

before you forget...

7. Conducting an interviewRecording responses: audio/videos

• Complete, but expensive (transcribing and interpreting responses)

• Video is tricky, but gets useful information

• Consider audio + still pictures• Respect privacy and anonymity

7. Conducting an interviewDuring the Interview

• You're the Host: Build Rapport• First few minutes are crucial• Be professional, friendly, respectful, nonjudgmental

• If so, interviewees are inclined to trust you and provide honest and useful feedback

• Listen carefully• Providing refreshments is a good idea (e.g., water, snacks)

• Avoid loud, crunchy food though

• Outline• Briefly introduce research goals (use script if possible to

be consistent)• Tell that you’re recording the session and how• Complete paperwork (informed consent)• Simple questions first, hard questions later

7. Conducting an interviewDuring the Interview

• Be flexible• If your interview is not fully

structured• But, try to keep things on track• Explain why you are asking each

question• Define terms, avoid jargon• Ask for clarification

7. Conducting an interviewDuring the Interview

Read between the lines…◦Is the interviewee telling you what

they think you want to hear? (hoping to please researchers)

◦If so, make a note of it◦Might want to downplay in

interpretation

7. Conducting an interviewDuring the Interview

Challenges of focus groups◦Manage the room. Be prepared to

deal with Digressions Arguments

◦Give everyone a chance to talk Address them directly “Joan, what do you think about...?”

7. Conducting an interviewDuring the Interview

Promoting discussion◦What if they won't talk?◦Fully-structured – not much to do◦Otherwise

Rephrase questions Dig deeper into specifics

◦Use props and probes to stimulate feedback

◦Focus groups – ask for dissenting or concurring feedback

7. Conducting an interviewClosing it out: DebriefingAsk for any final comments

◦ Interviewees might make additional useful comments

Provide more details about research goals

Brief summary of findingsSay “thanks”!After thanking the users, spend some

time to reflect and summarize notes immediately

Consider turning off recording devices

8. Electronically mediated interviews Phone, web chat, email,

conference calls◦Pros

Easy, inexpensive Reach out more people with less effort

◦Cons Lack of face-to-face contact Fewer non-verbal cues Pacing can be harder

9. Analyzing interview dataDo it as soon as possibleAvoid “cherry-picking”Fully-structured, closed-ended: tabulate

answersOpen-ended questions require coding

(audio transcribe data, written notes)◦Organizing responses to various categories

e.g., affinity diagrams: hierarchical groupings of structures and themes, built from the bottom up

Critical incidence analysis: if interviews are used for examining stories (e.g., a notable failure of an excising system)

Summary

1. Pros and cons of interviews2. Applications of interview research3. Who to interview4. Interview strategies5. Interviews vs focus groups6. Types of questions7. Conducting an interview8. Electronically mediated interviews 9. Analyzing interview data