Conducting Longitudinal Diary Studies

Amanda Nance

Sage

June 10, 2009

Early diary study

Goal of this presentation

• How to run a better diary study

– Create effective template

– Understand the data

– Keep users engaged

• See benefits of the method

Topics for this presentation

• Reason for the project

• About the diary study

• Diary study tips

• Unique contributions of the diary method

Reasons for the project

• Assess “getting started” experience of our desktop accounting software

• Focus on tasks after initial set up

• Create requirements for next release

Used multiple methods to get truer picture

Site visits Support calls Survey analysis

Researcher calls Diary studyDiary from: http://www.flickr.com/photos/bdorfman/15846725/

About the diary study

• 5 participants: small business owners, bookkeepers

• Diaries kept for up to 8 weeks beginning at install

• Diaries submitted weekly by email

• Compensation – at least $40/week

About the diary study

• Provided template in Word to record

– Daily Work: tools they used, other tasks

– Issues and Comments: problems with our product

Tips

Recruit extra participants

• Recruited 12

• 5 completed

• Reasons for drop outs were interesting findings, too

Use a recruiter to save time

• Pros and cons

• They can recruit and more– Email documents– Remind of deadlines

– Collect diaries

Communicate deadlines

• Communicate due date for first submission

• Disqualify participants who don’t respond

• Close the loop

Template tips

Make it easy for participants

• Keep it short

– This study: 3 questions daily, 6-7 per incident

• Separate questions from answers

Separate questions from answers

Show sample entries

What did you do for your business today?

• I answered several customer phone calls.

• Called automated bank system to make sure we have enough money for payroll.

• Called in payroll to ADP (we outsource payroll).

• Ordered parts for one of our jobs by phone.

Ask for screenshots

“Looking at Aged Receivables…. They do not match. Why are they different?”

Tips for screenshots

• If they are important, require them

• Include instructions

Structure template for easy analysis

• Separate daily entries from critical incidents

• Include participant name and week number in footer and file name

Iterate on the template

• Prior to launching

• As you receive user feedback

Process tips

Require daily entries

• Keeps participants engaged

• Gives you a reason to pay them

• Opportunity to learn about the broader work context

Talk to the participants yourself

• Intro call to understand business and product expectations

• Follow up on unclear entries

• Provide support

Research tips

• Read entries as soon as possible

• Leave time to research

Data analysis

• No different from other methods

• Evaluate data– Excel to categorize theme, second theme, week #

• Share and act on data– Summarize key findings, provide recommendation

for each

– Use customer quotes

– Share complete diaries

Unique contributions of diary study

• Infrequent tasks

• Help use

• Users resolving issues

• Tasks over time

Reasons to run a diary study

• No travel required

• Non-intrusive, in context

• Cheaper than site visits

Engagement and meaningful data

• Pay for more complete responses

• Encourage useful responses through feedback on recent entries

• Continue participants who are useful

Key Points

• Daily entries

• Explain what you expect

• Sample entries

• Reasonable number of questions

• Personal contact with researcher

• Questions?

Contact

More tips in conference proceedings

anance@gmail.com

Twitter: @amandaux