A Scientific Approach to Improving VUI Design
description
Transcript of A Scientific Approach to Improving VUI Design
A Scientific Approach to Improving VUI Design
Peter U. Leppik, CEORick Rappe, VP of Business Development
Vocal Laboratories Inc.
Agenda
1. Principles of Gourmet Customer Service
2. Data Collection Toolbox
3. The Project Life-Cycle
4. Conclusion/Q&A
Principles of Gourmet Customer Service
• Nearly every effort to improve service makes at least one of these mistakes:
1. No data is collected from actual callers.2. The data collected does not relate to business
goals.3. There is not enough data collected to be
statistically significant.4. There is no context or understanding of the data
(for example: what is a “good” satisfactions score?).
5. No action is taken based on the data collected.
The Scientific Approach1. Set Goals
o Use relevant business goals, not technical metrics
2. Gather datao Multiple kinds of tests may be needed to get a complete
picture
3. Take action
4. Validateo Repeat the tests from Step 2.
5. Commit: Go back to Step 1.
Data Collection Toolbox
• There is no one way to collect data about a VUI or call center.o A VUI designer who only uses one test method
is like a carpenter who only owns a hammer.
The Toolbox: General Tools• Test Goals
• Statistical Considerationso Margin of Erroro Probability of Finding Problems
• Prototypeso Functional Prototype (aka “Rapid Prototype”)o Wizard-of-Oz Prototype
The Toolbox: Idea Generating Tools
• Heuristicso “Best Practices”o Expert Opinions, “Audits,” “Assessments”
• Agent Feedback
• Call Recording and Monitoring
The Toolbox: Data Generating Tools
• Controlled Testingo Lab Testso Rapid Assessmentso Large Controlled Tests
• Follow-Up Surveys
• End-of-Call Surveyso Not Recommended due to extreme sample bias
The Toolbox: More Data Generating Tools
• Friends & Family & Employee Testso Not Recommended due to unrepresentative sample
• Automated Load Testing
• Traversal Testing
• Call Logs and Stats
The Project Life-Cycle
• Every project goes through certain phases.o Analysiso Designo Codeo Testo Releaseo Operational
• Every phase has different testing requirements
Analysis Phase• Lay out the project scope, goals, and budget• Ask the big questions
o Who are these callers, and how do they live?o Why are they using the phone and not (e.g.) the
web?o What is the competition doing?
• Look at the big picture and big ideas:o Generate ideas from consultants, vendors,
agents, and customerso Develop baselines & benchmarks
Analysis Phase Tools• Controlled Testing
o Benchmarking, competitive analysis, baseline measurementso Need to understand your callers before you start
• Agent Feedback
• Follow-up Surveys
• Call Recordings
• Heuristicso “Best practices,” consultants’ opinions, etc.
Design Phase
• This is when you plan the details of the VUI
• Use a Prototype Test Prototype Test iterative design cycle
• The keys to success are speed and flexibilityo Test several designs, sometimes in the same dayo Don’t blow the entire testing budget on The
Usability Test
Design Phase Tools
• Prototypeso WOZ or Functional Prototype
• Controlled testingo Rapid Assessment or Lab Test o Lets you use an iterative design processo Nothing ends arguments faster than solid
data.
Code Phase
• Implement the design
• Most testing is for debuggingo Unit testing and some functional testing
• Relatively little VUI testing takes place hereo Implementation may require some VUI changes;
these should be tested
Test Phase
• Last stage before rolling out the application to live callers
• This is where the most thorough testing takes place
Test Phase Tools
• Traversal testo Ensures that the implementation matches the
design
• Large controlled testso Designed to catch even small usability glitches
• Automated load testingo Done last because changes in the first two test
phases can affect load handling
The Test Phase can also be the Acceptance Phase
• Business goals are what are important, not the exact designo They are what should be in the contract.o Tests which measure business goals should be
the acceptance criteria.
• Many of these tests can be used in the design phase.
Release Phase
• This is the process of rolling out the application
• By this time, you should be confident that the system workso Confident doesn’t mean cocky
• This is the time for a phased roll-outo Some call it a “pilot test”o This is a deployment strategy, not a testing strategy
Operational Phase
• The work doesn’t end when the system is live!o Application Drifto External changes
• Company• Caller Lifestyle• Competition
• This looks a lot like the analysis phase for the next project….
Operational Phase Tools• Benchmarking (Controlled Testing)
• Competitive analysis (Controlled Testing)
• Follow-up surveys
• Agent feedback
• Monitoring call stats and logs
• Listening to call recordings
Lessons Learned1. Good service does not have to be more
expensive than bad service
2. Customers are smarter than machines
3. Every system can stand some improvement
4. The best VUI designs come from paying attention to the right things at the right time
Q & A