Confidential 1

SpeechCycle Confidential

Optimizing Natural Language Optimizing Natural Language Interfaces: Interfaces:

No Data Like More DataNo Data Like More Data

SpeechTEK New York, 2007SpeechTEK New York, 2007

Jonathan Bloom & Roberto PieracciniJonathan Bloom & Roberto Pieraccini

Confidential 2

SpeechCycle ConfidentialSpeechCycle Confidential

Just as speech scientists crunch data to optimize a speech recognizer,

speech companies need to crunch data to optimize a call at the dialog level

as well.

Voice user interface (VUI) design needs to be based on quantitative data as

much as possible.

We will provide an example of quantitative research in the area of VUI

design.

We will ask, “What is the future of this research-based VUI approach?”

Executive Summary

Confidential 3

SpeechCycle ConfidentialSpeechCycle Confidential

VUI = RELIGION + SCIENCE

Confidential 4

SpeechCycle ConfidentialSpeechCycle Confidential

Where there is data, we use it. At those times, it is a science.

All else is faith-based.

Religion is a fine thing. But our customers do not pay us for theology

lessons. They pay us to save them $$$ and keep their customers happy.

VUI = RELIGION + SCIENCE

Confidential 5

SpeechCycle Confidential

As much as possible, speech companies need to collect data in order to optimize their voice user interfaces at

every level – and in every facet - of the interaction, from recognition accuracy to prompt wording to dialog structure. In doing so, VUI will become less anecdotal

and more scientific.

The Point

Confidential 6

SpeechCycle Confidential

One needs the ability to compare two or more versions of an application…

…running at the same time.

…taking calls from the same population of callers.

…taking calls from that population at random.

…gathering data at a fast enough pace to meet customer deadlines.

Requirements for Exploration

Confidential 7

SpeechCycle Confidential

Designing Exploration Alternatives

Confidential 8

SpeechCycle Confidential

Example

“All of our agents are currently helping other customers. Let’s get started with our automated internet troubleshooter, so you don’t have

to wait. [chime] To begin, briefly describe the problem, saying something like “I can’t connect to the internet”, or you can say “What

are my choices”.

“All of our agents are currently helping other customers. Let’s get started with our automated internet troubleshooter, so you don’t have

to wait. [chime] Are you calling because you’ve lost your internet connection? [pause] Please say yes or no.”

Confidential 9

SpeechCycle Confidential

Example

RESULTS

INTRO followed by SLM = 20.4% automation rate

INTRO followed by YES/NO question = 22.6% automation rate

(CHI square) Statistically significant at .05 level

Confidential 10

SpeechCycle Confidential

Summary (Up To Now)

Strong opinions and qualitative usability tests should not be the only sources of VUI knowledge.

With the right tools, continual access to data, and with enough data, speech companies can make dialog experimentation a regular part of the product lifecycle.

At this point, other than the randomization script in the call flow, a lot of this process is manual, so now we need to ask, “How can we make this process more automatic?”

What’s next?

Confidential 11

SpeechCycle Confidential

Reinforcement Learning theory

A finite choice of actions

What the system does

A finite or infinite set of interaction states

Identify the factors that can influence the choice of the best action

Policy

Choose the action based on current state of the interaction

Maximization of a return function

Reinforce the choice of the actions that provide a positive return at any particular state

The process converges to a locally optimal policy

Confidential 12

SpeechCycle Confidential

Reinforcement Learning in dialog research

Research has proven that reinforcement learning can be used for automating the design of dialog systems

(Markov Decision Processes) E. Levin and R. Pieraccini (AT&T) , 1997

(Partially Observable Markov Decision Processes) S. Young (Univ. Cambridge, 2004)

Unfortunately, full dialog learning and optimization requires a lot of interactions

Academic research uses simulated users

Restricting the optimization to a small number of reasonable “competing designs”

Alternative competing designs at different points in the application

Exploration and Exploitation principle

Confidential 13

SpeechCycle Confidential

Conclusions

VUI design—from religion to science

Use data to validate and chose optimal design among competing alternatives

Reinforcement Learning—from theory to practice

Use data to optimize applications while they interact