1© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Applications of distributed dialogue systems:

The KTH Connector

Jens Edlund Anna Hjalmarsson

Aalborg, November 10th, 2005

2© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Why spoken interfaces?

Speech is good because it is• Hands free• Eyes free• Intuitive – already known• Robust – e.g. redundancy, grounding• Flexible• Responsive• Efficient• …

3© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

What do they do?• Purpose

— Problem solving— Information seeking— Transactions— Control— …

• Initiative — System— User— Mixed

• Modality— Multimodal (input and/or output)— Unimodal (input and/or output)

• …

4© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

What else do they do?

Transactions, e.g. ATIS

Web forms (and sales reps)

Call routing, e.g. HMIHY

DTMF (and operators)

Info services, e.g. weather

Web forms, DTMF

Control, e.g. smart rooms

Buttons, remote controls

Spoken interfaces often replace or complement existing automated interfaces, for example:

5© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Alternative interface systems

• Speech is an alternative or substitute• Commonly built to be as good as or

better than the corresponding system• Symmetry often required – what can

be done with the original system should be doable with speech and vice versa

6© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Why spoken interfaces?

• Hands free• Eyes free• Intuitive – already known• Robust – e.g. redundancy,

grounding• Flexible• Responsive• Efficient• …

• Hands free• Eyes free• Intuitive – already known• Robust – e.g. redundancy,

grounding• Flexible• Responsive• Efficient• …

These aspects are often not exploited much in alternative interface systems

7© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

What metaphor to rely on?

• The voice as an input device?—”You may use your voice to order”

From a travel booking instruction in Swedish

—”It didn’t give me any alternatives”From a post interview with call routing user

• The computer as a human?—Problem: Turing test

8© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Summary

• Speech can be used successfully as an alternative or complementary interface to other interfaces, particularly when hands and/or eyes are occupied, disabled, or otherwise impratcical to use

• The advantages of speech promised by analogies to human-human communication may not be fully exploited in such domains

9© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

The KTH Connector

• Background• Domain• System

10© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Background: CHIL

• CHIL: Computers in the Human Interaction Loop (EU funded, IP506909)

• The dialogue system as an unobtrusive conversational partner in a group of humans

11© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

A telephony based secretary

Wide range of complexity• From answer phone…

12© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

A telephony based secretary• …to meeting assistant

13© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Dialogue setup

Multimodal, multiparty,

system barge-in

Multipartytelephony

14© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

System highlights

Interpretation Generation

Skype IP telephony

User

System

ASR /nailon/

PICKERING

GALATEAKnowledgebase

DialogueManager

TextGeneration

TTS

1a) words

2) concepts

3) discourseresolved concepts 4) discourse

resolved concepts

5) worldresolved concepts

1b) potentialend-points

6) concepts

7a) text

7b) concepts

VAC VAC

0) Voice input 8) Synthesised speech output

Interpretation Generation

Skype IP telephony

User

System

ASR /nailon/

PICKERING

GALATEAKnowledgebase

DialogueManager

TextGeneration

TTS

1a) words

2) concepts

3) discourseresolved concepts 4) discourse

resolved concepts

5) worldresolved concepts

1b) potentialend-points

6) concepts

7a) text

7b) concepts

VAC VAC

0) Voice input 8) Synthesised speech output

System output is included in

discourse model(incrementally)

Prosody enhanced

endpointing

System may dial users

15© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Research highlight: Responsiveness

• When should we respond?

• Turn yielding & turn holding cues: —Prosody—Gaze —…

16© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Research highlight: Incrementality

• When can we respond?— After a ”long enough” silence?— At some semantic or syntactic completeness?— After any ”word”?— Anytime?

• What has actually happened?— System and user barge-in— Keep track of what we say, as well as what

the user says

17© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Research highlight: Unobtrusiveness

• With what should we respond?—Long prompts can be annoying—Short ones may be insufficient

• How?—Efficiently or politely?—Speech, gesture, other?

• Do we have to ”take turn”? —Backchannels—Grounding

18© 2005 CHIL KTH ASIDE 2005, Aalborg, 2005-11-10

Applications of distributed dialogue systems: The KTH Connector Jens Edlund & Anna Hjalmarsson

Thank you for your attention.