Speech-to-Speech MT in the JANUS System Lori Levin and Alon Lavie Language Technologies Institute...

Speech-to-Speech MT in the JANUS System

Lori Levin and Alon Lavie

Language Technologies Institute

Carnegie Mellon University

4/24/00 UMD Seminar 2

Outline

• Design and Engineering of the JANUS/C-STAR speech-to-speech MT system– Fundamentals of our approach

– System overview

– Engineering a multi-domain system

• The C-STAR Travel Domain Interlingua (IF)• Evaluation and User Studies• Conclusions, Current and Future Research


JANUS Speech Translation

• Translation via an interlingua representation

• Main translation engine is rule-based

• Semantic grammars

• Modular grammar design

• System engineered for multiple domains

• Incorporate alternative translation engines


Multilingual Interlingual Machine Translation


Advantages of Interlingua

• Avoid the n-sqared problem for all-ways translation.

• Mono-lingual grammar development teams.

• Add a new language easily and automatically get all-ways translation to all previous languages.


The C-STAR Travel Planning Domain

General Scenario:

• Dialogue between one traveler and one or more travel agents

• Focus on making travel arrangements for a personal leisure trip (not business)

• Free spontaneous speech


The C-STAR Travel Planning Domain

Natural breakdown into several sub-domains:

• Hotel Information and Reservation

• Transportation Information and Reservation

• Information about Sights and Events

• General Travel Information

• Cross Domain


A Travel DialogueTranslated from Italian

A: Albergo Gabbia D’Oro. Good evening.

B: My name is Anna Maria DeGasperi. I’m calling from Rome. I wish to book two single rooms.

A: Yes.

B: From Monday to Friday the 18th, I’m sorry, to Monday the 21st.

A: Friday the 18th of June.

B: The 18th of July. I’m sorry.

A: Friday the 18th of July to, you were saying, Sunday.

B: No. Through Monday the 21st.


A Travel Dialogue(Continued)

B: So with departure on Tuesday the 22nd.

A: Then leaving on the 22nd. Yes. We have two singles certainly.

B: Yes.

A: Would you like breakfast?

B: Is it possible to have all meals?

A: No. We serve meals only in the evening.

B: Ok. If you can do breakfast and dinner.

A: Ok.

B: Do you need a deposit?


A Travel Dialogue(Continued)

A: You can give me your credit card number.

B: Ok. Just a moment. Ok. My name is Anna Maria DeGaperi. The card is 005792005792.

A: Good.

B: Expiration 2002.

A: 2002. Good. Thank you. We need a confirmation on the 18th of July before 6pm.

B: Goodbye.

A: Thanks. Goodbye.

B: Thanks. Goodbye.


A Non-Task-Oriented Dialogue(We can’t translate this.)

A: Are you cooking?

B: My father is cooking. I’m cleaning. I just finished cleaning the bathroom.

A: Look. What do you know about Monica?

B: I don’t know anything. Look. I don’t know anything.

A: You don’t know anything? I wrote her three weeks ago, but if she hasn’t received the letter, they would have returned it. I hope she received it.

B: Because Celia told me that the address that Monica had given us was wrong. She said that if I was going to write to her, well, ….


Semantic Grammars

• Describe structure of semantic concepts instead of syntactic constituency of phrases

• Well suited for task-oriented dialogue containing many fixed expressions

• Appropriate for spoken language - often disfluent and syntactically ill-formed

• Faster to develop reasonable coverage for limited domains


Semantic Grammars

Hotel Reservation Example:

Input: we have two hotels available

Parse Tree:

[give-information+availability+hotel]

(we have [hotel-type]

([quantity=] (two)

[hotel] (hotels)

available)


The JANUS-III Translation System


The SOUP Parser

• Specifically designed to parse spoken language using domain-specific semantic grammars

• Robust - can skip over disfluencies in input• Stochastic - probabilistic CFG encoded as a

collection of RTNs with arc probabilities• Top-Down - parses from top-level concepts of the

grammar down to matching of terminals• Chart-based - dynamic matrix of parse DAGs

indexed by start and end positions and head cat


The SOUP Parser

• Supports parsing with large multiple domain grammars

• Produces a lattice of parse analyses headed by top-level concepts

• Disambiguation heuristics rank the analyses in the parse lattice and select a single best path through the lattice

• Graphical grammar editor


SOUP Disambiguation Heuristics

• Maximize coverage (of input)• Minimize number of parse trees (fragmentation)• Minimize number of parse tree nodes• Minimize the number of wild-card matches• Maximize the probability of parse trees• Find sequence of domain tags with maximal

probability given the input words: P(T|W), where T= t1,t2,…,tn is a sequence of domain tags


Modular Grammar Design

• Grammar development separated into modules corresponding to sub-domains (Hotel, Transportation, Sights, General Travel, Cross Domain)

• Shared core grammar for lower-level concepts that are common to the various sub-domains (e.g. times, prices)

• Grammars can be developed independently (using shared core grammar)

• Shared and Cross-Domain grammars significantly reduce effort in expanding to new domains

• Separate grammar modules facilitate associating parses with domain tags - useful for multi-domain integration within the parser


Translation with Multiple Domain Grammars


Analysis with Multiple Domain Grammars

• Parser is loaded with all domain grammars

• Domain tag attached to grammar rules of each domain

• Previously developed grammars for other domains can also be incorporated

• Parser creates a parse lattice consisting of multiple analyses of the input into sequences of top-level domain concepts

• Parser disambiguation heuristics rank the analyses in the parse lattice and select a single best sequence of concepts


A SOUP Parse Lattice


Alternative Analysis Approach: SALT

SALT - Statistical Analyzer for Lang. Translation• Combines ML trainable and rule-based analysis

methods for robustness and portability• Rule-based parsing restricted to well-defined set of

argument-level phrases and fragments• Trainable classifiers (NN, Decision Trees, etc.) used to

derive the DA (speech-act and concepts) from the sequence of argument concepts.

• Phrase-level grammars are more robust and portable to new domains


SALT Approach

• Example:Input: we have two hotels available

Arg-SOUP: [exist] [hotel-type] [available]

SA-Predictor: give-information

Concept-Predictor: availability+hotel

• Predictors using SOUP argument concepts and input words

• Preliminary results are encouraging


Design Criteria of the Interchange Format

• Suitable for task oriented dialogue• Based on speaker’s intent, not literal meaning• Domain independent framework with domain-

specific parts• Simple and reliable enough to use:

– at multiple research sites. – with widely varying type of parsers and

generators


Domain Actions: Extended, Domain-Specific Speech Acts

Examples:

c:request-information+availability+room

a:give-information+personal-data

c:give-information+temporal+arrival


Task Oriented Sentences

• Perform an action in the domain.

• Are not descriptive.

• Contain fixed expressions that cannot be translated literally.


Components of the Interchange Format

speaker a: a: (agent)

speech act give-informationgive-information

concept* +availability+room+availability+room

argument* (room-type=(single & double), (room-type=(single & double), time=md12)time=md12)


Examples no that’s not necessary c:c:negatenegate

yes I am c:c:affirmaffirm

and I was wondering what you have in the way of rooms available during that time c:c:request-informationrequest-information+availability+room+availability+room

my name is alex waibel c:c:give-informationgive-information+personal-data+personal-data (person-name=(given-name=alex, family-(person-name=(given-name=alex, family-

name=waibel))name=waibel))

and how will you be paying for this a:a:request-informationrequest-information+payment+payment (method=question(method=question))

I have a mastercard c:c:give-informationgive-information+payment+payment (method=mastercard)(method=mastercard)


Speaker Tag

Client says: Do you take credit cards?

c:request-information+payment (method=credit-card)

Agent says: Will you be paying with a credit card?

a:request-information+payment (method=credit-card)


Size of IF

May 1999

Speech acts 54 Concepts 84 Arguments 118


Speech Acts

acceptacknowledgeacknowledge-actionaffirmaffirm-actionapologizeclosingdelay-actionend-actiongive-certaintygive-information

I’ll take that, Sounds goodOkay, Sure, YeahHere you go, This is itYes, That is correctYes, please do, Go aheadSorry, I’m sorryBye, See you next weekI’ll get back to you on thatThat’s all for nowI’m sureI have 2 singles available


Speech Acts

greetinggreeting-nice-meetgreeting-requestgreeting-response-badgreeting-response-goodgreeting-welcomeintroduce-selfintroduce-topicnegatenegate-actionnot-understand

Hello, Good morningNice to meet youHow are youI’m not goodI’m fineWelcome to PittsburghThis is Brian, Best WesternAbout that flight…NoNo, don’tI don’t understand


Speech Acts offeroffer-informationoffer-repeatplease-waitreject (e.g., offer)

request-actionrequest-affirmationrequest-certaintyrequest-delay-actionrequest-informationrequest-introduce-self

How about it?Let me get you the informationLet me repeat thatJust a minute, Let me seeNo, I don’t want thatCan you reserve that for me?Is that correct?Are you sure?Can I get back to you later?Do you accept visa?Who am I speaking with?


Speech Acts request-knowledgerequest-neg-affirmationrequest-repeatrequest-suggestionrequest-verificationreturn-from-delaysuggestthankverifywelcomex-exclamation

Do you know?Is that bad?Could you repeat that?Which hotel should I get?Right?, That was 40 dollars?I’m backHow about a single?Thank you very muchYes, that is 40 dollars.You’re welcomeThat is beautiful! (ETRI only)


Meta-Demo Speech acts

testing

testing-problem

testing-start

testing-stop

testing-proceed

testing-request-proceed

testing-ready

testing-present

testing-request-present

Testing 1 2 3, This is a test

We have a problem

Let’s start

Let’s stop

Go ahead!

Would you go first

Ready here

We are here, CMU is on line

Are you there?


Some Concepts

• Actions: change, reservation, confirmation, cancellation, help, purchase, view, display, preference

• Attributes: availability, price, temporal, price, location, size, features etc.

• Objects: room, hotel, flight, tour, event, attraction, web-page etc.

• Other: arrival, departure, numeral, expiration-date, payment


Using Concepts to Represent Information Focus

Is there a hotel in Pittsburgh?

c:request-information+availability+hotel (location=pittsburgh)

Is the hotel in Pittsburgh?

c:request-information+location+hotel (location=pittsburgh)


Topic vs Focus

The Hilton Hotel is in Verona.

a:give-information+location+hotel

(hotel-name=hilton, location=verona)

The hotel in Verona is the Hilton Hotel.

a:give-information+location+hotel

(hotel-name=hilton, location=verona)


The Interchange Format Database

61.2.3 olang I lang I Prv IRST “telefono per prenotare delle stanze per quattro colleghi”

61.2.3 olang I lang E Prv IRST “I’m calling to book some rooms for four colleagues”

61.2.3 IF Prv IRST c:request-action+reservation+features+room (for-whom= (associate, quantity=4))

61.2.3 comments: dial-oo5-spkB-roca0-02-3

d.u.sdu olang X lang Y Prv Z “sdu-in-language-Y on one line”d.u.sdu olang X lang E Prv Z “sdu-in-English on one line”d.u.sdu IF Prv Z dialogue-act-on-one-lined.u.asdu comments: your commentsd.u.asdu comments: go here


The Interchange Format Database

English Dialogues

English Sentences

Korean Dialogues

Korean Sentences

Italian Dialogues

Italian Sentences

Japanese Dialogues

Japanese Utterances

Distinct Dialogue Acts

36

2466

70

1142

5

233

124

5887

554 (310 agent, 244 client)


Phenomena Not Covered

Relative Clauses

Plurality

Descriptive Sentences

Scope (negation and modifiers)

Comparative Constructions

Anaphora


Expressivity vs Simplicity

• If it is not expressive enough, components of meaning will be lost.

• If it is not simple enough, it can’t be used reliably across sites.


Coverage

• The database includes about 550 distinct dialogue acts.

• About 60 dialogue acts cover about 70% of the data.

• About 5% of unseen data wasn’t covered (as judged by human experts)


Consistency of Use Across Sites

• Successful international demo.

• After testing English-Italian and English-Korean, Italian-Korean worked without extra effort.

• Inter-coder agreement for each component of IF individually (speech acts, concepts, arguments) around 85%

• Cross-site evaluation same as intra-site evaluation: 60% spoken; 75% transcribed.


User Studies• We conducted three sets of user tests• Travel agent played by experienced system user• Traveler is played by a novice and given five

minutes of instruction• Traveler is given a general scenario - e.g., plan a trip

to Heidelberg• Communication only via ST system, multi-modal

interface and muted video connection • Data collected used for system evaluation, error

analysis and then grammar development


Evaluations

• Accuracy Based Evaluation– Translation preserves original meaning

• Task Based Evaluation– goal success or failure– user effort: how many attempts before

succeeding or giving up


Accuracy Based Evaluation

• End-to-end evaluations conducted at the SDU (sentence) level

• Multiple bilingual graders compare the input with translated output and assign a grade of: Perfect, OK or Bad

• OK = meaning of SDU comes across• Perfect = OK + fluent output• Bad = translation incomplete or incorrect


Task Based Evaluation

I would like to reserve #1s a single room #2frequest-action+reservation+hotel (room-type=single)

Translation: I would like to reserve a seating room.


Task Based Evaluation

Scoring Scheme:For goals that succeed: 1/n

For goals that fail: -(1-1/n)

where n is the number of attempts

Overall score: average for all goals


August-99 Evaluation

• Data from latest user study - traveler planning a trip to Japan

• 132 utterances containing one or more SDUs, from six different users

• SR word error rate 14.7%

• 40.2% of utterances contain recognition error(s)


Accuracy and Task Based Evaluations

Agent Traveler All

ABE 58.7% 44.7% 51.8%

TBE score .75 .56 .65

TBEsuccess

82.8% 64.7% 73.8%


Accuracy Based Evaluation

Method OutputLanguage

PercentAcceptable

Grader Number ofGraders

1. Speech Rec. English 78% CMU 32. Transcr. English 74% CMU 33. Rec. English 59% CMU 34. Transcr. Japanese 77% CMU 25. Rec. Japanese 62% CMU 26. Transcr. German 70% CMU 27. Rec. German 58% CMU 28. Transcr. German 67% IRST 29. Rec. German 59% IRST 210. Transcr. Italian 73% IRST 611. Rec. Italian 61% IRST 6


Evaluation - Progress Over Time

Method OK+Perfect Perfect

Jan-99 Transcribed 69% 46%

Apr-99 Transcribed 70% 49%

Aug-99 Transcribed 74% 54%

Jan-99 Recognition 55% 36%

Apr-99 Recognition 57% 38%

Aug-99 Recognition 59% 42%


Current and Future Work• Expanding the travel domain: covering descriptive

as well as task-oriented sentences• Development of the SALT statistical approach and

expanding it to other domains• Full integration of multiple MT approaches:

SOUP, SALT, Pangloss• Disambiguation: improved sentence-level

disambiguation; applying discourse contextual information for disambiguation


Conclusions• We started skeptically with tools that we thought

were too simple: context-free parser, semantic grammar, interlingua based on domain actions.

• We were surprised that they worked adequately for some types of task oriented dialogue.

• We improved portability.• We are now working on embedding the simple

task-oriented system into a more complete system.


The JANUS/C-STAR Team

• Project Leaders: Lori Levin, Alon Lavie, Monika Woszczyna, Alex Waibel

• Grammar and Component Developers: Donna Gates, Dorcas Wallace, Taro Watanabe, Boris Bartlog, Ariadna Font-Llitjos, Marsal Gavalda,Chad Langley, Marcus Munk, Klaus Ries, Klaus Zechner,Detlef Koll, Michael Finke, Eric Carraux, Celine Morel,Alexandra Slavkovic, Susie Burger, Laura Tomokiyo,Takashi Tomokiyo, Kavita Thomas, Mirella Lapata,Matthew Broadhead, Cortis Clark, Christie Watson,Daniella Mueller, Sondra Ahlen

Speech-to-Speech MT in the JANUS System Lori Levin and Alon Lavie Language Technologies Institute...

Documents

Transcript of Speech-to-Speech MT in the JANUS System Lori Levin and Alon Lavie Language Technologies Institute...