Lecture 16: Dialoguearitter.github.io/courses/5525_slides_v2/lec16-dialogue.pdfFull Dialogue Task...

Lecture16:Dialogue

AlanRi4er(many slides from Greg Durrett)

ThisLecture

‣ Chatbotdialoguesystems

‣ Task-orienteddialogue

‣ OtherdialogueapplicaAons

Chatbots

TuringTest(1950)‣ ImitaAongame:AandBarelockedinroomsandanswerC’squesAonsviatypewriter.BotharetryingtoactlikeB

OriginalInterpretaAon:

trainedjudge

trainedjudgeC trainedjudge

OriginalInterpretaAon: StandardInterpretaAon:

trainedjudgeC trainedjudge

OriginalInterpretaAon: StandardInterpretaAon:

‣ Thetestisnot“doesthiscomputerseemhuman-liketorandompeoplewithawebbrowser?”

ELIZA‣ Created1964-1966atMIT,heavilyscripted

‣ DOCTORscriptwasmostsuccessful:repeatsuser’sinput,asksinanequesAons

Weizenbaum(1966)

‣ IdenAfykeyword,idenAfycontext,applytransformaAonrule

(.*)you(.*)me

WhydoyouthinkI$2you?

(.*)you(.*)me

My(.)(.*)

Whatelsecomestomindwhenyou thinkofyour$1?

(.*)you(.*)me

‣ Veryli4leneedtogeneratenewcontent,butcanonlyhaveonetypeofconversaAon

My(.)(.*)

Whatelsecomestomindwhenyou thinkofyour$1?

Cleverbot‣ Carpenter(1986),onlinesystembuiltin2006

User:Hitherecleverbot.Cleverbot:Howareyou?User:Doingokay.whatplansdoyouhavetoday?Cleverbot:Idorevision.User:Whatareyourevisingfor?Cleverbot:Mathshistory.User:Howoldareyou?Cleverbot:That'snoneofyourbusiness.User:Whatismathshistory?Cleverbot:Idon’tknow,farmingmaybe

‣ “Nearestneighbors”:humansaysstatementA,findahumanresponseinhuman-humanorhuman-computerchatstostatementA,repeatthat

‣ Canofengivesensibleanswers,butthebotdoesn’treallyimposehigh-level discoursestructure

Data-DrivenApproaches‣ CantreatasamachinetranslaAonproblem:“translate”fromcurrentu4erancetonextone

Ri4eretal.(2011)

‣ Filterthedata,usestaAsAcalmeasurestopruneextractedphrasestogetbe4erperformance

Ri4eretal.(2011)

Data-DrivenApproaches

Ri4eretal.(2011)

Seq2seqmodels

Whatareyoudoing

am going home [STOP]

‣ JustlikeconvenAonalMT,cantrainseq2seqmodelsforthistask

Seq2seqmodels

Whatareyoudoing

‣Whymightthismodelperformpoorly?Whatmightitbebadat?

Seq2seqmodels

Whatareyoudoing

‣Whymightthismodelperformpoorly?Whatmightitbebadat?

‣ Hardtoevaluate:

LackofDiversity

Lietal.(2016)

‣ Trainingtomaximizelikelihoodgivesasystemthatpreferscommonresponses:

LackofDiversity

Lietal.(2016)

‣ SoluAon:mutualinformaAoncriterion;responseRshouldbepredicAveofuseru4eranceUaswell

LackofDiversity

Lietal.(2016)

‣ StandardcondiAonallikelihood: logP (R|U)

LackofDiversity

Lietal.(2016)

‣MutualinformaAon:

‣ StandardcondiAonallikelihood: logP (R|U)

logP (R,U)

P (R)P (U)= logP (R|U)� logP (R)

‣ logP(R)canreflectprobabiliAesunderalanguagemodel

LackofDiversity

Lietal.(2016)

‣ OpenSubAtlesdata

Futureofchatbots‣ HowdeepcanaconversaAonbewithoutmoresemanAcgrounding?Basicfactsaren’tevenconsistent…

Lietal.(2016)Persona…

Futureofchatbots‣ HowdeepcanaconversaAonbewithoutmoresemanAcgrounding?Basicfactsaren’tevenconsistent…

‣ Canforcechatbotstogiveconsistentanswers,butsAllprobablynotveryinteresAng

Futureofchatbots

‣ XiaoIce:MicrosofchatbotinChinese,20Musers,averageuserinteracts60Ames/month

‣ HowdeepcanaconversaAonbewithoutmoresemanAcgrounding?Basicfactsaren’tevenconsistent…

Futureofchatbots

‣ XiaoIce:MicrosofchatbotinChinese,20Musers,averageuserinteracts60Ames/month

‣ Peopledoseemtoliketalkingtothem…?

‣ HowdeepcanaconversaAonbewithoutmoresemanAcgrounding?Basicfactsaren’tevenconsistent…

Task-OrientedDialogue

Task-OrientedDialogue‣ QuesAonanswering/search:

Google,what’sthemostvaluable

Americancompany?

‣ QuesAonanswering/search:

Americancompany?

WhoisitsCEO?

Americancompany?

WhoisitsCEO?

TimCook

Task-OrientedDialogue‣ Personalassistants/APIfront-ends:

Siri,findmeagoodsushi restaurantinChelsea

‣ Personalassistants/APIfront-ends:

SushiSekiChelseaisasushirestaurantinChelseawith4.4stars

onGoogle

Howexpensiveisit?

onGoogle

Howexpensiveisit?

Entreesarearound$30each

onGoogle

Howexpensiveisit?

Findmesomethingcheaper

Task-OrientedDialogue‣ Personalassistants/APIfront-ends:

HeyAlexa,whyisn’tmyAmazon orderhere?

Letmeretrieveyourorder. Yourorderwasscheduledtoarrive

at4pmtoday.

Itnevercame

at4pmtoday.

Itnevercame

Okay,Icanputyouthroughtocustomerservice.

AirTravelInformaAonService(ATIS)‣ Givenanu4erance,predictadomain-specificsemanAcinterpretaAon

DARPA(early1990s),FigurefromTuretal.(2010)

‣ CanformulateassemanAcparsing,butsimpleslot-fillingsoluAons(classifiers)workwelltoo

FullDialogueTask‣ Parsing/languageunderstandingisjustonepieceofasystem

Youngetal.(2013)

‣ Dialoguestate:reflectsanyinformaAonabouttheconversaAon(e.g.,searchhistory)

Youngetal.(2013)

‣ Dialoguestate:reflectsanyinformaAonabouttheconversaAon(e.g.,searchhistory)

‣ Useru4erance->updatedialoguestate->takeacAon(e.g.,querytherestaurantdatabase)->saysomething

FullDialogueTask

FindmeagoodsushirestaurantinChelsea

FullDialogueTask

restaurant_type <- sushi

FullDialogueTask

location <- Chelsea

FullDialogueTask

location <- Chelsea

curr_result <- execute_search()

FullDialogueTask

location <- Chelsea

SushiSekiChelseaisasushirestaurantinChelseawith4.4starsonGoogle

FullDialogueTask

location <- Chelsea

Howexpensiveisit?

FullDialogueTask

location <- Chelsea

Howexpensiveisit?get_value(cost, curr_result)

FullDialogueTask

location <- Chelsea

POMDP-basedDialogueSystems

Youngetal.(2013)

‣ POMDP:useristhe“environment,”anu4eranceisanoisysignalofstate

Youngetal.(2013)

‣ Dialoguemodel:canlooklikeaparseroranykindofencodermodel

Youngetal.(2013)

‣ Generator:usetemplatesorseq2seqmodel

Youngetal.(2013)

‣ Generator:usetemplatesorseq2seqmodel

‣Wheredorewardscomefrom?

RewardforcompleAngtask?

location <- Chelsea

make_reservation(curr_result)

Howexpensiveisit?

OkaymakemeareservaAon!

RewardforcompleAngtask?

location <- Chelsea

make_reservation(curr_result)

Howexpensiveisit?

OkaymakemeareservaAon!

Veryindirectsignal ofwhatshould happenuphere

Usergivesreward?

location <- Chelsea

Usergivesreward?

location <- Chelsea

Howdoestheuserknowtherightsearchhappened?

Wizard-of-Oz

Kelley(early1980s),FordandSmith(1982)

‣ LearningfromdemonstraAons:“wizard”pullstheleversandmakesthedialoguesystemupdateitsstateandtakeacAons

FullDialogueTask

location <- Chelsea

curr_result <- execute_search(){wizardenters

FullDialogueTask

location <- Chelsea

SushiSekiChelseaisasushirestaurantinChelseawith4.4starsonGoogle{wizardtypesthis

outorinvokes templates

FullDialogueTask

location <- Chelsea

SushiSekiChelseaisasushirestaurantinChelseawith4.4starsonGoogle{wizardtypesthis

outorinvokes templates

‣Wizardcanbeatrainedexpertandknowexactlywhatthedialoguesystemsissupposedtodo

LearningfromStaAcTraces

Bordesetal.(2017)

‣ Usingeitherwizard-of-OzorotherannotaAons,cancollectstaActracesandtrainfromthese

FullDialogueTask

location <- Chelsea

FullDialogueTask

location <- Chelsea

curr_result <- execute_search()stars <- 4+

FullDialogueTask

location <- Chelsea

‣ Useraskedfora“good”restaurant—doesthatmeanweshouldfilterbystarraAng?Whatdoes“good”mean?

stars <- 4+

FullDialogueTask

location <- Chelsea

‣ Useraskedfora“good”restaurant—doesthatmeanweshouldfilterbystarraAng?Whatdoes“good”mean?

‣ HardtochangesystembehavioriftrainingfromstaActraces,especiallyifsystemcapabiliAesordesiredbehaviorchange

stars <- 4+

Goal-orientedDialogue

‣ BigCompanies:AppleSiri(VocalIQ),GoogleAllo,AmazonAlexa,MicrosofCortana,FacebookM,SamsungBixby,TencentWeChat

‣ Startups:

‣ Lotsofcoolworkthat’snotpublicyet

‣ Tonsofindustryinterest!

OtherDialogueApplicaAons

Search/QAasDialogue

‣ “HasChrisPra4wonanOscar?”/“HashewonanOscar”

QAasDialogue‣ DialogueisaverynaturalwaytofindinformaAonfromasearchengineoraQAsystem

Iyyeretal.(2017)

‣ QAishardenoughonitsown

‣ Challenges:

Iyyeretal.(2017)

‣ QAishardenoughonitsown

‣ Usersmovethegoalposts

‣ Challenges:

QAasDialogue‣ UWQuACdataset:QuesAonAnsweringinContext

Choietal.(2018)

SearchasDialogue

‣ Googlecandealwithmisspellings,somoremisspellingshappen—Googlehastodomore!

DialogueMissionCreep

System

Erroranalysis

Be4ermodel

MostNLPtasks

System

Erroranalysis

Be4ermodel

‣ FixeddistribuAon(e.g.,naturallanguagesentences),errorrate->0

MostNLPtasks

System

Erroranalysis

Be4ermodel

MostNLPtasks

System

Erroranalysis

Be4ermodel

Dialogue/Search/QA

System

Erroranalysis

Be4ermodel

DataHarderData

MostNLPtasks

System

Erroranalysis

Be4ermodel

Dialogue/Search/QA

System

Erroranalysis

Be4ermodel

‣ Errorrate->???;“missioncreep”fromHCIelement

HarderData

MostNLPtasks

System

Erroranalysis

Be4ermodel

Dialogue/Search/QA

‣ Highvisibility—yourproducthastoworkreallywell!

Takeaways

‣ Somedecentchatbots,applicaAons:predicAvetextinput,…

‣ Task-orienteddialoguesystemsaregrowinginscopeandcomplexity

‣Moreandmoreproblemsarebeingformulatedasdialogue—interesAngapplicaAonsbutchallengingtogetworkingwell

Lecture 16: Dialoguearitter.github.io/courses/5525_slides_v2/lec16-dialogue.pdfFull Dialogue Task...

Documents

Transcript of Lecture 16: Dialoguearitter.github.io/courses/5525_slides_v2/lec16-dialogue.pdfFull Dialogue Task...

Lec16 Modules

October 2008csa3180: Setence Parsing Algorithms 1 1 CSA350: NLP Algorithms Sentence Parsing I The Parsing Problem Parsing as Search Top Down/Bottom Up.

Lec16 Realtime PCR

Lecture 16 Interval, Scales, Tuning and Temperament - Ifaculty.tamuc.edu/cbertulani/music/lectures/Lec16/Lec16.pdf · 3 SCALE – A sucession of notes in ascending order (e.g., Pythagorean,

Lec16 Synthesis

Transport Phenomena LEC16 21b

Parsing Constituency: Nodes in a Parse Treehollingk/classes/lec16.CL1-f11.pdf · 2 Top-Down Search • Observation: trees must be rooted with an S node • Parsing strategy: • Start

Top-Down Parsing - recursive descent - predictive parsing

Parsing III (Top-down parsing: recursive descent & LL(1) )

Lec16: Medical Image Registration (Advanced): Deformable Registration

Lec16 memoryhier

Lec16, 3D Technologies, v1.01.ppt - Sharifce.sharif.edu/courses/92-93/2/ce342-1/resources/root/Lectures/Lec16... · 3D Technologies Course Presentation Mahdi Amiri May 2014 Sharif

LEC16 13 Politics

Statistical Dependency Parsing - Uppsala Universitynivre/docs/StatState.pdf · I Methods for statistical dependency parsing I Chart parsing techniques I Parsing as constraint satisfaction

Lec16 Sept10 Maxwell Eqns

Learning for semantic parsing using statistical syntactic parsing techniques

Chart Parsing and Probabilistic Parsing - SourceForgenltk.sourceforge.net/doc/en/advanced-parsing.pdfChart Parsing and Probabilistic Parsing 9.1 Introduction ... Furthermore, it is

Top Down Parsing, Predictive Parsing

Parsing III (Top-down parsing: recursive descent & LL(1) )

– 1 – CSCE 531 Spring 2006 Lecture 8 Bottom Up Parsing Topics Overview Bottom-Up Parsing Handles Shift-reduce parsing Operator precedence parsing Readings: