Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots...
Transcript of Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots...
![Page 1: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/1.jpg)
Talking to Robots
Speech-Directed Motion Planning
Anil VenkateshSUNFEST 2008
![Page 2: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/2.jpg)
Project Goals
• Economy of spoken commands• Customization with little programming
![Page 3: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/3.jpg)
• Spoken Language Understanding Shell (SLUSH)
• Interactive language model• No training or learning
![Page 4: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/4.jpg)
What SLUSH does
• Most Likely Sequence (MLS) output every 10ms
[GO,den.office]STA/N_PP;[IDENT,e0]N/den;;/N_SIL? 5
_ 1 2 _
![Page 5: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/5.jpg)
Linear Temporal Logic
ϕ ϕ ϕ
ϕϕ1 ϕ2
True if is true in the next stateϕ
True if is true until becomes trueϕ1 ϕ2
ϕϕ
True if is true in some next stateϕ
True if is true in every next stateϕ
![Page 6: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/6.jpg)
• Linear Temporal Logic Motion Planner (LTLMop)
• Structured English => Controller
If you are sensing Sandwich then do Eat
![Page 7: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/7.jpg)
Linking SLUSH and LTLMop
• Grammar and Lexicon files• Pronunciation file• MLS processing
![Page 8: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/8.jpg)
Grammar and Lexicon
Go to the room.
G N -> (ROOM) roomG NP -> the NG Simp -> go to NP (GO)
L GO : (source-set)L ROOM : (set-of-all i in (source-set) s-t ((ilk of i) is (room)))
![Page 9: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/9.jpg)
If-Then grammar and lexicon
If you see the sign, go to the room.
G Sind -> you see NPG PPcond -> if Sind (IF)G Simp -> PPcond SimpG N -> (SIGN) sign
L IF : (context-set)L SIGN : (set-of-all i in (source-set) s-t ((ilk of i) is (sign)))
![Page 10: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/10.jpg)
MLS processing
• Throw away phonetic transitions• Cross-reference in pronunciation file for
words• Extract referents and locations
[GO,e0.e1]STA/N_PP*;[IDENT,e0]N/room;;/R_UW_M_SIL? 2
...[IF-SEE,e12]PPcond/PP_PP*;[INPRIME,e0]PP/sil;;/S_AA_IH_N_SIL? 3
![Page 11: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/11.jpg)
Linking to LTLMop
• Identify the kind of command• Choose appropriate template sentences
1. If you are in Location1 then do Location1Flag2. If you activated Location1Flag and you were not in Location2
then do Location1Flag3. {If you are activating Location1Flag then visit Location2}4. If you were in Location2 do not Location1Flag
If you are in Location1, go to Location2
![Page 12: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/12.jpg)
If you see the alarm in the office,catch the convict in the forest.
If you catch the convict in the forest, lock him up in the jail.
Environment starts with falseRobot starts with false
Visit yardVisit forestVisit officeVisit jailVisit bridge
Environment starts with falseRobot starts with false
Visit yardVisit forestVisit officeVisit jailAlways not bridge
![Page 13: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/13.jpg)
Results
• Expressivity
• Objectives met
IF THENSENSOR [in LOCATION_1] go to LOCATION_2in LOCATION_1 ACTION_2 [in LOCATION_2]ACTION_1 [in LOCATION_1]
![Page 14: Talking to Robots - University of Pennsylvania · 2018. 7. 17. · Talking to Robots Speech-Directed Motion Planning Anil Venkatesh SUNFEST 2008. Project Goals • Economy of spoken](https://reader035.fdocuments.in/reader035/viewer/2022081615/5fdfe946775cda72983c4ddb/html5/thumbnails/14.jpg)
Acknowledgments
My advisorsWilliam Schuler (University of Minnesota)The SUNFEST program
Images fromhttp://b-bcs.com/http://mohitdhawan.blogspot.com/2005_04_01_archive.htmlhttp://scary-manilow.livejournal.com/