Introduction to Talking Robots - ling.helsinki.figwilcock/Doshisha-2016/docs/5-SpeechDialogu… ·...

www.helsinki.fi/yliopisto

Introduction toTalking Robots

Graham Wilcock

Adjunct Professor, Docent Emeritus

University of Helsinki

2/12/2016Graham Wilcock / Introduction to Talking Robots 1


Robot Speech and Dialogues

Graham Wilcock



Spoken Dialogue Systems:Closed-Domain Dialogues

• Example: Flight reservations

• Using finite state transitions

• Easily implemented by finite state machine (FSM)

• Fixed-domain database

• Flights, cities, days

• Easy to add new flights, new cities, …

• Difficult to switch domains



FSM for a Closed Domain



Spoken Dialogue Systems:Annual Conferences

• SIGDIAL (since 2000)

• Special Interest Group on Discourse and Dialogue

• http://sigdial.org/sigdial-meetings

• IWSDS (since 2009)

• International Workshop on Spoken Dialogue Systems

• http://www.iwsds.org/


http://sigdial.org/sigdial-meetings

http://www.iwsds.org/


Spoken Dialogue Systems:Books

• K. Jokinen and M. McTear

Spoken Dialogue Systems

Morgan & Claypool, 2009

• K. Jokinen

Constructive Dialogue Modelling

Wiley, 2009



Robot Dialogue Systems:Books

• K. Jokinen and G. Wilcock

(editors)

Dialogues with Social Robots

Springer, 2017



WikiTalk: Talking Robots and Wikipedia



• Visits to MTA-SZTAKI, Budapest 2011

• Hungarian Academy of Sciences

• Advanced 3-D Virtual Reality with Nao robot

• “It would be nice if it could talk about Shakespeare”

• How could that be possible?

• By using Wikipedia (key idea)

• I made a prototype with Python in 2011

• Windows speech (Windows XP)

2/12/2016

History of WikiTalk (1):Hungary 2011

Graham Wilcock / Introduction to Talking Robots 10


New Direction:Open-Domain Dialogues

• Need open-domain knowledge source

• Get information from the web

• Wikipedia has articles on almost any topic

• Example: to talk about Shakespeare

• Get Wikipedia article about Shakespeare

• Start reading out the first paragraph



FSM for Open-Domain (draft)



Topic Shifts and Topic Chains

• Hyperlinks -> topic shifts -> topic chains

• Robot: ”Shakespeare was born and raised in Stratford-upon-Avon” (link)

• Human: ”Stratford-upon-Avon?” (topic shift)

• Robot: ”Stratford-upon-Avon is a market townand civil parish in Warwickshire, England”

• Human: ”Warwickshire?” (topic shift)

• Otherwise, continue same topic


http://en.wikipedia.org/wiki/Stratford-upon-Avon

http://en.wikipedia.org/wiki/Market_town

http://en.wikipedia.org/wiki/Civil_parishes_in_England

http://en.wikipedia.org/wiki/Warwickshire

http://en.wikipedia.org/wiki/England


FSM for Open-Domain (WikiTalk)



Speech Recognition in WikiTalk

• Open Domain but not Open Vocabulary

• Open vocabulary speech recognition basically does not work (about 300,000 words in English)

• Recognize phrases from small phrase sets

• Dynamically predicted from Wikipedia links

• Recognize small set of navigation commands

• ”Repeat”, ”Continue”, ”Enough”, ”Previous”, etc.



• International summer school project, Metz 2012

• Led by Prof. Kristiina Jokinen (Tartu U.) and me

• 5 PhD students, 4 Nao robots, 4 hot weeks in July

• Transferred Python WikiTalk code to Nao robots

• Added robot gesturing and multimodal interaction

• Demos in Metz and Budapest 2012

2/12/2016

History of WikiTalk (2):France 2012


www.helsinki.fi/yliopisto 2/12/2016

International Summer Workshop eNTERFACE 2012, Metz, France



Making WikiTalk on 4 Nao robots at eNTERFACE 2012



First Demo of WikiTalk on Nao (Metz, France 2012)


https://www.youtube.com/watch?v=A7XFcnJE29A

http://www.iltalehti.fi/iltvdigi/201411290140927_v4.shtml

www.helsinki.fi/yliopisto 2/12/2016 24Graham Wilcock / Introduction to Talking Robots

ATIS Spelling Alphabet(used with ”Alphabet” command)


• Finnish WikiTalk, Helsinki 2014

• Prof. Jokinen, PhD student Niklas Laxström and me

• 2 robots borrowed from SASKY, Sastamala

• Demo at Robottiviikko 2014 exhibition

• Japanese WikiTalk, Kyoto 2015

• Prof. Yamamoto, MSc student Okonogi and me

• 3 robots in Prof. Yamamoto’s lab, Doshisha University

• Demo at SIGDIAL 2015 conference

2/12/2016

History of WikiTalk (3):Multilingual versions



EU Robotics Week 2014at ”AiRo Island” Helsinki



Multilingual WikiTalk:Finnish and English

2/12/2016 27

https://www.youtube.com/watch?v=q0aSgPns4dE

Graham Wilcock / Introduction to Talking Robots

http://www.iltalehti.fi/iltvdigi/201411290140927_v4.shtml


Multilingual WikiTalk:English and Japanese

2/12/2016 28

https://www.youtube.com/watch?v=NkMkImATfYQ

Graham Wilcock / Introduction to Talking Robots


• Talking Robot + Wikipedia

• Instant talking on any topic (open-domain)

• Community-validated talking

• Extend robot communicative capabilities


Open-Domain Talking:WikiTalk


• Anticipated developments in ASR

• Continuous listening capability

• Better cloud-based speech recognition

• Open vocabulary speech recognition

• Wikipedia-based listening

• Open-domain named entity recognition

• Wikification: Wikipedia reference linking

• Extend robot communicative capabilities again


Open-Domain Listening:Towards WikiListen


• Example: North Sami

• Spoken by 20,000 people in Lapland

• DigiSami project

• To support revitalization of North Sami language usingmodern language technologies

• Prof Kristiina Jokinen (University of Helsinki)

• Funded by Academy of Finland


Speech Technology for Under-resourced Languages


The Sami Languages


DigiSami Corpus of North Sami Conversational Speech


Speech Synthesizer and Recognizer for North Sami

https://www.youtube.com/watch?v=14jYeViJ0X0


First Demo of SamiTalk(IWSDS 2016 in Lapland)

https://drive.google.com/file/d/0B-D1kVqPMlKdeXhXbnlUMmtYcU0

Introduction to Talking Robots - ling.helsinki.figwilcock/Doshisha-2016/docs/5-SpeechDialogu… ·...

Documents

Transcript of Introduction to Talking Robots - ling.helsinki.figwilcock/Doshisha-2016/docs/5-SpeechDialogu… ·...