Jan Šedivý - Intelligent Personal Assistants
-
Upload
machine-learning-prague -
Category
Technology
-
view
186 -
download
0
Transcript of Jan Šedivý - Intelligent Personal Assistants
![Page 1: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/1.jpg)
Intelligent Personal Assistant Jan Šedivý
Petr Baudiš, Tomáš Gogár, Tomáš Tunys
April 2016
![Page 2: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/2.jpg)
Agenda
CommunicationIPA
SegmentsFunctionalityUse casesExpectationsInteractionModesContextPrivacyTechnologiesOur FocusRule based IPAYodaQAFuture
goo.gl/NiSF0I
![Page 3: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/3.jpg)
Intelligent Personal Assistants
Intelligent Personal Assistants on the market today
Apple Siri, Google Now, Microsoft Cortana and Amazon Echo applications/appliances are the best known Intelligent, Virtual or Personal Assistants (IPA).
In this presentation I will discuss the use cases, challenges and basic architecture of the future intelligent assistants.
![Page 4: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/4.jpg)
IPA Basic Definition
Predict users' needs, helps, alerts, answers questions and ultimately acts autonomously on our behalf.
To achieve it this goal: IPAs needs to communicate and be connected to the cloud.
How can IPA help us?
![Page 5: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/5.jpg)
IPA Segments
Mobile applications
Car interface
Wearables
Chat bots
Home automation
Robots, appliances
Where is IPA currently used?
![Page 6: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/6.jpg)
IPA Functionality
Alerting, reminding
Command control - mobile, car, wearables
IR front end, Search UI
Simple, factoid question answering
Simple chat bots, Avatars
Advising, Robo advisers
Suggesting
How can IPA help us?
![Page 7: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/7.jpg)
How to communicate with IPA?
People met shook hands and made a deal
Call centers, credit cards and parcel delivery
Internet commerce web sites
WeChat, FaceBook, Alexa call me uber
A short history of making business
![Page 8: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/8.jpg)
Relationships start with conversation …
Language
![Page 9: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/9.jpg)
IPA interaction channels
Input: Text, Speech, Haptic, Gestures, Image recognition,
Output: Text, Voice, Graphics, Haptic,
Mic, Camera, Touch screen, Wearables sensors, Brain waves
Multimodal: 5 senses, combines voice and GUI
Use not only language!
![Page 10: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/10.jpg)
IPA Interaction
ModesIPA initiated: Alerts, Suggests,
User initiated: Execute command, Answer question (voice), Carry a dialog
Directed dialog
Who starts the interaction?
![Page 11: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/11.jpg)
IPA Interaction
Modes
Open dialog
Mixed-initiative: The initiative is changing. IPA or user starts the dialog.
Disambiguation: IPA clarifies the question through a dialog
Conversational
Topic changing system
Who is leading the dialog?
![Page 12: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/12.jpg)
User Model - Context
Internal, embedded
Location, Time,
History - query, commands, situations, …
Future - calendar, email,...
User’s profile, preferences, usage modes, …
Affective computing - Emotional models
People know context!
![Page 13: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/13.jpg)
User Model - Context
External - environmentSocial, family, friendsconnected == IoTSensors, actuators, LANs
Private - with limited access
Recorded phone calls,Credit card transactions,Utilities ...
![Page 14: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/14.jpg)
IPA Privacy
IPA may know almost all
Supertrust relation
How much of private information do we want to share?
User need control
The information may only be shared with trust, (Norms of human relationships)
How much private data do we need to share to let IPA act
for us!
![Page 15: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/15.jpg)
IPA is one of the most complicated examples of the AI technologies
![Page 16: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/16.jpg)
IPA Technologies
Speech recognition, Speaker, language recognitionImage recognition,Haptics, gestures, face gesture, emotion recognitionEmotion recognition TTS, automatic speech generationGraph, picture, haptic generationUser modeling
NLP, NLU, Information retrieval, Knowledge management, Dialog managementInternet, APIsIoT etc. EvaluationAffective computing, Emotional
models
To meet the user's expectations we need to combine many AI technologies:
![Page 17: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/17.jpg)
IPA Architecture
Rule based If Sentence Pair Match is high=> intent do this
Statistical MLQuestion analysis,Knowledge base and Internet Answer Hypothesis
Answer scoring
Rule based or Statistical
![Page 18: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/18.jpg)
Rule Based IPA
Spoken input - ASR - Text - Entity extraction - Intent detector - Normalization - Execution
These systems assume questions with clear goal
If the question is beyond the system capabilities “I can’t answer this question”
Or it does the WEB search
![Page 19: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/19.jpg)
Intent Reco
Answer Sentence Selection
Next Utterance Ranking
Semantic Textual Similarity
Paraphrase Identification
Recognizing Textual Entailment
Basic models: TF-IDF, BM25, word, sentence embeddings
Sentence Pair Scoring
![Page 20: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/20.jpg)
The YodaQA System
● Universal end-to-end QA
● Searching databases and documents
● Open source research system
● Machine learning no manual rules!
● Java, Apache UIMA, Apache Solr
● Proof-of-concept web+mobile interface, public live demo
![Page 21: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/21.jpg)
Factoid Question Answering
Naturally phrased question instead of keywords
Output is not a whole document, but just the snippet of information
Voice interaction
![Page 22: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/22.jpg)
Factoid Question Answering
We cover the basic factoid questions!
When was J. R. R. Tolkien born?
What is the population of Brazil?
Who played Marge in The Simpsons?
Where was she born? (Julie Kavner)
How do I get to Wall Street?
Turn on the green light!
Tune BBC World News!
You are the last one, do you want me tu turn on the alarm?
![Page 23: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/23.jpg)
IPA Building Steps
Intet identification
Data collection,
Labeling,
Feature engineering,
Models building,
(Active learning,)
Model evaluation
![Page 24: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/24.jpg)
Future
Implement norms of human relationship: mutual value, respect, trust
What makes a better conversation?
How to carry an effective dialog, negotiation?
How to design an engine recognizing emotions?
How to learn habits?
How to make the IPA more human like?
![Page 25: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/25.jpg)
Future
How to make IPA adaptable to the user?
How to make IPA automatically configurable and integrate in a new environment?
How to make IPA enough flexible?
![Page 26: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/26.jpg)
IPA unified interface to mobile applications
Millions of mobile apps
Navigation, login-chaos, and unified bad notification
leveraging the context-of-consumption
leverage sensory and multimodal inputs
Gartner: By 2020, IPA will facilitate 40 percent of mobile interactions and it will begin to dominate the postapp era.
![Page 27: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/27.jpg)
Thank you
![Page 28: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/28.jpg)
TeamČVUT FEL - dept. Of Cybernetics
![Page 29: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/29.jpg)
Human behaviour
Content
People ask questions
What
Why
When
How
...
Mobility
People need help everywhere
Small real estate
Navigation, cross-app API, password chaos
UI has to change
iPhone introduced 2007
![Page 30: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/30.jpg)
IPA Segments
Automotive.
Utilities.
Banking.
Health.
Retail.
Real estate.
...
What are the industries benefiting from IPAs?
![Page 31: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/31.jpg)
IPA Developmen
t
Collect utterances
Define the answers
Label utterances
Build the model (ML)
Evaluate
Iterate to improve
![Page 32: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/32.jpg)
Users Expectations
Mustn't forcing to memorize commands.
It must understand natural language.
Helps solving everyday tasks.
Must be non obtrusive giving suggestions.
Answers questions.
![Page 33: Jan Šedivý - Intelligent Personal Assistants](https://reader036.fdocuments.in/reader036/viewer/2022062822/5885bb911a28ab6f168b5b33/html5/thumbnails/33.jpg)
IPA Use Cases
Complex questions,
Conversational, dialog
Complex robo advisers,
Presentation commerce,
Digital, enterprise, media asset management
Automatic generating documents, stats, news, tweets based on content on the web ….