Emotive Driver Advisory System (EDAS) - ICINCO...• Emotive Driver Advisory System – Emotive...
Transcript of Emotive Driver Advisory System (EDAS) - ICINCO...• Emotive Driver Advisory System – Emotive...
Emotive Driver Advisory System
(EDAS)
Oleg GusikhinFord Research & Advanced Engineering
Dearborn, Michigan, USA
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
Ford Motor Company
• Global automotive industry leader based in Dearborn, MI.
• Manufactures and distributes automobiles in 200 markets across automobiles in 200 markets across six continents.
Ford Brands 2010
Ford Technology
Ford SYNC MyFord Touch
Ford Research & Advanced EngineeringDearborn, USA
Aachen, Germany
� Information Services & Connectivity
Manufacturing, Vehicles Design & Safety Research Lab
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
The Challenge
Emotive Driver Advisory System (EDAS)
Aibo
Emotions in Engineering
• Affective (Kansei) Engineering– a method for translating feelings and
impressions into product parameters
• Affective Computing– design of systems and devices that can
recognize and/or simulate human emotions
Emotive Driver Advisory System
• Natural Interface• Personalization• Customization• Context-Awareness• Context-Awareness• Adaptable Interaction and Intelligence• Seamless Integration of Vehicle
Infotainment and Cloud-based Services
Ford SYNC
(CAN & USB) (Bluetooth) www.syncmyride.com
CAN BUS
SYNC Services
• Standard Services
�911 Assist
�Vehicle Health Report
• Subscription Services• Subscription Services
�Traffic
�Directions
�Business Search, News, Sports, Weather
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
EDAS Overview
Speech Recognition
Semantic Interpretation
Dialog Manager
Spoken Dialog System
Input Manager
Task ManagerApplication
3d party Web
services
Home/
Display NAV Audio
Buttons
Voice
Traffic/Weather AlertsRefueling AdviceSpeed Limit Changes Maintenance ConcernsPersonalized InfotainmentLocation-Based Notes Social Networking
Emotive Response
Emotive Response
Vehicle CAN BUS
Data ManagerDriver State & Driving Conditions Analyzer
Vehicle
Portal
User/ Vehicle Profile
Application Services Office
Server Speech Portal
Vehicle Apps Safety Performance Convenience
Video
Customization Preferences Learning Collaborative Filtering
Driving Conditions Driver State
Emotive Text-to-Speech
Response Generator
Response
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System– Emotive Spoken Dialogue System– Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
Spoken Dialogue System (SDS)• Conversational • Communication does not need to be
hierarchical / menu driven• Supports Open Mic / Push-to-Talk• Supports Open Mic / Push-to-Talk• Multimodal (speech and/or button based
interaction)• Context aware & emotive dialogue
Emotion Recognition of Driver Ways to detect emotions:
– Facial expressions: captured by camera– Voice: input through microphone
Emotion Vector
– Biometrics: steering wheel, heart monitor– Vehicle operating pattern: aggressiveness,
workload
Emotive Conversational DialogKey Word Dialogue Interface Conversational Dialogue Interface
Speech Recognition:
converts recognized key
word to text
keyword Natural phrase
Speech Recognition: acoustic +
language models
words
keyword
Dialogue Manager
Input manager: Semantic
interpretation
Dialogue Manager
Output Manager
Output Manager
emotions
Emotion-based dialogue strategy
If Mood = good &
Driver sends request: Monitor Fuel Economy
Using Driver’s Emotions to Modify Responses
If Mood = bad &Performance = bad
If Mood = good &Performance = bad
Check Driver’s mood
Your hard acceleration is negatively affecting
your gas mileage.
The way you’re driving right now is hurting your fuel economy.
Please ease up on the pedal.
Dialogue Interface Architecturedriver speaks into microphone
SpeechRecognition
[Nuance speech recognition engine
InputManager
[Semantic
DialogueManager
[Determines system responses, e.g. send
Words Interpretation
Spoken Dialogue System (SDS)
Speech Signal Emotion
recognition software
Driver’s emotional state
EDAS systemqueries the web
sends key words in text file Query
ResultStorage
CereProc text to speech
recognition engine +Language Model built using Regulus]
interpretation, built using Regulus]
responses, e.g. send keywords to EDAS or navigate existing query result]
Output Manager
Actions tobe performed
sends xml file in special format
sends emotional tag
Output message
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
Motivation for Avatar• Universal Multivariate Gage (Chernoff
Face)• Reference Point of Vehicle Intelligence • Agent of Customization • Agent of Customization • Emotive Supplement to Text-to-Speech• More natural interface
EDAS Prototype 2005
Avatars
Customizable Avatars, Voices, Names, & Background Scene
Jackie
Toby Shouting
Output Responses from Avatar
Driver Emotion Task
Status
Request
Emotions Status
Facial expressions
Tone of Voice
Emotive TTS
Emotion Recognition & Emotive Feedback
Playful
Emotion Recognition
Hey Jackie
SeriousJacki
FormalHello Jacki
e
SeriousJackie
Example clinic results
Example clinic results
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
Benefits of Integration of Vehicle Infotainment and Cloud-based Services• Enable on-demand, personalized infotainment services
• Access to real-time information
• Enable continuum of personalized information and entertainment services between on-vehicle and off-vehicle activities
• Utilize power of social network collaborative filtering in recommendation services
• Outsource Computing Power to Remote Server
• Simplify maintenance and upgrade of infotainment applications
Examples of EDAS Services
• POIs• News• Music• Refueling advice• Refueling advice
POIs & News
• POIs– Facilitates Social Network Collaborative Filtering –
based Recommendations– Allow consolidation of relevant information scattered
throughout the internet for enhanced decision analysis and supportanalysis and support
• News– Personalized Interactive News Radio– Support Continuum of Personalized Information
Services between in-vehicle and off-vehicle environment
POI & NEWS Services DEMO
Personalized Music Radio
Private Collection Broadcast
Existing In-Vehicle Music Services
Personalized Music Radio
Internet Personalized Music Services
• Creates custom stations based on artists or individual songs
• Learns preferences by analyzing the musical qualities of songs listener like and dislike
Pandora Music Radio
of songs listener like and dislike
• Plays Music that matches personal preferences using AI and song genome database for song selection
EDAS Pandora Player
Intelligent Refueling Gauge
• Given a set of routes over a time period – When to buy gas– Where to buy gas– How much gas to buy
I recommend that you stop today at the
Sunoco station because gas prices
will be going up tomorrow.
System Architecture
Current Gasoline
Prices
OPIS (US)
Experian Catalist (UK)
Historical Gasoline
Prices
Wholesale Gasoline
Prices
Optimization
GPS Speed Fuel Consumption Fuel Level
Vehicle Profile Predicted Routes & Fuel Consumption
Price ForecastGasoline Station Markup ($/gallon)
(Oaks Petro Mart, 9 mile road)
-0.5
0
0.5
1
9/9
/200
7
9/1
6/2
007
9/2
3/2
007
9/3
0/2
007
10
/7/2
007
10/1
4/2
007
10/2
1/2
007
10/2
8/2
007
Date
Mar
ku
p ($
/gal
lon
)
Markup ($/gallon)
M
Sa
Home/ Office
Outline
• Introduction • Background and Ford SYNC• Emotive Driver Advisory System
– Emotive Spoken Dialogue System – Emotive Spoken Dialogue System – Avatar as an Advanced Automotive HMI– Cloud-based Infotainment
• Marketing & Business Impact
EDAS at CES & NAIAS 2009
EDAS Media Coverage
SYNC Pandora
Music Genome Project Database
Pandora Server
User Profile
SYNC API
AcknowledgmentsY. Chen Vehicle DesignD. Filev Active SafetyS. Di Cairano P/T ControlsT.J. Giuli Vehicle DesignE. Klampfl Systems AnalyticsI. Kolmanovsky P/T ControlsP. MacNeille Systems AnalyticsP. MacNeille Systems AnalyticsJ. Michelini P/T ControlsK. Prakah-Asante Active SafetyN. Rychtyckyj ITF. Syed V&B ControlS. Szwabowski P/T ControlsC. Teslak P/T ControlsK. Theisen Systems AnalyticsB. Tonshal Vehicle Design