A Cognitive Navigation Assistant for the Blind fileThe technologies developed in our proposal will...
Transcript of A Cognitive Navigation Assistant for the Blind fileThe technologies developed in our proposal will...
A Cognitive Navigation Assistant for the BlindKris Kitani Manuela Veloso Chieko Asakawa Dragan Ahmetovic Hernisa Kacorri Hironobu Takagi
Motivation
� For individuals with visual impairments, navigation in a new environment is challenging as much of the visual information is not accessible
� Our goal is to provide guidance to visually impaired users in unexplored environments using machine perception and co-robot interaction
� The technologies developed in our proposal will be leveraged to help visually impaired users to build a cognitive map of the environment
� A cognitive map is a mental model of the environment, essential for independent mobility, which depends on the understanding of:• Geometrical structure of the environment (e.g. pathways, door and stairs)
• Functional attributes of the surroundings (e.g. place to eat, rest or socialize)
Cyber-physical Cognitive Assistant
� We develop a cognitive assistant which is able to augment human cognitive capabilities, in particular for people with visual impairments
� The cognitive assistant enables contextual hyper-awareness and guidance through a portfolio of sensing modalities and algorithms
� We focus on a cyber-physical instantiation of a navigation assistant for visually impaired users, through two concrete manifestations:
Compact Wearable Interface
� Smartphone interface, relies on existing consumer devices
� Audio feedback facilitates the creation of a cognitive map
� Highly accessible through audio-based interaction paradigms
Embodied Robotic Interface
� Hardware platform that implements a co-robot motion model
� Lessens the cognitive load of the user through haptic feedback
� Aware of situational context and user’s preferential dynamics
Portfolio of Algorithms for Context Awareness
� We propose a portfolio of algorithms, accessed by both wearable and robotic interfaces to make sense of the user’s surroundings
� Our goal is to provide to the user the information about the knowledge landscape of the environment. In particular in terms of:
Location of the user in the environment
� Accurate localization that relies on:
� Computer vision-based algorithms
� Bluetooth low energy (BLE) beacons
Landmarks of the environment
� Collect first-person navigation videos
� Identify important visual landmarks
� Use identified landmarks for guidance
Frames
Mot
ion
ampl
itude
(ver
tical
)
When seeing landmarksMotion vectors
t=550 t=900
t=1100 t=2000
SD=1/30sec
SD=1sec
SD=3sec
Functional attributes of the environment
� Collect video examples of activities
� Map locations to possible activities
� Predict environment “Action Maps”
Goals and Broader Impact
� We aim to improve the quality of life for people with visual impairments through sensory and cognitive assistance
� In the long term, we envision an approach towards human augmentation through machine intelligence
� This work will shed light on the automatical acquisition of environmental knowledge using machine perception
� We investigate how that information can be conveyed through an embodied co-robot or a wearable assistant