Post on 28-Jul-2018
A navigation system for Blind PeopleWhen the smartphone becomes a Guide DOG
Laura GiarréThe Blind Leading the Blind, P. Brugeul the elder. 1568, Napoli Capoimonte musuem
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 1
Problem
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
Visually impaired are 300.000.000 (source WHO)• And increasing with aging of the population• Social inclusion enhanced with autonomous
mobility• Today: tactile pavements (very expensive!)
2
Can technology help a Visually Impaired Person• to live a more independent life, working and
socializing• to move autonomously and safely in an
unknown environment• to retrieve information about the places
nearby• to recognize the persons they are interacting
with and understand their feelings• to perform day life activities with increased
autonomy
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
Challenges for VIP
3
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
S
Machine LearningFiltering
Signal DetectionFeature ExtractionComputer vision
Haptic InterfaceVibrationBraille PadMusicAudio signalWearable vibratingbracelet/belts
Camera Sonar/laserAcceleremoterRFIDIMUiBeacon
CYBER SIGHT SENSE
Sensors Actuators
Computation
Solution
4
Stairway to Heaven
√ How Much Information√ Given Task√ Ergonomic/Functionality√ Cosmetic Acceptability√ Tests in Real World √ Embed Blind Experience
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 5
• Small in size• Hardware robust but light• Optimized Battery life• Appeal of exterior design• Cosmetic acceptability is more importantthan technological potential benefits• Functionality: TRANSMIT INFORMATION to theremaining sense without INTERFERING
Ergonomic
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 6
Laboratory vs real world• Early success with simplified tasks and conditions has bred
optimism about eventual success in performing complex tasks in real-world conditions:
• Walking through environment cluttered with small low contrastobstacles
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 7
• Use of other SPARED sensory modality to supply information normally gathered by the impairedsense• Aids employed regardless of the different clinicalfeatures of blindness•Basic Examples:
•Braille reading: fluently read through raised patterns
• Long cane: uses tactile information to explore the surroundings
Non invasive Solutions
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 8
• Miniaturization of actuators and electronics :new systems embedded into clothing,
devices worn on the finger, hands, wrist, forearm, tongue, head, chest, abdomen and feet
• Hearing and touch is the first and second major senses, and it gather enough information from the environment to perform daily tasks.
• Assistive devices provide acoustical and tactile feedback to compensate for visual information.
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
Wearable Solutions
9
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 10
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
• What is the easiest way to let blind people use Sensory Substitution Devices?– minimize the external infrastructure– use their own smartphone– develop cheap (computationally) algorithms
running on smartphones– make use of all the sensors already
embedded in the phones
A case study:
11
State of the art on indoor navigation
Laura Giarré University of Modena and Reggio Emilia
March 1st, 2018
• Indoor navigation:– Triangulation of RF signals (WiFi)– Direct sensing (RFIDs, ultrasound, Bluetooth, iBeacon,...)– Pattern matching– Dead‐reckoning (accelerometers, gyroscopes, …)
• Often dedicated hardware is required– Expensive devices, complex deployment
• Low accuracy (sometimes <5m)– Insufficient for navigation of the visually impaired
• Audio feedback is annoying!!!12
: what is not?
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
A system to avoid obstaclesCameras, ultrasounds, sonar, haptic belts, interfaces
A system to give back sightto directly stimulate the deficient sensorysystem: Invasive solutions to send to theoptical nerve acquired signals from avideocamera
13
: what is?
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
Arianna is a system for low vision and/or blind person,A smartphone-based solution for assisted navigation:• goal: improving social inclusion, accompanying the
users along the path• From mute paths to talking paths
– ICT for improving user interactions (robotics&vision)– Haptic interfaces
14
A smartphone-based solution for assisted navigation:• A set of paths and tags deployed in the environment • A smartphone (instrument of mediation between the
reality and the user to access the information • A vibration feedback signal to the users to follow pre-
defined path
system (autonomous mobility)
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 15
March 1st, 2018
Tape and QRcode laying
User arianna app installation
Map creator and server configuration o app configuration
System usage
Installation step:1) Tape laying
2) Map creator
3) Map selection
4) System usage
What is needed
Laura Giarré University of Modena and Reggio Emilia 16
1. The smartphone camera captures the image of the scene in front of the user;2. A computer vision algorithm identifies the path by recognizing two colored
lines on the floor;3. The user receives a vibration signal through the telephone to understand the
direction to follow;4. Some special references (tags) like QR code or ibeacon allow to identify
contents associated to the position
What the camera sees
Path identified by Arianna
MORE details..
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 17
Inside
Laura Giarré University of Modena and Reggio EmiliaMarch 1st, 2018
The algorithms
18
1. How to detect the path?– There are various functions in computer vision
that can be combined for our problem, taking into account some fundamental aspects:
– the identification must be fast, to allow the user who moves the phone in his hand not to perceive discontinuous signals;
– the battery life must be maximized – What can we use?
• the geometry of the strip (actually three parallel lines) • the colors of the strip
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 19
Classic Approach• Basic idea:
– Detect the Contours (Canny algorithm)– Find contour points along a line (Hough transformation)
Contour foundwith Canny
Lines for eachcontour point
rho
theta
Identification of aligned points(quantized rho, theta)
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 20
Example: Optimization to select only the path
Pre‐filteringto elimatedetails
Tresholds to define the maincontours
Lines with the majority of ‘scores’
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 21
A colour based approach• It is based on the representaion HSV (hue saturation
value) (more robust to ambient light changing with respect to RGB)
• Simple range filtering intervals– Es. filtering all the points in the intervals (H:0.4-0.6, S:0.23-0.53,
V:0.4-0.6)• Double filtering to increase robustness• No complex matrix operations: only “if”s• Low robustenss to light condition• Indoor and Outdoor
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 22
A mixed approach• At regular intervals or when variation in lighting is
detected..– Geometry analisys to identify path and sample/resample the
colours
• Regularly– Coulour based identification
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 23
2. If the path is lost?• Basic Idea:
– Indication if it has been lost from left or right (computer vision)
– Estimation of the relative movement of the user with respect to the path (inertial samrtphone sensors: (dead-reckoning)
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 24
Inertial sensors measurements• Gyroscope angular direction (at regular intervals T)
– alpha_k• Accelerometer filtered velocity (assumption: the user is
moving in 2D)– from [a_x, a_y, a_z] and [alpha, beta, gamma] to |v_k|
• Activity estimation (walking/standing) to stop the acquisition when the user is.
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 25
Sensor fusion: integration of Computer vision and dead‐reckoning
• When the path is visible again:– Increase the quality of the measurement of alpha (from the angle
theta detected with hough)– Reste the cumulative error (from the estimated rho with hough)
• To use the camera data it is important to know the angualr reference in the
• Conversion between the absoulte and relative orientation
Beta
Alpha=f(Beta,Theta)
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 26
Example of ‘fusion’Real Experiment
only dead‐reckoning dead‐reckoning + CVMarch 1st, 2018 Laura Giarré University of Modena and
Reggio Emilia 27
3. How to improve the system by modeling user reaction (HIL)?
• Assumptions: – User is walking at constant speed (noisy)– Corretions:
1) Proportional to the deviatuon from the path and to the direction to go back onto the path in a time T_X, whenthe path is visible to the camera2) Fixed correction (the sign depends if we lose the pathfrom right or left)
– The correction of the user is characterized by a proportional term (m, n)
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 28
User position estimation with EKF(Estimated Kalman Filter)
• Inertial sensors, camera, correction model u_k
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 29
User Simulator
Perception direction noise uniformely distributed in
[-π/6, π/6]
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 30
User simulator (wrong initialdirection)
Heading correction m=0.6 Distance correction n=0.4
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 31
Inside
Laura Giarré University of Modena and Reggio EmiliaMarch 1st, 2018
The System
32
The APP ( Android and iOS)
Drawer menuFollow the line
Places
Help Settings
Navigation
• Changeable colours (for low vision people)• Not only path recognition but also navigation from a to b.• Reproduction of contents (POI)• Voice-over (upon request)
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 33
Laura Giarré University of Modena and Reggio Emilia
I-beacon integrationThe point of interests (POI) or TAGS can be QR code (CV) or I-beacon (radio) • Ibeacon vs Qrcode:
• More precise recognition in a wider aerea• Not on the path (Hidden for cosmetic reasons)• More costly and they need battery
March 1st, 2018 34
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
Where to install ARIANNA
Airports
Offices/Mall
Sidewalks
Museums
35
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia
How to evaluate the system?
• Run specific tests: decide a task (a pathpresenting curves, crossing points, beginningand end, Qr codes)
• Define some metrics: time, number of tagsmissed, etc (quantitative)
• Run follow up with questionnaires (qualitiative)• Importance of feedback from BlindCommunities!!!
36
Tests with Blind Communities• Istituto dei ciechi Milano, Bologna• Unione Ciechi Florence, Palermo• Demos & exhibitions:
– Project FIC3 App Launch Day, 2016– Taiwan Computex, 2016– EXPO Milan, 2015– National prize for Innovation, 2014– Smart City Exhibition, 2014– SZDIF14 China international industrial design fair, 2014– ABF MIT Challenge workshop, 2013 & 2015
• Collected subjective and objective feedback with questionnaires
Laura Giarré University of Modena and Reggio EmiliaMarch 1st, 2018 37
FIWARE accelerator
Recent installations (2017)
FARM CULTURAL PARKFavara, Agrigento
(Italy)
Laura Giarré University of Modena and Reggio Emilia
Blindinstitute of Palermo
March 1st, 2018 38
The team: University of Palermo, University of Modena and Reggio Emilia, In.sight, National Chiao Tung University,
University of Roma Tre
March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 39
Web: in.sight.srlSocial: @insightsrl
References• Tzu-Kuan Chuang*, Ni-Ching Lin, Jih Shi Chen, Chen-Hao Hung, Yi-Wei
Huang, Chunchih Teng, Haikun Huang, Lap-Fai Yu, Laura Giarré, Hsueh-Cheng Wang Deep Trail Following Robotic Guide Dog in PedestrianEnvironments for People Who Are Blind and Visually Impaired - Learningfrom Virtual and Real Worlds 2018 IEEE International Conference onRobotics and Automation
• Giovanni Galioto, Ilenia Tinnirello, Daniele Croce, Federica Pascucci, Federica Inderst, Laura Giarré Sensor Fusion Localization and Navigation for Visually Impaired People 2018 European Control Conference
• Croce D., Giarré L., La Rosa F. G., Montana E., Tinnirello I.“Enhancingperformance in a smartphone-based navigation system for visually impairedpeople”. In Proceedings of the 24nd Mediterranean Conference on Controland Automation (MED’2016), 2016.
• Croce D., Gallo P., Garlisi D., Giarré L. Mangione S., Tinnirello I.“ARIANNA: a smartphone-based navigation system with human in theloop“, in Proceedings of the 22nd Mediterranean Conference on Controland Automation (MED’2014), 2014.
•March 1st, 2018 Laura Giarré University of Modena and Reggio Emilia 40