Augmented Reality Tactile Map with Hand Gesture Recognition · – Collaborating with a printing...
Transcript of Augmented Reality Tactile Map with Hand Gesture Recognition · – Collaborating with a printing...
Augmented Reality Tactile Map with Hand Gesture Recognition
Ryosuke Ichikari1, Tenshi Yanagimachi2 and Takeshi Kurata1
1: National Institute of Advanced Industrial Science and Technology (AIST), Japan
2: Toyohashi University of Technology, Japan
1
13 July 2016, ICCHP 2016@ University of Linz, Austria
Walking Assists for the visually-impaired people
Assisting technologies when they are walking • White canes & guide dogs • Assisting technologies with computer
– Remote Sensing by RGB(-D) cameras or LRF – Voice navigation system based on localization
technology such as GNSS Pre-Journey learning and re-viewing before/after walking • Tactile Map
2 Tactile trajectory
Tactile Map
3 UV Offset printing PIAF
• Maps with tactile presentation for the visually-impaired people
• UV offset printing:Printing transparent plastic on the color maps
• PIAF(Picture in a flash):Using heat sensitive papers which can swell black-printed area (lines/images)
• Casio’s 2.5D printer:All-in-on solution for color and detailed tactile expression.
Casio’s 2.5D printer
Problems about tactile map
• Difficulties for constructing mental image of the contents with haptic sense – Needs experience and training
• Limitation of the amount of contents • Production issues
– Cost: UV offset printing is expensive – Needs know-how to design contents ⇒Tactile Map Automated Creation System (TMACS) [Watanabe et. al, ICCHP2014 ]
4
Proposed system:Utilizing AR for tactile map
• Augmented Reality; AR – Seamless integration of real
world and virtual world – Visual augmentation is main
stream, but AR concept can be applied for audio sense
• AR Tactile Map(Proposed System): – Provides audio-visual feedbacks in response to
interactive operations – Enables dynamic and multimodal contents – Can be utilized for learning tactile map
5
Audio-visual AR [Higa et. al 2007]
Characteristics of AR tactile map • Augmentation for the tactile map by audio representation
(for completely blind people) • Enlarging/enhancing display for people with week sight • Providing POR(Point of reference) information • Intuitive interaction by actually touching tactile maps • Configuration with off-the-shelf and varsatile products • Hands-free arrangement
6 Showing enlarged map with voice
Providing POR(Point of reference) information
• Target contents of the AR tactile map: POI(Point of Interest)and POR(Point of Reference)
• POR:Specific point location where it is easily recognized for confirming routes such as characteristic shape and material of ground (steps, stairs, sloop, door), sound/noise, and scent/odor) [Okuno et. al, CSUN2015] [Denoncin,CSUN2014] [Theberge, CSUN2015]
• Integration with POR/POI data base – Voice navigation system with POR/POI data base [Okuno et. al,
CSUN2015]
7
System configuration System configuration of the prototype system • Tablet PC ⇒ For Visual-Audio feedbacks • RGB-D camera:DepthSense DS325 (ToF, Range:15㎝
~100㎝) ⇒For finger gesture recognition and robust extraction of hands area, measurement of the distance between the map and finger
• PC for recognition • Arm for fixing tablet • Tablet mount parts
8
Related Works • ATMaps (Audio-Tactile Maps) using
IVEO’s Interactive tactile graphics. (ICCHP2016 workshop)
• Image to Sound Converter [Kuroda] – Image is converted to sound by changing key
and difference of L/R sound – Transparent tactile sheet can be overlaid – Display size and UI are limited by touch display
• Kiko-AR by cybernet systems – Mobile AR system based on character recognition and audio
presentation – Collaborating with a printing company to produce printed material
with tactile presentation and AR augmentation. – Need to shoot with smartphone’s camera
9
Characteristics of AR Tactile Map: - Using versatile products - Various sizes and types of tactile maps can be used - Intuitive interface for the system are focused
Recognition of gesture • Recognizing “Tapping”, “Double
Tapping”, “Flat-hand gesture” gestures as interaction commands
• Algorithm for detecting “Tapping” – Extracting hand area by RGB-D camera with background differencing
of depth image and distance threshold – Detecting state transition of distances within time window (①->②->①)
10
①Finger on the map ②Finger in the air for tapping time
Distance between map and finger ②
①
Size of time window
• Flat-hand gesture: Tracking flat-hand position when user touches maps by gradients and distances
Automatic identification and tracking of tactile map
11
• ORB Feature point detector/local feature descriptor is used for identifying tactile map with RGB image
• Estimating homography matrix between rectified image templates and input image
Mapping party
fo
Previewing by printing as tactile map
Walking assistance by voice navigation
Gathering acctual sites for making map (Mapping Party)
Virtual Mapping Party with AR Tactile Map
Omni-directional image and 3D-Sound Recording
Pre-learning of the walking route with the AR Tactile Map
Utilized at voice navigation
Mapping Party Anytime, Anywhere
Requests for Mapping
Workshop on Virtual Mapping Party • Held at Miraikan, Tokyo, Japan • March 18th-19th, 2016. 1h x 6 times (total 42 participants) • Collecting POR/POI by experiencing omnidirectional
images(360 deg.) and 3D sounds with HMD(Oculus, GearVR. Etc.), PC and headphone
• Confirming and requesting POR/POI by AR tactile Map • The number of the registered POR/POI: 598
Usage of the AR tactile map for virtual mapping party • Allowing the visually-impaired people to join
the mapping party by gesture – Flat-hand: Confirming the existence of POR/POI where
user touches – Tap: Confirming POR/POI
with Text-to-Sound when user taps the specific point
– Double Tap: Requesting the POR/POI for the specific points of the map
Summery of a questionnaire for the AR tactile map • Accuracy of the gesture recognition has a room for
improvement • Requesting function is good because the it can reflect
visually-impaired people's demand. • Scale size of the map was not fitted. Considering to
accuracy of the gesture recognition, the scale size of tactile maps and the density of the distribution of the contents need to be carefully considered.
• It seems to be interesting if the voice recognition can be combined with gesture interface.
• A participant with visual impairment like annotations printed in Braille, but another participant with visual impairment does not like them because they disturbed perception of the shapes of roads and buildings.
17
On going works • Implementing another configuration of the AR
tactile map with HP’s Sprout. • Sprout is a PC which has
– A projector – A touch pad – depth and color cameras
• Sprout enables – touch recognition through
the map – Projection onto map – Sheet tracking – 3D scanning
18
AR tactile map with Sprout
• Tapping: Checking registered POR/POI
• Dragging: Searching • Double Tapping:
Requesting POR/POI • Projecting POR/POI
points on the map
No other device or PC is required
19
The proposed system AR tactile map with Sprout
Gesture recognitions
hand gesture recognition by RGB-D camera
touch detection by touch pad
Information augmentation
audio + enlarging image audio + enlarging Image + projection onto map
System configuration & cost
PC($500~) + RGB-D camera($200) → can be replaced with a tablet PC with RGB-D camera
Sprout($1500)
Limitation: no visual augmentation onto map
Size of map and UI for the map
Advantages mobility, flexibility of UI and size of map, moderate price
visual augmentation, accurate gesture recognition
Comparison with the proposed system
Conclusions • AR tactile map: Audio-visual augmentation for tactile map
by utilizing hand gesture interface with RGB-D camera. • Virtual Mapping Party with the AR tactile Map • AR tactile map by HP’s Sprout • Future works
– Increasing accuracy of the gesture recognition – Covering individual difference of gesture – Noise and unclear situation (sunlight, close to paper) – Other configurations of AR tactile maps (Sprout etc.)
• Acknowledgement – This research is conducted as a part of the project sponsored by
JST RISTEX.
20
Thank you!
• Ryosuke Ichikari, Ph.D. – Postdoctoral Researcher at AIST – Contact: [email protected]
21