HoloToolkit for Unity - Gaze & Voice
-
Upload
dmitri-bouianov -
Category
Technology
-
view
696 -
download
0
Transcript of HoloToolkit for Unity - Gaze & Voice
PowerPoint Presentation
HoloToolkit for UnityGaze & VoiceDmitri BouianovFounder @ Context VR
BackgroundComputer Science, Software Engineering
Unity + AR/VR/MR + full-stack development
Multiple startups, technical & product roles
Looking for experts: Unity, graphics, shaders
OverviewHoloToolkit is an open-source framework brought to you by Microsoft HoloLens team
Encapsulates and simplifies key concepts of working with HoloLens: Gaze, Gesture, Voice, Spatial Audio, Spatial Perception
Today: Gaze & Voice Recognition
Gaze & Keyword ManagersHoloToolkit provides GazeManager prefab for getting information about objects hit with users gaze
HoloLens Unity SDK provides KeywordRecognizer API to register voice commands and trigger event callbacks
Example: Bigger, Smaller to adjust object size
HoloToolkit adds KeywordManager prefab for registering keywords and handlers in Unity Editor
ProblemScenario: Looking at objects instantiated at runtime and sending voice commands to those objects
Problem: Objects need to exist in scene hierarchy to register keywords and event handlers in KeywordManager
Workaround: OnSelect sent by GestureManager, OnGazeEnter & OnGazeLeave sent by GazeManager + multiple dynamically created KeywordRecognizers
Ideal: No boilerplate code when using voice commands + solution that scales with large number of objects
Problem
SolutionFocusedObjectMessageSender component is a proxy that sends keyword events to any currently focused object
FocusedObjectMessageSender
SolutionSelectedObjectMessageSender component adds persistent selection to explicitly select objects for voice interaction
SelectedObjectMessageSender
OutcomeNow: Sending voice commands to focused and selected objects is easy, no code required
Simple: Modular architecture decouples message senders from message receivers
Next: Automatic registration of keywords from holographic UI elements and other objects
Challenges: String message performance + type safety
Call to ActionDiscuss issues and challenges on HoloLens Developer Forum & at WinHUGR meetups
Recognize common problems that can be solved for everyone with some minimal code
Fork HoloTookit-Unity on Github, open issues, submit pull requests, address code reviews
Spread the word and help grow the community
Thank You!@dbouianov
@contextvr
http://contextvr.com
ResourcesHoloToolkit for Unityhttps://github.com/Microsoft/HoloToolkit-Unityhttps://github.com/Microsoft/HoloToolkit-Unity/issues/71https://github.com/Microsoft/HoloToolkit-Unity/pull/72https://github.com/Microsoft/HoloToolkit-Unity/pull/69
HoloLens Developer Forumshttp://forums.hololens.com/
WinHUGRhttp://www.meetup.com/Windows-Holographic-User-Group-Redmond/