UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop...
Transcript of UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop...
![Page 1: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/1.jpg)
UUIs – Ubiquitous User Interfaces
Alexander Nelson
April 16th, 2018
University of Arkansas - Department of Computer Science and Computer Engineering
![Page 2: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/2.jpg)
The Problem
“As more and more computation is woven into the fabric of our
lives, our interaction with such ubicomp systems cannot be the
focus of our attention. As noted previously, if every physical object
in our home demanded attention in terms of alerts, updates, and
configurations in the way our current personal computers do, we
would become quickly overwhelmed. Indeed, it has been noted
that the benefits of inexpensive Ubiquitous Computing may be
overwhelmed by its personal cost (Heiner et al., 1999). Instead,
the computational and activity support offered by ubicomp systems
must reside in the periphery of our attention and should remain
unnoticed until required.”
Aaron Quigley – Ubiqutous Computing Fundamentals Chapter 6
1
![Page 3: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/3.jpg)
Traditional User Interfaces
Traditional Computing Interface
• Monitor (Visual Output)
• Keyboard (Text Input)
• Mouse & Cursor (View
Context)
• Speaker (Audio Output)
2
![Page 4: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/4.jpg)
Ubiquitous User Interfaces
Two user-in-the-loop interaction styles:
• Notification – Ubicomp system obtains user’s attention
• Modification – User modifies context of the ubicomp system
User out-of-the-loop interaction?
System makes intercessions for user – Does not necessitate UI or
UI may be undesirable
(e.g. Nest changing temperature)
3
![Page 5: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/5.jpg)
Modification
![Page 6: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/6.jpg)
Modification
Modification – Interaction pattern where user intentionally
modifies state of ubicomp system
Traditional GUI analog – Mouse/Keyboard Input
4
![Page 7: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/7.jpg)
Ubiquitous Computing Input Technologies
Input technologies for ubicomp systems are many and varied
Example inputs:
• Infrared Remote Controller
• Wall-mounted switch panels
• Web-based dashboards
• Touchscreen
• Game Console Controllers
Can cause confusion in how to interact with systems
5
![Page 8: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/8.jpg)
Usability Overlap
Figure 6.3 – “Controller Hell”1
1This lecture is developed from text and images provided in our optional
textbook – Ubiquitous Computing Fundamentals, Chapter 6, by Aaron Quigley
6
![Page 9: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/9.jpg)
Sensor-based Input
Ubiquitous Interfaces often make use of sensed contexts to
facilitate/aid interaction
Common sensor-based inputs:
• Physiological Measurement
• Environmental Measurement
• Location
• Identity
• Audio/Video
• Gesture
• Touch
7
![Page 10: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/10.jpg)
Sensor-based Input
How does the system designer make use of these sensed values for
input?
So far we covered sensors in terms of:
• Context-Awareness
• WBANs/MBANs
These interaction patterns are typically more stochastic and
inferred/implicit rather than explicit and intentional
8
![Page 11: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/11.jpg)
Gestures – Intentional Sensor Based Input
Gesture – Movement of part of the body to express meaning
Human-to-Human Gesture – Aid/Facilitate ideas in lieu/support of
speech (e.g. Sign Language)
Gesture Recognition – Interpreting human gestures through
computation
Gestures can be interpreted as intentional ambient
human-computer interaction
9
![Page 12: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/12.jpg)
Example: Handshake
Haddock et al. 2009 – Handshake detection for social network
development
10
![Page 13: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/13.jpg)
Pen/Stylus Gestures
Mobile computers (e.g. Palm Pilot and others) gave rise to
pen/stylus based gestures
11
![Page 14: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/14.jpg)
Inertial Sensor-based Gestures
Inertial Sensors (e.g. Accelerometer/Gyroscope) can establish 6
degrees of freedom for gestures
Move Up/Down, Left/Right, In/Out
Rotate Yaw, Pitch, Roll
12
![Page 15: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/15.jpg)
Smartphone Gestures
Touchscreen + Inertial Sensor Input
13
![Page 16: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/16.jpg)
Vision-based Gestures
Kinect – Vision + Depth
14
![Page 17: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/17.jpg)
Physiological Sensor-based Gestures
Physiological Sensors as gesture input:
• Electromyography (EMG) – Myo Band
• Electrooculagraphy (EOG) – Eye Motions
• Electroencephalography (EEG) – Brain-Computer Interface
15
![Page 18: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/18.jpg)
Interpreting Gestures
Sensors provide a time-series set of data
Types and frequency depend on set of sensors
Example: Wii-Mote w/ Remote Plus
Sensors – Accelerometer, Gyroscope, Optical Sensor (for
triangulation)
Single point in time defined by the 8-dimensional tuple:
Si = 〈Ax ,Ay ,Az , ωx , ωy , ωz ,Px ,Py 〉Where A is acceleration, ω is angular velocity, and P is the
position of the pointer (can be null)
16
![Page 19: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/19.jpg)
Interpreting Gestures
Gesture defined as the set of time-series points:
G = {S0, S1,S2, ...,Sn}Gesture Classification can be continuous or discrete
Discrete – Pre-segmented gestures (i.e. Touch-based Smartphone
gesture)
Continuous – Must be recognized during continuous sensor
sampling
Discrete gestures are typically segmented through distinct
end-points
“Pen-down” and “pen-up” – Distinct segmenting time points
(from stylus being touched until removed)
17
![Page 20: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/20.jpg)
Gesture Segmentation
For continuous Gesture Recognition, time series is often segmented
for classification 2
Example: Sliding Window
2Some on-line recognition continuously update probabilities and do not need
segmentation
18
![Page 21: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/21.jpg)
Gesture Segmentation Strategies
Common segmentation strategies:
• Sliding Window
• On-line gesture likelihood classifier
Detect probability of any gesture occurring, based on feature
extraction (reduces # of times classifier runs)
• Start-Stop gesture actions
19
![Page 22: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/22.jpg)
Interpreting Gestures
Gestures match against an “alphabet” – The available gestures
Alphabets need to be accurate (both precision & recall)
Alphabets need to be memorable
20
![Page 23: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/23.jpg)
Interpreting Gestures
How do you classify gestures to the “alphabet?”
Classifier – Determine gesture based on features of input data
Classifier can be programmer defined or from Machine Learning
21
![Page 24: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/24.jpg)
Usability
![Page 25: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/25.jpg)
Interface Usability
Usability often defined by five quality components3:
• Learnability – Training Time
• Efficiency – Execution Time
• Memorability – Retraining Time
• Errors – Precision/Recall
• Satisfaction – Pleasant to use (Qualitative)
3https://www.nngroup.com/articles/usability-101-introduction-to-usability/
22
![Page 26: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/26.jpg)
Ubiquitous Interface Usability
Usability for UUIs is slightly different
Quigley defines usability for UUIs in terms of:
• Conciseness – Few actions in brief time to achieve task
• Expressiveness – Consistency of use
• Ease – Learning curve for UUI
• Transparency – How well UUI conveys state information
• Discoverability – Can user make mental model of interface
• Invisibility – How well does UUI stay in the periphery
• Programmability – Can UUI be extended for other applications
23
![Page 27: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/27.jpg)
Notification
![Page 28: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/28.jpg)
Notification
Notification – Ubicomp system obtains user’s attention
Primarily in calm ubiquitous interfaces
i.e. A system that is in the periphery of attention that wants to
provide information
24
![Page 29: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/29.jpg)
Ambient User Interfaces
Ambient Interfaces are intended to be “ignorable”/“glanceable”
Information that typically resides in the periphery of attention
Periphery – based on visual analog where certain forms can be
detected outside line-of-sight
25
![Page 30: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/30.jpg)
Criteria for Ambient UIs
Quigley defines five characteristics of a Ubiquitous User Interface
to be a tangible interface:
1. Provide a clear coupling of physical artifact to relevant and
related computation, state, and information
2. Ensure contextually appropriate physical affordances for
computational interaction
3. Ensure contextually sensitive coupling of physical artifact to
intangible representation (audio/graphics)
4. Support “invisibility in action” (not literal invisibility) and
natural behavior
5. Ensure a grounding of the UI interaction design in the fields of
ethnography, industrial design, kinesthesiology, and
architecture
26
![Page 31: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/31.jpg)
Ambient Notification
Peripheral information can be conveyed through:
• Static Visual
• Dynamic Visual
• Sounds
• Touch/Tactile/Haptic
• Smell
• Temperature Changes
Based on the five traditional senses (i.e. Sight, Hearing, Smell,
Touch, Taste(?))
27
![Page 32: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/32.jpg)
Ambient Interface – Sight
Power Aware Cord – Gustafsson
28
![Page 33: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/33.jpg)
Ambient Interface – Sight
Information Percolator – Heiner 1999
29
![Page 34: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/34.jpg)
Ambient Interface – Sight
Ambient Umbrella
30
![Page 35: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/35.jpg)
AmbientROOM – MIT Media Lab
1997 Exploration of ambient information delivery through light,
shadow, sound, airflow, water movement
Click photo to navigate to project page
31
![Page 36: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/36.jpg)
Speech Recognition
![Page 37: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/37.jpg)
Speech Recognition as UUI
Speech Recognition – Computational interpretation of human
speech through audio input
Has emerged as a popular UUI in recent years
32
![Page 38: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/38.jpg)
Speech Recognition History
Speech Recognition has a long history:
• 1952 – Bell Labs single digit recognition (“Eight”)
• Late 1960s – Raj Reddy Continuous Speech Recognition
• Late 1960s – Dynamic Time Warping allows 200-word
vocabulary
• 1972 – IEEE Acoustics, Speech, and Signal Processing
Conference begins
• 1970s – Hidden Markov Models used for speech recognition
• Mid 1980s – IBM produces Tangora w/ 20k word vocabulary
• 1990s – First commercial successful speech recognition
technologies
33
![Page 39: UUIs Ubiquitous User Interfaces · 2018-04-23 · Ubiquitous User Interfaces Two user-in-the-loop interaction styles: Noti cation { Ubicomp system obtains user’s attention Modi](https://reader034.fdocuments.in/reader034/viewer/2022042812/5fa999778cf96c79ec6e28f3/html5/thumbnails/39.jpg)
Limitations
End-to-End Automatic Speech Recognition – Joint model
combining pronunciation, acoustic, and language models
Benefits: HMM-based systems require an n-gram language model,
taking several GB of memory
Siri/Google Assistant require network connection to cloud holding
n-gram model
34