01/09/2018 CPEN541: Copyright from 2019, Sidney Fels
Human Measurement Technologies
!1
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• HMT: Introduction – categorizations – physical properties
• Introduction to Sensors • Tracking Technologies
– magnetic – optical – mechanical – video based – other
• Primary user input – head tracking – eye tracking – hand tracking – pointing and selecting – other
HMT: Overview
!2
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• How to categorize? – By human sense organs?
• Visual, audio, proprioception, taste, smell • doesn’t relate to which physical property is being measured
– By physical properties/physical principle? • Electrical sensor, optical sensor, magnetic sensor, mechanical sensor • one effect can be used to measure many attributes
– By relation of source to reference? • Inside-in
– sensor and source on body • Inside-out
– sensors on body which sense off-body source (artificial or natural) • Outside-in
– external sensor measures artificial or natural markers • no relation to what is being measured
HMT: Introduction
!3
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• For this class: – start with discussion of some physical sensors – task separation
• general (large object) body tracking • primary user input techniques
– pointing and selecting • mouse+
– head tracking – eye tracking – hand tracking
• Other interesting devices
HMT: Introduction
!4
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• General HMT framework:
• For HMT we’ll focus on: – input/data acquisition – some pattern/gesture recognition
• these will be good topics for student lectures
HMT: Introduction
!5
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Physical quantities available for transduction – electrical
• voltage • resistance • impedance
– optical • colour • intensity
– magnetic • induced current • field direction
– mechanical force
HMT: Sensors
!6
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• How to pick a sensor? – Decide which human part/action you want to measure – determine which attributes are related to the part – check what sensors you have available
• possibly modify it to suit your needs – if none available
• determine how to transduce quantity • build sensor
HMT: Sensors
!7
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Transduced physical quantity may need – conversion to voltage – filter
• band limiting • outliers removed
– reduce complexity
HMT: Signal Conditioning
!8
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Possibly need to do some data acquisition – use a data acquisition board plugged into
your computer • e.g. National Instruments DAQ
– Up to 16 analog inputs; 12-bit resolution; up to 500 kS/s sampling rate
– Two 12-bit analog outputs; 8 digital I/O lines; two 24-bit counters
-Icube (voltage->MIDI signal) -Arduino board - audio port (single analog channel) - video capture board (frame grabber)
HMT: DAQ
!9
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Piezoelectric Sensors • Force Sensing Resistors • Accelerometer (Analog Devices ADXL50) • Biopotential Sensors • Microphones • Photodetectors • CCDs and CMOS cameras • Electric Field Sensors (Fish) • More Sensors
HMT: Sensors
!10
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Piezoelectric effect – energy converted between mechanical
force /pressure and electrical form • piezoelectric microphone • piezoelectric speaker
– deformation of piezoelectric causes charge build up on surface
– put material in capacitor and measure voltage change
• Q_f is charge change due to pressure
HMT: Piezoelectric Sensors
!11
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• About 10-15% error • useful for many applications
HMT: Force Sensing Resistors
!12
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Measure acceleration of sensor • several techniques available to do this
– use a mass with some springs – use gyroscopes – use vibrating tuning forks – heated air cooling
• Basic principle: – Hooke’s law: F = kx – Newton: F = ma – displacement is proportional to acceleration – single axis accelerometer
• Analog Devices ADXL50 accelerometer •
HMT: Accelerometer
!13
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Popular model – Analog Devices ADXL50 accelerometer
• new models available: ADXL150 and 250 ($15-$24) • 10 mg resolution, 10mg to 50g
– dual axis – micromachined silcon bar – capacitance used to measure displacement – intended application:
• airbags
HMT: Accelerometer
!14
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Other techniques used measuring acceleration: – Electromechanical – Piezo-Film – Piezoelectric – Piezo Resistive Bulk Micromachined – Capacitive Bulk Micromachined
• Applications: – tilt sensing – inertial sensing
HMT: Accelerometer
!15
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Carbon: – lightly packed carbon – compression increases conduction
• needs power supply
• Capacitor (condenser): – capacitor between a stationary metal plate, and a light metallic
diaphragm – compression changes capacitance by moving diaphragm
• need power supply
• Electret and Piezoelectric: – already talked about these – no external power needed
• Magnetic (moving coil): – induction - moving conductor in magnetic field
HMT: Microphones
!16
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Output voltage proportional to light – photodiodes (use voltage bias, light affects
concentration of electron/hole pairs -> current ) – photovoltaic effect (P-N junction, light produces
voltage, electron/hole pairs separate) • also used for solar panels
• can arrange in array – photodiode array (PDA)
• used in Optotrak
Current changes with light
HMT: Photodectors
!17
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• CCD - charge coupled device – 2D array of capacitors on silicon substrate – light creates electron/hole pairs and accumulates charge – read charge after some interval
• similar to PDA – more sensitive to low light
HMT: CCD and CMOS cameras
!18
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Some cameras use different sizes of CCD element • some use 3 CCDs
– red, green and blue separated • Lots of different styles and products • Very versatile
– low light sensitivity • astronomy applications, microscopes
– Example: Marshall Electronics, V1055 • 0.05 lux, HAD sensor technology • 510 * 492 pixels (60 fields/sec) • 380 lines of resolution • Automatic electronic iris • 12 VDC, 100 mA, 1.2W • 1/3” CCD
• Consumes lots of power – need various sub-voltages
Can be board mounted
HMT: CCD Cameras
!19
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• CCDs have to transfer charge rows and columns one at a time
• CMOS photodiode arrays put amplifier at each pixel – about 3 transistors (photodiode version) – 1 transistor (photogate version)
• amplifier circuitry takes up a lot of room – only viable recently as sub-micron tech. gets better – only useful for low-end still
• cheap (<$100), low power (10-50mW vs. 1-2W) • offer single chip solution
HMT: CMOS Cameras
!20
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Example: Marshall Electronics – Single chip 1/3" format video camera – 310 TV line resolution – Composite NTSC/PAL video output – On-chip auto exposure – Automatic gain control – Auto white balance – 7 VDC-12 VDC, 20mA @ 9 VDC
HMT: CMOS Cameras
!21
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Sense electric field – shunting
• external EF grounded by person – transmitting
• low frequency energy coupled to person • person is emitter
• uses: – to transmit data, Personal Area Network (PAN) – position sensing
• some problems – proximity sensing
• easy to make • cheap
HMT: Electric Field Sensing
!22
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Here’s a few that find common usage: – Bend - piezo-resistive – Close - IR reflection, 1-7” – FarReach - ultrasonic (50Hz update) – Flash - phototransistor – Gforce - piezo-electric single axis accelerometer – Hot - zener effect (thermocouple)
• -40 to 100deg C – Light - photo-resistive
HMT: More Sensors
!23
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Reach - EMF disturbance • Slide - resistive • TapTile - Force senstive resistor • Tilt
– electrolytic, single axis (-70-+70 deg) • Touch - 0 travel FSR • TouchGlove
– several touch sensors • TouchStrip
– long touch sensor • Turn
– potentiometer
HMT: More Sensors
!24
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Tracking user location is very important for many applications – want 3 dof position
• X, Y, and Z – want 3 dof orientation
• roll, pitch, yaw (Euler angles) – sometime better to use:
• quaternions (or angle axis) • rotation matrix
– Trackers used to measure 3-6 dof typically.
HMT: Tracking Interfaces
!25
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Head • Eyes • Hand • Body • Other
HMT: Things to Track
!26
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Main characteristics – resolution – accuracy – range – System responsiveness
• sample rate - sensors checked (frequency) • data rate - positions computed (frequency) • update rate - system reports new positions (frequency) • latency (lag) - hopefully in msec • repeatability - same bend, same data value (variance) • linearity - response characteristic • drift - temperature, mechanical force (% of max)
HMT: Head Tracking Interfaces
!27
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Some Head Tracking Technologies also good for tracking: – Hand – Body
• but, some only good for heads • Uses:
– viewpoint tracking • immersive VR with HMDs • attention
– motion parallax systems • fish tank VR, Cubby,
HIT: Head Tracking Interfaces
!28
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• In head tracking: – latency is probably most important system attribute
• >10msec may cause sickness (Durlach, 1994) • >60msec impair adaptation and immersion (Durlach, 1994) • >500msec are not interactive (Bryson, 1993)
– situation determines importance • I.e. hand tracking latency more critical in non-immersive VEs
– (Ware and Balakrishnan, 1994)
– Sources: • tracker, communication delays, computation delays, graphic
rendering, (Bryson, 1993) • video sync.
HMT: Head Tracking Interfaces
!29
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Head movements – max: 1000 deg/sec (33 deg/33msec) – typical:
• 600 deg/sec (20 deg/33msec) yaw • 300 deg/sec (10 deg/33msec) pitch and roll
– most energy below 8 Hz and all below 15 Hz • -> update rate of 30 Hz is sufficient
HMT: Head Tracking Interfaces
!30
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Polhemus Fastrak is very popular sensor – see also Ascension for
competition – many different instantiations
• long range, cheap version, small, etc.
• Uses alternating magnetic field • Receiver has three wire loops • As loops move in field voltage
is induced • Measure voltage and determine
HMT: Polhemus Fastrak
!31
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Specs ($5000 with one receiver, $1000/receiver) – full accuracy 30”, reduced accuracy 10” – latency, 4msec - really?
• 8.5 msec, unfiltered (Adelstein, Johnston, Ellis, 1995) – Update Rate
• 120 Hz / number of receivers – Linear Accuracy
• 0.03 in – Angular Accuracy
• 0.15 degrees – Resolution
• 0.0002 in/in, 0.025 degrees orientation – RS232 interface, – worst user manual in the world – only works in one hemisphere
HMT: Polhemus Fastrak
!32
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Optotrak 3020 from Northern Digital Inc. (about $100,000)
• Uses IR based receivers with IR LED markers on body • three 1-D CCD track IR LEDs
– 3 dof of markers calculated from CCD readings • can use off line or in real time mode • Specs:
– data rate: • 3500Hz (raw), 600Hz (real-time 3D)
– Accuracy • 0.1mm for x, y, 0.15 for z at 2.25m distance
– x,y Resolution • 0.01 mm at 2.25m distance
– Max markers: 256 – FOV: 1.28m x 1.34m at 2.25m
• Good for very accurate position detection at high data rates
HMT: Optotrak
!33
01/09/2019
• OptiTrack • Vicon • others
Many other optical motion tracking system out there
!34
01/09/2019
• Motion tracking with body model – head, arms and feet – body geometry
• 20 joints per person – face recognition
• RGB camera – 30 Hz
• depth sensor – Kinect: Infrared pattern light + camera – Kinect 2: infrared time-of-flight camera
• microphone array – directional sound localization, speech
recognition and noise cancelation • Cheap!
Kinect (2011) and Kinect 2 (2014)
!35
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Eye tracking is important in: – motion parallax based systems
• fish-tank VR • Cubby
– attention – collaboration systems – camera focus and stabilization
• Techniques: – optical (reflection) – electrical (electrooculogram EOG) – magnetic (wire in lens on eye)
HMT: Eye Tracking
!36
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Many different attempts – Levine, 1981; Bolt, 1982l; Ware and Mikaelian, 1987; Jacob, 1990 – Neural network technique, Baluja and Pomerleau, 1993 – MAGIC (Manual and Gaze Input Cascaded Pointing), Zhai,
Morimoto, Ihde, CHI99 • Main problems are
– fovea is 1 degree and there is jitter • 25” viewing distance, 0.44 in • not precise enough for UI widgets
– Eye has not evolved as control organ • considerable non-volitional movement • select techniques not natural to eye
– dwell time, blink
HMT: Eye Tracking
!37
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Solution – separate what eye is good at from what it is bad at
• use gaze to indicate region for cursor • manual input for target acquisition and selection
– Liberal positioning • move cursor to any new location more than 120 pixels from current
location – Conservative
• move cursor only when manual device activated – cursor appears moving toward target in same direction as manual device – interaction now with input device dynamics
– Is eye tracking a Fitts’ Law task? • Pointing time is proportional to log(A/W) • Ware and Mikaelian, 1987 - yes • Silbert and Jacob, 1996 - no • if not, time to acquire should be faster
HMT: MAGIC
!38
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Implementation – IBM Almaden Eye Tracker
• tried ASL Model 5K, not good enough • use two light sources for reflection off retina
– one on-axis and one off axis – synchronize two Near-IR sources with odd and even camera
fields – most systems use only one light source – Tomono et al., 1989, Ebisawa and Satoh, 1993
• Use corneal glint plus pupil location to determine gaze direction
• filter to select only fixation points – ignore saccadic movements
HMT: MAGIC
!39
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: MAGIC
!40
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Pilot study – didn’t use mouse
• “intelligent” mode is problem since clutching needed • used… isometric pointing stick… surprise! • Did typical Fitts’ pointing tasks in experiments • Manipulated
– target sizes (20 and 60 pixels) – target separation (200 pixels, 500 pixels and 800 pixels) – target directions (horizontal, vertical and diagonal)
• Three conditions – manual only – conservative MAGIC – liberal MAGIC
HMT: MAGIC
!41
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Results: – liberal slightly faster than manual only (6.8%) – conservative slightly slower than manual only (4.3%) – fastest time was achieved with conservative (1.03s) – Both generally liked by test group – subjects noticed engineering problems with tracker – not clear how much of an advantage can be had
• should be able to do much cheaper now
HMT: MAGIC
!42
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Eye tracking will become cheap – will it be useful?
• Hands free environments • disabled users
– probably don’t want to use it as point and select device • attention tracking rather than precise pointing device
HMT: Eye Tracking Summary
!43
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Hands/fingers are main techniques for manipulation • positions and orientations all very valuable • multiple degrees of freedom can be easily manipulated
simultaneously – in 2D and 3D
• coordination of two hand (and feet) can be very good (see Guiard’s work on bimanual control)
• quiet, unobtrusive, embedded mode identification etc. etc. – pointing and selecting (group presentation)
• mouse, touchpad, isometric joystick, isotonic, touch screen, etc. • see Buxton’s discussions on pointing - does 1+1=2?
– The details of pointing devices are critical. • see MacKenzie and Oniszczak, CHI’99 for some comparisons on
touchpads
HMT: Hand/Arm/Body Tracking
!44
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Hands can do other things – see McNeill for example
• deitic and symbolic gesturing • how to use in a CHI?
• Technologies for tracking: – Fastrak, Flock of Birds – Cameras (Utsumi) – Cyberglove, VPL Dataglove, Exos glove
• cheaper versions – LEDs
HMT: Hand/Arm/Body Tracking
!45
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Applications – 2D and 3D input devices – Remote control manipulators – puppetry and computer animation – musical performance – surgical simulation – scientific simulation
HMT: Hand/Arm/Body Tracking
!46
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
– gestural interface examples • sign language recog., finger spelling
– Kramer and Leifer, 1989 – Glove-Talk: Fels and Hinton, 1990
• Music controller – foot controller (Paradiso, 1998) – sound sculpting (Mulder, Fels, and Mase, 1999)
• gesture mappings – Glove-TalkII: Fels and Hinton, 1998
HMT: Hand/Arm/Body Tracking
!47
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: Hand/Arm/Body Tracking
!48
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: Hand/Arm/Body Tracking
!49
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: Polhemus Fastrak
!50
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: CyberGlove
!51
01/09/2019
Inertial Measurement Unit (IMU)
!52
Xsense
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• 5 & 14 sensors
• fibre optic based
• USB/RS232
• 75 Hz sample rate
HMT: 5DT Glove
!53
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: Pinch
!54
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• New Technologies – Ring interface (Fukumoto and Tonomura - CHI97) – Electric Fish Technology (variation on Theremin)
• Zimmerman et al., CHI95 and now used in PAN – Virtex uses flex sensors for body suit – Electromagnetic Articulograph (EMA) - see Zierdt,
Hoole and Tillmann
Hand/Arm/Body Tracking
!55
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Put ring mounted accelerometers at the base of each finger. – No acquire time
• currently still using wires – use direct coupling
• similar to EF body coupled like PAN
• Chords used to specify characters – novice: 27 symbols/hand, 130 symbols/min – expert: 52 symbols/hand, 210 symbols/min
HMT: FingeRing
!56
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: FingeRing
!57
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Besides 6dof trackers
3D Input Devices
!58
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: Immersion Probe
!59
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: FingerMouse
!60
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: SpaceBall
!61
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT: Toronto EGG
!62
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• GTII • SoundSculpting • Bricks • Virtual Chopsticks • Instrumented Footwear
HMT: Gesture I/F Examples
!63
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Bricks: Laying the Foundations for Graspable User Intefaces – Fitzmaurice, Ishii and Buxton, CHI’95
• idea – use physical artifacts (bricks/handles) to attach to
virtual objects • graspable user interface • affordance of handle leveraged
– two-handed, spatial caching, parallel position and orientation control
• objects are on ActiveDesk
HMT Examples: Bricks
!64
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Bridges gap (somewhat) between – time-multiplexed input – space-multiplexed input
• Graspable UIs – two handed interactions – specialized, context sensitive input devices – parallel input – leverage prehensile behaviours – externalizes internal computer representations
• aka tangible bits – interface elements more direct – leverage spatial reasoning – space multiplexing – affords multi-person/collaborative use
HMT Examples: Bricks
!65
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Implementation – Lego sized objects’ position tracked
• use Ascension birds and mock ups – physical object associated/”attached” to virtual object
• i.e. MacDraw objects – One handle Operations
• place brick on object -> attach • lift brick off object -> detach • moving brick moves attached objects • rotating brick rotates object (c.o.g. centre of brick)
– Two handles • stretching, other deformations
HMT Examples: Bricks
!66
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT Examples: Bricks
!67
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Experiments – used mock-ups for exploratory hand
manipulation tasks • block sorting tasks, stretchable square, MacDraw
comparison and Curve matching – Prototype used for GraspDraw
• physical tray for function selection – select, delete, rectangle, line, circle, colour – use audio feedback to indicate selection
• developed concept of – anchor: origin of frame of reference – actuator: operate in anchors frame of reference
HMT Examples: Bricks
!68
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT Examples: Bricks
!69
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
HMT Examples: Bricks Design Space
!70
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
– Like LEGO behaviour construction kits (Resnick, 1993),
– AlgoBlocks (Suzuki and Kato) – 3-Draw (Sachs, Roberts and Stoops) – Props (Hinkley et. al. - look in readings, 1994) – Digital Desk (Wellner, 1993)
HMT Examples: Bricks
!71
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Many technologies available to measure human activity
• Knowledge of technologies critical for research and development in – pointing and selecting – ubiquitous computing – musical controllers – 2D and 3D direct manipulation interfaces – etc.
HMT Summary
!72
01/09/2019 EECE518: Copyright from 2002, Sidney Fels
• Important issues: – measurement side
• resolution, accuracy, update rates, lag, range etc. – application side
• continuous vs. discrete • mapping issues • intrusive • matching degree of freedom of control with task • and others
HMT Summary
!73
Top Related