Behavioral Engineering and Brain Science in VRaliceabr/VR-class-lecture.pdfrelated mechanical...

Behavioral Engineering and

Brain Science in Virtual

Reality

Bradly Alicea

http://www.msu.edu/~aliceabr

Introduction

Introduction

How can you use Virtual Reality to “engineer” and measure behavior?

Augmented Cognition

Introduction


Augmented Cognition

Brain-Computer Interfaces

COURTESY: http://cnel.ufl.edu

Introduction


Augmented Cognition


VR for therapy Jeka Lab, University of Maryland


PTSD Research, USC

Introduction


Augmented Cognition


VR for therapy

VR for neuroimaging Montague et.al, Nature 431, 760-767 (2004)

Jeka Lab, University of Maryland


PTSD Research, USC

Background Readings

Bohil, C., Alicea, B., and Biocca, F. (2011). Virtual Reality in Neuroscience

Research and Therapy. Nature Reviews Neuroscience, 12, 752-762.

Background Readings

Bohil, C., Alicea, B., and Biocca, F. (2011). Virtual Reality in Neuroscience

Research and Therapy. Nature Reviews Neuroscience, 12, 752-762.

Alicea, B. (2011). Naturally Supervised Learning in Motion and Touch-driven

Technologies. arXiv: 1106:1105.

Behavioral

Engineering

The “Closed-Loop” System

Virtual

World Individual

Work with hybrots (Figure 1, Artificial

Life, 6, 307 -- 2000).


Virtual

World Individual

Loop is closed by FEEDBACK:

Physiological State of User


Life, 6, 307 -- 2000).


Virtual

World Individual

Loop is closed by FEEDBACK:

Physiological State of User

Adaptive Control of Virtual World


Life, 6, 307 -- 2000).

Figure 1,

Autonomous

Robots, 11,

305-310

(2001).

A Cybernetic Approach

“Design for an Intelligence Amplifier”

W.R. Ashby, 1958

X is the User (with a

brain and senses),

receiving inputs from

S via U.



W.R. Ashby, 1958


brain and senses),


S via U.

G is the feedforward from X to

S. Ashby thought of this as

“intelligence”.



W.R. Ashby, 1958


brain and senses),


S via U.

S is the amplifier.

Operates within a

range of values,

and uses selection

criterion on input

G.



“intelligence”.



W.R. Ashby, 1958


brain and senses),


S via U. U is improvement in

performance due to S.

S is the amplifier.

Operates within a

range of values,

and uses selection

criterion on input

G.



“intelligence”.

Examples from

Arousal and

BCI Design

Mitigation of Sub-optimal Arousal Yerkes-Dodson Curve:

* model of arousal (attention,

wakefulness) in terms of performance.

* single performance optimum

(convexity).

OPTIMUM





(convexity).

Mitigation Strategy:

* when performance deviates too far

(BLACK) from optimal point (RED),

performance is actively corrected.

OPTIMUM

Heads-up Display (Siemens VDO)





(convexity).

Mitigation Strategy:

* when performance deviates too far

(BLACK) from optimal point (RED),

performance is actively corrected.

OPTIMUM

Heads-up Display

Optimal Range


Figure 1, J. Neural Engineering, 4,

R32 (2007).

“Feedforward” signal

involves much

computation, post-

processing.

* signal processing

component (brain signals

to machine input).


aaaaa


R32 (2007).


involves much

computation, post-

processing.

* signal processing


to machine input).

Signal trace, acquisition


aaaaa


R32 (2007).


involves much

computation, post-

processing.

* signal processing


to machine input).

Signal trace, acquisition Segmentation, Classification

COURTESY: Mathematical Biosciences,

157(1–2), 303–320 (1999)


aaaaa


R32 (2007).


involves much

computation, post-

processing.

* signal processing


to machine input).

Signal trace, acquisition Segmentation, Classification Control Signal

COURTESY: Mathematical Biosciences,

157(1–2), 303–320 (1999)

Measuring

Neural Activity

What are we measuring?

Time-course Data and Elicited Responses

COURTESY: Gallant Lab,

Berkeley

COURTESY: MathWorks

COURTESY: Biopac

Building a Controlled Environment

Two advantages to using virtual

environments as stimuli:

1) high degree of control and

selective manipulation.

* design custom objects with unique

physical properties.

2) introduce naturalistic contexts.

* observe behavior as it occurs in the

real world.

Physiological Measurement

How do you measure nervous system activity related to behavior?

EMG (electrical impulses in muscle)



EMG (electrical impulses in muscle) EEG (electrical fields in the brain)

COURTESY:

Nature 397,

430-433 (1999)




fMRI, fNIR (functional neuroimaging –

cerebral blood flow)

EEG (electrical fields in the brain)

COURTESY:

Nature 397,

430-433 (1999)




fMRI, fNIR (functional neuroimaging –

cerebral blood flow)

EEG (electrical fields in the brain)

Force sensors, accelerometers (movement-

related mechanical signals)

COURTESY:

Nature 397,

430-433 (1999)

Examples of

Nervous System

Measurement

Touch- and Motion-driven

Technologies What if we could decouple human motion from “real-world” physics?

* perturbation – interference with typically-experienced physics during a movement,

activity.

* removal of perturbation – removal of forces associated with interfereing with

typically experienced physics (loaded = perturbation vs. unloaded = removal).


Technologies What if we could decouple human motion from “real-world” physics?

* perturbation – interference with typically-experienced physics during a movement,

activity.

* removal of perturbation – removal of forces associated with interfereing with

typically experienced physics (loaded = perturbation vs. unloaded = removal).

Pre-trial Trials Trials Trials Trials Trials

Real-world L U U L L

LOADED:

composite golf club

driver with weight

(forcing chamber).

UNLOADED:

performance of activity

without associated

rotational, inertial forces

(motion capture only).


Technologies

Measurement techniques:

Mapped Physiological Output (MPO):

Can the subject move a virtual object

to a target? Related to magnitude of

force production:

* subject produces x amount of force,

mapped to virtual world object

movement.

BEHAVIORAL


Technologies



Unmatched Muscle Power (UMP):



force production:



movement.

EMG amplitude (RP) and MPO are

used as a proxy for force production :

* Subject produces x amount of force,

muscles contract, “force” is measured

indirectly in two ways.

BEHAVIORAL

HYBRID


Technologies



Unmatched Muscle Power (UMP):

EMG signal spikiness (z):



force production:



movement.

EMG amplitude (RP) and MPO are

used as a proxy for force production :

* Subject produces x amount of force,

muscles contract, “force” is measured

indirectly in two ways.

BEHAVIORAL

PHYSIOLOGICAL

HYBRID

z parameter used to characterize

variance in the amplitude of a signal

(time-series) over interval i.


Technologies Surface simulations: “virtual” surface reaction forces may allow us to manipulate

muscle function.

Applications to touch screens, motion capture systems (Kinect API), physical

therapy.


Technologies Surface simulations: “virtual” surface reaction forces may allow us to manipulate

muscle function.

Applications to touch screens, motion capture systems (Kinect API), physical

therapy.

COURTESY: Magic Vision Lab

“Virtual” Animal Model Research

Virtual Environments can also be

used to study animal nervous

systems (basic science):

* insect VR – an arena that simulates

visual and olfactory cues in an

adaptive (e.g. closed-loop) fashion.

Clockwise from top left:

Flight arena for insects

(COURTESY: Janelia

Farm).








* manipulate specific sensory

systems, provides a means to study

animal cognition and neurobiology

for which animal models are used.



(COURTESY: Janelia

Farm).

Airborne Cue Control:

Nature Methods, 9(3),

290-296 (2012).








* manipulate specific sensory

systems, provides a means to study

animal cognition and neurobiology

for which animal models are used.

* creates a naturalistic environment

that can be easily controlled

(ecological validity).



(COURTESY: Janelia

Farm).

Airborne Cue Control:

Nature Methods, 9(3),

290-296 (2012).

Setup for moth VR. From

NRN paper.

Sensory Engineering and VR Design

Two design principles: 1) Multisensory Integration: Senses are combined to produce cognitive behaviors (vision + audition, vision + touch).

* virtual stimuli presented to user, senses merged in their brains leads to synthetic,

suppressive cognitive effects. Decoupling senses can have as big of an effect as

strategically combining them.

2) Exploit Neural and Sensory Processing: Use the immediate and long-term features of sensory and other neural processing to build more effective virtual worlds. * arXiv paper experiments: altering experienced forces in bursts uncovers sensory and neuromuscular constraints of individual users. Interactions between short-term adaptations and longer-term memory.

Behavioral Engineering and Brain Science in VRaliceabr/VR-class-lecture.pdfrelated mechanical...

Documents

Transcript of Behavioral Engineering and Brain Science in VRaliceabr/VR-class-lecture.pdfrelated mechanical...