A Framework for Perceptive, Attentive, Cognitive, and Social-Interaction Enhancement From...

A Framework

Perceptive, Attentive, Cognitive,

Social-Interaction Enhancement

From User-Centred

Ego-Centred Design

Morris Goldberg

PACSIE

Sreekar KrishnaProf. Panchanathan

tion C

Attention

Social Interaction

•Perception:• Hear a baby cry.

•Attention:•Listen so as to hear one’s baby cry

•Cognition:•Identify the reason for crying

•Social-Interaction:•Rock the baby to stop the baby from crying

Friday, April 21, 2023 PACSIE: Ego-Centred Design 3

Perception, Attention, Cognition, Social-Interaction

•Movement:•Turn head to better hear the baby cry

•Move towards the baby, inspect baby

•Pickup baby, rock the baby

•Autonomic:• Posture: as baby may cry

•Heart-rate: if baby cries

Behaviour Responses:Movement and Autonomic

•Applications• Cerebral: injury, disease, and aging

• Autism, prosopagnosia

• « Extra eyes »

•Training, drugs, invasive• Practice makes perfect,…

•Assistive device• Non-invasive, customized, adaptive, lifelong

Enhancement

Motivation: The Carpenter Who Lost His Tools

Visual agnosia for toolssees tool but lost knowledge of tool

name, function, memory

Haptic tool recognition feels tool, knowledge recovered

Infancy: body(hands) calibrates vision

Maturity:

vision: dominates

body:mostly below conscious awareness

•CO poisoning heater•« Blind » to many objects•Recognized her mother through voice•Recognized colours, texture•Name some surafces

–Red plastic, shiny metal•Spatial acuity intact

–Gratings, but not directions•Basic problem recognizing shapes

•MA Goodale, AD Milner, Oxford Univ. Press

–Sight Unseen: An Exploration of Conscious and Unconscious Vision, 2004Friday, April 21, 2023 PACSIE: Ego-Centred Design 7

Patient DF: Visual form agnosia

Patient DF: Tests

• Object recognition (haptic)

• Copying objects

• Yellow blind to orientation

• Grasped any orientation

Visual processing streams: what, where, how

•Visual agnositics• rare, encumbered, difficult, non-stable

•The blind• visualize objects by touch, stable, intelligent

•Reader• NSF, focus groups

Surrogates: The visually impaired

•Vibro-tactile glove •Sensory substitution: distal visual to proximal haptic

•Haptic rendering not possible

•Observe haptic manipulation

•Uncover distinguishing haptic features/descriptors

•Deliver features by haptic cueing

•Supervised user learning: haptic cues

•Object visualization: user imagination

Haptic object visualisation

Kahol K., Panchanathan S.,“Neurocognitively inspired haptic user interfaces”,

Springer J. of MM Tools and App, v 3, pp 15-38 2007.

Kahol K, Smith M., “Neuro-cognitively Designed Dynamic Simulations For Laparoscopic Surgical Skills”. MODSIM. Virginia Beach, 2007. pp. 34-39.

•Design methodology •Participatory design

focus groups -scenario building

•Prototypic User •Average user

•Learning •Designer discovery

usage patterns, blog, competition..

•User discovery

short-cuts, hidden functionality

User-Centred Design

•Assistive device• white cane.. extension of self…peripersonal space

• left-neglect…stroke…line bisection..finger, laser-pointer, ,stick

•Haptic exploration • haptic glove 6 blind people pattern varies

• depends on degree and onset of blindness, other factors

•Ego-centred design• understanding of the individual

perceptive, attentional, cognitive, social (PACS)

Framework

•1st Principle: Sensory-Motor Coupling

•2nd Principle: Polymodal Reinforcement•Edelman, GM, « Neural Darwinisim: The Theory of Neuronal Group Selection », 1989.

•3rd Principle: Graft Onto Existing Scaffolds•Thelen and Smith, “A Dynamic Systems Approach to the Development of Cognition and Action”, 1996.

•4th Principle: Self Evaluation

Ego-Centred Design: Principles

Active sensing sensory adaptation to constant signal

adapt to pressure on body, smell, even vision

shift body, sniff, eye saccade, ….

« What » is being sensed? behavior is visible

deduce « what »

haptic manipulation- descriptors

Haptic rendering motor-sensory coupling

individual moves, assistive devices provides stimuli

1st Principle: Sensory-Motor Coupling

Modes vision: colour, shape, orientation, motion….

somatosensory: light and heavy pressure, pain, vibration…

auditory, proprioceptive, olfactory:…….

Existence-pathological cases Agnosia: loss of mode/function

Synaesthesia: extra function-e.g see sound, hear color

2nd Principle: Polymodal Reinforcement

Examples texture: somatosensory, visual, and auditory

taste: smell and somatosensory(tongue)

Associative memory Bartlett: memory is a re-creative process

Move hands, visualize, hear , feel

Exploitation Wagner: musical identities

rehabilitation: colour code objects, audio prompts

Haptic glove audio and haptic cues for textures

Child development reaching: months of effort, muscles, vision proprioceptive

orienting to people: instintive attraction to face-like patterns

Continuumscaffolding: basis for new functionalities

development nevers stops

Blind and body languageauditory cues: nervous movement, breathing, voice inflection

assistive device: correlate/integrate

3rd Principle: Graft Onto Existing Scaffolds

Gait variesload, terrain, physical condition

trial and error: self evaluation (pain, comfort)

Continuous self-evaluation

Innate Behavioral Responses attraction to eye-like objects

move eyes- get reward (smile,hug)-

emotional-based value system

4th Principle: Self Evaluation

Everyday Social InteractionsWhat do we know about them?

Social Interactions: Complex Visuo-Motor Interplay

Social Interactions

SensorySensory

PerceptualPerceptual

CognitiveCognitive

MotorMotor

Handshake

Eye Contact

Move towards each other(Keep in mind Proxemics)

Intent to interact

Shake hands

Conversation distance

Psychological Basis of Social Interactions• Emotions

• Interpersonal Attributes• Dominance• Trust• Like/Dislike

• Personality

• Communication• Turn Taking• Feedback• Attention• Iconic Cues, like Victory or Stop hand gestures.• Intrinsic Cues, like moving close to someone

M. Argyle, The Psychology of Interpersonal Behaviour, Penguin UK, 1999.

Psychological Aspects – Dyadic Interactions

FeelFeel SendSend ReceiveReceive FeelFeel

M.G. Almore, “Dyadic Communication,” The American Journal of Nursing, vol. 79, Jun. 1979, pp. 1076-1078.

A. Teeters, R.E. Kaliouby, and R. Picard, “Self-Cam: feedback from what would be your social partner,” ACM SIGGRAPH 2006 Research posters, Boston, Massachusetts: ACM, 2006, p. 138.

ExpressionsBody MannerismsCommunicative Gestures

Insula and the Mirror Neurons

Mirror NeuronsMirror

Neurons InsulaInsula Limbic SystemLimbic System

Simulate Facial

Expression

Feel the Emotion

L. Carr, M. Iacoboni, M. Dubeau, J.C. Mazziotta, and G.L. Lenzi, “Neural mechanisms of empathy in humans: A relay from neural systems for imitation to limbic areas,” Proceedings of the National Academy of Sciences of the United States of America, vol. 100, Apr. 2003, pp. 5497-5502.

Derived out of fMRI studies on human viewing facial images Enhanced activity in the Mirror Neuron Area, Insula and Limbic System observed Findings are just anatomical patterns Exact Pathway is still not discovered

Everyday Social Interactions – The PACSIE Model

Bringing it all together

Friday, April 21, 2023 27PACSIE: Ego-Centred Design

Principle 1: Sensory-Motor Coupling

• Social Interactions are integration of complex sensory and motor actuations and responses.

Enactor (Encoding) Recipient (Decoding)

Visual EngagementVisual Engagement

Coordinated Motor Interaction – Eye Gaze,

Head Nod

Coordinated Motor Interaction – Eye Gaze,

Head Nod

Communication – Asynchronous Exchange

Complex Social Engagement

PerceptionPerception

AttentionAttention

CognitionCognition

Social InteractionSocial Interaction

Enactor (Encoding) Recipient (Decoding)

Speech

27%19%

18%35%

Face Visual

Principle 2: Polymodal Reinforcement

• Social Interactions are integration of multichannel information (from the enactor and the recipient’s perspective).

N. Ambady and R. Rosenthal, “Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.,” Psychological Bulletin. Vol. 111(2), vol. 111, Mar. 1992, pp. 256-274.

Principle 3: Use existing skills (i.e. Scaffolding)

• Social Interactions capabilities are very individualistic based on when the person becomes blind.

Congenitally Blind

Adventitiously Blind

BirthBirth Developmental YearsDevelopmental Years Professional YearsProfessional Years Geriatric YearsGeriatric Years

Social Skills

Social assistance need varies depending on when the person loses his/her sight.

Principle 4: Recalibration by Self Evaluation

• Social Mirroring is the process by which effective social interactions are learned and propagated.

Hello !!!Welcome to

ActionAction

Social MirrorSocial Mirror

PerceptionPerception

Mirror NeuronsMirror

Neurons

Self Evaluate

M. Iacoboni, Mirroring People: The Science of Empathy and How We Connect with Others, Picador, 2009.

Social Solutions Through PACSIEBody Rocking Condition

Role of vision

Learning through

observation

Learning through

observation

Feedback on social

mannerisms

Feedback on social

mannerisms

Visual Impairment affects both aspects of social interactions

Stereotypic Mannerisms

Hand - 60.3%

Head- 13.1%

Trunk - 10%

Body - 6.9%

Leg - 6.4%

Face - 3.4%

Though body rocking is not the

most occurring mannerism, it is the

most noticed in a social scene

V.J. Eichel, “Mannerisms of the blind: A review of the literature,” Journal of Visual Impairment & Blindness, vol. 72, 1978, pp. 125-130.

Body Rocking in Adults

Rocking is a natural development behavior of the cerebellum’s balance system

Body rocking in adults is due to the lack of social feedback

Solutions

InterventionIntervention Self MonitoringSelf Monitoring

Immediate feedbackDoes not address the problemReduces the symptomChance of recurrence is high

Replace with Cognitive TaskProvides for symptom

substitutionChance of recurrence is low

Focus on Controlling the problem and not curing it !!!

M. Celeste, “Social Skills Intervention for a Child Who Is Blind,” Journal of Visual Impairment & Blindness, vol. 101, 2007, pp. 521-533.

R.L. Loftin, S.L. Odom, and J.F. Lantz, “Social Interaction and Repetitive Motor Behaviors.,” Journal of Autism & Developmental Disorders, vol. 38, Jul. 2008, pp. 1124-1135.

Social Solution

PACSIEPACSIE

Body Rocking

Social Solution

Social Feedback

Self Evaluate

Self EvaluateControlControl

Social Interaction Assistant: Work-inProgress

Person, Gaze, Expression, Identity Detector

•1st Principle: Sensory-Motor Coupling•Head movement controls sensing(far-near,left-right)

•2nd Principle: Polymodal Reinforcement•vibration and sound

•3rd Principle: Graft Onto Existing Scaffolds•mood recognition, voice recognition, footstep recognition..

•4th Principle: Self Evaluation•user evaluates/calibrates the signals from the SIA

•eventually SIA will also self-evaluate

Ego-Centred Design: Principles

A Framework for Perceptive, Attentive, Cognitive, and Social-Interaction Enhancement From...

Documents

Transcript of A Framework for Perceptive, Attentive, Cognitive, and Social-Interaction Enhancement From...

Perceptive Content Update Guide v 7 - Hyland · Perceptive Content 7.1.1 ... Perceptive Content Update Guide ... you must install Perceptive Integration Server as part of the Business

Francis Bacon’s “perceptive” instrumentsphilsci-archive.pitt.edu/12320/1/Perceptive instruments_submitted.pdf · 1 Francis Bacon’s “perceptive” instruments Abstract Francis

INTERLANGUE : UNE ANALYSE PERCEPTIVE DU PHONÈME …

PERCEPTIVE INSTITUTE - PAREXEL · · 2016-01-18WELCOME TO PERCEPTIVE INSTITUTE ... Perceptive MyTrials® Data-Driven Monitoring Courses 41 ... • Log into DataLabs and sign a patient

Perceptive Content Update Guide v 7 - Hyland · Update Integration Server ... Perceptive Content 7.1.1 ... RabbitMQ Clustering Guide.

Perceptive media for #CanvasConf

Objects, responsive and perceptive media

Attentive to the Word

Perceptive Content Server - Hyland information, see the inow.ini file table. Perceptive Content Server Installation and Setup Guide 9 Install Perceptive Content Server on Linux Important

“Active Listening” Lost Art or Learnable Skill?€¦ · 1.Attentive listening is a perceptual, cognitive and social act 2.Attentive listening is an activeprocess 3.Attentive listening

Perceptive Analytics - Creating Compelling Dashboards

Perceptive media

Perceptive media 2016 for wcsfp

Perceptive media thinking for Drama labs

RHYTHM TYPOLOGY: ACOUSTIC AND PERCEPTIVE STUDIES

PSYCHOLOGICAL, PHYSIOLOGICAL, … PHYSIOLOGICAL, PERFORMANCE AND PERCEPTIVE RESPONSES TO BRAZILIAN JIU-JITSU COMBATS ... physiological, performance and perceptive responses

Perceptive Document Filters - VirtualWorksdownload.virtualworks.com/helpdocs/Perceptive+Document+Filters... · Perceptive Document Filters Supported Formats Page 9 of 20 Document

Perceptive futures

Jesus Centred FAITH Jesus Centred THINKING Jesus Centred UNIVERSE.

WHAT’S NEW IN PERCEPTIVE CONTENT (IMAGENOW) 7 › wp-content › uploads › RPI... · WHAT’S NEW IN PERCEPTIVE CONTENT (IMAGENOW) 7.3 Hyland just released Perceptive Content