Neuro–Information Science

Jacek Gwizdka & Michael Cole iSchool @ Rutgers University, NJ, USA

jacek@neuroinfoscience.org

http://jsg.tel

June 5, 2012

Towards

Information Science

 Another IS   Information Science is about : ◦  understanding information seeking behavior (why/how/where/…) ◦  helping people find information they need

2  

Information Systems vs. iScience

 A lot of common concerns and constructs: ◦  information is digital accessed via information systems ◦  technology – task – individual ◦  IT usefulness, user interface design, usability … ◦  trust … ◦  decision making … ◦  affective and cognitive factors ◦  information search (e.g., stopping behavior) …

3  

Information Systems vs. iScience

 Also new opportunities: ◦  neural-correlates of constructs specific to Information Science ◦  Information Relevance : most commonly refers to topical relevance

or aboutness, that is: to what extent the content of a search result matches the topic of the query or a person’s information need (e.g., Saracevic, 2007)   relevance judgment decision making   information stopping

4  

Opportunities for Neuroscience to Inform IS Seven opportunities for cognitive neuroscience to inform IS research: 1.  localize the neural correlates of IS constructs to better understand

their nature and dimensionality; 2.  complement existing sources of IS data with neuroscientific data; 3.  capture hidden (automatic) processes that are difficult to measure

with existing measurement methods; 4.  identify antecedents of IS constructs by exploring the specifics of

how IT stimuli (e.g., the design of graphical user interfaces) are processed by the brain;

5.  test the outcomes of IS constructs by showing how brain activation predicts behavior (e.g., decisions);

6.  infer causality among IS constructs by examining the timing of brain activations due to a common stimulus;

7.  challenge existing IS assumptions and enhance IS theories that do not correspond to the brain’s functionality

(Dimoka et al. 2010)

5  

Neuroscience and Information Science?

 Eye-tracking +++  Galvanic skin response (GSR) ++  Heart –rate variability (HRV) +  EEG   fNIRS   fMRI

Recent and Current Projects

1.  eye-tracking: modeling reading + cognitive effort 2.  fMRI + eye-tracking: information relevance

7  

Part I: Eye-tracking

 General research goal: infer and predict mental states and context of a person engaged in interactive information searching

  Influence system design adaptive systems

Macro   user  task  characteris�cs,  cogni�ve  effort,  domain  knowledge  

Meso   reading  pa�erns  

Micro   eye-­‐gaze  posi�ons  +  �ming  

8  

Eye-movement Presentation

9  

Eye-tracking Data

10  

State2  State1  

State3  

àà Patterns

Eye-movement Patterns

 New methodology to analyze eye-movement patterns ◦ Model reading and Measure cognitive effort ◦ Correlate with higher-level constructs

user task characteristics, user knowledge, etc.

11  

Reading Model Origins

12  

 Based on E-Z Reader model Rayner , Pollatsek, Reichle

◦  Serial reading

◦  Words can be identified in parafovial region

◦  Early lexical access (word familiarity) + Complete lexical processing (word identification)

2o (70px) foveal region parafoveal region

MORE…

Two-State Reading Model

13  

◦  Filter fixations < 150ms (min time required for lexical processing) ◦  Model states characterized by:   probability of transitions; number of lexical fixations; duration   length of eye-movement trajectory, amount of text covered

Scan  Read  

1-q

p

1-p

q

MORE…

isolated fixations fixation

sequences

Example Reading Sequence

14  

Fixation sequence: (F F F) F (F F F) F F F F (F F F F F F) F Reading model states: R S R S S S S R S

Reading state – R | Scanning state – S

Cognitive Effort Measures of Reading

 Reading Speed

  Fixation Regression

 Perceptual Span

  Fixation Duration (“lexical processing excess”)

15  

foveal region

a b c d

Perceptual span = Mean(a,b,c,d)

regression

excess

User Study 1: Cognitive Effort and Tasks

16  

OBI:  advanced  obituary  INT:  interview  prepara�on  CPE:  copy  edi�ng  BIC:  background  informa�on  

N = 32

MORE…

Journalists’ Information Search

 Do the cognitive effort measures correlate with: task difficulty (by design), observable search effort, user’s subjective perception of task difficulty

 Can we detect differences between task characteristics from eye-movement patterns?

Eye-data and Cognitive Effort Measures

Cognitive effort measures derived from eye-tracking

reading speed mean fixation duration perceptual span total fixation regressions

Task difficulty by design Copy Editing (CPE) Advance Obituary (OBI)

Search effort task time pages visited queries entered

Subjective Task Difficulty

CPE        INT      BIC        OBI  

As expected: Copy Editing CPE easiest Advance Obituary OBI most difficult Sig: Kruskal-Wallis χ2 =46.1, p<.0001

17  

Eye-data and Task Characteristics

18  

Measure   Related  Task  Characteris�cs  

Frequency  of  reading  state  transi�ons  

SR bias  to  read   Advanced  obituary  and  Interview  prepara�on  tasks:    search  for  document;  task  goal  not  specific  

RS bias  to  scan   Copy  Edi�ng  task:  search  for  segment  and  task  goal  specific  

Scan  Read  

1-q

p

1-p

q

MORE…

Copy  Edi�ng   Interview  prepara�on

Summary: Eye-tracking Methodology

 Domain independent ◦ Document content is not involved

 Culturally* and individually independent  Real-time modeling of user and tasks is possible  Adaptive systems feasible  Eye-tracking is coming to us!

Tobii

19  

Part II: Current fMRI+eye-tracking Study

  Information Relevance : refers to topical relevance or aboutness, that is: to what extent the content of a document (webpage) matches the topic of the query or a person’s information need (e.g., Saracevic, 2007) ◦ Relevance multi-dimensional: topical, meaningful, useful, trust, affective…

 Neural correlates of topical relevance judgments  Hypothesis ◦ Brain regions that are activated when relevant information is found are

different from regions activated when no relevant info is found and when person does “low-level” visual word search (orthographic matching)   but no hypothesis in a sense where the brain activity is located

 Exploratory research

  (also: a similar experiment with eye-tracking, EEG, GSR) 20  

fMRI + eye-tracking lab   Lab Equipment: ◦  fMRI: 3T Siemens TRIO ◦  eye-tracker: Eyelink-1000   non-ferromagnetic optimized design; up to 2000 Hz sampling rate

21  

fMRI + eye-tracking

22  

fMRI + eye-tracking

Eye-tracking imposes additional constraints on projection (geometry)

projected screen

mirror

eye-tracker

23  

Current Experimental Design   Two blocks (types of tasks, balanced) ◦ WS – word search: find target word in a short news story – press yes/no ◦  IS – information search: find information that answers given question –

press yes/no. Three types of trials: relevant (R), topical (T), irrelevant (I) ◦ TR cycle: 2s

xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek dkdkdkkd kdkddk dkdkdk dkdkdkd kkdkd d d dd d djdj djdjdj rjrjr rjjweje ejejej ejej kek ekeke wej e ejej eje j

xmx ssms nsns snsns jsdjsd ke ekek dkdkdkk kdkddk dkdkdkdkdkdkd kkdkd d rjr jweje ejeje ekeke wej e ejej fjfjf fjfjfjfjf fjfjrjr rreje j

xmx ssms nsns snsns jsdjsd ke ekek dkdkdkkd

kdkddk dkdkdk dkdkdkd kkdkd d rjr jweje ejejej ejej kekekek ekeke wee ejej fjfjf fjfjfjfjf fjfjrjr rreje j

+ target: info

target: info

target: info

xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek kdkddk dkdkdk dkdkdkd kkdkd d d dd d djdj djdjdj rjrjr rjr jweje ejejej ejej kekekek ekeke wej e eej eje j

+ + target: word

21 x

21 x

WS task instruc-

tions

IS task instruc-

tions

+

+ + + + + +

30s 4s 6s 4s 20s max 4s

30s 4s 8s 20s max 4s 20s max 4s 20s max

24  

Planned Analysis

  Two blocks (types of tasks, balanced) ◦ WS – word search: find target word in a short news story ◦  IS – information search: find information that answers given question –

Three types of trials: relevant (R), topical (T), irrelevant (I)

  The main contrasts of interests are: ◦  IS-R - WS ◦  IS-R - IS-T ◦  IS-R - IS-I

25  

A Very, Very Preliminary Analysis

  For one participant, aggregated for all trials in each of two blocks (tasks)

 Word search (WS)

  Information Search – Relevant (IS-R)

26  

Stay Tuned for Results…

Neuro – Information Science

28  

Funding: Google, HP, IMLS (now funded by IMLS CAREER) Collaborators: Drs. Nicholas Belkin, Art Chaovalitwongse (U Wash), Xiangmin Zhang,

Ralf Bierig (Post Doc); PhD students: Michael Cole (co-author), Chang Liu, Jingjing Liu, Irene Lopatovska + many Master and undergraduate students …

Acknowledgements:

Fragen?

29  

More info & contact http://jsg.tel