Post on 28-Dec-2015
2
Pittsburgh Brain Activity Interpretation Competition Inferring Experience Based Cognition from fMRI
Welcome to the presentation of the Competition Overview
This is a short auditory graphical overview of the competition.
For details see, please see our web page at:
http://www.ebc.pitt.edu/competition.html
The guidebook is available at:http://www.ebc.pitt.edu/Competition/PittsburghBrainInterpretationCompetitionGuide2006.pdf
Speaker: Walter Schneider
University of Pittsburgh
3
Goals of Competition
• Advance the understanding of how the brain represents and manipulates information
• Enable scientists from many disciplines and nations to develop new brain interpretation methodologies
• Provide focus, data, and educational materials to expand research in this area
• Give top groups visibility for advances
4
Overview of competition
• Who can compete– People from many disciplines, nations, and type of
position (students, faculty, scientists, engineers)
– Individuals or groups
– Cross disciplinary teams
– Classes
(note only one cash prize per institution)
5
Overview of what to do
• You will be examine the brain activity and feature ratings of 3 people watching 3 twenty minute videos from a TV series
• Develop classifier systems such that for Movie1 and Movie2 predict the related feature rating data
• Apply that classifier to the Movie3 brain activity data to predict the ratings of the same people watching Movie3.
6
Brain Activity and Eye Movement Data Collection
• Data was collected at the Brain Imaging Research Center in Pittsburgh http://www.birc.pitt.edu/
• Data was collected on 3 subjects viewing three videos while collecting eye movement data.
• Rating data was collected on 13 features rated by each subject, plus 7 actor ratings and 7 location ratings and some processing of the video (e.g., RMS amplitude of the sound track).
Auditory
Visual
7
Diagram of VideoRuns
Drawings by: Sue Schneider
Brain activation data 34x64x64x860 (1.75s)
Subject and expert rating data 23x860(1.75 s) after hemodynamic lag
8
Create ROI x TimeTable
(Provided)
Do PostPost Process
Clean-Up
Identify Movie ActiveRegion of Interest
ROIs(Provided)
Submit Data
MovieRun2
Basic Analysis StepLoad
MovieRun2 Sub14D Functional Data
Regress ROI Table to
Feature Hemodynamics
9
Example Linear Prediction Approach
=
For a linear system you can solve for betas by taking inverse
ACTIVITY(ROI,Time)
To predict each feature calculate the betas to linearly predict feature strength from Activation Table (ROI,Time)
n – number of time points
k – number brain areas
Note this approach is meant as an illustration only and can be done as an exercise to learn to work with the data
BETA(ROI)
0 0.2 0.4 0.6 0.8
1
2
3
4
5
6
7
8
FEATURE(Time)
10
Some Example Post processing steps
• Predict new feature Is Movie Playing
• If Not Movie Playing set feature predictions to Not active state.
11
Developing/training techniques• Use the data from Movie1 & Movie2 to
develop the ability to go from the brain activation data to the behavioral rating data.
Drawings by: Sue Schneider
Brain activation data 34x64x64x860 (1.75s)
Movie1
Brain activation data 34x64x64x860 (1.75s)
Movie2
12
Prediction of rating data for Movie3Do not know what scenes were presented
? ? ?? ?
Eye movement x,y, pupil 3x9000 (0.016s)
Optional prediction of eye movement dataDrawings by: Sue Schneider
Use techniques to predict rating data
Subject and expert rating data 23x860(1.75 s) after hemodynamic lag
Brain activation data 34x64x64x860 (1.75s)
13
Example Advanced Analysis
For Top ROIs of Each
Feature
Use Advanced Techniques To Predict
Feature
Do PostPost Process
Clean-Up
Create Local Flat or Structure
Maps Submit Data
Regress Advanced Data to
Feature Hemodynamics
16
Actor
Create Feature Specific Prediction
Example Prediction of individual actor’s speech
Predicted Feature(A1) = Distance(A1) - Average (Distance(A2,A3,A4))
Predict Actor = Max of (A1, A2, A3, A4)
17
Keep this a Collegial Competition
• You might work with multiple people at your site.
• We will be providing resources (e.g., readings, additional formats, routines) that people wish to share.
• If you have a processing step you would like others to try and comment on contribute them.
• We will be doing special events (web conferences) if requested.
• Make suggestions for this and next years competition.
18
What are the Feature Vectors
• Subject rated vectors (all scored in competition)– Content: Body Parts, Environmental Sounds, Faces, Food,
Language, Laughter, Motion, Tools
– Reaction: Amusement, Arousal, Attention, Sadness
• Content questions (optionally scored)– Actors
– Locations
• Quantitiative measurement of content (not scored)– Auditory RMS amplitude
– Visual mean pixel brightness
19
Sub 1 Movie 1 Raw
0
2
4
6
8
10
12
14
16
0 500 1000 1500 2000
Amusement
Attention
Arousal
Body Parts
Environmental Sounds
Faces
Food
Language
Laughter
Motion
Music
Sadness
Tools
Blank
Subject FeaturesRating Data Season 1 Sub 1 Sub 1 Movie 1
0
2
4
6
8
10
12
14
16
0 500 1000 1500 2000
Amusement
Attention
Arousal
Body Parts
Environmental Sounds
Faces
Food
Language
Laughter
Motion
Music
Sadness
Tools
Blank
Hemodynamic Lag Output
20
time [s]0.5÷2 4 10
stimulus
0-0.5
15
% s
igna
l cha
nge
Blood Oxygenation Level Dependent contrast fMRI
Neural pathway Hemodynamics MR scanner
22
Keep in Contact
• We will provide posting on the discussion board and major notices via email.
• We will be posting updates on procedures and corrections of documents.
23
Submit Predicted Movie3 data to competition before May 1, 2006
• Note you are only allowed to submit the Movie3 predictions 3 times.
• Test out the methods on Movie1 and Movie2 data sets.
24
Comment on Selection of Video
The video was selected with the intent to provide a test bed for data mining methods. We used excepts from the Home Improvement TV series video by Disney Studios (http://www.ultimatedisney.com/tvshows.html, http://tvplex.go.com/touchstone/homeimprovement/ ) for a description of the series see http://www.morepower.com/) . You can obtain the source video from local stores or Amazon. COM (http://www.amazon.com/gp/product/B000AJJNIQ/102-8197023-4912152?n=130). The series provided 36 hours of material to choose from fairly well segmented scenes in a common professional video recording. This TV video provided long shots and a repeating use of a small number of actors in a small number of sets that allows common elements to reoccur. Also the materials (character types, settings, events, objects) are typical of what the subjects would be expected to have experience with (relative to weapons in action Videos). The specific scenes were selected through a series of rating efforts to rank order the clips that presented the categories uniquely and in combination and selected to make Movie1, Movie2, Movie3
Home Improvement Season 1
Home Improvement Season 2
Home Improvement Season 3
25
Example functional data sets
DICOM slices Extracted Brain Inflated Brain Flat Mapped Brain
Raw Data Pre-processed data
We provide multiple formats to minimize the start up time and allow those without specific brain imaging experience to get the benefits from experts on the preprocessing stages. For example, some computer science data-mining students that may enter the competition may have no brain imaging experience. Given that, we will make available the data preprocessed (and raw) so they do not have to create software for all the data handling. We will also provide raw data allowing participants to apply whatever methods they wish to analyze the data.
DICOM, Analyze or Brain Voyager formats provided
26
Suggested Entry Methods• We recommend that you start by predicting a single feature
(e.g., faces) on a single subject (subject 1) on a given movie (Movie2), developing methods and then add additional features.
• Test the methods developed on Movie2 at predicting Movie1 data improving your methods.
• Notes– You can submit entries with subsets for the features predicted (though the
accuracy of others will treated as zero).
– Teams can divide features and submit as a joint effort (e.g., each member work on 4 features but submit a combined effort). Note: the team will need to create a joint document detailing the methods and determine how to share a prize if awarded.
– Watch the Competition Message Board for updates of materials, readings, and suggestions.
28
Credit & Contact• For questions, e-mail ebc@pitt.edu
• This research is being managed by the Experience Based Cognition project at the University of Pittsburgh. Grant number: N00014-05-1-0881
• Experience Based Cognition team members: – PI: Walter Schneider
Co-PIs: Greg Siegle, Mark Wheeler and Kwan-Jin JungUniversity of Pittsburgh
– Rainer Goebel and Elia Formisano Maastricht University, Netherlands
– Rob GoldbergUniversity of Pennsylvania
– Tom Landauer and Peter Foltz Pearson Knowledge Technologies
– Daniel LevinVanderbilt University
• Pittsburgh Technical staff: Maureen McHugo, Ikumi Suzuki, Lori Koerbel, Lena Gemmer, Kate Fissell, Dan Jones, Sharanya Nama
Presentation includes images from Elia Formisano & Rainer Goebel Maastricht University, Netherlands