Eva Mohedano, "Investigating EEG for Saliency and Segmentation Applications in Image Processing"

Investigating EEG for Saliency and Segmentation Applications in Image

Processing

Eva Mohedano

1

CONTENT

1 - Problem statement

2 – Related Work

3 – Local exploration of the image

4 – Experimental set-up

5 – Signal Processing of EEG Signals

6 – Conclusions

2

CONTENT


2 – Related Work




6 – Conclusions

3

1- PROBLEM STATEMENT

Design a system based on a Brain Computer Interface (BCI) wich measureElectroencephalography (EEG) signals to answer the following questions:

BCI Data Processing

Visual stimulus

EEG Signals

2 - Are the EEG signals useful to images segmentation?

1 - Are the EEG signals to compute Saliency Maps?

4



Data Processing



Visual stimulus

BCI

5



Data Processing



Visual stimulus

BCI

6



Data Processing

Visual stimulus

BCI

Motivation:

- New way to compute maps of the atention of the imagebased directly in the reaction of the brain and not in thefeatures of the images (Niebur and Koch (1996) algorithm).

7



- Reduce the user interaction to the minimun expression.

- Measure the brain reaction at local scale of the image.

Motivation:

Data Processing

Visual stimulus

BCI

8

CONTENT


2 – Related Work




6 – Conclusions

9

2- RELATED WORK

2.1 – BCI in image processing applications

The oddball paradigm

10

2- RELATED WORK



P300

11

2- RELATED WORK



P300

• Speed rate around 10Hz

• Usually experiements centered tofind target images not target regions

12

2- RELATED WORK

•8 electrodes placed mainly in theposterior points on the scalp.

• Which is consistent with thediscriminating activity typicallyproduced by a P300 ERP.[Optimising the Number of Channels inEEG-Augmented Image Search. GrahamHealy]

Event-Related Potential

13

2- RELATED WORK

•8 electrodes placed mainly in theposterior points on the scalp.

• Which is consistent with thediscriminating activity typicallyproduced by a P300 ERP.[Optimising the Number of Channels inEEG-Augmented Image Search. GrahamHealy]

Event-Related PotentialHow to present the image to generate and detect this signal?

14

CONTENT


2 – Related Work




6 – Conclusions

15

First Design: Sliding Window

http://www.youtube.com/watch?v=bKTGKVx58Ps

3- LOCAL EXPLORATION OF THE IMAGE

16

http://www.youtube.com/watch?v=bKTGKVx58Ps

CHALLENGE 1

• Eyes movement affect to the EEG signals – Introduce Artifacts to the signal

Opened eyes / Closed eyes. Image from the slides Dr. Ranjith Polusani


17

http://www.slideshare.net/ranjithpolusani/artifacts-in-eeg-final

CHALLENGE 2

• Progressive inspection may not generate a useful reaction in the EEG waves.

- Follow Oddball Paradigm and perform RSVP od the windows

P300

Suggestions meeting Thomas Ward and Nima Bidgely Shamlo :

SNAP - Simulation and Neuroscience Application Platform


18

CHALLENGE 3

•Syncronitzation Problem


19

CHALLENGE 4

•Size of the object / window

Grabcut Dataset – Objects of different size

Suggestions meeting Thomas Ward and Nima Bidgely Shamlo :

To use images with an homogeneus background with a salient object.The number of distractors (windows with background) must be higher than the number of targets (windows with object).


20

CHALLENGE 5

•What am I seeing?


21

CHALLENGE 5

•What am I seeing?


22

CHALLENGES SOLUTIONS

1 - Eyes movement

2 - Progressive inspection

3 - Syncronitzation Problem

5 - What am I seeing?• Is it just noise?• Am I able to detect something?

Display a fixed window on the screen

SNAP to perform a random RSVP

First test with flashes to find ERPS

Real time visualitzation of the signal

4 - Size of the object / window Generate my own dataset


23

CONTENT


2 – Related Work




6 – Conclusions

24

Second Design: Starting from the easiest case

a) Device CalibrationI. Real time visualization of Alpha wavesII. Detecting ERPS

b) Synthetic ImagesI. RSVP synthetic images fitted in the window.

c) RSVP real images

4- EXPERIMENTAL SET-UP

http://www.youtube.com/watch?v=KsgtvQkOElQ&feature=youtu.be

25

a) Device Calibration


Is the device wellconnected?

Is the syncronitzationmethod correct ?

Am I able to detectsomething?

?

26



SIGNAL EXPECTED - Closed eyes – Alpha waves (8-12 Hz)

Closed-eye EEG alpha waves (10-20 channels Pz-Top, Fz-Bottom) extracted from http://blog.grahamhealy.com/

27

http://blog.grahamhealy.com/



SIGNAL OBTAINED

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-40

-20

0

20

40

Time (sec)

Am

plit

ude (

uV

)

5 seconds Closed Eyes

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-40

-20

0

20

40

Time (sec)

Am

plit

ude (

uV

)

5 seconds Opened Eyes

28


Finding ERPS response after a white flash

Presenting a serie of white flashes (2 seconds between the flashes)

SIGNAL EXPECTED: After the flash P100 and a negative peak between 150-200ms


29

SIGNAL OBTAINED: 60 Flashes to get the response.

Channel P100 (ms) N1 (ms)

1 130 320

2 90 210

3 90 210

4 10 220

5 110 220

6 90 210

7 10 220

8 100 22

Mean 80 23

0 100 200 300 400 500 600 700 800 900 1000-15

-10

-5

0

5

10

Time (ms)

Am

plit

ude (

uV

)

Averaged ERP waveform per channel


Finding ERPS response after a white flash


30

CONTENT


2 – Related Work




6 – Conclusions

31

5- SIGNAL PROCESSING OF EEG SIGNALS

1 Target

99 Distractors

100 windows per Image

Data adquisition

32


Preprocessing: Single trial

1 2

3 4

5

7

6

8

One Image

33


Preprocessing: Single trial - PROBLEM

- Signal very noisy

- Single Targets and Single Distractors very similar

34


Preprocessing: Feature

Mean Absolute Amplitude looks different

Energy from 0 to 600ms

96 Distractors

96 Targets

Feature for the window presented

35


Preprocessing: Averaged trials

1 averaged target

99 averaged distractors

Energy from 0-600 ms

36



1 x 32 single repeats


1 x 32 window repeats




Average by 32

1 averaged target


1 averaged target


1 averaged target


SINGLE AVERAGED

37



Problem:

Too few (target) samples for training

1 averaged target


1 averaged target


1 averaged target


38


Bootstrapping







SINGLE

Bootstrapping

1 x 32 averaged target

99 x 1 averaged distractors




99 x 1averaged distractors

AVERAGED

39


Classification

Problem:

Unbalanced dataset for binary classification







AVERAGED

40


Classification

1 x 32 averaged targets






AVERAGED

Subsample

Subsample

Subsample


32 x 1 avgd distractors





AVERAGED

41


Classification







AVERAGED HISTOGRAM

42


Classification







AVERAGED

SVMTRAIN(linear kernel)

Classifier

43


Classification

100 x 32 single repeatsAverage by 32

100 avgd windows

44


Classification


SVMPREDICT

ClassifierAverage by 32

100 avgd samples

45


8 samples feature vectors

Cross validation approach 3 train + 1 test

CONTENT


2 – Related Work




6 – Conclusions

47

48

6- CONCLUSIONS

- Results from sythetic images provide and evidence that BCI devices could beused to located an object into an image.

- Simplicity of the system: Energy value from 8 channels to train SVM withlineal kernel.

Future work

-Study the impact of the number of repetitions.

- Extract better features.

-Analize data from real images.

-Tool to evaluate and compare the EEG mask (ROC, Jaccard index)

Eva Mohedano, "Investigating EEG for Saliency and Segmentation Applications in Image Processing"

Technology

Transcript of Eva Mohedano, "Investigating EEG for Saliency and Segmentation Applications in Image Processing"