Depth Matters: Influence of Depth Depth Matters: Influence of Depth Cues onCues on

Visual SaliencyVisual Saliency

Congyan Lang , Tam V. Nguyen, Harish Katti , Karthik Yadati ,Mohan Kankanhalli , and Shuicheng Yan

Todays’ Presenter : Daniel Segal

Computer Vision – ECCV 201212th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II

Congyan Lang Tam V. Nguye Harish Katti

AuthorsAuthors

Problem motivation

Challenges

Related work

Proposed solution

Pros & Cons

Limitations

Future ideas

Seminar outlineSeminar outline

Problem presentation

Does depth affects saliency?

If so How to incorporate depth data?

Problem motivationProblem motivation

Why saliency?SurveillanceSearch and RescueMedical applicationImage automated croppingAdvertisingTarget recognitionVideo summarizationsPreprocess for other algorithms

Problem motivationProblem motivation

Why incorporate depth?

Are observers fixation different when viewing 3D images?

Human natural visual attention evolved in 3D environment

Absent of 3D fixation data

Problem motivationProblem motivation

Finding efficient saliency models

Difficult to model top-down processes effect

Integrating depth additional information

Absent datasets for 3D stimulus

Absent of images dataset with corresponding depth maps

Subjectivel

ChallengesChallenges

Which one is more conspicuous?

Computing Visual Attention from Scene Depth (2000) in ICPRAuthors :Authors :Nabil Ouerhani and Heinz Hiigli

Related workRelated work

Intro

Based on Itti & Koch saliency algorithm

Easy to compute in parallel

Easy to incorporate depth features

Only visual evaluation method

Top-Down overview

VideoCamera

Range Finder

Feature Map 1

Feature Map i

Feature Map n+m

Center Surround

Center Surround

Center Surround

Conspicuity Map 1

Conspicuity Map i

Conspicuity Map n+m

Feature extraction Integration

SaliencyMap

Assign weights that promotes conspicuity

Σ Wi

M

m

Wi

Conspicuity map i

Integration

step

Limitation

No statistical analysis to prove

improvements

No comparison with other methods

No quantities evaluation method

Proposed solution advantages

A new 3D dataset was created for statitical analysis

Evaluation and comparison with different methods

Experiments to investigate 3D saliency

Related workRelated workPre-Attentive detection of depth saliency using stereo vision(2010) in AIPRAuthors :Authors :M. Zaheer Aziz and Barbel Mertsching

Intro

Depth approximation using stereo images

Relates only to depth saliency

Top-Down overview

Algorithm

IR

IL

Depth saliency magnitude

Preprocessing

IR

IL

Depth saliency magnitude

Clipp

ing

Smooth

ing

Segmentat

ion

1 1 1

1 1 1

1 1 1

1/9

IR

Key insight

IRIL

IR IL

Main Algorithm explained/1

IR

IL

Segmentation

-

Δ(x, y) Remove extra stripes

Main Algorithm explained/2

Remove occluded stripes Assign region depth

Human subjects marked depth saliency

labels

Efficiency factor defined

Capability factor defined

Experiments and results

Nf =# of labels found

Ns =# of labels

Nf

Ns

Nf

Ns

Nf<Ns

Nf>NsPenalty

Experiments and results

Related workRelated work

Limitation

Requires stereo images

Approximated depth calculation

Emphasis on runtime

Experiments somehow dubious

Self invented evaluation index

Hard to compare with other methods

Ignoring fusion with contrast saliency algo.

Related workRelated work

Proposed solution advantages

Messured depth data

Statistical analysis and comparison

Applicable on all 2D saliency algorithms

Using conventional evaluation index

Top-Down overview

Dataset collection (3D/2D) and analysisExtracting Stereoscopic image pair generation for 3D display

Perform experiments to gain fixation maps

Observations and Statistics

Incorporating depth priors

Experiments and results

Proposed solutionProposed solution

Experiment setup

Rejected images from dataset images overlapping content with other images Images with significant artifacts after the smoothing process

Dataset Examples

1 1 11 0 11 1 1

D==0 ?

1/8∙

D==0 ?

In the same super pixel

LaplacianSmoothDepth map

Depth Map

Apply smoothing to depth mapsApply smoothing to depth maps

Stereoscopic image pair generation for 3D display

Pixels translation:xl = xp+ρ, xr = xp−ρρ = parallax/2

Experiments conducted usingactive shutter glasses on a3D LCD display

Three quantitative methods for performance evaluation

Correlation coefficient(for all images in dataset)

o Correlation between to given maps for different depth range bins

Saliency Ratio

o Saliency mean energy in depth bin

AUC of the ROC

o Use each one saliency map to predict another

Quantitative evaluation methods/1

Correlation coefficient

Saliency Ratio

AUC of the ROC

Quantitative evaluation methods/2

Observation 1

Depth cues modulate visual saliency at farther depth ranges

humans fixate preferentially at closer depth ranges

Observations and statistics

Observation 2

few objects account for majority of the fixations

The average AUC for the entire 3D fixation dataset is 0.7399and 0.7046 for 2D fixation dataset

Observations and statistics

Using fixation maps as predictor for labeled object

Observations and statistics

Observation 3

The relationship between depth and saliency is non-linear and

characteristic for low and high depth-of-field scenes

CC – Correlation coefficient

Observations and statistics

Observation 4:Fixation distribution discrepancy when multiple salient stimuli present different depth planes

GMM model ParametersFor each DOF bin

P(s(x)|d(x), DOF)

Training set(80% of thedataset)

200x200 pix

Incorporating depth priors/1

EMAlgo.

DownSample

S(x) = ψ(x)( / )P(s/d,DOF)⊕ ⊗

Proposed pdf P(s,d/DOF) is GMM (Gaussian Mixture Model)

640x480 pix

GMM PdfFrom training

Saliency mapSaliency

Algo.

Incorporating depth priors/2

Input image

+/*ImprovedSaliency

map

Depth mapCalcDOF

SelectedGMM Pdf

Experiments and results

Experiments and results

Experiments and results

Proposed solutionProposed solution

Depth saliency

Pros & Cons Pros & Cons

ProsNovel and provides a basis for further researchSimple approachGives an overall improvement for existing alg.

ConsFew missing implementation detailsNo source code/other descriptive materialRequires a depth mapStill no basis for comparison with other methods

Future ideasFuture ideasIntegrate depth priors in various algo. instead of late fusion

Questions ?

Thanks