8/20/2019 Saliency map presentation
1/38
Introduction of Saliency
Map
Presenter: Chien-Chi Chen
Advisor: Jian-Jiun Ding
1
8/20/2019 Saliency map presentation
2/38
Outline
• Introduction of saliency map• Button-up approach
– L. Itti’s approach – Frequency-tuned
– Multi-scale contrast – Depth of eld – Spectral !esidual approach – "lo#al contrast #ased
• $op-do%n approach – &onte't-a%are
• Information ma'imum – Measurin( )isual saliency #y site entropy rate
*
8/20/2019 Saliency map presentation
3/38
Outline
• Introduction of saliency map• Button-up approach
– L. Itti’s approach – Frequency-tuned
– Multi-scale contrast – Depth of eld – Spectral !esidual approach – "lo#al contrast #ased
• $op-do%n approach – &onte't-a%are
• Information ma'imum – Measurin( )isual saliency #y site entropy rate
+
8/20/2019 Saliency map presentation
4/38
Introduction of saliency map
• Lo%-le)el,contrast – &olor
– Orientation
– Sie
– Motion
– Depth
• /i(h-le)el – 0eople
– &onte't
Important
2 3udd etal4
Lo%-
le)el
7ithface
detection
8/20/2019 Saliency map presentation
5/38
Outline
• Introduction of saliency map• Button-up approach
– L. Itti’s approach – Frequency-tuned
– Multi-scale contrast – Depth of eld – Spectral !esidual approach – "lo#al contrast #ased
• $op-do%n approach – &onte't-a%are
• Information ma'imum – Measurin( )isual saliency #y site entropy rate
8
8/20/2019 Saliency map presentation
6/38
L. Itti’s approach
• 9rchitecture: "aussian
0yramids
!4"4B4 ; "a#orpyramids
for θ < =5>428>4 65>41+8>?
8/20/2019 Saliency map presentation
7/38
L. Itti’s approach
• Center-surround Diference• 9chie)e center-surround di@erence throu(h across-scale
di@erence
• Operated denoted #y Θ: Interpolation to ner scale and point-to-point su#traction
• One pyramid for each channel: I(σ), R(σ), G(σ), B(σ), Y(σ)%here σ ∈ A5..C is the scale
8/20/2019 Saliency map presentation
8/38
8/20/2019 Saliency map presentation
9/38
L. Itti’s approach
• Center-surround Diference•Color Feature Maps
!ed-"reen and ;ello%-Blue
Center-surround DiferenceOrientation Feature Maps
•
+R-G
+R-G+G-R
+G-R +B-Y
+Y-B
+Y-B
+B-Y
+B-Y
Same c and s as %ith
intensity
),(),(),,( θ θ θ sOcO scO −=
RG(c, s) = | (R(c) - G(c)) Θ (G( s) - R( s)) | BY (c, s) = | (B(c) - Y(c)) Θ (Y( s) - B( s)) |
8/20/2019 Saliency map presentation
10/38
L. Itti’s approach
• Norali!ation Operator• 0romotes maps %ith fe% stron(
peaHs
• Surpresses maps %ith manycompara#le peaHs
1. ormaliation of map to ran(e A0… M C
*. &ompute a)era(e m of all local ma'ima+. Find the (lo#al ma'imum M
2. Multiply the map #y , M – m*
8/20/2019 Saliency map presentation
11/38
L. Itti’s approach
Inhi"ition o# ret
J'ample ofOperation:
8/20/2019 Saliency map presentation
12/38
Frequency-tuned
1*
Image Average
Gaussian blur
L
I a
b
µ
µ µ
µ
−
( , )hc
hc hc
hc
L
I x y a
b
ω
ω ω
ω
=
( , ) ( , )hc
S x y I I x y µ ω = −
8/20/2019 Saliency map presentation
13/38
Multi-scale contrast
• Saliency al(orithm
Ima(e
Salienc
y map
Multi-scalecontrast
&enter-surroundhisto(ra
m
&olorspatial-distri#uti
on
&onditional!andom
Field
1+
8/20/2019 Saliency map presentation
14/38
Multi-scale contrast
Multi-scale contrast
• Local summation oflaplacian pyramid
Center-surround histogra
• Distance #et%eenhisto(rams of !"B color:
2
1 ( )( , ) || ( ) ( ) ||
L l l c
l x N x f x I I x I x∑ ∑
′= ∈′= − 22 ( )1( , )
2 ( )
i i s s i i
s
R R R R
R R χ ∑ −=
+
* 2
( )( ) arg max ( ( ), ( )) s
R x R x R x R x χ =
*
2 * *
{ | ( )}
( , ) ( ( ), ( ))h xx s x x R x
f x I R x R xω χ ∑ ′′ ′∈
′ ′µ
12
8/20/2019 Saliency map presentation
15/38
Multi-scale contrast
• &olor spatial-distri#ution
Ima(e,!
"B
"MMDistancefrom pi'el '
to ima(ecenter
$he)ariance of&oordinateof pi'el 'and y
( , ) ( | ) (1 ( )) (1 ( )) s xc
f x I p c I V c D c∑µ × − × −
18
8/20/2019 Saliency map presentation
16/38
Multi-scale contrast
• Jner(y term:
• Saliency o#Kect:
1 ,( | ) ( , ) ( , , )
K
k k x x x x k x x
E A I F a I S a a I λ ∑ ∑ ∑ ′′=
= +
( , ), 0
( , ) 1 ( , ), 1
k x
k xk x
f x I a
F a I f x I a
=
= − =
• 0air%isefeature:
,( , , ) | | exp( ) x x x x x xS a a I a a β ′ ′ ′= − × −
, || ||, 2 x x x x I I L !"#m′ ′= −
2 1(2 || || ) x x I I β −
′= < − >
1
8/20/2019 Saliency map presentation
17/38
Multi-scale contrast
• &!F:
• $he deri)ati)e of the lo(-liHelihood%ith respect to
1( | ) exp( ( | )) $ A I E A I
% = −
* arg max log ( | ; )! !
! $ A I
λ λ λ ∑= rr r
k λ
1
8/20/2019 Saliency map presentation
18/38
Depth of eld
• 9s the spread of sin(le lens ree'camera4 more and more lo% depth ofeld,DOF ima(es are captured.
• /o%e)er4 current saliency detectionmethods %orH poorly for the lo% DOFima(es.
1
8/20/2019 Saliency map presentation
19/38
Depth of eld
• 9l(orithm:
16
8/20/2019 Saliency map presentation
20/38
Depth of eld
• &lassication: • Focal 0oint: In a lo%DOF ima(e
DO"
!ectan(le %ith thehi(hest ed(e density4and center is initial
focal point2( , ) ( , )
S i & S i & A' σ −
=
• &omposition
9nalysis:se(mentation !e(ion
1 2 3
#
i
A !
A m# # S S '
σ σ σ − − −
=
*5
8/20/2019 Saliency map presentation
21/38
Spectral !esidual 9pproach
• First scalin( ima(e to 2'2.
• $hen %e smoothed the saliency map%ith a (aussian lter (,' , < .σ
*1
8/20/2019 Saliency map presentation
22/38
"lo#al contrast-#ased
• /isto(ram #ased contrast,La#:
2( )O N 2( ) ( )O N O !+
Nuantiation of La#
Jach channel toha)e 1* di@erent)alue
312 172=
**
8/20/2019 Saliency map presentation
23/38
"lo#al contrast-#ased
• !e(ion #ased contrast – Se(ment the Ima(e
– AJcient (raph-#ased ima(ese(mentationC
*+
8/20/2019 Saliency map presentation
24/38
Outline
• Introduction of saliency map• Button-up approach
– L. Itti’s approach – Frequency-tuned
– &enter-surround – Depth of eld – Spectral !esidual approach – "lo#al contrast #ased
• $op-do%n approach – &onte't-a%are• Information ma'imum
– Measurin( )isual saliency #y site entropy rate
*2
8/20/2019 Saliency map presentation
25/38
&onte't-9%are
• "oal: &on)ey the ima(e content
*8
Liu et al4 *55
8/20/2019 Saliency map presentation
26/38
&onte't-9%are
• Distance #et%een a pair of patches:
( , )( , )
1 ( , )
c"l"# i &
i & p"si(i"! i &
p p p p
c p p=
+ ×
salient
/i(h &∀
8/20/2019 Saliency map presentation
27/38
&onte't-9%are
• Distance #et%een a pair of patches:
/i(h for P mostsimilar
Saliency
k
# ) = P most similar patches atscale #
1
11 exp ( , )
i &
K # # #
i
k
S p ) K =
= − −
∑
8/20/2019 Saliency map presentation
28/38
&onte't-9%are
• Salient at: – Multiple scales fore(round
– Fe% scales #acH(round
1
1
M #
# i i
# #
S S M =
= ∑Scale 1 Scale 2
8/20/2019 Saliency map presentation
29/38
&onte't-9%are
• Foci <
• Include distance map
0!iS >
1 ( ) f"ci
i−
$
iS
( )" 1 ( )i i f"ciS S i= −
8/20/2019 Saliency map presentation
30/38
Outline
• Introduction of saliency map• Button-up approach
– L. Itti’s approach – Frequency-tuned
– &enter-surround – Depth of eld – Spectral !esidual approach – "lo#al contrast #ased
• $op-do%n approach – &onte't-a%are• Information ma'imum
– Measurin( )isual saliency #y site entropy rate
+5
8/20/2019 Saliency map presentation
31/38
Measurin( )isual saliency #ysite entropy rate
+1
1
8/20/2019 Saliency map presentation
32/38
Measurin( )isual saliency #ysite entropy rate
+*
9 fully-connected (raph representation is adopted
for each
*
8/20/2019 Saliency map presentation
33/38
Su#-#and (raphrepresentation
++
8/20/2019 Saliency map presentation
34/38
Su#-#and (raphrepresentation
+2
8/20/2019 Saliency map presentation
35/38
Measurin( )isual saliency #ysite entropy rate
+8
9 random %alH is adopted on each su#-#and (raph. 9nd
Site entropy rate,SJ! is measured the a)era(e informationfrom a node to the other
+
8/20/2019 Saliency map presentation
36/38
$he site entropy rate
•
•
+
i&i&
i& &
$ ω
ω ∑=
, #, ,2
i
i i i& i& & i & & i
*
* * * π ω ω ∑ ∑ >= = =
8/20/2019 Saliency map presentation
37/38
&onclusion
• Ima(e processin( is funny
• Qnusual in its nei(h#orhood %illcorrespond to hi(h saliency %ei(ht
• &ontrast is the Hey of saliency
+
8/20/2019 Saliency map presentation
38/38
!eference
A1C !. 9chanta4 F. Jstrada4 0. 7ils4 and S. SRusstrunH. Salient re(iondetection and se(mentation. In I&S4 pa(es T8. Sprin(er4 *55.2154 21*4 212
A*C !. 9chanta4 S. /emami4 F. Jstrada4 and S. SRusstrunH. Frequency-tuned salient re(ion detection. In &0!4 pa(es 186T1524 *556. 25642154 21*4 21+4 2124 218
A+C L. Itti4 &. Poch4 and J. ie#ur. 9 model of saliency #ased )isualattention for rapid scene analysis. IJJJ $09MI4 *5,11:1*82T1*864166. 2564 2154 21*4 212
A2C U. /ou and L. Vhan(. Saliency detection: 9 spectral residualapproach. In &0!4 pa(es 1T4 *55. 2154 21*4 21+4 212
A8C S. "oferman4 L. VelniH-Manor4 and 9. $al. &onte't-a%are saliencydetection. In &0!4 *515. 2154 21*4 21+4 2124 218
AC MM &hen(4 "U Vhan(4 . 3. Mitra4 U. /uan(4 S.M. /u. "lo#al &ontrast#ased Salient !e(ion Detect. In &0!4 *511 .AC $. Liu4 V. ;uan4 3. Sun4 3.7an(4 . Vhen(4 $. U.4 and S. /.;. Learnin( to
detect a salient o#Kect. IJJJ $09MI4 ++,*:+8+T+4 *511. 215AC 7. 7an(4 ;. 7an(4 N. /uan(4 7. "ao4 Measurin( isaul Saliency #y
Site Jntropy !ate4 In &0!4 *515.
+
Top Related