October 22, 20151 EFFECT OF PHYSICO- CHEMICAL PROPERTIES OF DRUG ON ABSORPTION By Dr. A. S. Adebayo.
Supplementary Materials for “Understanding Deep Networks...
17
Supplementary Materials for “Understanding Deep Networks via Extremal Perturbations and Smooth Masks” Ruth Fong † University of Oxford * Mandela Patrick † University of Oxford Andrea Vedaldi Facebook AI Reseach Contents 1. Proof for Lemma 1 1 2. Visualizations and Details for Area Constraint and Parameteric Family of Smooth Masks 1 2.1. Differentiability of vecsort .......... 1 3. Comparison Visualizations 2 3.1. Comparison with other visualization methods 2 3.2. Comparison with [2] ............. 2 3.3. Area Growth ................. 2 3.4. Sanity Checks [1] ............... 2 3.5. Pointing Game Examples .......... 2 4. Visualizations of Channel Attribution at Interme- diate Layers 3 List of Figures 1 Area constraint ............... 2 2 Generating smooth masks .......... 2 3 Comparison with other visualization methods 4 4 Comparison with other visualization meth- ods (more examples) ............ 5 5 Comparison with [2] ............ 6 6 Area growth (up to 5%). .......... 7 7 Area growth (up to 10%). .......... 7 8 Area growth (up to 20%). .......... 8 9 Area growth (up to 40+%). ......... 9 10 Cascading randomization sanity check (ours). 10 11 Independent randomization sanity check (ours). .................... 11 12 Cascading randomization sanity check for “jungo’ (comparison). ............ 12 13 Independent randomization sanity check for “jungo’ (comparison). .......... 13 14 Pointing Examples from PASCAL ..... 14 15 Pointing Examples from COCO ...... 14 * Work done as a contractor in FAIR. † equal contributions. 16 Per-instance channel attribution visualiza- tion (1-14) .................. 15 17 Per-instance channel attribution visualiza- tion (15-28) ................. 16 18 Per-instance channel attribution visualiza- tion (29-42) ................. 17 1. Proof for Lemma 1 Proof. For the first property m(u) = max v∈Ω k(u - v)¯ m(v) ≥ k(u - u)¯ m(u) = ¯ m(u). For the second property, if k is Lipschitz continuous with constant K, then |k(u) - k(u 0 )|≤ Kku - u 0 k. We whish to bound |m(u) - m(v)|. Assuming without loss of generality that m(u) ≥ m(v), we have |m(u) - m(v)| = m(u) - m(v)= max v∈Ω k(u - v)¯ m(v) - max v 0 ∈Ω k(u 0 - v 0 )¯ m(v 0 ) ≤ max v∈Ω (k(u - v)¯ m(v) - k(u - v 0 )¯ m(v)) ≤ max v∈Ω m(v) ·|k(u - v) - k(u - v 0 )|≤ max v∈Ω m(v) · Kku 0 - uk≤ Kku 0 - uk. 2. Visualizations and Details for Area Con- straint and Parameteric Family of Smooth Masks We have included a few illustrations of our techni- cal innovations. Figure 1 visualizes our novel area con- straint loss. Figure 2 demonstrates how we use the max- convolution in practice to generate smooth masks. 2.1. Differentiability of vecsort The vecsort function is differentiable for our purposes. We use the PyTorch sort function, which returns a tensor of sorted values and a tensor of sorted indices; the sorted values are differentiable but the indices are not. The back- ward function for the sorted values tensor simply “unsorts” 1
Transcript of Supplementary Materials for “Understanding Deep Networks...
![Page 1: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/1.jpg)
Supplementary Materials for “Understanding Deep Networks via ExtremalPerturbations and Smooth Masks”
Ruth Fong†
University of Oxford∗Mandela Patrick†
University of OxfordAndrea Vedaldi
Facebook AI Reseach
Contents
1. Proof for Lemma 1 1
2. Visualizations and Details for Area Constraintand Parameteric Family of Smooth Masks 12.1. Differentiability of vecsort . . . . . . . . . . 1
3. Comparison Visualizations 23.1. Comparison with other visualization methods 23.2. Comparison with [2] . . . . . . . . . . . . . 23.3. Area Growth . . . . . . . . . . . . . . . . . 23.4. Sanity Checks [1] . . . . . . . . . . . . . . . 23.5. Pointing Game Examples . . . . . . . . . . 2
4. Visualizations of Channel Attribution at Interme-diate Layers 3
List of Figures1 Area constraint . . . . . . . . . . . . . . . 22 Generating smooth masks . . . . . . . . . . 23 Comparison with other visualization methods 44 Comparison with other visualization meth-
ods (more examples) . . . . . . . . . . . . 55 Comparison with [2] . . . . . . . . . . . . 66 Area growth (up to 5%). . . . . . . . . . . 77 Area growth (up to 10%). . . . . . . . . . . 78 Area growth (up to 20%). . . . . . . . . . . 89 Area growth (up to 40+%). . . . . . . . . . 910 Cascading randomization sanity check (ours). 1011 Independent randomization sanity check
(ours). . . . . . . . . . . . . . . . . . . . . 1112 Cascading randomization sanity check for
“jungo’ (comparison). . . . . . . . . . . . . 1213 Independent randomization sanity check
for “jungo’ (comparison). . . . . . . . . . . 1314 Pointing Examples from PASCAL . . . . . 1415 Pointing Examples from COCO . . . . . . 14
∗Work done as a contractor in FAIR. † equal contributions.
16 Per-instance channel attribution visualiza-tion (1-14) . . . . . . . . . . . . . . . . . . 15
17 Per-instance channel attribution visualiza-tion (15-28) . . . . . . . . . . . . . . . . . 16
18 Per-instance channel attribution visualiza-tion (29-42) . . . . . . . . . . . . . . . . . 17
1. Proof for Lemma 1Proof. For the first property m(u) = maxv∈Ω k(u −v)m(v) ≥ k(u − u)m(u) = m(u). For the secondproperty, if k is Lipschitz continuous with constant K,then |k(u) − k(u′)| ≤ K‖u − u′‖. We whish to bound|m(u) −m(v)|. Assuming without loss of generality thatm(u) ≥m(v), we have
|m(u)−m(v)| = m(u)−m(v) =
maxv∈Ω
k(u− v)m(v)−maxv′∈Ω
k(u′ − v′)m(v′) ≤
maxv∈Ω
(k(u− v)m(v)− k(u− v′)m(v)) ≤
maxv∈Ω
m(v) · |k(u− v)− k(u− v′)| ≤
maxv∈Ω
m(v) ·K‖u′ − u‖ ≤ K‖u′ − u‖.
2. Visualizations and Details for Area Con-straint and Parameteric Family of SmoothMasks
We have included a few illustrations of our techni-cal innovations. Figure 1 visualizes our novel area con-straint loss. Figure 2 demonstrates how we use the max-convolution in practice to generate smooth masks.
2.1. Differentiability of vecsort
The vecsort function is differentiable for our purposes.We use the PyTorch sort function, which returns a tensorof sorted values and a tensor of sorted indices; the sortedvalues are differentiable but the indices are not. The back-ward function for the sorted values tensor simply “unsorts”
1
![Page 2: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/2.jpg)
0% 20% 40% 60% 80% 100%Mask Values Sorted in Ascending Order
(Percentage of Image as Area)
0.0
0.2
0.4
0.6
0.8
1.0M
ask
Valu
e
Figure 1: Area constraint. The blue line is the referencevector ra representing the target area a as a proportion of“on” versus “off” pixels (here a = 20%). The red dot-ted lines represent mask values in ascending, sorted order,vecsort(m), at different points of the optimization, staringby initializing m to 1/2. Over time, the optimization en-courages the mask to match the target area. Even in thelimit, there is a slight difference between ra as m as thelatter is forced to be smooth at the boundaries; this has thefurther beneficial effect of minimizing the boundary length,encouraging more compact masks.
A. Low-Res Mask Parameterization B. Expand to High-Res, Smooth Gaussians C. Apply Smooth Max Operator
Figure 2: Generating smooth masks. a. We first definea low resolution parameterization mask, m ∈ [0, 1]H×W
(shown as a binary mask in this illustration). b. We thenrepresent each location in m as a high resolution, smoothGaussian, weighted by the value in m. c. Finally, we applythe smooth max operator across all locations to get our finalmask.
the gradient being backpropagated. That is, for functionsorted, indices = torch.sort(x), the backward function is asfollows below; we only use the sorted values in our areaconstraint (Figure 1a).
def s o r t b a c k w a r d ( c tx , g r a d o u t p u t ) :# Get i n d i c e s f o r s o r t e d t o u n s o r t e d .
, i n d i c e s = t o r c h . s o r t ( c t x . i n d i c e s )re turn g r a d o u t p u t [ i n d i c e s ] , None
3. Comparison Visualizations3.1. Comparison with other visualization methods
Figure 3 and fig. 4 show comparisons between ourmethod and other popular visualization methods, such as
gradient [5], guided backprop [6], Grad-CAM [4], andRISE [3].
3.2. Comparison with [2]
Figure 5 show more examples comparing our methodwith [2]. We compare our masks generated using VGG16trained in PyTorch against their masks generated usingGoogLeNet trained in Caffe, as that is what their methodwas optimized for. One can see in fig. 5 that [2]’s masksconsistently are one connected component and are unableto identify multiple, distinct objects of the same class. Onecan also note the instability of [2] in some of their learnedmask, most likely in part due to the tradeoff between theirregularization terms (i.e., L1 sparsity term and total varia-tion smoothness term).
3.3. Area Growth
The following figures include area growth pro-gressions for the first 50 validation images in Ima-geNet: fig. 6, fig. 7, fig. 8, fig. 9. The bar graph visualizesΦ(ma x) as a normalized fraction of Φ0 = Φ(x) (i.e.,class score on original image) and Φmin = Φ(m0x) (i.e.,class score on fully perturbed image) and saturates after ex-ceeding Φ0 by 25% as follows:
v = max(Φ(ma x)− Φmin
Φ0 − Φmin, 1.25) (1)
3.4. Sanity Checks [1]
Adebayo et al. [1] proposed several “sanity checks” forevaluating the specificity of attribution methods to modelweights. We show qualitative experiments for our methodin fig. 10 and fig. 11 and for other methods in fig. 12and fig. 13 for [1]’s model parameter randomization test.Cascading randomization successively randomizes a mod-els’ learnable weights, from the end of a model (e.g., fc8in VGG16) to the beginning of a model (e.g., conv1 1 inVGG16), whereas independent randomization randomizes asingle layer’s random weights. When randomizing a layer’sweights, we draw from same distribution that we used toinitialize the model during training.
Implementation Details. We use the “hybrid” formula-tion instead of “preservation” formulation used elsewherein the paper. The input images were resized to (224, 224).
3.5. Pointing Game Examples
Figure 14 and fig. 15 show a few examples of our masksbeing discriminative for different classes on PASCAL andCOCO respectively.
![Page 3: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/3.jpg)
4. Visualizations of Channel Attribution at In-termediate Layers
Figure 16, fig. 17, and fig. 18 show saliency heatmapsand feature inversions of our channel attribution masks.By comparing the difference in feature inversions betweenthe original and perturbed activations, we can identify thesalient features that our method highlights. To our knowl-edge, this is the first work that shows side-by-side “diffs” offeature inversions.
References[1] Julius Adebayo, Justin Gilmer, Ian Goodfellow, Moritz Hardt,
and Been Kim. Sanity checks for saliency maps. In Proc.NeurIPS, 2018. 1, 2
[2] Ruth Fong and Andrea Vedaldi. Interpretable explanations ofblack boxes by meaningful perturbation. In Proc. ICCV, 2017.1, 2, 6
[3] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomizedinput sampling for explanation of black-box models. In Proc.BMVC, 2018. 2
[4] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das,Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In Proc. ICCV, 2017. 2
[5] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.Deep inside convolutional networks: Visualising image clas-sification models and saliency maps. In Proc. ICLR workshop,2014. 2
[6] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox,and Martin Riedmiller. Striving for simplicity: The all convo-lutional net. 2015. 2
![Page 4: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/4.jpg)
gree
n liz
ard
Orig Img20%
Mask Gradient Guided Grad CAM RISE
swim
min
g tru
nks
10%
Afgh
an h
ound
20%
car w
heel
20%
cinem
a
40%
Mad
agas
car c
at 10%
pass
enge
r car
20%
Figure 3: Comparison with other visualization methods.
![Page 5: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/5.jpg)
snow
mob
ile
Orig Img10%
Mask Gradient Guided Grad CAM RISE
oil f
ilter
5%
pill
bottl
e
5%
maz
e
60%
may
pole
20%
race
r
10%
dial
tele
phon
e 5%
bloo
dhou
nd
20%
skun
k
5%
Figure 4: Comparison with other visualization methods (more examples).
![Page 6: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/6.jpg)
10%sea snake
40%alp
10%Shetland sheepdog
40%soup bowl
10%garter snake
20%porcupine
20%bakery
5%mousetrap
10%Angora
60%brain coral
20%cougar
20%guenon
20%recreational vehicle
20%harvester
20%grey whale
20%sea anemone
80%vulture
40%carton
20%crane
20%porcupine
10%Ibizan hound
10%Granny Smith
40%planetarium
20%vulture
20%table lamp
20%white wolf
20%English foxhound
20%green snake
20%tile roof
10%cabbage butterfly
20%go-kart
10%sea lion
10%ballplayer
20%half track
20%vestment
5%tailed frog
10%abacus
20%scabbard
10%garden spider
40%rain barrel
20%spoonbill
20%standard schnauzer
10%Newfoundland
40%pitcher
40%freight car
10%English foxhound
10%pickup
5%sturgeon
20%bee eater
40%axolotl
20%switch
40%handkerchief
20%polecat
10%cab
10%Pomeranian
20%stinkhorn
20%tripod
10%hamper
Figure 5: Comparison with [2]. Odd rows: Our masks at the optimal area on the first 60 images in the ImageNet validationsplit. Even rows: [2]’s masks. We can see that our masks are able to pick out distinct object instances, whereas [2]’s masksonly highlight one connected component due to their smoothness regularization. We also see qualitatively that our method ismore stable than [2]’s (where sometimes the mask is completely off the object and/or has an unnatural speckled nature).
![Page 7: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/7.jpg)
stur
geon
5%
mou
setra
p
5%
taile
d fro
g 5%
Figure 6: Area growth (up to 5%).
sea
snak
e
5% 10%
garte
r sna
ke
5% 10%
Ango
ra
5% 10%ab
acus
5% 10%
sea
lion
5% 10%
Shet
land
shee
pdog 5% 10%
pick
up
5% 10%
ballp
laye
r
5% 10%
Ibiza
n ho
und
5% 10%
Newf
ound
land
5% 10%
cabb
age
butte
rfly 5% 10%
Shet
land
shee
pdog 5% 10%
gard
en sp
ider
5% 10%
Figure 7: Area growth (up to 10%).
![Page 8: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/8.jpg)
porc
upin
e
5% 10% 20%
guen
on
5% 10% 20%
recr
eatio
nal v
ehicl
e 5% 10% 20%
harv
este
r
5% 10% 20%
sea
anem
one
5% 10% 20%
cran
e
5% 10% 20%
porc
upin
e
5% 10% 20%
vest
men
t
5% 10% 20%
spoo
nbill
5% 10% 20%
half
track
5% 10% 20%
grey
wha
le
5% 10% 20%
coug
ar
5% 10% 20%
tile
roof
5% 10% 20%
stan
dard
schn
auze
r 5% 10% 20%
go-k
art
5% 10% 20%
bake
ry
5% 10% 20%
vultu
re
5% 10% 20%
Engl
ish fo
xhou
nd
5% 10% 20%
white
wol
f
5% 10% 20%
gree
n sn
ake
5% 10% 20%
tabl
e la
mp
5% 10% 20%
scab
bard 5% 10% 20%
Figure 8: Area growth (up to 20%).
![Page 9: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/9.jpg)
alp
5% 10% 20% 40%
soup
bow
l
5% 10% 20% 40%
rain
bar
rel
5% 10% 20% 40%
pitc
her
5% 10% 20% 40%
freig
ht c
ar
5% 10% 20% 40%
vultu
re
5% 10% 20% 40% 60% 80%
Figure 9: Area growth (up to 40+%).
![Page 10: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/10.jpg)
junco5%or
igfc
8fc
7fc
6co
nv5_
1co
nv4_
1co
nv3_
1co
nv2_
1co
nv1_
1
junco10%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
junco20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
corn5%or
igfc
8fc
7fc
6co
nv5_
1co
nv4_
1co
nv3_
1co
nv2_
1co
nv1_
1
corn10%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
corn20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Irish terrier5%or
igfc
8fc
7fc
6co
nv5_
1co
nv4_
1co
nv3_
1co
nv2_
1co
nv1_
1
Irish terrier10%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Irish terrier20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Figu
re10
:C
asca
ding
rand
omiz
atio
nsa
nity
chec
k(o
urs)
.L
ayer
sw
ithle
arna
ble
wei
ghts
are
succ
essi
vely
re-i
nitia
lized
,sta
rtin
gfr
omfc
8to
conv
11.a∗
=20
%,1
0%,a
nd5%
for
“jun
go,”
“cor
n,”
and
“Iri
shte
rrie
r”re
spec
tivel
y.T
hevi
sual
izat
ions
look
rand
omst
artin
gat
conv
51
orco
nv4
1.B
ecau
seth
em
axim
ally
activ
ated
outp
utne
uron
isbe
ing
optim
ized
,whe
non
lyFC
laye
rsar
era
ndom
ized
,the
outp
utne
uron
effe
ctiv
ely
code
sfo
ra
rand
omro
tatio
non
top
ofco
nv5
3(w
hich
still
enco
des
obje
ctch
arac
teri
stic
s).
![Page 11: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/11.jpg)
junco5%or
igfc
8fc
7fc
6co
nv5_
1co
nv4_
1co
nv3_
1co
nv2_
1co
nv1_
1
junco10%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
junco20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
corn5%or
igfc
8fc
7fc
6co
nv5_
1co
nv4_
1co
nv3_
1co
nv2_
1co
nv1_
1
corn10%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
corn20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Irish terrier5%or
igfc
8fc
7fc
6co
nv5_
1co
nv4_
1co
nv3_
1co
nv2_
1co
nv1_
1
Irish terrier10%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Irish terrier20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Figu
re11
:In
depe
nden
tra
ndom
izat
ion
sani
tych
eck
(our
s).
Asi
ngle
laye
r’s
lear
nabl
ew
eigh
tsar
ere
-ini
tializ
ed(a
sop
pose
dto
prog
ress
ivel
yra
ndom
izin
gw
eigh
tsas
infig
.10)
.a∗
=20
%,1
0%,a
nd5%
for“
jung
o,”
“cor
n,”
and
“Iri
shte
rrie
r”re
spec
tivel
y.T
hevi
sual
izat
ions
look
rand
omar
ound
fc6
orco
nv5
1.
![Page 12: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/12.jpg)
Grad
orig
fc8
fc7
fc6
conv
5_3
conv
5_2
conv
5_1
conv
4_3
conv
4_2
DeConv Guided G-CAM MWP c-MWP RISE junco20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Figu
re12
:Cas
cadi
ngra
ndom
izat
ion
sani
tych
eck
for
“jun
go”
(com
pari
son)
.Bot
tom
row
:our
extr
emal
pert
urba
tions
.
![Page 13: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/13.jpg)
Grad
orig
fc8
fc7
fc6
conv
5_3
conv
5_2
conv
5_1
conv
4_3
conv
4_2
DeConv Guided G-CAM MWP c-MWP RISE junco20%
orig
fc8
fc7
fc6
conv
5_1
conv
4_1
conv
3_1
conv
2_1
conv
1_1
Figu
re13
:Ind
epen
dent
rand
omiz
atio
nsa
nity
chec
kfo
r“j
ungo
”(c
ompa
riso
n).B
otto
mro
w:o
urex
trem
alpe
rtur
batio
ns.
![Page 14: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/14.jpg)
cow
pers
on
pers
on
mot
orbi
ke
boat
pers
on
pers
on
tvm
onito
r
chai
r
dini
ngta
ble
chai
r
pers
on
tvm
onito
r
chai
r
dini
ngta
ble
pers
on
Figure 14: Pointing Examples from PASCAL. Here wesee a few examples of where our method is able to local-ize well to different objects in the image. The overlaid vi-sualization is the average mask after Gaussian smoothing(σ = 20).
benc
h
elep
hant
boat
cat
pers
on
surfb
oard
frisb
ee
pers
on
car
stop
sign
truck
bus
car
truck
car
park
ing
met
er
truck
Figure 15: Pointing Examples from COCO. The overlaidvisualization is the average mask after Gaussian smoothing(σ = 20).
![Page 15: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/15.jpg)
chai
n m
ail
10 channels
skun
k
80 channels
Italia
n gr
eyho
und
1 channels
stee
l dru
m
10 channels
cass
ette
pla
yer
20 channels
junc
o
20 channels
park
ing
met
er
1 channels
hone
ycom
b
1 channels
blue
tick
1 channels
meg
alith
50 channels
gold
finch
10 channels
barn
spid
er
10 channels
coug
ar
1 channels
Figure 16: Per-instance channel attribution visualization (1-14). Left: input image overlaid with channel saliency map(see eq. 13 in main paper). Middle: feature inversion of original activation tensor. Right: feature inversion of activationtensor perturbed by optimal channel mask ma∗ . By comparing the difference in feature inversions between un-perturbed(middle) and perturbed activations (right), we can identify the salient features that our method highlights. Notable examplesare the crib (3rd row, left) and snakes (1st row, left and 3rd row, right). These are visualizations for the ImageNet validationimages 1-14.
![Page 16: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/16.jpg)
fireb
oat
100 channels
stin
gray
80 channels
med
icine
che
st
1 channels
abay
a
25 channels
quill
10 channels
elec
tric
fan
40 channels
junc
o
20 channels
Lhas
a
10 channels
lace
wing
1 channels
padd
le
5 channels
abay
a
10 channels
leaf
bee
tle
1 channels
echi
dna
10 channels
gold
en re
triev
er
1 channels
Figure 17: Per-instance channel attribution visualization (15-28). Left: input image overlaid with channel saliency map.Middle: feature inversion of original activation tensor. Right: feature inversion of activation tensor perturbed by optimalchannel mask ma∗ . Notable examples are the harvester (1st row, left), porcupines (4th row, left), and the bird (2nd to lastrow, left). These are visualizations for the ImageNet validation images 15-28.
![Page 17: Supplementary Materials for “Understanding Deep Networks ...openaccess.thecvf.com/content_ICCV_2019/... · ;1:25) (1) 3.4. Sanity Checks [1] Adebayo et al. [1] proposed several](https://reader034.fdocuments.in/reader034/viewer/2022042222/5ec8dacfdeb7a255c451243b/html5/thumbnails/17.jpg)
cash
mac
hine
1 channels
scho
oner
1 channels
mat
chst
ick
90 channels
Afric
an h
untin
g do
g 1 channels
indi
go b
untin
g
10 channels
bana
na
1 channels
mal
amut
e
35 channels
shop
ping
bas
ket
5 channels
cloak
1 channels
elec
tric
loco
mot
ive 1 channels
soup
bow
l
1 channels
iron
1 channels
trifle
25 channels
Figure 18: Per-instance channel attribution visualization (29-42). Left: input image overlaid with channel saliency map.Middle: feature inversion of original activation tensor. Right: feature inversion of activation tensor perturbed by optimalchannel mask ma∗ . Notable examples are the baseball player (3rd row, right), abacus (3rd to last row, right), crib (last row,left), and rain barrel (last row, right). These are visualizations for the ImageNet validation images 29-42.
