Processing Megapixel Images with Deep Attention-Sampling ...11-11-00)-11-11-25-4512... ·...
Transcript of Processing Megapixel Images with Deep Attention-Sampling ...11-11-00)-11-11-25-4512... ·...
Processing Megapixel Images withDeep Attention-Sampling Models
Angelos Katharopoulos & Francois Fleuret
ICML, June 11, 2019
Funded by
How do DNNs process large images?
Cropping and downsampling to a manageableresolution (e.g. 224× 224)
Dividing the image into patches and processingthem separately
∗image taken from the Imagenet dataset
A. Katharopoulos Deep Attention-Sampling Models 2/9
Our contributions
I Sample from a soft attention to only process a fraction of the image in highresolution.
I Derive gradients through the sampling for all parameters which allows to trainour models end-to-end.
I Disentangle the computational and memory requirements from the inputresolution.
A. Katharopoulos Deep Attention-Sampling Models 3/9
Soft Attention
Given an input x we define a neural network Ψ(x) that uses attention
Ψ(x) = g
(K∑i=1
a(x)i f (x)i
)= g
(EI∼a(x)[f (x)I ]
),
where f (x) ∈ RK×D are the features and a(x) ∈ RK+ is the attention distribution.
A. Katharopoulos Deep Attention-Sampling Models 4/9
Attention Sampling
We approximate Ψ(x) by Monte Carlo
Ψ(x) ≈ g
1
N
∑q∈Q
f (x)q
whereQ = {qi ∼ a(x) | i ∈ {1, 2, . . . ,N}}.
We show that
I Sampling from the attention is optimal to approximate Ψ(x) if‖f (x)i‖ = ‖f (x)j‖ ∀ i , j
I We can compute the gradients both for the parameters a(·) and f (·)
A. Katharopoulos Deep Attention-Sampling Models 5/9
Processing Megapixel Images with Deep Attention-Sampling Models
A. Katharopoulos Deep Attention-Sampling Models 6/9
Processing Megapixel Images with Deep Attention-Sampling Models
A. Katharopoulos Deep Attention-Sampling Models 6/9
Processing Megapixel Images with Deep Attention-Sampling Models
A. Katharopoulos Deep Attention-Sampling Models 6/9
Processing Megapixel Images with Deep Attention-Sampling Models
A. Katharopoulos Deep Attention-Sampling Models 6/9
Processing Megapixel Images with Deep Attention-Sampling Models
A. Katharopoulos Deep Attention-Sampling Models 6/9
Processing Megapixel Images with Deep Attention-Sampling Models
A. Katharopoulos Deep Attention-Sampling Models 6/9
Qualitative evaluation of the attention distribution (1)
Full Image Epithelial Cells Ilse et al. (2018) Attention Sampling
A. Katharopoulos Deep Attention-Sampling Models 7/9
Qualitative evaluation of the attention distribution (2)
Ground Truth Ilse et al. (2018) Attention Sampling
Extracted patch
A. Katharopoulos Deep Attention-Sampling Models 8/9
Thank you for your time!
Speed limit sign detection
0 500 1000 1500
Memory/sample (MB)
0.10
0.15
0.20
0.25
0.30
Tes
tE
rror
20 40 60 80 100
Time/sample (s)
0.10
0.15
0.20
0.25
0.30
Tes
tE
rror
Come talk to us at poster #3 at Pacific Ballroom.
A. Katharopoulos Deep Attention-Sampling Models 9/9