![Deep Parametric Continuous Convolutional Neural Networks€¦ · Graph Neural Networks: Graph neural networks (GNNs) [25] are generalizations of neural networks to graph structured](https://static.fdocuments.in/doc/165x107/5f7096c356401635d36dbe30/deep-parametric-continuous-convolutional-neural-networks-graph-neural-networks.jpg)
Practical Black-box Attacks on Deep Neural Networks using ...abhagoji/files/eccv_poster.pdf ·...
1
Practical Black-box Attacks on Deep Neural Networks using Efficient Query Mechanisms Arjun Nitin Bhagoji 1 , Warren He 2 , Bo Li 3 and Dawn Song 2 1 Princeton University 2 University of California, Berkeley 3 University of Illinois, Urbana-Champaign Motivation Adversarial examples for deep neural networks have so far been largely demonstrated in the white-box setting [1,2,3]. References [1] Szegedy et al., Intriguing properties of neural networks, ICLR 2014 [2] Goodfellow et al., Explaining and harnessing adversarial examples, ICLR 2015 [3] Carlini and Wagner, Towards evaluating the robustness of neural networks, IEEE S&P 2016 [4] Liu et al., Delving into transferable adversarial examples, ICLR 2017 [5] Papernot et al., Practical black-box attacks against deep learning systems using adversarial examples, AsiaCCS 2017 [6] Wright and Nocedal, Numerical Optimzation, Springer Science, 1999 [7] Hildebrand, Advanced calculus for applications (Volume 63), Prentice-Hall Englewood Cliffs, NJ, 1962. Conclusion Uses gradient knowledge Target Model Frog White-box access Current black-box attacks use transferability [4,5]. Target Model Local Model Frog? Training Data Attacks… Successful adversarial examples Commercial ML offerings (Google Vision API, Clarifai etc.) provide query access to pre-trained models. Usually no access to training data. Q: Can purely query-based attacks be carried out? Gradient Estimation Query Reduction White-box attacks rely on model gradients with respect to a loss function, e.g. the logit loss [3]: where is the i th logit. Decreasing the loss at the target class, the white-box adversarial example is `(x,y ) = max{φ f (x) i : i 6= T } - φ f (x) T , φ f (x) i x adv = x - ✏ · sign(r x (max(φ(x) i : i 6= T ) - φ(x) T )). Using Finite Differences [4], the gradient of any function can be estimated using queries to function values: FD x (g (x), δ )= g (x + δ e 1 ) - g (x - δ e 1 ) 2δ , ··· , g (x + δ e d ) - g (x - δ e d ) 2δ , where are the canonical basis vectors. The black-box Gradient Estimation based adversarial example is then e i x adv = x - ✏ · sign(FD x (max(φ(x) i : i 6= T ) - φ(x) T , δ )) Issue: Huge number of queries needed for high- dimensional images Results MNIST CIFAR-10 Single-step Iterative White-box Finite Diff. Query-reduced White-box Finite Diff. Query-reduced Attack on Clarifai Moderation API Benign: ‘Drug’ Adversarial: ‘Safe’ Generated using Iterative Gradient Estimation attack Sample adversarial images 0 20 40 60 80 100 0 0.05 0.10.15 0.20.25 0.30.35 0.4 ✏ L 1 0 20 40 60 80 100 0 4 8 12 16 20 24 28 32 ✏ L 1 Figure 2: Effectiveness of various single step black-box attacks on Model A (MNIST) an constrained strategies on Model A Difference-of-means Random-perturbation Finite-difference xent Finite-difference logit Query-reduced PCA-100 logit Transfer Model B FGS xent Transfer Model B FGS logit White-box FGS logit White-box FGS xent Difference-of-means Random-perturbation Finite-difference xent Finite-difference logit Query-reduced PCA-400 logit Transfer Resnet-28-10 FGS xent White-box FGS logit White-box FGS xent Model A (MNIST) Resnet-32 (CIFAR-10) constrained strategies on Resnet-32 Adversarial success (%) Adversarial success (%) Gradient Estimation matches white-box attack success rates! Comparison of attack success rate vs. perturbation magnitude Random grouping: Estimate gradient along directions corresponding to random grouping of dimensions " +ℎ " −ℎ & +ℎ & −ℎ 4 queries, more accurate ( " +ℎ, & +ℎ) ( " −ℎ, & −ℎ) 2 queries, less accurate ( " , & ) For differentiable functions, the estimate of the gradient along direction is equivalent to its projection along that direction. Both query reduction methods utilize this idea. Principal Component based: Estimate gradient along directions corresponding to principal components of data. More effective since PCA minimizes reconstruction error in terms of norm. Caveat: Assumes access to subset of training data Models (Top 2: MNIST Bottom 2: CIFAR-10) White-box Gradient Estimation Gradient Estimation (Query Reduction) FGS FGS (Iter .) FD FD (Iter.) RG PCA RG (Iter.) PCA (Iter.) A 30.1 99.6 29.9 99.7 15.9 23.2 73.8 96.2 B 29.6 98.7 29.3 98.7 17.8 29.0 73.7 93.9 Resnet-32 23.5 100.0 23.0 100.0 19.0 21.0 97.0 81.0 Resnet-28-10 27.6 100.0 28.0 100.0 20.0 23.0 94.0 72.0 Targeted attack success rates v Gradient Estimation attacks achieve high attack success rates comparable with white-box attacks. Query reduction methods reduce the number of queries required to a dimension-independent constant. These attacks are effective even against deployed real-world classifiers. API provides class-wise confidence scores. Aim: Misclassify image of drugs as ‘safe’. L 2 Classified as 3 Classified as ’Frog’
Transcript of Practical Black-box Attacks on Deep Neural Networks using ...abhagoji/files/eccv_poster.pdf ·...
Practical Black-box Attacks on Deep Neural Networks using Efficient Query Mechanisms
Arjun Nitin Bhagoji 1, Warren He 2, Bo Li 3 and Dawn Song 21 Princeton University 2 University of California, Berkeley 3 University of Illinois, Urbana-Champaign
MotivationAdversarial examples for deep neural networks have so far been largely demonstrated in the white-box setting [1,2,3].
References[1] Szegedy et al., Intriguing properties of neural networks, ICLR 2014[2] Goodfellow et al., Explaining and harnessing adversarial examples, ICLR 2015 [3] Carlini and Wagner, Towards evaluating the robustness of neural networks, IEEE S&P 2016[4] Liu et al., Delving into transferable adversarial examples, ICLR 2017
[5] Papernot et al., Practical black-box attacks against deep learning systems using adversarial examples, AsiaCCS 2017[6] Wright and Nocedal, Numerical Optimzation, Springer Science, 1999[7] Hildebrand, Advanced calculus for applications (Volume 63), Prentice-Hall Englewood Cliffs, NJ, 1962.
Conclusion
Uses gradient knowledge
Target Model Frog
White-box access
Current black-box attacks use transferability [4,5].
Target Model
Local Model
Frog?Training
Data
Attacks… Successful adversarial examples
Commercial ML offerings (Google Vision API, Clarifaietc.) provide query access to pre-trained models. Usually no access to training data.
Q: Can purely query-based attacks be carried out?
Gradient Estimation Query ReductionWhite-box attacks rely on model gradients with respect to a loss function, e.g. the logit loss [3]:
where is the ith logit. Decreasing the loss at the target class, the white-box adversarial example is
`(x, y) = max{�f (x)i : i 6= T}� �f (x)T ,<latexit sha1_base64="i8vJIAGFfD6YQ+FhqK/cxsPTqzU=">AAADtXicfZJda9swFIbVeB9d95V2l7sRC4UWsmCXQUuhUJabXXaQpGaRl8my3IhKtmfJiY3wr9mv2e12tX8zWclgsdkOGB2f5331ecKMM6lc99dez3nw8NHj/ScHT589f/Gyf3g0k2mREzolKU9zP8SScpbQqWKKUz/LKRYhp7fh/bjhtyuaS5YmE1VlNBD4LmExI1iZ0qJ/hSjnJyiMy2F1eoUELpE2f58+x7Z4umCXkEGU0K9wguq3O2gyXPQH7ujCbQJ2E29kR3cAtnGzOOx9QVFKCkETRTiWcu65mQo0zhUjnNYHqJA0w+Qe39G5SRMsqAy0PWcNj00lgnGamy9R0Fb/dmgsZENkDTdxDENuUJjiPIICqyWUlQhTLtsutRRDaMYG123YGFs7U/FFoFmSFYomZLOxuOBQpbC5ZBixnBLFK5NgkjNzNkiWOMdEmafYmT4UZuaErkkqBE4ijWb13As0atYMYz3w6hYP43FtVHrcBYUFRRdMLZh2wdqCdResLFh1gW+B3wWlBWUXVBZUbVDiqLa+hUaKlkrjyCzXEvlbkf8/kWnJRiQ0ypasrk1P/mk8+O9kdjby3JH38d3g+v22O/fBa/AGnAAPnINr8AHcgCkg4Bv4Dn6An865EziRE2+kvb2t5xXYCSf9DSl/QCc=</latexit><latexit sha1_base64="i8vJIAGFfD6YQ+FhqK/cxsPTqzU=">AAADtXicfZJda9swFIbVeB9d95V2l7sRC4UWsmCXQUuhUJabXXaQpGaRl8my3IhKtmfJiY3wr9mv2e12tX8zWclgsdkOGB2f5331ecKMM6lc99dez3nw8NHj/ScHT589f/Gyf3g0k2mREzolKU9zP8SScpbQqWKKUz/LKRYhp7fh/bjhtyuaS5YmE1VlNBD4LmExI1iZ0qJ/hSjnJyiMy2F1eoUELpE2f58+x7Z4umCXkEGU0K9wguq3O2gyXPQH7ujCbQJ2E29kR3cAtnGzOOx9QVFKCkETRTiWcu65mQo0zhUjnNYHqJA0w+Qe39G5SRMsqAy0PWcNj00lgnGamy9R0Fb/dmgsZENkDTdxDENuUJjiPIICqyWUlQhTLtsutRRDaMYG123YGFs7U/FFoFmSFYomZLOxuOBQpbC5ZBixnBLFK5NgkjNzNkiWOMdEmafYmT4UZuaErkkqBE4ijWb13As0atYMYz3w6hYP43FtVHrcBYUFRRdMLZh2wdqCdResLFh1gW+B3wWlBWUXVBZUbVDiqLa+hUaKlkrjyCzXEvlbkf8/kWnJRiQ0ypasrk1P/mk8+O9kdjby3JH38d3g+v22O/fBa/AGnAAPnINr8AHcgCkg4Bv4Dn6An865EziRE2+kvb2t5xXYCSf9DSl/QCc=</latexit><latexit sha1_base64="i8vJIAGFfD6YQ+FhqK/cxsPTqzU=">AAADtXicfZJda9swFIbVeB9d95V2l7sRC4UWsmCXQUuhUJabXXaQpGaRl8my3IhKtmfJiY3wr9mv2e12tX8zWclgsdkOGB2f5331ecKMM6lc99dez3nw8NHj/ScHT589f/Gyf3g0k2mREzolKU9zP8SScpbQqWKKUz/LKRYhp7fh/bjhtyuaS5YmE1VlNBD4LmExI1iZ0qJ/hSjnJyiMy2F1eoUELpE2f58+x7Z4umCXkEGU0K9wguq3O2gyXPQH7ujCbQJ2E29kR3cAtnGzOOx9QVFKCkETRTiWcu65mQo0zhUjnNYHqJA0w+Qe39G5SRMsqAy0PWcNj00lgnGamy9R0Fb/dmgsZENkDTdxDENuUJjiPIICqyWUlQhTLtsutRRDaMYG123YGFs7U/FFoFmSFYomZLOxuOBQpbC5ZBixnBLFK5NgkjNzNkiWOMdEmafYmT4UZuaErkkqBE4ijWb13As0atYMYz3w6hYP43FtVHrcBYUFRRdMLZh2wdqCdResLFh1gW+B3wWlBWUXVBZUbVDiqLa+hUaKlkrjyCzXEvlbkf8/kWnJRiQ0ypasrk1P/mk8+O9kdjby3JH38d3g+v22O/fBa/AGnAAPnINr8AHcgCkg4Bv4Dn6An865EziRE2+kvb2t5xXYCSf9DSl/QCc=</latexit><latexit sha1_base64="i8vJIAGFfD6YQ+FhqK/cxsPTqzU=">AAADtXicfZJda9swFIbVeB9d95V2l7sRC4UWsmCXQUuhUJabXXaQpGaRl8my3IhKtmfJiY3wr9mv2e12tX8zWclgsdkOGB2f5331ecKMM6lc99dez3nw8NHj/ScHT589f/Gyf3g0k2mREzolKU9zP8SScpbQqWKKUz/LKRYhp7fh/bjhtyuaS5YmE1VlNBD4LmExI1iZ0qJ/hSjnJyiMy2F1eoUELpE2f58+x7Z4umCXkEGU0K9wguq3O2gyXPQH7ujCbQJ2E29kR3cAtnGzOOx9QVFKCkETRTiWcu65mQo0zhUjnNYHqJA0w+Qe39G5SRMsqAy0PWcNj00lgnGamy9R0Fb/dmgsZENkDTdxDENuUJjiPIICqyWUlQhTLtsutRRDaMYG123YGFs7U/FFoFmSFYomZLOxuOBQpbC5ZBixnBLFK5NgkjNzNkiWOMdEmafYmT4UZuaErkkqBE4ijWb13As0atYMYz3w6hYP43FtVHrcBYUFRRdMLZh2wdqCdResLFh1gW+B3wWlBWUXVBZUbVDiqLa+hUaKlkrjyCzXEvlbkf8/kWnJRiQ0ypasrk1P/mk8+O9kdjby3JH38d3g+v22O/fBa/AGnAAPnINr8AHcgCkg4Bv4Dn6An865EziRE2+kvb2t5xXYCSf9DSl/QCc=</latexit>
�f (x)i<latexit sha1_base64="UDr1aTnlEdzXpQuJNKQjL+Xa+Rs=">AAADhHicfZLPbtQwEMbdDbSl/GvhyMViValI1SopoPaEKvbCsUhkG7FJF9txulbtJLKd3URW3oMrvBVvg+NdJJoIRopmNL/vsyfW4JIzpX3/187Ie/Bwd2//0cHjJ0+fPT88ejFTRSUJDUnBCxlhpChnOQ0105xGpaRIYE6v8d2049crKhUr8i+6KWki0G3OMkaQtq2bGGdfb7ITm+o3C7Y4HPuTC78LOCyCicv+GGzjanE0+hanBakEzTXhSKl54Jc6MUhqRjhtD+JK0RKRO3RL57bMkaAqMW7sFh7bTgqzQtov19B1/3YYJFRHVAs3cQwxtwgXSKZQIL2EqhG44Krv0ktxCm3ucNuHnbE3mc4uEsPystI0J5vBsopDXcDuzWDKJCWaN7ZARDL7b5AskURE25e9dzwW9uScrkkhBMpTE8/aeZCYuLsTZ2YctD2Os2lrVWY6BJUD1RCEDoRDsHZgPQQrB1ZDEDkQDUHtQD0EjQNNH9QobZ1vYWJNa21Qaq/riaKtKPqfyG5kJxImLpesbe1O/lk8+O9idjYJ/Enw+d348uN2O/fBK/AanIAAnINL8AlcgRAQIMF38AP89Ha9U++t934jHe1sPS/BvfA+/AYJZC40</latexit><latexit sha1_base64="UDr1aTnlEdzXpQuJNKQjL+Xa+Rs=">AAADhHicfZLPbtQwEMbdDbSl/GvhyMViValI1SopoPaEKvbCsUhkG7FJF9txulbtJLKd3URW3oMrvBVvg+NdJJoIRopmNL/vsyfW4JIzpX3/187Ie/Bwd2//0cHjJ0+fPT88ejFTRSUJDUnBCxlhpChnOQ0105xGpaRIYE6v8d2049crKhUr8i+6KWki0G3OMkaQtq2bGGdfb7ITm+o3C7Y4HPuTC78LOCyCicv+GGzjanE0+hanBakEzTXhSKl54Jc6MUhqRjhtD+JK0RKRO3RL57bMkaAqMW7sFh7bTgqzQtov19B1/3YYJFRHVAs3cQwxtwgXSKZQIL2EqhG44Krv0ktxCm3ucNuHnbE3mc4uEsPystI0J5vBsopDXcDuzWDKJCWaN7ZARDL7b5AskURE25e9dzwW9uScrkkhBMpTE8/aeZCYuLsTZ2YctD2Os2lrVWY6BJUD1RCEDoRDsHZgPQQrB1ZDEDkQDUHtQD0EjQNNH9QobZ1vYWJNa21Qaq/riaKtKPqfyG5kJxImLpesbe1O/lk8+O9idjYJ/Enw+d348uN2O/fBK/AanIAAnINL8AlcgRAQIMF38AP89Ha9U++t934jHe1sPS/BvfA+/AYJZC40</latexit><latexit sha1_base64="UDr1aTnlEdzXpQuJNKQjL+Xa+Rs=">AAADhHicfZLPbtQwEMbdDbSl/GvhyMViValI1SopoPaEKvbCsUhkG7FJF9txulbtJLKd3URW3oMrvBVvg+NdJJoIRopmNL/vsyfW4JIzpX3/187Ie/Bwd2//0cHjJ0+fPT88ejFTRSUJDUnBCxlhpChnOQ0105xGpaRIYE6v8d2049crKhUr8i+6KWki0G3OMkaQtq2bGGdfb7ITm+o3C7Y4HPuTC78LOCyCicv+GGzjanE0+hanBakEzTXhSKl54Jc6MUhqRjhtD+JK0RKRO3RL57bMkaAqMW7sFh7bTgqzQtov19B1/3YYJFRHVAs3cQwxtwgXSKZQIL2EqhG44Krv0ktxCm3ucNuHnbE3mc4uEsPystI0J5vBsopDXcDuzWDKJCWaN7ZARDL7b5AskURE25e9dzwW9uScrkkhBMpTE8/aeZCYuLsTZ2YctD2Os2lrVWY6BJUD1RCEDoRDsHZgPQQrB1ZDEDkQDUHtQD0EjQNNH9QobZ1vYWJNa21Qaq/riaKtKPqfyG5kJxImLpesbe1O/lk8+O9idjYJ/Enw+d348uN2O/fBK/AanIAAnINL8AlcgRAQIMF38AP89Ha9U++t934jHe1sPS/BvfA+/AYJZC40</latexit><latexit sha1_base64="UDr1aTnlEdzXpQuJNKQjL+Xa+Rs=">AAADhHicfZLPbtQwEMbdDbSl/GvhyMViValI1SopoPaEKvbCsUhkG7FJF9txulbtJLKd3URW3oMrvBVvg+NdJJoIRopmNL/vsyfW4JIzpX3/187Ie/Bwd2//0cHjJ0+fPT88ejFTRSUJDUnBCxlhpChnOQ0105xGpaRIYE6v8d2049crKhUr8i+6KWki0G3OMkaQtq2bGGdfb7ITm+o3C7Y4HPuTC78LOCyCicv+GGzjanE0+hanBakEzTXhSKl54Jc6MUhqRjhtD+JK0RKRO3RL57bMkaAqMW7sFh7bTgqzQtov19B1/3YYJFRHVAs3cQwxtwgXSKZQIL2EqhG44Krv0ktxCm3ucNuHnbE3mc4uEsPystI0J5vBsopDXcDuzWDKJCWaN7ZARDL7b5AskURE25e9dzwW9uScrkkhBMpTE8/aeZCYuLsTZ2YctD2Os2lrVWY6BJUD1RCEDoRDsHZgPQQrB1ZDEDkQDUHtQD0EjQNNH9QobZ1vYWJNa21Qaq/riaKtKPqfyG5kJxImLpesbe1O/lk8+O9idjYJ/Enw+d348uN2O/fBK/AanIAAnINL8AlcgRAQIMF38AP89Ha9U++t934jHe1sPS/BvfA+/AYJZC40</latexit>
xadv = x� ✏ · sign(rx(max(�(x)i : i 6= T )� �(x)T )).<latexit sha1_base64="0Bsx5y+4oXBHcrJ3kOmqm2pd9DY=">AAAD33icfZJNixMxGMezO76s9a2rRy/BstCCW2YWwUUQFnvxuELbLTalZjKZNnSSGSeZdkrI2Zt49UP5AfwcXhXMpBV2Z9AHMnl4fv8nL5N/mCVMKt//cXDo3bp95+7Rvdb9Bw8fPW4fPxnLtMgJHZE0SfNJiCVNmKAjxVRCJ1lOMQ8TehWuBhW/WtNcslQM1TajM44XgsWMYGVL8/YClTiCbyAK4xKeQkQzyZJUQESiVEGkaKm0ZAthukjgMMFzXSkN7CKOy67NP1SfsjdnryGDSNBPcNg7vVbXQ9Pr9Vvzdsfvn/tVwGYS9N3sd8A+LufHhx9RlJKCU6FIgqWcBn6mZhrnipGEmhYqJM0wWeEFndpUYE7lTLs/YuCJrUQwTnM7hIKuer1DYy4rIg3cxQm0lyOrMMV5BDlWSyi3PEwTWe9SS/4C2rnCpg6rxtrJVHw+00xkhaKC7A4WFwlUKayeA0Ysp0QlW5tgkjN7N0iWOMdE2Ue7sXzI7cqCbkjKORaRRmMzDWYaVXuGse4EpsbDeGCsSg+aoHCgaIKRA6Mm2DiwaYK1A+smmDgwaYLSgbIJtg5s68Aa1Lg+6z3nRxzZ7WqiyV40+Z/IurIScY2yJTPGevKv8eC/k/FZP/D7wfuXnYu3e3cegWfgOeiCALwCF+AduAQjQMB38BP8Ar897H32vnhfd9LDg33PU3AjvG9/AD+wUBQ=</latexit><latexit sha1_base64="0Bsx5y+4oXBHcrJ3kOmqm2pd9DY=">AAAD33icfZJNixMxGMezO76s9a2rRy/BstCCW2YWwUUQFnvxuELbLTalZjKZNnSSGSeZdkrI2Zt49UP5AfwcXhXMpBV2Z9AHMnl4fv8nL5N/mCVMKt//cXDo3bp95+7Rvdb9Bw8fPW4fPxnLtMgJHZE0SfNJiCVNmKAjxVRCJ1lOMQ8TehWuBhW/WtNcslQM1TajM44XgsWMYGVL8/YClTiCbyAK4xKeQkQzyZJUQESiVEGkaKm0ZAthukjgMMFzXSkN7CKOy67NP1SfsjdnryGDSNBPcNg7vVbXQ9Pr9Vvzdsfvn/tVwGYS9N3sd8A+LufHhx9RlJKCU6FIgqWcBn6mZhrnipGEmhYqJM0wWeEFndpUYE7lTLs/YuCJrUQwTnM7hIKuer1DYy4rIg3cxQm0lyOrMMV5BDlWSyi3PEwTWe9SS/4C2rnCpg6rxtrJVHw+00xkhaKC7A4WFwlUKayeA0Ysp0QlW5tgkjN7N0iWOMdE2Ue7sXzI7cqCbkjKORaRRmMzDWYaVXuGse4EpsbDeGCsSg+aoHCgaIKRA6Mm2DiwaYK1A+smmDgwaYLSgbIJtg5s68Aa1Lg+6z3nRxzZ7WqiyV40+Z/IurIScY2yJTPGevKv8eC/k/FZP/D7wfuXnYu3e3cegWfgOeiCALwCF+AduAQjQMB38BP8Ar897H32vnhfd9LDg33PU3AjvG9/AD+wUBQ=</latexit><latexit sha1_base64="0Bsx5y+4oXBHcrJ3kOmqm2pd9DY=">AAAD33icfZJNixMxGMezO76s9a2rRy/BstCCW2YWwUUQFnvxuELbLTalZjKZNnSSGSeZdkrI2Zt49UP5AfwcXhXMpBV2Z9AHMnl4fv8nL5N/mCVMKt//cXDo3bp95+7Rvdb9Bw8fPW4fPxnLtMgJHZE0SfNJiCVNmKAjxVRCJ1lOMQ8TehWuBhW/WtNcslQM1TajM44XgsWMYGVL8/YClTiCbyAK4xKeQkQzyZJUQESiVEGkaKm0ZAthukjgMMFzXSkN7CKOy67NP1SfsjdnryGDSNBPcNg7vVbXQ9Pr9Vvzdsfvn/tVwGYS9N3sd8A+LufHhx9RlJKCU6FIgqWcBn6mZhrnipGEmhYqJM0wWeEFndpUYE7lTLs/YuCJrUQwTnM7hIKuer1DYy4rIg3cxQm0lyOrMMV5BDlWSyi3PEwTWe9SS/4C2rnCpg6rxtrJVHw+00xkhaKC7A4WFwlUKayeA0Ysp0QlW5tgkjN7N0iWOMdE2Ue7sXzI7cqCbkjKORaRRmMzDWYaVXuGse4EpsbDeGCsSg+aoHCgaIKRA6Mm2DiwaYK1A+smmDgwaYLSgbIJtg5s68Aa1Lg+6z3nRxzZ7WqiyV40+Z/IurIScY2yJTPGevKv8eC/k/FZP/D7wfuXnYu3e3cegWfgOeiCALwCF+AduAQjQMB38BP8Ar897H32vnhfd9LDg33PU3AjvG9/AD+wUBQ=</latexit><latexit sha1_base64="0Bsx5y+4oXBHcrJ3kOmqm2pd9DY=">AAAD33icfZJNixMxGMezO76s9a2rRy/BstCCW2YWwUUQFnvxuELbLTalZjKZNnSSGSeZdkrI2Zt49UP5AfwcXhXMpBV2Z9AHMnl4fv8nL5N/mCVMKt//cXDo3bp95+7Rvdb9Bw8fPW4fPxnLtMgJHZE0SfNJiCVNmKAjxVRCJ1lOMQ8TehWuBhW/WtNcslQM1TajM44XgsWMYGVL8/YClTiCbyAK4xKeQkQzyZJUQESiVEGkaKm0ZAthukjgMMFzXSkN7CKOy67NP1SfsjdnryGDSNBPcNg7vVbXQ9Pr9Vvzdsfvn/tVwGYS9N3sd8A+LufHhx9RlJKCU6FIgqWcBn6mZhrnipGEmhYqJM0wWeEFndpUYE7lTLs/YuCJrUQwTnM7hIKuer1DYy4rIg3cxQm0lyOrMMV5BDlWSyi3PEwTWe9SS/4C2rnCpg6rxtrJVHw+00xkhaKC7A4WFwlUKayeA0Ysp0QlW5tgkjN7N0iWOMdE2Ue7sXzI7cqCbkjKORaRRmMzDWYaVXuGse4EpsbDeGCsSg+aoHCgaIKRA6Mm2DiwaYK1A+smmDgwaYLSgbIJtg5s68Aa1Lg+6z3nRxzZ7WqiyV40+Z/IurIScY2yJTPGevKv8eC/k/FZP/D7wfuXnYu3e3cegWfgOeiCALwCF+AduAQjQMB38BP8Ar897H32vnhfd9LDg33PU3AjvG9/AD+wUBQ=</latexit>
Using Finite Differences [4], the gradient of any function can be estimated using queries to function values:
FDx(g(x), �) =
g(x+ �e1)� g(x� �e1)
2�, · · · , g(x+ �ed)� g(x� �ed)
2�
�,
<latexit sha1_base64="978vREFljNIzlcjKidwzyixZ6K4=">AAAEUXichZPfb9MwEMedtsDo+NHBIy8W1aRWdFUyIbEXxEQR4nFItIuoo+I4ThvNiaPY6Q9Z/it52p/CG06aSTSBcVJ0p/t872zHZz9lkZC2fWu12p0HDx8dPe4eP3n67Hnv5MVM8DwjdEo445nrY0FZlNCpjCSjbppRHPuMXvs3k4Jfr2kmIp58k7uUejFeJlEYESxNatG7RZJupfr8SS8U8sOtHiwHhR+OUECZxEP4HiJGQwnnEIUZJmrP4Ru4F0AUY7nyQ0X1whmeVfTsr1Sr8yqv4aiLSMClKIL/9Q3u7Rsc9EVZtFxJ6I0Wvb49vrALg83AGZfe7oPKrhYnrR8o4CSPaSIJw0LMHTuVnsKZjAijuotyQVNMbvCSzk2Y4JgKT5V3oOGpyQQw5Jn5EgnL7J8VCseiIELDvZ1Cnxnkc5wFsDgNFLvY50zUq+QqHkHjC6zrsCis7UyGF56KkjSXNCH7jYU5g5LDYgBgEGWUSLYzASZZZM4GyQqbG5BmTA7a+7HpnNAN4XGMk0ChmZ47nrr79X1H17gfTrRRqUkT5CXIm2BagmkTbEqwaYJ1CdZN4JbAbYJtCbZNsCvBrg62ONBlnXkT5fPAgVmuJnIrkXufyA+/F6JYoXQVaW1m8m7w4L+D2fnYscfO17f9y4/VdB6BV+A1GAAHvAOX4Au4AlNArA8WtRKLt3+2f3VAp7WXtqyq5iU4sM7xbx7vdbs=</latexit><latexit sha1_base64="978vREFljNIzlcjKidwzyixZ6K4=">AAAEUXichZPfb9MwEMedtsDo+NHBIy8W1aRWdFUyIbEXxEQR4nFItIuoo+I4ThvNiaPY6Q9Z/it52p/CG06aSTSBcVJ0p/t872zHZz9lkZC2fWu12p0HDx8dPe4eP3n67Hnv5MVM8DwjdEo445nrY0FZlNCpjCSjbppRHPuMXvs3k4Jfr2kmIp58k7uUejFeJlEYESxNatG7RZJupfr8SS8U8sOtHiwHhR+OUECZxEP4HiJGQwnnEIUZJmrP4Ru4F0AUY7nyQ0X1whmeVfTsr1Sr8yqv4aiLSMClKIL/9Q3u7Rsc9EVZtFxJ6I0Wvb49vrALg83AGZfe7oPKrhYnrR8o4CSPaSIJw0LMHTuVnsKZjAijuotyQVNMbvCSzk2Y4JgKT5V3oOGpyQQw5Jn5EgnL7J8VCseiIELDvZ1Cnxnkc5wFsDgNFLvY50zUq+QqHkHjC6zrsCis7UyGF56KkjSXNCH7jYU5g5LDYgBgEGWUSLYzASZZZM4GyQqbG5BmTA7a+7HpnNAN4XGMk0ChmZ47nrr79X1H17gfTrRRqUkT5CXIm2BagmkTbEqwaYJ1CdZN4JbAbYJtCbZNsCvBrg62ONBlnXkT5fPAgVmuJnIrkXufyA+/F6JYoXQVaW1m8m7w4L+D2fnYscfO17f9y4/VdB6BV+A1GAAHvAOX4Au4AlNArA8WtRKLt3+2f3VAp7WXtqyq5iU4sM7xbx7vdbs=</latexit><latexit sha1_base64="978vREFljNIzlcjKidwzyixZ6K4=">AAAEUXichZPfb9MwEMedtsDo+NHBIy8W1aRWdFUyIbEXxEQR4nFItIuoo+I4ThvNiaPY6Q9Z/it52p/CG06aSTSBcVJ0p/t872zHZz9lkZC2fWu12p0HDx8dPe4eP3n67Hnv5MVM8DwjdEo445nrY0FZlNCpjCSjbppRHPuMXvs3k4Jfr2kmIp58k7uUejFeJlEYESxNatG7RZJupfr8SS8U8sOtHiwHhR+OUECZxEP4HiJGQwnnEIUZJmrP4Ru4F0AUY7nyQ0X1whmeVfTsr1Sr8yqv4aiLSMClKIL/9Q3u7Rsc9EVZtFxJ6I0Wvb49vrALg83AGZfe7oPKrhYnrR8o4CSPaSIJw0LMHTuVnsKZjAijuotyQVNMbvCSzk2Y4JgKT5V3oOGpyQQw5Jn5EgnL7J8VCseiIELDvZ1Cnxnkc5wFsDgNFLvY50zUq+QqHkHjC6zrsCis7UyGF56KkjSXNCH7jYU5g5LDYgBgEGWUSLYzASZZZM4GyQqbG5BmTA7a+7HpnNAN4XGMk0ChmZ47nrr79X1H17gfTrRRqUkT5CXIm2BagmkTbEqwaYJ1CdZN4JbAbYJtCbZNsCvBrg62ONBlnXkT5fPAgVmuJnIrkXufyA+/F6JYoXQVaW1m8m7w4L+D2fnYscfO17f9y4/VdB6BV+A1GAAHvAOX4Au4AlNArA8WtRKLt3+2f3VAp7WXtqyq5iU4sM7xbx7vdbs=</latexit><latexit sha1_base64="978vREFljNIzlcjKidwzyixZ6K4=">AAAEUXichZPfb9MwEMedtsDo+NHBIy8W1aRWdFUyIbEXxEQR4nFItIuoo+I4ThvNiaPY6Q9Z/it52p/CG06aSTSBcVJ0p/t872zHZz9lkZC2fWu12p0HDx8dPe4eP3n67Hnv5MVM8DwjdEo445nrY0FZlNCpjCSjbppRHPuMXvs3k4Jfr2kmIp58k7uUejFeJlEYESxNatG7RZJupfr8SS8U8sOtHiwHhR+OUECZxEP4HiJGQwnnEIUZJmrP4Ru4F0AUY7nyQ0X1whmeVfTsr1Sr8yqv4aiLSMClKIL/9Q3u7Rsc9EVZtFxJ6I0Wvb49vrALg83AGZfe7oPKrhYnrR8o4CSPaSIJw0LMHTuVnsKZjAijuotyQVNMbvCSzk2Y4JgKT5V3oOGpyQQw5Jn5EgnL7J8VCseiIELDvZ1Cnxnkc5wFsDgNFLvY50zUq+QqHkHjC6zrsCis7UyGF56KkjSXNCH7jYU5g5LDYgBgEGWUSLYzASZZZM4GyQqbG5BmTA7a+7HpnNAN4XGMk0ChmZ47nrr79X1H17gfTrRRqUkT5CXIm2BagmkTbEqwaYJ1CdZN4JbAbYJtCbZNsCvBrg62ONBlnXkT5fPAgVmuJnIrkXufyA+/F6JYoXQVaW1m8m7w4L+D2fnYscfO17f9y4/VdB6BV+A1GAAHvAOX4Au4AlNArA8WtRKLt3+2f3VAp7WXtqyq5iU4sM7xbx7vdbs=</latexit>
where are the canonical basis vectors. The black-box Gradient Estimation based adversarial example is then
ei<latexit sha1_base64="MMOrl7C2dzw9PYSWmxLUVjTZSIw=">AAADgnicfZLPbtQwEMbdTaGl/GvhyMViVQkhtEoKEj30ULEXjkUi24hNtNiO07VqJ5Ht7G5k+TW4wmvxNjjerUQTwUiRR/P7PnsyGlxzpnQY/t4bBfsPHh4cPjp6/OTps+fHJy9mqmokoTGpeCUTjBTlrKSxZprTpJYUCczpNb6ddvx6RaViVflVtzXNBLopWcEI0q6UpgLpJS4MtQu2OB6Hk/OwCzhMook/wzHYxdXiZPQ9zSvSCFpqwpFS8yisdWaQ1Ixwao/SRtEakVt0Q+cuLZGgKjO+aQtPXSWHRSXdV2roq387DBKqI8rCbZxCzB3CFZI57PqGqhW44qrv0kvxDrqzw7YPO2OvM12cZ4aVdaNpSbaNFQ2HuoLdxGDOJCWaty5BRDL3b5AskUREu7neux4Ld3NJ16QSApW5SWd2HmXmbsjjyPY4LqbWqcx0CBoPmiGIPYiHYO3BeghWHqyGIPEgGYKNB5shaD1o+2CDcut9C5NqutEG5e65nijZiZL/iXDxrRMJk9ZLZq3bybvFg/9OZmeTKJxEXz6MLz/ttvMQvAKvwRsQgY/gEnwGVyAGBNTgB/gJfgX7wdsgCt5vpaO9necluBfBxR+qRy4c</latexit><latexit sha1_base64="MMOrl7C2dzw9PYSWmxLUVjTZSIw=">AAADgnicfZLPbtQwEMbdTaGl/GvhyMViVQkhtEoKEj30ULEXjkUi24hNtNiO07VqJ5Ht7G5k+TW4wmvxNjjerUQTwUiRR/P7PnsyGlxzpnQY/t4bBfsPHh4cPjp6/OTps+fHJy9mqmokoTGpeCUTjBTlrKSxZprTpJYUCczpNb6ddvx6RaViVflVtzXNBLopWcEI0q6UpgLpJS4MtQu2OB6Hk/OwCzhMook/wzHYxdXiZPQ9zSvSCFpqwpFS8yisdWaQ1Ixwao/SRtEakVt0Q+cuLZGgKjO+aQtPXSWHRSXdV2roq387DBKqI8rCbZxCzB3CFZI57PqGqhW44qrv0kvxDrqzw7YPO2OvM12cZ4aVdaNpSbaNFQ2HuoLdxGDOJCWaty5BRDL3b5AskUREu7neux4Ld3NJ16QSApW5SWd2HmXmbsjjyPY4LqbWqcx0CBoPmiGIPYiHYO3BeghWHqyGIPEgGYKNB5shaD1o+2CDcut9C5NqutEG5e65nijZiZL/iXDxrRMJk9ZLZq3bybvFg/9OZmeTKJxEXz6MLz/ttvMQvAKvwRsQgY/gEnwGVyAGBNTgB/gJfgX7wdsgCt5vpaO9necluBfBxR+qRy4c</latexit><latexit sha1_base64="MMOrl7C2dzw9PYSWmxLUVjTZSIw=">AAADgnicfZLPbtQwEMbdTaGl/GvhyMViVQkhtEoKEj30ULEXjkUi24hNtNiO07VqJ5Ht7G5k+TW4wmvxNjjerUQTwUiRR/P7PnsyGlxzpnQY/t4bBfsPHh4cPjp6/OTps+fHJy9mqmokoTGpeCUTjBTlrKSxZprTpJYUCczpNb6ddvx6RaViVflVtzXNBLopWcEI0q6UpgLpJS4MtQu2OB6Hk/OwCzhMook/wzHYxdXiZPQ9zSvSCFpqwpFS8yisdWaQ1Ixwao/SRtEakVt0Q+cuLZGgKjO+aQtPXSWHRSXdV2roq387DBKqI8rCbZxCzB3CFZI57PqGqhW44qrv0kvxDrqzw7YPO2OvM12cZ4aVdaNpSbaNFQ2HuoLdxGDOJCWaty5BRDL3b5AskUREu7neux4Ld3NJ16QSApW5SWd2HmXmbsjjyPY4LqbWqcx0CBoPmiGIPYiHYO3BeghWHqyGIPEgGYKNB5shaD1o+2CDcut9C5NqutEG5e65nijZiZL/iXDxrRMJk9ZLZq3bybvFg/9OZmeTKJxEXz6MLz/ttvMQvAKvwRsQgY/gEnwGVyAGBNTgB/gJfgX7wdsgCt5vpaO9necluBfBxR+qRy4c</latexit><latexit sha1_base64="MMOrl7C2dzw9PYSWmxLUVjTZSIw=">AAADgnicfZLPbtQwEMbdTaGl/GvhyMViVQkhtEoKEj30ULEXjkUi24hNtNiO07VqJ5Ht7G5k+TW4wmvxNjjerUQTwUiRR/P7PnsyGlxzpnQY/t4bBfsPHh4cPjp6/OTps+fHJy9mqmokoTGpeCUTjBTlrKSxZprTpJYUCczpNb6ddvx6RaViVflVtzXNBLopWcEI0q6UpgLpJS4MtQu2OB6Hk/OwCzhMook/wzHYxdXiZPQ9zSvSCFpqwpFS8yisdWaQ1Ixwao/SRtEakVt0Q+cuLZGgKjO+aQtPXSWHRSXdV2roq387DBKqI8rCbZxCzB3CFZI57PqGqhW44qrv0kvxDrqzw7YPO2OvM12cZ4aVdaNpSbaNFQ2HuoLdxGDOJCWaty5BRDL3b5AskUREu7neux4Ld3NJ16QSApW5SWd2HmXmbsjjyPY4LqbWqcx0CBoPmiGIPYiHYO3BeghWHqyGIPEgGYKNB5shaD1o+2CDcut9C5NqutEG5e65nijZiZL/iXDxrRMJk9ZLZq3bybvFg/9OZmeTKJxEXz6MLz/ttvMQvAKvwRsQgY/gEnwGVyAGBNTgB/gJfgX7wdsgCt5vpaO9necluBfBxR+qRy4c</latexit>
xadv = x� ✏ · sign(FDx(max(�(x)i : i 6= T )� �(x)T , �))<latexit sha1_base64="HiMZOzU9mfv2Y/TyvhgBbQFKpp4=">AAAD6XicfZLfatswFMaVeGu77l+6Xe5GLBQSaINdBiuDQVnG2GUHSWsWhUyW5UREkl1LThyMHmJ3Y7d7qLGHGUx2MmhttgOyjs/vO5JsfUHCmdKu+6vVdu7d39s/eHD48NHjJ087R8+uVJylhI5JzOPUD7CinEk61kxz6icpxSLg9DpYDkt+vaKpYrEc6U1CpwLPJYsYwdqWZp0blOMQvoUoiHJ4ChFNFOOxhIiEsYZI01wXis2lgb3d24f3ZlaUcltCAuc9m38uH3l/xt4wiCS9gaP+6a1yMTInEIWUa9zvzzpdd3DulgGbiTeoZrcLdnE5O2p/QWFMMkGlJhwrNfHcRE8LnGpGODWHKFM0wWSJ53RiU4kFVdOi+jcGHttKCKM4tUNqWFVvdxRYqJIoA7dxDANuURDjNIQC6wVUGxHEXNW79EKcQDuX2NRh2Vg7mY7OpwWTSaapJNuDRRmHOoblxcCQpZRovrEJJimz3wbJAqeYaHt9d5YPhF1Z0jWJhcAyLNCVmXjTApV7BlHR9UyNB9HQWFUxbIKsAlkTjCswboJ1BdZNsKrAqgn8CvhNkFcgb4JNBTZ1YK1qqj5rwMqLOLTb1UT+TuT/T2S9WYpEgZIFM8Z68q/x4L+Tq7OB5w68T6+6F+927jwAL8BL0AMeeA0uwEdwCcaAgJ/gd2uvte8sna/ON+f7Vtpu7Xqegzvh/PgD2YFSXQ==</latexit><latexit sha1_base64="HiMZOzU9mfv2Y/TyvhgBbQFKpp4=">AAAD6XicfZLfatswFMaVeGu77l+6Xe5GLBQSaINdBiuDQVnG2GUHSWsWhUyW5UREkl1LThyMHmJ3Y7d7qLGHGUx2MmhttgOyjs/vO5JsfUHCmdKu+6vVdu7d39s/eHD48NHjJ087R8+uVJylhI5JzOPUD7CinEk61kxz6icpxSLg9DpYDkt+vaKpYrEc6U1CpwLPJYsYwdqWZp0blOMQvoUoiHJ4ChFNFOOxhIiEsYZI01wXis2lgb3d24f3ZlaUcltCAuc9m38uH3l/xt4wiCS9gaP+6a1yMTInEIWUa9zvzzpdd3DulgGbiTeoZrcLdnE5O2p/QWFMMkGlJhwrNfHcRE8LnGpGODWHKFM0wWSJ53RiU4kFVdOi+jcGHttKCKM4tUNqWFVvdxRYqJIoA7dxDANuURDjNIQC6wVUGxHEXNW79EKcQDuX2NRh2Vg7mY7OpwWTSaapJNuDRRmHOoblxcCQpZRovrEJJimz3wbJAqeYaHt9d5YPhF1Z0jWJhcAyLNCVmXjTApV7BlHR9UyNB9HQWFUxbIKsAlkTjCswboJ1BdZNsKrAqgn8CvhNkFcgb4JNBTZ1YK1qqj5rwMqLOLTb1UT+TuT/T2S9WYpEgZIFM8Z68q/x4L+Tq7OB5w68T6+6F+927jwAL8BL0AMeeA0uwEdwCcaAgJ/gd2uvte8sna/ON+f7Vtpu7Xqegzvh/PgD2YFSXQ==</latexit><latexit sha1_base64="HiMZOzU9mfv2Y/TyvhgBbQFKpp4=">AAAD6XicfZLfatswFMaVeGu77l+6Xe5GLBQSaINdBiuDQVnG2GUHSWsWhUyW5UREkl1LThyMHmJ3Y7d7qLGHGUx2MmhttgOyjs/vO5JsfUHCmdKu+6vVdu7d39s/eHD48NHjJ087R8+uVJylhI5JzOPUD7CinEk61kxz6icpxSLg9DpYDkt+vaKpYrEc6U1CpwLPJYsYwdqWZp0blOMQvoUoiHJ4ChFNFOOxhIiEsYZI01wXis2lgb3d24f3ZlaUcltCAuc9m38uH3l/xt4wiCS9gaP+6a1yMTInEIWUa9zvzzpdd3DulgGbiTeoZrcLdnE5O2p/QWFMMkGlJhwrNfHcRE8LnGpGODWHKFM0wWSJ53RiU4kFVdOi+jcGHttKCKM4tUNqWFVvdxRYqJIoA7dxDANuURDjNIQC6wVUGxHEXNW79EKcQDuX2NRh2Vg7mY7OpwWTSaapJNuDRRmHOoblxcCQpZRovrEJJimz3wbJAqeYaHt9d5YPhF1Z0jWJhcAyLNCVmXjTApV7BlHR9UyNB9HQWFUxbIKsAlkTjCswboJ1BdZNsKrAqgn8CvhNkFcgb4JNBTZ1YK1qqj5rwMqLOLTb1UT+TuT/T2S9WYpEgZIFM8Z68q/x4L+Tq7OB5w68T6+6F+927jwAL8BL0AMeeA0uwEdwCcaAgJ/gd2uvte8sna/ON+f7Vtpu7Xqegzvh/PgD2YFSXQ==</latexit><latexit sha1_base64="HiMZOzU9mfv2Y/TyvhgBbQFKpp4=">AAAD6XicfZLfatswFMaVeGu77l+6Xe5GLBQSaINdBiuDQVnG2GUHSWsWhUyW5UREkl1LThyMHmJ3Y7d7qLGHGUx2MmhttgOyjs/vO5JsfUHCmdKu+6vVdu7d39s/eHD48NHjJ087R8+uVJylhI5JzOPUD7CinEk61kxz6icpxSLg9DpYDkt+vaKpYrEc6U1CpwLPJYsYwdqWZp0blOMQvoUoiHJ4ChFNFOOxhIiEsYZI01wXis2lgb3d24f3ZlaUcltCAuc9m38uH3l/xt4wiCS9gaP+6a1yMTInEIWUa9zvzzpdd3DulgGbiTeoZrcLdnE5O2p/QWFMMkGlJhwrNfHcRE8LnGpGODWHKFM0wWSJ53RiU4kFVdOi+jcGHttKCKM4tUNqWFVvdxRYqJIoA7dxDANuURDjNIQC6wVUGxHEXNW79EKcQDuX2NRh2Vg7mY7OpwWTSaapJNuDRRmHOoblxcCQpZRovrEJJimz3wbJAqeYaHt9d5YPhF1Z0jWJhcAyLNCVmXjTApV7BlHR9UyNB9HQWFUxbIKsAlkTjCswboJ1BdZNsKrAqgn8CvhNkFcgb4JNBTZ1YK1qqj5rwMqLOLTb1UT+TuT/T2S9WYpEgZIFM8Z68q/x4L+Tq7OB5w68T6+6F+927jwAL8BL0AMeeA0uwEdwCcaAgJ/gd2uvte8sna/ON+f7Vtpu7Xqegzvh/PgD2YFSXQ==</latexit>
Issue: Huge number of queries needed for high-dimensional images
Results
MNIST
CIFAR-10
Single-step IterativeWhite-box Finite Diff. Query-reduced White-box Finite Diff. Query-reduced
Attack on Clarifai Moderation API
Benign: ‘Drug’ Adversarial: ‘Safe’
Generated using Iterative Gradient Estimation attack
Sample adversarial images
0
20
40
60
80
100
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4✏
L1
0
20
40
60
80
100
0 4 8 12 16 20 24 28 32✏
L1
Figure 2: Effectiveness of various single step black-box attacks on Model A (MNIST) andResnet-32 (CIFAR-10). The y-axis for both figures gives the variation in adversarial success as ✏is increased. The most successful black-box attack strategy in both cases is the Gradient Estimationattack using Finite Differences with the logit loss (FD-logit), which coincides almost exactly with thewhite-box FGS attack with the logit loss (WB FGS-logit). Also, the Gradient Estimation attack withquery reduction using PCA (GE-QR (PCA-k, logit)) performs well for both datasets as well.
FD-T and IFD-T achieve the highest adversarial success rates in the targeted setting: For tar-geted black-box attacks, IFD-xent-T achieves 100% adversarial success rates on almost all models asshown by the results in Table 2. While FD-xent-T only achieves about 30% adversarial success rates,this matches the performance of single-step white-box attacks such as FGS-xent-T and FGS-logit-T
(Table 9). The average distortion for samples generated using gradient estimation methods is similarwith that of white-box attacks.
Parameter choices: We use � = 1.0 for FD-xent and IFD-xent for both datasets, while using� = 0.01 for FD-logit and IFD-logit. We find that a larger value of � is needed for xent loss basedattacks to work. The reason for this is that the probability values used in the xent loss are not assensitive to changes as in the logit loss, and thus the gradient cannot be estimated since the functionvalue does not change at all when a single pixel is perturbed. For the Iterative Gradient Estimationattacks using Finite Differences, we use ↵ = 0.01 and t = 40 for the MNIST results and ↵ = 1.0and t = 10 for CIFAR-10 throughout. The same parameters are used for the white-box Iterative FGSattack results given in Appendix D. This translates to 62720 queries for MNIST (40 steps of iteration)and 61440 queries (10 steps of iteration) for CIFAR-10 per sample. We find these choices work well,and keep the running time of the Gradient Estimation attacks at a manageable level. However, we findthat we can achieve similar adversarial success rates with much fewer queries using query reductionmethods which we describe in the next section.
3.2 QUERY REDUCTION
The major drawback of the approximation based black-box attacks is that the number of queriesneeded per adversarial sample is large. For an input with dimension d, the number of queries will beexactly 2d for a two-sided approximation. This may be too large when the input is high-dimensional.So we examine two techniques in order to reduce the number of queries the adversary has to make.Both techniques involve estimating the gradient for groups of features, instead of estimating it onefeature at a time.
The justification for the use of feature grouping comes from the relation between gradients anddirectional derivatives (Hildebrand, 1962) for differentiable functions. The directional derivative of afunction g is defined as rvg(x) = limh!0
g(x+hv)�g(x)h . It is a generalization of a partial derivative.
For differentiable functions, rvg(x) = rxg(x) · v, which implies that the directional derivative isjust the projection of the gradient along the direction v. Thus, estimating the gradient by groupingfeatures is equivalent to estimating an approximation of the gradient constructed by projecting italong appropriately chosen directions. The estimated gradient r̂xg(x) of any function g can becomputed using the techniques below, and then plugged in to Equations 3 and 6 instead of the finitedifference term to create an adversarial sample. Next, we introduce the techniques applied to groupthe features for estimation.
9
constrained strategies on Model A
Difference-of-meansRandom-perturbationFinite-difference xentFinite-difference logit
Query-reduced PCA-100 logit Transfer Model B FGS xentTransfer Model B FGS logit
White-box FGS logitWhite-box FGS xent
Difference-of-meansRandom-perturbationFinite-difference xentFinite-difference logit
Query-reduced PCA-400 logitTransfer Resnet-28-10 FGS xent
White-box FGS logitWhite-box FGS xent
Model A (MNIST) Resnet-32 (CIFAR-10)
constrained strategies on Resnet-32
Adv
ersa
rial s
ucce
ss (%
)
Adv
ersa
rial s
ucce
ss (%
)
Gradient Estimationmatcheswhite-box attack success rates!
Comparison of attack success rate vs. perturbation magnitude
Random grouping: Estimate gradient along directions corresponding to random grouping of dimensions
𝑥" + ℎ𝑥" − ℎ
𝑥& + ℎ
𝑥& − ℎ
4 queries,
more accurate
(𝑥" + ℎ, 𝑥&+ℎ)
(𝑥" − ℎ, 𝑥&−ℎ)
2 queries,
less accurate
(𝑥", 𝑥&)
For differentiable functions, the estimate of the gradient along direction is equivalent to its projection along that direction. Both query reduction methods utilize this idea.
Principal Component based: Estimate gradient along directions corresponding to principal components of data. More effective since PCA minimizes reconstruction error in terms of norm.Caveat: Assumes access to subset of training data
Models(Top 2: MNIST
Bottom 2: CIFAR-10)
White-box Gradient EstimationGradient Estimation(Query Reduction)
FGS FGS (Iter.) FD FD (Iter.) RG PCA RG (Iter.) PCA (Iter.)
A 30.1 99.6 29.9 99.7 15.9 23.2 73.8 96.2
B 29.6 98.7 29.3 98.7 17.8 29.0 73.7 93.9
Resnet-32 23.5 100.0 23.0 100.0 19.0 21.0 97.0 81.0Resnet-28-10 27.6 100.0 28.0 100.0 20.0 23.0 94.0 72.0
Targeted attack success rates
v<latexit sha1_base64="WZDsTV28dK21n+V4MtFlW2eQiWQ=">AAACT3icdVBNSyQxEE2Puuroro4evQSHBU+z3YugR/EDvIgKzig73Ug6Uz0G89Ek6ZEh9L/wqv/Jo7/Em5geW1BXC5J6vKpKXr0058zYMHwMGlPTMz9m5+abC4s/fy0tt1Z6RhWaQpcqrvRFSgxwJqFrmeVwkWsgIuVwnl7vVfXzEWjDlDyz4xwSQYaSZYwS66l/sSD2Ks3cqLxcboed7bAK/D+IOpMctlEdJ5et4E88ULQQIC3lxJh+FOY2cURbRjmUzbgwkBN6TYbQ91ASASZxE8kl/u2ZAc6U9kdaPGHfTzgijBmL1HdWEs3nWkV+VesXNttOHJN5YUHS14+ygmOrcLU/HjAN1PKxB4Rq5rViekU0oda71Iwl3FAlBJEDF/fKfpS4N4vaUel32ge/q4Yjzx3noIlV2sWGDWXpJvfHFw7Kejx1B2Vl8ZuP+HvQ+9uJwk50utne2a3NnkNraB1toAhtoR10iE5QF1Ek0S26Q/fBQ/AUPDfq1kZQg1X0IRrzL6oWtVQ=</latexit><latexit sha1_base64="WZDsTV28dK21n+V4MtFlW2eQiWQ=">AAACT3icdVBNSyQxEE2Puuroro4evQSHBU+z3YugR/EDvIgKzig73Ug6Uz0G89Ek6ZEh9L/wqv/Jo7/Em5geW1BXC5J6vKpKXr0058zYMHwMGlPTMz9m5+abC4s/fy0tt1Z6RhWaQpcqrvRFSgxwJqFrmeVwkWsgIuVwnl7vVfXzEWjDlDyz4xwSQYaSZYwS66l/sSD2Ks3cqLxcboed7bAK/D+IOpMctlEdJ5et4E88ULQQIC3lxJh+FOY2cURbRjmUzbgwkBN6TYbQ91ASASZxE8kl/u2ZAc6U9kdaPGHfTzgijBmL1HdWEs3nWkV+VesXNttOHJN5YUHS14+ygmOrcLU/HjAN1PKxB4Rq5rViekU0oda71Iwl3FAlBJEDF/fKfpS4N4vaUel32ge/q4Yjzx3noIlV2sWGDWXpJvfHFw7Kejx1B2Vl8ZuP+HvQ+9uJwk50utne2a3NnkNraB1toAhtoR10iE5QF1Ek0S26Q/fBQ/AUPDfq1kZQg1X0IRrzL6oWtVQ=</latexit><latexit sha1_base64="WZDsTV28dK21n+V4MtFlW2eQiWQ=">AAACT3icdVBNSyQxEE2Puuroro4evQSHBU+z3YugR/EDvIgKzig73Ug6Uz0G89Ek6ZEh9L/wqv/Jo7/Em5geW1BXC5J6vKpKXr0058zYMHwMGlPTMz9m5+abC4s/fy0tt1Z6RhWaQpcqrvRFSgxwJqFrmeVwkWsgIuVwnl7vVfXzEWjDlDyz4xwSQYaSZYwS66l/sSD2Ks3cqLxcboed7bAK/D+IOpMctlEdJ5et4E88ULQQIC3lxJh+FOY2cURbRjmUzbgwkBN6TYbQ91ASASZxE8kl/u2ZAc6U9kdaPGHfTzgijBmL1HdWEs3nWkV+VesXNttOHJN5YUHS14+ygmOrcLU/HjAN1PKxB4Rq5rViekU0oda71Iwl3FAlBJEDF/fKfpS4N4vaUel32ge/q4Yjzx3noIlV2sWGDWXpJvfHFw7Kejx1B2Vl8ZuP+HvQ+9uJwk50utne2a3NnkNraB1toAhtoR10iE5QF1Ek0S26Q/fBQ/AUPDfq1kZQg1X0IRrzL6oWtVQ=</latexit><latexit sha1_base64="WZDsTV28dK21n+V4MtFlW2eQiWQ=">AAACT3icdVBNSyQxEE2Puuroro4evQSHBU+z3YugR/EDvIgKzig73Ug6Uz0G89Ek6ZEh9L/wqv/Jo7/Em5geW1BXC5J6vKpKXr0058zYMHwMGlPTMz9m5+abC4s/fy0tt1Z6RhWaQpcqrvRFSgxwJqFrmeVwkWsgIuVwnl7vVfXzEWjDlDyz4xwSQYaSZYwS66l/sSD2Ks3cqLxcboed7bAK/D+IOpMctlEdJ5et4E88ULQQIC3lxJh+FOY2cURbRjmUzbgwkBN6TYbQ91ASASZxE8kl/u2ZAc6U9kdaPGHfTzgijBmL1HdWEs3nWkV+VesXNttOHJN5YUHS14+ygmOrcLU/HjAN1PKxB4Rq5rViekU0oda71Iwl3FAlBJEDF/fKfpS4N4vaUel32ge/q4Yjzx3noIlV2sWGDWXpJvfHFw7Kejx1B2Vl8ZuP+HvQ+9uJwk50utne2a3NnkNraB1toAhtoR10iE5QF1Ek0S26Q/fBQ/AUPDfq1kZQg1X0IRrzL6oWtVQ=</latexit>
Gradient Estimation attacks achieve high attack success rates comparable with white-box attacks. Query reduction methods reduce the number of queries required to a dimension-independent constant. These attacks are effective even against deployed real-world classifiers.
API provides class-wise confidence scores.
Aim: Misclassify image of drugs as ‘safe’.
L2<latexit sha1_base64="WkiWjvzowue9LYXlWscYLg+nJ6Q=">AAACSHicdVDLSiwxEE2PetXRe30t3QQHwdXcbhF0KT7AhaKiMwrTzZDOVI/BPJokrQyhP8Gt/pN/4F+4E3emxxHUey1IcjhVJ1V10pwzY8PwKaiNjU/8mpyars/M/v4zN7+w2Daq0BRaVHGlL1NigDMJLcssh8tcAxEph4v0erfKX9yANkzJczvIIRGkL1nGKLGeOjvsrnfnG2FzK6wC/wui5vANG2gUJ92F4G/cU7QQIC3lxJhOFOY2cURbRjmU9bgwkBN6TfrQ8VASASZxw1lLvOqZHs6U9kdaPGQ/KxwRxgxE6isFsVfme64i/5frFDbbShyTeWFB0vdGWcGxVbhaHPeYBmr5wANCNfOzYnpFNKHW21OPJdxSJQSRPRe3y06UuLjqkWauEZV+pz3wu2o48txxDppYpV1sWF+Wbnh//WG/HMlTt1+W3uIPH/HPoL3ejMJmdLrR2N4ZmT2FltEKWkMR2kTb6ACdoBaiqI/u0D16CB6D5+AleH0vrQUjzRL6ErXaG1tfsq4=</latexit><latexit sha1_base64="WkiWjvzowue9LYXlWscYLg+nJ6Q=">AAACSHicdVDLSiwxEE2PetXRe30t3QQHwdXcbhF0KT7AhaKiMwrTzZDOVI/BPJokrQyhP8Gt/pN/4F+4E3emxxHUey1IcjhVJ1V10pwzY8PwKaiNjU/8mpyars/M/v4zN7+w2Daq0BRaVHGlL1NigDMJLcssh8tcAxEph4v0erfKX9yANkzJczvIIRGkL1nGKLGeOjvsrnfnG2FzK6wC/wui5vANG2gUJ92F4G/cU7QQIC3lxJhOFOY2cURbRjmU9bgwkBN6TfrQ8VASASZxw1lLvOqZHs6U9kdaPGQ/KxwRxgxE6isFsVfme64i/5frFDbbShyTeWFB0vdGWcGxVbhaHPeYBmr5wANCNfOzYnpFNKHW21OPJdxSJQSRPRe3y06UuLjqkWauEZV+pz3wu2o48txxDppYpV1sWF+Wbnh//WG/HMlTt1+W3uIPH/HPoL3ejMJmdLrR2N4ZmT2FltEKWkMR2kTb6ACdoBaiqI/u0D16CB6D5+AleH0vrQUjzRL6ErXaG1tfsq4=</latexit><latexit sha1_base64="WkiWjvzowue9LYXlWscYLg+nJ6Q=">AAACSHicdVDLSiwxEE2PetXRe30t3QQHwdXcbhF0KT7AhaKiMwrTzZDOVI/BPJokrQyhP8Gt/pN/4F+4E3emxxHUey1IcjhVJ1V10pwzY8PwKaiNjU/8mpyars/M/v4zN7+w2Daq0BRaVHGlL1NigDMJLcssh8tcAxEph4v0erfKX9yANkzJczvIIRGkL1nGKLGeOjvsrnfnG2FzK6wC/wui5vANG2gUJ92F4G/cU7QQIC3lxJhOFOY2cURbRjmU9bgwkBN6TfrQ8VASASZxw1lLvOqZHs6U9kdaPGQ/KxwRxgxE6isFsVfme64i/5frFDbbShyTeWFB0vdGWcGxVbhaHPeYBmr5wANCNfOzYnpFNKHW21OPJdxSJQSRPRe3y06UuLjqkWauEZV+pz3wu2o48txxDppYpV1sWF+Wbnh//WG/HMlTt1+W3uIPH/HPoL3ejMJmdLrR2N4ZmT2FltEKWkMR2kTb6ACdoBaiqI/u0D16CB6D5+AleH0vrQUjzRL6ErXaG1tfsq4=</latexit><latexit sha1_base64="WkiWjvzowue9LYXlWscYLg+nJ6Q=">AAACSHicdVDLSiwxEE2PetXRe30t3QQHwdXcbhF0KT7AhaKiMwrTzZDOVI/BPJokrQyhP8Gt/pN/4F+4E3emxxHUey1IcjhVJ1V10pwzY8PwKaiNjU/8mpyars/M/v4zN7+w2Daq0BRaVHGlL1NigDMJLcssh8tcAxEph4v0erfKX9yANkzJczvIIRGkL1nGKLGeOjvsrnfnG2FzK6wC/wui5vANG2gUJ92F4G/cU7QQIC3lxJhOFOY2cURbRjmU9bgwkBN6TfrQ8VASASZxw1lLvOqZHs6U9kdaPGQ/KxwRxgxE6isFsVfme64i/5frFDbbShyTeWFB0vdGWcGxVbhaHPeYBmr5wANCNfOzYnpFNKHW21OPJdxSJQSRPRe3y06UuLjqkWauEZV+pz3wu2o48txxDppYpV1sWF+Wbnh//WG/HMlTt1+W3uIPH/HPoL3ejMJmdLrR2N4ZmT2FltEKWkMR2kTb6ACdoBaiqI/u0D16CB6D5+AleH0vrQUjzRL6ErXaG1tfsq4=</latexit>
Classified as 3
Classified as ’Frog’
![Do Deep Neural Networks Suffer from Crowding? - CBMM · Do Deep Neural Networks Suffer from ... Despite stunning successes in many computer vision problems [1–5], Deep Neural Networks](https://static.fdocuments.in/doc/165x107/5ac1231e7f8b9aca388cb550/do-deep-neural-networks-suffer-from-crowding-cbmm-deep-neural-networks-suffer.jpg)
