Cyber Attack Intent Recognition and Active Deception using ...
Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical...
Transcript of Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical...
![Page 1: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/1.jpg)
Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection
Zhenyu (Edward) Zhong, Yunhan Jia, Weilin Xu, Tao Wei
![Page 2: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/2.jpg)
Our Team X-Lab
Chief Security ScientistDr. Tao Wei
Dr. Yunhan Jia
Dr. Zhenyu(Edward) Zhong edwardzhong [at] baidu DOT com
Weilin Xu
AI Security Research
System Security Research
https://github.com/baidu/rust-sgx-sdk
https://github.com/mesalock-linux
https://github.com/mesalock-linux/mesapy
https://github.com/mesalock-linux/mesalink
https://www.mesatee.org
https://github.com/baidu/AdvBox
Scan Me
![Page 3: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/3.jpg)
Disclaimer
• This talk doesn’t target any commercial autonomous driving systems.
• We don’t provide any comments to the vulnerabilities of the perceptions of existing autonomous driving systems.
• We focus on state-of-the-art object detection methods, all the results/techniques are proof-of-concept.
![Page 4: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/4.jpg)
Car Safety – Unintended Acceleration
https://www.cbsnews.com/news/toyota-unintended-acceleration-has-killed-89/
…the large throttle opening UAs as described in submitted VOQs could not be found…
…does not mean it could not occur…
https://www.eetimes.com/document.asp?doc_id=1319903&page_number=2
…experts identified ”unprotected critical variables.”…
…the defects we found were linked to unintended Acceleration through vehicle testing, …
![Page 5: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/5.jpg)
Car Safety – Autonomous Driving
![Page 6: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/6.jpg)
Car Safety – Rear Ended Into Fire Truck
![Page 7: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/7.jpg)
Imgsrc: http://www.crosslinksolutions.co.uk/service-listing/adas-calibration-equipment/
Car Perception While Driving
$$ $$$$ $Cost Estimate: $$ $
![Page 8: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/8.jpg)
Behind Perception: End2End Object Detection
Object Detection: a technology related to computer vision and image processing that deals with instances of semantic objects of certain class in digital images and videos.
https://software.intel.com/en-us/articles/a-closer-look-at-object-detection-recognition-and-tracking
![Page 9: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/9.jpg)
State-of-the-Art Vision-based Object Detection
Conv Feature Map YOLO (You Only Look Once)
416
4163
3
3208
208 16
33
104
104 32
3
3
13
13256
3
3
13
13
numAnchors x (5 + numClasses)
3
3
![Page 10: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/10.jpg)
Pick Our Target – YOLOv3
Accuracy on MS COCO
YOLOv3: An Incremental Improvement. Joseph Redmon, Ali Farhadihttps://arxiv.org/pdf/1804.02767.pdf
![Page 11: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/11.jpg)
Pick Our Target – YOLOv3
Performance
YOLOv3: An Incremental Improvement. Joseph Redmon, Ali Farhadihttps://arxiv.org/pdf/1804.02767.pdf
![Page 12: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/12.jpg)
Current Status of Adversarial Example
Definition: For an input image !, minimize " !, ! + % , &. (. ) ! + % = (, ! + % ∈ [0,1]0
The most well-studied distance metric: 12 3456 Perturbations• 78 -- each pixel is allowed to be changed by up to a limit• 79 -- number of pixels altered that matter most• 7: -- many small changes to many pixels
![Page 13: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/13.jpg)
Adversarial Examples & !" NormPerturbations Impact to DNN
#$%& !" based Perturbation Method
Source Image Perturbed Images
'( = ' − + , -./0(2!3--#,5('))Intuition: each pixel is allowed to change by up to a limit
Still in Digital Context
Perturbations
![Page 14: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/14.jpg)
Adversarial Examples & !" NormPerturbations Impact to DNN
#$%& !" based Perturbation Method
Source Image YOLOv3 Detection
'( = ' − + , -./0(2!3--#,5('))Intuition: each pixel is allowed to change by up to a limit
Still in Digital Context
Perturbations
![Page 15: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/15.jpg)
Adversarial Examples & !" NormPerturbations Impact to DNN
#$%A !" based Perturbation Method
Source Image Perturbations Perturbed Image
Intuition: # of pixels altered that matter the most
Still in Digital Context
![Page 16: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/16.jpg)
Adversarial Examples & !" NormPerturbations Impact to DNN
#$%A !" based Perturbation Method
Source Image Perturbations YOLOv3 Detection
Intuition: # of pixels altered that matter the most
Still in Digital Context
![Page 17: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/17.jpg)
Adversarial Examples & !" NormPerturbations Impact to DNN
Source Image Perturbed ImagePerturbations
Still in Digital Context
#$" !" based Perturbation Method Intuition: many small changes to many pixels %&'&%&() ∥ + − +- ∥""+ / 0 1(+-)
![Page 18: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/18.jpg)
Adversarial Examples & !" NormPerturbations Impact to DNN
Source Image Perturbed ImagePerturbations
Still in Digital Context
#$" !" based Perturbation Method Intuition: many small changes to many pixels %&'&%&() ∥ + − +- ∥""+ / 0 1(+-)
![Page 19: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/19.jpg)
Digital Perturbations Realistic Enough?
FGSM JSMA CW2
Alter pixel value of the
sky?
Alter pixel value of the
tree?Alter any
pixel value in the image?
![Page 20: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/20.jpg)
Identify Opportunities by Completely Understanding YOLOv3 Inference Mechanism
Explore Chances of Physical White Box Attack against YOLOv3
![Page 21: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/21.jpg)
Deep Dive into YOLOv3
YOLO v3
Object Detection Model
[147 Layers, 62M Parameters]
Input
[416x416x3]
Output
[10,647
Bounding
Boxes]
Image: http://media.nj.com/traffic_impact/photo/all-way-stop-sign-that-flashes-in-montclairjpg-30576ab330660eff.jpg
13x13
26x26
52x52
![Page 22: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/22.jpg)
• Common Objects in Context• 80 Classes: person, [car, truck, bus], [bicycle, motorcycle], [stop sign,
traffic light], etc.
Training Dataset – MS COCO Dataset
![Page 23: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/23.jpg)
car
0.01%
116x
90
156x
198373x326
Anchor
Boxes
!" = $ %" + '"!( = $ %( + '(!) = *)+,-
Center
Point
Object
Size !. = *.+,/
*.*)
13 x 13 Grid
('", '() = (11,2)
Prediction
Vector
Bounding Box 80 Class ConfidenceObjectness
%" %( %) %. *567 '8 '9 … … ':; '<=
■
stop sign
99%
×
car
0.01%
car
0.01%
YOLOv3 Prediction23
![Page 24: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/24.jpg)
Threat Model : Physical Image Patch Attack
CompanyLogo Image Patches
![Page 25: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/25.jpg)
Our Physical Attack Approach & Objectives
• Input Patch Construction• Differentiable to craft adversarial examples
• Attack Objectives• Make YOLOv3 detect fake object
• Make object disappear in front of YOLOv3
![Page 26: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/26.jpg)
CompanyLogo
Differentiable Input Patch Construction
Resize Perspective
Transformation
![Page 27: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/27.jpg)
Our Physical Attack Approach & Objectives
• Input Patch Construction• Differentiable to craft adversarial examples
• Attack Objectives• Object Fabrication: make YOLOv3 detect fake object
• Object Vanishing: make object disappear in front of YOLOv3
![Page 28: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/28.jpg)
Attack Objective 1 – Object Fabrication
A. Naive Fabrication• Push more detections towards a certain object
B. Precise Fabrication• Produce fake object at specific location
CompanyLogo
CompanyLogo
![Page 29: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/29.jpg)
Attack Objective 2 – Object Vanishing
Make a certain object class disappear in the whole image.
![Page 30: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/30.jpg)
5 Inaccurate Patch Location
Challenges to the Success of Physical Attack
Controlled Perturbation Area1
2
Object appearance changesat various distances, angles
3
Various Light conditions:e.g. glaring, dimming
4 Color Distortion on various devices
Digital color palette32 x 21
Kyocera Taskalfa3551 ci
Captured by iPhoneXfrom a distance
![Page 31: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/31.jpg)
Tactics to the Challenges:
• [Controlled Perturbation Area] Image-patch based Attack
• [Color Distortion] Color Management with the Non-Printability Loss (NPS)
• [Inaccurate Patch] Random Transformation (RT) during optimization iterations
• [Various Distances & Angles] RT + Total Variation regularization instead of Expectation-Over-Transformation
• [Various Light Condition] Get a stable environment
• More …
![Page 32: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/32.jpg)
Color Management with Non-Printability Loss
Accessorize to a Crime: Real and Stealthy Attacks on State-Of-The-Art Face Recognition. Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael ReiterIn. In Proceedings of CCS 2016
Given ! ⊂ 0,1 & , a set of printable RGB triplets. '!( *̂ = ∏-∈/ |*̂ − *|For the perturbated 2, NPS(2) = ∑ 5- ∈ 6 '!((*̂) . NPS(2) ↓, color reproducibility ↑
No NPS With NPSPrinted&Captured by iPhone
![Page 33: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/33.jpg)
Solution: Introduce Random Perspective Transformation
Random Transformation During Optimization Iterations
Generated Perturbation PatchPerfectly positioned?
!"#$%"&'(&
!"#$%"&'()
!"#$%"&'(*
![Page 34: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/34.jpg)
RT + TV for Various Distances & Angles
• Random Transformation + Total Variance Regulation�a different approach from EOTSimulate the transformations using RT + TV for various distances & angles instead of drawing from a distribution
![Page 35: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/35.jpg)
Put Everything Together: An Iterative Optimization
NPS RT TV
!"#$%"&'(& !"#$%"&'() !"#$%"&'(*
![Page 36: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/36.jpg)
![Page 37: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/37.jpg)
Conclusion & Takeaway
• With careful setup, physical attacks are achievable against DNN-based object detection methods in a white box setting
• Defense is hard, a good safety and security metric has to be explored
• We call out efforts for a robust, adversarial example resistant model that is required in safety critical system like autonomous driving system
![Page 38: Perception Deception: Physical Adversarial Attack Challenges … · Perception Deception: Physical Adversarial Attack Challenges and Tactics for DNN-based Object Detection Zhenyu(Edward)](https://reader031.fdocuments.in/reader031/viewer/2022041017/5ecb22a8c4bb4006b457bb89/html5/thumbnails/38.jpg)
Scan Me