Model Inversion Attacks AgainstCollaborative Inference

Zecheng He1, Tianwei Zhang2, and Ruby B. Lee1

1Princeton University2Nanyang Technological University

1

Model Inversion Attacks AgainstCollaborative Inference

•Motivation and Background•White-box Attack• Black-box Attack• Access-free Attack• Defense and Conclusion

2

q Deep learning is everywhere• Computer Vision, Natural Language Processing, Robotics,

Medical, Autonomous Driving…

q Cloud-based deep learning systems become prevalent• Google Cloud AI, Amazon Sagemaker, Microsoft Azure AI…• Collaborative Training, Collaborative Inference

Motivation

3

Collaborative Training

Parameter server

Client

…Client

4

g2t

𝜽𝒕

gnt

𝜽𝒕

𝜽𝒕

gn-1t

Client

Client

Collaborative Inferenceq Split the DNN to multiple parties

• Hybrid IoT / Edge-Cloud Computation• Easy parts computed on the edge device, hard parts on the cloud• Save power and reduce latency

Cloud

Edge Device

Edge Device Edge Device

Edge Device

Attacks against Training Data Privacy

q Model inversion attacks• Recover representative input of each class• ‘Average’ of data instead of a specific input. • Not work well for deep NNs

q Membership attacks• Whether a given sample is in the training set?• Need to know candidate training samples

q Attribute/property attacks• Whether the training data has a property, e.g. color, gender• Obtain property information, not an individual sample

6

q Inference data privacy is less studied• More severe problem:

ü Unlike fixed number of training data, the number of inference data is increasing over time

ü Inference samples could be more sensitive

• More challenging problem:ü Trained model does not depend on inference dataü Inference samples vary significantly

No Attacks against Collaborative Inference

Is it possible to recover useful information of individual inference data input?

7

Threat Model

q Adversary: Untrusted or compromisedcloud provider

q Target:Recover sensitive inputs during inference

q Adversary’s capabilities1) White-box of the edge-side model2) Black-box of the edge-side model3) No query to the edge-side model

Edge-side model Cloud-side model

Attack works in all scenarios!8

Model Inversion Attacks AgainstCollaborative Inference

•Motivation and Background•White-box Attack• Black-box Attack• Access-free Attack• Defense and Conclusion

9

?0.850.020.410.19...

[

[

White-box Attack

Intermediate values𝑓$% 𝑥'

Insight: find an input x, such that1)  𝑓$% 𝑥 is close to 𝑓$% 𝑥'2) Domain knowledge: 𝑥 is a natural input

10

Known parameters 𝑓$%

White-box Attack

Regularized Maximum Likelihood Estimation (rMLE)

1) 𝐟𝛉𝟏 𝐱 is close to 𝐟𝛉𝟏 𝐱𝟎 => Euclidean Distance (ED)

2) 𝐱 is a natural sample => Small Total Variation (TV)

ED(𝑥, 𝑥') = ||𝑓$% 𝑥 − 𝑓$% 𝑥' ||88

TV(𝑥) = ;(|𝑥<=%,> − 𝑥<,>|8 + |𝑥<,>=% − 𝑥<,>|8)@/8<,>

𝒙∗ = 𝒂𝒓𝒈𝒎𝒊𝒏𝒙  𝑬𝑫 𝒙,𝒙𝟎 + 𝝀𝑻𝑽(𝒙)

S𝐨𝐥𝐯𝐞  𝒙 with Gradient Descent11

White-box Evaluation

Dataset MNIST CIFAR10Target Model 2 conv + 3 fc (LeNet5) 6 conv + 2 fcSplit Point • 1st conv layer (conv1)

• 2nd conv layer after activation (ReLU2)

• 1st conv layer (conv11)• 4th conv layer after

activation (ReLU22)• 6th conv layer after

activation (ReLU32)

q Target models and datasets

12conv11 conv12 pool1 conv21 conv22 pool2 conv31 conv32 pool3 FC1 FC2

White-box Evaluation

13

Model Inversion Attacks AgainstCollaborative Inference

•Motivation and Background•White-box Attack• Black-box Attack• Access-free Attack• Defense and Conclusion

14

?0.850.020.410.19...

[

[

𝑓$% 𝑥'

Black-box Attack

Insight: Train 𝑓$%S%, an ‘inverse network’ of 𝑓$%, 𝑚𝑎𝑝𝑝𝑖𝑛𝑔

𝑓$% 𝑥 back to x

The adversary can query 𝑓$%

15

Known parameters 𝑓$%

Black-box Attack

q Inverse Network 𝒇𝜽𝟏S𝟏1) Generate training samples2) Train the inverse network

𝑓$%S% = 𝑎𝑟𝑔𝑚𝑖𝑛\  

1𝑚; ||𝑔 𝑓$% 𝑥< − 𝑥<||8]

<^%

𝑓$% 𝑥% , … , 𝑓$% 𝑥] ~(𝑥%,… , 𝑥])

𝑓$% 𝑓$%S%

16

Black-box Attack Evaluation

17

Black-box Attack Evaluation

q Is knowing training data / distribution important?

1) Train Inverse Network with same training data

2) Train Inverse Network with data from the same distribution

3) Train Inverse Network with random noise

𝑓$%S% = 𝑎𝑟𝑔𝑚𝑖𝑛\  

1𝑚; ||𝑔 𝑓$% 𝑥< − 𝑥<||8]

<^%

18

Black-box Attack Evaluation

q Is knowing training data / distribution important?

Inverse Network trained with the same training data

Original inputs

Inverse Network trained with the same training distribution

Inverse Network trained with random noise 19

Black-box Attack Evaluation

q The data distribution is important• Random noise to train the inverse network can only

partially recover the inputs

20

q The exact training samples are not necessary

q Quantitative results in the paper• Peak Signal Noise Ratio (PSNR)• Structural Similarity (SSIM)

Model Inversion Attacks AgainstCollaborative Inference

•Motivation and Background•White-box Attack• Black-box Attack• Access-free Attack• Defense and Conclusion

21

?0.850.020.410.19...

[

[

𝑓$% 𝑥'

Access-free Attack

Known parameters 𝑓$%The adversary can query 𝑓$%

Insight: Reconstruct a shadow model 𝑓′$% , then perform a white-box attack on 𝑓′$%

22

Shadow Model Construction

The cloud-side model 𝑓$8, frozen

The shadow (edge-side) model 𝑓′$% , trainable

Cat

𝑓′$% = 𝑎𝑟𝑔𝑚𝑖𝑛\ ;CrossEntropy(𝑓$8 𝑔 𝑥< , 𝑦<)  ]

<^%

𝑓′$% and 𝑓$8 (frozen) jointly perform well on the original classification task

23

White-box Attack on the Shadow Model

ED′(𝑥, 𝑥') = ||𝑓′$% 𝑥 − 𝑓$% 𝑥' ||88

𝒙∗ = 𝒂𝒓𝒈𝒎𝒊𝒏𝒙  𝑬𝑫′ 𝒙, 𝒙𝟎 + 𝝀𝑻𝑽(𝒙)

0.850.020.410.19...

[

[

𝑓′$% 𝑥'𝑓′$%

White-box rMLE attack against 𝑓′$%

24

Access-free Attack Evaluation

25

Access-free Attack Evaluationq Is knowing exact training data important?

26

Original input

Same training data

Same distribution

Original input

Same training data

Same distribution

Where should the model be split between edge and cloud?

PSRN and SSIM of query-free attackagainst v.s. split points

Query-free attack against v.s. split points

27

Model Inversion Attacks AgainstCollaborative Inference

•Motivation and Background•White-box Attack• Black-box Attack• Access-free Attack• Defense and Conclusion

28

Defenses

q Running FC layers before sending to the cloud, or make the edge-side network deeper• High overhead on edge-side

q Homomorphic Encryption• Too complex, not practical

q Differentially private model inference• Adding noise to the inference input: 𝑣 = 𝑓$%(𝑥 + 𝜀)• Adding noise to the intermediate value: 𝑣 = 𝑓$% 𝑥 + 𝜀• Tradeoff between accuracy (usability) and privacy

29

Conclusion

q Three new attacks in collaborative inference scenarios• White-box attack• Black-box attack• Query-free attack• Adversary can successfully recovers sensitive inputs• Exact training data is not important to the success of the attack, but

training data distribution is important

q Data privacy should be considered in collaborative inference system design

30

Code available

https://github.com/zechenghe/Inverse_Collaborative_Inference

Questions?

31