Matryoshka: Strengthening Software Protection via Nested Virtual Machines

Sudeep Ghosh (Microsoft Research) Jason D. Hiser (University of Virginia)

Jack W. Davidson (University of Virginia)

The Problem

2

0101110 00111001010 001 01011001000110001110110 0101110110110010111010 101101000101101111111 100010101101100111110 01010111001110010111 1 11111111111111110

•  Disassembler •  Debuggers •  Static Analyzers

•  Coverage tools •  Simulators

Software is now used to perform critical functionality.

•  Banks •  Power grids •  Medical software •  Transportation systems •  Internet of Things Researchers must continually upgrade and

enhance software protection approaches

Threat Model

n  Software is created in a secure environment. n  White-box attack model

n  Adversary has access to multiple tools including debuggers, simulators and emulators.

n  They can modify the OS to return inaccurate information. n  As such, the adversary can inspect, modify or forge any

information. n  Given enough time and resources, the adversary can succeed in

manually inspecting and modifying programs. n  Boaz Barak , Oded Goldreich , Russell Impagliazzo , Steven Rudich , Amit

Sahai , Salil P. Vadhan , Ke Yang: On the (Im)possibility of Obfuscating Programs, Proceedings of the 21st Annual International Cryptology Conference on Advances in Cryptology, 2001.

n  However, most attacks use algorithmic solutions to disable security features in programs.

3

Virtualization

Virtualizing Software (e.g. Xen)

Operating System

Hardware

Applications

System-level VM

Hardware

Process-level VM

Applicationn Application1

Operating System

Virtualizing Software

(e.g. Strata)

Virtualizing Software

(e.g. Strata)

4

Strata

n  Infrastructure designed for building process-level virtualization systems

n  Designed with extensibility, portability, and application-independence in mind

n  Implement new systems by customizing the VM

n  Binary only n  No source code required n  Can be applied to any

application regardless of source language, compiler used, libraries used, etc.

n  Provides common services necessary for software dynamic translation

5

Process-level Virtualization

Context Capture

Context Switch Next PC

Translate Decode Fetch

New Fragment

Finished?

Dynamic Translator

Cached? New PC

Application Binary

System Start (first PC)

F$

Direct Conditional branch Trampoline Non-control instruction

PC

7

See: Hiser et al., Fragment Cache Construction Policies for Software Dynamic Translation Systems, VEE 2006.

Strata

0

0.5

1

1.5

2

2.5

Ove

rhea

d N

orm

aliz

ed t

o N

ativ

e R

un

DynamoRIO

PIN

HDTrans

Strata

8

Strata Related Work n  SDT

n  Evaluating Indirect Branch Handling Mechanisms in Software Dynamic Translation Systems. CGO 2007.

n  Evaluating Fragment Construction Policies for Software Dynamic Translation Systems. VEE 2006.

n  Retargetable and Reconfigurable Software Dynamic Translation. CGO 2003. n  Obfuscation and anti-tamper

n  What’s the PointISA? IH 2014. n  Replacement Attacks against VM-protected Applications. VEE 2006. n  A Secure and Robust Approach to Software Tamper Resistance. IH 2010

n  Security n  ILR: Where did my gadgets go? Oakland 2012. n  Secure and Practical Defense Against Code-injection Attacks Using Software

Dynamic Translation. VEE 2006. n  Safe Virtual Execution Using Software Dynamic Translation, ACSAC 2002.

9

Matryoshka: Nested PVMs

10

APPLICATION

PVM2 PVM4

dPVM1 PVM3

G1

G2

G3

G4

Software Protection via Virtualization

Application

Protection Scheme (Guards,

encryption)

Builder (Diablo)

Protected Binary

11

PVMs

PVM1 PVM2 PVM2

Nested PVMs Disk Image

12

F$ (SC1)

Strata1

Strata2

Translated Application Code

Memory

X86 (Hardware)

F$ (SC2)

Translated Application Code

Translated Application Code

Evaluation

F$ Diversity

13

Software Cache Addresses App

Strata2

n  Use compression ratio as a proxy for diversity and obfuscation.

n  Single PVM: 149; N-PVM: 15.63

Evaluation

Cyclomatic Complexity n  Developed by McCabe in 1976 as a measure

of software complexity (TSE Vol. 2, No. 4) n  M = E – N + 2P

14

Benchmark CC for PVM CC for N-PVM Increase

176.gcc 1,604 80,109 49X

181.mcf 351 9828 27X

256.perlbmk 803 32,903 40X

179.Art 181 5,130 27X

Evaluation

Run-time Overhead n  With a nesting level of two, the base run-time

overhead was 35X. n  The problem is trampoline patching (i.e., self-

modifying code), which causes excessive F$ flushes.

15

Nested PVMs Disk Image

17

F$ (SC1)

Strata!

Strata2

Translated Application Code

Memory

X86 (Hardware)

F$ (SC2)

Translated Application Code Translated Application Code

Translated Application Code

Translated Application Code

Super Patching

n  When Stratan patches a trampoline, the patch information is sent to Stratan-1

n  When a patched (in F$2) target block is translated to F$n-1 by Stratan-1, Stratan-1 patches its F$ (F$n-1), thereby avoid the F$ flush

18

Super Patching Overhead

19

0

0.5

1

1.5

2

2.5

3

3.5

4

164

.gzip

175

.vpr

176

.gcc

181

.mcf

197

.par

ser

253

.per

lbm

k

256

.bzip

2

300.

twol

f

177

.mes

a

179

.art

183

.equ

ake

188

.am

mp AV

G

Strata N-PVM with Super-patching

Perf

orm

ance

Ove

rhea

d (n

orm

alize

d to

nat

ive)

Related Work

n  Collberg and Nagra [Pearson 2006] provide an excellent overview of the area

n  Anckaert et al [DRM 2006] showed the promise of virtualization for software protection

n  Themida [1] and VMProtect [2] use interpreted virtual machines for software protection. Nested VMs apply to them as well.

n  Anckaert et al [QoP 2007] discuss metrics for metrics for software protection

20

Summary

n  Nested PVMs can significantly increase the complexity of software that is the target of crackers

n  More research is needed to determine when and how to apply nested VMs to software to balance run-time performance and the strength of the protection provided

21