Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md....
-
Upload
iris-simmons -
Category
Documents
-
view
218 -
download
1
Transcript of Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md....
![Page 1: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/1.jpg)
Analyzing Memory Analyzing Memory Accesses in Accesses in
Obfuscated x86 Obfuscated x86 ExecutablesExecutables
Michael VenableMohamed R. Choucane
Md. Enamul KarimArun Lakhotia (Presenter)
DIMVA 2005Wien
![Page 2: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/2.jpg)
Talk ContentsTalk Contents
Motivation
Previous work
VSA (Balakrishnan and Reps, 2004)
Abstract Stack Graph (Lakhotia and Kumar 2004)
VSA+ASG
Domain
Interpretation example
Application
Detect Call obfuscations
Implementation
Future and Conclusions
![Page 3: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/3.jpg)
MotivationMotivation
• Past project: detecting malicious behavior– Match models of malicious behavior
• models typically involved system calls
• Typical application of accepted techniques– implemented prototype using IDA Pro– standard application of formal theory
(model checking)– tried against common virus (Win32.Evol)
![Page 4: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/4.jpg)
Result: Result: FAILUREFAILURE
• Our original tool didn’t stand a chance– our technology wasn’t even tested– virus writer fouled up the entire analysis pipeline
• Success out of failure:– showed us where a major battle lies– refocus on hardening analysis techniques
![Page 5: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/5.jpg)
Code analysis: benign/adversarialCode analysis: benign/adversarial
• Half a century of program analysis– compilers, optimizers, checkers, refactoring tools– read in programs, analyze, visualize, transform
• Mostly for friendly programs– yet these are often used to battle malware– not prepared to face adversarial code
• soft targets • fragile infrastructures
– would we send Mouseketeers to battlefield?
![Page 6: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/6.jpg)
certify /reject
disassemble
Typical analysis pipelinesTypical analysis pipelines
extractprocedures
extract control& data flow
verifyproperty
VIRUS DATABASE
![Page 7: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/7.jpg)
certify /reject
disassemble
Problem: Not hardenedProblem: Not hardened
extractprocedures
extract control& data flow
verifyproperty
DATABASE
SILENTFAILURE!
D I S A B L E D !D I S A B L E D !
![Page 8: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/8.jpg)
AttackAttack: Disassembly: Disassembly
decode machine instructions (byte seq)
disassemble extractprocedures
extract control& data flow
verifyproperty
401063: 5d pop %ebp401064: c3 ret 401065: 55 push %ebp401066: 89 e5 mov %esp,%ebp401068: 83 ec 08 sub $0x8,%esp40106b: eb 05 jmp 0x40107240106d: e8 ee ff ff ff call 0x401060401072: e8 e9 ff ff ff call 0x401060401077: c7 45 fc 00 00 00 00 movl $0x0,0xfffffffc(%ebp)40107e: 81 7d fc e7 03 00 00 cmpl $0x3e7,0xfffffffc(%ebp)
ORIG BYTES ASSEMBLY
401063: 5d pop %ebp401064: c3 ret 401065: 55 push %ebp401066: 89 e5 mov %esp,%ebp401068: 83 ec 08 sub $0x8,%esp40106b: eb 05 jmp 0x40107240106d: c7 ee ff ff ff e8 mov $0xe8ffffff,%esi401073: e9 ff ff ff c7 jmp 0xc8401077401078: 45 inc %ebp401079: fc cld
malicious func
jump over junkbad disassembly(no jump target)
![Page 9: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/9.jpg)
AttackAttack: Extract procedures: Extract procedures
determine procedure boundaries (entry & exit)
disassemble extractprocedures
extract control& data flow
verifyproperty
401063: 5d pop %ebp401064: c3 ret 401065 <_malicious>:401065: 55 push %ebp401066: 89 e5 mov %esp,%ebp401068: 83 ec 08 sub $0x8,%esp40106b: eb 05 jmp 401072 <_malicious+0xd>40106d: e8 ee ff ff ff call 401060 <_sendLotsOfEmail>401072: e8 e9 ff ff ff call 401060 <_sendLotsOfEmail>401077: c7 45 fc 00 00 00 00 movl $0x0,0xfffffffc(%ebp)
malicious func
![Page 10: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/10.jpg)
401063: 5d pop %ebp401064: c3 ret 401065 <_malicious>:401065: 55 push %ebp401066: 89 e5 mov %esp,%ebp401068: 83 ec 08 sub $0x8,%esp40106b: ff 35 78 10 40 00 pushl 401078 <_malicious+0x13>401071: ff 35 60 10 40 00 pushl 401060 <_sendLotsOfEmail>401077: c3 ret 401078: c7 45 fc 00 00 00 00 movl $0x0,0xfffffffc(%ebp)
AttackAttack: Extract CF & DF: Extract CF & DF
trace call structure (control flow)
disassemble extractprocedures
extract control& data flow
verifyproperty
401063: 5d pop %ebp401064: c3 ret 401065 <_malicious>:401065: 55 push %ebp401066: 89 e5 mov %esp,%ebp401068: 83 ec 08 sub $0x8,%esp40106b: eb 05 jmp 401072 <_malicious+0xd>40106d: e8 ee ff ff ff call 401060 <_sendLotsOfEmail>401072: e8 e9 ff ff ff call 401060 <_sendLotsOfEmail>401077: c7 45 fc 00 00 00 00 movl $0x0,0xfffffffc(%ebp)
L0: call F L0: push L1L1: push F L1: ret
instr. substitution
no call found
![Page 11: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/11.jpg)
40106b: ff 35 78 10 40 00 pushl 401078 <_malicious+0x13>401071: ff 35 60 10 40 00 pushl 401060 <_sendLotsOfEmail>401077: c3 ret 401078: c7 45 fc 00 00 00 00 movl $0x0,0xfffffffc(%ebp)
AttackAttack: Verify property: Verify property
verify security or match pattern/signature
• Transformations destroy signature/pattern match– eg metamorphic viruses: self-transforming– instruction substitution, nop insertion, etc.
disassemble extractprocedures
push x push xpush y push zret pop push y
ret
extract control& data flow
verifyproperty
![Page 12: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/12.jpg)
Motivation summaryMotivation summary
• Soft target attacks– disassembler disruption– obfuscated calls– metamorphic transformations
• Pipeline disruptions– silent failures from every stage
![Page 13: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/13.jpg)
A shift in research focusA shift in research focus
Question
How to make program analysis battle-ready?
![Page 14: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/14.jpg)
Advertisement: Portfolio of resultsAdvertisement: Portfolio of results
Create malware phylogeny
Deobfuscate Calls
Reverse self transformations
All results are “Patent pending”
![Page 15: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/15.jpg)
Talk ContentsTalk Contents
Motivation
Previous work
VSA (Balakrishnan and Reps, 2004)
Abstract Stack Graph (Lakhotia and Kumar 2004)
VSA+ASG
Domain
Interpretation example
Application
Detect Call obfuscations
Implementation
Future and Conclusions
![Page 16: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/16.jpg)
Value-Set AnalysisValue-Set Analysis
• Determines values of program variables statically
• RIC: Reduced Interval Congruence– Interval: [0,4] = {0, 1, 2, 3, 4}– RIC: 4[0,4] + 10 = {10, 14, 18, 22, 26}
• Can ensure memory accesses align to memory boundaries
![Page 17: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/17.jpg)
ExampleExample
MOV eax, 4CMP eax, ebxJE Label
SUB eax, 2ADD eax, 2
MOV [ADDR], eax
eax = 0[0,0]+4
eax = 0[0,0]+6
eax = 0[0,0]+2
eax = 4[0,1]+2
![Page 18: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/18.jpg)
OperationsOperations
• Easily supported operations– ADD– SUB– MOV– others
• Unsupported Operations– DIV– OR– AND– …
![Page 19: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/19.jpg)
VSA LimitationsVSA Limitations
• Assumes executable conforms to standard conventions
• Depends on control-flow graph
• We would like to remove these two requirements
![Page 20: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/20.jpg)
Abstract StackAbstract Stack
• Each node stores an instruction address
• Example:
L1: PUSH 20L2: PUSH 10L3: POP ebxL4: PUSH 5 20 L1
10 L2 5 L4
Actual Stack Abstract Stack
![Page 21: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/21.jpg)
TOP
TOP
TOP
TOP
Abstract Stack GraphAbstract Stack Graph
• Contains all possible abstract stack
• Example:
L1: PUSH 20L2: PUSH 10L3: POP ebxL4: PUSH 5 L1
L2 L4
Abstract Stack Graph
![Page 22: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/22.jpg)
Abstract Stack GraphAbstract Stack Graph
• Apply set of rules to detect obfuscations
• Limitations– Does not know register/memory contents– Can’t handle instructions like:
• sub esp, eax• mov esp, eax
– Can’t handle indirect jumps/calls
![Page 23: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/23.jpg)
Talk ContentsTalk Contents
Motivation
Previous work
VSA (Balakrishnan and Reps, 2004)
Abstract Stack Graph (Lakhotia and Kumar 2004)
VSA+ASG
Domain
Interpretation example
Application
Detect Call obfuscations
Implementation
Future and Conclusions
![Page 24: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/24.jpg)
Our GoalOur Goal
• VSA+ASG
• Achieve:– Support more instructions (ex. sub esp, eax)– Support indirect jumps/calls– Not rely on CFG– Do not assume binary obeys common standards
![Page 25: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/25.jpg)
DomainDomain
• RIC
• STACK-LOCATION
• VALUE := 2-tuple– RIC⊤
– P(STACK-LOCATION)⊤
• STATE := 3-tuple– REGISTER VALUE– STACK-LOCATION VALUE– STACK-LOCATION STACK-LOCATION
![Page 26: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/26.jpg)
Operation: AdditionOperation: Addition
+: VALUE × VALUE × STATE VALUEPairwise addition of both components of VALUE
+: RIC × RIC RIC+: STACK-LOCATION × STACK-LOCATION = NULL+: RIC × STACK-LOCATION P(STACK-LOCATION)
• RIC+RIC– Equals approximation of the sum of each value in the two
RICs
• Stack-location + Stack-location– Unable to do, since requires knowing actual addresses
![Page 27: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/27.jpg)
Operation: AdditionOperation: Addition
• Stack-location + RIC– Travel down the stack
N0
N1
N2
N0
N1
N2esp
esp+4
BEFORE AFTER
![Page 28: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/28.jpg)
Operation: SubtractionOperation: Subtraction
• RIC – RIC, Stack-location – Stack-location– Similar to addition
• Stack-location – RIC– Travel up the stack or add new nodes
N0
N1
N0
N1
N2
esp
esp – 4BEFORE AFTER
![Page 29: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/29.jpg)
Operation: Memory OperationsOperation: Memory Operations
• Push– Creates a new stack-location– Stores pushed value at stack-location
• Pop– Retrieves value at top of stack– Increments top of stack
![Page 30: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/30.jpg)
Operation: Memory OperationsOperation: Memory Operations
• Store– Places a value in memory
• Load– Retrieves a value from memory
• Other operations defined in paper
![Page 31: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/31.jpg)
Interpretation ExampleInterpretation Example
• Main– Pushes two values
onto stack– Calls Max
• Max– Places max of
parameters in eax
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
![Page 32: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/32.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N0})
![Page 33: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/33.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N1})
N0:
N1: (4, ⊤)
![Page 34: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/34.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N2})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
![Page 35: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/35.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N3})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 36: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/36.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (2, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N3})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 37: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/37.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (2, ⊤)ebx = (4, ⊤)esp = (⊤, {N3})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 38: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/38.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (2, ⊤)ebx = (4, ⊤)esp = (⊤, {N3})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 39: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/39.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (2, ⊤)ebx = (4, ⊤)esp = (⊤, {N3})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 40: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/40.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (4, ⊤)ebx = (4, ⊤)esp = (⊤, {N3})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 41: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/41.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (2[0,1]+2, ⊤)ebx = (4, ⊤)esp = (⊤, {N0})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 42: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/42.jpg)
Interpretation ExampleInterpretation Example
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 RET 8
STATEeax = (2[0,1]+2, ⊤)ebx = (4, ⊤)esp = (⊤, {N0})
N0:
N1: (4, ⊤)
N2: (2, ⊤)
N3: (103, ⊤)
![Page 43: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/43.jpg)
Talk ContentsTalk Contents
Motivation
Previous work
VSA (Balakrishnan and Reps, 2004)
Abstract Stack Graph (Lakhotia and Kumar 2004)
VSA+ASG
Domain
Interpretation example
Application
Detect Call obfuscations
Implementation
Future and Conclusions
![Page 44: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/44.jpg)
ApplicationApplication
• Detect– Call replaced with equivalent instruction(s)– Return address manually removed from stack
• Flag as obfuscation if– instruction retn AND topOfStack.creator call– instruction pop AND topOfStack.creator call
• Rules are applied to each instruction
![Page 45: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/45.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N0})
![Page 46: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/46.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N1})
N1: (4, ⊤)100
![Page 47: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/47.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N2})
N1: (4, ⊤)100
N2: (2, ⊤)101
![Page 48: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/48.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 49: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/49.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 50: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/50.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (2, ⊤) ebx = (⊤, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 51: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/51.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (2, ⊤) ebx = (4, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 52: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/52.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (2, ⊤) ebx = (4, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 53: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/53.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (2, ⊤) ebx = (4, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 54: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/54.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (4, ⊤)ebx = (4, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 55: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/55.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (2[0,1]+2, ⊤)ebx = (4, ⊤)esp = (⊤, {N0})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
![Page 56: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/56.jpg)
Replace CALL with JMPReplace CALL with JMP
VAR1 EQU 4VAR2 EQU 2
Main:100 PUSH VAR1101 PUSH VAR2102 PUSH 104103 JMP Max104 …
Max:105 MOV eax, [esp+4]106 MOV ebx, [esp+8]107 CMP eax, ebx108 JG Done109 MOV eax, ebx
Done:110 RET 8
STATE
N0:
eax = (2[0,1]+2, ⊤)ebx = (4, ⊤)esp = (⊤, {N0})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (104, ⊤)102
Return address placed on stack by push
![Page 57: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/57.jpg)
Removal of Return AddressRemoval of Return Address
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N0})
![Page 58: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/58.jpg)
Removal of Return AddressRemoval of Return Address
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N1})
N1: (4, ⊤)100
![Page 59: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/59.jpg)
Removal of Return AddressRemoval of Return Address
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N2})
N1: (4, ⊤)100
N2: (2, ⊤)101
![Page 60: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/60.jpg)
Removal of Return AddressRemoval of Return Address
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (⊤, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
![Page 61: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/61.jpg)
Removal of Return AddressRemoval of Return Address
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (2, ⊤)ebx = (⊤, ⊤)esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
![Page 62: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/62.jpg)
Removal of Return AddressRemoval of Return Address
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (2, ⊤) ebx = (4, ⊤) esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
![Page 63: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/63.jpg)
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (2, ⊤) ebx = (4, ⊤) esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
Removal of Return AddressRemoval of Return Address
![Page 64: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/64.jpg)
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (2, ⊤) ebx = (4, ⊤) esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
Removal of Return AddressRemoval of Return Address
![Page 65: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/65.jpg)
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (4, ⊤) ebx = (4, ⊤) esp = (⊤, {N3})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
Removal of Return AddressRemoval of Return Address
![Page 66: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/66.jpg)
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (2[0,1]+2, ⊤) ebx = (4, ⊤) esp = (⊤, {N2})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
Removal of Return AddressRemoval of Return Address
![Page 67: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/67.jpg)
Main:100 PUSH 4101 PUSH 2102 CALL Max103 …
Max:104 MOV eax, [esp+4]105 MOV ebx, [esp+8]106 CMP eax, ebx107 JG Done108 MOV eax, ebx
Done:109 POP ebx110 ADD esp, 8111 JMP ebx
STATE
N0:
eax = (2[0,1]+2, ⊤) ebx = (4, ⊤) esp = (⊤, {N2})
N1: (4, ⊤)100
N2: (2, ⊤)101
N3: (103, ⊤)102
Return address is popped off stack
Removal of Return AddressRemoval of Return Address
![Page 68: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/68.jpg)
Talk ContentsTalk Contents
Motivation
Previous work
VSA (Balakrishnan and Reps, 2004)
Abstract Stack Graph (Lakhotia and Kumar 2004)
VSA+ASG
Domain
Interpretation example
Application
Detect Call obfuscations
Implementation
Future and Conclusions
![Page 69: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/69.jpg)
OllyDbg
Eclipse
PrototypePrototype
• Implemented on Eclipse platform
Disassembler
Parser
Interpreter
Analyzer
Binary image
Assembly instructions
AST
AST annotated with state
Set of obfuscations
![Page 70: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/70.jpg)
Obfuscations shown as list
Obfuscations shown on ruler
Register and stack contents
![Page 71: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/71.jpg)
![Page 72: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/72.jpg)
![Page 73: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/73.jpg)
![Page 74: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/74.jpg)
![Page 75: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/75.jpg)
![Page 76: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/76.jpg)
![Page 77: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/77.jpg)
![Page 78: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/78.jpg)
![Page 79: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/79.jpg)
![Page 80: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/80.jpg)
![Page 81: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/81.jpg)
![Page 82: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/82.jpg)
![Page 83: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/83.jpg)
![Page 84: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/84.jpg)
Pushes return address and function address
![Page 85: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/85.jpg)
Talk ContentsTalk Contents
Motivation
Previous work
VSA (Balakrishnan and Reps, 2004)
Abstract Stack Graph (Lakhotia and Kumar 2004)
VSA+ASG
Domain
Interpretation example
Application
Detect Call obfuscations
Implementation
Future and Conclusions
![Page 86: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/86.jpg)
Prototype LimitationsPrototype Limitations
• More Memory Support– mov [eax], ebx– GetModuleHandle/GetProcAddress
PUSH offset DLL_NAME ; “kernel32.dll”CALL GetModuleHandleA
PUSH offset PROC_NAME ; “ExitProcess”PUSH eaxCALL GetProcAddress
PUSH 0CALL eax
![Page 87: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/87.jpg)
Prototype LimitationsPrototype Limitations
• Efficiency– Many paths to each instruction– More efficient algorithms needed 1
1 1
2
2 2
4
4 4
8
8 8
16
CFG with branches
![Page 88: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/88.jpg)
General LimitationsGeneral Limitations
• More RIC operations– Too many undefined RICs
• What about obfuscated call–obfuscated return pairs– Not a problem with system calls
![Page 89: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/89.jpg)
General LimitationsGeneral Limitations
• Structure Exception Handling
Main: PUSH offset Handler ; set up exception handler PUSH dword ptr FS:[0] MOV FS:[0], esp
MOV edx, 0 ; force an exception MOV eax, 0 DIV eax ; (edx / eax)
PUSH 0 CALL ExitProcess
Handler: ... ; real code goes here
![Page 90: Analyzing Memory Accesses in Obfuscated x86 Executables Michael Venable Mohamed R. Choucane Md. Enamul Karim Arun Lakhotia (Presenter) DIMVA 2005 Wien.](https://reader038.fdocuments.in/reader038/viewer/2022103006/56649f285503460f94c41172/html5/thumbnails/90.jpg)
ConclusionConclusion
• Method– Combines VSA+ASG– Interpret using abstract values– Apply rules to abstract stack graph to detect obfuscations
• Results– Can handle more instructions (mov esp, eax)– Can handle indirect jumps/calls– Anticipate support for GetModuleHandle/GetProcAddress
• Prototype performed as expected, though more work is needed