Malware Obfuscation Techniques:...
Transcript of Malware Obfuscation Techniques:...
Malware Obfuscation Techniques: Packing
November 18, 2014
Malware and packing
Not packed (20%) •
80% of new malware are packed with various packers
50% of new malware samples are simplyrepacked versions of existing malware
Malware Obfuscation Techniques: Packing 2
Malware and packing
Not packed (20%) •
80% of new malware are packed with various packers
50% of new malware samples are simplyrepacked versions of existing malware
Malware Obfuscation Techniques: Packing 2
Code packing
I A technique to hide the real code of a program through one ormore layers of compression/encryption
I At run-time the unpacking routine restores the original code inmemory and then executes it
Maliciouscode
The effectiveness of malware detectors depends on the abilityto recover the “real” malicious code, but recovery often fails!
Malware Obfuscation Techniques: Packing 3
Code packing
I A technique to hide the real code of a program through one ormore layers of compression/encryption
I At run-time the unpacking routine restores the original code inmemory and then executes it
Maliciouscode
The effectiveness of malware detectors depends on the abilityto recover the “real” malicious code, but recovery often fails!
Malware Obfuscation Techniques: Packing 3
Code packing
I A technique to hide the real code of a program through one ormore layers of compression/encryption
I At run-time the unpacking routine restores the original code inmemory and then executes it
Maliciouscode
Maliciouscode
Unpackingroutine
The effectiveness of malware detectors depends on the abilityto recover the “real” malicious code, but recovery often fails!
Malware Obfuscation Techniques: Packing 3
Code packing
I A technique to hide the real code of a program through one ormore layers of compression/encryption
I At run-time the unpacking routine restores the original code inmemory and then executes it
Maliciouscode
Maliciouscode
Unpackingroutine
Unpackingroutine
The effectiveness of malware detectors depends on the abilityto recover the “real” malicious code, but recovery often fails!
Malware Obfuscation Techniques: Packing 3
Code packing
I A technique to hide the real code of a program through one ormore layers of compression/encryption
I At run-time the unpacking routine restores the original code inmemory and then executes it
Maliciouscode
Maliciouscode
Unpackingroutine
Unpackingroutine
The effectiveness of malware detectors depends on the abilityto recover the “real” malicious code, but recovery often fails!
Malware Obfuscation Techniques: Packing 3
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program
????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program
????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed code
Unpacked code
Malware Obfuscation Techniques: Packing 4
Traditional approaches to deal with packed code
Algorithmic unpacking
Use of specific unpacking routines to recover the original code (i.e.,one routine per packing algorithm)
Malicious program ????
????
Malware detector
Malicious
Generic unpacking
Emulation/tracing of the execution until the unpacking routineterminates (e.g., PolyUnpack [ACSAC 06] and Renovo [WORM 07])
Packed codeUnpacked code
Malware Obfuscation Techniques: Packing 4
A simple generic unpacker
I Track all memory writes and the program counter
I The execution of a previously written memory location denotesthe end of an unpacking stage
I All written-then-executed memory locations should then beanalyzed by a malware detector
Extend this idea to design an iterative unpackingalgorithm that achieves low overhead yet does not
compromise the security of the system
Malware Obfuscation Techniques: Packing 5
A simple generic unpacker
I Track all memory writes and the program counter
I The execution of a previously written memory location denotesthe end of an unpacking stage
I All written-then-executed memory locations should then beanalyzed by a malware detector
Extend this idea to design an iterative unpackingalgorithm that achieves low overhead yet does not
compromise the security of the system
Malware Obfuscation Techniques: Packing 5
Goals of Real-Time Unpackers
I Generic unpacking with low-overhead by using existinghardware mechanisms
I Precise unpacking by running the program on the native OS
I A new malware detection strategy, independent of packing,where the malware detector analyzes new pieces of code beforethey are executed.
Malware Obfuscation Techniques: Packing 6
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Malicious code
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Malicious code
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
Efficient tracking of memory accesses
Coarse-grained memory access tracking (at page level), throughthe use of hardware mechanisms
Malicious code
Mem
ory
Executed page
Written page
Executed memory location
Written memory location
Unfortunately...
I Written-then-executed locations are indicative of unpacking butnot indicative of the end of unpacking
I Coarse-grained memory accesses tracking further increases thechances to detect spurious unpacking stages (up to hundreds ofthousands stages)
Malware Obfuscation Techniques: Packing 7
The overhead introduced by invoking the malware detectorevery time a written page is executed is prohibitive!
Better approximating the end of an unpacking stage
Ideally:
Scan
Start
Halt
With coarse-grained memory access tracking:
Start Scan Scan Scan
Halt
Mitigate the imprecision of the coarse-grained memory accessestracking by considering an unpacking stage concluded when theexecution of a previously written page is followed by a dangeroussystem call
Start Scan
Halt
Malware Obfuscation Techniques: Packing 8
Better approximating the end of an unpacking stage
Ideally:
Scan
Start
Halt
With coarse-grained memory access tracking:
Start Scan Scan Scan
Halt
Mitigate the imprecision of the coarse-grained memory accessestracking by considering an unpacking stage concluded when theexecution of a previously written page is followed by a dangeroussystem call
Start Scan
Halt
Malware Obfuscation Techniques: Packing 8
Better approximating the end of an unpacking stage
Ideally:
Scan
Start
Halt
With coarse-grained memory access tracking:
Start Scan Scan Scan
Halt
Mitigate the imprecision of the coarse-grained memory accessestracking by considering an unpacking stage concluded when theexecution of a previously written page is followed by a dangeroussystem call
Start Scan
Halt
Malware Obfuscation Techniques: Packing 8
Dangerous system calls
To achieve its malicious goals, the malware has to interact withthe system (through system calls)
Only few system calls are dangerous
A system call is dangerous if its execution can leave the system inan unsafe state
Start • • • Scan
Halt
NtOpenFile NtOpenKey NtDeleteFile
Malware Obfuscation Techniques: Packing 9
Dangerous system calls
To achieve its malicious goals, the malware has to interact withthe system (through system calls)
Only few system calls are dangerous
A system call is dangerous if its execution can leave the system inan unsafe state
Start • • • Scan
Halt
NtOpenFile NtOpenKey NtDeleteFile
Malware Obfuscation Techniques: Packing 9
Dangerous system calls
To achieve its malicious goals, the malware has to interact withthe system (through system calls)
Only few system calls are dangerous
A system call is dangerous if its execution can leave the system inan unsafe state
Start • • • Scan
Halt
NtOpenFile NtOpenKey NtDeleteFile
Malware Obfuscation Techniques: Packing 9
Unpacker algorithm
Input: an execution trace 〈e0, e1, . . .〉
where a trace event can be:
w(p) write access to a memory page p
x(p) instruction execution from a memory page p
s invocation of the system call s
Malware Obfuscation Techniques: Packing 10
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1), x(1), s1, x(2), s2, . . .〉
.
Memory pages status
Page Access# W WX012
. . .
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈 x(0) , w(2), s0, w(1), x(1), s1, x(2), s2, . . .〉
.
Memory pages status
Page Access# W WX012
. . .
The memory page 0 is executed
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2) , s0, w(1), x(1), s1, x(2), s2, . . .〉
.
Memory pages status
Page Access# W WX012 •
. . .
The memory page 2 is writtenThe page is recorded in the set W of written pages
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0 , w(1), x(1), s1, x(2), s2, . . .〉
s0 is NtOpenFile
Memory pages status
Page Access# W WX012 •
. . .
The system call s0 is executed (not dangerous and WX is empty)
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1) , x(1), s1, x(2), s2, . . .〉
.
Memory pages status
Page Access# W WX01 •2 •
. . .
The memory page 1 is writtenThe page is recorded in the set W of written pages
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1), x(1) , s1, x(2), s2, . . .〉
.
Memory pages status
Page Access# W WX01 • •2 •
. . .
The memory page 1 is executedThe page is recorded in the set WX of written-then-executed pages
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1), x(1), s1 , x(2), s2, . . .〉
s1 is NtOpenKey
Memory pages status
Page Access# W WX01 • •2 •
. . .
The system call s1 is executed (not dangerous)
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1), x(1), s1, x(2) , s2, . . .〉
.
Memory pages status
Page Access# W WX01 • •2 • •
. . .
The memory page 2 is executedThe page is recorded in the set WX of written-then-executed pages
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1), x(1), s1, x(2), s2 , . . .〉
s2 is NtDeleteFile
Memory pages status
Page Access# W WX01 • •2 • •
. . .
The system call s2 is executed (dangerous)
The malware detector is invoked to scanall the memory pages in W
Malware Obfuscation Techniques: Packing 11
Unpacker algorithm
Execution trace〈x(0), w(2), s0, w(1), x(1), s1, x(2), s2, . . . 〉
.
Memory pages status
Page Access# W WX012
. . .
If the program is not malicious the sets W and WX are emptiedand the execution is resumed
Malware Obfuscation Techniques: Packing 11
OmniUnpack for Microsoft Windows XP
Kernel
User
Suspiciousprogram
ClamAVMalwaredetector
OmniUnpackkerneldriver
Memoryaccessmonitor
System-callmonitor
Malware Obfuscation Techniques: Packing 12
OmniUnpack for Microsoft Windows XP
Kernel
User
Suspiciousprogram
ClamAVMalwaredetector
OmniUnpackkerneldriver
Memoryaccessmonitor
System-callmonitor
I The W ⊕ X policy is enforced on the memory pages of thesuspicious program
I Page-fault exceptions are trapped by OmniUnpack
I Non executable pages can be emulated via software
Malware Obfuscation Techniques: Packing 12
OmniUnpack for Microsoft Windows XP
Kernel
User
Suspiciousprogram
ClamAVMalwaredetector
OmniUnpackkerneldriver
Memoryaccessmonitor
System-callmonitor
I Any malware detection strategy can be used to scan the codegenerated during the previous stage
Malware Obfuscation Techniques: Packing 12
Food for Thoughts & Exercises
I Try to find out a method in order to evade Omninpacksystem
I Following the parassite developed for the last homeworktry to patch the got table on-the-fly and wrap some functionand logs the parameters.
I Add a layer of protection to the parassite against thestatic analysis the parassite should be able to unpack yourselfduring the execution of the binary.
Malware Obfuscation Techniques: Packing 13
Q&A
Thank You!Q&A?
Malware Obfuscation Techniques: Packing 14