Statistical Tools for Linking Statistical Tools for Linking Engine-generated Malware to its Engine-generated Malware to its

EngineEngine

Edna C. MilgoEdna C. MilgoM.S. Student in Applied Computer ScienceM.S. Student in Applied Computer Science

TSYS School of Computer ScienceTSYS School of Computer Science

Columbus State UniversityColumbus State UniversityNovember 19November 19thth, 2009, 2009

MalwareMalware

State of the Threat: Anti-virus firms are having to analyze

between 15,000 and 20,000 new malware instances a day. [AV Test Lab 2007, McAfee2009]

1.6 million malware instances were detected in 2008. [F-Secure2008]

Professionals are being recruited to make stealthier malware. [ESET2008]

Automation is being used to generate malware. [ESET2008]

Generic Detection is not good. 630 out of 1000 instances of new malware went unnoticed. [Team-cymru2008]

Classes: Viruses, Worms, TrojansMalware-generating Engines: Script Kiddies, Morphers, Metamorphic, and Virus Generating Toolkits

2/20

• Static Program Analysis:Static program analysis may be imprecise and

inefficient (e.g., Def-Use analysis).

Static program analysis may be challenged by obfuscation.• Dynamic Program Analysis:May be challenged by testing the patience of the emulator.[Aycock2005]

Extract Procedures

DisassemblySignature Verification

Control Flow Graphs

Malicious

Benign

Suspect Program

Malware is Hard to Detect as Malware is Hard to Detect as it is…it is…

Read x

Entry

Call Proc 1

Call Proc 2

Call Proc 2

Call Proc 1

Exit

YESNO

Dead code

insertion

If (x*x* (x+1) *(x+1) % 4==0)

…Stop early in the program analysis pipeline.

3/20

Variant 2Variant

1Variant 3 Variant n

IN

OUT

Engines-Generated Engines-Generated MalwareMalware

ENGINE

Variant 0

MALWARE DETECTOR

Engine generates new variants at a high rate.Malware detectors typically store one signature per variant. Too many signatures challenge the detector.

4/20

Proposal: View Engine as Proposal: View Engine as AuthorAuthorGoal 1: Reduce the number of steps required in the

program analysis pipeline.Goal 2: Eliminate the need for signature per variant.Goal 3: Must be satisfactorily accurate.

Proposed Model [Chouchane2006]

ENGINE

Variant 2Variant

1Variant 3 Variant n

MALWARE DETECTOR

Engine Signatu

re

OUT

Source: Google Images

Proposed approach was inspired by Keselj’s work on authorship analysis of natural text produced by humans. [V. Keselj2003]

5/20

Example Program: P

IFV(P) Normalized IFV(P)

Feature 1: Instruction Frequency Feature 1: Instruction Frequency VectorVector

6

add,push,pop,add,and,jmp,pop,and,mov,jmp,mov,push,jmp,jmp,push, jmp,add,pop,mov,add,mov,push,jmp,mov,mov,jmp,push

mov push add and jmp pop

0.22 0.19 0.15 0.07 0.26 0.11

mov push add and jmp pop

6 5 4 2 7 3

6/20

IFV ClassificationIFV Classification

threshold

Malicious

Benign

Suspect program

STEPS1. Given a sample of malicious

programs and a sample of benign ones select sets of trainers from each sample.

2. Compute the IFVs of all trainers.3. Choose a threshold ε.4. Input : IFVsuspect , where ‘suspect’

is a program that is not among the trainers.

5. Count the number of malicious training IFVs within ε of IFVsuspect .

6. Count the number of benign training IFVs within ε of IFVsuspect .

7. Output: The family that has the highest number of trainers within ε is declared to be that of the suspect program. If there is a tie, pick one at random.

7/20

Distance Measure

1

0

2iyixyx ))(IFV)((IFV)IFV,d(IFV

n

i

Experimental SetupExperimental Setup

Metamorphic Malware (from vx.netlux.org)

W32.Simile (100 samples)

Benign Programs (from download.com, sourceforge.net)

(100 samples)

8/20

Thanks to Jessica Turner for extracting the original variant of W32.Simile.

Classifying W32.Simile vs. Classifying W32.Simile vs. BenignsBenigns

RI is the number of instructions considered in the IFVFor RI=4: 0.1 ≤ ε ≤ 0.7, 98 % ≤ accuracy ≤ 100% For RI=5: 0.1 ≤ ε ≤ 0.7, 96 % ≤ accuracy ≤ 100% Very small signatures (4 and 5 doubles per IFV) But does not use single signature

9/20

Feature 2: N-gram Frequency Feature 2: N-gram Frequency VectorVector

Example Program: P

NFV(P) Normalized NFV(P)addpush pushcall callpop popcall callmov movadd

3 5 4 6 1 7addpush pushcall callpop popcall callmov movadd

0.12 0.19 0.15 0.23 0.04 0.27

add,push,call,pop,call,add,push,call,pop,call,add,mov,add,add,mov,add,add, mov,add,push,call,push,call,call,pop,call,push,mov,add,mov,add,push,call,pop,pop,call, pop,call,pop,call,mov,add,mov,add

10/20

N-Gram Authorship Attribution N-Gram Authorship Attribution (Proposed)(Proposed) Malware Detector

DS

DE

DV

DN

FSB

FSS

FSE

FSV

FSN

STEPS1. Choose a set of trainers from each of

the families. 2. For each family, compute the

average of the NFVs of the family’s trainers to create a Family Signature(FS) for each family.

3. Input : NFVsuspect , where ‘suspect’ is a program that is not among the trainers.

4. Compute the distance between each of the FS’s and NFVsuspect.

5. Output: The suspect program classified as a member of the family with the shortest distance. If there are ties, choose one at random.

NFVsuspect

11/20

Prediction for = MIN (DB,DS,DE,DV,DN)

DB

Distance Measure2

1)

)()(

)()(*2(),(

nm

iiyix

iyixyx NFVNFV

NFVNFVNFVNFVd

k-nn Classification k-nn Classification STEPS

1. Given a sample of malicious programs and a sample of benign ones select sets of trainers from each sample.

2. Choose k > 0.3. Input: NFVsuspect of suspect

program.4. Find the k closest training NFVs

(neighbors of NFVsuspect).5. Output: The suspect program

classified as a member of the family with the most neighbors. If there are ties, choose one at random.

NGVCK

Evol

VCL

Simile

Benign

12/20

Distance Measure2

1)

)()(

)()(*2(),(

nm

iiyix

iyixyx NFVNFV

NFVNFVNFVNFVd

Experimental SetupExperimental SetupMetamorphic Malware (from vx.netlux.org)

W32.Simile + W32.Evol (100 samples each)Malware Generation Toolkits (from

vx.netlux.org) VCL + NGVCK (100 samples each)

Benign Programs (from download.com, sourceforge.net). (100 samples)

13/20

Thanks to Yasmine Kandissounon for collecting the NGVCK and VCL variants.

Ten-fold Cross Ten-fold Cross ValidationValidation

Divide each family into a training set of 90 instances and a testing set of 10 instances.

Perform 10-fold cross validation using a new testing set and a new training set each time.

Cross accuracy equals the average accuracy across all the validation accuracies.

14/20

Bigram Selection (Relevant Bigram Selection (Relevant Instructions) Instructions)

RI is the number of most relevant instructions across the samples used to construct the features. Best Accuracy 85% for RI =3, RI=4 and RI=9.

15/20

Bigram Selection(Relevant Bigram Selection(Relevant Bigrams)Bigrams)

RB is the number of most relevant bigrams across the samples used to construct the features. Best accuracy 95% for 17 doubles.Accuracies of 94.8 % for 6, 8 and 14 doubles. 16/20

Successful Evaluation…Successful Evaluation…A single, small family signature of 17 doubles

for each family induced a 95% detection accuracy.

W32.Simile's Engine signature = (0.190, 0.030, 0.155, 0.048, 0.043, 0.057, 0.063, 0.020,0.076, 0.022, 0.0, 0.041, 0.109, 0.0, 0.122, 0.022, 0.0)

W32.Evol's Engine signature = (0.074, 0.026, 0.006, 0.326, 0.208, 0.014, 0.024, 0.073,0.043, 0.048, 0.0, 0.071, 0.042, 0.0, 0.026, 0.019, 0.0)

W32.VCL's Engine signature = (0.111, 0.238, 0.142, 0.027, 0.076, 0.063, 0.063, 0.033,0.009, 0.018, 0.018, 0.054, 0.042, 0.0, 0.040, 0.052, 0.013)

W32.NGVCK's Engine signature = ( 0.132, 0.113, 0.106, 0.048, 0.203, 0.018, 0.055,0.038, 0.022, 0.017, 0.070, 0.122, 0.007, 0.0, 0.007, 0.020, 0.017)

Benign's “Engine signature” = (0.165, 0.173, 0.091, 0.061, 0.052, 0.060, 0.052, 0.046, 0.060,0.028, 0.019, 0.043, 0.024, 0.029, 0.02, 0.031, 0.029)

A single, small family signature of 6 doubles for each family induced a 94.8% detection accuracy.

W32.Simile's Engine signature = (0.362, 0.058, 0.295 , 0.093 , 0.082, 0.110 )

W32.Evol's Engine signature = (0.113, 0.039, 0.010, 0.497, 0.319, 0.021 ) W32.VCL's Engine signature = (0.176, 0.358, 0.212 , 0.041, 0.115 , 0.100 ) W32.NGVCK's Engine signature = (0.212, 0.182, 0.171, 0.078, 0.327,

0.029 ) Benign's “Engine signature” = (0.265, 0.279, 0.147, 0.102 , 0.098,

0.098 )

17/20

MALWARE GENERATING

ENGINE

Variant 2

Variant 1

Variant 3Variant

n

MALWARE DETECTOR

ES

OUT

… Re-examining our goals Goal 1: Simplified analysis. Analysis involves only

the disassembly and signature verification stages of the program analysis pipeline.

Goal 2: One signature per family (Family Signature). Goal 3: Accuracy of 95% using only 17 doubles as a

signature.

Successful Evaluation Successful Evaluation cont’d…cont’d…

18/20

Directions for Future WorkDirections for Future WorkExperiment with other malware instances and

familiesAddress scalability issue?

Experiment with other feature selection methodsCould we do “better” than 95% for a signature of 17

doubles? Try other classifiers

Other distance measure?Try byte NFV’s instead of opcode NFV’s

Take into account malware that comes as binary. Import existing forensic linguistics methods to

malware detection 19/20

ReferencesReferences Paper documenting this work has already been submitted for possible

publication at the Journal in Computer Virology.• E. Milgo. A Fast Approximate Detection of Win.32 Simile Malware. Columbus

State University Colloquium Series, Feb.’09 and Best Paper Award, 2nd Place-Masters category: ACM MidSE, ’08 .

M. R. Chouchane, A. Lakhotia. Using Engine Signature to Detect Metamorphic Malware. WORM, ‘06.

M. R. Chouchane. Approximate Detection of Machine-morphed Malware. Ph.D. Dissertation, University of Louisiana at Lafayette, ’08.

P. Ször. The Art of Computer Virus Research and Defense. 2005. J. D. Aycock. Computer Viruses and Malware. 2005. V. Keselj, F. Peng, N. Cercone, and C. Thomas. N-gram-based Author Profiles for

Authorship Attribution. PACL, ’03. T. Abou-Assaleh, N. Cercone, V. Keselj, and R. Sweidan. N-gram Based Detection

of New Malicious Code. CMPSAC, ‘04. http://www.av-test.org/, 2007 . http://resources.mcafee.com/content/AvertReportQ109, 2009. http://www.eset.com, 2008. http://www.f-secure.com/en_US/, 2008 http://www.team-cymru.org, 2008.

20/20