1

Forecasting Cyber Attacks with ImbalancedData Sets and Different Time Granularities

Ahmet Okutan∗, Shanchieh Jay Yang∗, Katie McConky†∗Computer Engineering, Rochester Institute of Technology, Rochester, NY, USA.

†Industrial and Systems Engineering, Rochester Institute of Technology, Rochester, NY, USA.

Abstract—If cyber incidents are predicted a reasonable amount of time before they occur, defensive actions to prevent theirdestructive effects could be planned. Unfortunately, most of the time we do not have enough observables of the malicious activitiesbefore they are already under way. Therefore, this work suggests to use unconventional signals extracted from various data sourceswith different time granularities to predict cyber incidents for target entities. A Bayesian network is used to predict cyber attacks wherethe unconventional signals are used as indicative random variables. This work also develops a novel minority class over samplingtechnique to improve cyber attack prediction on imbalanced data sets. The results show that depending on the selected timegranularity, the unconventional signals are able to predict cyber attacks for the anonimyzed target organization even though the signalsare not explicitly related to that organization. Furthermore, the minority over sampling approach developed achieves betterperformance compared to the existing filtering techniques in the literature.

Index Terms—Cyber security, Bayesian networks, forecasting, unconventional signals.

F

1 INTRODUCTION

The number and diversity of cyber attacks have increasedrecently compared to the previous years. Denial of Service(DOS), Malware, attack on Internet facing service (Deface-ment), malicious Email, and malicious URL are some of theattack types that organizations and Internet users commonlyobserve. PwC reports that the number of security incidentsincreased by 38% in 2015 across all industries [1]. Predictionof cyber attacks before they are executed is very importantto take necessary defensive actions beforehand. This workproposes to forecast cyber attacks towards an entity byusing unconventional signals from various data sourcesthat may or may not be related to that target entity. Forunconventional signals, this paper uses social media, i. e.,Twitter and the open source GDELT [2] project that followsthe media from almost every corner of the world, in over100 languages, and almost every moment of each day.

The idea of using unconventional signals to forecastcyber events is based on the premise that the signals are notdirectly linked to specific exploits or vulnerabilities withinthe targeted organization. There might be many potentiallyuseful unconventional signals, but each signal may notbe particularly effective by itself. Furthermore, continuousenhancement to condition or aggregate these signals is nottrivial and requires a significant amount of research. Thiswork proposes a methodology based on Bayesian networksthat can treat a variety of unconventional signals to forecastevents that do not necessarily have balanced positive andnegative ground truth instances. The feasibility of the pro-posed approach is demonstrated using the selected uncon-ventional signals. Additional and potentially better signalsmay be used in conjunction with the method presented inthis paper.

The main motivation behind this work is to address three

important cyber security problems: Use unconventional sig-nals rather than the direct cyber observables to forecastcyber attacks, research the significance of changing timegranularities for signals, and improve the performance ofcyber attack forecasting on imbalanced cyber data sets. First,unlike some cyber attack prediction techniques which ana-lyze the results of cyber incidents or suggest approaches toreact to the attacks already underway, this research focuseson unconventional signals that have a potential to be indica-tive of a cyber incident towards a target entity. Bakdash etal. [3] emphasize the importance of moving from reactiveand passive defense systems to more proactive ones andshow that the frequency of attacks from the previous weekcan be used to predict the number of attacks for the nextweek. Ramakrishnan et al. [4] develop an automated andcontinuously running system named EMBERS (Early ModelBased Event Recognition using Surrogates) to forecast civilunrest events. They use open source indicators such astweets, news sources, blogs, economic indicators, and otherdata sources as evidence and show that EMBERS is able tosuccessfully forecast significant public unrest events across10 countries of Latin America.

While using unconventional signals to forecast futurecyber incidents, one needs to decide on the aggregationperiod of each signal and the entity ground truth, in order togenerate training and test sets for target entities. Assuming trepresents a time stamp and tx and tg represent time granu-larities for the unconventional signals and the entity groundtruth respectively, we investigate if signals aggregated overa period of time between t − tx and t could be indicativeof cyber events for the succeding time period starting at tand ending at t+ tg . For instance, if tx is one week and tg is24 hours, signals averaged over the last week could be usedto predict attacks that will occur in the next 24 hours. Ourprevious study [5] calculated unconventional signals on a

arX

iv:1

803.

0956

0v1

[cs

.CR

] 2

6 M

ar 2

018

2

daily basis and used them with the ground truth data ofthe following day to predict cyber attacks before they areobserved. This paper uses different time granularities forthe unconventional signals (tx) and the entity ground truth(tg) and shows that the performance of a prediction modelis dependent on the selected tx and tg .

Depending on the update frequency of a particulardata source, choosing a very small or large tx affects theindicative power or quality of a specific signal. Similarly,contingent upon the type or reputation of an entity, thenumber and frequency of cyber incidents observed for thatentity could be different. Moreover, some attack types couldbe observed more frequently compared to some other typeswithin an entity. Therefore, the training and test data setsof each attack type could be more balanced or imbalanceddepending on the chosen tg value. If a malicious emailattack is observed a couple of times per week, then traininga classifier with the ground truth data of each day (or 2 or3 days) could be a better approach compared to trainingwith the ground truth of a week. Similarly, if an attacktype is seen each and every day consistently, training amodel with attacks observed in 6 or 12 hours of granularitymight be better. Based on this intuition, this work developsa general cyber attack forecasting framework for entities,i. e., organizations and industries that uses different tx andtg time granularities to forecast cyber attacks for differentattack types.

The infrequency of certain types of attacks may causechallenges where the positive instances are much fewercompared to the negative instances (no attack). Dealing withimbalanced data sets is challenging, because it is not oftenpossible for a classifier to learn the minority class. Thereare several approaches to deal with imbalanced data sets.Apart from choosing a proper performance metric or clas-sifier, other approaches include changing the distribution ofthe instances, which is a very delicate process. Moreover,there are many accompanying questions that need to beanswered, like which instances to under or over sampleand how to generate synthetic new instances? This workproposes a new approach to improve cyber attack forecast-ing performance on imbalanced data sets. The proposedalgorithm is a modified version of the Synthetic Minor-ity Over-sampling Technique (SMOTE) [6], where some ofthe majority instances are under sampled before applyingSMOTE. We show that this algorithm improves the predic-tion performance of the proposed method on imbalanceddata sets.

The unconventional signals used to predict cyber attacksare not directly related to the target entity for which theprediction is made. There is no explicit relationship betweenthese signals and the target entity. In fact, it is not easyto measure the level of correlation between any of the un-conventional signals and the ground truth events. Bayesiannetworks are probabilistic graphical models that are moresuccessful in marginalizing the unknown uncertainties com-pared to other models. They are helpful in accounting forthe uncertainty among the variables whose distribution ordependencies are unknown and also extracting the influen-tial or causal relationships among features if any. Therefore,Bayesian approach is used to predict cyber attacks withunconventional signals, as the relationship of these signals

and the target entities is not straightforward.This paper is organized as follows: Section 2 presents

a brief review of the previous approaches in cyber attackprediction. In Section 3, the problem is formulated bygiving a background on Bayesian networks and definingthe different time granularities used for signal and groundtruth calculation. Section 4 explains the proposed approach,Section 5 gives the experiment results and Section 7 presentsconclusions and future work.

2 PREVIOUS WORK

Traditional intrusion detection systems rely on a mis-use based approach where monitored events are matchedagainst the signatures of the previously observed incidents[7], [8], [9], [10], [11]. One of the drawbacks of such systemsis their inability to detect new events whose signatures arenot known to the detection systems. Furthermore, deployingthe signatures of the new attacks across the whole networktakes some time and may not be effective. Anomaly detec-tion aims at detecting deviations from normal behaviourand labels them as malicious [12], [13], [14], [15], [16]. Theseapproaches typically have a high false alarm rate, becausethey may label normal but previously unseen behaviors asanomalies.

Yang et al. [17] describe several attack projection frame-works which model how an attack might transpire overtime. This modeling is broader than the traditional defini-tion of the intrusion detection systems where observablesof ongoing attacks are used to predict the next maliciousactions based on system vulnerabilities and attacker behav-ior. In attack projection, the focus is on the footprints ofthe multistage cyber attacks. Extracted footprints and thesequence or causal dependencies among these footprintsare used to generate hypothesized attack strategies thatrepresent a multistage attack. These attack strategies arethen modeled to project future actions of ongoing attacks.Wang et al. [18] suggest that efficient algorithms are neededfor multi-step intrusions to correlate isolated alerts intoattack scenarios and propose to use attack graphs. Theyuse attack graphs to identify possible cases where vulner-abilities can be exploited in a network and show that theirmethod is more successful in correlating isolated attacksinto attack scenarios compared to the traditional intrusiondetection systems. Similarly, Qin and Lee [19] suggest to useprobabilistic inference with a dynamic Bayesian Networkapproach to correlate attacks. Using the Grand ChallengeProblem (GCP) data set of the Defense Advanced ResearchProjects Agency, they show that their method is successfulin correlating isolated attacks, identifying attack strategies,and predicting future attacks.

A broad range of Machine Learning techniques havebeen used so far for intrusion detection and cyber attackprediction. Artificial Neural Networks [7] [13], Clustering[14], Decision Trees [8] [15], Ensemble Learning [9], SupportVector Machines [10] are some of the methods that havebeen used in the cyber security literature. Besides the tra-ditional approaches, there are some studies that elaborateon the prediction of multi stage cyber attacks. For example,Cheng et al. [20] use a novel method that is based on theLongest Common Subsequence (LCS) technique to measure

3

the similarities between attack progressions, correlate secu-rity alerts with specific patterns, and predict multi stageattacks. They show that their method is able to projectpossible attacks better when compared to the previous LCSbased approaches. Furthermore, Fava et al. [21] presentthe use of a variable-length Markov model (VLMM) thatcaptures the sequential properties of attacks to project futureactivities of ongoing cyber attacks and show that the methodthey propose is able to adapt to the newly observed attacksequences.

In most of the existing intrusion detection or projectionsystems, the observables are direct outcomes of the mali-cious activities on the computing systems. These observ-ables may be referred to as conventional signals. However,our research uses unconventional signals that are observedbefore any malicious activity occurs. They may be due to in-creasing negative tones toward an organization or news re-ports, and sometimes may not have explicit relations to thefuture victim of the cyber attacks. Preliminary studies haveshown the viable concept of using unconventional signalsto forecast cyber attacks [5] and attack intensities [22]. Silver[23] investigates how a true signal can be distinguished froma universe of noise data and states that most predictions faildue to a poor understanding of probability and uncertainty.He believes that as the appreciation of uncertainty improves,better prediction results could be achieved. Furthermore,Tetlock and Gardner [24] state that creating good forecastinvolves gathering evidence from a variety of sources ratherthan using very powerful computing resources or arcanetechniques.

Bayesian methods have also been used in the cybersecurity literature to a limited extent [11], [16]. Dua and Du[25] state that the false alarm rate of the rule based misusedetection systems are high since the signature of a maliciousand normal user might overlap to activate a rule. As analternative solution they suggest to use Bayesian networksand state that Bayesian approaches have a relative resiliencein conditional probability table parametrization and allowthe anomalousness of an event to be directly related to itsprobability.

3 PROBLEM FORMULATION

A Bayesian network is defined as a directed acyclic graph(DAG) that is composed of n random variables (nodes) ande edges that represent the conditional dependencies amongthese variables. Let X = {X1, X2, ...Xn} be n randomvariables (unconventional signals) with nominal or numericvalues for a Bayesian network B where n >= 1. Assumingπi represents the parents of node Xi, the probability dis-tribution of Xi is calculated by P (Xi|πi). Each node Xi

in the Bayesian network is associated with a conditionalprobability table that shows the probabilities for each valueofXi based on all combinations of values of its parent nodesand is represented as PXi = P (Xi|πi).

To calculate the joint probability distribution of X , thechain rule is used, i. e.,P (X) = P (X1|X2, X3, ..., Xn)P (X2, X3, ..., Xn)

= P (X1|X2, ..., Xn)P (X2|X3, ..., Xn)P (X3, ..., Xn)

= P (X1|X2, ..., Xn)P (X2|X3, ..., Xn)...P (Xn−1|Xn)P (Xn)

=

n∏i=1

P (Xi|Xi+1, ..., Xn) (1)

Knowing that Xi is independent from the variables otherthan its parents πi

P (X) =

n∏i=1

P (Xi|πi) (2)

Consider the Bayesian network in Figure 1. Bayesian infer-

Fig. 1. An example Bayesian Network showing a hypothetical relation-ship among the Entity Mentions (EM), Cyber Mentions (CM) and Attack(A)

ence can be used to calculate the probability of observingan attack towards an entity based on all possible valuesof Entity Mentions (EM) and Cyber Mentions (CM). Forexample, assuming that EM and CM can take values eitherLow (L) or High (H),

P (A|EM = H) = P (A|EM = H,CM = H)P (CM = H|EM = H)

+ P (A|EM = H,CM = L)P (CM = L|EM = H)

P (CM = H|EM = H) is P (CM = H) and P (CM =L|EM = H) is P (CM = L), because the variables EMand CM are independent. Then,

P (A|EM = H) = P (A|EM = H,CM = H)P (CM = H)

+ P (A|EM = H,CM = L)P (CM = L)

Consider a set of target entities composed of organizationsor industries E = {E1, E2, ...Et}, a set of cyber attacksignals X = {X1, X2, ...Xn}, and a set of attack typesA = {A1, A2, ...Am}. For any given entity Ei, if the setof attack types is defined as A(Ei) where A(Ei) ⊆ A andthe set of cyber attack signals is defined as X(Ei) whereX(Ei) ⊆ X , Bayesian inference is used to calculate theprobability of each attack type Ai ∈ A(Ei) based on thecyber attack signals in X(Ei).

3.1 Signal and Ground Truth Time Granularity

Depending on how frequent cyber incidents are observedfor an entity Ei, making predictions using different groundtruth granularities could be helpful. If a couple of cyberattacks of a certain type are observed almost every dayconsistently, it would not be beneficial to train and test aclassifier to predict attacks for each day. Instead, one maymake predictions for 6 or 12 hours of granularity. On theother hand, depending on the availability, quality or updatefrequency of the unconventional signals, one may think tocalculate signals for different time granularities, i. e., the lastcouple of days, weeks or months. If the data source is notbeing updated on a daily basis, it might not be beneficial tocalculate signals on a daily basis. Different time granularitiesare used to calculate the unconventional signals and theground truth. The time granularity of signals is defined as txand five cases are considered where each signal is calculatedby averaging its previous values observed in the last threedays (3d), one week (1w), one month (1m), three months(3m), and six months (6m).

4

Similarly, the time granularity of the ground truth isdefined as tg and four cases are evaluated where tg takes avalue of 6, 12, 24, or 48 hours (The tg values are representedas 6hr, 12hr, 24hr, and 48hr respectively in the followingsections). Taking tg less than 6 hours leads to a very sparsedata set, since a typical organization may not see certaincyber attacks every hour of the day. Similarly, taking tglarger than 48 hours causes the training data set to havepositive ground truth for the vast majority of instances.Depending on how frequent cyber incidents are observedfor an entity or attack type, different tg values may beassessed for various entities and attack types.

For each attack type of an entity Ei, each of the txvalues defined above is used to calculate the unconventionalsignals in X(Ei). Similarly, to train a Bayesian classifier foreach attack type, different tg time granularities (6, 12, 24,and 48 hours) are used to consider real cyber incidents forEi. For example, if tx is selected as one week and tg as 24hours, then for a given time t, signals calculated for the lastweek are used together with the ground truth of the next 24hours (tg) to train a Bayesian classifier. A Bayesian classifieris built for each attack type Ai ∈ A(Ei) of a target entity Ei,considering all possible combinations of the defined tx andtg values.

4 PROPOSED METHOD

Bayesian inference is used to calculate the probability ofcyber attacks towards a specific target entity using uncon-ventional signals that may or may not be related to thattarget entity. These signals are pulled from various datasources, including social media (Twitter) and open sourceglobal event tracking projects like GDELT.

The prediction is regarded as a classification problemwhere the value of the target class is either 1 or 0 dependingon having a cyber attack or not. For each entity Ei andits attack type Ai ∈ A(Ei), the unconventional signals inX(Ei) are used as random variables and the ground truthof Ei for Ai as target class to train a Bayesian classifier fordifferent tx and tg pairs. For a given time t, the training setof each Ai is represented as SAi and includes the unconven-tional signals calculated for the time period (t − tx, t) andthe real cyber incidents reported for the period (t, t + tg).For each attack type Ai of a target entity Ei and for eachtx and tg time granularity pair, we learn the structure ofthe Bayesian network for Ai using the previously observedunconventional signals in (t − tx, t) and the ground truthincidents observed in (t, t + tg). Then, for a given time tunconventional signals are calculated for the time period(t − tx, t) to predict the cyber incidents for the upcomingtime period, i. e., (t, t+ tg).

4.1 Unconventional Signals Used

Increased discussion of past or potential cyber attacks maybe due to a significant change in the intent or capability ofattackers or software vulnerabilities. Similarly, an increase inthe number of mentions of a target entity might be due to anincrease in the surveillance towards it, and may lead to anincrease in the probability of a cyber attack for that targetentity. Furthermore, the number of mentions of negative

events, the tone of the negativity of these events or thenumber of source documents referring to them might be in-dicative of a cyber incident. These example unconventionalsignals are used to predict cyber attacks for an anonymizedprivate company nicknamed KNOX. These unconventionalsignals are by no means comprehensive, but are good ex-amples to test our methodology. These signals are describedbelow, which are fed into a Bayesian classifier for each txand tg time granularity pair, to predict attacks for eachattack type defined for KNOX: Defacement, Malware, DOS,and Malicious Email/URL (MEU).

• Twitter Cyber Mentions (TCM): Given a time t anda signal time granularity tx, a set of cyber keywordsincluding Hack, Malware, DDOS, and Malicious areused to count the number of mentions of these key-words on Twitter for each signal calculation period,i. e., (t− tx, t).

• Twitter Entity Mentions (TEM): The number ofmentions of KNOX related keywords in Twitter iscounted for each signal calculation period (t − tx, t)to check if these mentions are indicative of a cyberincident towards KNOX.

• GDELT Event Mentions (GEM): GDELT keeps trackof the media from every country, in over 100 lan-guages. As a measure of event significance it countsthe total number of mentions of specific events acrossall of its source documents. GDELT also attaches anaverage tone value for each mentioned event. Thevalue of the average tone could be between -100 and+100 and it indicates the degree of the negativityof the associated event. For each signal calculationperiod (t − tx, t), the total mentions of events thathave a negative average tone are calculated.

• GDELT Event Articles (GEA): GDELT counts thetotal number of source documents containing one ormore mentions of an event for assessing the “impor-tance” of an event. The more an event is discussed,the more likely it is regarded as significant. For agiven time t, the total number of source documentsmentioning events with an average negative tone arecounted in the signal calculation period (t− tx, t).

• GDELT Event Tone (GET): Given a time t and asignal granularity tx, the average tone of the negativeevents is calculated for the time period (t− tx, t).

4.2 Signal and Ground Truth Calculation

For a given time t, the signals for the past period (t − tx, t)are aggregated to predict the cyber attacks for the next tghours. Most of the signals are general and not specific toKNOX. The ground truth data of KNOX was available be-tween April 1 and October 30 2016, therefore the predictionmodels are trained and tested in this period. Step by stepdetails of the signal and ground truth generation process areprovided in Algorithm 1. The variables gtStartT ime andgtEndT ime are set to 04.01.2016 and 10.30.2016 respectively.For each attack type and time granularity pair (tx and tg), aseparate data set is generated. All ground truth counts thatare greater than 1 are set to 1 to perform binary prediction.In the inner most while loop, the unconventional signals are

5

aggregated over a time period of length tx, i. e., starting atsignalStartT ime and ending at currentT ime.

Algorithm 1: The signal and ground truth calculationprocess for different tx and tg .

Function GenerateDataSets (gtStartT ime, gtEndT ime)attackTypes := [Malware,Defacement,DOS,MEU];signalGranularity := [6m,3m,1m,1w,3d];groundTruthGranularity := [48hr,24hr,12hr,6hr];

foreach ai ∈ attackTypes doforeach tx ∈ signalGranularity do

foreach tg ∈ groundTruthGranularity docurrentT ime := gtStartT ime;while currentT ime <= gtEndT ime− tg do

signalStartT ime := currentT ime− tx;

. Calculate average of signalsTCM, TEM, GEM, GEA, GET

let signals be the signals calculated fortime period(signalStartT ime, currentT ime);

. Calculate ground truthgtEnd := currentT ime+ tg ;let gt be number of cyber incidents

observed in the time period(currentT ime, gtEnd);

if gt > 1 thengt := 1;

end. signal and gt is appended tothe file for ai

write(ai, signals, gt);

currentT ime := currentT ime+ tg ;end

endend

end

4.3 Working with Imbalanced Cyber Data Sets

Cyber incidents towards a specific entity are rare events thatcan be marked as anomalies especially when filtered for aspecific attack type. Therefore, the data sets for cyber inci-dents could be highly skewed towards negative instancesthat have a ground truth value of zero. Sometimes it mightbe very difficult for a classifier to predict positive instanceswith a cyber attack based on such data sets. There areseveral approaches to overcome the imbalanced data setproblem, including:

• Using different performance metrics like F-Measureor AUC (the area under the ROC curve) rather thanaccuracy.

• Using different classifiers like decision tree basedalgorithms that could perform well on imbalanceddata sets.

• Over sampling the minority class.• Under sampling the majority class.• Introducing new synthetic instances for the minority

class.

A new algorithm named SMOTE++ is proposed to improvethe cyber attack prediction performance on imbalanced datasets. It is a hybrid approach where the under sampling,instance weighing, and over sampling techniques are usedtogether.

It is highly desirable for a classifier to have the negativeand positive instances apart from each other. In other words,if it is easier to separate the negative and positive instanceswith a line, plane or hyperplane, one may expect a classi-fier to perform better. Unconventional signals that are notexplicitly related to the target entity are used. Therefore,some of the training instances may not be sufficient inexplaining the relationship of the unconventional signalsand the cyber event ground truth. To be able to distinguishthe negative and positive instances better, we suggest toremove a certain percentage of the majority class (negative)instances that are near to the minority class (positive) in-stances. To find such instances, a simple approach could beto filter out the negative instances that are near to the meanof the positive instances. k-Means clustering algorithm isused to find the main cluster of the positive instances andremove some portion of the negative instances that arenear to that mean where the percentage to remove is aconfigurable parameter in the algorithm. In case there isno cluster for the positive instances, i. e., they are scatteredaround, the mean of all positive (minority) instances isused to remove negative (majority) instances. Removingsome of the negative instances causes a change in the totalweight of the negative classes. To compensate for that, theremaining negative instances are reweighed to maintainthe same total weight for the negative instances. On theother hand, to make the class weights of the negative andpositive classes equal, a hybrid approach is followed wherethe weights of the existing positive instances are reweighedand new synthetic positive instances with a lower weightare introduced. This is done to differentiate the existingminority instances from the newly genareted synthetic ones.Assume that we need to introduce three more instances foreach positive instance to achieve an equal weight for thenegative and positive classes. We first multiply the weightof each existing positive instance by two and then introducetwo new synthetic instances for every positive instance. Tointroduce new synthetic instances k−NN algorithm is usedin the same way it is used in SMOTE [6]. A step by stepdescription of the proposed method is shown in Algorithm2.

5 EXPERIMENTS AND RESULTS

The unconventional signals defined in Section 4.1 that areextracted from various data sources and are not explicitlyrelated with KNOX are used to predict the probability ofdifferent attack types for KNOX. For each attack type, a sep-arate data set is created for every tx and tg time granularitypair. Using five tx and four tg time granularity values, a totalof 20 different data sets are created for each attack type. It isalways desirable for a classifier to have a high true positive(TP) and a low false positive (FP) rate at the same time. Thearea under the ROC curve (AUC) gets higher, when TP ishigh and FP is low. Therefore, AUC is used to compare theperformance of the proposed model on different data sets.Moreover, the BayesNet classifier in Weka [26] is used with10× 10 folds cross validation to calculate the average AUCvalue for each attack type and the tx and tg time granularitypair.

6

Algorithm 2: SMOTE++ algorithm.Input : An imbalanced data set denoted as allInstances

Percentage of majority instances to remove, i. e., pThe number of nearest neighbors to consider togenerate synthetic minority instances, i. e., k2

Output : A new data set with a uniform majority andminority class distribution

Function SMOTE++ (allInstances, p, k2)let majInstances be the set of majority instances inallInstances;

let minInstances be the set of minority instances inallInstances;

let sMin be the size of the minInstances;let sMaj be the size of the majInstances;

. Find the first minority cluster usingk−Means Clustering with Euclidian distance

k := 2;minorityClusterFound := false;while minorityClusterFound 6= true do

let clusters be the first k clusters in allInstances;if clusters includes a minority cluster then

let cMin be the centroid of the minority cluster inclusters;

minorityClusterFound := true;else

k := k + 1;endif k = sMin then

break;end. If a minority class cluster is not foundthen use the mean of all minority instances

if minorityClusterFound 6= true thenlet cMin be the mean of all minority instances inminInstances;

. Filter majority instancesremove p percent of majInstances that are nearest tocMin;

let majInstancesNew be the remaining instances inmajInstances;

. Reweigh majority instancesmajWeight := 100/(100− p);set the weight of each instance in majInstancesNew tomajWeight;

. Reweigh existing minority instancesminW := sMaj/sMin / 2;set weight of each instance in minInstances to minW ;

. Generate new synthetic minority instancesgenerate minW ∗ sMin synthetic minority instances usingk−NN with k := k2 [6];

let minInstancesSyn be the set of created syntheticminority instances;

returnmajInstancesNew∪minInstances∪minInstancesSyn;

5.1 Data Sets Used

We create a separate data set for each attack type and the txand tg time granularity pair. The distribution of the negativeand positive classes in these data sets are different for dif-ferent tg values. For each attack type, as tg gets lower, fewerground truth incidents are observed for KNOX, thereforethe data sets become more imbalanced. Similarly, as tg getslarger more cyber incidents are observed and the ratio of thepositive instances with a cyber attack increases. The list ofthe created data sets and their negative and positive classdistribution for each attack type and tg are shown in Table1. For example, the data sets for the DOS attack type are

highly imbalanced for all tg values. However, the data setsfor the Malware attack type are more balanced compared tothe DOS and MEU attack types. Moreover, the data sets ofthe MEU and Defacement attack types are skewed when tgis low.

TABLE 1The percentage (%) of the positive instances for the data sets of

different attack types for different tg time granularities: 6, 12, 24, and 48hours. A positive instance corresponds to an instance with a cyber

attack.

6 hr 12 hr 24 hr 48 hrMalware 36 51 72 80Defacement 15 26 48 64DOS 2 4 9 15MEU 10 17 32 50

5.2 Using Different tx and tg Time GranularitiesThe average AUC values for each attack type for differenttx and tg pairs are shown in Figures 2, 3, 4, and 5. Wepresent the AUC values when tx is three days (3d), oneweek (1w), one month (1m), three months (3m), and sixmonths (6m) for different tg . Each line in these figuresshows the AUC values when tg is 6, 12, 24, and 48 hoursrespectively. First, we observe that when an optimum txand tg time granularity is chosen, it is possible to achievean AUC value of at least 0.70 for all attack types defined forKNOX, using unconventional signals that are not directlyrelated with KNOX. The results suggest that if a predictionmodel calculates signals for different time granularities andkeeps track of the best performing granularity pair, it mightbe possible to achieve a better performance to predict cyberincidents for different attack types.

Fig. 2. Average AUC values for the Malware attack type for different txand tg signal and ground truth time granularities.

A data set might be more imbalanced for specific attacktypes (like DOS or MEU in our case) or when createdwith a lower tg value. For example, when tg is 24 hours,the rate of the positive instances with a cyber attack andthe negative instances with no attack are different for eachattack type. One might expect a lower performance on animbalanced data set compared to a data set that has auniform class distribution. Although the skewness of datasets are different for each attack type when tg is 24 hours, ahigher classification performance is observed for all attacktypes for tg = 24 hours. For example, except for Malware,

7

the performance of the prediction model is better when tgis 24 hours for almost all attack types and tx values. ForMalware, a comparable performance is observed when tg is24 and 48 hours.

Fig. 3. Average AUC values for the Defacement attack type for differenttx and tg signal and ground truth time granularities.

Fig. 4. Average AUC values for the DOS attack type for different tx andtg signal and ground truth time granularities.

Similarly, the performance of the classifier is also im-pacted by the choice of tx. For example, for the MEU andDOS attack types the AUC value of the classifier is higherwhen tx is one month. However, Malware and Defacementfollow a different pattern. For the Defacement attack type,a higher AUC value is observed when tx is three monthsand the performance is comparable when tx = 6 months.Similarly, for the Malware data set, although the perfor-mance of the model is slightly better when tx is six months,a comparable performance is observed when tx is threemonths.

5.3 Relationships of the Signals and Cyber IncidentsTo explore the relationship of each unconventional signaland the cyber incidents, we take a closer look at theBayesian networks generated. Twenty Bayesian networksare trained for each attack type considering each possiblepair of defined tx and tg values. One of the best performinggranularity pairs is used for each attack type where tg is 24hours and tx is either one month or three months dependingon the average AUC value found. Therefore, the Bayesiannetwork for tx = 1m is used for the DOS and MEU attacktypes, and the Bayesian network for tx = 3m is used for theMalware and Defacement attack types.

Fig. 5. Average AUC values for the MEU attack type for different tx andtg signal and ground truth time granularities.

Using a Bayesian network, assuming a dependency ex-ists between a signal and the target class, we can find outthe probability of having a high or low signal value whenan attack is observed or not. Furthermore, by looking at theprobability values in the conditional probability tables forthe low and high values of each signal, one may commenton the importance of a specific signal. If a signal has arelationship with the target class and the probabilities for itshigh and low values are different for cases with and withouta cyber attack, that signal might be regarded as important.Similarly, if the probabilities for its high and low values areclose to each other for cases with and without a cyber attack,it has less discriminative power. For a signal to be morediscriminative, if the probability of observing its high valueis higher when a cyber attack is seen, it is expected to belower when there is no cyber attack.

We look at the conditional probability tables of eachsignal in the Bayesian networks learned for each attacktype. For the Malware attack type, a relationship is observedbetween each unconventional signal and the cyber attacks.The values in the conditional probability table when tx is3 months and tg is 24 hours is shown in Figure 6. Whenan attack is observed, the probability of having a highervalue of TCM, TEM, GEM, GET, and GEA signal is muchhigher. However, for the TCM signal, when there is noattack, the probability of having a high and low signal valueis equal, therefore TCM is less predictive compared to othersignals when the ground truth is zero. On the other hand,GEA tends to be lower when the ground truth is zero andhigher when the ground truth is one, therefore it is morediscriminative for the Malware attack type compared toother signals.

The relationship of the signals and the target class for theDefacement, DOS, and MEU attack types is shown in thecorresponding graphs in Figure 6. The tg time granularityis 24 hours for each of the data sets used for these attacktypes. The tx time granularity is one month for the DOS,MEU, and 3 months for the Defacement attack type. Onlythe unconventional signals that have a relationship with thetarget class are included in the graphs. We find that whena Defacement attack is observed, the value of the TCM,TEM, GEM, and GEA signals tend to be higher. For theDOS and MEU attack types, there are fewer signals thathave a relationship with the target attack class. For the

8

Fig. 6. The probability (p) of having a low or high value of a signal when there is an attack or not. All probabilities in each graph are based on theconditional probability tables of the Bayesian networks learned for the corresponding attack types

DOS attack type, GEM is more discriminative compared tothe GET signal, as its probabilities for the low and highsignal values are very much different when the groundtruth is 0 and 1. A similar case is observed for the MEUattack type with the GEA signal. GEA tends to have alower value when the ground truth is zero, and a highervalue when the ground truth is one. Although GEM ismarked as an important signal for the MEU attack type, it isless discriminative compared to GEA. The analysis aboveprovides a means to assess the quality of the signals todiscriminate positive and negative instances. This is madepossible with the use of Bayesian networks. Similar analyseswith additional unconventional signals could be helpful toexplore other important signals for different attack types.

Fig. 7. The probability (p) of observing a DOS attack depending on thehigh or low values of the unconventional signals for different tx valueswhen tg is 24 hours.

For a given attack type, having a higher AUC valuefor a specific tx may not mean that all unconventionalsignals are more discriminative for that tx. The importanceor discriminative power of a signal might also be changingfor different tx values. We use the DOS attack type as an

example, to show how the quality or importance of eachsignal changes with different tx. The probabilities of thehigh or low values of each signal is shown in Figure 7.GEA signal seems to be important when tx is three andsix months and it is more discriminative when tx is threemonths. However, GEA is not among the signals that havea dependency with the target class when tx is one month.Therefore, we use the GEA signal calculated for tx = 3mand other signals calculated for tx = 1m, to repeat theexperiment for tx = 1m. As a result, the AUC value of theBayesNet classifier increases from 0.806 to 0.824 for tx = 1m,when GEA signal calculated with tx = 3m is used (See Table2).

Similarly, the quality of each signal calculated with dif-ferent tx granularities is analyzed for other attack types. Forexample, when tx = 3m and tg = 24hr the AUC valueis 0.696 for the Defacement attack type (See Figure 3). Theconditional probability tables for each signal in the Bayesiannetworks learned for Defacement for different tx show thatthe GEM signal is more powerful when tx = 1m (See Figure8). Therefore, the GEM signal (calculated with tx = 1m) isused together with the remaining signals calculated withtx = 3m and this increases the AUC value from 0.696to 0.714. On the other hand, for the Malware and MEUattack types, all signals that have a dependency with thetarget class seem to be more predictive when tx = 3m andtx = 1m respectively. The AUC values found when pickingup a more predictive signal with a different tx is shown inthe “Variable tx” column of Table 2 for each attack type. Wefind that depending on an entity or attack type, there mightbe a benefit if more discriminative signals with different txare used to built the Bayesian prediction model.

To cross check our findings about the importance of eachunconventional signal for each attack type, feature selectiontechnique is applied to the same data sets with 10×10 foldscross validation. The CfsSubsetEval algorithm [27] is usedwith the BestFirst search method to select more predictive

9

Fig. 8. The probability (p) of observing a Defacement attack depending on the high or low values of the unconventional signals for different tx valueswhen tg is 24 hours.

TABLE 2The AUC values for the Defacement and DOS when using signals with

fixed and variable tx

Fixed tx Variable txDefacement 0.696 0.714DOS 0.806 0.824

TABLE 3The number of times each signal is selected in the 10-fold cross

validation experiment of each attack type.

TCM TEM GEM GET GEAMalware 0 1 10 2 1Defacement 9 10 4 5 10DOS 0 1 10 10 0MEU 0 0 8 0 3

signals in each data set. CfsSubsetEval selects the subsetsof signals that are highly correlated with the class in eachfold. The results for the CfsSubsetEval is shown in Table 3.For each attack type, the number of folds a signal is selectedby CfsSubsetEval is shown for each signal. For the Malwareattack type, the GEM signal is selected in all folds and theGET, TEM, and GEA signals are selected in at least onefold. Similarly, signals that are differentiative according tothe conditional probability tables of the Bayesian classifierare selected by the CfsSubsetEval for other attack types.For example, for the DOS attack type only the GEM andGET signals are selected. Similarly, CfsSubseteval selectsthe GEM and GEA signals for the MEU attack type. Asa summary, we show that the feature selection techniqueconfirms our previous observations with the conditionalprobability tables, with regards to the importance of theunconventional signals.

5.4 Working with Imbalanced Cyber Data SetsAs tg gets lower, the percentage of the positive instancesdecreases in all data sets. For instance, when tg is 6 hours,only 2% and 10% of the instances are positive in the datasets for the DOS and MEU attack types. In fact the DOSdata set is still highly imbalanced when tg is 48 hours.Observation of a cyber incident is an abnormal situationfor a target entity, therefore most of the time the cyber datasets are highly imbalanced. There might also be cases where

one needs to use a smaller tg . In addition to DOS, whichis inherently imbalanced, other attack types may also sufferfrom the imbalanced ground truth for smaller tg values. It isalways more difficult for a classifier to classify instances ina highly imbalanced data set. Notice the poor performanceof the Bayesian classifier (lower AUC values) for lower tg .For instance, when tg is 6 hours, a lower AUC value isobserved for the skewed Defacement, DOS, and MEU datasets in the Figures 3, 4, and 5. Therefore, any technique thathas a potential to improve the performance of a classifieron the imbalanced data sets is valuable. We use the widelyused techniques in the literature together with the proposedSMOTE++ method, to check if the performance of theBayesian classifier gets better on the imbalanced data sets.Unless a data set has a completely uniform class distributionit can be regarded as imbalanced, therefore existing filteringtechniques and SMOTE++ are applied to all data sets for allattack types. Some of the widely used approaches to combatthe imbalanced data set problem are:

• SMOTE: This technique generates synthetic minorityinstances using k − NN algorithm [6]. One canspecify the percentage of the minority instances togenerate, to convert a highly imbalanced data set toa completely balanced one. During our experiments,we choose the proper percentage parameter for eachdata set to make the distribution of the positive andnegative classes equal.

• Spread Sub Sample: This technique randomly subsamples the majority class to decrease the number ofthe majority instances. During our experiments, thedistribution parameter of the method is chosen tobe 1, to achieve an equal positive and negative classdistribution in the data sets.

In addition to the existing filtering approaches,SMOTE++ is used to achieve a uniform class distributionbefore applying the Bayesian classifier. In the SMOTE++approach, a certain percentage of the majority instancesthat are nearest to the minority instances are removed. Tovisualize the changes made on a data set by SMOTE++, thescattered view of the MEU data set instances is shown inFigure 9.A, before any filter is applied (when tx is one monthand tg is 6 hours). Then, the scattered view of the same dataset is shown after the SMOTE filter is applied in Figure 9.B.

10

Fig. 9. A 2-D view of the data set for the MEU attack type before andafter the SMOTE and SMOTE++ filters are applied. The plot of the MEUdata set before any filters are applied is shown in (A). (B) shows the dataset after the SMOTE filter is applied, and (C) shows the data set afterthe SMOTE++ filter is applied.

Similarly, the status of the data is shown in Figure 9.C, afterthe SMOTE++ technique is applied.

We observe that the number of positive instances (darkones) increased after the SMOTE filter is applied, comparedto the original data set shown in Figure 9.A. A similarsituation is observed for the data set when SMOTE++ isapplied too. But, an important difference exists betweenthe two approaches when the highlighted parts of Figure9.B and 9.C are considered. With SMOTE, the generatedsynthetic positive instances are mixed with the existingnegative majority instances. However, when SMOTE++ isapplied, the main cluster of the positive instances does notinclude that many negative majority instances.

Does cleaning the main cluster of the positive (minor-ity) instances from the negative (majority) ones help theBayesian classifier to separate the positive and negativeinstances better? SMOTE++ is applied in parallel with otherfiltering techniques in the literature, to see if the perfor-mance of the Bayesian classifier is improved. During eachfiltering, the BayesNet classifier is used with the Filtered-Classifier algorithm in Weka [26] to ensure that the testset is not touched when a filter is applied on each dataset. Furthermore, the percentage parameter is tuned beforeapplying SMOTE++ to a data set. The average AUC valuesof different methods, for different attack types and tx andtg values are shown in Figures 10, 11, 12, and 13. The AUCvalues of the SMOTE++, SMOTE, and the RandomSubsam-ple are shown with dark solid (blue), light solid (green)and dotted (blue) lines respectively. Additionally, the AUCvalues of the BayesNet without any filters is shown with adashed (red) line in Figures 10, 11, 12, and 13.

SMOTE++ performs better than other techniques interms of AUC in almost all cases for all attack types and alltx and tg time granularities. To compare different methodsand check if a difference of AUC between these methods issignificant or not, a two tailed t-test is used with a p-value of0.05 in all experiments. For the MEU attack type, the AUCvalue of the BayesNet is around 0.5 when tg is 6, 12, or48 hours before any filter is applied. Although we do notobserve a consistent and significant improvement with otherfiltering techniques, using SMOTE++ a significant increaseis observed in the average AUC value of the BayesNet

classifier when tg is 6, 12, or 48 hours. For the DOS attacktype, SMOTE++ achieves a better performance compared toSMOTE for all tx and tg values, but the differences in theAUC are not significant. Both methods lead to a significantimprovement in the AUC value of the Bayesian classifier forall tg values and for almost all tx except when tx = 1mand tg = 24hr. Although the RandomSubsample methodimproves the prediction performance of the Bayesian clas-sifier in some cases, in some other cases its AUC value iscomparable with the AUC value of the ordinary Bayesianclassifier (See Figure 11).

For the Defacement attack type, SMOTE++ achieves thebest AUC scores in almost all cases compared to otherfiltering techniques but the differences are not significantexcept the cases when tg = 48hr and tx is three days orone week. When compared to the Bayesian classifier withno filtering, SMOTE and SMOTE++ leads to a significantimprovement in the AUC values when tg is 12 and 48 hours(See Figure 12). Similar to our findings for other attack types,SMOTE++ achieves the highest AUC for the Malware attacktype with most of the tx and tg combinations. For somedata sets like when tx is one week and tg is 12 or 24 hours,the improvement is significant compared to the ordinaryBayesNet classifier as it is seen in Figure 13.

6 TYPICAL RISKS FOR RESEARCH VALIDITY

In research studies there are three types of validity risks, i. e.,internal, construct, and external that should be considered.Below we explain the measures taken to mitigate each ofthese validity risks.

If causal relationships between a dependent target andits independent variables are not defined properly in anexperiment, then its internal validity is low. Although weuse unconventional signals that are not directly related tothe target entity, we still observe a similar and encouragingperformance for all attack types with different skewness.Observing similar promising results for all attack typeswith different skewness and ground truth is important tomitigate the internal validity threats. In addition to that,using Bayesian networks clarifies the probabilistic relation-ships between each model variable and the target classbetter compared to other models, and supports the internalvalidity of the experiments further. Depending on how wellan experiment or a tool measures the intended construct,a construct validity threat might be observed. To mitigateconstruct validity threats, the unconventional signal gen-eration process is automated. External validity is relatedto the consistency of the claims made in a research study.It is hard to conclude that the results observed in thisresearch could be generalized. Therefore, further researchwith additional unconventional signals and target entities isneeded to justify our observations.

7 CONCLUSION

This paper uses unconventional signals extracted from theTwitter and GDELT data sources and shows that they canbe used to predict various cyber attacks for the anonymizedtarget entity KNOX. Furthermore, different time granulari-ties (defined as tx and tg) are used to calculate the uncon-ventional signals and the entity ground truth for KNOX.

11

Fig. 10. Average AUC values for the data sets of the MEU attack type with different tx and tg values when SMOTE++ (S++), RandomSubsample(SS), and SMOTE (S) filters are applied before BayesNet classifier. The dashed line shows the AUC values with an ordinary BayesNet classifier (B)and the Density shows the percentage of positive instances in each data set.

Fig. 11. Average AUC values for the data sets of the DOS attack type with different tx and tg values when SMOTE++ (S++), RandomSubsample(SS), and SMOTE (S) filters are applied before BayesNet classifier. The dashed line shows the AUC values with an ordinary BayesNet classifier (B)and the Density shows the percentage of positive instances in each data set.

Each time the signals are averaged over a period of time oflength tx, to predict the cyber incidents for the succeedingperiod of time of length tg . Although the extracted signalsare not explicitly related with the target entity KNOX, thispaper shows that if appropriate tx and tg time granularitiesare chosen, the combined use of unconventional signalsis able to achieve 70 % AUC (Area Under ROC Curve)performance for Malware, Defacement, DOS, and Mali-cious Email/URL attack types for KNOX between April toNovember 2016. Although the set of signals that are related

to the target attack class is different for each attack type,GEM, GET, and GEA signals seem to be more indicative ofcyber incidents when compared to other signals used in thisstudy. Moreover, this work also shows that using a differentversion of the same signal calculated with a different txcould improve the overall prediction model.

Furthermore, common Machine Learning filtering meth-ods like SMOTE and RandomSubsample are used to seeif the attack prediction performance is improved whena skewed data set is balanced. This work developed

12

Fig. 12. Average AUC values for the data sets of the Defacement attack type with different tx and tg values when SMOTE++ (S++), RandomSub-sample (SS), and SMOTE (S) filters are applied before BayesNet classifier. The dashed line shows the AUC values with an ordinary BayesNetclassifier (B) and the Density shows the percentage of positive instances in each data set.

Fig. 13. Average AUC values for the data sets of the Malware attack type with different tx and tg values when SMOTE++ (S++), RandomSubsample(SS), and SMOTE (S) filters are applied before BayesNet classifier. The dashed line shows the AUC values with an ordinary BayesNet classifier (B)and the Density shows the percentage of positive instances in each data set.

SMOTE++ that uses a hybrid approach where under sam-pling, synthetic minority instance generation and reweigh-ing techniques are used together to balance the distributionof the majority and minority instances in an imbalanceddata set. We show that the performance of the Bayesianclassifier on a skewed data set could be improved by makingthe data set balanced. We also show that the proposedSMOTE++ technique improves the prediction performancefor most of the data sets for different attack types andthe improvement amount is significant in some of these

cases. As a future work we plan to use more signals fromseveral other open data sources and include ground truth ofdifferent target entities. The proposed approach can be usedwith additional unconventional signals to achieve a higherforecast performance.

ACKNOWLEDGMENTS

This research is supported by the Office of the Director ofNational Intelligence (ODNI) and the Intelligence Advanced

13

Research Projects Activity (IARPA) via the Air Force Re-search Laboratory (AFRL) contract number FA875016C0114.

REFERENCES

[1] PwC, “The global state of information security survey 2016,”2016, [Online; accessed 6-February-2017]. [Online]. Available:http://www.pwc.ru/gsiss2016

[2] GDELT, “The gdelt project,” 2017, [Online; accessed 6-February-2017]. [Online]. Available: http://www.gdeltproject.org/

[3] J. Z. Bakdash, S. Hutchinson, E. G. Zaroukian, L. R. Marusichet al., “Malware in the future? forecasting analyst detectionof cyber events,” CoRR, vol. abs/1707.03243, 2017. [Online].Available: http://arxiv.org/abs/1707.03243

[4] N. Ramakrishnan, P. Butler, S. Muthiah, N. Self et al., “’beatingthe news’ with embers: Forecasting civil unrest using open sourceindicators,” in Proceedings of the 20th ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining, ser. KDD’14. New York, NY, USA: ACM, 2014, pp. 1799–1808. [Online].Available: http://doi.acm.org/10.1145/2623330.2623373

[5] A. Okutan, S. J. Yang, and K. McConky, “Predicting cyber attackswith bayesian networks using unconventional signals,” in Proceed-ings of the Cyber and Information Security Research (CISR) Conference,2017, pp. 1–4.

[6] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer,“Smote: Synthetic minority over-sampling technique,” Journal OfArtificial Intelligence Research, vol. 16, no. 1, pp. 321–357, Jun. 2002.[Online]. Available: http://dl.acm.org/citation.cfm?id=1622407.1622416

[7] J. Cannady, “Artificial neural networks for misuse detection,” inProceedings of National Information Systems Security Conference, 1998,pp. 443–456.

[8] C. Kruegel and T. Toth, Using Decision Trees to Improve Signature-Based Intrusion Detection. Berlin, Heidelberg: Springer BerlinHeidelberg, 2003, pp. 173–191.

[9] J. Zhang, M. Zulkernine, and A. Haque, “Random-forests-basednetwork intrusion detection systems,” IEEE Transactions onSystems, Man, and Cybernetics, Part C, vol. 38, no. 5, pp.649–659, Sep. 2008. [Online]. Available: http://dx.doi.org/10.1109/TSMCC.2008.923876

[10] Y. Li, J. Xia, S. Zhang, J. Yan et al., “An efficient intrusion detectionsystem based on support vector machines and graduallyfeature removal method,” Expert Systems with Applications,vol. 39, no. 1, pp. 424–430, Jan. 2012. [Online]. Available:http://dx.doi.org/10.1016/j.eswa.2011.07.032

[11] C. Livadas, R. Walsh, D. Lapsley, and W. T. Strayer, “Using ma-chine learning techniques to identify botnet traffic,” in Proceedingsof 31st IEEE Conference on Local Computer Networks, Nov 2006, pp.967–974.

[12] D. E. Denning, “An intrusion-detection model,” IEEE Transactionson Software Engineering, vol. SE-13, no. 2, pp. 222–232, Feb 1987.

[13] R. P. Lippmann and R. K. Cunningham, “Improving intrusiondetection performance using keyword selection and neural net-works,” Computer Networks, vol. 34, p. 2000, 2000.

[14] M. Blowers and J. Williams, Machine Learning Applied to CyberOperations. New York, NY: Springer New York, 2014, pp. 155–175.

[15] L. Bilge, S. Sen, D. Balzarotti, E. Kirda, and C. Kruegel, “Exposure:A passive dns analysis service to detect and report maliciousdomains,” ACM Transactions on Information and System Security,vol. 16, no. 4, pp. 14:1–14:28, Apr. 2014. [Online]. Available:http://doi.acm.org/10.1145/2584679

[16] C. Kruegel, D. Mutz, W. Robertson, and F. Valeur, “Bayesian eventclassification for intrusion detection,” in Proceedings of 19th AnnualComputer Security Applications Conference, Dec 2003, pp. 14–23.

[17] S. J. Yang, H. Du, J. Holsopple, and M. Sudit, “Attack projection,”in Cyber Defense and Situational Awareness, 2014, pp. 239–261.

[18] L. Wang, A. Liu, and S. Jajodia, “Using attack graphs forcorrelating, hypothesizing, and predicting intrusion alerts,”Computer Communications, vol. 29, no. 15, pp. 2917–2933, Sep.2006. [Online]. Available: http://dx.doi.org/10.1016/j.comcom.2006.04.001

[19] X. Qin and W. Lee, “Attack plan recognition and predictionusing causal networks,” in Proceedings of the 20th Annual ComputerSecurity Applications Conference, Dec 2004, pp. 370–379.

[20] B. C. Cheng, G. T. Liao, C. C. Huang, and M. T. Yu, “A novelprobabilistic matching algorithm for multi-stage attack forecasts,”IEEE Journal on Selected Areas in Communications, vol. 29, no. 7, pp.1438–1448, August 2011.

[21] D. S. Fava, S. R. Byers, and S. J. Yang, “Projecting cyberattacksthrough variable-length markov models,” IEEE Transactions onInformation Forensics and Security, vol. 3, no. 3, pp. 359–369, Sept2008.

[22] G. Werner, S. Yang, and K. McConky, “Time series forecasting ofcyber attack intensity,” in Proceedings of the 12th Annual Conferenceon Cyber and Information Security Research, ser. CISRC ’17. NewYork, NY, USA: ACM, 2017, pp. 18:1–18:3. [Online]. Available:http://doi.acm.org/10.1145/3064814.3064831

[23] N. Silver, The Signal and the Noise: Why So Many Predictions Fail-butSome Don’t. Penguin Publishing Group, 2012.

[24] P. E. Tetlock and D. Gardner, Superforecasting: The Art and Science ofPrediction. New York, NY, USA: Crown Publishing Group, 2015.

[25] S. Dua and X. Du, Data Mining and Machine Learning in Cybersecu-rity, 1st ed. Boston, MA, USA: Auerbach Publications, 2011.

[26] M. Hall, E. Frank, G. Holmes, B. Pfahringer et al., “Theweka data mining software: An update,” SIGKDD Explorations,vol. 11, no. 1, pp. 10–18, Nov. 2009. [Online]. Available:http://doi.acm.org/10.1145/1656274.1656278

[27] M. A. Hall, “Correlation-based feature subset selection for ma-chine learning,” Ph.D. dissertation, University of Waikato, Hamil-ton, New Zealand, 1998.

Ahmet Okutan Dr. Ahmet Okutan receivedhis B.S.degree in Computer Engineering fromBosphorus University, Istanbul, Turkey in 1998.He received his M.S. and Ph.D. degrees inComputer Engineering from Isik University, Is-tanbul, Turkey in 2002 and 2008 respectively.Dr. Okutan is currently a Post Doctoral Re-search Fellow in the Computer Engineering De-partment at Rochester Institute of Technology.He worked as analyst, architect, developer, andproject manager in more than 20 large scale

software projects. He has professional experience regarding softwaredesign and development, mobile application development, databasemanagement systems, and web technologies for more than 18 years.His current research interests include cyber attack forecasting, softwarequality prediction and software defectiveness prediction.

Shanchieh Jay Yang Dr. S. Jay Yang receivedhis B.S. degree in Electronics Engineering fromNational Chiao-Tung University, Hsin-Chu, Tai-wan in 1995, and his M.S. and Ph.D. degreesin Electrical and Computer Engineering fromthe University of Texas at Austin in, 1998 and2001, respectively. He is currently a Professorand the Department Head for the Departmentof Computer Engineering at RIT. He and hisresearch group has developed several systemsand frameworks in the area of cyber attack mod-

eling for predictive situation, threat and impact assessment. He haspublished more than sixty papers and was invited as a keynote speaker,a panelist, and a guest speaker in various venues. He was a co-chair forIEEE Joint Communications and Aerospace Chapter in Rochester NYin 2005, when the chapter was recognized as an Outstanding Chapterof Region 1. He has also contributed to the development of two Ph.D.programs at RIT, and received Norman A. Miles Award for AcademicExcellence in Teaching in 2007.

Katie McConky Dr. Katie McConky received herB.S. and M.S. degrees in Industrial Engineeringfrom Rochester Institute of Technology in 2005and 2007 respectively, and her Ph.D. in IndustrialEngineering from SUNY Buffalo in 2013. Dr. Mc-Conky is currently an Assistant Professor in theIndustrial and Systems Engineering Departmentat Rochester Institute of Technology, and hasseven years of industry experience working inthe fields of information fusion and predictivedata science prior to joining RIT. Currently Dr.

McConky‘s research work focuses on applying operations research anddata analytic techniques to energy and security applications, includingcyber security.