University of Virginia Department of Systems & Information Engineering Feature Selection,...

University of VirginiaUniversity of VirginiaDepartment of Systems & Information EngineeringDepartment of Systems & Information Engineering

Feature Selection, Prediction, and Feature Selection, Prediction, and Change Detection in Terrorist/Insurgency Change Detection in Terrorist/Insurgency Processes Using a Spatial Point Process Processes Using a Spatial Point Process

ApproachApproach

Feature Selection, Prediction, and Feature Selection, Prediction, and Change Detection in Terrorist/Insurgency Change Detection in Terrorist/Insurgency Processes Using a Spatial Point Process Processes Using a Spatial Point Process

ApproachApproach

Michael D. Porter and Donald E. Brown{mdp2u,brown}@virginia.edu

Department of Systems and Information EngineeringUniversity of Virginia


2

AgendaAgendaAgendaAgenda

• Problem Statement

• Approach & Specification

• Feature Selection (Review)

• Prediction (Review)

• Change Detection

• IED Example

/42


3

Terrorism Against the U.S.Terrorism Against the U.S.Terrorism Against the U.S.Terrorism Against the U.S.

• Terrorist attacks take many forms

– suicide bombings– improvised explosive

devices– hostage taking– mortar & rocket attacks– Complex attacks

• The incident on the right was a suicide bombing at a police station in Iraq that occurred on February 12, 2004 & killed 47 people

• How can we become proactive in addressing these threats?

Hull, Bryson, “100 die in two Iraq suicide bombings,” The Age, February 12, 2004, http://www.theage.com.au/

/42


4

Modeling Terrorism and Modeling Terrorism and InsurgencyInsurgency

Modeling Terrorism and Modeling Terrorism and InsurgencyInsurgency

PREDICTION

• Proactive in force deployment

• Awareness of threat locations and levels

• Should include terrorist decision making

CHANGE DETECTION

• New terrorist group

• Interventions effects (arrests, patrols, etc)

• New terrorist strategy/preferences

/42


5

Intelligent Site SelectionIntelligent Site SelectionIntelligent Site SelectionIntelligent Site Selection

Assumptions:

• Terrorists judiciously plan the location of their attacks according to their preferences or perceived utility of that location

– Site selection is not made by considering spatial coordinates alone

• Site selection is made based on the attributes or features of each possible location

• Capturing these features that are considered by the terrorist will lead to better (more accurate) models

/42


6

Criminal Ecological TheoryCriminal Ecological TheoryCriminal Ecological TheoryCriminal Ecological Theory

• In criminology, ecological theories seek to describe the motivations and acts of crime based on the general features of one’s environment

– Environment of criminal

– Environment of crime scene

• This points to the explanation that there are some features of the location of crimes which are important to criminal decision making

• We extend this thought to terrorist decision making

/42


7

Feature SetFeature SetFeature SetFeature Set

• To successfully model the terrorist attacks, we should attempt to model their decision making process or preferences for attack locations

• Thus we include features that are thought to influence the terrorist site selection process (or that are associated (correlated) with such features) in our models

• Since we usually don’t know the terrorist’s preferences we must discover these from previous attack locations

– Observe past attack locations and associated feature values for that location

• Examples of possible features– Census (Socio-economic)

– Proximity (Distance to landmarks or structures)

– Military or Police Patrols (times and locations)

/42


8

Point ProcessesPoint ProcessesPoint ProcessesPoint Processes

• A stochastic model governing the location

and number of events in some set X [Cressie, 1993]

• A point process P = {xi : xi X }, where the xi

are the events and X is the set over which the point process is defined

• Specified by

where - mean measure and - intensity function

•(Poisson) !

)]([)}(exp{))(Pr(

nBB

nBNn

B

dxxB )()(

/42


9

Spatially Defined Point Spatially Defined Point ProcessesProcesses

Spatially Defined Point Spatially Defined Point ProcessesProcesses

• For spatially defined PP, events are located in a bounded plane, A 2

– where A is the study region and the events are

• In our processes, time cannot be ignored– Spatio-temporal model (full model)

– Spatial model (time integrated out)

– Time series model (space integrated out)

NisssAss iiiii ,,1,,,: 2,1,

tdstsfttsfts whereA ,1)|(,)()|(),( Spatial Density (Preferences)

Temporal Rate

T

dttss0

),()(

)(),()( tdstst AA

/42


10

Feature SpaceFeature SpaceFeature SpaceFeature Space

• The feature space G, defines the additional attribute information relating to the locations in the study region

– g(s,t)=[g1(s,t),g2(s,t),…,gp(s,t)]

• G defines p other spatially defined stochastic processes

– Real (perhaps discrete), ordered, or categorical valued

– High Dimensional (large p)

• The PP is specified conditionally on the values of the other stochastic processes

– The intensity is dependent upon the feature values

(s,t) (s,t|g(s,t))

/42


11

Feature Space FormulationFeature Space FormulationFeature Space FormulationFeature Space Formulation

• Since we condition on the value of g it is assumed that given any (s,t), the value g(s,t) can be determined

– Using GIS

(s,t)=h(g(s,t))– g is a realization of G

– h is the function we want to determine

• Two ways to examine model– Define the PP in feature space

– Look at association between PP and random processes

/42


12

Overview of the point process Overview of the point process modelmodel

Overview of the point process Overview of the point process modelmodel

0 t1 t2 t3 t4 t5 t6 t7 TTimeAxis

. . .

f1

f2

fi

fp

...

.

..FeatureSpace

x1 x3

x4

x2x5

x6

x7

GeographicSpace

s1

s2s1 s3

s4

s2 s5

s6 s7

Liu and Brown 1999

/42


13

Spatial Association / Feature Spatial Association / Feature SelectionSelection

Spatial Association / Feature Spatial Association / Feature SelectionSelection

• Find the features that are associated with the point process

– Correlated with PP, but not necessarily causal

– This is preference discovery

• Two procedures– Find important features

– Remove redundant (correlated) features

Important Features (Foxall and Baddeley 2002)

• J(r)=[1-G(r)]/[1-F(r)]

=Pr(event>r)/Pr(grid>r)» J®<1 suggests attraction, J>1 suggests repulsion» Feature is important if J(r) is much different than

1 for small r

/42


140 0.02 0.04 0.06 0.08 0.1 0.120

1

2

3

4

5

6

7Feature g

6

r

J(r)

0 0.02 0.04 0.06 0.08 0.1 0.120

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Feature g

6

r

CD

F

G-EventsF-Grid

0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.040

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Feature g

5

r

J(r)

0 0.005 0.01 0.015 0.02 0.025 0.03 0.035 0.040

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Feature g

5

r

CD

FG-EventsF-Grid

Values much lower that 1 suggest the PP is associated with feature g5

Values close to 1 suggest the PP is not associated with feature g6


15

Removing Redundant FeaturesRemoving Redundant FeaturesRemoving Redundant FeaturesRemoving Redundant Features

• If a feature is redundant, its intensity can be explained by another feature

• Rescaling Theorem– Rescale events {ei} {ei’} so ei’ form homogeneous Poisson

with rate =1

• Estimate intensity based on one feature m (perhaps the most important from the last step)

• Use this to test for homogeneity of rescaled events in feature k (Let rki=gk(si) )

/42

ie

i duue0

)(

duurki

m

r

gki )(ˆ0

hrr

N

ihrp

i

mrg

1

1)(

1)(ˆ (Diggle, 1985 Applied Statistics)


16

0 50 100 1500

5

10

15

20

25Histogram of g

3 based on intensity from g

5

Fre

qu

en

cy

r'3

0 50 100 1500

5

10

15

20

25Histogram of g

12 based on intensity from g

5

Fre

qu

en

cy

r'12

Redundant Features – con’tRedundant Features – con’tRedundant Features – con’tRedundant Features – con’t

• Tests for homogeneity or uniform– K function (Cressie 1993)– Nearest Neighbor– KS– Scan Statistics

/42


17

PredictionPredictionPredictionPrediction

• Build space-time prediction model

(s,t+1 | g(s,t), t)where t is the history of the PP up to time t

Possible assumptions:1. Feature values stationary / deterministic

• Implies g(s,t)=g(s) for all t

2. Intensity (or preferences) stationary for next time period• Implies (s,t+1)=(s,t) or f(s|t+1)=f(s|t)

3. Intensity is space-time separable• Implies (s,t+1)=f(s)A(t+1)• Model spatial as density and temporal as time series

/42


18

Prediction ModelsPrediction ModelsPrediction ModelsPrediction Models

• Poisson/ regression– Log{(s)}=0 + 1 g1(s)+ 2 g2(s)+ . . .– Should use GAM or transformation since distribution of

g is non-uniform on A

• Mixed models1

(s)=ifi(s;i)

• Spatial Choice Models2

C

V

V

sdeMseMs

sfs

i

}Pr{}Pr{

)(

1 Liu,H. and Brown, D.E. (2004). A New Point Process Transaction Density Model for Space-Time Event Prediction. Systems, Man and Cybernetics, Part C, IEEE Transactions on, Volume 34, Issue 3, Pg.310 - 324 2 Smith, M.A. (2005). CHOICE MODELING OF BOMBING ATTACK SITE SELECTION, Masters Thesis, Dept of Systems and Information Engineering, University of Virginia, May.

/42


19

Change DetectionChange DetectionChange DetectionChange Detection

• Statistically detect a region, B, of change between two time periods– Detect changes between two spatial point processes

• Steps1. Form hypothesis test

2. Select an appropriate test statistic to test hypothesis

3. Specify how region B will be identified for testing

4. Establish significance testing

5. Evaluate the results

• Method also applicable for detecting differences between two types of events or case-control studies

/42


20

Formulation: Marked Point Formulation: Marked Point ProcessProcess

Formulation: Marked Point Formulation: Marked Point ProcessProcess

Let the terrorist process be represented by P = {(si,ki)}, a marked PP on space X = A x K, A2, K={1,2}

Assuming the ground process is a nonhomogeneous Poisson spatial PP and the marks are independent of each other, P can be specified by:

An observation is:

}}2,1{,:)|()(),({ kAsskfsks g

nNksksks nn ),,(,),,(),,( 2211

/42


21

Hypothesis Test Hypothesis Test (Assuming equal time periods and no temporal trends)(Assuming equal time periods and no temporal trends)


Change Detection

1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate

If unequal time periods or temporal trends, set

where i is the time period i. Then under the null:

i

dttTiT )()(

)()(

)(

)()2,(

)()1,(

:

2

1

21

T

T

TT

o

s

ksksH

* If unequal time periods and no trends, T(1 )/ T(2 )=(1)/(2), where (i) is the length of time period i.

/42


22



Change Detection

1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate/42


23



Change Detection

1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate/42


24

Statistical Test – Generalized Statistical Test – Generalized Likelihood RatioLikelihood Ratio

Statistical Test – Generalized Statistical Test – Generalized Likelihood RatioLikelihood Ratio

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 3) Identify region 3) Identify region B B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate

Assuming equal time periods and no temporal trends{

/42


25

Finding Change RegionFinding Change RegionFinding Change RegionFinding Change Region

• B defines the geographical region in A where change has occurred in the intensity function of the point process

• For testing the null hypothesis, (s)= Ho, we must identify the region B

• We do this by searching over A for the region B* that provides the minimum value of T(,B)

• Procedure of scanning for extrema regions for significance testing is termed scan process (e.g. scan statistic)

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 2) Test statistic 3) Identify region B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate/42


26

Scan ProcessScan ProcessScan ProcessScan Process

• In general, scan processes are used to detect significant clusters of events while accounting for multiple hypothesis testing

• Create a window, Wx of some geometry and move it over the entire region of interest ( x X) and calculate some score for each window, S(Wx).

• The multiple hypothesis problem is resolved by testing on the scan statistic, SS=maxxX S(Wx)

– Dependent on geometry selected for W

– But multiple geometries can be considered

Change Detection



27

Modified Scan Process for Modified Scan Process for Change DetectionChange Detection

Modified Scan Process for Modified Scan Process for Change DetectionChange Detection

• Here, we want to find window location and geometry producing the minimum value of the likelihood ratio statistic

• Thus our scan statistic becomes

SS() = minBB T(,B)

= T(,B*)

where B is the set of windows that we searched over

• The windows are restricted to be connected sets in map and feature space (i.e. BX=AxG)

– One change region

– Hyper-rectangles in the real variables

– No restriction on size of window

Change Detection



28

Approximation to scan statisticApproximation to scan statisticApproximation to scan statisticApproximation to scan statistic

• When the number of events and/or dimensionality of G is large, searching over all possible windows is computationally prohibitive.

• Instead, find good approximation to scan statistic

• Search over restricted set B’ B and hope this set includes B*, or a large portion of it

• Use PRIM to search for B* and thus the value of our scan statistic

Change Detection



29

Patient Rule Induction Method Patient Rule Induction Method (PRIM)(PRIM)

Patient Rule Induction Method Patient Rule Induction Method (PRIM)(PRIM)

• Patient Rule Induction Method

• PRIM handles high-dimensional and mixed data well

• Find the boxes (hyper-rectangles) where the response (i.e. T) is low

• Procedure is to produce a series of boxes, {Bk}, decreasing in size, by successively peeling away a subbox in such a manner that each new box Bk+1 has the lowest value for T(,Bk+1) among all possible subboxes.

Change Detection



30

PRIM – con’tPRIM – con’tPRIM – con’tPRIM – con’t

• Define the subboxes

• xj(k) is the k–quantile of the xj values for the data within the current box Bk

• k (0,1) dictates the size of the subboxes

• The optimal subbox for peeling becomes

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 2) Test statistic 3) Identify region B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate

)(:

)1(:)(

)(:)(

mjxxb

xxxb

xxxb

jmjk

kjjkjk

kjjkjk

(For catagorical variables)

),(maxarg ,*

},,{}2,,2,1{

kjk bTbmpj

/42


31

PRIM – con’tPRIM – con’tPRIM – con’tPRIM – con’t

• This creates a new box Bk+1 = Bk-bk*• Continue peeling until T can no longer be

decreased beyond minimum at step k• After peeling procedure, pasting step can

further improve box– Add subbox in similar manner as peeling

• To help find global minimum, allow variable k and run PRIM for J iterations

– Produces

• Leads to test of significance of

for change in region

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 2) Test statistic 3) Identify region B 4) Significance test 5) Evaluate4) Significance test 5) Evaluate

jJjSSSS~

min

),(minarg *

}:{

*~

*j

JjB

BTBj

~~

*, jj BSS

/42


32

Significance – Monte CarloSignificance – Monte CarloSignificance – Monte CarloSignificance – Monte Carlo

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 5) Evaluate 5) Evaluate/42


33

Simulated ExampleSimulated ExampleSimulated ExampleSimulated Example

• Examine region A on unit square– Six features identified: G=[g1,g2,…,g6]– Features {g1,g2,g3,g6} are proximity to nearest landmark– Features {g4,g5} are values of some variable recorded at

census tract level

• Group of insurgents operating in region A 1(s)=exp{-cys2-c1g1(s)+c5g5(s)}/C1

– cy=0.5, c1=100, c5=1,C1=15.5– E[N(A)]=100

• Random attacks with intensity r=15• New group active in 2

2(s)=I(g2(s)0.10,g3(s)0.05, g4(s)0.30)/C2

– I is indicator function and C2=12– E[N2(A)]=35

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 4) Significance test 5) Evaluate/42


34

Simulated Feature ValuesSimulated Feature ValuesSimulated Feature ValuesSimulated Feature Values

Change Detection



35

ProcedureProcedureProcedureProcedure

• Thus, (s,k=1)= 1(s)+ r for 1 and (s,k=2)= 1(s)+ r + 1(s) for 2

• Let (1)=(2) so under null, (s)=1 everywhere in A

• Generate 100 realizations of the PP and for each apply PRIM with k [0.05,0.15]

• For each original observation J=500 iterations were performed

• For each realization of the simulated PP, 99 simulations under H0 are created by random labeling with f(k=i|s)=0.50 and p-values are calculated

• Results in set Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 4) Significance test 5) Evaluate

100

1

* ˆ,

iii pB

/42


36

MeasuresMeasuresMeasuresMeasures

• Let Ei be the events from insurgency group i, with random events assigned to group 1

• (R2,R4) – region where change occurs

• (R1,R3) – events that constitute change

Change Detection



37

ResultsResultsResultsResults

Change Detection

1) Hypothesis test1) Hypothesis test 2) Test statistic 3) Identify region 2) Test statistic 3) Identify region B B 4) Significance test 4) Significance test 5) Evaluate

ROC

PD-Prob to Detect

Detection declared if dRp 1andˆ

/42


38

IED Attacks in IraqIED Attacks in IraqIED Attacks in IraqIED Attacks in Iraq

• Major method of attacking U.S. forces in Iraq

• Responsible for more U.S. deaths than any other attack mode

• Inexpensive, easy to deploy, and deadly

• Picture on right shows U.S. troops with IED on March 15, 2004

Picture from http://www.middle-east-online.com/english/?id=9250

/42


39

IED Change RegionIED Change RegionIED Change RegionIED Change Region

• N1=151 events, N2=91 events

• |1|=|2|• J=500, M=999 Monte Carlo

/42


40

IED Change Region - con’tIED Change Region - con’tIED Change Region - con’tIED Change Region - con’t

• Region B* contains 48 events from 1 and 0 events from 2

• PRIM peeled 16 times on 14 features

• B* composes approx. 21% of the total area of A• Estimated p-value is 0.001 so reject null and conclude

change has occurred in region B*

/42


41

ConclusionsConclusionsConclusionsConclusions

• Modeling terrorist processes is difficult yet important

• Point process approach provides small scale analysis

• Including terrorist decision making process leads to better models

• Non-model based change detection method can detect presence of new terrorist group or changing terrorist attack strategies

/42


42

Background: Recent papers modeling Background: Recent papers modeling criminal or terrorist attackscriminal or terrorist attacks

Background: Recent papers modeling Background: Recent papers modeling criminal or terrorist attackscriminal or terrorist attacks

1. Y. Xue and D.E. Brown, “Spatial Analysis with Preference Specification for Latent Decision Makers for Criminal Event Prediction,” Decision Support Systems, forthcoming.

2. Porter, M.D. and Brown, D.E., “Detecting Local Regions of Change in High-Dimensional Criminal or Terrorist Point Processes”, forthcoming

3. Porter, M.D. and Brown, D.E., “Finding Changing Crime Regions: Use of High Dimensional Geographic Feature Space and Classification Trees”. Proceedings of the Eight Crime Mapping Research Conference, Sept. 7-10, 2005 Savannah, GA.

4. Smith, M.A., “Choice Modeling of Bombing Attack Site Selection,” Masters Thesis, Dept of Sys. and Info Engineering, University of Virginia, May 2005.

5. H. Liu and D.E. Brown, “A New Point Process Transition Density Model for Space-Time Event Prediction,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews,Vol.34, No.3, 2004, pp. 310-324.

6. D.E. Brown, J. Dalton, and H. Hoyle, “Spatial Forecast Methods for Terrorist Events in Urban Environments,” Second NSF/NIJ Symposium on Intelligence and Security Informatics, Tucson, AZ, June 2004, pp. 426-435.

7. Liu, H. and D.E. Brown, “Criminal Incident Prediction Using a Point-pattern Based Density Model” Int. Journal of Forecasting, vol. 19 (4), 2003, pp. 603-622.

8. Xue, Y. and D.E. Brown, “A Decision Model for Spatial Site Selection by Criminals: A Foundation for Law Enforcement Decision Support,” IEEE Transactions on System, Man, and Cybernetics, Part C: Applications and Reviews, Vol. 33, No. 1, February 2003, pp. 78-85.

/42

University of Virginia Department of Systems & Information Engineering Feature Selection,...

Documents

Transcript of University of Virginia Department of Systems & Information Engineering Feature Selection,...