Outlier Selection and One Class Classification by Jeroen Janssens

Outlier Selection andOne-Class Classification

Jeroen Janssens @jeroenhjanssens

. . . . . . . . . . . .Anomalies and outliers

. . . . . . . . . . . . . . . . . . . . . . .Stochastic Outlier Selection

. . . . . . . . . . .Experiments and results

Overview

• Anomalies and outliers

• Stochastic Outlier Selection

• Experiments and results

Outlier Selection and One-Class Classification Jeroen Janssens

Anomalies and outliers

Definition (Anomaly)

An anomaly is an observation or event that deviates qualitatively from what isconsidered to be normal, according to a domain expert.

Detecting anomalies is important

• Expensive

• Dangerous

• Mess up your model

Detecting anomalies is important

• Expensive

• Dangerous

• Mess up your model

Human anomaly detection may suffer from

• Fatigue

• Information overload

• Emotional bias

Computers work with numbers

height

. ..apple

. ..orange.

Visualisation

width height label8.4 7.3 apple6.7 7.1 orange8.0 6.8 apple7.4 7.2 apple9.6 9.2 orange .

Data points

Observations

height

. ..apple

. ..orange.

Visualisation

Data points

Observations

.....6

height

. ..apple

. ..orange.

Visualisation

Data points

Observations

From anomaly to outlier

speed over ground

rateofturn

. ..anomalous vessel

. ..normal vessel

Definition (Outlier)

An outlier is a data point that deviates quantitatively from the majority of thedata points, according to an outlier-selection algorithm.

Three standard deviations

.....−3σ

.−2σ

.−1σ

. ..outlier

. ..inlier

.. 3σ..

Euler diagram

Set legend

Set notation

real-world observations

X.X.(a)

labeled by expert as anomalous

labeled by expert as normal

X ∩A

data points

unrecorded

X ∩D

.(d) .

anomalies represented as data points

. normalities represented as data points.

CA = D ∩A. CN = D ∩A.A . D.

classified by algorithm as an outlier

classified by algorithm as an inlier

CI = D ∩ CO

misses

false alarms

correct rejects

H= CA ∩ CO

FA = CN ∩ CO

M= CA ∩ CI

CR = CN ∩ CI

Euler diagram

Set legend

Set notation

X.X.(a) .

X ∩A

data points

unrecorded

X ∩D

.(d) .

CA = D ∩A. CN = D ∩A.A . D.

CI = D ∩ CO

misses

false alarms

correct rejects

H= CA ∩ CO

FA = CN ∩ CO

M= CA ∩ CI

CR = CN ∩ CI

Euler diagram

Set legend

Set notation

X.X.(a) .

X ∩A

data points

unrecorded

X ∩D

.(d) .

CA = D ∩A. CN = D ∩A.A . D.

CI = D ∩ CO

misses

false alarms

correct rejects

H= CA ∩ CO

FA = CN ∩ CO

M= CA ∩ CI

CR = CN ∩ CI

Euler diagram

Set legend

Set notation

X.X.(a) .

X ∩A

data points

unrecorded

X ∩D

.(d) .

CA = D ∩A. CN = D ∩A.A . D.

CI = D ∩ CO

misses

false alarms

correct rejects

H= CA ∩ CO

FA = CN ∩ CO

M= CA ∩ CI

CR = CN ∩ CI

Euler diagram

Set legend

Set notation

X.X.(a) .

X ∩A

data points

unrecorded

X ∩D

.(d) .

CA = D ∩A. CN = D ∩A.A . D.

CI = D ∩ CO

misses

false alarms

correct rejects

H= CA ∩ CO

FA = CN ∩ CO

M= CA ∩ CI

CR = CN ∩ CI

Euler diagram

Set legend

Set notation

X.X.(a) .

X ∩A

data points

unrecorded

X ∩D

.(d) .

CA = D ∩A. CN = D ∩A.A . D.

CI = D ∩ CO

misses

false alarms

correct rejects

H= CA ∩ CO

FA = CN ∩ CO

M= CA ∩ CI

CR = CN ∩ CI

Confusion matrix

..hitHi

..false alarmFA

..missMi

..correct rejectCR

Anomaly (CA)

Normality (CN)

.Outlier(

.Inlier(

Expert labels the observation as a(n)

classi

esthedatapo

intasa

Data set

. ..data point

Labels by the expert

. ..anomaly. ..normality

.. .....

Classi cations by the algorithm

. ..inlier. ..outlier

Outcome

. ..hit. ..false alarm. ..miss. ..correct reject

Stochastic Outlier Selection

• Unsupervised outlier selection algorithm

• Employs concept of affinity

• Computes outlier probabilities

• One parameter: perplexity

• Inspired by t-SNE

A data point is selected as an outlier when all the other data pointshave insufficient affinity with it.

From input to output

..1. 2.1.

.. .. 0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.

..1.1.

.. .. 0.0.2

4.2.4 – 6

Subsections:

Demo: SOS on thecommand-line

(see http://sos.jeroenjanssens.com)

From feature matrix to dissimilarity matrix

...........

d2,6 ≡ d6,2 =5.056

rst feature x1

second

feature

. ..data point

..1. 2.1.

.. .. 0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.2

Equation 2.1

¿ÁÁÀ

∑k=1(xjk − xik)2

From feature matrix to dissimilarity matrix

...........

d2,6 ≡ d6,2 =5.056

rst feature x1

second

feature

. ..data point

..1. 2.1.

.. .. 0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.2

Equation 2.1

¿ÁÁÀ

∑k=1(xjk − xik)2

Affinity between data points

.....0.

10.0 .

dissimilarity dij

affinity

. ..σ 2i = 0.1

. ..σ 2i = 1

. ..σ 2i = 10

..1. 2. 3. 4. 5. 6.1.

.. .. 0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.2

Equation 4.2

aij =⎧⎪⎪⎨⎪⎪⎩

exp (−d 2ij / 2σ 2

i ) if i ≠ j0 if i = j

Affinity between data points

.....0.

10.0 .

dissimilarity dij

affinity

. ..σ 2i = 0.1

. ..σ 2i = 1

. ..σ 2i = 10

..1. 2. 3. 4. 5. 6.1.

.. .. 0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.2

Equation 4.2

aij =⎧⎪⎪⎨⎪⎪⎩

exp (−d 2ij / 2σ 2

i ) if i ≠ j0 if i = j

Smooth neighborhoods

...........

.−1.

9.−1 .

rst feature x1

second

feature

. ..σ 21. ..σ 2

2. ..σ 2

3. ..σ 2

4. ..σ 2

5. ..σ 2

From affinity to binding probability

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.

Equation 4.5 / 4.6

bij = p (i→ j ∈ EG)∝ aij

bij =aij

∑nk=1 aik

From affinity to binding probability

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.2

..1. 2. 3. 4. 5. 6.1.

.. .. 0.

Equation 4.5 / 4.6

bij = p (i→ j ∈ EG)∝ aij

bij =aij

∑nk=1 aik

Binding probabilities

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

.v1.v2.

. .04.

..1. 2. 3. 4. 5. 6.1.

.. .. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.05

..1. 2. 3. 4. 5. 6.1.

.. 0.0.01

Stochastic Neighbor Graph

G = (V,EG)

p(G) =∏i→j ∈EG

CO ∣G = {xi ∈ X ∣ deg−G(vi) = 0}= {xi ∈ X ∣ ∄vj ∈ V ∶ j→ i ∈ EG}= {xi ∈ X ∣ ∀vj ∈ V ∶ j→ i ∉ EG}

G = (V,EG)

.v1.v2.

p(Ga) = 3.931 ⋅ 10−4

.CO∣Ga = {x1, x4, x6}

p(Gb) = 4.562 ⋅ 10−5

CO∣Gb = {x5, x6}

p(Gc) = 5.950 ⋅ 10−7

CO∣Gc = {x1, x3}

Set of all SNGs

.....0 .

⋅10−4

abilitymass

.. ..0 .

.cumulativeprob

abilitymass

. ..h = 4.0

. ..h = 4.5

. ..h = 5.0

Approximating outlier probabilities by sampling SNGs

... ..100.

106.0 .

sampling iteration

outlierprob

ability

. ..x1

. ..x2

. ..x3

. ..x4

. ..x5

. ..x6

p(xi ∈ CO) = limS→∞

∑s=1

I{xi ∈ CO ∣G(s)} , G(s) ∼ P(G)

Demo: Sampling SNGs inCoffeeScript and D3(see http://sos.jeroenjanssens.com)

Computing outlier probabilities through marginalisation

p(xi ∈ CO) =∑G∈G

I{xi ∈ CO ∣G} ⋅ p(G)

=∑G∈G

I{xi ∈ CO ∣G} ⋅∏q→r ∈EG

∣G∣ = (n − 1)n

Computing outlier probabilities in closed form

...........

rst feature x1

second

feature

. ..data point

..1. 2. 3. 4. 5. 6.1.

.... 0.

..1.1.

.... 0.0.2

Equation 4.14

p(xi ∈ CO) =∏j≠i(1 − bji)

xi ∈ CO ∣G ⇐⇒ deg−G(vi) = 0 (1)

p(xi ∈ CO) = EG [I{deg−G(vi) = 0}] (2)

p(xi ∈ CO) = EG [∏j≠i

I{ j→ i ∉ EG}] (3)

p(xi ∈ CO) = EG [∏j≠i(1 − I{ j→ i ∈ EG})] (4)

p(xi ∈ CO) =∏j≠i(1 −EG [I{ j→ i ∈ EG}]) (5)

p(xi ∈ CO) =∏j≠i(1 − p( j→ i ∈ EG)) (6)

p(xi ∈ CO) =∏j≠i(1 − bji) (7)

Selecting outliers

...........

.−1.

9.−1 .

rst feature x1

second

feature

. ..outlier

. ..inlier

f(x) =⎧⎪⎪⎨⎪⎪⎩

outlier if p(x ∈ CO) > θ,inlier if p(x ∈ CO) ≤ θ.

Adaptive variances via the perplexity parameter

... ..10−1

variance σ 2i

perplexity

h(b i)

.. ..5

......

variance σ 2i

. ..h = 4.5

. ..x1

. ..x2

. ..x3

. ..x4

. ..x5

. ..x6

h(bi) = 2H(bi) , H(bi) = −n

∑j=1j≠i

bij log2 (bij)

Continuous binary search

.............

variance

. ..x1

. ..x2

. ..x3

. ..x4

. ..x5

. ..x6..

binary search iteration

perplexity

h(b i)

Perplexity influences outlier probabilities

.....0.

perplexity h

outlierprob

ability

. ..x1

. ..x2

. ..x3

. ..x4

. ..x5

. ..x6

Experiments and results

......

Banana

Densities

KNNDD(k=

F(k=10)

SOS(h=10)

One-class datasets

Sepal length

Petalw

DM = Iris ower data set

. ..C1 = Setosa

. ..C2 = Versicolor

. ..C3 = Virginica

......

. ..CA = Versicolor ∪ Virginica

. ..CN = Setosa

......

. ..CA = Setosa ∪ Virginica

. ..CN = Versicolor

.. ....

. ..CA = Setosa ∪ Versicolor

. ..CN = Virginica

Weighted AUC

.....0 .

outlierscore

. ..hit. ..false alarm. ..miss. ..correct reject

θ′.

.....0

false alarm rate

hitrate

.. ... 0.2.

θ′= 0.57

Real-world datasets

Breast W. Original.

Delft Pump.

Waveform Generator.

Wine Recognition.

Colon Gene.

BioMed.

Vehicle Silhouettes.

Glass Identi cation.

Boston Housing.

Heart Disease.

Arrhythmia.

Haberman’s Survival.

Breast W. New.

Liver Disorders.

SPECTF Heart.

Hepatitis.

. ..SOS . ..KNNDD . ..LOF . ..LOCI . ..LSOD

Synthetic datasets

Parameter λ

Data set Determines λstart step size λend

(a) Radius of ring 5 0.1 2(b) Cardinality of cluster and ring 100 5 5(c) Distance between clusters 4 0.1 0(d) Cardinality of one cluster and ring 0 5 45(e) Density of one cluster and ring 1 0.05 0(f ) Radius of square ring 2 0.05 0.8(g) Radius of square ring 2 0.05 0.8

.......

......

.......

......

.......

......

.......

......

.......

......

.......

......

.......

......

λstart

. λinter

...anom

..normality

Results on synthetic datasets

.. ...

....10 .20 .30 .40 .........

.2 .3 .4

....λ

.........

.0.6 .0.7

.........λ

..0.8 .1 .1.2 .1.4 .0.6

. ..SOS

. ..KNNDD

. ..LOF

. ..LOCI

. ..LSOD

....10 .20 .30 .40 .........

.2 .3 .4

....λ

.........

.0.6 .0.7

.........λ

..0.8 .1 .1.2 .1.4 .0.6

. ..SOS

. ..KNNDD

. ..LOF

. ..LOCI

. ..LSOD

Results on synthetic datasets

....10 .20 .30 .40 .........

.2 .3 .4

....λ

.........

.0.6 .0.7

.........λ

..0.8 .1 .1.2 .1.4 .0.6

. ..SOS

. ..KNNDD

. ..LOF

. ..LOCI

. ..LSOD

....10 .20 .30 .40 .........

.2 .3 .4

....λ

.........

.0.6 .0.7

.........λ

..0.8 .1 .1.2 .1.4 .0.6

. ..SOS

. ..KNNDD

. ..LOF

. ..LOCI

. ..LSOD

SOS performs significantly better

cd (p < .05)

cd (p < .01)

Conclusion

• Outlier selection can support the detection of anomalies

• SOS is an intuitive and probabilistic algorithm to select outliers

• SOS has a very good performance

• No free lunch

Outlier Selection andOne-Class Classification

Jeroen Janssens @jeroenhjanssens

Outlier Selection and One Class Classification by Jeroen Janssens

Technology

Transcript of Outlier Selection and One Class Classification by Jeroen Janssens

16.40 o4 s janssens

Outlier Detection for Temporal Data Outlier D for Temp Outlier for Tem

Frieke Janssens

Multivariate outlier detection

Class Outlier Mining

ISTD Response Outlier Report - chem-agilent.com · ISTD Response Outlier Report High Low High RT 2.453 Outlier High Response 20506 Min 18000 Max 20000 RT 2.128 Outlier Low Data …

Traffic Outlier Detection by Density-Based Bounded Local Outlier … · 2017-11-04 · Traffic Outlier Detection by Density-Based Bounded Local Outlier Factors Jialing Tang and Henry

Ronny Janssens Validation G 185 Incubator

Presentation sustain-it - Laurent Janssens

Outlier Detection Techniques For Wireless Sensor Networks ... · Additional Key Words and Phrases: Outlier, outlier detection, wireless sensor networks, taxonomy framework, decision

Cv Jeroen Oskam

Robust outlier detection

IMGENT - Tom Janssens

Open City, Smart City - Tom Janssens

Outlier brochure

JEROEN NELEMANS

The Outlier - Swamy

2011 jeroen vanhoudt_ngs

Jeroen Stage Director

Presentatie Jeroen Smits