T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

OUTLIERS

HOW TO PREDICT A UMBC UPSETTHE FIRST 16 SEED WIN OVER A 1 SEED

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

JUST KIDDING!

IT RUINED MY BRACKET AS WELL…

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

59 TES

ANOMALY DETECTION: AIRCRAFT

SYSTEM HEALTH DATA STABILITY

REPORTING

PRESENTED BY 1LT KYLE “KODAC” GARTRELL

06/23/16This Briefing is:

UNCLASSIFIED

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Start with a Definition:

“An outlying observation, or outlier, is one that

appears to deviate markedly from other

members of the sample in which it occurs”

- Grubbs, 1969

“An observation which deviates so much from

other observations as to arouse suspicions that

it was generated by a different mechanism”.

- Hawkins, 1980

Going forward keep in mind the second

definition

ANOMALIES

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Health Code Reports (HRC)

Intelligent Queuing Data (ICUE)

Pilot Reports

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Asses System Stability Discovery

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Types of Anomalies:

Point Anomaly

Contextual Anomaly

Collective Anomaly (small cluster)

Anomalies Discovery

Outlier Categories:

Local Outlier - Density Based

Example Technique: LOF

Global Outlier - Distance Based

Example Technique: K-means

*The presentation on assessing system stability will be held Feb 29, 2019

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Local Outlier Factor

K-means

Pattern based Outlier

Detection using Support Vector Machines

(POD-SVM)

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Ex. a mission system fails or reports

degradation in a flight several

deviations outside the norm

Easy to see to yet hard to diagnose as

random or a new trend

Mission dependent

Distance and density techniques work

well

Point Anomaly: A single instance or observation different with

respect to the data set features

System Fail = a system loses total functionality and requires a reboot

System Degrade = a system loses partial functionality, but is still usable

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Contextual Anomaly: Considered abnormal when viewed against meta-information associated with the data points

Numerically similar clusters

When taken in context easy to

see

Computers have a hard time

analyzing categorical features

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Collective Anomaly: A collective anomaly designates a group of instances that exhibits an anomalous behavior compared to the other

groups of instances

Group of instances or actions

found nonconforming

Not necessarily a point anomaly

May be found base on distance

techniques, but density

techniques may struggle to

identify as anomalous

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Step One: Retrieve the system data from the Jet (ICUE)

Step Two: Subset the ICUE data for relevant mission systems and features

Step Three: Merge the Pilot Reports with the ICUE data and tidy

Step Four: Perform analysis and identify anomalous flights

Step Five: Use HRC data to identify the cause

Flights: 1489

Flight Hours: 2165

Time Frame: 2016-Present

Jets: 23

Mission Systems: 46

Location: Nellis AFB

The Data

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Created a new column of the sum total of all the degrades and fails

Took the Log transform of the totals to achieve a more “normal” distribution

Found the St. Dev of each data point and picked the top 10 outliers

Used these specific outliers to rank LOF, K-means, and POD-SVM

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Top Outliers were Flights with

least system failures (10 flights)

The top outlier is ranked 21st

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Centers = 6

Identified 4 of the top 10 outliers

(3, 4, 7, 8)

The top outlier is ranked 7th

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Identified 5 of the top 10 outliers

(2406, 2458, 2382, 2472, 2405)

The top outlier is ranked 1st

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

Top Outlier = 2406

1. MADOF: Top Outlier 1st

2. K-Means: Top Outlier 7th

3. LOF: Top Outlier 21st

*Each entry in the data table is the row id and corresponds to a specific flight

*Column header is the order rank for each outlier

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

SOURCES

Barnett, V., Lewis, T., 1994. Outliers in Statistical Data, 3rd ed, 584.

Mandar, Katdare, 2011. Pattern based Outlier Detection in Mixed-Attribute Datasets.

Hayes, A. Michael, Capretz, AM. Miriam, 2014. Contextual anomaly work for big

sensor data.

Retrieved from https://journalofbigdata.springeropen.com/articles/10.1186/s40537-

014-0011-y

Vijayakumar Jawaharlal

JawaHarlal, Vijayakumar, 2014. Knn Using caret R package. Retrieved from

https://rpubs.com/njvijay/16444

Saxena, Rahul, 2017. K-Nearest Neighbor Implementation in R Using Caret Package

KNN R. Retrieved from http://dataaspirant.com/2017/01/09/knn-implementation-r-

using-caret-package/

https://journalofbigdata.springeropen.com/articles/10.1186/s40537-014-0011-yhttps://rpubs.com/njvijay/16444http://dataaspirant.com/2017/01/09/knn-implementation-r-using-caret-package/

T e s t i n g - T a c t i c s - T r a i n i n g - I n n o v a t i o n - I n t e g r a t i o n

SOURCES

http://www.cse.ust.hk/~leichen/courses/comp5331/lectures/LOF_Example.pdf

Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, Jörg Sander, 2000. LOF:

Identifying Density-Based Local Outliers. Retrieved from

http://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf

ANALYSIS

Rstudio Version 1.1.419 – © 2009-2018 RStudio, Inc.

R i386 3.4.3

Packages:

ellipse, Rlof, caret, AppliedPredictiveModeling, DMwR, FNN, factoextra, gridExtra,

probsvm,stats, tidyr, e1071, distances, ggplot2, dplyr

http://www.cse.ust.hk/~leichen/courses/comp5331/lectures/LOF_Example.pdfhttp://www.dbs.ifi.lmu.de/Publikationen/Papers/LOF.pdf