ECE 494 Capstone DesignFinal Design Presentation
Smartphone Based Human Behavioral Analysis Andrew Jackson Michael Armstrong Robbie Rosati Andy McWilliams Aaron Stewart
April 18, 2014 Advisor: Dr. Fei Hu
Cyber Physical
Team
2
Outline
4/18/2014Smartphone Based Human Behavioral Analysis
Project Recap & Goal System Diagram Subsystem Breakdown & Team Roles Sensor Data Extraction & Analysis App Development & Activity Recognition DTW Algorithm Environment Sensors NN Algorithm Raw Memory Extraction & Analysis HMM Algorithm Administration
3
Project Recap & Goal
4/18/2014Smartphone Based Human Behavioral Analysis
Create a system for tracking and detecting a user’s behavioral patterns though the phone’s sensors and internal memory logs.
Can be used in multiple areas: Healthcare Activity Monitoring Homeland Security
4
System Diagram
4/18/2014Smartphone Based Human Behavioral Analysis
Sensors
Hidden Markov Model
Graphs
Binary Data
Acoustic
Accelerometer
Light
Reverse Engineering Hidden Markov Model
Neural Networks
Dynamic Time Warping
Call log, Deleted phone numbers, deleted texts
Speech recognition, shouting, snoring, coughing
Walking, Running, Going up stairs, Sitting, Standing
Daytime/nighttime, inside/outside, in or out of pocket
Internal Memory
5
Behavior Recognition
Phone Data
Sensor Data Extraction &
Analysis
Accelerometer
Light
Sound
Proximity
GPS
Raw Memory Extraction &
Analysis
dc3dd
Call Logs
Text Messages
Machine Learning
Algorithms
Hidden Markov Model
Neural Networks
Dynamic Time Warping
Subsystem Breakdown & Team Roles
4/18/2014Smartphone Based Human Behavioral Analysis
Andrew Jackson• Android Sensor App• DTW AlgorithmAaron Stewart• Raw Memory Analysis• Team LeaderRobbie Rosati • Machine Learning
Algorithms• Raw Memory ExtractionMichael Armstrong• Raw Memory Extraction• HMM AlgorithmAndy McWilliams• HMM Algorithm • Raw Memory Analysis
Sensor Data Extraction & Analysis
Aaron Stewart
7
Android, iOS, or Windows PhoneWe chose to go with
Android. No development license
fees Majority of market Many sensors available Open file system Previous Android
programming experience Cheaper phone prices
4/18/2014Smartphone Based Human Behavioral Analysis
8
Samsung Galaxy S4 Mini
4/18/2014Smartphone Based Human Behavioral Analysis
New phone with many sensors
Fairly inexpensive for an unlocked phone
Smaller size great for testing in pockets
Smartphone Based Human Behavioral Analysis9
Accelerometer
Light
SoundProximity
GPS
4/18/2014
Sensors
10
Sensor Types
4/18/2014Smartphone Based Human Behavioral Analysis
Accelerometer• Measures acceleration with gravity
value• Reports data in 3 values, one for
each axis• Example: Could tell if user is walking
or stationaryGPS• Reports GPS coordinates• See changes in location• Example: Could tell if user is driving
11
Sensor Types
4/18/2014Smartphone Based Human Behavioral Analysis
Light• Intensity of light• Reports data in a single value• Example: Indoors/Outdoors, Weather, Time of
daySound• Detects sound waves• Reports loudness• Example: Voice recognition
Proximity• Changes depending on surrounding objects• Boolean value• Example: Device is in tight space such as pocket
or purse
12
Behavior Analysis App Version 1.0
4/18/2014Smartphone Based Human Behavioral Analysis
Shows values from each sensor in real-time as the sensor updates
Can save the data to a file on phone
Move file to computer to analyze with MATLAB
Graph the data Validate that it makes
sense
13
Testing with App
3/10/2014Smartphone Based Human Behavioral Analysis
Most sensor data comes in a set of 3 points
Few have single values
Tests related to typical smartphone user behaviors
Used MATLAB for data graphing and filtering
14
Light Sensor Test (Different Exposure)
4/18/2014Smartphone Based Human Behavioral Analysis
15
Accelerometer Test (Answer Call)
4/18/2014Smartphone Based Human Behavioral Analysis
X
Y
Z
App Development and Activity Recognition
Andrew Jackson
4/18/2014Smartphone Based Human Behavioral Analysis
17
Sensors
Hidden Markov Model
Graphs
Binary Data
Acoustic
Accelerometer
Light
Reverse Engineering Hidden Markov Model
Neural Networks
Dynamic Time Warping
Call log, Deleted phone numbers, deleted texts
Speech recognition, shouting, snoring, coughing
Walking, Running, Going up stairs, Sitting, Standing
Daytime/nighttime, inside/outside, in or out of pocket
Internal Memory
18
Behavior Analysis App Version 2.0
4/18/2014Smartphone Based Human Behavioral Analysis
Integrates DTW machine learning algorithm
Allows training of the algorithm from the phone – no computer required
Once trained, will recognize where the phone is with almost 100% accuracy
Visual Upgrades All features from previous
app rolled over
19
Further App Improvements
4/18/2014Smartphone Based Human Behavioral Analysis
Increased accuracy of activity recognition by Increasing sensor sensitivity Polling each sensor more times a
second Tweaking DTW algorithm code Using multiple sensors at the same
time Continued adding new activities Added speech recognitionBy demo day:
Add in GPS coordinates Further explore light, sound, and
proximity sensor possibilities
20
Accelerometer, Light, and Proximity
4/18/2014Smartphone Based Human Behavioral Analysis
Chose to work with accelerometer first because it can give some of the most useful data about the phone
Applying DTW on accelerometer data
Next, applied light and proximity sensors to get better results
21
Recognized ActivitiesLast Presentation Now
4/18/2014Smartphone Based Human Behavioral Analysis
Walking (with phone in hand)
Talking on phone Sitting on table Holding in hand
Walking (with phone in pocket)
Running Stairs Sitting Driving
Smartphone Based Human Behavioral Analysis22 4/18/2014
App Demonstration
Dynamic Time Warping
Andrew Jackson
Smartphone Based Human Behavioral Analysis
24
Dynamic Time Warping Measures similarity between two sequences
which may vary in time or speed Calculates an optimal match between the two
given sequences or time series A distance-like quantity is measured between
the two series
4/18/2014
Time
25
DTW Time Series
4/18/2014Smartphone Based Human Behavioral Analysis
Time series can be accelerated/decelerated as much as necessary to give an optimal match.
Cost: 3.3084e+05
Cost: 2.7239e+06
Smartphone Based Human Behavioral Analysis
26
Applying DTW to the Project
4/18/2014
Allows training for different activities with disregard to time
Supports three-dimensional data Provides extremely accurate results Runs in O(n2) time
27
FastDTW
4/18/2014Smartphone Based Human Behavioral Analysis
Open-source algorithm based on original DTW
Runs in O(n) time
Slightly less accurate
Easy to port to Android
Execution Time of FastDTW on Large Time Series
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 180,000Length of Time Series
Tim
e (s
econ
ds)
DTWFastDTW (radius=100)FastDTW (radius=20)FastDTW (radius=0)
Environmental Sensors
Robbie Rosati
29
Proximity Sensor
4/18/2014Smartphone Based Human Behavioral Analysis
Gives proximity in cm Most phones only return binary values “near”
and “far” Ability to check whether phone is pressed
against ear, in pocket, etc.
30
Light Sensor
4/18/2014Smartphone Based Human Behavioral Analysis
Detects ambient luminosity in lux
Useful for indoor/outdoor detection
Could be included in gestures, used with DTW
31
GPS
4/18/2014Smartphone Based Human Behavioral Analysis
Can get user’s latitude and longitude
Could improve detection for if user is driving
Worst sensor with battery life
Need to only use it occasionally
32
Microphone
4/18/2014Smartphone Based Human Behavioral Analysis
Speaker recognition via NN algorithm Passive or active detection Could use to detect loudness of rooms Also see things in sound waveform – like
snoring
Neural Networks
Robbie Rosati
34
Transition from Support Vector Machine
to Neural Networks
4/18/2014Smartphone Based Human Behavioral Analysis
Implemented SVM into our Behavior Analysis app
Ran too inefficiently for phone hardware
Difficult to train with the sound sensor
Therefore, decided to use
an alternative algorithm that would fit our needs. Decided on Neural
Networks, a popular machine learning algorithm for speech recognition
35
Neural Networks
4/18/2014Smartphone Based Human Behavioral Analysis
Algorithm used for machine learning and pattern recognition
Inspired by the way the brain recognizes objects and sound
Presented as systems of interconnected neurons that can compute values
Difficult to train in a short time so used an API from Google to offload processing from the phone
Integrates speech recognition into the app
36
Google Speech API
4/18/2014Smartphone Based Human Behavioral Analysis
Open source API for speech recognition
Could be coded into our Behavior Analysis app
Uses NN to interpret speech
Can be presented in text with further coding
However, the app requires an internet connection for this function since it streams audio to remote servers
Smartphone Based Human Behavioral Analysis37 4/18/2014
App Demonstration
Raw Memory Extraction & Analysis
Michael Armstrong
4/18/2014Smartphone Based Human Behavioral Analysis
39
Sensors
Hidden Markov Model
Graphs
Binary Data
Acoustic
Accelerometer
Light
Reverse Engineering Hidden Markov Model
Neural Networks
Dynamic Time Warping
Call log, Deleted phone numbers, deleted texts
Speech recognition, shouting, snoring, coughing
Walking, Running, Going up stairs, Sitting, Standing
Daytime/nighttime, inside/outside, in or out of pocket
Internal Memory
40
Overview
4/18/2014Smartphone Based Human Behavioral Analysis
Retrieving the physical image of a device is our goal. Immense variety of phones with an array of OS
and applications. Current solutions are time consuming and/or very
expensive. Access to deleted data
A logical image is easier to obtain, but it omits deleted data, and logical extraction interfaces usually enforce access rules and may modify data upon access.
41
Samsung Galaxy S3 Mini
4/18/2014Smartphone Based Human Behavioral Analysis
Relatively inexpensive
Compatible with team’s SIM cards
SD card slot
Has some sensors in case we need it as back up for sensor testing
42
Testing Options
4/18/2014Smartphone Based Human Behavioral Analysis
Flashing is interpreted as a dump of the phones memory into a format that is either hexadecimal or binary. Backup Software Flashing box Linux Forensics Software dc3dd
43
dc3dd
4/18/2014Smartphone Based Human Behavioral Analysis
Terminal based utility for Linux Parses a partition bit-by-bit and creates binary
image Advantages: Exactly what we need, easy to
use Disadvantage: Parses through empty space
and fills it with zeroes, creating a very large image file full of nothing however, there is a workaround for this issue
44
Steps for Extraction
4/18/2014Smartphone Based Human Behavioral Analysis
Gain root access to Android using simple utility
Extract a copy of the database file Shrink the partition of a USB flash drive to the
smallest possible size and copy the database file to the partition
Use dc3dd to parse the partition and create the binary image for the database file
45
Viewing the Image File
4/18/2014Smartphone Based Human Behavioral Analysis
Any hex viewer application should able to view the data inside the image
Chose GHex because it is easy to obtain with Ubuntu and has search functions for finding the data we are looking for
46
Finding Desired Data
4/18/2014Smartphone Based Human Behavioral Analysis
Messages and numbers are encoded as ASCII
Locate corresponding hex data using the search function in Ghex
Verify by comparing the located hex data to ASCII values
Process should work for any type of file because the ASCII data will there regardless of file type
47
Example: Text Message
4/18/2014Smartphone Based Human Behavioral Analysis
48
‘Strings’ and ‘Grep’ Commands
4/18/2014Smartphone Based Human Behavioral Analysis
Strings - Captures groups of letters separated by spaces in a file
Grep - Searches for an expression matching the user’s input expression
Hidden Markov Model
Andy McWilliams
Smartphone Based Human Behavioral Analysis
50
Hidden Markov Model
4/18/2014
Way of predicting the next possible output in a sequence of events
Equivalent to recognizing or interpreting that particular sequence
Slightly complicated because there can be multiple ways to produce the same observed output
51
Hidden Markov Model
4/18/2014Smartphone Based Human Behavioral Analysis
Accomplished through MATLAB There will be different matrices such as the
transition matrix and emission matrix Transition matrix – probability of transitioning
from one state to the next Emission matrix – probability that one state
emits the next state
52
Hidden Markov Model
4/18/2014Smartphone Based Human Behavioral Analysis
1. Model Evaluation Problem What is the probability of the observation? Given an observed sequence and an HMM, how
probable is that sequence? Forward algorithm
2. Path Decoding Problem What is the best hidden state sequence for the
observation? Given an observed sequence and an HMM, what is the
most likely state sequence that generated it? Viterbi algorithm
3. Model Training Problem How to estimate the model parameters? Given an observation, can we learn an HMM for it?
Smartphone Based Human Behavioral Analysis53
Example with Binary
4/18/2014
54
HMM MATLAB
4/18/2014Smartphone Based Human Behavioral Analysis
Answers received contain new transition and emission matrices with probabilities
There are three files in which we work with to execute the code
Multiple Assumptions must be made in order to create the three files
55
Three Files of MATLAB
4/18/2014Smartphone Based Human Behavioral Analysis
1. Markovsource is a data file that contains multiple fields of phone numbers, dates, etc.
2. The transfer file contains matlab code to convert the data into the necessary values
3. The markovTEmatrix file contains the transition and emission matrices
These three files are used to receive the state probabilities for each matrix
56
Assumptions
4/18/2014Smartphone Based Human Behavioral Analysis
There are three kinds of fields within the data file such as phone numbers with 10 digits, dates, and other non-numerical alphabetic fields
Within the matrices, the first ten rows represent the date field’s probabilities
The next ten rows represent the phone number fields
There is an extra 1 or 2 rows that are used for the non-numerical alphabetic fields
57
Transition and Emission Matrices
4/18/2014Smartphone Based Human Behavioral Analysis
58
Multiple MATLAB Functions
4/18/2014Smartphone Based Human Behavioral Analysis
Hmmdecode - used to calculate the state probabilities of a sequence of emissions
Hmmviterbi – used to calculate the most probable state path for a hidden Markov model
Hmmtrain – used to calculate likelihood estimates for the transition and emission matrices from a certain sequence
Hmmestimate – can be used if the sequence is already known, but it has something missing
59
State Probabilities
4/18/2014Smartphone Based Human Behavioral Analysis
Administration
Andy McWilliams
61
Validation Plan
4/18/2014Smartphone Based Human Behavioral Analysis
Sensor Data Extraction & Analysis Retrieve real-time data from all chosen sensors Graph and analyze each set of data by performing
tests Raw Memory Extraction & Analysis
Test each raw memory extraction option Retrieve binary image of databases
Machine Learning Algorithms Use algorithms to recognize patterns in user’s
behavior Use HMM algorithm to retrieve deleted data
62
Budget
4/18/2014Smartphone Based Human Behavioral Analysis
Samsung Galaxy S3
Mini
$225
$75
Capstone Budget
Also received $400 from Dr. Hu’s budget to
purchase Samsung Galaxy S4 Mini
Smartphone Based Human Behavioral Analysis
63
Schedule
4/18/2014
Research and Purchase
Build Sensor Testing App
Test Sensors and Analyze Data
Add DTW Support for App
Train DTW With Gestures
Build App Version 2.0
Add Speech Recognition Support
Research Image Extraction Tools
Create and View a Generic Binary Image
Root Android and Extract Text Database
Create Binary Image of Database File
Succesfully Create HMM Matrices
Use HMM on Binary Data
8/24/2
013
10/13
/2013
12/2/
2013
1/21/2
014
3/12/2
014
CompletedIn - ProgressNot Started
64
Impacts Healthcare, Activity Monitoring, Forensics National Science Foundation Dr. Hu will use project findings to help
teach CPS at rural schools in a virtual environment
4/18/2014Smartphone Based Human Behavioral Analysis
Thank you! Questions?
Top Related