Smart Homes for the Elderly Dementia Sufferers: Identification and ...
Transcript of Smart Homes for the Elderly Dementia Sufferers: Identification and ...
Journal of Ambient Intelligence and Humanized Computing(September 2012, Volume 3, Issue 3, pp 205-218)
Smart Homes for the Elderly Dementia Sufferers:Identification and Prediction of Abnormal Behaviour
Ahmad Lotfi · Caroline Langensiepen ·Sawsan M. Mahmoud · M. J. Akhlaghinia
Postprint
Abstract In this paper we have described a solution for supporting independent liv-
ing of the elderly by means of equipping their home with a simple sensor network
to monitor their behaviour. Standard home automation sensors including movement
sensors and door entry point sensors are used. By monitoring the sensor data, im-
portant information regarding any anomalous behaviour will be identified. Different
ways of visualizing large sensor data sets and representing them in a format suitable
for clustering the abnormalities are also investigated. In the latter part of this paper,
recurrent neural networks are used to predict the future values of the activities for
each sensor. The predicted values are used to inform the caregiver in case anomalous
behaviour is predicted in the near future. Data collection, classification and predic-
tion are investigated in real home environments with elderly occupants suffering from
dementia.
Keywords Smart home · Dementia · Alzheimer · Assistive technology · Prediction ·Abnormality detection · Time series · Sensor network · Intelligent environment
Ahmad Lotfi · Caroline Langensiepen · Sawsan M. Mahmoud
School of Science and TechnologyNottingham Trent UniversityClifton Lane, Nottingham, NG11 8NSUnited KingdomTel:+44 115 8488390E-mail: [email protected]
Caroline LangensiepenE-mail: [email protected]·Sawsan M. MahmoudE-mail: [email protected]·MJ AkhlaghiniaCentre for Innovation and Technology ExploitationNottingham Trent UniversityE-mail: [email protected]
2
1 Introduction
The European Union considers dementia to be one of the most important causes of
disability in the elderly. Its figures show that between 1% and 2% of people aged 65-69
suffer dementia, but this proportion more than doubles for people in the age band
70-74, and studies across a number of countries show that its prevalence “increases
almost exponentially with age” [1]. The socio-economic costs of dementia are large and
increasing, and an international study by Anders Wimo of the Karolinska Institute
suggest, that 72.5 billion euros per annum accross the Europe is the cost of the informal
care provided by family and other carers to dementia sufferers [2]. A further study
by Wimo showed that carers have to spend many hours per day assisting dementia
sufferers [3], and any technology that would reduce this would help to ease the costs -
both financial and emotional.
Most patients would prefer to use a non-intrusive technology to help them with
their day-to-day activities. For example usage of surveillance cameras for patient mon-
itoring is not welcomed and in most cases it is ruled out completely [4]. Research and
development have focused mainly in the utilisation of different low-key technological
devices which are readily available [5] [6] [7]. The major players in patient monitoring
systems rely heavily on the use of monitored call centres rather than carers, with stan-
dard telephone lines for logging data and require significant installation. Some other
companies (e.g. JustChecking Ltd. [8]) have realised the importance to patient care of
an individualised, personal system whereby the carers, relatives and others who know
the dementia sufferers can monitor them; only intervening when the information and
their personal knowledge indicates that the situation has changed significantly.
The aim of our research is to find means of improving the lifestyle of older citizens
by integrating intelligence into their existing homes (making their homes smart). The
smart home could contain different sensors (movement sensors, door entry point, taps,
kettles, cookers sensors, etc.) to determine different classes of context which would help
to identify patterns of use and movement, and eventually allow the categorisation of the
user’s behaviour. When the behavioural pattern is learned, any anomalous behaviour
could then be detected. The most important factor in designing a smart environment
for the elderly is that the technology should not interfere with the normal activities of
the patient. Thus all devices should operate autonomously. We intend to use only low
cost and readily available sensors which could be installed by the user themselves or
their informal carers. We believe developing a technological solution easily post-fitted
in existing homes would definitely assist the patients in gaining independence without
altering their lifestyle or losing their personal dignity.
The structure of this paper is as follows: related work in this field are reported
in Section 2 followed by our methodology in anomaly detection in Section 3. Our
data collection system and the way data is represented and visualised are reported in
Sections 3.1, 3.2 and 3.3 respectively. In Section 4 the prediction technique is explained
followed by case studies in Section 5. Final conclusions are drawn in Section 6.
2 Related Work
A comprehensive survey published in [9] reports on state of the art technologies to
support people at the early stages of dementia during the night. Extensive research
3
has been reported on smart homes with a variety of applications including monitoring
systems for elderly independent living, accident and fall detection [10] [11].
Research in independent living is not limited to dementia sufferers. Many published
works address the issue of independent living in a broader sense [12] [13] [14] [15]. The
smart environment can also help to identify and model progression of dementia of
the Alzheimer’s type by evaluating performance in the execution of activities of daily
living [16]. The smart home can either monitor and collect the activities information
of the user by means of sensors, or communicate and control the environment. The
former approach is widely used for monitoring [17], anomalous behaviour detection
[18], behaviour diagnosis and prediction of activities in an ambient intelligence (AmI)
environment [19] [20] [21] [22]. The latter approach is used to intervene and interact
with the user as a means of preventing accidents and reminding the user.
In [23] the activities extracted from sensors are distinguished and the relationships
between them are established by using data mining techniques such as model based
clustering and association rules. Also, temporal relations for the most frequent events
are used in [24] to identify abnormal events. Moreover, many statistical methods are
used to monitor the daily activities of an inhabitant in an intelligent environment. For
example, Naive Bayesian classifiers are used in [25] to classify and detect activities
using a “tape-on and forget” sensor system.
Hidden Markov Model (HMM) is widely used to detect and predict activities in
an AmI environment. For example, the authors in [18] employed a HMM technique to
model behaviour based on the sensor data collected from a smart home environment.
In [26], the HMM technique is employed to model the occupant behaviour after using
unsupervised classification techniques to group his/her daily routine activities. As a
result, the model is able to detect the anomalies behaviour. Hierarchical Hidden Semi-
Markov Models [27] are also used to identify the daily activities of the occupants in an
assisted living community. However, HMMs do not take into account the relationships
between activities happened in sequential or parallel. Also, for each individual activity,
the sequence of sensor events cannot be separated using these models. Furthermore,
processing large data sets generated from low level sensors (i.e. temporal data from
different time scales) using HMM has shown some difficulties [28] [29].
One alternative candidate is to use Artificial Neural Networks (ANNs) to deal with
temporal patterns generated from sensors. In general, different kinds of ANNs are used
to track and predict the daily routine activities of the occupant in an AmI environ-
ments. For example, an approach named One-Pass Neural Network (OPNN) is used in
[29] in intelligent environment embedded agents to detect the user’s activities. In [30] a
competitive neural network called Growing Self Organizing Map (GSOM) is proposed
to cluster the daily behaviour of humans within a smart environment. Recurrent neu-
ral networks are also proven to be useful tools to solve the difficulties of the temporal
relationships of inputs between observations at different time steps, by maintaining
internal states that have memory. There are a few works that have attempted to use
temporal neural network algorithms to detect, recognize and classify human activities
in an intelligent environment. For example, the authors in [31] [32] developed a tempo-
ral neural-network based agent to identify human behaviour according to the temporal
order of their activities.
4
Fig. 1: An overview of the monitoring and interaction system architecture.
3 Anomaly Detection in Smart Homes
Our research aim is to design an unobtrusive “activities of daily living” (ADL) monitor-
ing system to allow identification and prediction of abnormal behaviour. We approach
this by gathering data on the routine activities of the elders e.g., getting out of bed,
going to the bathroom, preparing meals etc. without altering the elders’ normal be-
haviour. Wireless sensors are used, along with a computerised base station to collect
data that are then analysed and transferred to a secure central web site for viewing by
the carer/relative. Therefore, the aim is that adult children of frail elders living alone
and at a distance could be sent reports or alerts daily/weekly in the form of e-mail or
phone calls and even they should also be informed if any abnormality in the near future
is predicted. An overview of the monitoring and interaction system architecture is de-
picted in Fig. 1. By enriching an environment with sensors and devices interconnected
through a network, an AmI can be formed to take decisions to benefit the users of that
environment based on real-time information gathered and historical data accumulated
[33]. This should help the elders retain independence and to remain in their own homes
longer than they might otherwise.
The challenge we face is to understand human behaviour from low level sensory
data. This could be achieved using common-sense knowledge or using computational
5
Fig. 2: Phases in the data handling work flow.
intelligence integrated with sensory data. An individual user model can be learned from
the sensory data which eventually represents the behaviour model of the user.
Fig. 2 represents the phases in the data handling work flow. The initial data capture
results in a large number of data items which, though time ordered, are not evenly
distributed in time, and are initially labelled only by sensor ID, time and boolean state.
The data must then be re-factored so that it can be more easily accessed, enumerated
and represented. The data representation and clustering/identification phases feed from
each other, as it is only through using different data representations that the separate
activities of clustering and abnormal behaviour identification can be easily carried out
and assessed.
By monitoring the sensor data, important information regarding any irregular be-
haviour will be identified. Anomalies are those odd patterns of data which do not match
the normal behaviour. Anomalies can be recognised using different anomaly detection
techniques. In many real life applications, these kinds of pattern are also called outliers,
discordant observations, exceptions, surprises or peculiarities. Amongst all mentioned
terminology, anomalies and outliers are the most frequently used terms within the con-
text of human behaviour detection. For example, Fig. 3 shows anomalies within two
dimensional data sets representing the sleeping pattern (from bed pressure sensor) of
an occupant. Most values of the data are in two regions N1 and N2 representing night
time sleeping and afternoon nap sleeping respectively. These regions are considered as
normal. At the same time the points in region A1 and points A2 and A3 are considered
to be anomalies because these points are at different time of the day and different from
the normal pattern in the regions N1 and N2. In our study anomalies are detected us-
ing clustering techniques [34]. In Section 3.3 clustering techniques are used to identify
any anomalies within collected sensory data.
3.1 Data Collection
As stated earlier, we rely on a data collection system which provides both sensation
and transmission. The data acquired includes the occupancy of different areas, environ-
6
Fig. 3: Anomalies in a simple 2-dimensional data set.
mental attributes, and interactions between occupants and devices. Sensory devices are
responsible for data collection and a variety of sensors are readily available to perform
this task. Typical sensors are as follows:
– Passive Infra-red Sensors (PIR) or motion detectors are sensitive to the movements
of living objects. They are normally used to monitor the occupancy of different
areas.
– Door/Window entry point sensors are on/off switches which can detect the open
and closed status of a door/window.
– Electricity power usage sensors which can monitor the activity of electrical devices
by measuring their electrical current consumption.
– Bed/sofa pressure sensors to measure the presence in and usage of these areas.
– Flood sensors provide early warning of overflows and leaks.
The choice of sensors for different environments vary. Without the loss of generality,
the discussion presented in this paper concentrate on the usage of PIR and door entry
point. PIR motion sensors responds to changes in heat in the form of infra-red radiation.
They are used to identify the movement and then the movement pattern is interpreted
as the occupancy. It is important to place PIR sensors in locations where the most
effective form of movements are captured. On the other hand, door entry point sensors
are relatively reliable as they clearly represent the movement activities.
The data collected from the sensor network are communicated with a base station
and eventually stored in a central database. The communication between the sensor
network and the base station could be in either wired or wireless format. Wireless
technology for sensor communication is a preferred option, as it is easier to fit wire-
less sensors in existing homes. However, we do not rule out the use of X10 technology
or other well established wired sensor networks in which sensory devices can commu-
nicate with the base station via electrical power lines. Wireless sensors networks, in
7
comparison with wired sensors networks, are more flexible in terms of deployment and
the required infrastructure of the network in a smart home environment. However, in
wireless sensor networks power consumption is the most important concern mainly be-
cause all sensory devices are powered by batteries. A system where the occupant was
required to change batteries frequently is not ideal.
Using either of these two technologies should not make any differences in the results
of our research in identification of anomalies in behaviour. For the sake of simplicity
and ease of installation, we have used a wireless sensor network comprising movement
sensors and door contact sensors.
Fig. 4: A sample of activity of daily living data collected by sensory devices.
Fig. 5: A sample of activity of daily living for a single user.
3.2 Sensor Data Representation
Different techniques have been used for activities representation and interpretation. For
instance, in [36] an activity path for a single continuous vertical trajectory is applied
to identify the activities for a period of time. Whereas the authors in [37] propose a
technique based on binary tree structure called a Routine Tree. In [38] activities are
divided into adjacent subsequences of length n called n-grams. These activities are
regarded as a histogram of their event n-gram. As the value of n is increased, the order
of information of events are more accurately captured. However, increasing the value
of n affects the dimensionality of the histogram.
8
Sensory data collected from a smart home environment needs to be represented in
an appropriate format before any further analysis is carried out. The raw sensory data
is often difficult to understand. This becomes even more complicated when sensory
data from multiple sensors are gathered. A snapshot of the binary data collected in the
base station from two occupancy sensors (PIR sensors) is illustrated in Fig. 4. Due to
the fact that only one occupant is present in the monitored environment, no parallel
activity in different areas is detected.
Fig. 6: Sample occupancy chart for 10 days of data for four different sensors.
Fig. 7: Layout of the house and the location of sensors for sample data set.
9
Fig. 8: Combined activity of daily living signal as a time-series.
The data represented in Fig. 4 can be integrated together to form the ADL chart.
A snapshot of the activity of daily living for a single user is depicted in Fig. 5. Similar
patterns can be obtained for a home with multiple occupants. This is achieved by using
RFID or other tagging devices to tag different users in the environment. A detailed
analysis is reported in [35].
To identify any abnormal behaviour of a user, we need to collect sufficient data of
daily activities to be able to establish the correlation between different events and ac-
tivities. Furthermore, a trend analysis of the information could be obtained if sufficient
data are available. It should be noted that sensor data are collected approximately ev-
ery second and when this frequency of data collection is repeated for multiple sensors,
we would be facing the difficult challenge of interpretation of large amounts of sensor
data.
To illustrate the complexity of the sensor data, Fig. 6 shows the occupancy signals
from 4 PIR sensors over a sample 10 day period. The layout of the environment and
the location of sensors are illustrated in Fig. 7.
In this paper, the process of modelling a large binary data set collected from a
sensor network is represented using two different techniques which are proven to be
useful to summarise the data. These techniques are tested with multiple binary sensors
(occupancy, door, windows, and etc. sensors). The data extracted from those sensors
are usually sparse and contain many repeated constant values.
3.2.1 Combined activity of daily living signal
Assuming different levels for each activity, a combined signal can be shaped. Each level
of the combined signal represents one of the sensors. A sample of combined activity
signal is illustrated in Fig. 8. This signal represents a non-stationary time series and
available techniques in time series prediction may be used to predict the future values
of the signal, which could be interpreted as prediction of the activities of the patient.
This method is explained in more detail in [19].
3.2.2 Start time and Duration
In this method, the signal is represented by the start time and the duration of an event.
For example, we use the start time that the person enters a room and duration
that he/she stays in a specific location. Thus, each observation has three parameters
[39]. These are:
1. Sensor ID (position of the sensor e.g. in the bedroom);
2. Start time (the time of day the observations is occurred); and
10
(a) bedroom
(b) corridor
(c) lounge
(d) kitchen
Fig. 9: Plots of 365 days of the sample data for the four sensors.
11
Table 1: Start time and duration of a sample of bedroom sensor.
Start time (hrs.) 13:01 23:23 14:06 23:15 13:45Duration (hrs.) 2:20 10:20 2:30 8:15 2:05
Fig. 10: Start time, duration and time of bedroom PIR activities for one year sample
occupancy chart.
3. Duration (the amount of time the observation is occurred) of specific room is oc-
cupied.
Table 1 demonstrates a sample of observations representing the time that the pa-
tient spent in the bedroom at each day for a duration of three days. From this sample
data, a sleeping pattern is easily identifiable.
The graphical representation of these attributes shows significant information about
the daily movement behaviour of the occupant. For example, consider Fig. 6 and Fig. 9
which show the plots of a sample data set of four sensors for a single occupant. As can
be seen, the behaviour of the occupant is more easily interpreted in Fig. 9 than Fig.
6. For instance, in Fig. 9-a, the bedroom sensor plot shows that the occupant always
goes to bed at midnight for around 7-10 hours, and he/she usually spends about two
hours period of time in nap sleeping. It is almost impossible to achieve this level of
understanding from the raw sensory data represented in Fig. 6.
Consider the bedroom movement sensor shown in Fig. 9-a which represents the
projection of the sensory data collected over a long period of time into a 2D graph,
collapsing out the axis referring to the individual days. In Fig. 10 the same activity is
illustrated in a 3D graph where start time and duration of each activity over one year
period, with the individual days shown.
Using the above two forms of data representations have provided us with useful
information and this provides the basis of our work for classification of activities. We
intent to use the start-time and duration form of representation in the rest of this
paper.
12
(a) bedroom (b) corridor
(c) lounge (d) kitchen
Fig. 11: Clusters of activities for 14 days of the sample data for the four sensors.
3.3 Sensor Data Visualisation and Clustering
Data visualisation has the potential to help in understanding and recognising large
volumes of data and also detect patterns and anomalies that are not obvious using
non-graphical forms of representation. Good data visualisation eases the examination
of large volumes of data, and allows deduction to be made from the relationships within
the data [40]. If the daily routine activities are clustered together, then odd activities
can then be identified as anomalous behaviour. Clustering is an important process for
condensing and summarising information because it can provide an overview of the
stored data [41]. To identify the abnormalities within the sensory data using clustering
based animality detection, the following three categories are identified [34]:
1. The data that reflect the regular or normal data are grouped in clusters, while
the data that do not fit in any clusters are treated as anomalies. In this case, any
clustering technique can be used and any data that do not find in any cluster are
considered to be anomalies.
2. The data that are near their closed cluster centroid are considered as normal data,
while the data that are located far away from their cluster centroid are treated as
anomalies. In this case, the data are first clustered and then the anomaly score,
which is the distance to its closed cluster, is calculated.
3. The data found in large clusters are considered as normal data, while the small or
sparse clusters contain the anomalies. In this case, depending on a threshold value,
anomalies can be detected. If the size of any cluster is below this value then the
stored data is considered to be anomalous.
In this paper, anomaly detection techniques have been applied using different
clustering algorithms [41][45]. Examples of such algorithms are: self organising maps
(SOM), K-means clustering, and fuzzy C-means (FCM) to cluster training data and
13
then use the clusters to classify test data set. For example, in Fig. 11 the clusters for
the same sample data illustrated earlier in Fig. 9 are shown. In Fig. 11-a and Fig. 11-c
there are some instances of data that do not belong to any cluster (see the first cate-
gory above), so these data are considered as abnormal data. The clusters that contain
a large data set represent the most frequent pattern in these clusters, while the clusters
that contain small datasets are treated as anomalies. It can be concluded that large
volume of data can be easily represented and visualised by using clustering techniques.
The number of clusters could vary from one algorithm to another. For example,
unlike self organizing map algorithms, the number of clusters in fuzzy C-means need to
be known in advance. In our experiments the maximum number of clusters was set for
supervised algorithms depending on the duration times that the occupant spent in a
particular area. In fuzzy C-means clustering, objects on the boundaries between several
clusters do not belong to a specific cluster. They belong to more than one cluster with
a certain degree of belonging. Every time a new activity is recorded, the Euclidean
distance between the new data and cluster centre is calculated. The distance matrix
will represent whether if the data should be considered as normal or abnormal.
4 Sensor Data Prediction
Using the above mentioned data interpretation, we would be able to separate the
anomalous behaviours when an activity has happened. However, this would not help the
carer to make necessary arrangements in advance. Data interpretation does, however,
help us to better understand the activities of daily living. This would also be useful
if we want to generate a report to summarise the activities of the patient over a long
period of time. To improve the proposed system, a predictive method will be utilised
to predict the future values of the activities (start time and duration) based on the
historical data available from activities recorded by each sensor.
Activities of daily living presented either in combined (e.g. see Fig. 8) or separate
signals (e.g. see Fig. 6) represent a time series. To predict the future values of time
series, the hidden Markov model (HMM) has been used widely to find the relationships
between the temporal or sequential data extracted from sensors and identify the routine
activities of an occupant. HMM is also used in time series prediction to predict future
values of the series. The disadvantages of using HMM in time series prediction is
that increasing the length of time series needs large volume of time series runs from
HMM [42][43][44]. Many other statistical techniques in time series prediction have been
reported [41].
In this study, we have investigated different techniques for prediction of stationary
time series. The investigated techniques included; Echo State Network (ESN), Back
Propagation Through Time (BPTT) and Real Time Recurrent Learning (RTRL). Re-
current neural networks (RNNs) are widely used to deal with many dynamical and
non-linear problems, such as time series forecasting. They have feedback connections
which address the temporal relationship of inputs by maintaining internal states that
have memory. RNNs are proven to be effective in learning time-dependent signals that
have short-term structure. RNNs are computationally more powerful than feed forward
neural networks and better approximation results have been obtained for prediction
problems. Based on our empirical investigation, it was found that ESN is a suitable
technique for prediction of binary time series and a short summary of the technique is
presented in the following section.
14
Fig. 12: Structure of an Echo State Network approach. Only the output weights Wout
are adapted, all other weights (input, reservoir and feedback) are chosen randomly.
4.1 Echo State Network
In this section, the recurrent neural network, Echo State Network (ESN) is described.
It was developed recently by Jaeger [46]. The basic architecture of ESN is illustrated
in Fig. 12 which consists of three layers. These include input, hidden and output layer.
The input layer is connected to the hidden layer. Both the input and hidden layer are
fully connected to the output layer. On the other hand, the output layer is backward
connected to the hidden layer only. It is a discrete-time, continuous state where the
activation function for all neurons is the sigmoid function [47].
An ESN consists of a reservoir of conventional processing elements, which are re-
currently interconnected with untrained random weights, and a readout (output) layer,
which is trained using linear regression methods. The key advantage of the ESN is its
ability to model systems without the need to train the recurrent weights [48]. For train-
ing an ESN with an input u(n), a reservoir state x(n) with M processing elements,
and an output y(n), the equations are calculated as follows:
x(n+ 1) = tansig(wx.x(n) + win.u(n) + v(n+ 1)) (1)
and
y(n) = w.x(n) (2)
where x(n) denotes the hidden layer or the internal state. The input and output to the
ESNs are denoted by u(n) and y(n) respectively. tansig denotes hyperbolic tangent
sigmoid function which is applied element wise, v(n + 1) is an optional noise vector.
wx, win and w are respectively the internal connection weights of the reservoir, the
input weights to the reservoir and the readout (output) weights from the reservoir[49].
The ESN approach differs from other methods in that a large RNN is used (on
the order of 50 to 1000 neurons) and in that only the synaptic connections from the
RNN to the output neurons are updated i.e. weights coming from the hidden layer
15
Fig. 13: ESN predicted values for bedroom sensor.
(the reservoir) to the output layer are updated in order to achieve the learning task.
As a result, large datasets are learnt in only few minutes or even seconds [46]. Also,
there are neurons in the reservoir connected in loops [see Fig. 12], therefore the past
states ’echo’ in the reservoir. The convergence of training in ESN is much faster than
other RNN [49]. This has made ESN an attractive model for a wide range of signal
processing and control applications (e.g. time series prediction, pattern generation,
event detection and classification and non-linear control). For instance, in prediction
of chaotic time series, ESN has proven to be a very accurate and valuable tool. The
prediction is accomplished using a black box model, i.e. it only depends on past data
since no further information is used. In addition, no explicit model is given in order to
create a new situation [50] [51].
In this paper, ESN is used as a model to predict and extract behavioural patterns
while keeping learning complexity at a low level. In addition, ESN is a very good choice
for the modelling because in the methodology of sensor networks, new data are arriving
at any time, whilst other approaches need all data input at the same time steps in order
to compute the output.
4.2 Predictive Techniques Comparison
Initially ESN was employed to predict the future values based on the sample data
shown earlier in Fig. 9. In our experiments, instead of using a separate ESN for each
sensor, all available sensors are connected at the same time as inputs to the network
and compute the prediction of the sensors. The advantages of using just one ESN for
all sensors are to reduce the amount of memory and computation time. In this case, the
number of input and output units depends on the number of sensors, which is driven
by the actual sensor value at time t. The output unit is the value of the same sensor at
time t+τ with different hidden units (reservoir) sizes. A number of parameters are used
in the ESN learning algorithm including number of neurons in the hidden layer, the
Root Mean Square Error (RMSE) for training and testing datasets, number of epochs
and time required for training. For training the ESN network, 50 hidden neurons are
16
Fig. 14: ESN predicted values for corridor sensor.
used. The results for τ = 6 step ahead (6 hours) prediction for both bedroom and
corridor sensors are shown in Fig. 13 and Fig. 14 respectively. Results shown are only
for a sample of 14 days dataset split into training and testing data. Samples of 10 days
data are used as the training and the samples of 4 days are used for testing.
The bedroom sensor shows a very good match between the predicted and actual
sensor values. This is slightly less accurate for corridor sensor, as the corridor signal is
relatively more chaotic. Our observation is that more chaotic signals are expected to
be more difficult to predict. Different sizes of reservoir (number of hidden neurons) are
also used to test the performance of ESN. In Fig. 15 the training time using different
reservoir sizes (hidden neurons) are shown. As can be seen, ESN is relatively fast
and datasets are trained in only a few minutes or even seconds. In addition, ESN is
compared with other RNNs used in time series prediction. These techniques are back
propagation through time (BPTT) and real time recurrent learning (RTRL). Table 2
compares the results of all sensor datasets using ESN with the other recurrent neural
networks techniques. ESN prediction results are better as compared with BPTT and
RTRL in that the training time is significantly shorter. The other approaches suffer
from slow convergence as the number of neurons are increased.
Table 2: Prediction results of all sensor datasets using ESN, RTRL and BPTT. (⊕:
No. of hidden neurons; ⊗: Training RMSE; : Testing RMSE; and �: Time(Sec.).)
Method ⊕ ⊗ �ESN 10 0.0556 0.0556 0.0116ESN 50 0.0556 0.0556 0.0519RTRL 10 0.0987 0.0964 281.8231RTRL 50 0.0731 0.0740 1541.0427BPTT 10 0.0759 0.0766 12.7315BPTT 50 0.0803 0.0811 218.0986
17
Fig. 15: ESN training time for different reservoir sizes.
5 Case Studies
In this section two separate case studies are reported. In both case studies, the data
collection system consists of an array of wireless motion sensors (kitchen, bedroom,
bathroom, lounge) and door entry point sensors (front door and back door) were used.
For both case studies, JustChecking [8] equipment is used. More details about the
technical characteristics of the hardware equipment is available from [8]. However, we
should emphasise that the results and discussion presented below is not restricted to
the usage of this equipment only. In both cases only one resident who suffered from
dementia living in their own home is monitored. The caregiver has access to the daily
activities of the patient and they are monitored from a secure web interface. The output
of all sensors are discrete values. These discrete values represent presence or absence
from a specific area. When the patient moves from one room to another, the status of
the sensor will vary.
5.1 Case Study 1 - Anomaly Detection
The aim of this case study is to establish whether the system can detect abnormality
within the behaviour of an occupant living in a real environment. In this case study,
the data is representing the occupancy in different areas of a house and for an elder
person where her health deteriorated during the course of our research. To demonstrate
the difference in the activities based on the sensor network measurement only, the
data set are represented into separate groups. Fig. 16-a and Fig. 17-a show the start
time and duration time of occupancy activities for the lounge and bathroom motion
sensors respectively. In both figures, only a sample of three days are depicted. She was
prescribed with some medications earlier and it worsened her health status. She was
wandering around during the early hours of morning and her behaviour was considered
as abnormal. As can be seen from Fig. 16-a, the occupant has often been in the lounge
area many times after midnight. Frequently the occupant stays 20 minutes or less in
the lounge room but on some occasions she spent more than one hour in this area. She
was prescribed with new medications and just after 20 days from when the first data
set is shown, occupant’s health improved. The normal occupancy behaviour for both
lounge and bathroom are shown in Fig. 16-b and 17-b respectively.
18
(a) Abnormal
(b) Normal
Fig. 16: A sample of three days activities for the lounge area. (a) Abnormal (b) Normal.
It is straightforward to see the difference in the pattern for the lounge area (Fig.
16-a and Fig. 16-b). However, this might not be so obvious for other areas of the house.
For example, the duration of bathroom occupancy for the abnormal behaviour period
is just slightly more than the expected normal bathroom occupancy. Therefore, it is
important to make a collective decision based on all available sensory data. All data
points are clustered and clusters are shown with different markers. A threshold value is
used to detect the anomalies. The unexpected clusters for both lounge and bathroom
are shown in dotted lines in Figs. 16-a and 17-a respectively.
As discussed earlier in Section 3, using an appropriate form of signal representation
and ultimately clustering the activities, it is possible to identify the abnormality from
low level sensory signals even if detailed knowledge about the subject is not available.
5.2 Case Study 2 - Anomaly Prediction
In this case study, the data was collected over a one and half year period. The layout
of the house and where sensors are mounted is shown Fig. 18. The house consists
19
(a) Abnormal
(b) Normal
Fig. 17: A sample of three days activities for the bathroom. (a) Abnormal (b) Normal.
of a lounge, corridor, kitchen, and bedroom (upstairs). Therefore, only three motion
sensors and two door contact sensors (front door and back door) are used to monitor
the occupancy and activities of the resident.
The first stage of our research is to identify the normal behaviour and distinguish
any abnormality. Using the data representation method discussed in Section 3.2, the
sensor data are clustered. Fig. 19-a and Fig. 19-b show start time and duration graph
for front door and back door sensors respectively. From the clusters highlighted in these
two graphs, it is evident that for most days the front and back doors were opened for a
short period of time. However, in some instances both doors were left open for a long
period of time.
Using this form of data representation and visualisation, we have managed to look
at each sensor activity independently and identify the abnormal behaviour for that
specific activity. However, all these activities are interdependent and we should be able
to establish the dependency between the activities. To achieve this, an ESN is used to
learn the temporal relationship of all 5 sensors.
Figs. 20-a and 20-b show the ESN training results for back door and kitchen re-
spectively. The prediction is based on a one month data set where 20 days are used for
20
Fig. 18: Layout of the house and location of sensors for our case study.
training and 10 days for testing. 50 hidden neurons and 6 hours ahead prediction are
used. The results shown here is a sample of prediction for one day only.
The root mean square error (RMSE) for training of these sensors shown in Fig.
20 is about 1% and 7% for back door and kitchen sensors respectively. For instance,
consider the back door sensor shown in Fig. 20-a. The back door is opened only once
during this day and it is closed after 3 seconds. As can be seen from this figure, the
predicted duration and start time is very close to the actual data. There is a very small
error between the predicted data and the actual sensor data.
6 Conclusions and Future Work
The results presented in this paper show that the start-time/duration is the most
effective way of representing a large sensor data set. This will also help with the clas-
sification of the activities to identify the abnormal behaviour. Furthermore, we have
investigated different recurrent neural network technique to predict the future activi-
ties. The results presented in this paper show that Echo State Network (ESN) is a very
promising approach for binary datasets collected from smart environments. Datasets
investigated here are based on a single inhabitant environment equipped with appro-
priate motion and door contact sensors. These sensors are used to record the activities
representing the behaviour of the occupant, and allow the caregiver to observe any
changes to patterns. It can be concluded that using large number of hidden neurons
in ESN yielded a good results in terms of the error and time required for training and
testing.
Based on the results shown here, it appears that a home equipped with some low-
level sensors can provide important information about the status of the occupant. The
proposed approach works better for elderly residents when more routine activities are
expected. We cannot suggest whether this approach would work for a young or more
active occupant.
21
(a) front door sensor
(b) back door sensor
Fig. 19: Start time and duration activities.
Our future work aims at multiple occupancy and prediction of abnormal behaviour.
The approach presented in this paper would not be effective in the presence of visitors
or even when the elderly people have a pet which is true for some cases. We are also
aiming to continue our research in semantic modelling of the behaviour where the
predicted values are communicated with the elderly and carer in linguistic terms. For
the work presented in this paper, only a limited number of discrete sensors were used.
However, more research is required when a combination of discrete sensors (occupancy,
door entry point, ...) and continuous sensors (temperature, humidity, ..) are used.
22
(a) back door sensor
(b) kitchen sensor
Fig. 20: Predicted values for start time and duration activities: A sample of one day
activities.
Acknowledgements
This research was partially supported by Nottingham Trent University’s Stimulating
Innovation for Success (SIS) programme. The authors would like to thank Just Check-
ing Ltd. (www.justchecking.co.uk) for their support of this work.
References
1. European Comission,(2010), Europa Public Health [online]. Available: “http://ec.europa.eu/ health/ph information/ dissemination/ diseases/ alzheimer en.htm” [ac-cessed 16 August 2010]
2. A. Gustavsson, L. Jonsson, R. McShane, M. Boada, A. Wimo, AS. Zbrozek, “Willingness-to-pay for reductions in care need: estimating the value of informal care in Alzheimer’sdisease” Int J Geriatr Psychiatry. 2009 Sep 14.
23
3. A. Wimo, B. Winblad, L. Jonsson, “An estimate of the total worldwide societal costs ofdementia in 2005”, Alzheimers Dement. Apr; 3(2):81-91 2007.
4. A. Kenner1, “Securing the Elderly Body: Dementia, Surveillance, and the Politics of Agingin Place,” Surveillance and Society, Vol. 5, no. 3, pp. 252-269, 2008.
5. Marilyn Cash, At Home with AT (Assistive Technology), Research report available from:“http://www.dementia-voice.org.uk/Projects/At Home with AT main.pdf”, 2004.
6. C. Nugent, M. Mulvenna, F. Moelaert, B. Bergvall-Kareborn, F. Meiland, D. Craig, R.Davies, A. Reinersmann, M. Hettinga, A. -. Andersson, R. -. Droes and J. E. Bengtsson,“Home based assistive technologies for people with mild dementia,” in 5th InternationalConference on Smart Homes and Health Telematics,pp. 63-9, 2007.
7. R. Orpwood, C. Gibbs, T. Adlam, R. Faulkner and D. Meeeahawatte, “The design ofsmart homes for people with dementia - user-interface aspects,” Universal Access in theInformation Society, vol. 4, pp. 156-64, 12, 2005.
8. JustChecking, (2010), Supporting Independence People with Dementia online]. Available:http://www.justchecking.co.uk [accessed 16 August 2010]
9. P.J. McCullagh, W. Carswell, J.C. Augusto, et al., “State of the Art on Night-Time Careof People with Dementia,” IET Assisted Living Conference, 24-25 March, 2009, LondonUK.
10. P. Boissy, S. Choquette, M. Hamel and N. Noury, “User-based motion sensing and fuzzylogic for automated fall detection in older adults,” Telemedicine Journal and e-Health, vol.13, pp. 683-93, 12/01. 2007.
11. X. Yu, “Approaches and principles of fall detection for elderly and patient”, in 2008 10thInternational Conference on e-Health Networking, Applications and Services (Healthcom),2008, pp. 42-7.
12. MJ. Akhlaghinia, A. Lotfi, C. Langensiepen and N. Sherkat, “A fuzzy predictor modelfor the occupancy prediction of an intelligent inhabited environment,” in 2008 IEEE 16thInternational Conference on Fuzzy Systems (FUZZ-IEEE), 2008, pp. 939-46.
13. V. Callaghan, G. Clarke, M. Colley, H. Hagras, J. S. Y. Chin and F. Doctor, “Inhabitedintelligent environments,” BT Technology Journal, vol. 22, pp. 233-247, 2004.
14. H. Hagras, “Embedding computational intelligence in pervasive spaces,” IEEE PervasiveComputing, vol. 6, pp. 85-9, 07. 2007.
15. A. Helal, J. King, R. Bose, et al., “Assistive Environments for Successful Aging,” inAdvanced Intelligent Environments, Achilles D. Kameas, Victor Callagan, Hani Hagras,Michael Weber and Wolfgang Minker (Eds.), Springer US, 2009.
16. A. Serna, H. Pigot and V. Rialle, “Modeling the progression of Alzheimer’s disease forcognitive assistance in smart homes,” User Modeling and User-Adapted Interaction, vol.17, pp. 415-38, 2007.
17. H. Medjahed, D. Istrate, J. Boudy and B. Dorizzi, “A fuzzy logic system for home elderlypeople monitoring (EMUTEM),” in 10th WSEAS International Conference on Fuzzy Sys-tems (FS’09), 2009, pp. 69-75.
18. D. N. Monekosso and P. Remagnino, “Anomalous Behavior Detection: Supporting In-dependent Living” in Intelligent Environments, Methods, Algorithms and Applications,Monekosso, Dorothy; Remagnino, Paolo; Kuno, Yoshinori (Eds.), Springer London, 2009.
19. MJ. Akhlaghinia, A. Lotfi, C. Langensiepen and N. Sherkat, “Occupant Behaviour Predic-tion in Ambient Intelligence Computing Environment,” International Journal of UncertainSystems, 2008.
20. D. J. Cook, “Making sense of sensor data,” IEEE Pervasive Computing, vol. 6, pp. 105-8,04. 2007.
21. Sawsan M. Mahmoud, Ahmad Lotfi, N. Sherkat, C. Langensiepen and T. Osman, “Echostate network for occupancy prediction and pattern mining in intelligent environment,”in Proceedings of the 5th International Conference on Intelligent Environments, 2009, pp.474-481.
22. Dante I. Tapia, Ajith Abraham, Juan M. Corchado and Ricardo S. Alonso, “Agents andambient intelligence: case studies,” Journal of Ambient Intelligence and Humanized Com-puting, Vol. 1, No. 2, pp. 85-93, 2010.
23. Tracy Barger, Majd Alwan, Steve Kell and Beverely Turner, Objective Remote Assessmentof Activities Of Daily Living: Analysis Of Meal Preparation Patterns, Medical AutomationResearch Center, University of Virginia Health System, 2002.
24. Vikramaditya Jakkula, Diane J. Cook and Aaron S. Crandall: Temporal Pattern Discoveryfor Anomaly Detection in a Smart Home, 3rd IET International Conference on IntelligentEnvironments, pp. 339 -345, 2007.
24
25. Emmanuel Munguia Tapia, Stephen S. Intille, Kent Larson, Activity Recognition in theHome using Simple and Ubiquitous Sensors, F. (eds.) PERVASIVE, LNCS, Vol. 3001, pp.158-175. Springer, Heidelberg , 2004.
26. Dorothy N. Monekosso and Paolo Remagnino, Anomalous Behavior Detection: Support-ing Independent Living, Intelligent Environments, Advanced Information and KnowledgeProcessing, Springer, 2009.
27. H. Kautz, O. Etziono, D. Fox and D. Weld: Foundations of assisted cognition systems.Technical report, University of Washington, Department of Computer Science and Engi-neering, 2003.
28. Geetika Singla, Diane J. Cook and Maureen Schmitter-Edgecombe: Recognizing inde-pendent and joint activities among multiple residents in smart environments, Journal ofAmbient Intelligent Human Computing, Vol. 1, pp. 57–63, 2010.
29. H. Li, Q. Zhang and P. Duan: A Novel One-Pass Neural Network Approach for Activi-ties Recognition in Intelligent Environments. Proceedings of the 7th World Congress onIntelligent Control and Automation, pp. 50-54, 2008.
30. H. Zheng, H. Wang and N. Black: Human Activity Detection in Smart Home Environ-ment with Self-Adaptive Neural Networks. IEEE International Conference on Networking,Sensing and Control, pp. 1505-1510, 2008.
31. Rivera Illingworth, V. Callagha and H. Hagras: Towards the Detection of Temporal Be-havioural Patterns in Intelligent Environments. 2nd IET International Conference on In-telligent Environments, pp. 119-125, 2006.
32. Rivera Illingworth, V. Callaghan and H. Hagras: A Neural Network Agent Based Approachto Activity Detection in AmI Environments. IEE International Workshop, Intelligent En-vironments (IE05), Colchester, UK, pp. 1-12, 2005.
33. Juan Carlos Augusto, Ambient Intelligence: Basic Concepts and Applications, Softwareand Data Technologies, Communications in Computer and Information Science, Vol. 10,pp. 16-26, 2008.
34. Varun Chandola, Arindam Banerjee and Vipin Kumar, “Anomaly detection: A survey”,ACM Computing Surveys (CSUR), vol. 41, No. 3, 2009.
35. MJ. Akhlaghinia, A. Lotfi, C. Langensiepen, “Localising Agents in Multiple-OccupantIntelligent Environments”, in Proc. WCCI 2010 IEEE World Congress on ComputationalIntelligence, Barcelona, Spain, July, 18-23, pp. 2202-2208, 2010.
36. K. Vrotsou, K. Ellegard and M. Cooper, “Everyday Life Discoveries: Mining and Visu-alizing Activity Patterns in Social Science Diary Data,” Information Visualization, 11thInternational Conference, pp. 130-138, 2007
37. Raza Ali, Mohamed ElHelw, Louis Atallah, Benny Lo and Guang-Zhong Yang, “PatternMining for Routine Behaviour Discovery in Pervasive Healthcare Environments,” 2008.
38. Raffay Hamid, Siddhartha Maddi, Amos Johnson, Aaron Bobick, Irfan Essa and CharlesIsbell: “A Novel Sequence Representation for Unsupervised Analysis of Human Activities,”Elsevier Science, Artificial Intelligence, Vol. 173, Issue 14 pp. 1221-1244, 2009
39. Fabien Cardinaux, Simon Brownsell1 Mark Hawley, and David Bradley, “Modelling ofbehavioural patterns for abnormality detection in the context of lifestyle reassurance”,Proceeding CIARP, Havana, Cuba, 2008.
40. Michael D. Lee, Rachel E. Reilly and Marcus A. Butavicius, “An Empirical Evaluationof Chernoff Faces, Star Glyphs, and Spatial Visualizations for Binary Data”, In CRPITS24:Proceedings of the Australian symposium on Information visualization, pp. 110, 2003
41. T. Warren Liao, “Clustering of time series data a survey”, Pattern Recognition, vol. 38,pp. 1857-1874, 2005.
42. Sang-Ho Park, Ju-Hong Lee, Jae-Won Song and Tae-Su Park, “Forecasting Change Direc-tions for Financial Time Series Using Hidden Markov Model,” vol. 5589, Lecture Notes inComputer Science, Springer, pp. 184-191, 2009
43. Osman N. Yogurtcu, Engin Erzin and Attila Gursoy, “Extracting Gene Regulation In-formation from Microarray Time-Series Data Using Hidden Markov Models,” vol. 4263,Lecture Notes in Computer Science, Springer, pp. 144-153, 2006
44. H. Li, Q. Zhang and P. Duan, “A Novel One-Pass Neural Network Approach for Activi-ties Recognition in Intelligent Environment”. Proceedings of the 7th World Congress onIntelligent Control and Automation, pp. 50-54, 2008
45. Eamonn Keogh Jessica Lin, Michail Vlachos and Dimitrios Gunopulos, “Iterative incre-mental clustering of time series”, In EDBT, pp. 106-122, 2004.
46. Herbert Jaeger and Harald Haas, “Harnessing Nonlinearity: Predicting Chaotic Systemsand Saving Energy in Wireless Communication”, Appril, VOL 304, SCIENCE, (2004).
25
47. Alexandre Devert, Nicolas Bredeche and Marc Schoenauer, “Unsupervised Learning ofEcho State Networks: A Case Study in Artificial Embryogeny”, Artificial Evolution, Vol.4926, (2008), pp. 278–290.
48. Mark D. Skowronski and John G. Harris, “Minimum mean squared error time series clas-sification using an echo state network prediction model”, IEEE International Symposiumon Circuits and Systems, (2006).
49. Zhiwei Shi and Min Han, “Support Vector Echo-State Machine for Chaotic Time-SeriesPrediction”,IEEE Transactions on Neural Networks, March, Vol. 18, Issue: 2, (2007), pp.359–372.
50. Sven Hellbach, Soren Straussl, Julian P. Eggert, Edgar Korner and Horst-Michael Gross1,“Echo State Networks for Online Prediction of Movement Data - Comparing Investiga-tions”, (2008).
51. Obst, O., Wang, X.R. and Prokopenko, M., “Using Echo State Networks for AnomalyDetection in Underground Coal Mines”, Proceedings of the 7th international conferenceon Information processing in sensor networks, (2008).
52. K. Sawai and M. Yoshida, “Algorithm to detect abnormal states of elderly persons forhome monitoring,” Syst. Comput. Jpn., vol. 38, pp. 34-42, 06/15. 2007.