ScienceDirect 1. Introduction

ScienceDirect

Available online at www.sciencedirect.com

Procedia Computer Science 147 (2019) 338–348

1877-0509 © 2019 The Authors. Published by Elsevier B.V.This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)Peer-review under responsibility of the scientific committee of the 2018 International Conference on Identification, Information and Knowledge in the Internet of Things.10.1016/j.procs.2019.01.234

10.1016/j.procs.2019.01.234

© 2019 The Authors. Published by Elsevier B.V.This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/)Peer-review under responsibility of the scientific committee of the 2018 International Conference on Identification, Information and Knowledge in the Internet of Things.

1877-0509


ScienceDirect

Procedia Computer Science 00 (2019) 000–000 www.elsevier.com/locate/procedia

1877-0509 © 2019 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the 2018 International Conference on Identification, Information and Knowledge in the Internet of Things

2018 International Conference on Identification, Information and Knowledge in the Internet of Things, IIKI 2018

A data-driven framework for learners’ cognitive load detection using ECG-PPG physiological feature fusion and XGBoost

classification Chixiang Wang, Junqi Guo*

College of Information Science and Technology, Beijing Normal University, Beijing, 100875, P. R. China

Abstract

Cognitive load condition is of great significance for judging learners’ learning state and improving the learning and teaching effects. This paper proposed a feature fusion based processing framework for high cognitive load detection, which includes heart rate variability (HRV) and pulse rate variability (PRV) acquisition, data preprocessing, feature extraction, feature selection, feature fusion by linear feature dependency modeling (LFDM) and high cognitive load detection by XGBoost classifier. This paper experiment on simulated learning paradigm, and the experimental results show that the proposed framework for detection of high cognitive load outperforms conventional processing approaches that uses HRV or PRV only. This paper compared the effects of using different feature fusion algorithms (PCT, SKRRR, ADMM, LFDM) and different classification algorithms (KNN, SVM, DT, RF, XGBoost), and the final proposed framework outperforms other schemes. The proposed framework achieves approximately 97.2% accuracy of high cognitive load detection. © 2019 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (https://creativecommons.org/licenses/by-nc-nd/4.0/) Peer-review under responsibility of the scientific committee of the 2018 International Conference on Identification, Information and Knowledge in the Internet of Things Keywords: Cognitive load; HRV; PRV; physiological signal processing; feature fusion; classifier; LFDM; XGBoost.

* Corresponding author. Tel.: +86-138-1180-5682.

E-mail address: [email protected]


ScienceDirect







Abstract




2 Chixiang Wang, Junqi Guo/ Procedia Computer Science 00 (2019) 000–000

1. Introduction

Cognitive load is a theoretical notion with an increasingly central role in educational research literature these years. It can be defined as a multidimensional construct representing the load that performing a particular task imposes on the learner’s cognitive system[1]. During student's learning process in the classroom, it is an extremely hard task for students to focus their attention on what the teacher and other students are talking about and meanwhile keep thinking. Their cognitive system suffers from high cognitive load and as the learning time accumulates, their attention, responding rate and correct rate gradually decrease. Too much cognitive load has a dramatic bad impact on learning and teaching and previous studies have proved this association, e.g. It is suggested that increasing student’s cognitive load results in the ineffectiveness of problem solving[2]. Also, the famous cognitive load theory proposes that cognitive capacity in working memory is limited, so that if a learning task requires too much capacity, learning will be hampered [3]. Instructional methods to decrease extraneous cognitive load have thus been the central focus of educational studies so that available cognitive resources can be fully devoted to learning[4]. Therefore, this paper aims to construct a framework that can detect students’ cognitive load in the classroom, based on physiological signal acquisition and feature fusion, to guide teaching and learning.

As for the detection of cognitive load, the three most commonly used measurement methods are subjective measurement[5], task performance measurement and physiological measurement[6]. Besides methods based on subjective scales and statistical analysis, methods based on physiological signals are gaining more and more attention – machine learning techniques can better assist in detection and prediction. We get experience from existing studies focusing on mental fatigue and learning. Mental fatigue refers to a physiological and psychological change experienced by people when they perform cognitive activities that require sustained mental attention for a long time. Prolonged cognitive load can cause mental fatigue in general[7], which is a common case in classroom. The subjective measure of mental fatigue has also been proved highly correlated with cognitive load[8]. For physiological cognitive load measurements, the two main perspectives are HRV (heart rate variability)[9] and PR (pupillary responses)[10]. A large number of studies have found that heart rate (HR) and heart rate variability (HRV) can distinguish different states of mental fatigue[11]. Time-frequency analysis (TFA) of HRV found that, HRV can distinguish whether the subject is under normal state or mental fatigue state[11].The study found that the high spectral band (0.15-0.4 Hz) under mental stress was reduced and shifted to higher spectral bands. Scholars have also confirmed that HRV (total power, very low power, high power) is significantly different between relax state and mental fatigue state [12]. For the study for combination of HRV and PRV, Hayano proposed a Pulse Frequency Demodulation (PFDM) method and found that, through HRV correlation analysis, whether it is low frequency or high frequency part, high similarity (ICC ≈ 0.99) between HRV and PRV is exhibited[13].

In this paper, we proposed a framework for high cognitive load detection based on physiological signal ECG and PPG. First we estimate heart rate variability (HRV) and pulse rate variability (PRV) from pre-processed ECG and PPG signal. Then extract features from HRV and PRV. After that feature extraction and feature selection based on LDA is performed. Then we proposed feature fusion scheme using linear feature dependency modeling (LFDM) and cognitive load condition classification scheme using XGBoost classifier.

The contributions of this paper are as follows: (1) A novel framework, based on feature fusion and ML, is constructed to solve the problem of high cognitive load detection. (2) Previous studies have confirmed the redundancy between HRV and PRV features. This paper innovatively used the LFDM algorithm for the feature fusion of HRV and PRV. (3) The experimental paradigm was designed specifically to collect physiological signals under different cognitive load conditions. The experiment results demonstrate the effectiveness of the framework.

2. Methodology

The proposed framework consists of processing steps as detailed in the following sections. The overall flowchart is shown below in Figure 1.

http://crossmark.crossref.org/dialog/?doi=10.1016/j.procs.2019.01.234&domain=pdf

Chixiang Wang et al. / Procedia Computer Science 147 (2019) 338–348 339


ScienceDirect







Abstract





ScienceDirect







Abstract





1. Introduction

Cognitive load is a theoretical notion with an increasingly central role in educational research literature these years. It can be defined as a multidimensional construct representing the load that performing a particular task imposes on the learner’s cognitive system[1]. During student's learning process in the classroom, it is an extremely hard task for students to focus their attention on what the teacher and other students are talking about and meanwhile keep thinking. Their cognitive system suffers from high cognitive load and as the learning time accumulates, their attention, responding rate and correct rate gradually decrease. Too much cognitive load has a dramatic bad impact on learning and teaching and previous studies have proved this association, e.g. It is suggested that increasing student’s cognitive load results in the ineffectiveness of problem solving[2]. Also, the famous cognitive load theory proposes that cognitive capacity in working memory is limited, so that if a learning task requires too much capacity, learning will be hampered [3]. Instructional methods to decrease extraneous cognitive load have thus been the central focus of educational studies so that available cognitive resources can be fully devoted to learning[4]. Therefore, this paper aims to construct a framework that can detect students’ cognitive load in the classroom, based on physiological signal acquisition and feature fusion, to guide teaching and learning.

As for the detection of cognitive load, the three most commonly used measurement methods are subjective measurement[5], task performance measurement and physiological measurement[6]. Besides methods based on subjective scales and statistical analysis, methods based on physiological signals are gaining more and more attention – machine learning techniques can better assist in detection and prediction. We get experience from existing studies focusing on mental fatigue and learning. Mental fatigue refers to a physiological and psychological change experienced by people when they perform cognitive activities that require sustained mental attention for a long time. Prolonged cognitive load can cause mental fatigue in general[7], which is a common case in classroom. The subjective measure of mental fatigue has also been proved highly correlated with cognitive load[8]. For physiological cognitive load measurements, the two main perspectives are HRV (heart rate variability)[9] and PR (pupillary responses)[10]. A large number of studies have found that heart rate (HR) and heart rate variability (HRV) can distinguish different states of mental fatigue[11]. Time-frequency analysis (TFA) of HRV found that, HRV can distinguish whether the subject is under normal state or mental fatigue state[11].The study found that the high spectral band (0.15-0.4 Hz) under mental stress was reduced and shifted to higher spectral bands. Scholars have also confirmed that HRV (total power, very low power, high power) is significantly different between relax state and mental fatigue state [12]. For the study for combination of HRV and PRV, Hayano proposed a Pulse Frequency Demodulation (PFDM) method and found that, through HRV correlation analysis, whether it is low frequency or high frequency part, high similarity (ICC ≈ 0.99) between HRV and PRV is exhibited[13].

In this paper, we proposed a framework for high cognitive load detection based on physiological signal ECG and PPG. First we estimate heart rate variability (HRV) and pulse rate variability (PRV) from pre-processed ECG and PPG signal. Then extract features from HRV and PRV. After that feature extraction and feature selection based on LDA is performed. Then we proposed feature fusion scheme using linear feature dependency modeling (LFDM) and cognitive load condition classification scheme using XGBoost classifier.

The contributions of this paper are as follows: (1) A novel framework, based on feature fusion and ML, is constructed to solve the problem of high cognitive load detection. (2) Previous studies have confirmed the redundancy between HRV and PRV features. This paper innovatively used the LFDM algorithm for the feature fusion of HRV and PRV. (3) The experimental paradigm was designed specifically to collect physiological signals under different cognitive load conditions. The experiment results demonstrate the effectiveness of the framework.

2. Methodology

The proposed framework consists of processing steps as detailed in the following sections. The overall flowchart is shown below in Figure 1.

340 Chixiang Wang et al. / Procedia Computer Science 147 (2019) 338–348 Chixiang Wang, Junqi Guo/ Procedia Computer Science 00 (2019) 000–000 3

Figure 1 Overall flowchart of the proposed framework

The experimental paradigm is designed to simulate the learning procedure. ECG and PPG signals were recorded simultaneously during the experiment. Then, the HRV and PRV were estimated from the successive peak intervals of ECG and PPG, respectively. Feature fusion superimposed the features of HRV and PRV, then fed the fusion result into the XGBoost classifier (classification). The result of feature fusion method was compared with classification results that classifier was only fed with HRV or PRV individually. What’s more, this paper also compared different feature fusion algorithms (PCT, SKRRR, ADMM, LFDM) and classification algorithms (KNN, SVM, DT, RF, XGBoost) as shown in Figure 2, after which find out the ultimate feature fusion and classification scheme (LFDM for feature fusion and XGBoost for classification) that achieves the best accuracy for about 97.2%.

Figure 2. After extracting features, feature selection, feature fusion and classification is operated. In experiment, we used different feature fusion and classification algorithms and compared their performance.

2.1. Data acquisition module

Paradigm: 160 subjects were involved in the study. The subjects were teenagers aged 14(±1) years old who were healthy and had no cardiovascular disease, chosen from Tongzhou No.6 middle school of Beijing, including 80 boys and 80 girls. Each subject conducted a mathematics practice test with a duration of about 3 hours. We carefully selected the students to ensure that the mathematics level of each student is roughly the same.

ECG signal (HP 78354A patient monitor) and PPG signal (pulse oximeter module MP506, Nellcor Puritan Bennett) were collected in the process simultaneously and digitized with a sampling rate of 400Hz (Powerlab 4SP, AD Instruments). HRV was obtained from ECG signals after R-wave peaks were detected.

2.2. Preprocessing module

2.2.1 Smooth Prior Approach (SPA) Denoising

SPA is effective as a non-linear approach to detrend. Original PPG signal data contain periodic term and aperiodic term(trend term). SPA filters out the aperiodic term which has high frequency, and remains the periodic term as result.

Assume that the original signal contains two parts:

z = 𝑧𝑧$%&% + 𝑧𝑧%()*+ (1)

Where: 𝑧𝑧$%&% is periodic term, 𝑧𝑧%()*+ is aperiodic term (trendterm). Then there is:


𝑧𝑧%()*+ = 𝐻𝐻𝐻𝐻 + 𝜈𝜈 (2)

Where: 𝐻𝐻 ∈ 𝑅𝑅5×7is the observation matrix; N is the data length. θ ∈ 𝑅𝑅7 is the regression parameter. 𝜈𝜈 is the observation error. H is the unit matrix of choice. Let:

𝐻𝐻 = 𝐼𝐼5×7 (3)

We use single parameter to filter the target term. Calculate parameters by the optimal estimation method. Consequently:

λ= × �̈�𝑔 + 𝑔𝑔 = 𝑓𝑓 (4)

If 𝑓𝑓 = 𝑐𝑐𝑐𝑐𝑐𝑐 𝜔𝜔𝜔𝜔, then 𝑔𝑔 = G(GHIJK)

𝑐𝑐𝑐𝑐𝑐𝑐 𝜔𝜔𝜔𝜔.

It is a typical lowpass filter for discrete time series. Tune the regularization parameters λ, isolate periodic term and aperiodic term effectively. Then get rid of the high-frequency noise.

2.2.2 Median filter baseline drift removing

Take k as midpoint, and define the data window which possesses 𝑁𝑁 points. If 𝑁𝑁 is even, the window is:

M𝑘𝑘 −𝑁𝑁2 , 𝑘𝑘 +

𝑁𝑁2 − 1Q (5)

If N is odd, the window is:

M𝑘𝑘 −𝑁𝑁 − 1

2 , 𝑘𝑘 +𝑁𝑁 − 1

2Q (6)

For every 𝑘𝑘, reorder the data in the window from small to large, and replace x with the new midpoint. Tune the size 𝑁𝑁 of the window till the frequency component of baseline is elected. Get rid of it and then the PPG signal data become horizontal. After data preprocessing, a smooth and periodic signal curve is obtained which is easier to estimate index.

2.2.3 Cubic spline interpolation uniform resampling

Both the heartbeat interval sequence and the pulse wave interval sequence are non-uniform sampling sequences, therefore direct spectral analysis will cause many problems[14]. For example, the non-uniformity can affect the results of spectral analysis, which can obscure the meaning of certain signals[15]. This non-uniformity is also different for different subject and even for the same subject, when they are under different physiological conditions, the average sampling rate is very different. Therefore, the raw data must be uniformly resampled. Li Liping et al. studied the effect of resampling on HRV spectral analysis[16]. The results show that the method of re-sampling using cubic spline interpolation method and using Welch periodogram for spectral analysis is less suitable for spectral analysis of HRV and PRV.

R-R interval sequence is r. Original ECG signal sampling frequency is F(Hz). Sampling time is T(s). The original ECG signal data length is 𝐹𝐹𝐹𝐹. To interpolate the RR interval sequence r over the interval [1, 𝐹𝐹𝐹𝐹], the sample points should be:

Table 1 Resampling sample points

Time 𝑟𝑟G𝐹𝐹 (𝑟𝑟G + 𝑟𝑟=)𝐹𝐹 …… Y𝑟𝑟*𝐹𝐹

Samplingpoint 𝑅𝑅G 𝑅𝑅G …… 𝑅𝑅G

According to the theory of cubic spline interpolation, a piecewise interpolation function 𝑅𝑅(𝜔𝜔) can be calculated to make it a cubic polynomial in time interval Z𝑟𝑟[𝐹𝐹, 𝑟𝑟\HG𝐹𝐹]:

𝑅𝑅\(𝜔𝜔) = 𝐴𝐴\𝜔𝜔_ +𝑏𝑏\𝜔𝜔= +𝐶𝐶\𝜔𝜔 +𝐷𝐷\ (7)

According to the cubic spline interpolation theory, the function 𝑅𝑅(𝜔𝜔) that satisfies the conditions can be calculated and then resampled.

Chixiang Wang et al. / Procedia Computer Science 147 (2019) 338–348 341 Chixiang Wang, Junqi Guo/ Procedia Computer Science 00 (2019) 000–000 3

Figure 1 Overall flowchart of the proposed framework

The experimental paradigm is designed to simulate the learning procedure. ECG and PPG signals were recorded simultaneously during the experiment. Then, the HRV and PRV were estimated from the successive peak intervals of ECG and PPG, respectively. Feature fusion superimposed the features of HRV and PRV, then fed the fusion result into the XGBoost classifier (classification). The result of feature fusion method was compared with classification results that classifier was only fed with HRV or PRV individually. What’s more, this paper also compared different feature fusion algorithms (PCT, SKRRR, ADMM, LFDM) and classification algorithms (KNN, SVM, DT, RF, XGBoost) as shown in Figure 2, after which find out the ultimate feature fusion and classification scheme (LFDM for feature fusion and XGBoost for classification) that achieves the best accuracy for about 97.2%.

Figure 2. After extracting features, feature selection, feature fusion and classification is operated. In experiment, we used different feature fusion and classification algorithms and compared their performance.

2.1. Data acquisition module

Paradigm: 160 subjects were involved in the study. The subjects were teenagers aged 14(±1) years old who were healthy and had no cardiovascular disease, chosen from Tongzhou No.6 middle school of Beijing, including 80 boys and 80 girls. Each subject conducted a mathematics practice test with a duration of about 3 hours. We carefully selected the students to ensure that the mathematics level of each student is roughly the same.

ECG signal (HP 78354A patient monitor) and PPG signal (pulse oximeter module MP506, Nellcor Puritan Bennett) were collected in the process simultaneously and digitized with a sampling rate of 400Hz (Powerlab 4SP, AD Instruments). HRV was obtained from ECG signals after R-wave peaks were detected.

2.2. Preprocessing module

2.2.1 Smooth Prior Approach (SPA) Denoising

SPA is effective as a non-linear approach to detrend. Original PPG signal data contain periodic term and aperiodic term(trend term). SPA filters out the aperiodic term which has high frequency, and remains the periodic term as result.

Assume that the original signal contains two parts:

z = 𝑧𝑧$%&% + 𝑧𝑧%()*+ (1)

Where: 𝑧𝑧$%&% is periodic term, 𝑧𝑧%()*+ is aperiodic term (trendterm). Then there is:


𝑧𝑧%()*+ = 𝐻𝐻𝐻𝐻 + 𝜈𝜈 (2)

Where: 𝐻𝐻 ∈ 𝑅𝑅5×7is the observation matrix; N is the data length. θ ∈ 𝑅𝑅7 is the regression parameter. 𝜈𝜈 is the observation error. H is the unit matrix of choice. Let:

𝐻𝐻 = 𝐼𝐼5×7 (3)

We use single parameter to filter the target term. Calculate parameters by the optimal estimation method. Consequently:

λ= × �̈�𝑔 + 𝑔𝑔 = 𝑓𝑓 (4)

If 𝑓𝑓 = 𝑐𝑐𝑐𝑐𝑐𝑐 𝜔𝜔𝜔𝜔, then 𝑔𝑔 = G(GHIJK)

𝑐𝑐𝑐𝑐𝑐𝑐 𝜔𝜔𝜔𝜔.

It is a typical lowpass filter for discrete time series. Tune the regularization parameters λ, isolate periodic term and aperiodic term effectively. Then get rid of the high-frequency noise.

2.2.2 Median filter baseline drift removing

Take k as midpoint, and define the data window which possesses 𝑁𝑁 points. If 𝑁𝑁 is even, the window is:

M𝑘𝑘 −𝑁𝑁2 , 𝑘𝑘 +

𝑁𝑁2 − 1Q (5)

If N is odd, the window is:

M𝑘𝑘 −𝑁𝑁 − 1

2 , 𝑘𝑘 +𝑁𝑁 − 1

2Q (6)

For every 𝑘𝑘, reorder the data in the window from small to large, and replace x with the new midpoint. Tune the size 𝑁𝑁 of the window till the frequency component of baseline is elected. Get rid of it and then the PPG signal data become horizontal. After data preprocessing, a smooth and periodic signal curve is obtained which is easier to estimate index.

2.2.3 Cubic spline interpolation uniform resampling

Both the heartbeat interval sequence and the pulse wave interval sequence are non-uniform sampling sequences, therefore direct spectral analysis will cause many problems[14]. For example, the non-uniformity can affect the results of spectral analysis, which can obscure the meaning of certain signals[15]. This non-uniformity is also different for different subject and even for the same subject, when they are under different physiological conditions, the average sampling rate is very different. Therefore, the raw data must be uniformly resampled. Li Liping et al. studied the effect of resampling on HRV spectral analysis[16]. The results show that the method of re-sampling using cubic spline interpolation method and using Welch periodogram for spectral analysis is less suitable for spectral analysis of HRV and PRV.

R-R interval sequence is r. Original ECG signal sampling frequency is F(Hz). Sampling time is T(s). The original ECG signal data length is 𝐹𝐹𝐹𝐹. To interpolate the RR interval sequence r over the interval [1, 𝐹𝐹𝐹𝐹], the sample points should be:

Table 1 Resampling sample points

Time 𝑟𝑟G𝐹𝐹 (𝑟𝑟G + 𝑟𝑟=)𝐹𝐹 …… Y𝑟𝑟*𝐹𝐹

Samplingpoint 𝑅𝑅G 𝑅𝑅G …… 𝑅𝑅G

According to the theory of cubic spline interpolation, a piecewise interpolation function 𝑅𝑅(𝜔𝜔) can be calculated to make it a cubic polynomial in time interval Z𝑟𝑟[𝐹𝐹, 𝑟𝑟\HG𝐹𝐹]:

𝑅𝑅\(𝜔𝜔) = 𝐴𝐴\𝜔𝜔_ +𝑏𝑏\𝜔𝜔= +𝐶𝐶\𝜔𝜔 +𝐷𝐷\ (7)

According to the cubic spline interpolation theory, the function 𝑅𝑅(𝜔𝜔) that satisfies the conditions can be calculated and then resampled.


2.3. Feature extraction module

Malik’s research indicated a series of indications and measure methods for HRV[17]. In this paper, the indications and measurements are operated in accordance with this Standard. Features were extracted from time-domain analysis, frequency-domain analysis and nonlinear analysis.

Table 2 Definition of time domain analysis parameter indicators.

Features’meaning Calculationmethod

AVNN Average of all R-R intervals 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 =

∑ 𝑅𝑅𝑅𝑅\5\fG

𝐴𝐴 (8)

SDNN The standard deviation of all RR intervals, in ms. Overall estimate of HRV. 𝑆𝑆𝑆𝑆𝐴𝐴𝐴𝐴 =i

1𝐴𝐴 − 1

Yj𝑅𝑅𝑅𝑅\ − 𝑅𝑅𝑅𝑅kkkkl=5

\fG

(9)

rMSSD The root mean square of the difference between adjacent RR intervals, used to reflect the size of the rapidly changing component

𝑟𝑟𝑟𝑟𝑆𝑆𝑆𝑆𝑆𝑆 =i1

𝐴𝐴 − 1Yj𝑅𝑅𝑅𝑅\HG − 𝑅𝑅𝑅𝑅\l

=5

\fG

(10)

pNN50 The number of adjacent RR interval differences over 50ms as a percentage of the total

𝑝𝑝𝐴𝐴𝐴𝐴50 = 𝐴𝐴𝐴𝐴50𝐴𝐴 − 1 × 100% (11)

SDANN[18] The standard deviation of the mean RR interval in all 5min segments in a 24-hour record 𝑆𝑆𝑆𝑆𝐴𝐴𝐴𝐴𝐴𝐴 =i

1𝑟𝑟 − 1

Yj𝑅𝑅𝑅𝑅k\ − 𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑅𝑅𝑅𝑅kl=

7

[fG

(12)

SDNNIdx The average of the RR interval standard deviations for all 5 min segments in a 24-hour record

𝑆𝑆𝑆𝑆𝐴𝐴𝐴𝐴𝑆𝑆𝑆𝑆𝑆𝑆 = 1𝑟𝑟

Y𝑆𝑆𝑆𝑆[

7

[fG

(13)

Table 3 Definition of frequency domain analysis parameter indicators

Indicators Unit Indicatormeaning Frequencyband(𝐻𝐻𝐻𝐻)TP 𝑚𝑚𝑚𝑚= Total power 0~0.4𝐻𝐻𝐻𝐻 VLF 𝑚𝑚𝑚𝑚= Very low frequency power 0~0.04𝐻𝐻𝐻𝐻 LF 𝑚𝑚𝑚𝑚= Low frequency power 0.04~0.15𝐻𝐻𝐻𝐻 HF 𝑚𝑚𝑚𝑚= High frequency power 0.15~0.4𝐻𝐻𝐻𝐻

LF/HF / Ratio of LF to HF / The time domain and frequency domain analysis indicators of HRV can quantitatively reflect heart rate variability

at different time scales. At the same time, the nonlinear analysis index of HRV can quantitatively reflect the structure and complexity of RRI[19].

Nonlinear analysis of HRV and PRV was performed using a Poincare plot. Take HRV/PRV time series as 𝑋𝑋G, 𝑋𝑋=,… ,𝑋𝑋*. The Poincare plot is a scatter plot, which plots two consecutive data points (𝑋𝑋[, 𝑋𝑋[HG). The short axis is 𝑆𝑆𝑆𝑆G and the long axis is 𝑆𝑆𝑆𝑆=. The 𝑆𝑆𝑆𝑆G calculation is performed by (14) and 𝑆𝑆𝑆𝑆= is estimated by (15):

𝑆𝑆𝑆𝑆G= = 𝛾𝛾}(0) − 𝛾𝛾}(𝑚𝑚) ⇒ 𝑆𝑆𝑆𝑆G = 𝐹𝐹(𝛾𝛾}(0) − 𝛾𝛾}(𝑚𝑚)) (14)

𝑆𝑆𝑆𝑆== = 𝛾𝛾}(0) + 𝛾𝛾}(𝑚𝑚) − 2𝑋𝑋

=⇒ 𝑆𝑆𝑆𝑆= = 𝐹𝐹(𝛾𝛾}(0) + 𝛾𝛾}(𝑚𝑚)) (15)


𝑆𝑆𝑆𝑆G= is the ratio of the two can reflect the complexity of HRV. It shows the balance between sympathetic/ parasympathetic arms or between the short and long X interval variations [20]. The higher the complexity, the larger the value of 𝑆𝑆𝑆𝑆G=.

𝑆𝑆𝑆𝑆G= = 𝑆𝑆𝑆𝑆G𝑆𝑆𝑆𝑆=

� (16)

2.4. Feature selection module

Extracted features may contain a number of redundant and irrelevant features that if not properly removed, may negatively affect the accuracy of classification process [21, 22]. Therefore, this paper used Fisher’s Linear Discriminant Analysis (LDA)[23] to select an optimal feature subsets of HRV and PRV with minimum redundancy and maximum class discriminability from the large extracted set. LDA is calculated over every features, which is defined as

𝐽𝐽(𝑓𝑓) =|𝜇𝜇G − 𝜇𝜇ƒ|𝜎𝜎G

= + 𝜎𝜎ƒ= (17)

The feature selection process is considered successful if the dimensionality of the feature set is reduced while the accuracy of the classification is either improved or remains unaffected. Afterward, user can decide how much discriminant features they would like to select based on validation sets and pick up features with the highest J values.

2.5. Feature fusion module

As illustrated before, the EEG and PPG windows were pre-processed before being fed to the feature extraction unit. From the larger set of features extracted, a feature subset was selected using the LDA-based feature selection method. After that, The algorithm procedure for feature fusion of HRV and PRV is shown in Figure 3. This approach is based on Linear Feature Dependency Modeling (LFDM)[24], after which feature-level fusion is achieved.

Figure 3 The algorithm procedure of LFDM

This algorithm models the dependencies between classifiers by adding a linear dependency 𝛼𝛼[†𝑃𝑃(𝜔𝜔†)𝛿𝛿[

†, where


𝛼𝛼[† represents the dependency weight of the i-th classifier, and 𝛿𝛿[

† is a minimum value. Finally, a LFDM based on the posteriori probability summation form is described as

𝑃𝑃(𝜔𝜔†|𝑥𝑥GG,… , 𝑥𝑥G5‰,… , 𝑥𝑥75‰ ,… , 𝑥𝑥75Š) = 𝑃𝑃ƒ‹ Œ𝑃𝑃(𝜔𝜔†) + YY 𝛾𝛾[*

ç (𝑃𝑃(𝜔𝜔†|𝑥𝑥[*) − 𝑃𝑃(𝜔𝜔†)5Ž

*fG

7

[fG

)è(18)

LFDM is an “optimal” feature-level combination method in the sense of feature dependency modelling. And it’s not affected by the feature dimension. Compared with the classifier-level fusion algorithm, the feature-level fusion algorithm can directly use the feature data of the sample to fuse multiple features, which can solve the problem of small intra-class variance in the classification problem.

2.6. Classification module

The XGBoost (eXtreme Gradient Boosting) algorithm is an efficient implementation of the Gradient Boosting algorithm[25]. The XGBoost algorithm can be considered as a further optimization based on the GBDT algorithm. First, the XGBoost algorithm introduces a regular term in the base learner loss function, which reduces the over-fitting in the training process. Secondly, the XGBoost algorithm not only uses the first derivative to calculate the pseudo-residue, but also calculates the second derivative to approximate the fast pruning.

Define objective, calculate 𝑔𝑔[, ℎ[, feed it to the old tree learning algorithm we have for un-weighted version as followed formula. For objective:

𝑂𝑂𝑂𝑂𝑂𝑂 = Y 𝑙𝑙(𝑦𝑦[, 𝑦𝑦•–)*

[fG+ Y 𝛺𝛺(𝑓𝑓†)

˜

˜fG(19)

The former means training loss, and the latter means complexity of the trees. XGBoost uses additive training to find f. Start from constant prediction, add a new function each time:

𝑦𝑦•–(ƒ) = 0

𝑦𝑦•–(G) = 𝑓𝑓G(𝑥𝑥[) = 𝑦𝑦•–

(ƒ) + 𝑓𝑓G(𝑥𝑥[)

𝑦𝑦•–(=) = 𝑓𝑓G(𝑥𝑥[) + 𝑓𝑓=(𝑥𝑥[) = 𝑦𝑦•–

(G) + 𝑓𝑓=(𝑥𝑥[)

…

𝑦𝑦•–(%) = ∑ 𝑓𝑓†(𝑥𝑥[)%

˜fG = 𝑦𝑦•–(%™G) + 𝑓𝑓%(𝑥𝑥[) (20)

Define:

𝑂𝑂𝑂𝑂𝑂𝑂(%) = Y š›Y 𝑔𝑔[[∈œù

ž𝜔𝜔\ +12›Y ℎ[

[∈œù+ 𝜆𝜆ž𝜔𝜔\

= ¡

\fG+ 𝛾𝛾𝛾𝛾 = Y M𝐺𝐺\𝜔𝜔\ +

12j𝐻𝐻\ + 𝜆𝜆l𝜔𝜔\

=Q¡

\fG+ 𝛾𝛾𝛾𝛾 (21)

Assume the structure of tree 𝑞𝑞(𝑥𝑥) is fixed, the optimal weight in each leaf, and the resulting objective value are:

𝜔𝜔¤∗ = −

𝐺𝐺¤

𝐻𝐻\ + 𝜆𝜆(22)

𝑂𝑂𝑂𝑂𝑂𝑂 = −12Y

𝐺𝐺\=

𝐻𝐻\ + 𝜆𝜆 +¡

\fG𝛾𝛾𝛾𝛾 (23)

3. Experiment

3.1. Data acquisition

We chose 160 teenagers at the age of 14(±1) in Tongzhou No.6 middle school of Beijing, including 80 boys and 80 girls. As described in section 2.1 above, we collected the full ECG and PPG signals simultaneously from these


teenagers during the mathematics test experiment. These data are used for model training and testing of the learner's cognitive load.

3.2. Impact of high cognitive load on physiology

Some ECG signal samples and HRV are shown below in Figure 4. Here, I took some signal fragments as an example to show the changes of ECG and beats per minute (BPM) of the subject under relax and high cognitive load condition. It’s shown that the BPM of the subject rises up at the end of the experiment compared to that at the beginning of the experiment, which indicates high cognitive load condition.

Figure 4 ECG and HRV of a subject under normal and high cognitive condition.

3.3. The effect of preprocessing

In figure 5, it shows the comparison of PPG waveforms before and after pre-processing. It’s clear that after pre-processing, the waveform becomes more smooth, stable and benefits for subsequent analysis.

Figure 5 PPG waveform before and after pre-processing


In Figure 6, it shows the effect of cubic spline interpolation uniform resampling. In this figure, red spots means the original samples’ time interval, blue curve means the result of interpolation resampling. Y-axis means the time interval, X-axis means the time of the corresponding time interval.

Figure 6 The result of cubic spline interpolation uniform resampling

3.4. Detection of high cognitive load

The chart below shows the sensitivity and specificity comparison of the classification performance based on HRV, PRV and feature fusion, respectively. The feature fusion based classification achieved 92.5% sensitivity and 92.0% specificity, compared to the result of HRV features only (86.7% and 87.0%) and PRV features only (81.8% and 80.2%). This shows

We tested 4 different feature fusion algorithms (PCT, SKRRR, ADMM, LFDM) and 5 classification algorithms (SVM, KNN, DT, RF, XGBoost), aim to find the best algorithm combination. Figure 7 shows the accuracy of the accuracy performance of different fusion algorithms using XGBoost as the classifier. LFDM achieved an accuracy of 97.2%, which overperforms all the other methods.

Figure 7 Comparison of the performance of different feature fusion algorithms, using XGBoost as the classifier.

Table 4 and Figure 8 shows the accuracy comparison for predicting high cognitive load condition using different classification algorithms, and the accuracy using HRV or PRV features individually and in combination, taking LFDM as the feature fusion method. The prediction accuracy using LFDM feature fusion and XGBoost classifier was 97.2%, which overperforms all the other classifiers.


Table 4 Performance comparison of high cognitive load detection in different classifier, respectively using HRV and PRV individually and the proposed feature fusion method.

Accuracy SVM KNN DT RF XGBoost

HRVOnly 80.2% 72.8% 84.6% 82.0% 93.4%

PRVOnly 78.3% 70.1% 83.9% 86.8% 92.7%

Featurefusion 88.4% 81.4% 92.2% 94.6% 97.2%

Figure 8 The accuracy of different classifier and different data source, using LFDM as the feature fusion algorithm.

4. Conclusion

In this paper, we proposed the framework including both physiological signal acquisition, LDA-based feature selection, LFDM-based feature fusion and XGBoost-based classification. We designed an experimental paradigm that effectively simulates the accumulation of cognitive load. We innovatively used the LFDM feature fusion algorithm to fuse the HRV and PRV features and obtained a feature set with better detection effect. XGBoost classifier which gains popularity in recent years is used for learning and prediction and achieved better performance than other classic ML algorithms. This proposed framework achieves the performance of approximately 97.2% accuracy. It outperforms existing processing schemes for detection of cognitive load or mental fatigue that uses HRV or PRV features individually without feature fusion, and that uses antique conventional classifiers.

Acknowledgements

This research is sponsored by Beijing Advanced Innovation Center for Future Education (BJAICFE2016IR-004), “Educational Big Data R&D and its Application”, Major Big Data Engineering Project of National Development and Reform Commission 2017, and National Natural Science Foundation of China (No.61401029).


References

[1] Paas, Fred GWC, Jeroen JG Van Merriënboer, and Jos J. Adam. "Measurement of cognitive load in instructional research." Perceptual and motor skills 79.1 (1994): 419-430. [2] Sweller, John. "Cognitive load during problem solving: Effects on learning." Cognitive science 12.2 (1988): 257-285. [3] De Jong, Ton. "Cognitive load theory, educational research, and instructional design: Some food for thought." Instructional science 38.2 (2010): 105-134. [4] Van Merrienboer, Jeroen JG, and John Sweller. "Cognitive load theory and complex learning: Recent developments and future directions." Educational psychology review 17.2 (2005): 147-177. [5] Schnotz W, Kürschner C. A reconsideration of cognitive load theory. Educational psychology review. 2007 Dec 1;19(4):469-508. [6] Paas F, Tuovinen JE, Tabbers H, Van Gerven PW. Cognitive load measurement as a means to advance cognitive load theory. Educational psychologist. 2003 Mar 1;38(1):63-71. [7] Mizuno, Kei, et al. "Mental fatigue caused by prolonged cognitive load associated with sympathetic hyperactivity." Behavioral and brain functions 7.1 (2011): 17. [8] Nygren, Thomas E. "Psychometric properties of subjective workload measurement techniques: Implications for their use in the assessment of perceived mental workload." Human Factors 33.1 (1991): 17-33. [9]Paas, Fred GWC, and Jeroen JG Van Merriënboer. "Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach." Journal of educational psychology 86.1 (1994): 122. [10] Van Gerven, Pascal WM, et al. "Memory load and the cognitive pupillary response in aging." Psychophysiology 41.2 (2004): 167-174. [11] Taelman, Joachim, et al. "Instantaneous changes in heart rate regulation due to mental load in simulated office work." European journal of applied physiology 111.7 (2011): 1497-1505. [12] Xia, Lan, et al. "Mental fatigue assessment based on physiological signals." Nan fang yi ke da xue xue bao= Journal of Southern Medical University 32.6 (2012): 870-873. [13] Hayano J, Barros A K, Kamiya A, et al. Assessment of pulse rate variability by the method of pulse frequency demodulation[J]. Biomedical Engineering Online, 2005, 4(1):62. [14] Clifford, Gari D., and Lionel Tarassenko. "Quantifying errors in spectral estimates of HRV due to beat replacement and resampling." IEEE transactions on biomedical engineering52.4 (2005): 630-638. [15] Ling Li, Zhizhong, and Liyu. "Research of Unevenly Sampled HRV Signal." Department of Biomedical Engineering, Shanghai Jiaotong University 20.1 (2001): 47-50. [16] Liping Li, et al. "The effect of re-sampling on spectral analysis of pulse interval series." Shandong University 41.2 (2011): 102-106. [17] Malik, Marek, et al. "Heart rate variability: Standards of measurement, physiological interpretation, and clinical use." European heart journal 17.3 (1996): 354-381. [18] Ramshur, John T. Design, evaluation, and application of heart rate variability analysis software (HRVAS). Diss. University of Memphis, 2010. [19] Voss, Andreas, et al. "Methods derived from nonlinear dynamics for analysing heart rate variability." Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 367.1887 (2009): 277-296. [20] Guzik, Przemyslaw, et al. "Correlations between the Poincare plot and conventional heart rate variability parameters assessed during paced breathing." The Journal of Physiological Sciences 57.1 (2007): 63-71. [21] Malarvili, M. B., and M. Mesbah. "HRV feature selection based on discriminant and redundancy analysis for neonatal seizure detection." Information, Communications & Signal Processing, 2007 6th International Conference on. IEEE, 2007. [22] Malarvili, M. B., M. Mesbah, and B. Boashash. "HRV feature selection for neonatal seizure detection: a wrapper approach." Signal Processing and Communications, 2007. ICSPC 2007. IEEE International Conference on. IEEE, 2007. [23] F. Song, D. Mei, and H. L., “Feature Selection Based on Linear Discriminant Analysis,” Intelligent System Design and Engineering Application (ISDEA), 2010 Int’l. Conf., vol. 1, pp. 746–749 (2010) [24] Ma, Andy Jinhua, Pong C. Yuen, and Jian-Huang Lai. "Linear dependency modeling for classifier fusion and feature combination." IEEE transactions on pattern analysis and machine intelligence 35.5 (2013): 1135-1148. [25] Chen, Tianqi, and Carlos Guestrin. "Xgboost: A scalable tree boosting system." Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. ACM, 2016.

ScienceDirect 1. Introduction

Documents

Transcript of ScienceDirect 1. Introduction