RESEARCH POSTER PRESENTATION DESIGN © 2019

www.PosterPresentations.com

Introduction

Proposed Approach

• Proposed a novel technique for user verification using a smartwatch based on a writing pattern or style.

• Extensive experiments are carried out on the newly captured dataset using two different smartwatches with three different writing scenarios (or activities).

Contributions

Experiments and Results

Experiment 1

Experiment 2

Norwegian University of Science and Technology, Gjøvik, Norway

Raghavendra Ramachandra, Sushma Venkatesh, Kiran Raja, Christoph Busch

Handwritten Signature and Text based User Verification using Smartwatch

Research Questions• Q1: Does the verification performance of a user vary with the medium

of writing (electronic tablet or paper)?• Q2: Is the verification performance dependent on the type or make of

the wearable device (or smartwatch)?• Q3: Does the user verification performance vary when the user changes

the type of the smartwatch?• Q4: Does data captured while signing suffice the verification

performance need over the free writing of short sentences?

Database• The data of was captured using two devices (i) Sony Smartwatch 3 (referred to as W1) (ii) LG smartwatch (referred to as W2). • 30 unique data subjects• Three different experiments were designed to capture the writing styles envisaging both real- life applications and convince of user for

authentication. • -The first protocol recorded the signature from the participants; • -The second protocol recorded the short own-sentence that participants could write. • -The third protocol recorded the common sentence from each data subject. Both self-chosen and common sentences were carefully

chosen to be short with few words (3 to 5), the common sentence used for the entire corpus of users in this work is you are smart. The data subjects were also instructed to choose a sentence with less than five words.

• Ten samples were captured for each data subject corresponding to a signature, common sentence, and a self-chosen sentence using wearable at a time while writing on both iPad and paper.

• Thus, for each data subject in the dataset, 3 sessions × 3 activities × 10 samples × 2 smartwatches × 2 mediums (iPad and Paper) = 360 samples. Thus, the whole dataset is comprised of 30 data subjects × 360 = 10800 samples.

0 200 400 600 800 1000-10

-5

0

5

10

15

Acc. XAcc. YAcc. Z

0 200 400 600 800-10

-5

0

5

10

15

Acc. XAcc. YAcc. Z

0 50 100 150 200 250 300 350-15

-10

-5

0

5

10

15

Acc. XAcc. YAcc. Z

Common Sentence Own Sentence Signature

(a)

0 200 400 600 800 1000-10

-5

0

5

10

15

Acc. XAcc. YAcc. Z

0 200 400 600 800 1000 1200 1400-15

-10

-5

0

5

10

15

Acc. XAcc. YAcc. Z

0 100 200 300 400 500 600-10

-5

0

5

10

15

Acc. XAcc. YAcc. Z

(b)Fig. 1: Illustration of Accelerometer data captured from example data subject with iPad using (a) Sony Smart watch 3 (W1)(2) LG smart watch (W2)

authenticating the user. While both conventional and DeepConvolutional Neural Networks (DCNN) were benchmarkedfor user authentication from the captured accelerometer datawhile writing the text, the best performance was obtainedfrom Convolutional-BiLSTM. Despite exploring the free texthandwriting for user authentication, the considerably lengthytext was needed to verify user for authentication [3], [4].Nonetheless, both the earlier works indicated improved userexperience independent of the medium used for signing (e.g.,tablet and/or paper) or writing text.

Albeit the aptness of using the smartwatch to capture thedata while writing the text, the earlier works have not answereda set of questions to list the full potential for user verification.In an endeavour to benchmark the full potential of datacaptured from smartwatch for user verification while writinga text, we draft a set of research questions:

Q1: Does the verification performance of a user vary withthe medium of writing (electronic tablet or paper)?Q2: Is the verification performance dependent on the typeor make of the wearable device (or smartwatch)?Q3: Does the user verification performance vary whenthe user changes the type of the smartwatch?Q4: Does data captured while signing suffice the veri-fication performance need over the free writing of shortsentences?

In answering these questions and considering the lack ofpublicly available smartwatch datasets for such an investiga-tion, we introduce a new dataset of smartwatch data for bothsignature and free handwriting. The new dataset comprisesdata captured from 30 unique subjects collected in three differ-ent sessions using two smartwatches. Both signature and free

handwriting corresponding to each data subject are capturedusing two different writing mediums: paper and electronictablet (iPad). Further, we propose a novel method based ondeep learning and hand-crafted features for authenticating theuser. Extensive experiments are carried out to answer theabove-outlined research questions. The set of main contribu-tions of this work can therefore be listed as:

• Introduces a new dataset collected from 30 unique sub-jects using two different wearable devices. A diverse setof data is captured for each data subject while signing,writing a common sentence and a self-chosen sentence onboth paper and electronic device (iPAD). With the diversenature of dataset, the database serves many unansweredresearch questions beyond the scope of this work alone.

• Proposes a novel method based on Continuous WaveletTransform (CWT) and deep features extracted using thepre-trained ResNet50 to imporve the state-of-art userauthentication.

• Presents extensive evaluation on the newly collecteddataset with various protocols to benchmark the perfor-mance of the proposed method with the other state-of-the-art techniques.

In the rest of the paper, Section II describes the datacollection protocols employed in this work to construct thenew dataset. Section III presents the proposed method, SectionIV presents the quantitative results of the proposed methodtogether with state-of-the-art technique and Section V drawsthe conclusion.

0 50 100 150 200 250 300 350 400 450-2

-1.8

-1.6

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

Deep Features Ensemble Classifier

0 50 100 150 200 250 300 350 400 450 500-7

-6

-5

-4

-3

-2

-1

Deep Features Ensemble Classifier

0 50 100 150 200 250 300 350 400 450 5006.5

7

7.5

8

8.5

9

9.5

10

10.5

11

Deep Features Ensemble Classifier

X

Y

Z

∑Accept

Reject

CWT

Fig. 2: Block Diagram of the proposed Method

II. DATA COLLECTION PROTOCOLS

In this section, we describe the data collection protocols andthe devices used for data collection. The data collection wascarried out with 30 unique data subjects and most of themwere graduate students. The data of was captured using twodevices (i) Sony Smartwatch 3 (referred to as W1) (ii) LGsmartwatch (referred to as W2). All participants have wornthe smartwatch on their dominant hand and performed theexperiments. A data collection application was developed forthe smartwatches, which allowed the participants to start theapplication soon they started writing and stopped after endingthe writing process. The application was designed to gatherthe measurements across three-axis (x, y and z) from thesmartwatch accelerometer. Three different experiments weredesigned to capture the writing styles envisaging both real-life applications and convince of user for authentication. Thefirst experiment recorded the signature from the participants;the second experiment recorded the short own-sentence thatparticipants could write and the third experiment recorded thecommon sentence from each data subject. Both self-chosenand common sentences were carefully chosen to be short withfew words (3 to 5), the common sentence used for the entirecorpus of users in this work is you are smart. The data subjectswere also instructed to choose a sentence with less than fivewords.

Ten samples were captured for each data subject corre-sponding to a signature, common sentence, and a self-chosensentence using wearable at a time while writing on both iPadand paper. This was further repeated in three different sessionswith a time gap of 3-7 days to account for larger sessionvariability and avoid intra-day capture bias. Thus, for eachdata subject in the dataset, 3 sessions ⇥ 3 activities ⇥ 10samples ⇥ 2 smartwatches ⇥ 2 mediums (iPad and Paper) =360 samples. Thus, the whole dataset is comprised of 30 datasubjects ⇥ 360 = 10800 samples.

Further to study the classical signature capture set-up versusthe digital signature set-up, the data subjects were asked towear smartwatch while writing on both papers and iPad (elec-

tronic pads). When the data subject performed the activities(signature, common sentence and own sentence) on the paper,the paper was placed (vertical to a subject) on the table andthe data subject was asked to adjust the position before writingon the paper in a comfortable manner. However, the iPad wasplaced in an elevated position at an angle of approximately60 degrees to allow a better user experience and simulatethe most shopping kiosk signing experience. Both of theseuser interactions capture large user experiences across real-life scenarios like in banking, post office, or signing a receiptat a restaurant. An illustrative sample of the data is shown inFigure 1 depicting the captured accelerometer data for bothwatches with common sentence, own sentence and signature.The variations across mediums of writing, smartphone usedand the sentence is written can be noticed evidently.

III. PROPOSED METHOD

With the set of given data, we first evaluated the state-of-artalgorithms and to complement the existing state-of-art, we alsopropose a new approach to user verify the users as illustratedin Figure 2. The proposed method is designed to process theindividual accelerometer signals using the proposed featureextraction with Continuous Wavelet Transform (CWT) andthe deep features computed using off-the-shelf pre-trainedResNet50. The extracted features are further classified usingan off-the-shelf ensemble of classifiers whose comparisonscores are computed independently on X, Y and Z componentsof accelerometer data. To obtain a superior user verificationperformance, we combine the independent decisions using theSUM rule to verify the user in the final decision. The proposedmethod has four functional blocks: (i) Pre-processing and ROIextraction block, (ii)feature extraction block, (iii) classificationblock and, (iv) fusion block.

A. Pre-processing and ROI extraction

The pre-processing is the vital step to accurately segmentthe Region Of Interest (ROI), which localizes the activity per-formed using the smartwatch effectively. As the user enablesthe application before performing the activity, the collected

data has traces of the hand’s initial movements prior and postwriting activity. Pre-processing the signals helps to discard theunintended data pertaining to hand movements and retains thesignal components that absolutely represent the activities. Inthis work, we use the local minima-based operations to extractthe ROI reliably. Given the accelerometer signal, we firstsmoothen the signal and then divide the signal in two halves.We then compute the local minima points independently onboth halves of the signal. We then choose the maximum pointfrom the computed local-minima in both halves that servesas the anchor points for segmenting the signal. Since both ofthese anchor points detect the extreme local minima points,we augment both of these points by a value of 20 to locatethe exact points for the ROI extraction. The value of 20 ischosen manually based on visual analysis of a set of samplesand empirically validated for optimal results. Based on thesetwo computed points, the signal is cropped to extract the ROI.A similar procedure is used on both smartwatches to obtainthe ROI.

0 50 100 150 200 250 300 350 400-4

-3

-2

-1

0

1

2

3

4

5

0 50 100 150 200 250 300 350 400-15

-10

-5

0

5

10

0 50 100 150 200 250 300 350 400-4

-2

0

2

4

6

8

10

0 20 40 60 80 100 120 140 1601.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

3

0 20 40 60 80 100 120 140 160-9.5

-9

-8.5

-8

-7.5

-7

-6.5

-6

0 20 40 60 80 100 120 140 1601

1.5

2

2.5

3

3.5

4

4.5

(a)

(b)

Accelerometer-X Accelerometer-Y Accelerometer-Z

Fig. 3: Illustration of Pre-Processing and ROI extraction: (a)Raw Signal (b) ROI

B. Feature extraction

Given the localized ROI, we extract the features from thesignals through the proposed feature extraction approach. Thefeatures are extracted using the Continuous Wavelet Transform(CWT) given the nature of the continuous signal and tocomplement it with deeply learned features, we also extractfeatures using Convolutional Neural Network (CNN). Giventhe accelerometer signal, we compute the time-frequencyfeatures using CWT. We obtain the given signal’s scalogramby computing the absolute value of the CWT coefficientscorresponding to the signal. To create a scalogram of the givensignal, we compute the CWT filter bank based on the Morsewavelet. In this work, we use the first 2500 sample points fromthe given signal to obtain the scalogram. We employ the pre-trained ResNet50 to extract the deep features correspondingto the computed scalogram in the next step. The deep featuresare extracted from a fully connected layer of ResNet50 thatprovides the features of dimension 2048⇥ 1.

C. Classification

Given the random nature of the signal subspace, we employoff-the-shelf ensemble classifier to classify the features. Thechoice of such approach is based on inherent advantage ofusing less memory yet handling the missing feature values.It is therefore justified as an viable approach for the lowcomputation platforms like a smartphone or smartwatch. Fur-ther, we have trained the classifier for user authenticationindividually for each data subject independently on threedifferent accelerometer data (X, Y and Z).

D. Fusion

In order to further improve the verification performanceand also fully utilize all the available accelerometer signalscorresponding to X, Y and Z components, we fuse the scoresfrom each classifier using SUM rule fusion to make the finaldecision.

TABLE I: Quantitative performance of the proposed methodon Smartwatch-1 (W1)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 7.67

Sentence Paper 6.67

Common iPAD 9.36

Sentence Paper 6.14

SignatureiPAD 6.72

Paper 3.33

Proposed Method

Own iPAD 0

Sentence Paper 0

Common iPAD 0

Sentence Paper 0

SignatureiPAD 0

Paper 0

IV. EXPERIMENTS AND RESULTS

In this section, we report the quantitative results of theproposed method for smartwatch based user authentication.We also benchmark the proposed method’s performance withthe state-of-the-art techniques based on CNN-LSTM (or Bi-LSTM) introduced in [4]. Owing to the lack of publiclyavailable datasets, we have evaluated both state-of-the-artand proposed methods on our newly introduced dataset withmultiple activities and multiple smartwatches. The results arepresented using Equal Error Rate (EER%) that correspondsto the points where False Match Rate (FMR%) is equal toFalse Non-Match Rate (FMNR%). To answer the outlinedresearch questions (see introduction), we have carried outtwo main experiments (a) Experiment-1: We enrol and probethe data subjects with the same smartwatch independentlyfor each writing activity. (b) Experiment-2: Here, each data

vExperiment-1: We enrol and probe the data subjects with the same smartwatch independently for each writing activity.v Experiment-2: Here, each data subject is enroled with one type of smartwatch (for example, W1) and probed with another smartwatch (for example, W2) independently for each activity.

data has traces of the hand’s initial movements prior and postwriting activity. Pre-processing the signals helps to discard theunintended data pertaining to hand movements and retains thesignal components that absolutely represent the activities. Inthis work, we use the local minima-based operations to extractthe ROI reliably. Given the accelerometer signal, we firstsmoothen the signal and then divide the signal in two halves.We then compute the local minima points independently onboth halves of the signal. We then choose the maximum pointfrom the computed local-minima in both halves that servesas the anchor points for segmenting the signal. Since both ofthese anchor points detect the extreme local minima points,we augment both of these points by a value of 20 to locatethe exact points for the ROI extraction. The value of 20 ischosen manually based on visual analysis of a set of samplesand empirically validated for optimal results. Based on thesetwo computed points, the signal is cropped to extract the ROI.A similar procedure is used on both smartwatches to obtainthe ROI.

0 50 100 150 200 250 300 350 400-4

-3

-2

-1

0

1

2

3

4

5

0 50 100 150 200 250 300 350 400-15

-10

-5

0

5

10

0 50 100 150 200 250 300 350 400-4

-2

0

2

4

6

8

10

0 20 40 60 80 100 120 140 1601.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

3

0 20 40 60 80 100 120 140 160-9.5

-9

-8.5

-8

-7.5

-7

-6.5

-6

0 20 40 60 80 100 120 140 1601

1.5

2

2.5

3

3.5

4

4.5

(a)

(b)

Accelerometer-X Accelerometer-Y Accelerometer-Z

Fig. 3: Illustration of Pre-Processing and ROI extraction: (a)Raw Signal (b) ROI

B. Feature extraction

Given the localized ROI, we extract the features from thesignals through the proposed feature extraction approach. Thefeatures are extracted using the Continuous Wavelet Transform(CWT) given the nature of the continuous signal and tocomplement it with deeply learned features, we also extractfeatures using Convolutional Neural Network (CNN). Giventhe accelerometer signal, we compute the time-frequencyfeatures using CWT. We obtain the given signal’s scalogramby computing the absolute value of the CWT coefficientscorresponding to the signal. To create a scalogram of the givensignal, we compute the CWT filter bank based on the Morsewavelet. In this work, we use the first 2500 sample points fromthe given signal to obtain the scalogram. We employ the pre-trained ResNet50 to extract the deep features correspondingto the computed scalogram in the next step. The deep featuresare extracted from a fully connected layer of ResNet50 thatprovides the features of dimension 2048⇥ 1.

C. Classification

Given the random nature of the signal subspace, we employoff-the-shelf ensemble classifier to classify the features. Thechoice of such approach is based on inherent advantage ofusing less memory yet handling the missing feature values.It is therefore justified as an viable approach for the lowcomputation platforms like a smartphone or smartwatch. Fur-ther, we have trained the classifier for user authenticationindividually for each data subject independently on threedifferent accelerometer data (X, Y and Z).

D. Fusion

In order to further improve the verification performanceand also fully utilize all the available accelerometer signalscorresponding to X, Y and Z components, we fuse the scoresfrom each classifier using SUM rule fusion to make the finaldecision.

TABLE I: Quantitative performance of the proposed methodon Smartwatch-1 (W1)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 7.67

Sentence Paper 6.67

Common iPAD 9.36

Sentence Paper 6.14

SignatureiPAD 6.72

Paper 3.33

Proposed Method

Own iPAD 0

Sentence Paper 0

Common iPAD 0

Sentence Paper 0

SignatureiPAD 0

Paper 0

IV. EXPERIMENTS AND RESULTS

In this section, we report the quantitative results of theproposed method for smartwatch based user authentication.We also benchmark the proposed method’s performance withthe state-of-the-art techniques based on CNN-LSTM (or Bi-LSTM) introduced in [4]. Owing to the lack of publiclyavailable datasets, we have evaluated both state-of-the-artand proposed methods on our newly introduced dataset withmultiple activities and multiple smartwatches. The results arepresented using Equal Error Rate (EER%) that correspondsto the points where False Match Rate (FMR%) is equal toFalse Non-Match Rate (FMNR%). To answer the outlinedresearch questions (see introduction), we have carried outtwo main experiments (a) Experiment-1: We enrol and probethe data subjects with the same smartwatch independentlyfor each writing activity. (b) Experiment-2: Here, each data

Common Sentence Own Sentence Signature

(a)

(b)

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

Fig. 4: DET curves on W1 versus W2 (a) Input device: iPAD (b) Input device: Paper

TABLE II: Quantitative performance of the proposed methodon Smartwatch-2 (W2)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 6.67

Sentence Paper 3.33

Common iPAD 10

Sentence Paper 3.33

SignatureiPAD 3.33

Paper 1.78

Proposed Method

Own iPAD 0

Sentence Paper 0

Common iPAD 0

Sentence Paper 0

SignatureiPAD 0

Paper 0

subject is enroled with one type of smartwatch (for example,W1) and probed with another smartwatch (for example, W2)independently for each activity. In both experiments, we haveused the leave-one-out cross-validation such that each datasubject, 29 samples are used for training and 1 sample is usedfor testing. The leave-one-out K � fold cross-validation isrepeated with K = 5 and the average results are presented.We follow this evaluation protocol independently for all threeactivities in both Experiment-1 and Experiment-2.

A. Results on Experiment-1

Table I indicates the performance of the proposed methodwith the existing method for all three different activities cap-tured using the smartwatch W1. Table II shows the quantitativeperformance of both proposed and existing methods on allthree activities captured using smartwatch W2. Based on theobtained results, the following are the main observations:

• The proposed method indicates the best performance onboth W1 and W2 when compared to that of the existingmethod on all three different activities. This shows therobustness of the proposed method on both activities andthe type of the smartwatch.

Common Sentence Own Sentence Signature

(a)

(b)

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

Fig. 4: DET curves on W1 versus W2 (a) Input device: iPAD (b) Input device: Paper

TABLE II: Quantitative performance of the proposed methodon Smartwatch-2 (W2)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 6.67

Sentence Paper 3.33

Common iPAD 10

Sentence Paper 3.33

SignatureiPAD 3.33

Paper 1.78

Proposed Method

Own iPAD 0

Sentence Paper 0

Common iPAD 0

Sentence Paper 0

SignatureiPAD 0

Paper 0

subject is enroled with one type of smartwatch (for example,W1) and probed with another smartwatch (for example, W2)independently for each activity. In both experiments, we haveused the leave-one-out cross-validation such that each datasubject, 29 samples are used for training and 1 sample is usedfor testing. The leave-one-out K � fold cross-validation isrepeated with K = 5 and the average results are presented.We follow this evaluation protocol independently for all threeactivities in both Experiment-1 and Experiment-2.

A. Results on Experiment-1

Table I indicates the performance of the proposed methodwith the existing method for all three different activities cap-tured using the smartwatch W1. Table II shows the quantitativeperformance of both proposed and existing methods on allthree activities captured using smartwatch W2. Based on theobtained results, the following are the main observations:

• The proposed method indicates the best performance onboth W1 and W2 when compared to that of the existingmethod on all three different activities. This shows therobustness of the proposed method on both activities andthe type of the smartwatch.

Common Sentence Own Sentence Signature

(a)

(b)

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

Fig. 5: DET curves on W2 versus W1 (a) Input device: iPAD (b) Input device: Paper

• The existing method also indicates a reasonable perfor-mance of all three different activities on both smart-watches W1 and W2. Among the three different activ-ities, the existing method indicates the best performancewith signature, and the degraded performance is notedwhen the subject wrote his/her common sentence. Such adegraded performance indicates that, the use of commonsentence might not provide a reasonable user verificationsince the writing style captured from motion sensors ofsmartwatch fails to provide the discriminatory features.

B. Results on Experiment-2

In this section, we present the quantitative results on theExperiment-2 to provide the insights on the performance ofboth the proposed method and existing methods when thedata from the data subject is enroled from one smartwatchand probed using the different smartwatch. Table III showsthe quantitative performance of both the proposed method andthe existing methods for the same and Figure 4 shows theDET curves indicating the performance on all three differentactivities. Based on the obtained results, the following are theimportant observations:

• When compared to the Experiment-1, the performance ofboth the proposed and existing method indicates the de-

TABLE III: Quantitative performance of the proposed method:Smartwatch-1 (W1) versus Smartwatch-2 (W2)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 23.33

Sentence Paper 27.29

Common iPAD 26.26

Sentence Paper 33.33

SignatureiPAD 43.44

Paper 36.67

Proposed Method

Own iPAD 3.33

Sentence Paper 6.55

Common iPAD 3.39

Sentence Paper 2.72

SignatureiPAD 9.25

Paper 6.89

Common Sentence Own Sentence Signature

(a)

(b)

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5

1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

0.1 0.2 0.5 1 2 5 10 20 40 False Match Rate (FMR%)

0.1 0.2

0.5 1

2

5

10

20

40

Fals

e N

on-M

atch

Rat

e (F

NM

R%

)

DET curve

CNN-BiLSTMProposed Method

Fig. 5: DET curves on W2 versus W1 (a) Input device: iPAD (b) Input device: Paper

• The existing method also indicates a reasonable perfor-mance of all three different activities on both smart-watches W1 and W2. Among the three different activ-ities, the existing method indicates the best performancewith signature, and the degraded performance is notedwhen the subject wrote his/her common sentence. Such adegraded performance indicates that, the use of commonsentence might not provide a reasonable user verificationsince the writing style captured from motion sensors ofsmartwatch fails to provide the discriminatory features.

B. Results on Experiment-2

In this section, we present the quantitative results on theExperiment-2 to provide the insights on the performance ofboth the proposed method and existing methods when thedata from the data subject is enroled from one smartwatchand probed using the different smartwatch. Table III showsthe quantitative performance of both the proposed method andthe existing methods for the same and Figure 4 shows theDET curves indicating the performance on all three differentactivities. Based on the obtained results, the following are theimportant observations:

• When compared to the Experiment-1, the performance ofboth the proposed and existing method indicates the de-

TABLE III: Quantitative performance of the proposed method:Smartwatch-1 (W1) versus Smartwatch-2 (W2)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 23.33

Sentence Paper 27.29

Common iPAD 26.26

Sentence Paper 33.33

SignatureiPAD 43.44

Paper 36.67

Proposed Method

Own iPAD 3.33

Sentence Paper 6.55

Common iPAD 3.39

Sentence Paper 2.72

SignatureiPAD 9.25

Paper 6.89

TABLE IV: Quantitative performance of the proposed method:Smartwatch-1 (W2) versus Smartwatch-2 (W1)

Algorithm Data type Writing Mode EER

CNN-BiLSTM [4]

Own iPAD 26.67

Sentence Paper 32.78

Common iPAD 30.00

Sentence Paper 33.33

SignatureiPAD 46.67

Paper 43.34

Proposed Method

Own iPAD 2.58

Sentence Paper 3.33

Common iPAD 6.83

Sentence Paper 10.57

SignatureiPAD 6.89

Paper 9.54

graded performance. The loss can be primarily attributedto the device specific features which do not match exactlythe data from other device.

• The proposed method indicates improved performancewhen compared to that of the existing techniques un-der this scenario exemplifying the comparatively robustnature of proposed approach.

• The performance of the existing methods shows thedegraded performance on all three activities. However,the proposed method indicates the best performance onthe own sentence activity further supporting the robustnature of feature extraction and classification.

Table IV and Figure 5 presents the performance of both theproposed method and the existing method when data subject isenroled with a smartwatch (W2) and probed with a smartwatch(W1). Based on the obtained results we can note the similarobservations as before.

Based on the obtained results, we precisely answer theformulated research questions in this work:

Q1: Does the verification performance of a user vary withthe medium of writing (electronic tablet or paper)?We answer this with the experimental results obtainedon both paper and electronic mediums for writing. Ex-perimental results indicate the performance variations onboth Experiment-1 and Experiment-2. Even though theproposed method did not report the difference in perfor-mance with Experiment-1, the existing method shows thedegraded results with electronic medium with smartwatchW1 and W2 compared to paper medium. However, basedon the results from the Experiment-2, both existing andthe proposed method indicates the variation in perfor-mance with the writing mediums. Therefore, it can beconcluded that the medium of writing influences theperformance of user verification significantly.

Q2: Is the verification performance dependent on the typeor make of the wearable device (or smartwatch)?Based on the results obtained from Experiment-1 on indi-vidual smartwatches, it is clear that the user verificationperformance varies across the individual smartwatches,especially with the existing state-of-art. However, theproposed approach did not reflect the change in perfor-mance, indicating the need for robust feature extractionand classification approaches.Q3: Does the user verification performance vary whenthe user changes the type of the smartwatch?Based on Experiment-2, it can be noted that the veri-fication performance degrades with the cross-device (orsmartwatch), indicating the need for antagonistic deviceapproaches to handle cross-device verification. Alterna-tively, the re-enrolment/multiple-enrolment of user canbe conducted to overcome the performance loss in suchcross-smartwatch settings.Q4: Does data captured while signing suffice the veri-fication performance need over the free writing of shortsentences?Based on the results from Experiments-1 and 2, the useof own sentence indicates the improved performance oversignature, especially in cross-device verification scenario.While the approaches perform well under similar settingswith signature alone, self-chosen sentences help in distin-guishing the users in different settings such as differentmediums used to write on or different smartwatches.

V. CONCLUSION

This work has presented a novel technique for user ver-ification based on patterns captured while writing using asmartwatch. The proposed method is based on the deeprepresentation of the 2D-CWT features computed from themotion/accelerometer data using a smartwatch. We have alsointroduced a new dataset collected using two different smart-watches, capturing three different writing activities from 30unique data subjects. Extensive experiments are carried out onthe newly introduced data with various evaluation protocols.The performance of the proposed method is compared withthe current state-of-art. Obtained results demonstrated the bestperformance of the proposed method with EER = 0% when thesame smartphone is employed for all three writing activitiescaptured for two different smartwatches. The proposed methodalso indicates the best performance on the cross-smartwatchverification scenario for all three writing activities with scopefor improvement.

REFERENCES

[1] Microsoft Band, howpublished = https://www.microsoft.com/microsoft-band/en-us, note = Accessed: 2019-11-30.

[2] L. Arduser, P. Bissig, P. Brandes, and R. Wattenhofer. Recognizingtext using motion data from a smartwatch. In 2016 IEEE International

Conference on Pervasive Computing and Communication Workshops

(PerCom Workshops), pages 1–6, March 2016.[3] I. Griswold-Steiner, R. Matovu, and A. Serwadda. Handwriting watcher:

A mechanism for smartwatch-driven handwriting authentication. In 2017

IEEE International Joint Conference on Biometrics (IJCB), pages 216–224, Oct 2017.