A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

A Pen Based Intelligent System for Educating Arabic Handwriting



Introduction

Previous Related Work

Background and Preliminaries

Arabic Handwritten Digits Recognition System

Arabic Handwritten Characters Recognition System

Intelligent Arab Teaching System

Conclusion and Future Work


Problem Definition

ObjectivesHypothesis

Research Questions


Traditional Learning


Complexity of Arabic characters.

Expertise needs in traditional recognition systems.

Traditional hand-crafted features.

Arabic handwritten characters mistakes.


Description Template character Sample character

Missing stroke error

Extra stroke error

Broken stroke error


Description Template character Sample character

Concatenated stroke error

Stroke order error

Direction Error


Problem Definition


Research Questions


The main goal of this work is to build and develop an intelligent

tutor system for detecting Arabic preschool children handwriting

difficulty based on immediate feedback.

The second goal of this work is to use deep learning architectures

to recognize Arabic handwritten characters and digits.


Problem Definition


Research Questions


Our hypothesis is that applying Convolutional neural networks and

stacked auto-encoder to classify Arabic handwritten characters and

digits.

We expect that a simple Convolutional neural network and stacked

Autoencoder will success to obtain competitive results.

We believe that implementing deep learning for this domain

will be moderately easy.


Problem Definition


Research Questions


Is training a deep learning architectures on handwritten characters

and digits better than computing numeric features from handwritten

characters and digits and training a simpler classifier ?

Is deep learning feasible with the resources we have ? Is there any

advantage to use GPU acceleration ?

Can we simplify the pipeline for Arabic handwritten characters and

digits recognition ?


Can preprocessing and features extraction be replaced by more

layers on deep learning ?

What are the best parameters for our deep learning architectures ?

What are the advantages of using a automatic feedback to

determine handwriting stroke mistakes for Arab children ?

Are deep learning architectures are a good option for future

research ?


Introduction








Arabic Handwritten

Digits Recognition

Arabic Handwritten

Characters Recognition

Intelligent

Tutoring Systems


Authors Year Method Dataset Error Rate

Alwzwazy et al. 2016 CNN 46,612 4.3%

Kathirvalavakumarand Palaniappan

2015 K-NN 6670 1.3%

Takruri et al. 2014 SVM 3510 12%

AlKhateeb et al. 2014 DBN 70,000 14.74%

Majdi Salameh 2014 Fuzzy 2000 5%

CNN: Convolutional Neural Network SVM: Support Vector Machine DBN: Dynamic Bayesian Network



Pandi selvi andMeyyappan

2013 NN Samples 4%

Mahmoud 2008 SVM 211200.15% and

2.16%

Melhaoui et al. 2011 CL 600 1%

NN: Neural Network SVM: Support Vector Machine CL: Characteristics Loci


Arabic Handwritten

Digits Recognition

Arabic Handwritten


Intelligent

Tutoring Systems



Hussien et al. 2015 HNN 8 22.75%

ElAdel et al. 2015 CNN 6000 6.08%

Elleuch et al. 2015 DBN 6600 2.10%

Shatnawi and Abdallah

2015 K-NN 1824 26.6%

CNN: Convolutional Neural Network HNN: Hopfild Neural Network DBN: Dynamic Bayesian Network



Kef et al. 2015FuzzyNN

3840 6.20%

Alabodi and Li 2014 GF 3840 6.70%

Lawgali et al. 2014DCTNN

6033 9.27%

NN: Neural Network DCT: Discrete Cosine Transform GF: geometrical features


Arabic Handwritten

Digits Recognition

Arabic Handwritten


Intelligent

Tutoring Systems


Authors Year Language Method

Will Tang et al. 2014 Chinese special relation matrix

H. Bezine & A. Alimi

2013 Arabic Attributed Relational Graph

Fork and Chan 2013 Latin Algorithms

Priyankara et al. 2013 Latin logical and spatial relationships

Hammadi et al. 2012 ArabicGraph matching

A* algorithm

Chea et al. 2012 Latin Chain code and direction code


Authors Year Language Method

Neo et. al. 2012 Latin Chain code and direction code

Chen et al. 2007 ChineseFeature extraction

spatial relationships

Hu et al. 2007 Chinesegraph matching technique

A* algorithm

Kai-Tai Tang et al.

2006 ChineseHungarian methodEuclidean distance

Tang and Leung 2006 Chinese Feature extraction techniques


Introduction








Pattern Recognition

Neural Network

Activation FunctionsDeep Learning

Stacked Autoencoder

Convolutional Neural Network

Intelligent Tutoring Systems


Neural Network


Stacked Autoencoder



Pattern Recognition


Pattern recognition is a branch of machine learning

Machine learning is divided into two main types:

supervised learning

unsupervised learning

Supervised learning learn from labeled training data

UnSupervised learning learn from unlabeled training data


Neural Network


Stacked Autoencoder



Pattern Recognition


Artificial neural networks or simply neural networks are one of the

most important nonlinear recognition classifiers used today.

Axon

Terminal Branches

of AxonDendrites

S

x1

x2

w1

w2

wn

xn

x3 w3


Neural Network


Stacked Autoencoder



Pattern Recognition


Common

nonlinear

activation

functions


Most deep networks use Rectified Linear Unit (ReLU)

ReLU trains much faster

RelU more expressive than logistic function

ReLU prevents the gradient vanishing problem.


Recognition

Neural Network


Stacked Autoencoder




Deep learning (DL) is a hierarchical structure network which through

simulates the human brain’s structure to extract the internal and

external input data’s features


Neural Network


Stacked Autoencoder



Pattern Recognition


The main component in Stacked Autoencoder is combined

Autoencoder. Autoencoder is a simple three-layer neural network

including an encoder and a decoder where output units are directly

connected back to input units.

In Autoencoder: the number of input units equal the number of

output units.


Hidden Layer Equation:

Sigmoid Equation:

Output Layer Equation:


The first sparse auto-encoder produce the primary feature.


The second sparse auto-encoder produce the secondary feature.


Soft-max classifier:


Proposed Stack Auto-Encoder architecture:


Neural Network


Stacked Autoencoder



Pattern Recognition


Convolution Neural Networks (CNN) is supervised learning and a family

of multi-layer neural networks particularly designed for use on two

dimensional data, such as images and videos.

A CNN consists of a number of layers:

Convolutional layers.

Pooling Layers.

Fully-Connected Layers.


Convolutional layer acts as a feature extractor that extracts features

of the inputs such as edges, corners , endpoints.


Feature extraction layer

10-1

10-1

10-1

Convolve with Activation

Kernel


features Feature extraction layer


The pooling layer reduces the resolution of the image that

reduce the precision of the translation (shift and distortion) effect.


ConvInput Pooling


fully connected layer have full

connections to all activations in the

previous layer.

Fully connect layer act as classifier.


Neural Network


Stacked Autoencoder



Pattern Recognition


Intelligent tutoring systems (ITS) are computational agents whose

purpose is to facilitate learning, usually without the help of a

human teacher.

ITS products can be organized on three categories:

Read Systems

Guided Systems

Immediate Error Detection Systems


Read systems are static not interactive systems; read-only because it

cannot provide the practice of writing.


The Guided systems allow children to practice writing in a guided

method and on-line.


The immediate error detection Systems gives access to the

children to practice free writing mode, and provide

immediately a feedback to indicate if there are any errors

in the writing.


Introduction








Arabic Handwritten Digit

Dataset


based on LeNet-5

Stacked Autoencoder


Optimized


The MADBase is a modified version of the ADBase

benchmark that has the same format as MNIST benchmark.

MADBase is composed of 70,000 digits written by 700 writers.

The databases is partitioned into two sets:

60,000 Training Data

10,000 Testing Data


Training Data

Testing Data



Dataset


based on LeNet-5

Stacked Autoencoder


Optimized


CCN based on LeNet-5 architecture was used with an 8 layers

including one input layer, one output layer, two Convolutional

layers and two sub-sampling, two fully connected layers as multi-

layer perceptron hidden layers for nonlinear classification.


The experiments outcomes was performed by MATLAB 2016a

programming environment.

Why MATLAB 2016b:

Neural Network Toolbox (contain deep leaning algorithms)

Statistics and Machine Learning Toolbox

Image processing Toolbox


Misclassification Rate


Confusion Matrix:



Dataset


based on LeNet-5

Stacked Autoencoder


Optimized


A Stacked Autoencoder (SAE) is a neural network consisting of

multiple layers of sparse Auto-encoders in which the outputs of each

layer is wired to the inputs of the successive layer.


Size of input layer is 784 x 60,000

Hidden layer for primary feature is 392

Train Autoencoder with 60,000 training set

Encode The training data with Autoencoder to produce the features

Encoder outcome is 392 x 60,000 features

The First Autoencoder:



Hidden layer for secondary feature is 196

Train Autoencoder with 60,000 features set

Encode The training features with Autoencoder to produce the features

Encoder outcome is 196 x 60,000 features

The Second Autoencoder:


Train a soft-max layer to classify the 196 x 60,000 feature vectors.

The soft-max layer is trained to produce 10 output class.

The Soft-max Classifier:


The proposed stacked Autoencoder produce ten output Arabic digits


Confusion Matrix

Feed Testing dataset to

proposed Stacked Autoencoder

Misclassification error is 2.2%


To produce better outcomes,

fine-tuning was used to update

all SAE parameters.

Feed Training dataset to


Feed Testing dataset to


Misclassification error is 1.5%


Authors Database Images Error Rate

Takruri et al. Private 3510 12%

AlKhateeb et al. ADBase 70000 14.74%

Majdi Salameh Fonts 2000 5%

Melhaoui et al. Private 600 1%

Pandi selvi & Meyyappan Private Samples 4%

Mahmoud Private 21120 0.15%

Kathirvalavakumar & Palaniappan Private 6670 1.3%

Alwzwazy et al. Private 46,612 4.3%

Our Approach MADBase 70000 1.5%



Dataset


based on LeNet-5

Stacked Autoencoder


Optimized


We built a new CNN architecture:

INPUT → CONV → RELU → Max-pooling → CONV → RELU →

Max-pooling → FC → RELU → FC → Output


The CNN architecture


Confusion Matrix

Error Rate= 0.8%


The total of wrong

classification is 85 from 10k.


Authors Database Images Error Rate

Takruri et al. Private 3510 12%

AlKhateeb et al. ADBase 70000 14.74%

Majdi Salameh Fonts 2000 5%

Melhaoui et al. Private 600 1%

Pandi selvi & Meyyappan Private Samples 4%

Mahmoud Private 21120 0.15%

Kathirvalavakumar & Palaniappan Private 6670 1.3%

Alwzwazy et al. Private 46,612 4.3%

Our Approach MADBase 70000 0.85%


Introduction








Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network


We collect a dataset that composed of 16,800 characters written by 60

participants, the age range is between 19 to 40 years.

The forms were scanned at the resolution of 300 dpi. Each block is

segmented automatically using Matlab 2016a to determining the

coordinates for each block.

The database is partitioned into two sets: a training set (13,440

characters to 480 images per class) and a test set (3,360 characters to

120 images per class).


Each participant wrote each

character (from ’alef’ to

’yeh’) ten times on two forms


The different shapes of some

Arabic characters


Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network




Train Autoencoder with 13,440 training

set

Encode The training data with

Autoencoder to produce the features

Encoder outcome is 512 x 13,440

featuresThe First Autoencoder




Train Autoencoder with 13,440 features

Encode The features data with

Autoencoder to produce the features

Encoder outcome is 256 x 13,440

features

The Second Autoencoder


Train a soft-max layer to classify the 256

x 13,440 feature vectors.

The soft-max layer is trained to produce

28 output class.

The Soft-max classifier


The proposed stacked Autoencoder

produce 28 output Arabic characters


Confusion Matrix

Error Rate= 36.0%

# of Misclassification

= 1208 from 3,360

Class 1 2 3 4 5 6 7

Arabic Character alef beh teh theh jeem hah khah

Correct Classification 111 86 60 64 67 66 65

Wrong Classification 9 34 60 56 53 54 55

Classification Accuracy 92.5% 71.7% 50% 53.3% 55.8% 55% 54.2%

Miss-Classification 7.5% 28.3% 50% 46.7% 44.2% 45% 45.8%

Class 8 9 10 11 12 13 14

Arabic Character dal thal reh zain seen sheen sad



Classification Accuracy 70.8% 62.5% 77.5% 64.2% 65.0% 63.3% 53.3%

Miss-Classification 29.2% 37.5% 22.5% 35.8% 35.0% 36.7% 46.7%

Class 15 16 17 18 19 20 21

Arabic Character dad tah zah ain ghain feh qaf





Class 22 23 24 25 26 27 28

Arabic Character kaf lam meem noon heh waw yeh






Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network


The CNN architecture


Classification Matrix

Error Rate= 5.15%

Class 1 2 3 4 5 6 7

Arabic Character alef beh teh theh jeem hah khah



Classification Accuracy 100% 96.70% 91.70% 91.70% 95.80% 97.50% 93.30%


Class 8 9 10 11 12 13 14

Arabic Character dal thal reh zain seen sheen sad



Classification Accuracy 95.00% 91.70% 100% 87.50% 97.50% 95.80% 98.70%


Class 15 16 17 18 19 20 21

Arabic Character dad tah zah ain ghain feh qaf





Class 22 23 24 25 26 27 28

Arabic Character kaf lam meem noon heh waw yeh






The total of wrong

classification is 173 from

3,360.


Authors Database Images Accuracy Rate

Hussien et al. Private 8 Letters 77.25%

ElAdel et al. IESK-arDB 6000 93.92%

Elleuch et al. HACDB 6600 97.9%

Shatnawi and Abdallah Private 1824 73.4%

Kef et al. IFN/ENIT 3840 93.8%

Alabodi and Li IFN/ENIT 3840 93.3%

Lawgali et al. IFN/ENIT 6033 90.73%

Our Approach Private 16800 94.85%


Introduction








Interfaces

ArchitectureComponents

Database

Agents

Results


Interfaces


Database

Agents

Results


The AKT system implementation follows the MVC design pattern.

The Model–View–Controller (MVC) is a software architectural

pattern used to design a software system.

View

Controller

Model

Database


Interfaces


Database

Agents

Results


We Develop two interfaces:

Children & Tutor Interface

Learning Interface


Interfaces


Database

Agents

Results


Intelligent Arab Teaching is a multi-agent system based on

three components:

Learning Agent

Feedback Agent

Evaluation Agent


Learning agent based on four components:

1) stroke number

2) stroke similarity

3) stroke order

4) stroke direction


Stroke Number Arabic characters

One ،وهح،د،ر،س،ص،ط،ع،ل،م،

Two أ،ب،ج،خ،ذ،ز،ض،ظ،غ،ف،ك،ن

Three ت،ق،ي

Four ث،ش


Basic strokeSimilar

charactersBasic stroke

Similar

characters

ٮ ب،ت،ث ط ط،ظ

ح ج،ح،خ ص ص،ض

د د،ذ ع ع،غ

ر ر،ز ٯ ف،ق

س س،ش ل ل،ك


Arabic Character Stroke #1

ح ح

د د

ر ر

س س

ص ص

ع ع

ل ل

م م

ه ه

و و Stroke #2

أ ا ء

ب ب .

ج ح .

خ ح .

ذ د .

ز ر .

ط ـص ا

ض ص .

غ ع .

ف ٯ .

ك لـ ء

ن ں . Stroke #3

ت ٮ . .

ظ ـص ا .

ق ٯ . .

ي ى . . Stroke #4

ث ٮ . . .

ش س . . .


Feedback agent based on:

1) Stroke direction detector

2) Stroke order detector


Intelligent Arab

Teaching system

was used chain

code to encode

a movement.

Code Angle (θ) Direction

C0 355°<θ<5° Right

C1 5°<θ<85° Up Right

C2 85° <θ< 95° Up

C3 95°<θ< 175° Up Left

C4 175° <θ< 185° Left

C5 185° <θ< 265° Down Left

C6 265° <θ< 275° Down

C7 275° <θ< 355° Down Right


The freeman chain code algorithm is defined in the following three

steps:

Step 1: The absolute difference between y-axis

Step 2: The absolute difference between x-axis

Step 3: Calculate the angle


When the preschool children put his/her finger on touch

screen, the system detected the sequence of x−y point

coordinate.

When The children move his/her finger up, the system

store those sequence of points as stroke.


In this part, Intelligent Arab Teaching system indicates

the children level of understanding of learning

handwriting character concepts.

Intelligent Arab Teaching system use fuzzy logic to

evaluate Arabic children.


The Fuzzy Rules

Example


Fuzzy System & membership functions


Arabic Character Difficulty Membership function


Arabic Character Error Membership function


Time Consumed Membership Function


Arab Children Age Membership function


Children Evaluation Membership Function


Interfaces


Database

Agents

Results


Directional stroke error of

character Seen.


Stroke position of Arabic

character Teh.


Extra stroke error of

Arabic character Seen.


The Egyptian preschool children and their tutors from Benha city

were asked to use the Intelligent Arab Teaching system as part of

their learning of write Arabic alphabet.

200 questionnaires were collected from the educators & some

children in the experimental groups after the computerized training

program.

The results specify that educators rated the Intelligent Arab Teaching

very highly on acceptability for both likeability and ease of use.


1) Improve children handwritingاالطفالكتابةاداءتحسين

(a) No (b) Neutral (c) Yesال(أ)عادى(ب)نعم(ج)

2) Easy to learn Applicationالتطبيقتعلمسهولة


3) Easy to useاالستخدامسهولة


4) Interactive with kidsاالطفالمعمتفاعل


# Yes Neutral No

1 85% 10% 5%2 90% 5% 5%3 90% 5% 5%4 75% 10% 15%


Introduction








In this presentation, we have demonstrated the

effectiveness of deep learning for Arabic handwritten

Arabic characters and digits recognition.

Compared to other machine learning architectures, SAE

and CNN have better performance in both images and

big data of images.


In Arabic handwritten digits based on MADBase dataset:

We achieved misclassification error rates 12% using CNN

based on LeNet-5.

We achieved misclassification error rates 1.5% using SAE.

We achieved misclassification error rates 0.85% using CNN.


In Arabic handwritten characters based on our dataset:

We achieve 36% misclassification error rates using

SAE.

We achieve 5.15% misclassification error rates using

CNN.


An intelligent tutoring system for handwriting Education

was developed, called Intelligent Arab Teaching system.

The main purpose of Intelligent Arab Teaching is to help

Arab preschool children to diagnose their handwriting

mistakes.


Work on Arabic handwritten word recognition using deep

learning techniques.

Improving the performance of handwritten Arabic

character recognition.

Improving our Intelligent Arab Teaching system to detect

all types of the Arabic handwriting learning mistakes.


facebook.com/mloey

[email protected]

twitter.com/mloey

linkedin.com/in/mloey

[email protected]

A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

Education

Transcript of A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning