A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

A Pen Based Intelligent System for Educating Arabic Handwriting

Introduction

Previous Related Work

Background and Preliminaries

Arabic Handwritten Digits Recognition System

Arabic Handwritten Characters Recognition System

Intelligent Arab Teaching System

Conclusion and Future Work

Introduction

Problem Definition

ObjectivesHypothesis

Research Questions

Problem Definition

Research Questions

Traditional Learning

Complexity of Arabic characters.

Expertise needs in traditional recognition systems.

Traditional hand-crafted features.

Arabic handwritten characters mistakes.

Description Template character Sample character

Missing stroke error

Extra stroke error

Broken stroke error

Description Template character Sample character

Concatenated stroke error

Stroke order error

Direction Error

Problem Definition

Research Questions

The main goal of this work is to build and develop an intelligent

tutor system for detecting Arabic preschool children handwriting

difficulty based on immediate feedback.

The second goal of this work is to use deep learning architectures

to recognize Arabic handwritten characters and digits.

Problem Definition

Research Questions

Our hypothesis is that applying Convolutional neural networks and

stacked auto-encoder to classify Arabic handwritten characters and

digits.

We expect that a simple Convolutional neural network and stacked

Autoencoder will success to obtain competitive results.

We believe that implementing deep learning for this domain

will be moderately easy.

Problem Definition

Research Questions

Is training a deep learning architectures on handwritten characters

and digits better than computing numeric features from handwritten

characters and digits and training a simpler classifier ?

Is deep learning feasible with the resources we have ? Is there any

advantage to use GPU acceleration ?

Can we simplify the pipeline for Arabic handwritten characters and

digits recognition ?

Can preprocessing and features extraction be replaced by more

layers on deep learning ?

What are the best parameters for our deep learning architectures ?

What are the advantages of using a automatic feedback to

determine handwriting stroke mistakes for Arab children ?

Are deep learning architectures are a good option for future

research ?

Introduction

Arabic Handwritten

Digits Recognition

Arabic Handwritten

Characters Recognition

Intelligent

Tutoring Systems

Arabic Handwritten

Digits Recognition

Arabic Handwritten

Intelligent

Tutoring Systems

Authors Year Method Dataset Error Rate

Alwzwazy et al. 2016 CNN 46,612 4.3%

Kathirvalavakumarand Palaniappan

2015 K-NN 6670 1.3%

Takruri et al. 2014 SVM 3510 12%

AlKhateeb et al. 2014 DBN 70,000 14.74%

Majdi Salameh 2014 Fuzzy 2000 5%

CNN: Convolutional Neural Network SVM: Support Vector Machine DBN: Dynamic Bayesian Network

Pandi selvi andMeyyappan

2013 NN Samples 4%

Mahmoud 2008 SVM 211200.15% and

Melhaoui et al. 2011 CL 600 1%

NN: Neural Network SVM: Support Vector Machine CL: Characteristics Loci

Arabic Handwritten

Digits Recognition

Arabic Handwritten

Intelligent

Tutoring Systems

Hussien et al. 2015 HNN 8 22.75%

ElAdel et al. 2015 CNN 6000 6.08%

Elleuch et al. 2015 DBN 6600 2.10%

Shatnawi and Abdallah

2015 K-NN 1824 26.6%

CNN: Convolutional Neural Network HNN: Hopfild Neural Network DBN: Dynamic Bayesian Network

Kef et al. 2015FuzzyNN

3840 6.20%

Alabodi and Li 2014 GF 3840 6.70%

Lawgali et al. 2014DCTNN

6033 9.27%

NN: Neural Network DCT: Discrete Cosine Transform GF: geometrical features

Arabic Handwritten

Digits Recognition

Arabic Handwritten

Intelligent

Tutoring Systems

Authors Year Language Method

Will Tang et al. 2014 Chinese special relation matrix

H. Bezine & A. Alimi

2013 Arabic Attributed Relational Graph

Fork and Chan 2013 Latin Algorithms

Priyankara et al. 2013 Latin logical and spatial relationships

Hammadi et al. 2012 ArabicGraph matching

A* algorithm

Chea et al. 2012 Latin Chain code and direction code

Authors Year Language Method

Neo et. al. 2012 Latin Chain code and direction code

Chen et al. 2007 ChineseFeature extraction

spatial relationships

Hu et al. 2007 Chinesegraph matching technique

A* algorithm

Kai-Tai Tang et al.

2006 ChineseHungarian methodEuclidean distance

Tang and Leung 2006 Chinese Feature extraction techniques

Introduction

Pattern Recognition

Neural Network

Activation FunctionsDeep Learning

Stacked Autoencoder

Convolutional Neural Network

Intelligent Tutoring Systems

Neural Network

Stacked Autoencoder

Pattern Recognition

Pattern recognition is a branch of machine learning

Machine learning is divided into two main types:

supervised learning

unsupervised learning

Supervised learning learn from labeled training data

UnSupervised learning learn from unlabeled training data

Neural Network

Stacked Autoencoder

Pattern Recognition

Artificial neural networks or simply neural networks are one of the

most important nonlinear recognition classifiers used today.

Terminal Branches

of AxonDendrites

Neural Network

Stacked Autoencoder

Pattern Recognition

Common

nonlinear

activation

functions

Most deep networks use Rectified Linear Unit (ReLU)

ReLU trains much faster

RelU more expressive than logistic function

ReLU prevents the gradient vanishing problem.

Recognition

Neural Network

Stacked Autoencoder

Deep learning (DL) is a hierarchical structure network which through

simulates the human brain’s structure to extract the internal and

external input data’s features

Neural Network

Stacked Autoencoder

Pattern Recognition

The main component in Stacked Autoencoder is combined

Autoencoder. Autoencoder is a simple three-layer neural network

including an encoder and a decoder where output units are directly

connected back to input units.

In Autoencoder: the number of input units equal the number of

output units.

Hidden Layer Equation:

Sigmoid Equation:

Output Layer Equation:

The first sparse auto-encoder produce the primary feature.

The second sparse auto-encoder produce the secondary feature.

Soft-max classifier:

Proposed Stack Auto-Encoder architecture:

Neural Network

Stacked Autoencoder

Pattern Recognition

Convolution Neural Networks (CNN) is supervised learning and a family

of multi-layer neural networks particularly designed for use on two

dimensional data, such as images and videos.

A CNN consists of a number of layers:

Convolutional layers.

Pooling Layers.

Fully-Connected Layers.

Convolutional layer acts as a feature extractor that extracts features

of the inputs such as edges, corners , endpoints.

Feature extraction layer

Convolve with Activation

Kernel

features Feature extraction layer

The pooling layer reduces the resolution of the image that

reduce the precision of the translation (shift and distortion) effect.

ConvInput Pooling

fully connected layer have full

connections to all activations in the

previous layer.

Fully connect layer act as classifier.

Neural Network

Stacked Autoencoder

Pattern Recognition

Intelligent tutoring systems (ITS) are computational agents whose

purpose is to facilitate learning, usually without the help of a

human teacher.

ITS products can be organized on three categories:

Read Systems

Guided Systems

Immediate Error Detection Systems

Read systems are static not interactive systems; read-only because it

cannot provide the practice of writing.

The Guided systems allow children to practice writing in a guided

method and on-line.

The immediate error detection Systems gives access to the

children to practice free writing mode, and provide

immediately a feedback to indicate if there are any errors

in the writing.

Introduction

Arabic Handwritten Digit

Dataset

based on LeNet-5

Stacked Autoencoder

Optimized

Dataset

based on LeNet-5

Stacked Autoencoder

Optimized

The MADBase is a modified version of the ADBase

benchmark that has the same format as MNIST benchmark.

MADBase is composed of 70,000 digits written by 700 writers.

The databases is partitioned into two sets:

60,000 Training Data

10,000 Testing Data

Training Data

Testing Data

Dataset

based on LeNet-5

Stacked Autoencoder

Optimized

CCN based on LeNet-5 architecture was used with an 8 layers

including one input layer, one output layer, two Convolutional

layers and two sub-sampling, two fully connected layers as multi-

layer perceptron hidden layers for nonlinear classification.

The experiments outcomes was performed by MATLAB 2016a

programming environment.

Why MATLAB 2016b:

Neural Network Toolbox (contain deep leaning algorithms)

Statistics and Machine Learning Toolbox

Image processing Toolbox

Misclassification Rate

Confusion Matrix:

Dataset

based on LeNet-5

Stacked Autoencoder

Optimized

A Stacked Autoencoder (SAE) is a neural network consisting of

multiple layers of sparse Auto-encoders in which the outputs of each

layer is wired to the inputs of the successive layer.

Size of input layer is 784 x 60,000

Hidden layer for primary feature is 392

Train Autoencoder with 60,000 training set

Encode The training data with Autoencoder to produce the features

Encoder outcome is 392 x 60,000 features

The First Autoencoder:

Hidden layer for secondary feature is 196

Train Autoencoder with 60,000 features set

Encode The training features with Autoencoder to produce the features

Encoder outcome is 196 x 60,000 features

The Second Autoencoder:

Train a soft-max layer to classify the 196 x 60,000 feature vectors.

The soft-max layer is trained to produce 10 output class.

The Soft-max Classifier:

The proposed stacked Autoencoder produce ten output Arabic digits

Confusion Matrix

Feed Testing dataset to

proposed Stacked Autoencoder

Misclassification error is 2.2%

To produce better outcomes,

fine-tuning was used to update

all SAE parameters.

Feed Training dataset to

Feed Testing dataset to

Misclassification error is 1.5%

Authors Database Images Error Rate

Takruri et al. Private 3510 12%

AlKhateeb et al. ADBase 70000 14.74%

Majdi Salameh Fonts 2000 5%

Melhaoui et al. Private 600 1%

Pandi selvi & Meyyappan Private Samples 4%

Mahmoud Private 21120 0.15%

Kathirvalavakumar & Palaniappan Private 6670 1.3%

Alwzwazy et al. Private 46,612 4.3%

Our Approach MADBase 70000 1.5%

Dataset

based on LeNet-5

Stacked Autoencoder

Optimized

We built a new CNN architecture:

INPUT → CONV → RELU → Max-pooling → CONV → RELU →

Max-pooling → FC → RELU → FC → Output

The CNN architecture

Confusion Matrix

Error Rate= 0.8%

The total of wrong

classification is 85 from 10k.

Authors Database Images Error Rate

Takruri et al. Private 3510 12%

AlKhateeb et al. ADBase 70000 14.74%

Majdi Salameh Fonts 2000 5%

Melhaoui et al. Private 600 1%

Pandi selvi & Meyyappan Private Samples 4%

Mahmoud Private 21120 0.15%

Kathirvalavakumar & Palaniappan Private 6670 1.3%

Alwzwazy et al. Private 46,612 4.3%

Our Approach MADBase 70000 0.85%

Introduction

Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network

Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network

We collect a dataset that composed of 16,800 characters written by 60

participants, the age range is between 19 to 40 years.

The forms were scanned at the resolution of 300 dpi. Each block is

segmented automatically using Matlab 2016a to determining the

coordinates for each block.

The database is partitioned into two sets: a training set (13,440

characters to 480 images per class) and a test set (3,360 characters to

120 images per class).

Each participant wrote each

character (from ’alef’ to

’yeh’) ten times on two forms

The different shapes of some

Arabic characters

Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network

Train Autoencoder with 13,440 training

Encode The training data with

Autoencoder to produce the features

Encoder outcome is 512 x 13,440

featuresThe First Autoencoder

Train Autoencoder with 13,440 features

Encode The features data with

Autoencoder to produce the features

Encoder outcome is 256 x 13,440

features

The Second Autoencoder

Train a soft-max layer to classify the 256

x 13,440 feature vectors.

The soft-max layer is trained to produce

28 output class.

The Soft-max classifier

The proposed stacked Autoencoder

produce 28 output Arabic characters

Confusion Matrix

Error Rate= 36.0%

# of Misclassification

= 1208 from 3,360

Class 1 2 3 4 5 6 7

Arabic Character alef beh teh theh jeem hah khah

Correct Classification 111 86 60 64 67 66 65

Wrong Classification 9 34 60 56 53 54 55

Classification Accuracy 92.5% 71.7% 50% 53.3% 55.8% 55% 54.2%

Miss-Classification 7.5% 28.3% 50% 46.7% 44.2% 45% 45.8%

Class 8 9 10 11 12 13 14

Arabic Character dal thal reh zain seen sheen sad

Classification Accuracy 70.8% 62.5% 77.5% 64.2% 65.0% 63.3% 53.3%

Miss-Classification 29.2% 37.5% 22.5% 35.8% 35.0% 36.7% 46.7%

Class 15 16 17 18 19 20 21

Arabic Character dad tah zah ain ghain feh qaf

Class 22 23 24 25 26 27 28

Arabic Character kaf lam meem noon heh waw yeh

Arabic Handwritten

Characters DataSet

Stacked Autoencoder

Convolutional

Neural Network

The CNN architecture

Classification Matrix

Error Rate= 5.15%

Class 1 2 3 4 5 6 7

Arabic Character alef beh teh theh jeem hah khah

Classification Accuracy 100% 96.70% 91.70% 91.70% 95.80% 97.50% 93.30%

Class 8 9 10 11 12 13 14

Arabic Character dal thal reh zain seen sheen sad

Classification Accuracy 95.00% 91.70% 100% 87.50% 97.50% 95.80% 98.70%

Class 15 16 17 18 19 20 21

Arabic Character dad tah zah ain ghain feh qaf

Class 22 23 24 25 26 27 28

Arabic Character kaf lam meem noon heh waw yeh

The total of wrong

classification is 173 from

3,360.

Authors Database Images Accuracy Rate

Hussien et al. Private 8 Letters 77.25%

ElAdel et al. IESK-arDB 6000 93.92%

Elleuch et al. HACDB 6600 97.9%

Shatnawi and Abdallah Private 1824 73.4%

Kef et al. IFN/ENIT 3840 93.8%

Alabodi and Li IFN/ENIT 3840 93.3%

Lawgali et al. IFN/ENIT 6033 90.73%

Our Approach Private 16800 94.85%

Introduction

Interfaces

ArchitectureComponents

Database

Agents

Results

Interfaces

Database

Agents

Results

Interfaces

Database

Agents

Results

The AKT system implementation follows the MVC design pattern.

The Model–View–Controller (MVC) is a software architectural

pattern used to design a software system.

Controller

Database

Interfaces

Database

Agents

Results

We Develop two interfaces:

Children & Tutor Interface

Learning Interface

Interfaces

Database

Agents

Results

Interfaces

Database

Agents

Results

Intelligent Arab Teaching is a multi-agent system based on

three components:

Learning Agent

Feedback Agent

Evaluation Agent

Learning agent based on four components:

1) stroke number

2) stroke similarity

3) stroke order

4) stroke direction

Stroke Number Arabic characters

One ،وهح،د،ر،س،ص،ط،ع،ل،م،

Two أ،ب،ج،خ،ذ،ز،ض،ظ،غ،ف،ك،ن

Three ت،ق،ي

Four ث،ش

Basic strokeSimilar

charactersBasic stroke

Similar

characters

ٮ ب،ت،ث ط ط،ظ

ح ج،ح،خ ص ص،ض

د د،ذ ع ع،غ

ر ر،ز ٯ ف،ق

س س،ش ل ل،ك

Arabic Character Stroke #1

و و Stroke #2

أ ا ء

ب ب .

ج ح .

خ ح .

ذ د .

ز ر .

ط ـص ا

ض ص .

غ ع .

ف ٯ .

ك لـ ء

ن ں . Stroke #3

ت ٮ . .

ظ ـص ا .

ق ٯ . .

ي ى . . Stroke #4

ث ٮ . . .

ش س . . .

Feedback agent based on:

1) Stroke direction detector

2) Stroke order detector

Intelligent Arab

Teaching system

was used chain

code to encode

a movement.

Code Angle (θ) Direction

C0 355°<θ<5° Right

C1 5°<θ<85° Up Right

C2 85° <θ< 95° Up

C3 95°<θ< 175° Up Left

C4 175° <θ< 185° Left

C5 185° <θ< 265° Down Left

C6 265° <θ< 275° Down

C7 275° <θ< 355° Down Right

The freeman chain code algorithm is defined in the following three

steps:

Step 1: The absolute difference between y-axis

Step 2: The absolute difference between x-axis

Step 3: Calculate the angle

When the preschool children put his/her finger on touch

screen, the system detected the sequence of x−y point

coordinate.

When The children move his/her finger up, the system

store those sequence of points as stroke.

In this part, Intelligent Arab Teaching system indicates

the children level of understanding of learning

handwriting character concepts.

Intelligent Arab Teaching system use fuzzy logic to

evaluate Arabic children.

The Fuzzy Rules

Example

Fuzzy System & membership functions

Arabic Character Difficulty Membership function

Arabic Character Error Membership function

Time Consumed Membership Function

Arab Children Age Membership function

Children Evaluation Membership Function

Interfaces

Database

Agents

Results

Directional stroke error of

character Seen.

Stroke position of Arabic

character Teh.

Extra stroke error of

Arabic character Seen.

The Egyptian preschool children and their tutors from Benha city

were asked to use the Intelligent Arab Teaching system as part of

their learning of write Arabic alphabet.

200 questionnaires were collected from the educators & some

children in the experimental groups after the computerized training

program.

The results specify that educators rated the Intelligent Arab Teaching

very highly on acceptability for both likeability and ease of use.

1) Improve children handwritingاالطفالكتابةاداءتحسين

(a) No (b) Neutral (c) Yesال(أ)عادى(ب)نعم(ج)

2) Easy to learn Applicationالتطبيقتعلمسهولة

3) Easy to useاالستخدامسهولة

4) Interactive with kidsاالطفالمعمتفاعل

# Yes Neutral No

1 85% 10% 5%2 90% 5% 5%3 90% 5% 5%4 75% 10% 15%

Introduction

In this presentation, we have demonstrated the

effectiveness of deep learning for Arabic handwritten

Arabic characters and digits recognition.

Compared to other machine learning architectures, SAE

and CNN have better performance in both images and

big data of images.

In Arabic handwritten digits based on MADBase dataset:

We achieved misclassification error rates 12% using CNN

based on LeNet-5.

We achieved misclassification error rates 1.5% using SAE.

We achieved misclassification error rates 0.85% using CNN.

In Arabic handwritten characters based on our dataset:

We achieve 36% misclassification error rates using

We achieve 5.15% misclassification error rates using

An intelligent tutoring system for handwriting Education

was developed, called Intelligent Arab Teaching system.

The main purpose of Intelligent Arab Teaching is to help

Arab preschool children to diagnose their handwriting

mistakes.

Work on Arabic handwritten word recognition using deep

learning techniques.

Improving the performance of handwritten Arabic

character recognition.

Improving our Intelligent Arab Teaching system to detect

all types of the Arabic handwriting learning mistakes.

facebook.com/mloey

mohamedloey@gmail.com

twitter.com/mloey

linkedin.com/in/mloey

mloey@fci.bu.edu.eg

THANKS FOR YOUR TIME

A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

Education

Transcript of A Pen Based Intelligent System for Educating Arabic Handwriting Deep Learning

Impact of Handwriting Training on Handwriting Fluency ... · Impact of Handwriting Training on Handwriting Fluency, Spelling and Text Quality in Primary School Children ... Visual-motorIntegration

Handwriting Instruction

Handwriting without Tearssportwood.thevilla.org/files/2012/09/Handwriting-without...Handwriting without Tears Handwriting without Tears is our First Grade Handwriting curriculum. It

Better handwriting for adults...writing hand. 2 Grip the pen higher up so that you can see your writing. Hold it at least 2cm (about 2 finger-tips) away from the tip of the pen. 3

Speech handwriting

Pen Research Jay Pittman Development Lead Tablet PC Handwriting Recognition Microsoft Corporation Jay Pittman Development Lead Tablet PC Handwriting Recognition.

Julie A. Kientz Assistant Professor Technical ... · Overview of Cognitive Disabilities ... Record handwriting using Anoto digital pen to collect grades and timestamp end of trial.

Self starter… Check all marks and comments using a different colour pen make any necessary corrections. Answer questions * neatest handwriting Literacy.

Handwriting Worksheets

Handwriting Sheet

MANUSCRIPT PEN COMPANY ALL SAFETY DATA FOR MM6409 · Arian Farmer Manuscript Pen Company Ltd Highley Nr. Bridgbnorth Shropshire 6NN Richard Chadd HANDWRITING PENS S1002720LS H822

A Handwriting and Pen-Printing Classification System for ...

Handwriting Recognition for Genealogical Records · Handwriting Recognition • Two different fields: • Online Handwriting Recognition Writer's pen movements captured Velocity,

H WHISPER - Weeke Primary School€¦ · Riddles Easter Crafts . ... Friction Pen- These friction pens are handwriting savers! ... Electric Sharpener- Although expensive, ...

Large Vocabulary Unconstrained Handwriting Recognition J Subrahmonia Pen Technologies IBM T J Watson Research Center.

The NEXT STEP Welcome to Knutsford Academy€¦ · • Pencil case • Blue handwriting pen (and spare) – available from the Student Office for 20p • Pencil • Rubber • Pencil

Handwriting Recognition .

Teaching Handwriting

Handwriting Analysis

Handwriting model