Speech to Sign Language Interpreter System (SSLIS)

Khalid K. El-Darymli, Othman O. Khalifa and Hassan Enmsoah

Department of Computer Engineering, Faculty of EngineeringInternational Islamic University Malaysia, PO BOX 10, Kuala Lumpur, 50728, Malaysia

Phone: 03-6196-4433, Fax: 03-6196-4433, E-mail: wahyudi@iiu.edu.my

Sphinx 3.5 speech recognition

engine

ASL database

* Speech to text* Equivalent SE translation

Recognized textLive input speech

Figure 1: Flowchart depicts basic structure of SSLIS

Input constituents of recognized text separately

Is the input word a basic?

Is the basic word within the ASL video clips vocabulary?

YES

Extract the basic word out of the input word and temporally memorize

the word before extractionNO

ASL pre-recorded video clips

YES

Final Output

The American Manual Alphabet

NO

Equivalent ASL video clip

of input basic word

Fingerspelling of the input word

Is extracted word within ASL db

vocabulary?

NO

Depending on the type of temporally memorized word append a suitable

marker to its equivalent ASL video clip

YES

The temporally memorized word

Recognized text(The speech-recognition

engine's output)

Signal processing (FE)

Continuous input speech

Acoustic ModelP(A1,…,AT|p1,...pk)

DictionaryP(p1,…,pk|W)

Language ModelP(Wn|W1,W2,…,Wn-1)

Knowledge Base

Training

Extracted features vector

Hypothesis evaluation

Decoder

P(X|W)*P(W)

Search feedback

Decoding

Best Hypotheses

WBest

H={W1, W2, …,Wk}

X={x1, x2,…, xT}

Figure 2: Flowchart depicts simplified structure of Sphinx 3.5 speech engine

Regular past verbs:-edtalked, wanted, learned

Regular plural nouns: sbears, houses

3rd person singular: -swalks, eats, sings

Irregular past verbs:(sweep RH open B, tips

out, to the right )saw, heard, blew

Irregular plural nouns:(sign the word twice )

children, sheep, mice

mice

Possessive: -'scat's, daddy's, chair's

verb form: -ingclimbing, playing, running

Adjective: -ySleepy, sunny, cloudy

Adverb: -lyBeautifully, happily, nicely

Participle:Fallen, gone, grown

Comparative: -ersmaller, faster, longer

Superlative: -estSmallest, fastest ,

longest

Opposite of: un-, im-, in-, etc.

(made before the sign word, as a prefix )

unhappy, impatient, inconsiderate

Agent (person):(sign made near the body )

teacher, actor, artist

Agent (thing)(sign made away from the body )

washer, dryer , planter

Figure 3: Signed English markers

1. Research Goal and Objectives:Design and Manipulation of Speech to Sign Language Interpreter System. To fill the gap between deaf and nondeaf people in two senses. Firstly, by using this SW for educational purposes for deaf people and secondly, by facilitating the communication between deaf and nondeaf people.To increase independence and self-confidence of the deaf person.To increase opportunities for advancement and success in education, employment, personal relationships, and public access venues.To improve quality of life.

2. SSLIS Capabilities:Real time speech to text to video sign language.Text to video sign language.Automatic WER calculation.Text to computer generated voice with synchronized lips.Speed control of ASL movies in play.“Minimize to Auto” allows “drag and drop” from any text editor to be signed.Demonstration of SE manual as parallel to English.Demonstration of decoding process of speech.“Live Decode” Program allows real time speech recognition while “Live Pretend” and “Decode” allows speech recognition in batch mode.

3. ConclusionsThe research aim of offering freely available and open source SSLIS is fulfilled.Sphinx 3.5 was manipulated as the SR engine.Signed English manual was followed for translation.

4. Shortcomings and Further Work: Degradation in the speech recognition accuracy [an acoustic model generated from dictation tasks DB needs to be generated and employed],To highly improve the speech recognition accuracy, speech recognition engine can be adapted to different users through introducing enrollment session.

Figure 4: ASL alphabets

Figure 5: Demonstration of ASL in SSLIS

Figure 6: Some Snapshots of SSLIS