1

Evaluating the Effect of Predicting Oral Reading Miscues

Satanjeev Banerjee, Joseph Beck, Jack MostowProject LISTEN (www.cs.cmu.edu/~listen)

Carnegie Mellon University

Funding: NSF IERI

2

Why Predict Miscues?

• Reading Tutor helps children learn to read.• Speech recognizer listens for miscues (reading errors)

– E.g.: listen for “hat” if sentence to be read has word “hate”

• Accurate miscue prediction helps miscue detection.

3

Real Word Substitutions

• Miscues = substitutions, omissions, insertions

• Real word substitution = misread target word as another word– E.g. read “hat” instead of “hate”

• Most miscues are real word substitutions

• ICSLP-02: predicted real word substitutions

• Here: evaluate effect on substitution detection

4

How Evaluate Substitution Detection?

What child said I hat having milk every a

Correct text I hate having milk every day

What ASR heard I hate having mill every hate

substitution undetected false alarm substitution detected

substitution

1

2Substitution detection rate =

# substitutions detected

# substitutions child made=

1

4False alarm rate =

# false alarms

# words correctly read=

5

Evaluation Data

• Sentences read by 25 children aged 6 to 10

Correctly read Incorrectly read

Content tokens

5,981 335

Function tokens

3,166 147

6

Rote Method• Uses the University of Colorado miscue database.• For each target word

– Sort substitutions by # children who made them.

– Predict that the top n substitutions will reoccur, for this word.

Model typeSubstitution

detection rateFalse alarm

rate

No predicted substitutions

21.58 % 2.42 %

Top 1 22.82 % 2.88 %

Top 2 24.90 % 3.27 %

7

Extrapolative Method• Predict the probability that a word is a likely

substitution for another word– Pr ( substitution “hat” | target “hate”)

• Use machine learning to induce a classifier• Train using University of Colorado miscue database.

Some features (more in paper)

Candidate substitution

Target word

Spelling edit distance = 1 H.A.T.E H.A.T

Phonetic distance = 1 /HH EY T/ /HH AE T/

Rank in descending sorted frequency table

1364 1887

8

Extrapolative Method cont’d

Given a target word, predict substitution if

Pr ( substitution candidate | target word ) > threshold

Model typeSubstitution

detection rateFalse alarm

rate

No predicted substitutions

21.58 % 2.42 %

Pr >= 0.99 23.03 % 2.77 %

Pr >= 0.95 24.48 % 3.67 %

9

Combining Rote and Extrapolative

• Aim: Get n substitutions for a given word.

• Step 1: Use top n substitutions from rote.

• Step 2: If rote predicts k substitutions, k < n, – Then add top n – k substitutions from extrapolative.

• Intuition: rote is more accurate, so use when available. If not available, fall back on extrapolative.

10

Results from Combining Algorithms

Model typeSubstitution detection rate

False alarm rate

Top 1 25.73 % 3.77 %

Truncation 24.69 % 4.29 %

Top 2 31.54 % 4.62 %

Theoretical max

42.53 % 2.90 %

Truncation = The first 2 to n-2 phonemes of a word – models false starts. [/K AE/ and /K AE N/ for /K AE N D IY/; none for “hate”]

Theoretical max = use only those miscues the child actually made.

11

Conclusion

• Evaluated effect on substitution detection of – Two previously published algorithms– A combination of the two algorithms.

• Combined approach improved on current configuration (truncations) by– Reducing false alarms by 0.52% abs (12% rel)– Increasing miscue detection by 1.04% (4.2% rel)

• Take-home sound byte: Listening for specific reading mistakes can help detect them!