Geneva, 25 November 2011

The NER model to assess accuracyin respeaking

Pablo Romero-Fresco (Roehampton University, CAIAC research centre)

Juan Martínez (Respeaking consultant)

ITU-T Workshop on“Telecommunications relay services for persons

with disabilities ”

(Geneva, 25 November 2011)

Accuracy in Respeaking

Quality in respeaking

Delay

Accuracy

Accuracy in Respeaking

97-98% accuracy

Basic requirements for a model

1) Functional and easy to apply

2) Include the basic principles of WER calculations in SR

3) Different programmes, different editing

4) Possibility of edited and yet accurate respeaking

5) Compare subtitles with original spoken text

6) Include other relevant info (delay, position, speed)

7) Provide both percentage and food for thought in training

Traditional WER methods

US National Institute of Standards and Technology

N - ErrorsAccuracy Rate ------------------------ × 100 = %

NBut...

Well, you know, you have to try

and put out a good performance,

I mean, yeah, it’s kind of a

stepping stone, isn’t it, really?

You have to try to put out a good

performance. It’s a stepping stone.

Traditional WER methods

US National Institute of Standards and Technology

N - ErrorsAccuracy Rate ------------------------ × 100 = 16%

NBut...

Well, you know, you have to try

and put out a good performance,

I mean, yeah, it’s kind of a

stepping stone, isn’t it, really?

You have to try to put out a good

performance. It’s a stepping stone.

Spain = SDH guidelines

Different European countries

UN Accessibility Focus Group

N – E – R Accuracy ------------------------ ×

100 = % N

Correct editions:Serious errors:Assessment:

NER Model

205 – 3 – 2 Accuracy ------------------------ × 100 = 98.6%

205

NER Model

226 – 13 – 1 Accuracy ------------------------ × 100 = 93.8%

226

Assessment: poor editing (not quantity, but quality)

NER Model

257 – 1 – 13Accuracy ------------------------ × 100 = 94.3%

257

Assessment: poor recognition (including serious mistakes)

WGBH: “There is a wide range of error types in real time captioning and they are not all equal in their impact to caption viewers”.

“Treating all errors the same does not provide a true picture of caption accuracy”.

Types of errors (feedback from DTV4ALL project)

1) “There are errors, yes, but you can easily figure out what the correct form was meant to be. Now I’m bilingual –I can speak English and teletext”

2) “Live subtitles? - Sound like gobbledygook to me”

3) “As far as I’m concerned they are not errors, but lies”

Types of errors (feedback from DTV4ALL project)

1) Minor edition or recognition errors (0.25)

2) Normal edition or recognition errors (0.5)

3) Serious errors (1)

Minor Errors

What a great goal by a Ryan Giggs!

Simon brown has been appointed new chairman of Rolls Royce.

For people are still missing following Sunday’s tornado.

Standard Errors

He’s a buy you a bull asset.

Is it really attend Tatian?

Serious errors

Public funding for universities has been cut by 15% this year.

He never talks dirty.

Serious errors

Public funding for universities has been cut by 15% this year.

He never talks dirty.

He never talks to Rudy.

NER MODEL

N – E – RAccuracy ------------------------ ×

100 = % N

Correct editions:Comments:

Target = 98%

NER Orange with apples

Pablo Romero-Fresco (p.romero-fresco@roehampton.ac.uk)

Graciñas

Geneva, 25 November 2011

The NER model to assess accuracyin respeaking

Pablo Romero-Fresco (Roehampton University, CAIAC research centre)(p.romero-fresco@roehampton.ac.uk)

Juan Martínez (Respeaking consultant)

ITU-T Workshop on“Telecommunications relay services for persons

with disabilities ”

(Geneva, 25 November 2011)