Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco...
-
Upload
xavier-harper -
Category
Documents
-
view
218 -
download
2
Transcript of Geneva, 25 November 2011 The NER model to assess accuracy in respeaking Pablo Romero-Fresco...
Geneva, 25 November 2011
The NER model to assess accuracyin respeaking
Pablo Romero-Fresco (Roehampton University, CAIAC research centre)
Juan Martínez (Respeaking consultant)
ITU-T Workshop on“Telecommunications relay services for persons
with disabilities ”
(Geneva, 25 November 2011)
Accuracy in Respeaking
Quality in respeaking
Delay
Accuracy
Accuracy in Respeaking
97-98% accuracy
Basic requirements for a model
1) Functional and easy to apply
2) Include the basic principles of WER calculations in SR
3) Different programmes, different editing
4) Possibility of edited and yet accurate respeaking
5) Compare subtitles with original spoken text
6) Include other relevant info (delay, position, speed)
7) Provide both percentage and food for thought in training
Traditional WER methods
US National Institute of Standards and Technology
N - ErrorsAccuracy Rate ------------------------ × 100 = %
NBut...
Well, you know, you have to try
and put out a good performance,
I mean, yeah, it’s kind of a
stepping stone, isn’t it, really?
You have to try to put out a good
performance. It’s a stepping stone.
Traditional WER methods
US National Institute of Standards and Technology
N - ErrorsAccuracy Rate ------------------------ × 100 = 16%
NBut...
Well, you know, you have to try
and put out a good performance,
I mean, yeah, it’s kind of a
stepping stone, isn’t it, really?
You have to try to put out a good
performance. It’s a stepping stone.
Spain = SDH guidelines
Different European countries
UN Accessibility Focus Group
N – E – R Accuracy ------------------------ ×
100 = % N
Correct editions:Serious errors:Assessment:
NER Model
205 – 3 – 2 Accuracy ------------------------ × 100 = 98.6%
205
NER Model
226 – 13 – 1 Accuracy ------------------------ × 100 = 93.8%
226
Assessment: poor editing (not quantity, but quality)
NER Model
257 – 1 – 13Accuracy ------------------------ × 100 = 94.3%
257
Assessment: poor recognition (including serious mistakes)
WGBH: “There is a wide range of error types in real time captioning and they are not all equal in their impact to caption viewers”.
“Treating all errors the same does not provide a true picture of caption accuracy”.
Types of errors (feedback from DTV4ALL project)
1) “There are errors, yes, but you can easily figure out what the correct form was meant to be. Now I’m bilingual –I can speak English and teletext”
2) “Live subtitles? - Sound like gobbledygook to me”
3) “As far as I’m concerned they are not errors, but lies”
Types of errors (feedback from DTV4ALL project)
1) Minor edition or recognition errors (0.25)
2) Normal edition or recognition errors (0.5)
3) Serious errors (1)
Minor Errors
What a great goal by a Ryan Giggs!
Simon brown has been appointed new chairman of Rolls Royce.
For people are still missing following Sunday’s tornado.
Standard Errors
He’s a buy you a bull asset.
Is it really attend Tatian?
Serious errors
Public funding for universities has been cut by 15% this year.
He never talks dirty.
Serious errors
Public funding for universities has been cut by 15% this year.
He never talks dirty.
He never talks to Rudy.
NER MODEL
N – E – RAccuracy ------------------------ ×
100 = % N
Correct editions:Comments:
Target = 98%
NER Orange with apples
Pablo Romero-Fresco ([email protected])
Graciñas
Geneva, 25 November 2011
The NER model to assess accuracyin respeaking
Pablo Romero-Fresco (Roehampton University, CAIAC research centre)([email protected])
Juan Martínez (Respeaking consultant)
ITU-T Workshop on“Telecommunications relay services for persons
with disabilities ”
(Geneva, 25 November 2011)