Translectures
49
FP7-ICT-2011-7 - Language technologies (STREP) Duration: 36 months (1 Nov 2011 – 31 Oct 2014) No Name Short Country 1 UNIVERSITAT POLITECNICA DE VALENCIA UPVLC Spain 2 XEROX SAS XEROX France 3 INSTITUT JOZEF STEFAN JSI Slovenia 3+ KNOWLEDGE FOR ALL FOUNDATION K4A UK 4 RHEINISCH-WESTFAELISCHE TECHNISCHE HOCHSCHULE AACHEN RWTH Germany 5 EUROPEAN MEDIA LABORATORY GMBH EML Germany 6 Deluxe Digital Studios Ltd DDS UK
-
Upload
the-open-education-consortium -
Category
Education
-
view
164 -
download
0
description
Online collections of video material are fast becoming a staple feature of the Internet and a key educational resource. What we are working on at transLectures is a set of easy-to-use tools that will allow users to add multilingual subtitles to these videos. In doing so, they will make the content of these videos available to a much wider audience in a way that is cost-effective and sustainable over the vast collections of online video lectures being generated. Automatic transcription tools will provide verbatim subtitles of the talks recorded on video, thereby allowing the hard-of-hearing to access this content. Language learners and other non-native speakers will also benefit from these monolingual subtitles. Meanwhile, machine translation tools will make these subtitles available in languages other than that in which the video was recorded. Specifically, we will be developing tools for use on VideoLectures.NET, a collection of videos recorded at various academic events set up by JSI’s Centre for Knowledge Transfer, and for poliMedia, a lecture capture system designed and implemented at the UPVLC. Our tools will also be fully compatible with Matterhorn, a free, open-source platform for the management of educational audio and video content. The language pairs being targeted in this project are English, Spanish and Slovenian for transcription, and EnglishSpanish, EnglishSlovenian, English>French and English>German for translation. http://www.translectures.eu
Transcript of Translectures
FP7-ICT-2011-7 - Language technologies (STREP) Duration: 36 months (1 Nov 2011 – 31 Oct 2014)
No Name Short Country
1 UNIVERSITAT POLITECNICA DE VALENCIA UPVLC Spain
2 XEROX SAS XEROX France
3 INSTITUT JOZEF STEFAN JSI Slovenia
3+ KNOWLEDGE FOR ALL FOUNDATION K4A UK
4 RHEINISCH-WESTFAELISCHE TECHNISCHE HOCHSCHULE AACHEN
RWTH Germany
5 EUROPEAN MEDIA LABORATORY GMBH EML Germany
6 Deluxe Digital Studios Ltd DDS UK
Outline of the talk
• Motivation
• Goal and technology challenges
• Demo videos: project overview, intelligent interaction and Matterhorn integration
• Transcriptions and translations at the pilots
• Massive adaptation
• Intelligent interaction with users
• Integration and evaluation
• Large on-line repositories of video lectures are being established: VideoLectures.NET, poliMedia, OCW and Opencast Matterhorn-based repositories, etc.
• Video lectures are neither transcribed nor translated due the lack of cost-effective tools.
• Transcriptions and translations are needed to make them accessible to speakers of different languages and people with disabilities. They would also facilitate further search and analysis functions.
• Starting hypothesis: current ASR and MT techniques are not far from achieving acceptable results in educational repositories.
Motivation
Goal and technology challenges
Goal: To develop innovative, cost-effective solutions to produce accurate transcriptions and translations in VideoLectures.NET, with generality across other Matterhorn-related repositories. Technology challenges: • Improvement of transcription and translation quality by
massive adaptation. • Improvement of transcription and translation quality by
intelligent interaction. • Integration into Matterhorn to enable real-life evaluation.
Demo videos
• Project overview
https://www.translectures.eu/project-summary
• Intelligent interaction with users
https://www.translectures.eu/web/progress
• Matterhorn integration
https://www.translectures.eu/web/progress
Transcriptions and translations at the pilots
Language Nb. of lectures Nb. of hours
VideoLectures.NET
English 9148 5900
Slovenian 741 490
Total 9889 6390
poliMedia (live)
Spanish 7554 996
English 420 62
Catalan 91 11
Total 8065 1069
VideoLectures.NET: EnEs, EnFr, EnDe, EnSl, SlEn
poliMedia: EsEn
Massive adaptation
• Massive adaptation of acoustic models
• Massive adaptation of language models
• Massive adaptation of translation models
10
15
20
25
30
35
40
45
50
M06 M12 M18 M24 M30
WER (ASR)
Sl
Ca
En
Es 15
20
25
30
35
40
M12 M18 M24 M30
BLEU (MT)
EnEsEnFr
EsEn
EnDe
SlEn
EnSl
Intelligent interaction with users
• Fast constrained search
• Intelligent interaction for transcription
• Intelligent interaction for translation
• Incremental training for translation
Transcription of 23 videos (3h) with no interaction
1
1 2
2
1 2 3
3
1 2 3 4 5
4
1 2 3 4 5
5
1 2 3 4 5
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
Transcription of 23 videos (3h) with no interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER18.8
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
18.4 BI
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
18.0 BI
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
17.6 BI
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
17.2 BI
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.8 BI
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.8 BI
Transcription of 23 videos (3h) with batch interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
18.5 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
18.1 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
18.0 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
17.5 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI17.3 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI17.3 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI17.2 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI17.0 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI16.5 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI16.4 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI16.0 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI15.7 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI15.7 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI15.6 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
15.4 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
15.2 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
15.1 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
14.5 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
14.5 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
14.3 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
14.3 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
14.0 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
13.6 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
13.6 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
13.4 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
13.1 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
13.0 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
12.6 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
12.6 II
Transcription of 23 videos (3h) with intelligent interaction
1
1
CC
2
CC
2
1
CC
2
CC
3
CC
3
1
CC
2
CC
3
CC
4
CC
5
CC
4
1
CC
2
CC
3
CC
4
CC
5
CC
5
1
CC
2
CC
3
CC
4
CC
5
CC
12
13
14
15
16
17
18
19
0 3 6 9 12 15 18Supervised minutes
WER
16.5 BI
12.4 II
Integration and evaluation
• Integration and evaluation at the pilots
• Try our tools and transcribe your videos
www.translectures.eu
Free service opened by UPV on 13 March 2014
