Bootstrap TDNN for Classification of Voiced Stop Consonants (B,D,G) CAIP, Rutgers University Oct.13,...
-
Upload
amy-horton -
Category
Documents
-
view
217 -
download
0
Transcript of Bootstrap TDNN for Classification of Voiced Stop Consonants (B,D,G) CAIP, Rutgers University Oct.13,...
Bootstrap TDNN for Classification of Voiced Stop
Consonants (B,D,G)
CAIP, Rutgers University
Oct.13, 2006
Jun Hou, Lawrence Rabiner and Sorin Dusan
2
Outline
• Review TDNN basics
• Bootstrap TDNN using categories
• Model lattice
• Experiments
• Discussion and future work
3
4
1
2 3
5. Overall System Prototypes and Common Platform
ASAT Paradigm
4
Previous Research
• Frame based method– Used MLP to detect the 14 Sound Pattern of
English features for the 61 phoneme TIMIT alphabet using single frame of MFCC’s
• Major problem– Frames capture static properties of speech– Need dynamic information when detection
dynamic features and sounds– Need segment-based methods rather than
frame-based methods
5
Phoneme Hierarchy
Phonemes
Vowels Diphthongs Semivowels Consonants
AWAYEYOY
WL
RY
Nasals
MNNG Stops Fricat
ives
Affricates
JCH
Whisper
H
Voiced
BDG
Unvoiced
PTK
Voiced
VDHZZH
Unvoiced
FTHSSH
OWAOAELow
UHERAX
IHEH
Mid
UWAAIYHigh
BackMidFront
39 phonemes – 11 vowels, 4 diphthongs, 4 semivowels, 20 consonants
• Bottom-up approach
• Classify voiced stop consonants: /B/, /D/ and /G/ using segment based methods (this is the hardest classification problem among the consonants)
Step 1
Step 2
Step 3
Step 4
6
Voiced Stop Consonants Classification• B, D, and G tokens in the TIMIT training and test sets, without the SA
sentences– * C V form of tokens
• * - preceding phoneme can be any sound• C - B, D, or G• V - vowel or diphthong
– 10 msec windows, 150 msec segments (15 frames)– The beginning of the vowel is at the 10th frame
Training set Test set
B D G Total B D G Total
1567 1460 658 3685 638 537 243 1418
42.5 39.6 17.9 45.0 37.8 17.1
Any preceding phoneme (s)
burst vowel
10th frame
15th frame
1st frame
– Distribution (in # tokens and percentage)
7
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-5000
0
5000
time
Am
plitu
de
2 4 6 8 10 12 14
40
60
80
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-1000
0
1000
2000
time
Am
plitu
de
2 4 6 8 10 12 14
30
40
50
60
70
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-500
0
500
time
Am
plitu
de
2 4 6 8 10 12 14
55
60
65
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
Voiced Stop Consonants Classification• Example – /b/ (speech wave, 10 * log(energy), spectrogram)
Short stop gap Medium stop gap Long stop gap
“ … that big goose …”
Stop gap = 2260 (141 msec)
20580 20850 dh20850 22920 ae22920 25180 tcl25180 25460 b25460 27320 ih27320 29270 gcl29270 29850 g29850 32418 ux32418 34506 s
“… and become …”
Stop gap = 396 (24.74 msec)
48215 48855 ix48855 49684 n49684 50080 bcl50080 50321 b50321 51400 iy51400 52360 kcl52360 53463 k53463 54600 ah54600 55560 m
“… judged by …”
Stop gap = 1100 (68.75 msec)
27770 28970 jh28970 31960 ah31960 32619 dcl32619 33810 jh33810 34280 dcl34280 34840 d34840 35940 bcl35940 36180 b36180 37800 ay
8
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-500
0
500
1000
time
Am
plitu
de
2 4 6 8 10 12 14
50
60
70
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-1000
0
1000
time
Am
plitu
de
2 4 6 8 10 12 14
40
50
60
70
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-1000
0
1000
2000
time
Am
plitu
de
2 4 6 8 10 12 14
30
40
50
60
70
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
Voiced Stop Consonants Classification• Example – /d/ (speech wave, 10 * log(energy), spectrogram)
Short stop gap Medium stop gap Long stop gap
“… scampered across …”
Stop gap = 398 (24.88 msec)
16560 17100 pcl17100 17530 p17530 18162 axr18162 18560 dcl18560 18800 d18800 19360 ix19360 20150 kcl20150 21080 k21080 21640 r
“A doctor …”
Stop gap = 800 (50 msec)
2360 3720 ey3720 4520 dcl4520 4760 d4760 7060 aa7060 8920 kcl8920 9480 t9480 10443 axr
“Does …”
Stop gap = silence = 1960 (122.5 msec)
0 1960 h#1960 2440 d2440 3800 ah3800 5413 z
9
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400-1000
0
1000
time
Am
plitu
de
2 4 6 8 10 12 14
40
50
60
70
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400-4000
-2000
0
2000
4000
time
Am
plitu
de
2 4 6 8 10 12 14
40
60
80
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
200 400 600 800 1000 1200 1400 1600 1800 2000 2200 2400
-1000
0
1000
2000
time
Am
plitu
de
2 4 6 8 10 12 14
30
40
50
60
70
time
10*l
og(e
nerg
y)
time
freq
uenc
y
0 0.02 0.04 0.06 0.08 0.1 0.120
2000
4000
6000
8000
Voiced Stop Consonants Classification• Example – /g/ (speech wave, 10 * log(energy), spectrogram)
Short stop gap Medium stop gap
“ May I get …”
Stop gap = 430 (26.88 msec)
0 2080 h#2080 2720 m2720 4344 ey4344 6080 ay6080 6510 gcl6510 6930 g6930 8141 ih8141 9170 tcl9170 9492 t
“ … a good mechanic …”
Stop gap = 960 (60 msec)
30340 36313 pau36313 36762 q36762 37720 ah37720 38680 gcl38680 39101 g39101 40120 uh40120 41000 dcl41000 41600 m41600 42200 ix
“ …, give or take …”
Stop gap = 3022 (188.88 msec)
24800 27822 pau27822 28280 g28280 29080 ih29080 29960 v29960 30840 axr30840 31480 tcl31480 32360 t32360 34200 ey34200 34966 kcl34966 35400 k
Long stop gap
10
Time-Delay Neural Network• Developed by A. Waibel e
tc.• Effective in classifying dy
namic sounds, like voiced stop consonants
• Introduces delays into the input of each layer of a regular MLP
• The inputs of a unit are multiplied by the un-delayed weights and the delayed weights, then summed and passed through a nonlinear function
11
0 0.5 1 1.5 2 2.5
x 105
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7Mean Square Error for TDNN training, OLGD
No. of training epochs
MSE
3 6 9 24 99 249 780 3135
Jitter
Problems in TDNN Training
• Slow convergenceSlow convergence– during error back
propagation, the weight increments are smeared when averaging on the sets of time-delayed weights
• Requires staged Requires staged batch trainingbatch training– initially trained on a
small number of tokens
– after convergence, gradually add more tokens to the training set
Hand selected
balancedUnbalanced
Bootstrapping begins here
12
A well designed bootstrap training
method
Training Training SolutionSolution
13
Bootstrap Training Introduction
• A bootstrap sample utilizes tokens from the original dataset, by sampling with replacement
• The statistics are calculated on each bootstrap sample• Estimate standard error, etc. on the bootstrap replications
X=(x1, x2, …xn)
X*1 X*2
X*B
S(X*1) S(X*2)S(X*B)
Training dataset
Bootstrap samples
Bootstrap replications
14
Bootstrap TDNN
• Because of the slow convergence of a TDNN, it is difficult (and time consuming) to repeat the training of a TDNN many times (more than 200 times for normal bootstrap training experiments)
• Instead of resampling individual tokens, we build a bootstrap sample by resampling clusters of tokens
• Need to find a way to partition the input space into a small number of clusters which we call categories
15
Bootstrap TDNN – Concept• Begin with a good starting pointBegin with a good starting point
– use a small set of hand selected tokens to train the initial TDNN• Partition the training set into subsetsPartition the training set into subsets
– use the initial TDNN to partition the training set into several subsets which we call “Categories”, and a subsequent TDNN is trained on each category
• Expand each categoryExpand each category– iteratively use the TDNN to partition the remaining (unutilized) tr
aining data into categories; merge the tokens in the category with the previous training data, and train a new TDNN based on the merged data
• Merge final categoriesMerge final categories– merge the tokens in the categories (in a sequenced manner) and
train the final TDNNs based on the union of the categories• Use an Use an nn-best list to combine the TDNN scores to giv-best list to combine the TDNN scores to giv
e the final segment classificatione the final segment classification
16
Bootstrap TDNN – Notation
• Double thresholdsDouble thresholds – a high score threshold (φm
ax) and a low score threshold (φmin)• Good score and bad scoreGood score and bad score – if, and only if, one
phoneme has a score above φmax and the other two phonemes have scores below φmin, the classification score is considered as a good score. All other cases are treated as bad scores.
• Segmentation ruleSegmentation rule:– Category 1 — good score and correct classification – Category 2 — good score and incorrect classification– Category 3 — bad score and correct classification – Category 4 — bad score and incorrect classification
17
Bootstrap TDNN – Procedure• (1) Use the balanced training set of 99 hand-selected tokens (i.e., 33 t
okens each of /B/, /D/, and /G/) as the initial training set, and train a single TDNN
• (2) The current set of TDNNs (one TDNN initially, four TDNNs at later stages of training) is used to score and segment the complete set of training tokens into 4 Categories (based on the double threshold procedure)
• (3) Add selected and balanced (equal number of /B/, /D/ and /G/ tokens) training tokens from the above four categories to the old training data, and train a new set of four updated TDNNs.
• (4) Iterate steps (2) and (3) until a stopping criterion is met– stopping criteria: there are no more new tokens to be added; the desired T
DNN performance is met.
• (5) Merge the tokens from the four training categories in a sequenced manner to obtain a new TDNN. Use a beam search to select the best sequence for merging the data from the four categories
18
Bootstrap TDNN – Illustration• Use the 99 hand selected tokens to train a TDNN, and partition the
initial input space into 4 categories
I(1) II(1)
III(1) IV(1)
A circle denotes balanced tokens in this category
99 TDNN688
1081
954
525309
429
612
863
19
Bootstrap TDNN – Illustration• Merge 99 tokens and the balanced tokens in one category, and train
a TDNN• Use the TDNN to partition the remaining space into 4 categories
99 I(2) II(2)
III(2) IV(2)I(1)
99
I(n)
I(1)
TDNN
• Iterate until the TDNN performance is met, or there are no more new and balanced training tokens to be added to the previous training data
TDNN
II(n+1)
III(n+1)
IV(n+1)
……
…
20
Merge Categories – Model Lattice
• Use a forward lattice to merge the four different category TDNNs– after the initial categories are established, we create a
bootstrap sample consisting of some or all of the categories, selected in a sequenced manner
• Partial lattice – select category samples without replacement
• Full lattice – select category samples with replacement
21
Partial Lattice• Starting point: 4 TDNNs built on each of the 4 category TDNNs• At each step
– select a category without replacement of the previous categories– merge the data in this category and the data in previous step– build a TDNN on the union of the data
Step 1
Cat 1
Cat 2
Cat 3
Cat4
Step 2 Step 3 Step 4• Iterate until all the categories are merged together or the TDNN performance is met
• Use a beam search to select the best path(s) for merging categories to obtain the best set of TDNNs
22
Partial Lattice• Cross node beam search – compare TDNNs that are trai
ned on the same categories but in different sequences
Net 1 Net 2Cat (I) Cat (II)
Cat. (I) U Cat.(II)Net 12 Net 21
Net(1,2) If beam width = 1, select the best net between net12 and net21
• Beam search comparison criterion– performance on the complete training set; or– weighted sum of performances on the 99 hand selected tokens,
the training set, and the complete training set.
23
Full Lattice
Cat 1
Cat 2
Cat 3
Cat 4
……
……
……
……
Step 0 Step 1 Step 2 Step n-1 Step n
• Select categories with replacement• Regular beam search
24
Experiments
• Training set and test setTraining set and test set– TIMIT training set and test set without the SA sentences– *CV form tokens, where C denotes /B/, /D/ or /G/, V denotes any
vowel or diphthong and * denotes any previous sound• /B/ - 1567; /D/ - 1460; /G/ - 658. 3685 tokens in training set• /B/ - 638; /D/ - 537; /G/ - 243. 1418 tokens in test set.
– 13 MFCCs calculated on a 10 msec window with 5 msec window overlap
– average adjacent frames resulting in 10 msec frame rate– segment length: 150 msec (15 frames, with the beginning of the
succeeding vowel at the 10th frame)• TDNNTDNN
– inputs: 13 mfccs, 15 frames → 195 input nodes– 1st hidden layer: 8 nodes, delay D1 = 2;– 2nd hidden layer: 3 nodes, delay D2 = 4;– output layer: 3 nodes, one each for /B/, /D/, and /G/.
25
Baseline Results
• A single TDNN trained using staged batch training of 3, 6, 9, 24, 99, 249, 780, 3685 tokens.
• Train the TDNN on the 99 tokens and test on the same 99 tokens
B D G
B 32 0 1
D 0 32 1
G 0 2 31
• After staged batch training– performance on the complete training set: 91.8%– performance on the test set: 82.0%
• After bootstrapping training + lattice decision, need 68% of training set to achieve comparable results
27
0 1 2 3 440
50
60
70
80
90
100
Bootstrap iterations
Perc
enta
ge
Performance of bootstrapping of catagory 1
99traincompletetest
0 2 4 6 840
50
60
70
80
90
100
Bootstrap iterations
Perc
enta
ge
Performance of bootstrapping of catagory 2
99traincompletetest
0 2 4 6 840
50
60
70
80
90
100
Bootstrap iterations
Perc
enta
ge
Performance of bootstrapping of catagory 3
99traincompletetest
0 5 10 1540
50
60
70
80
90
100
Bootstrap iterations
Perc
enta
ge
Performance of bootstrapping of catagory 4
99traincompletetest
Results of the 4 Category TDNNs• TDNN performance after stopping criteria are met
280 2 4 6 8 10 120
200
400
600
800
1000
1200
Bootstrap iterations
Num
ber
of
tokens in t
rain
ing
Number of tokens used in bootstrapping of TDNN
cat 1cat 2cat 3cat 4
Results of the 4 Category TDNNs
• Number of tokens used in each category
31
Results on Partial Lattice
• Using an n-best list method to score all 4 models and choosing the highest score for each of /B/, /D/ and /G/. The maximum of the 3 highest scores provides the final classification decision.– performance on the complete training set: 93.1%– performance on the complete test set: 82.1%– 35% error reduction on the complete training set, over the best model
trained using data from all 4 categories– 18% error reduction on the complete training set compared with a single
TDNN trained using all the 3685 tokens that achieved 91.8% on the training set and 82.0% on the test set
Cat 1 Cat 2 Cat 3 Cat 4
Lattice Complete train 2423 (65.75%) 21 (0.57%) 1014 (27.52%) 227 (6.16%)
Test 630 (44.43%) 33 (2.33%) 534 (37.66%) 221 (15.59%)
Staged batch training
Complete train 2920 (79.24%) 105 (2.85%)
483 (13.11%) 177 (4.80%)
Test 977 (68.90%) 123 (8.67%)
184 (12.98%) 134 (9.45%)
32
Bootstrap - Discussion
• Bootstrapping is an effective procedure to guarantee convergence in robust training of TDNNs
• The problem with bootstrapping is that the TDNN needs to be trained several times, which is quite time consuming
• In order to reduce the total number of training cycles, we use a beam search method to prune the path for merging different categories of data
• The results showed that, although trained on a relatively small portion of all the training data (approximately 68%), the TDNN achieved better performance on the complete training set and concomitant improvement on the test set
35
A Few Issues – Shifting
• The difficulty in TDNN training lies in the shift-invariant nature of the TDNN. For the set of voiced stop consonants, the stop regions can appear at any 30 msec window during the 150 msec segment.
• The previous vowel affects the BDG articulation and provides information useful for classification
• We can make a TDNN converge faster (to a better solution) by appropriately shifting frames for tokens in categories (II), (III), and (IV).
36
Frame Shifting• Train TDNN on the 99 long stop gap hand selected tokens;
test on the same 99 tokens• Shift right 4 frames for tokens in categories (II), (III) and (IV)• Train the TDNN again, and test on the 99 tokens
Before shifting (# tokens) After shifting (# tokens)
B D G B D G
B 32 1 0 31 1 1
D 1 30 2 0 33 0
G 0 0 33 0 0 33
Category (i) (ii) (iii) (iv)
Before shift (# tokens) 87 0 8 4
After shift(# tokens) 94 0 3 2
Number of tokens in each category
37
Discussion and Future Work• Examine bootstrapping more closely, to reduce the total
number of bootstrap iterations, and to improve model accuracy
• Segment length affects classification accuracy – 150 msec can contain more than one short stop consonants → use DTW to map the input frames to a fixed number of frames and then use the aligned tokens to train a TDNN
• Investigate TDNN for classification of other phoneme classes, e.g., voiced fricatives, diphthongs, etc.
• Use frame-based methods for classification of static sounds (e.g. vowels, unvoiced fricatives); use segment-based methods to recognize dynamic sounds (e.g., voiced stop consonants, diphthongs)
• Develop a bottom-up approach to build small but accurate classifiers first, then gradually classify broader classes in the phoneme hierarchy
38
Thank you!