WORD TOPICAL MIXTURE MODELS FOR DYNAMIC LANGUAGEMODEL ADAPTATION

Hsuan-Sheng Chiu and Berlin Chen

Department of Computer Science & Information Engineering,National Taiwan Normal University, Taipei, Taiwan

{g93470240, berlin} @csie.ntnu.edu.tw

ABSTRACT

This paper considers dynamic language model adaptation forMandarin broadcast news recognition. A word topical mixturemodel (TMM) is proposed to explore the co-occurrencerelationship between words, as well as the long-span latenttopical information, for language model adaptation. Thesearch history is modeled as a composite word TMM modelfor predicting the decoded word. The underlyingcharacteristics and different kinds of model structures wereextensively investigated, while the performance of word TMMwas analyzed and verified by comparison with theconventional probabilistic latent semantic analysis-basedlanguage model (PLSALM) and trigger-based language model(TBLM) adaptation approaches. The large vocabularycontinuous speech recognition (LVCSR) experiments wereconducted on the Mandarin broadcast news collected inTaiwan. Very promising results in perplexity as well ascharacter error rate reductions were initially obtained.

Index Terms-word topical mixture model, probabilisticlatent semantic analysis, trigger-based language model,language model adaptation, speech recognition

1. INTRODUCTION

As we know, for complicated speech recognition tasks, suchas broadcast news transcription, it is difficult to build well-estimated language models, because the subject matters andlexical characteristics for the linguistic contents of broadcastnews speech to be transcribed are very diverse and are oftenchanging with time. Hence, there is always good reason todynamically adapt the language models for better speechrecognition results. On the other hand, the conventional n-gram modeling approach is inadequate. The n-gram modelingapproach can only capture the local contextual information orthe word regularity, and it will be a problem for n-grammodeling when there is mismatch of word usage betweentraining and test conditions.

In the recent past, the latent topic modeling approaches,which were originally formulated in information retrieval (IR)[1], have been introduced to dynamic language modeladaptation and investigated to complement the n-gram modelsas well. Among them, the latent semantic analysis (LSA) [2]and the probabilistic latent semantic analysis (PLSA) [3] havebeen widely studied. LSA has been demonstrated effective ina few of speech recognition tasks; however, its derivation isbased on linear algebra operations. Though PLSA-basedlanguage model (PLSALM) has a probabilistic framework formodel optimization, it merely targets on maximizing thecollection likelihood but not directly on its language modelprediction capability, and it also suffers from the problem that

part of its model parameters have to be dynamically estimatedon the fly during the speech recognition process, which wouldbe time-consuming and makes it impractical for real-worldspeech recognition applications. Moreover, there are alsoother approaches developed to complement the n-gram models,such as the trigger-based language model (TBLM) [4, 5], forwhich word trigger pairs are automatically generated tocapture the co-occurrence information among words. By usingTBLM, the long-distance relationship between the words inthe search history and the currently predicted word can becaptured to some extent.

Based on these observations, in this paper a word topicalmixture model (TMM), using words as the modeling units, isproposed to explore the co-occurrence relationship betweenwords, as well as the latent topical information inherent in thesearch histories, for language model adaptation. Theunderlying characteristics and different kinds of modelstructures were extensively investigated, while theperformance of word TMM was analyzed and verified bycomparison with the conventional PLSALM and TBLMmodels. The large vocabulary continuous speech recognition(LVCSR) experiments were carried out on the Mandarinbroadcast news transcription task [6].

The remainder of this paper is organized as follows. InSection 2, we briefly review the related work with thePLSALM and TBLM models. In Section 3, we present ourproposed word TMM and elucidate its difference with theother models. Then, the experimental settings and a series ofspeech recognition experiments conducted are presented inSections 4 and 5, respectively. Finally, conclusions and futurework are given in Section 6.

2. RELATED WORK

2.1. PLSA-based Language Model (PLSALM)PLSA is a general machine learning technique for modelingthe co-occurrences of words and documents, and it evaluatesthe relevance between them in a low-dimensional factor space[3]. When PLSA is applied to language model adaptation inspeech recognition, for a decoded word wi, we can interpreteach of its corresponding search histories H, as a history (ordocument) model MHW used for predicting the occurrenceprobability of wi:

(1)

where Tk is one of the latent topics and P(wi Tk) is theprobability of the word wi occurring in Ti The latent topicdistributions P(wI Tk) can be estimated beforehand bymaximizing the total log-likelihood of the training (oradaptation) text document collection. However, the searchhistories are not known in advance and their number could be

1-4244-0728-1/07/$20.00 ©2007 IEEE

KI'Vi MH -VV, ITPPLSA IVi yp( k-)P( k- IMHi .

k-l

IV - 169 ICASSP 2007

enormous and varying during speech recognition. Thus, thecorresponding PLSA model of a search history has to beestimated on the fly. For example, during the speechrecognition process, we can keep the topic factors P(wi Tk)unchanged, but let the search history's probability distributionover the latent topics P(Tk MHW be gradually updated aspath extension is performed, by using the expectation-maximization (EM) updating formulae [7]. Then, theprobabilities of the PLSALM and background n-gram (e.g.,trigram) language models can be combined through a simplelinear interpolation:

PAdapt (Wi Wi-2Wi -1 )

APPLSA(Wi|Hwi )+ (1 A) Pn-gram(wiwi-2wi-I),where , is a tunable interpolation weight.

(2)

2.2. Trigger-based Language Model (TBLM)To capture long-distance information, we also can use triggerpairs. A trigger pair (A->B) denotes that unit A issignificantly and semantically related to unit B within a givencontext window. The complexity can be reduced by usingwords as the modeling units. Instead of using the averagemutual information (MI) for the selection of trigger pairs [4],the TF/IDF measure which captures both local and globalinformation respectively from the document and thecollection can be used for this purpose [5]. Then, the triggerpairs whose constituent words have the average MI scores orthe TF/IDF scores higher than a predefined threshold can beselected for language modeling, and the associatedconditional probability of the selected trigger pair (w ,wi)can be estimated using the following equation:

PTrig ('Vi wj) (vj)_= 3

where n(w,,w1) is the count of co-occurrences of word wjand wi within a given context window in the training (oradaptation) text collection, and w/ is an arbitrary word thatco-occurs with wi within the context window. Therefore, thesearch history H,I for a decoded word wi can be viewed asa series of words wj and the probability of the search historyH,I predicting word wi can be expressed by linearlycombining the conditional probabilities of the trigger pairs(W ,,W) as follows:

PTrig (vi Hwi) 1H PTrig (wiw|)(4)

where HW is the length of the search history H, w1 isthe word iJnHW The TBLM probability PTrig (wv v j) canalso be combined with the background n-gram model througha simple linear interpolation similar to Eq.(2).

3. WORD TOPICAL MIXTURE MODEL

In this paper, we present an alternative probabilistic latenttopic approach by treating each word wj of the language as atopical mixture model (TMM) Mwj for predicting theoccurrences of the other word wi:

PTMM (Wi Imwj YE P(wi ITk)P(Tk IM,) (5)k=1

3.1. Training ofWord TMMs

Each word TMM M_ can be trained by concatenating thosewords occurring within a context window of size N (forsimplicity, N is set to 3 in this study) around each occurrence

Qwj ,2

WI

Qwj,N

Wix!.

Q. =QQ ., ,272*Q2' IN

Figure 1: A schematic depiction of the occurrences of a wordwj and its corresponding training observation sequence Q'j .

of wj , which are postulated to be relevant to wj, in thetraining (or adaptation) text document collection to form theobservation Q, for training Mw Figure l is a schematicdepiction of the occurrences 'of a word wj and itscorresponding training observation sequence Q, The wordsin Qj are assumed to be conditionally indep'endent givenMw Therefore, the TMM models of the words of thelanguage can be estimated by maximizing the total log-likelihood of their corresponding training observationsrespectively generated by themselves:

log LQ TrainSet = y log P QWIMl )

= E n(wn,Qwj )og PTMM (n4w wX)Q~wn'=wn

(6)

where ,w ) is the number of times a word wn occurringin Qj The word TMM parameters are estimated using thefollowing three EM updating formulae [7]:

wwGQ

3 (T. RMc)ogniion using Wr TMM)

E_ n(w Q,j )P(Tk lw, M waj)P(w, JTk, )= j n (wn,, Qw )P (Tk |vWn rM w

P(TkWH7M )=(T,|Mv)P(W,|r-)3.2. Speech Recognition using Word TMMs

(7)

(8)

(9)

During the speech recognition process, for a decoded word wi,we can again interpret it as a (single-word) observation. Whilefor each of its search histories Hi = w1,w2 i-. , we canlinearly combine the associated TMM models of the wordsinvolved in HJJ to form a composite word TMM model forpredicting wi:

PTMM, ( w i IM H W ) =i(XajPTMM (wil|M.Wj )EKV PKW ( )'

=zai PWjk(kMj(10)

where the values of the nonnegative weighting coefficientsao are empirically set to be exponentially decayed as theword Wj is being apart from wi and summed to 1( 'j=la=1=); and the search history's probability distributionover the latent topics P'(T, HK is thus represented as:

(11)

IV- 170

Qwj1l

P'(Tk IM H,,, ) = Y a jp Tk M "i ).i=l

Figure 2 is a schematic representation of speech recognitionusing a composite word TMM model.

It is noteworthy that unlike PLSALM where the topicmixture weights trained with the training (or adaptation)collection are entirely discarded during the speech recognitionprocess, the topic mixture weights of word TMM are insteadretained and exploited. A nice feature of word TMM is thatgiven a training set of speech utterances equipped withcorresponding correct and recognized transcripts, we canfurther explore the use of discriminative training algorithms,such as the minimum word error (MWE) training [8] and soforth, to train the mixture weights of the word TMM model tocorrectly discriminate the recognition hypotheses for the bestrecognition results rather than just to fit the modeldistributions. We also can combine the word TMM and thebackground n-gram probabilities through a simple linearinterpolation similar to Eq.(2).

3.3. Issue on the Amount of Training ObservationsAs mentioned previously, during the training of a word TMMMk , those words wi that occur within a context window ofsize N around each occurrence of wj in the training (oradaptation) text document collection will be included in thetraining observation Q, for training AIJ. However, thetraining observation QW 'will become considerably larger asthe context window increases. Therefore, in this paper, weinvestigated the use of two statistical measures to removewords wi that are not quite related or relevant to word wjfrom the training observation Q, of the word TMM MAl forthe purpose of speeding up the' training process. The firststatistical measure we used in the paper is mutual information(MI). The MI score of a word pair (w j,wi) co-occurringwithin the context window can be computed as follows:

Score., (wj, wi ) =log P( (12)

The second statistical measure is the geometrical average ofthe forward-backward (FB) bigrams [9] of a word pair(wj, wi) co-occurring within the context window. Forexample, the FB score of a word pair wj and wi can becomputed as follows:

ScoreFB (WIj, WJ)=p (W,i WV )Pb (WI WI)- (13)

Based on these two scores, respectively, we can rank the totalword pairs of the training collection and then select differentportions of the word pairs with higher statistical scores for thetraining of the word TMM models.

3.4. Comparison of Word TMM, PLSALM and TBLMModels

Word TMM, PLSALM and TBLM can be compared from fourperspectives. First, PLSALM models the co-occurrencerelationship between words and documents (or searchhistories), while word TMM and TBLM directly model the co-occurrence relationship between words. Second, PLSALMneeds to update the distributions of the search histories overthe latent topics on the fly; however, word TMM and TBLMcan be trained in an offline manner. Third, PLSALM and wordTMM model the topics of sentences or words with explicitdistributions, while TBLM models topics implicitly. Finally,word TMM has V xK x 2 parameters, PLSALM hasV x K + K xD parameters and TBLM at most has V x Vparameters; where V is the size of the vocabulary, K is the

A composite word TMM model for thesearch history H, wi,w2, ,wi-

A decoded word

wi

Figure 2: Speech recognition using a composite word TMMmodel.

number of the latent topics and D is the number of thedocuments used for training (or adaptation).

4. EXPERIMENTAL SETUP

The speech corpus consists of about 200 hours of MATBNMandarin broadcast news (Mandarin Across TaiwanBroadcast News), which were collected by Acdemia Sinicaand Public Television Service Foundation of Taiwan [10]. Allthe speech materials were manually segmented into separatestories. About 25 hours of gender-balanced speech data of thefield reporters collected during November 2001 to December2002 were used to bootstrap the acoustic training. Another setof 1.5-hour speech data of the field reporters collected within2003 were reserved for testing. On the other hand, the acousticmodels chosen here for speech recognition are 112 right-context-dependent INITIALs and 38 context-independentFINALs. The acoustic models were first trained with themaximum likelihood (ML) criterion and then by the minimumphone error rate (MPE) criterion [8].

The n-gram language models used in this paper consist oftrigram and bigram models, which were estimated using abackground text corpus consisting of 170 million Chinesecharacters collected from Central News Agency (CNA) in2001 and 2002 (the Chinese Gigaword Corpus released byLDC). The vocabulary size is about 72 thousand words. Then-gram language models were trained with the Katz backoffsmoothing using the SRI Language Modeling Toolkit (SRILM)[11]. The adaptation text corpus used for training word TMM,PLSALM and TBLM is collected from MATBN 2001 and2002, which consists of one million Chinese characters of theorthographic transcripts.

In this paper, the language model adaptation experimentswere performed in the word graph rescoring procedure. Theassociated word graphs of the 1.5-hour speech test data werebuilt beforehand by a tree search procedure [6] and using thebackground bigram language model.

5. EXPERIMENTAL RESULTS

The baseline trigram system results in a character error rate(CER) of 20.79% and a perplexity (PP) of 667.23. We firstcompare the performance of word TMM with those ofPLSALM and TBLM. The weights respectively for theinterpolation of word TMM, PLSALM and TBLM with thebackground trigram were all tuned at optimum values. As canbe seen from Table 1, the performance of both word TMMand PLSALM tends to be better as the topic mixture number

IV- 171

|| CER(%) PPBaseline (Trigram) 20.79 667.23

Word TMM CER (0) PP16 topics 19.80 520.3132 topics 19.76 510.2664 topics 19.69 507.13128 topics 19.55 499.30

PLSALM CER(%) PP16 topics 20.13 540.5232 topics 20.06 533.0764 topics 19.99 527.82128 topics 19.95 519.71

TBLM CER (%) PPMI (51,594) 20.11 507.54MI (465,722) 19.99 467.62TF/IDF (88,310) 20.63 614.70TF/IDF (914,159) 20.25 501.01

Table 1: The results of word TMM, PLSALM and TBLM.

increases, and word TMM slightly outperforms PLSALM inall cases. The best PP and CER results for word TMM are19.55% (5.96% relative reduction) and 499.30 (25.17%relative reduction), respectively, while for PLSALM are19.95% (4.04% relative CER reduction) and 519.71 (22.11%relative PP reduction), respectively. Our postulation for thesuperiority of word TMM over PLSALM is that, for wordTMM, both the latent topic distributions and the distributionsof words over the latent topics, estimated from the adaptationcorpus, were fully exploited during the speech recognitionprocess, while for PLSALM, only the latent topic distributionswere exploited. On the other hand, the results of TBLMachieved by using either MI or TF/IDF for the selection ofword trigger pairs are also listed in the lower part of Table 1,where the numbers in the parentheses represent the actual sizeof TBLM when different selection thresholds were applied.The CER performance of word TMM is always better thanthat of TBLM with different selection criteria and thresholds;however, the best PP result of word TMM is worse than thebest result (467.62) ofTBLM with the MI criterion.

We also try to investigate the influence of the amount oftraining observations on the performance of word TMM. Theresults are depicted in Figure 3. As can be seen, byappropriately using the FB scores for training observationselection, we can obtain almost the best CER results eventhough the word TMM models of different complexities (16 or128 mixtures) were trained with reduced observations.Moreover, when the ratio of training observations exploitedduring training is in the interval of 0.4 to 0.8, the resultsachieved by using the FB scores consistently outperform thatdone by using the MI scores in most cases. Therefore, it isconcluded here that with the aid of the FB scores for trainingdata selection, the word TMM models trained with reducedobservations still can retain the same performance level as theword TMM trained with the whole training observations.

6. CONCLUSIONS AND FUTURE WORK

In this paper we have presented a word topical mixture model(TMM) for dynamic language model adaptation. The

20.711I MI 16

20.620 5

FB 16

20 4 _ ~~~~~~~~~~~~MI12820.3

m m I ~~~~~~~~~~FB 12820.3

O 20.2U 20.1

19 9

19.819. 7

19.6

19.5

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Ratios of TrainMng Observations

Figure 3: The relationship between the performance, ratios oftraining observations and selection strategies for word TMM.

underlying characteristics and different kinds of modelstructures were extensively investigated and tested. Wecompared it with the PLSA- and TBLM-based approaches.Very promising results in both perplexity and character errorrate reductions were initially obtained. More in-deepinvestigation and analysis of the word TMM-based approaches,such as the discriminative training of the word TMM models,and their possible applications to spoken documentsummarization and organization, are currently undertaken [12].

7. REFERENCES

[1] W. B. Croft (editor), J. Lafferty (editor) LanguageModeling for Information Retrieval, Kluwer-AcademicPublishers, 2003.

[2] J. R. Bellegarda, "Latent Semantic Mapping," IEEESignal Processing Magazine 22(5), 2005.

[3] D. Gildea, T. Hofmann, "Topic-based language modelsusing EM," in Proc. Eurospeech 1999.

[4] R. Lau et al., "Trigger-based language models: Amaximum entropy approach," in Proc. ICASSP 1993.

[5] C. Troncoso, T. Kawahara, "Trigger-Based LanguageModel Adaptation for Automatic MeetingTranscription," in Proc. Eurospeech 2005.

[6] B. Chen et al., "Lightly Supervised and Data-DrivenApproaches to Mandarin Broadcast NewsTranscription," in Proc. ICASSP 2004.

[7] A. P. Dempster et al., "Maximum likelihood fromincomplete data via the EM algorithm," Journal ofRoyal Statistical Society B 39(1), 1977.

[8] J. W. Kuo, B. Chen, "Minimum Word Error BasedDiscriminative Training of Language Models," in Proc.Eurpspeech 2005.

[9] Saon, G. and M. Padmanabhan, "Data-Driven Approachto Designing Compound Words for Continuous SpeechRecognition," IEEE Trans. on Speech and AudioProcessing 9(4), 2001.

[10] H. M. Wang et al., "MATBN: A Mandarin ChineseBroadcast News Corpus," International Journal ofComputational Linguistics & Chinese LanguageProcessing 10(1), 2005.

[11] A. Stolcke, "SRI language Modeling Toolkit," version1.5, http://www.speech.sri.com/projects/srilm/.

[12] B. Chen et al., "Chinese Spoken DocumentSummarization Using Probabilistic Latent TopicalInformation," in Proc. ICASSP 2006.

IV- 172