(12) United States Patent (10) Patent No.: US 7,613,604 B1 ... · 8 fig. 1 b [20 prior art fzz...

US007613604B1

(12) United States Patent (10) Patent No.: US 7,613,604 B1 Malah et al. (45) Date of Patent: *Nov. 3, 2009

(54) SYSTEM FOR BANDWIDTH EXTENSION OF FOREIGN PATENT DOCUMENTS NARROW-BAND SPEECH

EP 0 287 104 A 4/1988

(75) Inventors: David Malah, Kiryat-Chayim (IL); Richard Vandervoort Cox, New (Continued) Providence, N] (U S)

OTHER PUBLICATIONS (73) Assignee: AT&T Intellectual Property II, L.P.,

New York’ NY (Us) Yasukawa H EdiBunnell H T et al. “Restoration of Wide band signal from telephone speech using linear prediction error processing”,

( * ) Notice: Subject to any disclaimer, the term of this tspolgleg Linguigli’l {199% gricgeghiggzg Fgurtglnlieglg' patent is extended or adjusted under 35 ‘on on ‘on 1 a ep la’ ’ C ' ' ’ ’ 6W or ’ ’

U'S'C' 154(1)) by 27 days‘ USA IEEE, US, Oct. 3, 1996, pp. 901-904.

_ _ _ _ _ (Continued)

Th1s patent 1s subject to a termmal d1s claimer. Primary ExamineriDaniel D Abebe

(21) Appl. No.: 11/691,160 (57) ABSTRACT

(22) Flled: Man 26’ 2007 A system and method are disclosed for extending the band

Related US Application Data Width of a narroWband signal such as a speech signal. The method applies a parametric approach to bandwidth exten

(63) Continuation of application NO- 11/113,463, ?led 011 sion but does not require training. The parametric represen APF- 25, 2005, HOW Pat- NO- 7,216,074, Which is a tation relates to a discrete acoustic tube model (DATM). The Continuation of application No- 09/ 971,375, ?led 011 method comprises computing narroWband linear predictive OCt- 4, 2001, HOW Pat- NO- 6,895,375- coef?cients (LPCs) from a received narroWband speech sig

nal, computing narroWband partial correlation coef?cients (51) Int- Cl- (parcors) using recursion, computing MHZ7 area coef?cients

G10L 21/ 00 (2006-01) from the partial correlation coe?icient, and extracting Mwb (52) US. Cl. ...................... .. 704/205; 704/217; 704/218 area Coefficients using interpolation~ Wideband parcoi's are (58) Field of Classi?cation Search ............... .. 704/205, computed from the Mwb area coef?cients and Wideband LPCs

704/217, 218 are computed from the Wideband parcors. The method further See application ?le for complete search history. comprises synthesizing a Wideband signal using the Wide

band LPCs and a Wideband excitation signal, highpass ?lter (56) References Cited mg the syntheslzed W1deband slgnal to produce a hlghband

U.S. PATENT DOCUMENTS

5,978,759 A 11/1999 Tsushima et al.

6,323,907 B1 11/2001 Hwang 6,691,083 B1 2/2004 Breen

signal, and combining the highband signal With the original narroWband signal to generate a Wideband signal. In a pre ferred variation of the invention, the Mnb area coef?cients are converted to log-area coef?cients for the purpose of extract ing, through shifted-interpolation, Mwb log-area coef?cients. The Mwb log-area coef?cients are then converted to Mwb area coef?cients before generating the Wideband parcors.

(Continued) 18 Claims, 20 Drawing Sheets

I1" [116 [115 no snb I-PC gnb AREA COEFF. Am’ *REAS/HLI‘§E-;REA /

ANALYSIS comm)" INTERPOLATION

A'lb - 112 124 INTERPOLATED AREA

mmibimoN / *2 f COEFF. T0 WIDEBAND /' ‘2° LPC PARAMETERS

[A 1111)] ,0 N “ — I134

Snb INVERSE WIDEBAND LPC Yvlb Shh FILTERING SYNTHESIS MN

126/ 122/ s“,

US 7,613,604 B1

US. PATENT DOCUMENTS

6,813,335 B2 11/2004 Shinbata 6,895,375 B2 5/2005 Malah et al. 7,317,309 B2 * 1/2008 Yamaguchi et al. .... .. 324/76.19

2001/0044722 A1* 11/2001 Gustafsson et al. ....... .. 704/258

2002/0193988 A1* 12/2002 Chennoukh et al. ....... .. 704/205

FOREIGN PATENT DOCUMENTS

JP 01292400

OTHER PUBLICATIONS

Atal, B.S. et al., “Speech Analysis and Synthesis by Linear Prediction of the SpeechWave”, Journal of the Acoustical Society of America, American Institute of Physics, New York, US, vol. 50, No. 2, Jan. 1971, pp. 637-655. Valimaki et al., “Articulartory Control of a Vocal Tract Model Based on Fractional Delay Waveguide Filters”, Speech, Image Processing and Neural Networks, 1994. Proceedings, ISSIPNN ’94, 1994 Intl. Symposium on Hong Kong, Apr. 13-16, 1994, NewYork, NY, USA, IEEE, Apr. 13, 1994, pp. 571-574. Nakatoh, Y. et al., “Generation of Broadband Speech from Nar rowband Speech Using Piecewise Linear Mapping”, Proc. European Conf. Speech Comm. and Technology, EUROSPEECH ’97, 1997. Cheng, YM. et al., “Statistical Recovery of Wideband Speech from Narrowband Speech,” IEEE Trans. Speech and Audio Processing, vol. 2, No. 4, pp. 544-548, Oct. 1994. Carl, H. et al., “Bandwidth Enhancement of Narrow-Band Speech Signals”, Proc. European Signal Processing ConfiEUSIPCO ’94, pp. 1178-1181, 1994. Yoshida, Y, “An Algorithm to Reconstruct Wideband Speech from Narrowband Speech Based on Codebook Mapping,” Proc. Intl. Conf. Spoken Language Processing, ICSLP ’94, 1994. Yasukawa, H. “Quality Enhancement of Band Limited Speech by Filtering and Multi-rate Techniques,” Proc. Intl. Conf. Spoken Lan guage Processing, ICSLP ’94, 1994, pp. 1607-1610. Hermansky, H. et al., “Speech Enhancement Based on Temporal Processing,” Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’95, pp. 405-408, 1995. Yasukawa, H., “Enhancement of Telephone Speech Quality by Simple Spectrum Extrapolation Method”, Proc. European Conf. Speech Comm. and Technology, EUROSPEECH ’95, 1995. Yasukawa, H. “Adaptive Filtering for Broad Band Signal Recon struction Using Spectrum Extrapolation,” Proc. IEEE Digital Signal Processing Workshop, pp. 169-172, 1996. Yasukawa, H. “Restoration of Wide Band Signal from Telephone Speech Using Linear Prediction Residual Error Filtering,” Proc. IEEE Digital Signal Processing Workshop, pp. 176-178, 1996. Yasukawa, H. “Implementation of Frequency Domain Digital Filter for Speech Enhancement”, Proc. Intl. Conf. Electronics, Circuits and Systems, ICECS ’96, pp. 518-521, 1996. Yasukawa , H. “Signal Restoration of Broad Band Speech Using Nonlinear Processing”, Proc. European Conf. Speech Comm. and Technology, EUROSPEECH ’96, pp. 987-990, 1996. Yasukawa, H. “Wideband Speech Recovery from Bandlimited Speech in Telephone Communications,” Proc. Intl. Syrnp. Circuits and Systems, ISCAS ’98, pp. IV-ZOZ-IV-Z05, 1998. Epps, J. et al., “A New Technique for Wideband Enhancement of Coded Narrowband Speech”, Proc. IEEE Speech Coding Workshop, SCW ’99, 1999. Enbom, N. et al., “Bandwidth Expansion of Speech Based on Vector Quantization of the Mel Frequency Cepstral Coef?cients,” Proc. IEEE Speech Coding Workshop, SCW ’99, 1999.

11/1989

Jax, P et al., “Wideband Extension of Telephone Speech Using a Hidden Markov Model”, Proc. IEEE Speech Coding Workshop, SCW ’00, 2000. Valin, J -M. et al., “Bandwidth Extension of Narrowband Speech for Low Bit-Rate Wideband Coding,” Proc. IEEE Speech Coding Work shop, SCW ’00, 2000. Park et al., K-Y et al., “Narrowband to Wideband Conversion of Speech Using GMM Based Transformation”, Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’00, pp. 1843-1846, 2000. Miet, G. et al., “Low-Band Extension of Telephone-Band Speech” Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’00, pp. 1851-1854, 2000. Chennoukh, S. et al., “Speech Enhancement Via Frequency Band width Extension Using Line Spectral Frequencies”, Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’01, 2001. Uncini, A. et al., “Frequency Recovery of Narrow-Band Speech Using Adaptive Spline Neutral Networks,” Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’99, 1999. McCree, A., “A 14 kb/ s Wideband Speech Coder with a Parametric Highband Model,” Proc. Intl. conf. Acoust., Speech, Signal Process ing, ICASSP ’00, pp. 1153-1156, 2000. Taori, R., “Hi-Bin: An Alternative Approach to Wideband Speech Coding”, Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’00, pp. 1157-1160, 2000. McCree, A., “An Embedded Adaptive Multi-rate Wideband Speech Coder”, Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’01, 2001. Erdmann, C., “A Candidate Proposal for a 3GPP Adaptive Multi-Rate Wideband Speech Coded,” Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’01, 2001. Makhoul, J. et al., “High-Frequency Regeneration in Speech Coding Systems,” Proc. Intl. Conf. Acoust., Speech, Signal Processing, ICASSP ’79, pp. 428-431, 1979. Wakita, H., “Direct Estimation of the vocal Tract Shape by Inverse Filtering of Acoustic Speech Waveforms,” IEEE Trans. Audio and Electroacoust., vol. AU-21, No. 5, pp. 417- 427, Oct. 1973. Wakita, H., “Estimation of Vocal-Tract Shapes from Acoustical Analysis of the Speech Wave: The State of the Art,” IEEE Trans. Acoustics, Speech, Signal Processing, vol. ASSP-27, No. 3, pp. 281 285, Jun. 1979. Schroeder, M.R., “Determination of the Geometry of the Human Vocal Tract by Acoustic Measurements”, Journal Acoust. Soc. Am., vol. 41, No. 4, (Part 2), 1967. Schroeter, J. et al., “Techniques for Estimating Vocal-Tract Shapes from the Speech Signal,” IEEE Trans. Speech and Audio Processing, vol. 2, No. 1, Part II, pp. 133-150, Jan. 1994. Baharav, Z. et al., “Hierarchical Interpretation of Fractal Image Cod ing and Its Applications,” Chapter 5, Y. Fisher, Ed., Fractual Image Compression: Theory and Applications to Digital Images, Springer Verlag, New York, 1995, pp. 97-117. Avendano, C., “Beyond Nyquist: Towards the Recovery of Broad Bandwidth Speech from Narrow-Bandwidth Speech,” proc. Euro pean Conf. Speech Comm. and Technology, EUROSPEECH ’95, pp. 165-168, Madrid, Spain 1995. Chan, C-F., “Wideband Re-Synthesis of Narrowband Celp-Coded Speech Using Multiband Excitation Model,” Proc. Intl. Conf. Spoken Language Processing, ICSLP ’96, pp. 322-325, 1996. Epps, J ., “Wideband Extension of Narrowband Speech for Enhance ment and Coding,” School of Electrical Engineering and Telecom munications, The University of New South Wales, Sep. 2000, pp. 1-155.

* cited by examiner

US. Patent Nov. 3, 2009 Sheet 1 0f 20 US 7,613,604 B1

FIG. 1A [To PRIOR ART /12

NARROWBAND V HIGHBAND SIGNAL SPEECH GENERATION

Snb '

[14 : 1:2 INTERPOLATION

fnb 8

FIG. 1 B [20 PRIOR ART fzz

NARROWBAND _ HIGHBAND SIGNAL

SPEECH / 24 GENERATIoN

Snb 1=2 INTERPOLATION 26

/

— DELAY

fnb 8

FIG. 2A

[36 [as [40 Snb UPSAMPLINGAZ _ HIGHBAND _ GAIN 3hb_

' (INsERT ZEROS) ' FILTERING ' ADJUSTMENT '

FIG. 2B [44 N [46 [48 [50

Snb 1=2 Snb NON-LINEAR HIGHBAND GAIN Shb INTERPOLATION V PROCESSING V FILTERING V ADJUSTMENT


F I C. 3 /54 GENERATION OF

~ WIDEBAND ENVELOPE

REPRESENTATION wb Cl

Snb _ _

—’ [56 [58 I so GENERATION OF r HIGHBAND 3

~ WIDEBAND "*1 ~ FILTERING l

EXCITATION AND GAIN

F I G. 4 f 64 f 66

Snb EXTRACTING gnb EXTRACTING gwb —> NARROWBAND ENvELOPE—> WIDEBAND ENvELOPE —>

REPRESENTATION REPRESENTATION

FIG. 5A nb Q

l- /72 [74 [76 NARROWBAND

Snb INvERSE rnb RESIDUAL SPECTRAL rwb —> FILTERING V BANDWIDTH V FLATTERING —

[A hm] EXTENSION (OPTIONAL) n

FIG. 5B gnbIzoRgwb

[a2 1 /a4 /as 188 S , “g WIDEBAND 7 RESIOuAL SPECTRAL ,

LINTERFIbZLAHON "b= INvERSE “b= BANDWIDTH = FLATTERING i FILTERING EXTENSION (OPTIONAL)


FIG. 6

GLOTTIS LIPS

FIG. '7

98/12345678910111213141516 LIPS GLOTTIS


220

378

$1

28:55 on: gimme;

£2

em em 225055 2E5 EIQE?

M as; 2

NE

a

mi zudnf; 2

21

20:32:00 v58 5% 21 m. 65%

Ul

on: :1


FIG. 9

FREQUENCY RESPONSE OF LPF INTERPOLATION FILTER [138 20 l l |

MAGNITUDE

[dB]

—100- -

-120 - - MMMMAMMU 0 2000 4000 0000 8000

FREQUENCY [Hz]

US. Patent

MAGNITUDE

[dB]

——————————————————— —- --- COMPENSATION FILTER —

Nov. 3, 2009 Sheet 6 0f 20 US 7,613,604 B1

FIG. 1 O

140 IRS AND IRS-COMPENSATION j

Ail’ 1'; "" ' "’ ._ §'~ ..... _./

"" IRS RESPONSE

— COMPENSATED RESPONSE

1500 2000 2500 5000 3500 4000

FREQUENCY [Hz]

US 7,613,604 B1 US. Patent Nov. 3, 2009 Sheet 7 0f 20

152, INITIALIZATION F I G- 11 F1

154 \Q

: READ SIGNAL FRAMEj \/

_ 1:2 SIGNAL $164 ‘, INTERPOLATION

156v‘ LPC ANALYSIS ‘I

_ INVERSE ,

A ' FILTERING 166

AREA 158“ PARAMETERS H

SHIFTED OPERATION 16°“ INTERPOLATION

II II

WIDEBAND LP _ 162% Com. 7 LPC SYNTHESIS @170

II

SPECTRAL SHAPING ’ 172 AND GAIN

US. Patent

NORMALIZED AREA VALUE


Nov. 3, 2009 Sheet 8 0f 20 US 7,613,604 B1

FIG. 12A

184 / LINEAR SHIFTED-INTERPOLATION — AREA COEFF.

<3‘, 8 SECT. —A— 16 SECT. —El— 16 INTERP.

SECTION NUMBER

FIG. 12B

LINEAR SHIFTED-INTERPOLATION - LOG-AREA COEFF. / 194

' ' ' 2 <3‘, 8 SECT.

—A— 16 SECT. -

—El— 16 INTERP.

SECTION NUMBER

US. Patent



Nov. 3, 2009 Sheet 9 0f 20 US 7,613,604 B1

F I G. 12 C

204 / SPLINE SHIFTED-INTERPOLATION — AREA COEFF.

é a SEICT. —A— 16 SECT. a 16 INTERP.

SECTION NUMBER

FIG. 12D

SPLINE SHlFTED-INTERPOLATION — LOG-AREA COEFF. / 214

' ' ' é 8 SEN. _

—A— 16 SECT. -

—El— 16 INTERP.

SECTION NUMBER


FIG. 13A

SPECTRAL ENVELOPES: LINEAR SHIFTED INTERPOLATION — LOG AREA

8000 6000 2000 4000 FREQUENCY [Hz]

FIG. 13B

SPECTRAL ENVELOPES: SPLINE SHIFTED INTERPOLATION — LOG AREA


FIG. 14A 238 l

8000

— Nb RESIDUAL

— Wb EXCITATION

244

HIGHBAND

6000

EXCITATION SPECTRA — VOICED SPEECH FRAME

240

2000 4000 FREQUENCY [Hz]

O o 1

- - _ - - - 0

0 o 0 0 o 0 0 0 9 8 7 6 5 4 3 2

E w] “B Nd G[ A

FIG. 14B 248 [ EXCITATION SPECTRA — UNVOICED SPEECH FRAME

8000

— NB RESIDUAL — Wb EXCITATION

254

HIGHBAND

6000

252

2000 4000 FREQUENCY [Hz]

R _ - ER - - - - 0

0 0 0 O 0 o 0 0 0 9 8 7 6 5 4 3 2 1 E W] .HB Nd G[ A M

US. Patent Nov. 3, 2009

FIG. 16A VOICED SPEECH SIGNAL 284

x104 NARROWBAND INPUT SIGNALS 1.2

12s 25s 3114 512

FIG. 16B

x104 ORIGINAL WIDEBAND SIGNALS 1.2 I

_1_2 . 0 128 250 384 512

FIG. 16C 288

2000 ORIGINAL HIGI-IBAND SIGNAL 1

0111010 128 255 384 512

SAMPLE N0. IN FRAME

Sheet 13 0f 20 US 7,613,604 B1

FIG. 16F UNVOICED SPEECH SIGNAL

296 NARROWBAND INPUT SIGNAL 8

1000

0

-1000 - - -

0 12s 250 384 512

FIG. 16G

1000 ORIGINAL WIDIEBAND SIGNAL 1

0

-1000 - - -

0 12s 250 384 512

FIG. 16H

50o ORIGINAL HIGHBAND SIGNAL 1

OMMWWWM 128 256 384 512

SAMPLE N0. IN FRAME

U.S. Patent

0 128 SAMPLE N0. IN FRAME

Nov. 3, 2009

FIG. 16D VOICED SPEECH SIGNAL

290

GENERATED HIGHBAND SIGNAL 8

NW 128 256 384 512

FIG. 16E 292

x104 GENERATED WIDEBAND SIGNALL

255 364 512

Sheet 14 0f 20 US 7,613,604 B1

FIG. 16I UNVOICED SPEECH SIGNAL

302

GENERATED HIGHBAND SIGNALS 50o

-5oo - - -

128 256 384 512

FIG. 16J 304

woo GENEIIQATED WIDEBAND SIGNALS

o

-1ooo - - -

0 128 256 384 512 SAMPLE N0. IN FRAME

US. Patent Nov. 3, 2009 Sheet 15 0120 US 7,613,604 B1

FIG. 17A “Which tea party did Baker go to”

310“

FIG. 1 '7 B NARROWBAND INPUT

/ 812 4000

FREQUENCY 3000 2000 1000 - 1 4 V _ _

00 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 SEC

FIG. 1 '7 C BANDWIDTH EXTENDED

0 0.2 .4 0.6 0.8 1 1.2 1.4 1.6 1.8 " 2

FIG. 1 7D WIDEBAND ORIGINAL

8000

6000 _

FREEJIEIJSNCY 4000 2000

0

US. Patent Nov. 3, 2009 Sheet 16 0f 20

FIG. 18

f 320 f 522

US 7,613,604 B1

LOWPASS FILTER

(HALF-BAND) vgnl x(n) NON-LINEAR

OPERATOR

FIG. 19

POWER SPECTRA 324 I — - - — INPUT HALFBAND

............. .............. .............. .............. ....... ............. a:

FULLWAVE

—GENERALIZED RECTIFICTION

............. RECTIF.

NORMALTZED FREQUENCY

0 0.05 0.1 0.15 0.2 0.25 0.5 0.55 0.4 0.45 0.5


FIG. 20A

352 I POWER SPECTRA — 0l=1

— - - — INPUT HALFBAND

GENERALIZED RECTIFICTION

NORMALIZED FREQUENCY


FIG. 2 OB

5 POWER SPECTRA - a=0 f 340

‘ —--—INPUT HALFBAND

0 I. —GENERALIZED RECTIFICTLON

0 0.1 0.2 0.3 0.4 0.5

NORMALIZED FREQUENCY

(12) United States Patent (10) Patent No.: US 7,613,604 B1 ... · 8 fig. 1 b [20 prior art fzz...

Documents

Transcript of (12) United States Patent (10) Patent No.: US 7,613,604 B1 ... · 8 fig. 1 b [20 prior art fzz...