Extraction of Listening Effort Correlates in the Oscillatory EEGActivity: Investigation of Different Hearing Aid Configurations

Corinna Bernarding1, Ronny Hannemann2, David Herrmann1, Daniel J. Strauss1,3,4,and Farah I. Corona–Strauss1,3

Abstract— A generally accepted objective measure for listen-ing effort in hearing aid fitting procedures is still missing. Thus,the focus of our research is the extraction of possible neuralcorrelates of listening effort by using electroencephalographicdata. Such an objective measure could optimize the hearing aidfitting procedures by reducing the listening effort in hearingaid wearers. In the current study, we tested different hearingaid configurations in 15 normal hearing persons. For this,we created a realistic listening situation using standardizedsentences embedded in multitalker babble noise at a fixedsignal to noise ratio. The main objectives were (i) to extractpossible neural correlates of listening effort using the previouslyproposed angular entropy measure; (ii) to find the respectiveelectrode locations and scales (frequencies) which best representthe subjectively rated listening effort. In order to decomposethe multiway data (electrode channel x number of sentences xscales) the parallel factor analysis (PARAFAC) was applied tothe ANOVA F-test values. The results indicate that the refinedangular entropy could serve as a possible correlate of listeningeffort in frontal electrode locations in the frequency range ofthe EEG theta band. Anyway further research is necessary tovalidate these findings.

I. INTRODUCTION

Until now, a generally accepted and validated methodto determine listening effort objectively in clinical settings(e.g., hearing aid fitting procedures) is still not available.Especially hearing-impaired persons have an increased effortto process natural signals (e.g., speech) in demanding lis-tening environments [1]. To correctly perceive the relevantinformation, cognitive as well as attentional resources arerequired [1], [2]. Thus, one part of our research focuses onthe attentional, effortful modulation of the incoming (speech)signal to process the presented information. In previous stud-ies [3], [4], we analyzed the instantaneous phase of auditoryevoked responses in the range of the N1 component indifferent listening conditions. Related to the findings of thesestudies, we assume that a higher synchronization of the phasereflects a higher effort to solve an auditory task. The focus ofour current research relies also on the instantaneous phase butextracted from the ongoing oscillatory EEG activity [5]. Forthe analysis of the data we propose the angular entropy. Here,the phase angles of the EEG data are calculated for differentscales (corresponding to the frequencies of the signal) bythe application of the complex wavelet transform. Then, the

1 Systems Neuroscience and Neurotechnology Unit at the Neurocenter,Saarland University Hospital and Saarland University of Applied Sciences,Homburg/Saar, Germany, strauss@snn-unit.de

2 Siemens Audiologische Technik GmbH, Erlangen, Germany3 Key Numerics GbR, Saarbrucken, Germany4 Leibniz-Institut for New Materials, Saarbrucken, Germany

entropy of these phase angles is determined. The entropywas considered as it describes the order and disorder of asystem or a process. We expect that smaller values of theangular entropy reflect a more ”ordered” process of the phasedistribution, i.e. the phase is more synchronized due to anincreased attention on the relevant (speech) signal. To extractthese possible listening effort correlates, different hearingaid features were tested. It is assumed that some of thesefeatures are useful to reduce the listening effort in hearingaid wearers. During the collection of the EEG data, thenormal hearing subjects had to perform an auditory task. Thisparadigm was composed of sentences taken from a Germansentence test (Oldenburger Sentence Test, OlSa [6]), whichwere embedded in a multitalker babble noise. Furthermore,we refined the calculation of the angular entropy and appliedthe parallel factor analysis (PARAFAC) [7] to the data,which was also used by Morup et. al [8] to analyze wavelettransformed evoked responses. This analysis was selected todecompose the multiway data structure (electrode channel xnumber of sentences x scales) in order to find the respectiveelectrode location(s) and scale(s) which best represent thesubjectively perceived listening effort.

II. MATERIALS AND METHODSA. Experimental Paradigm and Hearing Aid Settings

The speech material was taken from a German sentencetest (Oldenburger Sentence Test (OlSa) [6]) which is prin-cipally applied in clinical settings for the detection of thespeech intelligibility threshold. Each sentence consists ofthe following structure: subject - verb - numeral - adjective- object (e.g., Peter buys three red cups) and is spokenby a male voice. The same sentence test was used inorder to create a multitalker babble noise. Thus, five speechsequences were generated. Each sequence was composed of150 sentences of the OlSa test and shifted in time by thelength of one sentence. Finally, the sequences were addedtogether to create one noise file. To hamper the listeningsituation for the normal hearing subjects, the backgroundnoise and the speech signal were filtered (1/3 octave bandfilter bank, 18 channels, frequency range 250-6kHz [9])and attenuated in each frequency band in order to mimica moderate high frequency hearing loss according to [10].The task of the subject during the experiment was to followand to repeat the last word of each sentence. Thus, a sinustone (1kHz, duration: 40ms) was added after each of the 75sentences to indicate the point of time (silent gap with aduration of 2s) where the subject’s response was expected.

6th Annual International IEEE EMBS Conference on Neural EngineeringSan Diego, California, 6 - 8 November, 2013

978-1-4673-1969-0/13/$31.00 ©2013 IEEE 1258

The sentences (S) as well as the disturbing babble noise (N)were calibrated according to the norms [11] and presentedat 60dB SPL (resulting in a SNR of 0dB) via two loud-speakers (loudspeaker arrangements: S0N0, S0N180).The hearing aids were fitted using the same audiogram of themoderate high frequency hearing loss [10], which was alsoused to attenuate the speech signals. An upper comfortableloudness level (UCL) was set at 80 dB HL. In total 7different hearing aid configurations (conf.) were tested: 1.basis fit, 2. reduction of the master gain, 3. gain reductionin the speech area, 4. reduction of the speech enhancement,5. increase of the speech enhancement, 6. omnidirectionalmicrophones, 7. automatic microphone setting. In conf. 1.-5. the loudspeaker arrangement S0N0 and in conf. 6.-7.the arrangement S0N180 was used. Before the experimentstarted, each subject had an acclimatization period of approx.5min with the basis fit configuration. Because of this, the firststage (conf. 1) was repeated at the end of the experiment andthe EEG data were collected. In all conditions the subjectswere instructed to keep their eyes closed and to minimizemovements in order to avoid muscular artifacts. The subjectsresponses were documented by the experimenter. This wasdone to examine if the same speech intelligibility levelwas guaranteed in all conditions. Additionally, we used asubjective seven-step listening effort scale (no effort - verylittle effort - little effort - moderate effort - considerable effort- much effort - extreme effort) [12], where the subjects wereasked to rate their perceived effort. The whole experimentlasted around 40min.

B. Subjects and Data Acquisition

A total of 15 normal hearing subjects (hearing thresholds(below 15dB (HL)) participated in this study (mean age24.8±2.59 years, 8F/ 7M). The subjects were student volun-teers from the Saarland University and Saarland University ofApplied Sciences. All subjects were native German speakersand gave their informed consent after a detail explanationof the procedure. The continuous EEG was recorded witha commercially available amplifier (g.tec USBamp, GugerTechnologies Austria, sampling frequency: 512Hz). The 16active electrodes were placed according to the international10-20 system, with the left earlobe as reference and theground electrode was placed at the upper forehead. Afterthe data collection, the EEG was re-referenced to linkedearlobes. In all measurements electrode impedances werekept below 5kΩ. The data were bandpass-filtered from 0.5 to40Hz. A trigger signal indicated the onset of each sentence,so it was possible to extract only the EEG data duringthe presentation of the sentence. Artifacts were rejected ifeither the maximum amplitude threshold ± 70µV or thecorresponding standard deviation exceeded ± 40µV withina moving time window (window size: 50ms). The subjectswere included in the analysis, if they had (i) 50 % of correctlyrecognized words during the acclimatization period; and (ii)85 % artifact free EEG data.

C. Data Analysis

1) Wavelet Phase Entropy: For the quantification oflarge–scale phase synchronization processes, the angularentropy H was calculated as proposed in our previous study[5]. This angular entropy is based on the distribution ofthe instantaneous phase. The phase ϕa,b was extracted bythe application of the continuous wavelet transform. Letψa,b(·) = |a|−1/2ψ((· − b)/a)) where ψ ∈ L2(R) is thewavelet with 0 <

∫R |Ψ(ω)|2|ω|−1dω < ∞ (Ψ(ω) is the

Fourier transform of the wavelet), and a, b ∈ R, a = 0. Thewavelet transform Wψ : L2(R) −→ L2(R2, dadba2 ) of a signalx ∈ L2(R) with respect to the wavelet ψ is given by the innerL2–product (Wψx)(a, b) = ⟨x, ψa,b⟩L2 . The instantaneousphase of a signal x ∈ L2(R) can be achieved by takingthe complex argument from the complex wavelet transformwith the signal: ϕa,b = arg(Wψx)(a, b)). The determinationof the angular entropy was refined in the following way:The phase of each sentence was unwrapped to display it asa continuous function. Then, a linear function was fitted tothe unwrapped phase and subtracted. After this, the resultingstructure was taken to calculate the angular entropy H . Wedivided the values of the resulting structure into N binsand each bin had the probability pi, i = 1, 2, ..., N , with∑Ni pi = 1. Then, the normalized weighted angular entropy

can be defined by

H = exp(−(1− (−

∑i∈I

pi·ln pilnN ))2

0.01). (1)

We expect that for effortful listening conditions the weightedangular entropy reveals smaller values compared to easierlistening conditions. This could be seen as a more ”ordered”and synchronized process of the phase distribution of theongoing oscillatory activities.

2) Parallel Factor Analysis and Analysis of Variance:In order to analyze the multiway-data-matrices (electrodechannels x number of sentences x scale x hearing aid con-figuration x subjects), an analysis of variance test (ANOVA)and the parallel factor analysis (using the N-way toolboxfor MATLAB [13]) were applied to the data sets as in[8]. For the selection of the two hearing aid configurations,which required the largest difference in perceived effort fromthe subjects, the results of the listening effort rating scalewere taken into account. Then, the angular entropy valuesof these two conditions were used to perform the ANOVAtest. After this, the three-way (electrode channels x numberof sentences x scales) PARAFAC model of the ANOVA Fvalues was calculated using non-negativity constraints onall modalities and the alternating least squares algorithm asin [8]. The correct number of factors was estimated usingthe core consistency diagnostic [14] and the algorithm wasrun three times in order to proof that stable solutions werereached [8].

III. RESULTS AND DISCUSSION

One subject had to be excluded from the analysis dueto having too many EEG artifacts (<85% artifact free-EEGdata), so we had a total of 14 included subjects. In the upper

1259

panel of Figure 4, the median and the standard deviation ofthe subjective listening effort scale are depicted. It can beseen that there is a large difference of the subjectively per-ceived effort between conf. 6 (omnidirectional microphones,loudspeaker arrangement S0N180, considerable effort) and7 (automatic microphone setting, loudspeaker arrangementS0N180, litte effort). Due to this difference and the samefixed loud speaker arrangement (only the hearing aid settingsdiffered), the weighted angular entropy values for theses twohearing aid configurations were taken to perform the ANOVAtest as well as the PARAFAC analysis on the respective F-values (c.f. Section II-C.2). The weighted angular entropywas calculated for different scales a, ranging form 10 to 60in steps of one. The wavelet ψ used in this study was the 6th-derivative of the Gaussian wavelet as in [3]. Note that eachscale a can be associated with a ’pseudo’ frequency fa inHz by fa = Tfψ/a, where T is the sampling period and fψis the center frequency of the wavelet ψ. Thus, the analyzedscales covered a frequency range from 5.12Hz to 30.72Hz.The parallel factor analysis of the three-way (electrodechannels x scales x number of sentences ) ANOVA F-valuesrevealed a good fit in a one component model (explainedvariation 34.79%, results of the core consistency diagnostic:1st component = 100%, 2nd component = 76.28%). Figure 1shows from top to bottom the loading vectors of the electrodechannels, the scales and the number of sentences. It can benoted that there is a difference between the analyzed twohearing aid configurations in the electrode channels FC3,C3, CZ, and C4 for scale 51 (indicated by the highest peaksin the electrode channel and scale loadings). The scale 51corresponds to a pseudo frequency of 6.02Hz and is situatedin the EEG theta band. For a better comparison of the data,the mean difference of the weighted angular entropy (conf.6 - conf. 7) was calculated for the previously identified scale51. The result is illustrated in Figure 2. Here, the meandifference of the angular entropy for all electrode channelsover the number of sentences is depicted in the upper panel.The bright colors indicate a positive difference, whereasdark colors indicate the opposite. A positive difference canbe seen for the same electrode channels, namely FC3, C3,CZ, and C4, which were also identified by the parallelfactor analysis. The largest difference is visible for elec-trode FC3. Furthermore, these findings are supported by thecorresponding significance map (ANOVA-test) in the lowerpanel of the same figure. The white areas indicate significantdifferences p<0.05). In Figure 3, the mean angular entropyvalues for the electrode FC3 and scale 51 (top) for thetwo analyzed conditions (conf. 6 and 7) together with thecorresponding ANOVA-test (bottom) are shown. For thehearing aid conf. 7 (automatic microphone setting), wherethe subjects perceived ”very little effort” (c.f. Figure 4, upperpanel), the angular entropy yield higher values compared tothe values of conf. 6 (omnidirectional microphones), whichwas subjectively rated as a ”considerable effort” condition.The smaller entropy values (for the fixed scale 51) of conf.6 can be interpreted as a result of a more ”ordered” andsynchronized angular phase due to a more effortful listening

Fig. 1. One component PARAFAC model of the ANOVA F-values testingthe multiway difference between conf. 6 and 7 in the 14 subjects. Fromtop to bottom: Loading vectors of the electrode channels, the scales and thenumber of sentences. Note, the highest peaks of the loadings are for theelectrode channels FC3, C3, CZ and C4 and the highest peak in the scalemode is noticeable for scale 51.

Fig. 2. Top: Mean weighted angular entropy difference between conf. 6and 7 for scale 51 (6.02Hz) displayed for all electrode channels (y-axis)over the number of sentences (x-axis). Positive differences are encoded inbright colors and dark colors indicate the opposite. Bottom: Correspond-ing significance map (ANOVA-test). The white areas indicate significantdifferences (p<0.05) associated with the upper plot.

1260

Fig. 3. Top: Mean weighted angular entropy for electrode channel FC3and scale 51 (6.02Hz) for the hearing aid configuration 6 (black line) and7 (gray line). Bottom: Corresponding ANOVA-test. Note the separationof the entropy values for the two configurations over the duration of themeasurement.

condition, i.e., conf. 6 requires more effort from the hearingaid wearers in this study as conf. 7. According to currentliterature, the theta band can be associated with attentionand attentional resource allocation, especially in the frontalareas [15]. Thus, we can assume that our findings could alsobe related to an attentional, effortful modulation/processingof the incoming auditory information. Figure 4 comparesthe subjectively rated effort (top) with the mean weightedangular entropy values for each hearing aid configuration forchannel FC3 and scale 51 (bottom). Here, the entropy valueswere averaged over all 75 sentences and 14 subjects. It canbe noticed that similar pattern for the subjective listeningeffort scale is revealed for the objectively measured data.Note the y-axis is inverted in Figure 4 to ease the comparisonof the subjective data with the angular entropy values, i.e.,the rating scale ranges from extreme effort to no effort. Asthe subjects perceived listening with a particular hearing aidconfiguration more effortful, the weighted angular entropyvalue decreased and vice versa.

IV. CONCLUSION AND FUTURE WORKS

In the present study, we analyzed the weighted angularentropy of the ongoing EEG activity in normal hearingpersons wearing differently configured hearing aids. Theresults indicate that the angular entropy extracted in the EEGfrequency range of the theta band could serve as a possibleindex for listening effort and attention effects in general.However, further research is necessary to proof if the findingscan be reproduced in hearing impaired persons. Thus, a partof our future research includes a study with hearing impairedpersons testing different hearing aid settings.

REFERENCES

[1] M. K. Pichora-Fuller and G. Singh, “Effects of age on auditoryand cognitive processing: implications for hearing aid fitting andaudiologic rehabilitation,” Trends Amplif, vol. 10, pp. 29–59, 2006.

[2] P. A. Gosselin and J.-P. Gagne, “Use of a dual–task paradigm tomeasure listening effort,” Canadian Journal of Speech–LanguagePathology and Audiology, vol. 34:1, pp. 43–51, 2010.

Fig. 4. Top: Median and standard deviation of the subjectively rated effortfor each hearing aid configuration. Bottom: Corresponding mean weightedangular entropy values (averaged over all sentences and subjects) for therespective configuration. Note, smaller angular entropy values correspondto more effortful listening conditions.

[3] D. J. Strauss, F. I. Corona-Strauss, C. Trenado, C. Bernarding, W. Re-ith, M. Latzel, and M. Froehlich, “Electrophysiological correlates oflistening effort: Neurodynamical modeling and measurement,” CognNeurodyn, vol. 4, pp. 119–131, 2010.

[4] C. Bernarding, D. J. Strauss, R. Hannemann, H. Seidler, and F. I.Corona-Strauss, “Neural correlates of listening effort related factors:Influence of age and hearing impairment,” Brain Research Bulletin,vol. 91, pp. 21–30, 2013.

[5] C. Bernarding, D. J. Strauss, R. Hannemann, and F. I. Corona-Strauss, “Quantification of listening effort correlates in the oscillatoryEEG activity: A feasibility study,” in Proceedings of the AnnualInternational Conference of the IEEE Engineering in Medicine andBiology Society, EMBS, 2012, pp. 4615–4618.

[6] K. Wagener, V. Kuhnel, and B. Kollmeier, “Entwicklung und Evalu-ation eines Satztests in deutscher Sprache I: Design des OldenburgerSatztests,” Z Audiol, vol. 38, no. 1, pp. 4–15, 1999.

[7] R. Harshman, “Foundation of the PARAFAC procedure: models andconditions for an ’explanatory’ multi-model factor analysis,” UCLAwork. pap. phonetics, vol. 16, pp. 1–84, 1970.

[8] M. Mørup, L. K. Hansen, C. S. Herrmann, J. Parnas, and S. M.Arnfred, “Parallel factor analysis as an exploratory tool for wavelettransformed event-related EEG,” NeuroImage, vol. 29, no. 3, pp. 938–947, 2006.

[9] European Committee for Standardization, “Electroacoustics- octave-band and fractional octave-band filters,” DIN EN 61260:2003-3.,”Technical Report, 2008.

[10] ——, “Electroacustics- hearing aids. part 15: Signal processing inhearing aids.” Proposal. IEC 60118-15:2008.” Technical Report, 2008.

[11] European Committee for Standardization, “Audiometers - part 2:Equipment for speech audiometry,” EN 60645-2:1997,” TechnicalReport, January 1997.

[12] B. Gabriel and M. Meis, “Optimierung eines Messverfahrens fur dieHoranstrengung,” in Zeitschrift fur Audiologie/Audiological Acoustics,Westhofen and Doring, Eds., vol. Supplementum IV,. In 4. Jahresta-gung der Deutschen Gesellschaft fur Audiologie, 2001, pp. 100–103.

[13] C. A. Andersson and R. Bro, “The n-way toolbox for matlab,”Chemometrics and Intelligent Laboratory Systems, vol. 52, no. 1, pp.1–4, 2000.

[14] R. Bro and H. A. L. Kiers, “A new efficient method for determiningthe number of components in PARAFAC models,” Journal of Chemo-metrics, vol. 17, no. 5, pp. 274–286, 2003.

[15] G. Caravaglios, G. Castro, E. Costanzo, G. Di Maria, D. Mancuso,and E. Muscoso, “Theta power responses in mild Alzheimer’s diseaseduring an auditory oddball paradigm: lack of theta enhancement duringstimulus processing,” J. Neural Transm., vol. 117, pp. 1195–1208,2010.

1261