Draft-ietf-codec-results-00 Jan Skoglund, Christian Hoene November 15th, 2011...

draft-ietf-codec-results-00 1

draft-ietf-codec-results-00

Jan Skoglund, Christian HoeneNovember 15th, 2011


Diff todraft-valin-codec-results-00

• Added test results of Google and Universität Tübingen (Section 2.1.4, 2.1.5 and 2.4)

• Editorial changes – Moved Section 2 and 3 to the appendix – as

suggested by email comments– Moved Section 4 to the appendix as it also refers

to old implementations.


Results of Google

• Can Opus compress – tonal languages (in this case Mandarin) well?

• What is the impact of transcoding – with AMR and AMR-WB?

Measurement Methodology• Followed MUSHRA ITU-R BS.1384-1• Headphones


99,8

76,3

22,8

76,8 77,9

63,6

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0

Original LP 3.5 MNRU iLBC 15 Opus 11 Speex 11

Narrowband Coding -Mandarin Speech


99,0

54,6

79,5 81,6

98,1

53,6

72,6

93,4

0,0

20,0

40,0

60,0

80,0

100,0

Original LP 3.5 LP 7 Opus 20 Opus 32 Speex24

G.722.124

G.71932

Wideband and fullband codecs -Mandarin speech


99,5

63,5

14,9

54,551,1

54,150,9

47,8

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0

Narrowband transcoding -English speech


99,4

37,0

74,278,4

64,0 65,3 62,8

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0

Wideband transcoding -English speech


Results of Universität TübingenCan Opus compress

– stereo voice in the Silk and the Hybrid mode?– two simultaneous voices?– binaural content?

How does Opus perform compared to other stereo voice codecs? – No open source stereo voice codec available– thus, used AMR-WB+

Measurement Methodology– Followed MUSHRA ITU-R BS.1384-1– Headphones– Participants were unaware of a hidden reference


Reference Items

1. One Voice Stereo8s, stereo voice recording, female German speakers

2. Two Voices Stereo9s, two stereo female voices mixed together

3. One Voice Binaural13s, one female voice, rendered with HTRF and added room impulse response, moving

4. Two Voice Binaural13s, two female voices at different stationary positions, rendered with HTRF and added room impulse response

5. Acappella Song „Mein Fahrrad“ by „Die Prinzen“10.5s, mono


opus.12kamrwbp.12k

opus.16kanchor

amrwbp.16kopus.32k

amrwbp.32kopus.64k

reference

0

20

40

60

80

100

Items

MU

SH

RA

sc

ore

ba

se

d o

n 8

5 r

ati

ng

sResults: Codecs

CI 95%

opus.12kamrwbp.12k

opus.16kanchor

amrwbp.16kopus.32k

amrwbp.32kopus.64k

reference

0

20

40

60

80

100

one voice stereo two voices stereo one voice binaural

two voices binaural acappella

Items

MU

SH

RA

sc

ore

ba

se

d o

n 1

7 r

ati

ng

s

Codec and Item

CI 95%


Summary of both tests

• Encoding of Mandarin works well – Passed– However, Opus at 20 kbps did not outperformed

G.719 at 32 kbps as with English– Mandarin seems not to contain as much contain

as much high frequency-rich consonants as English• Transcoding via AMR and AMR-WB – Passed

– Opus better than AMR-WB• Stereo voice coding in CELT mode – Passed• Stereo voice coding in CELP mode – Fixed?


Next steps: Codec characterization• Codec characterization needed for

– People selecting codecs– Application programmers

• Work can only be started after code freeze.

• Old test results might be use but only after testing output of old version equals output of new version

• Shall we run objective tests on speech and audio quality as well as runtime and memory consumption?

• Merging with draft-hoene-quality-01 to include some more textual descriptions?


• Please send new listening-test results to the editor ([email protected])

• Thanks…

mailto:[email protected]



Draft-ietf-codec-results-00 Jan Skoglund, Christian Hoene November 15th, 2011...

Documents

Transcript of Draft-ietf-codec-results-00 Jan Skoglund, Christian Hoene November 15th, 2011...