QoE Software and Hardware Packaging

Jesse Frey

John Marts

Peter Fink

#PSCR2021

DISCLAIMER

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately.

Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

* Please note, unless mentioned in reference to a NIST Publication, all information and data presented is preliminary/in-progress and subject to change

#PSCR2021

INTRODUCTION

Jesse FreyJohn MartsPeter Fink

4

Outline

• Background

• Motivation

• Hardware improvements

• Software improvements

• Current status

5

BACKGROUND : OVERVIEW

• Measurement system for quality of experience (QoE)

• QoE key performance indicators (KPIs) for mission critical voice (MCV)• Mouth-to-Ear Latency

• Access Time

• Voice Quality/Intelligibility

• Probability of Successful Delivery

• More information :• MCV QoE Measurement Methods Overview presentation

• Mouth-to-Ear Latency Measurement System paper (NIST IR 8206)

• Access Time Measurement System paper (NIST IR 8328)

6

BACKGROUND : MEASUREMENT CONCEPT

Communications System

KPIs:• Mouth-to-Ear Latency• Access Time• Intelligibility• Probability of

Successful Delivery

7

THE USER EXPERIENCE: PTT COMMUNICATIONS

• Press the Push To Talk (PTT) button and speak into a device

• Listening to speech output from a device

• It’s all about speech

• Goal - Create measurement systems that are: • Based upon the user experience -- speech

• Comparable and fair across technologies

• This is not:• Analyzing internal system design/construction

8

MEASUREMENT SYSTEM

9

HARDWARE ISSUES

• Radio interface• Development board

• Perfboard and parts

• Not plug and play

• Radio cables• Standard headsets

• Modified with connectors

• Also not plug and play

10

SOFTWARE ISSUES

• Original software written in MATLAB• Excellent libraries and support

• Great for initial development

• MATLAB is expensive and less common

• Code re-use across projects is difficult

11

HARDWARE IMPROVEMENTS

• Radio interface• Create a Printed Circuit Board (PCB) design

• Might be purchasable in the future

• Radio cables• Rework hardware to use interoperability cables

• Available for purchase, ready to go

12

NEW HARDWARE

• Same MSP430F5529 processor• All input/output connections are the

same

• Same firmware

• Has 8P8C connectors for radio cables

• Includes transformers for audio• Used to be external

13

RADIO CABLES

• Standard cables for interoperability gateway

• Use 8P8C connector

• Mostly for LMR radios• Working on getting cables for LTE

PTT phones

14

NEW SOFTWARE

• New software written in Python• Popular

• Used in many major software applications

• Supported on a wide variety of platforms

• Common code is placed in packages• General common measurement functions : mcvqoe

• Intelligibility estimator : abcmrt

• Measurement code written as classes• Allows for re-use

• Also allows for simulations

15

PYTHON MODULES

• Packages all code and data together• Tracks dependencies so all needed packages can be installed

• Can be uploaded to the Python package index (PyPI)• Easy to download and install measurement code and dependencies with a single

command

• Packages will be created for measurement system code• Published to PyPI

• Common code shared across measurements• No more duplicated code for each measurement

16

MEASUREMENT CLASSES

• Each measurement has a class

• Settings for test• Used for logging

• Allows for code reuse during reprocessing

• Allows for simulations with test code

• Easy validation of measurement methods

17

STATUS

• Python code • In the final testing stages

• Should be published by the end of 2021

• First Hardware Revision• Complete

• Also needs testing

• Future hardware technology transfer options

18