Taxonomy of xRperformance

requirements and design of meaningful end to end

KPI for xRPresentation for the ETSI STQ Workshop on

Multimedia Quality in Virtual, Augmented or other RealitiesMay 2017

Dr. Wolfgang Balzer

Overview/ScopeVirtual Reality (VR) and Augmented Reality (AR) areon the rise in multiple fields of applications. Both technologies provide new means of datavisualization for e.g. data processing and man-machine interaction and will have an impact on manysectors of the economy.Mobile data access is a key element to project AR and VR solutions to the places it is used, andsufficient quality of user experience will be – perhapsmore than in other fields – a key success factor.

Scope

Categorization framework: xR use cases, functionality, and requirements for mobile packet data networksDesign of a comprehensive set of KPI fromrespective use cases. Interaction between projected evolution ofmobile networks and xR requirements.

Roles of mobile networks in xR• Access to required data, e.g.

• Download maps and other location-dependent data

• Download real-time environmental information (other vehicles)

• Access to non-local processing power, e.g.• Upload camera feeds and other sensor data• Download data-enriched images

• Maintenance• Software updates

• The trivial rest• General connectivity for users while they use xR

services

Chances and risks• Promises and chances

• Have lean, low-cost local devices with minimum energy consumption

• Do everything in the cloud: Optimization of maintenance cost

• Side effects and risks• Vulnerable (single point of failure) • Potentially extreme radio resource

requirements• In particular if the number of parallel users can vary

strongly

Some examples for VR• Gaming

• Multiplayer, e.g. Shooters or MMORG• Single player (interactive adventures/puzzles)

• Educational• Architecture and Engineering (build/explore large objects

such as buildings, ships,…)• Communication (e.g. immersive teleconferencing)• Sales and marketing• Immersive Data Processing

• (bordering AR; think of all the movies where people enter data worlds or just interact with virtual screens projected into the room)

Some examples of AR• Telemedicine: Guidance in diagnostics or surgery• Logistics: Help humans or robots find items in

warehouses• Maintenance: Assist local repair technicians

(interactive guidance)• Shopfinder, sightseeing assistance• Gaming (Ingress, Pokemon To Go,…) • Interior Design (How will furniture look in your home)

• “Machine AR” in Automotive: providing information about local traffic environment

Are there new QoS/QoEdimensions in xR?• Well-known

• Audio quality• Video quality

• New aspects:• Immersion („credibility“ of xR effects)• Physiological effects (Impact on

physical well-being such as balance, nausea)

xR QoS is not only a function ofcarrier network performance

• QoE result will depend on carrier network performance but translated by differences in implementation and equipment

• data transfer efficiency, transfer strategies, use of equipmentproperties,…

User QoE/QoSxRservice/function

Implementation

Equipment

Infrastructure performance

Carrier networkperformance

Impact of network functionality degradations on functionality or QoE

• VR (from a small survey with VR developers)• Delays > 10 ms can already cause substantial

degradation in QoE (e.g. nausea), Delays > 50 ms make such services practically unworkable

• Data rates below requirements cause visual impairments (actual effects will depend on implementation)

• Degree and shape of dependency is a design parameter (from “graceful degradation” to “hard failure”) if ideal zone is left

xR related bitpipecharacteristics

xR related bitpipecharacteristics

Data Volume MB

Latency From To Data rate RemarkAR processing, minimal local resources < 10 ms 0,001 1

3D FHD or 4K Video data rates

AR or VR fast environment< 1000 ms 3 60

max 450 MBpsburst typical 1 s of video

AR or VR basic environment 1..100 s 3 60Ca. 5 MBps

20 high‐res images

Algorithm or SW updatebackground 10 1000

typical high‐end gaming machine major SW update

Main 5G promises and questions• High data rates

• Load distribution within a given area?• Low latency

• Edge computing – can AR or VR engines be provided near RAN elements? Which other processes (maintenance, security) need to be provided?

• Where will xR take place? Is this consistent with projected coverage characteristics?

• Motion profile (stationary/nomadic, slow moving, fast moving; in-house/deep in-house)

QoS and QoE

Regions of QoE• Perfect/no noticeable QoE improvements

if carrier QoS improves further• Acceptable (small but tolerable

impairments; improvement of carrier QoS will improve QoE noticeably)

• Unpleasant (QoE critically degraded)• Unacceptable

Dimensions of QoE• Degree of impairment• Duration of impairment• Frequency of impairment• Proposed: Averaging of a single indicator will

not work well enough• Short-time strong impairments may be more

tolerable than extended medium or small impairments

• Exact relationship should be expected to depend on the actual use case

KPI families (“pyramid of needs” model)

• Primary• Can the service be accessed at all?• Service disruption rates• Critical range of service-dependent subjective

QoE indicators• Secondary

• Comfort range of QoE indicators• Set-up time (multiple if access is multi-phase)

From use case to KPIUse Case:- Goal (implicitly also providing criteria for

success/failure)- Related flow of activity

- Operational steps- Expected responses Sequence of events as seen from the user’s

perspective- Identify meaningful QoS KPI- Define mapping to QoE

Role of QoS KPI

• Define/identify business critical user perception

• QoE can serve as an instrument to optimally use resources

• Direct investment funds to areas where they produce the highest customer satisfaction

• Provide a framework for diagnostics• Not meaningful for all stakeholders

Specific KPI for VR

• Mostly subjective aspects (human perception)

• General audiovisual QoE (sound, image)• Immersion

• Image artifacts• Responsivity

• Physical well-being• Balance disturbance (due to imperfect

match of images/head motion etc.)

Specific KPI for AR

• Both data and human-experienced related aspects

• General audiovisual QoE (sound, image)• Immersion

• Image artifacts• Responsivity• Match between camera image and

additional information

How to measure xR KPI• Direct (human experience - MOS)

• Similar to video or audio telephony – rate the quality while using the service

• Unclear: Will there be perceptive models such as POLQA or PEVQ?

• Expected: Parametric models to predict xR MOS• Using generic “bit pipe” properties• Probably use specifically designed stimuli• Calibrate models with data from human MOS lab

or field tests• Can crowdsourcing play a role (e.g. embed QoE

into products?)

Upfront xR Testing Strategies; assumptionsApproach: Assume as little as possible about internal working of the service

• Chance: Save some effort (taking shortcuts)

• Risk: Partial blindness• Internal workings can 

change anytime

• Assume (within reasonable limits) as little as possible

• Robustness against functional changes of system under test

Get in Touch

Focus Infocom GmbHwww.focus-infocom.dewolfgang.balzer@focus-infocom.de