QoE & App Testing on Real Devices · Danish Naeem Vice President Business Strategy Rachel Obstler...

QoE & App Testing on Real DevicesIntroducing Hybrid CLU & App Testing in Cloud

Danish NaeemVice PresidentBusiness Strategy

Rachel ObstlerVice PresidentMobile Testing

Testing is our Competence

Former Device Anywhere

Keynote Mobile Testing joins SIGOS

in close cooperation

Page 2



Agenda

Market and market complexity

Comprehensive QoE Testing

App Performance Testing

Summary

Page 3



Agenda


Comprehensive QoE Testing

App Performance Testing

Summary

Page 4



Smartphone market

0,00%

10,00%

20,00%

30,00%

40,00%

50,00%

60,00%

70,00%

80,00%

2013 2014 2015 2016 2017 2018

Adoption Rate

World Africa Americas Asia Europe Oceania

Source:


Increasing network & service complexity

2.516 4.507 21.646

63.985

102.062

138.809

179.628

224.801

268.692

-

50.000

100.000

150.000

200.000

250.000

300.000

2009 2010 2011 2012 2013 2014 2015 2016 2017

Number of Apps Downloaded

50

1000

0

200

400

600

800

1000

1200

2010-2015 2016-2021

Huge demand of video- a QoE challenge

ExtaByte

15

180

0

50

100

150

200

2010-2015 2016-2021

How to sustain Social Network Quality?

ExtaByte

7

22

Increased signalling loadvs. features phones

Apple Android

X times

• Network Congestion

• Dropped Calls

• Overload

• Frequent capacity expansions

• Complicated servicese.g. VoLTE, VoWiFi, RCSe

• Service delay on apps

• Poor multimedia quality on OTTs

• Device specific issues

Ap

p P

erf

orm

ance

Tes

tin

g R

eal U

ser

Exp

erie

nce

Co

mp

reh

ensi

ve Q

oE

Test

ing

wit

h

Hyb

rid

CLU


Today’s users expect apps to just work on their device

Plethora of applications

Software diversity

Hardware diversity

Frequent changes

Testing on real devices


Testing and measurement with smartphones– existing approaches – Not SIGOS


Testing and measurement with smartphones– Problems with existing approaches – Not SIGOS

Frequent devicedisconnection

Patchy hardware work

Missing carrier grade reliability

Often broken due to battery issues

Not possible tomaintain remotely

No SIM Multiplexing


Agenda


Comprehensive QoE Testing with Hybrid CLU

App Performance Testing Real User Experience

Summary

Page 10



Introducing Hybrid CLU – SIGOS‘s solution for QoE testing with smartphones


Robust hardware for Real Device Testing

Unified QoS and QoE testing in one chassis

Smartphone Battery Emulation

Smartphone SIMMUX support

Remote hard-reset button for the smartphone

Virtualised LU for hosting the DG environment

Leverages existing SITE feature licenses

Supports SITE Interactive

Up to two smartphones and two wireless modules

Integration of Samsung Galaxy series

Hybrid CLU – Salient Features


Hybrid CLU – Available Test Cases


Hybrid CLU – Available Test Cases

Coming Soon

Facebook – newsfeed access, status change, image sharing

Instagram – image sharing

Viber – Viber to Viber messaging, Viber to Viber calls, Viber to/from 2G/3G calls

Twitter – Twitter direct messaging

DropBox* - access a folder, upload/download/delete a file

Skype enhancement* - Skype text messaging


Hybrid CLU – Samsung Galaxy S7 is already supported

Out-of-band control via mechanical fingers, for easier maintenance and support

SIMMUX enabled

USB connector for communication with SITE and powering of the device

Audio connector for Voice Quality

*Available in SITE 3.1.1


Hybrid CLU5 Example Reports – Smartphone Network Services Testing

Daily Success Rate ofservices beeing testedon S5/S6 and S7

3G/CSFB/VoLTEVoiceQuality MOS values

3G Multi-RAB MOS values

IP Interuption Duration in CSFB


Hybrid CLU5 Example Reports – Smartphone OTT Apps Testing

SpeedTest appreporting DL/UL Speed and RTT

Skype_MO_MT reportingMOS andCall_Establish_Duration

WhatsApp_MO_MT reportingMOS andCall_Establish_Duration


Hybrid CLU – SITE Interactive – Interactive GUI in a Browser

Remote screen view, handset can be controlled with mouse gestures and keyboard input

Control section buttons for playing a preset audio file on the handset, upload a preset audio to be played, record the audio session and close the audio session.

Audio streaming player, for the handset audio output

Browser


Hybrid CLU - SITE Interactive PCAPs, NAS and RRC Traces

PCAP available from the SITE Interactive session

NAS and RRC available in SITE trace inspector


Dialing between two smartphones on one screen


Agenda




Summary

Page 22



Confidential

App Testing in the Cloud – new approach

Monitor Anything, Your Way

Any transaction on any mobile app or website

Real smartphones and tablets on operator networks around the world

Understand Your Customers’ Experience

Exactly what happens on the device

Understand customer-perceived performance over time


Interacting with and scripting on Real Devices

Easy point-and-click recorder to create your unique customer transactions

Validation via object, text, image

Any application: web, native, hybrid, messaging

Single and multi-device test cases

Ad hoc testing anytime with one or more devices


Confidential

Device support

Many devices to choose from

Google Nexus 4 4.x WiFi, T-Mobile Available


Google Nexus 6 5.x WiFi Available

Google Nexus 6P 6.x WiFi Available

Google Nexus 7 (2013) 4.x WiFi Available


HTC One M8 4.x WiFi, AT&T, Verizon Available

HTC One M9 5.x WiFi, AT&T Available

LG G3 4.x AT&T, Verizon Available

LG G4 5.x WiFi, Verizon Available

LG G5 6.x WiFi, AT&T, Verizon, T-Mobile Available

LG G Flex D950 4.x AT&T Available

LG Leon H345 5.x T-Mobile Available

Motorola Droid MAXX 4.x Verizon Available

Motorola Droid RAZR 4.x Verizon Available

Motorola Moto G (2015) 5.x WiFi, AT&T Available

Motorola Moto X (2014) 4.x AT&T Available

Samsung Galaxy Tab 3 4.x WiFi Available

Samsung Galaxy Tab S (10.5) 5.x WiFi Available

Samsung Nexus S 4G SPH-D720 4.x Sprint Available

Samsung Galaxy S4 4.x AT&T Available

Samsung Galaxy S5 4.x WiFi, AT&T, Verizon, T-Mobile Available


Samsung Galaxy S6 Edge 5.x WiFi, AT&T, Verizon, T-Mobile Available

Samsung Galaxy S6 Edge+ 5.x WiFi, AT&T, Verizon Available

Samsung Galaxy S7 6.x WiFi, AT&T, Verizon Available

Samsung Galaxy S7 Edge 6.x WiFi, AT&T, Verizon Available

Samsung Note 4 4.x AT&T,Verizon Available

Samsung Note 5 5.x WiFi, AT&T Available

Samsung Note Edge 4.x WiFi Available

Google Nexus 9 5.x WiFi Coming Soon

Apple iPhone 6 9.x WiFi, AT&T Available

Apple iPhone 6S 9.x WiFi, AT&T Available

Apple iPhone 6 Plus 9.x WiFi, AT&T Available

Apple iPhone 6S Plus 9.x WiFi, AT&T Available

Apple iPhone SE 9.x WiFi, AT&T, Verizon, T-Mobile Available

Apple iPad Pro 9.x WiFi Coming Soon

Status

Android Devices

Manufacturer Model Name Model

Number

Supported

OS Version

Operator

iOS Devices




Google Nexus 6P 6.x WiFi Available

Google Nexus 7 (2013) 4.x WiFi Available


HTC One M8 4.x WiFi, AT&T, Verizon Available

HTC One M9 5.x WiFi, AT&T Available

LG G3 4.x AT&T, Verizon Available

LG G4 5.x WiFi, Verizon Available

LG G5 6.x WiFi, AT&T, Verizon, T-Mobile Available

LG G Flex D950 4.x AT&T Available

LG Leon H345 5.x T-Mobile Available

Motorola Droid MAXX 4.x Verizon Available

Motorola Droid RAZR 4.x Verizon Available

Motorola Moto G (2015) 5.x WiFi, AT&T Available

Motorola Moto X (2014) 4.x AT&T Available

Samsung Galaxy Tab 3 4.x WiFi Available

Samsung Galaxy Tab S (10.5) 5.x WiFi Available

Samsung Nexus S 4G SPH-D720 4.x Sprint Available

Samsung Galaxy S4 4.x AT&T Available



Samsung Galaxy S6 Edge 5.x WiFi, AT&T, Verizon, T-Mobile Available

Samsung Galaxy S6 Edge+ 5.x WiFi, AT&T, Verizon Available

Samsung Galaxy S7 6.x WiFi, AT&T, Verizon Available

Samsung Galaxy S7 Edge 6.x WiFi, AT&T, Verizon Available

Samsung Note 4 4.x AT&T,Verizon Available

Samsung Note 5 5.x WiFi, AT&T Available

Samsung Note Edge 4.x WiFi Available

Google Nexus 9 5.x WiFi Coming Soon

Apple iPhone 6 9.x WiFi, AT&T Available

Apple iPhone 6S 9.x WiFi, AT&T Available

Apple iPhone 6 Plus 9.x WiFi, AT&T Available

Apple iPhone 6S Plus 9.x WiFi, AT&T Available

Apple iPhone SE 9.x WiFi, AT&T, Verizon, T-Mobile Available

Apple iPad Pro 9.x WiFi Coming Soon

Status

Android Devices

Manufacturer Model Name Model

Number

Supported

OS Version

Operator

iOS Devices

Change orientation

Power on/off

Disconnect battery for remote recovery

Advanced device functionality


Trending

Availability

Performance


Detailed reports

Shows each individual result

Red dots denote failed transactions

Scatter Plot Chart

Click on a failed transaction to see detailed results including screenshots of each step



Availability

Demo & Product available from 1 July 2016

Product Details

This is a SaaS product

Can be subscribed to on a minimum half yearly period

Customers can chose type and number of devices, from a published device list. Will be restricted to their private account for the duration of contract

Device cost is included in the pricing

For US, Canada and EU, probes will be hosted in the Data centers. Outside these locations, customers will have to host the probe

Only Apple and Android devices are supported

Available in both: Fully Managed Service & Self scripting model

6-8 weeks time for service start (need for data center preparation, device roll out etc..)


Agenda




Summary

Page 29



Product Summary

Comprehensive QoE Testingwith Hybrid CLU

App Testing in Cloud

Use Case Tests and addresses network and network service issues

Understand user’s experience at the handset

Product Type Licensed – you buy the probe as part of SITE System

Cloud-based subscription offer independent to SITE or GR

Product Availability Today 1 July 2016

Measurements Network service issue identification and diagnosis via Deep KPIs (up to Layer 3)Live view on NAS and RRC trace and pcap

Handset-based availability based on what shows on-screen. Screen shots of each step. Live device screen view of running tests

Performance measurement

Network and service based On device (screen- or app-based)

App Use Case Coverage Pre-defined test cases on select applications Any transaction, any application can be scripted

Voice Quality MOS (PESQ/POLQA) NA

Network Services VoLTE, VoWiFi, RCSe, SMSoIP NA

SITE integration Fully integrated into SITE Separate application, scripting, reporting, etc.

SIM Multiplexing Supported Not supported

Device Support Android, select devices Android, iOS, many devices


Product outlook

Available Today Comprehensive QoE Testing with Hybrid CLU

Available from 1 July 2016 App testing in cloud

Available in 2017 – App testing features in Hybrid CLU Full featured Hybrid Evolution

Combination of current Hybrid CLU features with salient features from App testing such as : Record Replay funtionality

Fast integration of Devices

iPhone support

And many more


Thank you

[email protected]

QoE & App Testing on Real Devices · Danish Naeem Vice President Business Strategy Rachel Obstler...

Documents

Transcript of QoE & App Testing on Real Devices · Danish Naeem Vice President Business Strategy Rachel Obstler...