Combinatorial Approach to Testing Mobile Applications:

An Experimental Investigation

Brandi AmstutzAdvisor: Dr. Sergiy

Vilkomir

This research is supported by the National Science Foundation REU Site Award 1262933

Importance of Research

Mobile testing is a critical step in the development process where errors in an application can be discovered and corrected to increase reliability and functionality.

Traditional software testing strategies can be modified and implemented to solve the challenges associated with mobile device selection.

The amount of devices required to exhaustively represent today’s mobile market is extremely high due to a wide range of variability between available devices.

This investigative research analyzes the efficiency of mobile testing based on a combinatorial approach to device selection.

Introduction

Research Questions Addressed

Which device characteristics have the potential to influence application reliability?

Both hardware and software components have the potential to influence an application’s functionality.

▫ Hardware components considered for this approach include: Mobile device type Screen resolution Hardware manufacturer

▫ Software attributes included: Android OS version RAM memory

Introduction

Research Questions Addressed

How should device selection be executed to ensure an optimal set of devices?

This approach suggests parameter value specification based on user profiledata reflecting the current mobile market. Less popular parameter values can be grouped to reduce the total possible values without compromising proportionality

Introduction

Low11%

Medium23%

High36%

Extra High24.6%

Other (tvdpi and xxhigh)5.9%

Samsung; 55.9%

Sony; 8.1%

HTC; 6.3%

LG; 6.1%

Amazon; 3.6%Xiaomi; 2.5%Google; 2.5%Motorola; 2.4%

Pantech; 1.8%

Huawei; 1.2% Other; 9.5%

Figure 2. Device Manufacturer Distribution

Figure 1. Screen Resolution Distribution Source: Unity 3D Mobile Statistics

Research Questions Addressed

How many devices are sufficient for mobile testing?

Exhaustive testing strategies are limited by resource constraints and contain redundancy between device attributes. This research suggests a set of five devices selected based on device parameterization and combinatorial testing techniques for maximum coverage

Introduction

Device ParameterizationBased on user profile data we suggest to use the following device characteristics and their parameter values:

Materials and Methods: Step One

TypeDevice

ManufacturerAndroid OS

VersionScreen Resolution

RAM Memory

Smartphone HTCHoneycomb (3.0-

3.2)

Lowless than 426 dp x 320

dp

< 768 MB

Tablet MotorolaIce Cream

Sandwich (4.0)Medium

at least 470 dp x 320 dp≥ 768

MB

  Samsung Jelly Bean (4.1-4.3)High

at least 640 dp x480 dp 

  Other OtherExtra High

at least 960 dp x 720 dp 

Table 1. Selected Parameter Values

Device Selection Through Each-Choice coverage techniques and consideration of current market statistics we suggest the following two sets of selected devices:

Materials and Methods: Step Two

Based on user profile data the following devices were included due to their high level of popularity: Set A: Galaxy S2 i9100 and Galaxy Tab 2, 7.0Set B: Galaxy S2 i9100 and Galaxy Tab 10.1

All other devices have been added to provide total coverage of device characteristics

Table 2. Selected Device Sets

Application SelectionThree applications with similar functionality were selected for testing. The following applications used provide access to search, book and review travel accommodations ranging from car rental to discount vacation packages:

Materials and Methods: Step Three

Hotwire Travelocity BookIt.com

Device AccessPerfecto Mobile cloud services were used for remote access to devices for testing purposes. Tests were recorded and screen shots were saved for later analysis. http://www.perfectomobile.com/

Remote testing procedure using SaaS cloud-based device access:

Materials and Methods: Step Four

User provides

test input

Input is sent to devices selected by user

Audio & video of live test

execution is

recorded

Test results & video are sent to user for analysis

Device Access

Materials and Methods: Step Four

Figure 3. Perfecto Mobile Dashboard

Additional DevicesFor comparison, two sets of random devices equivalent in size to the Each-Choice sets were selected. The same testing procedures were executed on the ten additional devices:

Materials and Methods: Step Five

Table 3. Random Device Sets

Screenshot AnalysisScreenshots were analyzed from all sets of devices for fault identification

Results

Figure 4 (left) illustrates Fault H5 for the Hotwire application. The expected result is that the number of rooms should update on the main page when a user selects a new number of desired hotel rooms from the drop down menu. The fault occurs when the room quantity is not updated appropriately

Screenshot Analysis

Results

Figure 5 (right) illustrates fault T1 for the Travelocity.com application. The expectation is that the Travelocity menu bar with their gnome icon is displayed at the top of the home screen. The fault occurs when the menu bar is missing from view.

Screenshot Analysis

Results

Expected

Fault

Figure 6 (left) illustrates fault B1 of the Bookit.com application. The expectation is that a menu bar with current query's information and reference number is displayed at the top of the screen. The fault occurs when this menu bar is missing from view.

Fault Detection

Results

Fault detection totals are as follows:

Hotwire: 7 2 device independent 5 device dependent

Travelocity: 7 1 device independent 6 device dependent

Bookit.com: 4 1 device independent 3 device dependent

Fault Detection

Results

In 7 out of 12 cases (or 58%) the effectiveness of our approach was greater than the results of random testing.

In 4 out of 12 cases (or 33%) the effective ness of our approach was equivalent to the results of random testing.

Table 4. Fault Detection

Conclusions

The quantity of application faults detected by the selected set of devices was equal to or greater than the quantity detected by the random set of devices. Additionally, the random set of devices did not yield a fault that the selected set was not able to identify. This suggests that the device attributes selected were suitable based on their expected influence on application reliability.

Combinatorial testing strategies provided sufficient parameter value coverage with a test suite of five devices. This research suggests that this small set of devices is effective for mobile testing while also reasonable enough for resource constraints.

Conclusion

Questions?