Testdroid: Best Practices in Mobile Game Testing

2 December 2014

Ville-Veikko HelppiTechnical Product Manager

ville-veikko.helppi@bitbar.com

Learn the Best

Practices in Mobile

Game Testing

W E B I N A R

Agenda

• Different Types of Testing and How Game

Testing Differs from Regular App Testing?

• Frameworks, Tools and Technologies – What is

Important in Mobile Game Testing Today?

• Top Priorities for Mobile Game Testing

• Infrastructure for Successful Game Testing

• Hands-on Example (Image Recognition)

• Testdroid Update

• Q&A

Let's Define 'Testing'...

Different Types of Testing?

• Black-box testing vs. White-box testing

• Methods used in Mobile Game Testing

– Functional

– Compatibility

– Performance

– Localization

– Regression

– Load

– Smoke

– Acceptance

Defining Some Ways to Test

• Functionality of the game-play, featuresFunctional Testing

• Done always when new features /regressions are includedRegression Testing

• How game runs on different configurationsCompatibility Testing

• Different languages, geo-focused materialsLocalization Testing

• Endurance test to determine if system can handle the loadSoak Testing

• Measures the capacity of the system Stress Testing

• Simplest form of performance testing, measures how system handles certain loadsLoad Testing

• Isolation of the environment (e.g. from network) to see how game worksHermetic Testing

Feature

-based

testing

Performance

testing

End-user

testing

The Old-Fashion Thinking

Deterministic systems

Identifiable objects

Tool support(s)

User-driven actions

Simple inputs/outputs

Virtualization

Random setups/results

No handle on objects

No standards

User reacts to input

Complex inputs/outputs

Real hardware

High-level of automation Low-level of automation

Test Automation as a Solution

Random setups/results -> Easy to generate random events

No handle on objects -> Variety of options (e.g. image

recognition)

No standards -> Variety of frameworks (cross-platform

User reacts to input -> Test scripts can handle random

events

Complex inputs/outputs -> Test scripts can generate input/content

Real hardware -> Availability of real mobile hardwareTest Automation brings real value

with variety of options to mobile game testing

Challenges that Test Automation Tackles

• Proliferation of Devices, Brands, Operators,

Platforms

• Coupling the Testing and Development process

(Continuous Integration/Deployment/Delivery)

• Different Types of Testing (as automated)

– Stress, Load, Performance, Functionality etc.

• Test script & test asset usability

• Cons of Manual testing

– Tedious effort, Expensive human resources, Error-prone

(as done by humans) & Not reusable (done once)

versions

ChipsetsCPU + GPU

Tens of

Memory

Displays (resolutions,

physical

hardwar

e (connectivity,

calibration)

Relation

to other

software

Significant Benefits of Using Automation

Correct behaviour

across platforms and

browsers

Integration with web

back-ends

Typically need to

fully utilize HW

(CPU+GPU)

Resource (e.g.

battery)

consumption

OpenGL ES 2/3

Functionality

and usability

Screen

orientations,

connectivity,

user profiles

Robustness

Robustness and

security!

Compliances,

verification with back-

ends and data

Different 'Verticals' Require Different Practices

Importance of

Mobile Game Testing

Important Aspects in Game Testing

• User Interface and Functionality

– UI layouts/elements, Menus, Resolution,

Orientation, Overall visuality

• Graphics Performance

• Usability and User Experience

– Responsiveness, Real Occuring Events

• Multi-player/User

• Social Integrations

• Security and Liabilities

User Interface and Functional

• Compared to regular app testing, mobile game

testing may seem to be more challenging

– e.g. OpenGL ES content not action-identifiable

• Graphics and UI – Identical look&feel across all

devices and variants

• Usability controls in the context of game

• Navigation/game-play, Progress, Delays

between "game-stages"

• Animations, Fonts, Other graphics assets

• Screen Resolution and Orientation

Performance Testing

• Performance, Load, Stress, Spike, Soak

• Measure the real

user experience

• Execute tests with

all possible devices

(SW&HW)

• Discover the impact

of server loads, app

performance

(CPU/GPU

utilization)

• Studies: Majority of

users leave game due

sluggish, general

slowness or other bad

performance behavior

• It's very important to

understand how game

behaves on different

devices, under

different circumstances

Performance Makes +/- UX

Building the User Experience

UnderstandingHow quickly gamer assimilates what the

game is about, how to fully play with it, what

are the functionalities.

UsabilityThe main User Interface, menus, design and

how usable those are

Playing ExperienceIntuition to follow the game and go forward

with its instructions and game-play

MonetizationAre gamers willing to buy – invest – in game

• Graphics performance

capability is the most

important thing for mobile

games when forming UX

• Too many games fail to

work properly with low-

end/mid-range devices due

hardware performance

• The real understanding of

game's performance

capability can be only

figured out by testing on

real devices

• End-to-end between tests, AI or

more simple implementations

• Optimize the data communication

between server and clients

• Test simultaneously on several

(different) devices

• Check what happens if game gets

interrupted (platform)

• Location-based features – how to

properly mock up those?

• Hermetic environment – what

happens?

Multi-Player/User Features

• Important part of the after-

engagement (how gamers

share their results)

• Many of OEM branded

devices provide certain

SoMe apps pre-installed

(version?)

• Connectivity, login and

data transferring are

tested in context of game

• Hermetic environment –

what happens?

Social Integrations

Security and Liabilities

4 Top Priorities in

Mobile Game Testing

Localization

Device Diversity

By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/

Real Devices vs. Emulators

• Emulators/simulators won’t help you to test…

– User Experience and Usability

– Hardware

– Software

– Infrastructure

0 % = the percentage of your app users

that use emulator to run your app!

Why Real Devices are Must-to-Have?

Manual vs. Automation

Smaller coverage, More money burnt & time wasted, Error-prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.

Manual Automation

Infrastructure for

Mobile Game Testing

High-Level Infrastructure

Architecture / Infrastructure

Android

services

Server

Database

App Middleware Platform Connectivity/Network Back-end

Isolated, Hermetic Testing

Android

services

Server

Database

Hermetic Environment

Compatibility & Porting Testing

Android

services

Server

Database

Real Devices

Smoke/Reference Testing

Android

services

Server

Database

Reference Environment

Back-End Performance Testing

Android

services

Server

Database

Massive no. of users and significant load

Continuous Integration

Game Testing Process

Hands-on Example:

Integrating Development with

Testing Environment

• Continuous Integration (e.g. Jenkins)

• Appium framework (for Android and iOS)

• No need for devices – we'll use devices on

Testdroid Cloud service

• Test scripts / test cases for game

Continuous Integration

Appium Intro

Family Tree: Android Test Automation Frameworks

Appium works for both, Android and iOS

• Uses Selenium Webdriver (W3C standard) as a

scripting framework

• Supports native Android, native iOS and mobile

– Android via uiautomator (API level >=16) and

Selendroid (API level <16)

– iOS via UI Automation

– Mobile web as Selenium driver for Android and iOS

• You can write your Appium scripts on almost any

programming language

(Haskell/Go/Clojure/Java/Ruby)

Appium Basics

Test Script

Test Case

Desired Capabilities

“device”: “Android”,

“app”: “/Users/user/ApiDemos.apk”

“app-package”: “com.example.android.apis”

“app-activity”: “.ApiDemos”

Desired Capabilities / Android

Example

Test Script

Test Case

“app”: “com.bitbar.testdroid.BitbarIOSSample”

Example

Desired Capabilities / iOS

Test Script

Test Case

Appium @ Localhost

WebDriver

http://localhost_4723/wd/hubAppium Server

Device

Example

Test Script

Test Case

From Localhost to Testdroid

WebDriver

http://localhost_4723/wd/hub

*Testdroid Caps

http://appium.testdroid.com/wd/hub

Example

Test Script

Test Case

From Localhost to Testdroid

WebDriver {

“testdroid_username”: “user@domain.com”,

“testdroid_password”: “p4s$w0rd”,

“testdroid_project”: “My First Project”,

“testdroid_testrun”: “Test 1”,

“testdroid_device”: “iPad Mini 7.0.4 A1432”,

“testdroid_app”: “http://domain.com/app_v1.ipa”

“app”: “com.bitbar.testdroid.BitbarIOSSample”

Example

350+ Android & 20+ iOS Devices

Example

Projects and Test Runs

AppiumBroker

5 Appium Ready

sessionid

Test Script

Behind the Scene

Device Cluster

Device 1

Device 2

Device 3

1 WebDriver Session Request@ http://appium.testdroid.com/wd/hub/

Desired Caps, .apk / .ipa

2 Configure project

Wait for device to become available

4 Start Appium

Session Map

WebDriver Session response6

sessionid

7 Test Run

sessionid Proxy

Appium

Device 1

Example

Image Recognition

The basic example of Image Recognition

in Mobile Game Testing: Template

Matching

Code Example

• Test Script contains the following 'steps':

1. Identify the used platform (Android / iOS)

2. Search for 'Goldmine' graphical element

(using find_image to search for screen)

3. Script runs test to go to shop

4. Script runs test to buy a cannon

5. Script runs test to place the cannon (all

these three are defined in .png files)

6. Script runs test to start a battle!

7. Game is brought down and test ends.

Example: Clash of Clans

Image Recognition with Real Android and

iOS Devices in Local Environment

Testdroid Update

New UI in Testdroid Cloud 2.0

Testdroid Cloud's Interactive

Testdroid

PrivateCloud -

Running user-

selected devices

at any of our

hosting facilities

NEW HOSTING CENTER!Mountain View, CA

San Francisco

Mountain View

datacenter

Option 1:

Public Device Cloud

on-demand devices

(multitenant)

Mobile app testing on

over 400+ real

Android and iOS

devices hosted by

Bitbar

Option 2:

Private Device Cloud

reserved devices

Hosted by Bitbar in the

US and/or Europe

Devices chosen by and

reserved only for the

Customer

Option 3:

On-premise Device

Automated mobile

app testing on real

Android and iOS de-

vices hosted by the

customer, usually

30-500 devices

Testdroid Deployment Options

Video recording & slides of this webinar will

be soon available at Vimeo and Slideshare!

Please visit at www.testdroid.com for more

information.

THANK YOU!

W E B I N A R

We operate the largest global device

cloud with over 400 devices

instantly available for developers

Devices from the US, Europe, China, Japan and Korea

– global devices for global customers

Testdroid: Best Practices in Mobile Game Testing

Software

Transcript of Testdroid: Best Practices in Mobile Game Testing

Best Practices for Game Localization v21

Cross Platform Development Best Practices Matt Lee, Kev Gee Microsoft Game Technology Group.

Guide to Good Programming and Game Making Practices · PDF fileGuide to Good Programming and Game Making Practices with Game Maker ... Scripts as Functions c) Portability d) ... markov/gmaker/index.html

eBook with Apps and Pedagogical Practices using Game-based ... · eBook with Apps and Pedagogical Practices

LDNSE: Testdroid for Mobile App and Web Testing (London Selenium Meetup)

Examining Literacy Practices in the Game Magic: The Gathering · Examining Literacy Practices in the Game Magic: The Gathering ... (MTG) before I met Paul, who introduced me to the

Testdroid: Build a Large Scale In-House Test Lab for Mobile Apps

Testdroid: Tips and Tricks with Calabash for Mobile App Testing

Cross-Platform Game Development - Intel · Cross-Platform Game Development Best practices learned from Marmalade, Unreal, Unity, etc. ... (Facebook and Twitter -> Share Contracts).

Staying in the Game: Game localization practices for the mobile market

Game Architecture and best practices

Interactional practices and artifact orientation in mobile ...2-3)/PSYCHNOLOGY... · Interactional practices and artifact orientation in mobile augmented reality game play Steven

Five Game Changing Ideas for Re-Shaping Banking Practices

Guide to Good Programming and Game Making Practices with Game Maker

Staying in the Game Six Practices for Helping the Long-Term Unemployed

Testdroid: How to Leverage Appium in Your Mobile App Testing

Video Game Preservation in the UK: Independent …eprints.gla.ac.uk/98882/1/98882.pdfVideo Game Preservation in the UK: Independent Games Developers’ Records Management Practices

Game Journalism- Time to Reconsider Practices?

Best Practices for Game Localization v2 Practices for Game Localization Initial Draft by Richard Honeywood, Vice-Chair Game Localization Special Interest Group (SIG) International

Game Development Best Practices