How to Reliably Measure and Optimize Graphics Performance of Your Android Games
47
11 August 2015 Ville-Veikko Helppi Technical Product Manager ville- [email protected] How to Reliably Measure and Optimize Graphics Performance of Your Android Games W E B I N A R Sharif Sakr Director of Business Dev. [email protected]
-
Upload
bitbar -
Category
Technology
-
view
599 -
download
0
Transcript of How to Reliably Measure and Optimize Graphics Performance of Your Android Games
11 August 2015
Ville-Veikko HelppiTechnical Product [email protected]
How to Reliably Measure and Optimize Graphics Performance of Your Android Games
W E B I N A R
Sharif SakrDirector of Business [email protected]
Agenda• Testdroid Statistics from Q2'2015• Testing Mobile Games – Why and
How?• Optimize Graphics Performance
with Help of Gamebench - Introduction
• Demonstration• Q&A
TESTDROID STATISTICS
Q2'2015 – Test Run
Stats
Why Mobile Games
Fail?
iOS 8.0
iOS 8.0.2
iOS 8.1
iOS 8.1.1
iOS 8.2
iOS 8 ALL
KitKat 4.4
KitKat 4.4.2
KitKat 4.4.3
KitKat 4.4.4
KitKat ALL
Lollipop 5.0
Lollipop 5.0.1
Lollipop 5.0.2
Lollipop ALL
Android 'M'
0 5 10 15 20 25 30 35
Failed test runs. Percentage (%).
TREND
TREND
TREND
Some Observations1
2
Some Observations3
4
Some Observations5
6
Android Fragmentation
2015
by Open Signal report August 2015
Testdroid Cloud
>95%
Android Fragmentation
2015
by Open Signal report August 2015
Testdroid Cloud +
Gamebench
>85%
How Many Devices is
Enough?
~90%market coverage can
be achieved with
128 devices
~20%market coverage can
be achieved with
14 devices
US Market
30 Android devices= ~2/3 market
Global Market
80 Android devices= ~1/2 market
TESTING MOBILE GAMES
Correct behaviour across platforms and browsers
Integration with web back-ends
Typically need to fully utilize HW (CPU+GPU)
Resource (e.g. battery) consumption
OpenGL ES 2/3
Performance!
Functionality and usability
Screen orientations, connectivity, user profiles
Robustness
Robustness and security!
Brand
Compliances, verification with back-
ends and data
Different 'Verticals'
Manual Testing is not
Agile
Test Automation is
Agile
OS versions
ChipsetsCPU + GPU
Tens of OEMs
MemoryDisplays
(resolutions, physical hw)
OEM mods
Other hardware (connectivityc
alibration)
Relation to other software
Where Test Automation Can
Help
Why Real Devices Are Must-
to-Have• Emulators cannot help you to test...
• User Experience • Usability• Hardware• Software• Infrastructure
0 % = the percentage of your app users that use emulator to run your app!
Why Testing Games
Matter?• Graphics performance
capability is the most important thing for mobile games when forming UX
• Too many games fail to work properly with low-end/mid-range devices due hardware performance
• The real understanding of game's performance capability can be only figured out by testing on real devices
UnderstandingHow quickly gamer assimilates what the game is about, how to fully play with it, what are the functionalities.
UsabilityThe main User Interface, menus, design
and how usable those are
Playing ExperienceIntuition to follow the game and go
forward with its instructions and game-play
Monetization
Are gamers willing to buy – invest – in game
Types of Game Testing• Functionality of the game-play, featuresFunctional Testing• Done always when new features /regressions are
includedRegression Testing
• How game runs on different configurationsCompatibility Testing
• Different languages, geo-focused materialsLocalization Testing• Endurance test to determine if system can handle
the loadSoak Testing
• Measures the capacity of the system Stress Testing• Simplest form of performance testing, measures
how system handles certain loadsLoad Testing• Isolation of the environment (e.g. from network)
to see how game worksHermetic Testing
Feature-based testing
Performance testing
End-user testing
Gameflow -
Performance• Compared to regular app testing, mobile game
testing may seem to be more challenging• e.g. OpenGL ES content not action-identifiable
• Graphics and UI – Identical look&feel across all devices and variants
• Usability controls in the context of game• Navigation/game-play, Progress, Delays
between "game-stages"• Animations, Fonts, Other graphics assets• Screen Resolution and Orientation
• Studies: Majority of users leave game due sluggish, general slowness or other bad performance behavior
• It's very important to understand how game behaves on different devices, under different circumstances
Performance Makes +/-
UX
Performance Testing• Performance, Load, Stress, Spike, Soak
• Measure the real user experience
• Execute tests with all possible devices (SW&HW)
• Discover the impact of server loads, app performance (CPU/GPU utilization)
Components in Game
Testing
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Starts at Design Level
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Game Design, GFX Assets,Files, Optimization
Performance Testing
w/HW
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Real Devices and Platform
Compatibility Testing
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Variations
Reference Testing
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Back-End Performance
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Game with Back-End Integration / Google Play Services
Game Testing Process
Trends in Mobile Game
Testing• Image recognition
15 seconds
GAMEBENCH
A traffic light system for mobile apps and devices
50%
People notice performance
Performance = promotion
Thousands of developers and
enthusiasts
OEM and Chip makers
Developers – Games, Apps, VR
Journalists
Testing / QA
40k+ downloads in less than 1 yr since launch,
>250 users any day
Market traction
App and Game Developers
Chip makers OEM
Pre-release phase
In market / production
Consumers
Journalists and
Enthusiasts
Used to certify performance
before release
Used by retailers to match consumers’ usage profiles
to mobile devices
Helps with device and app performance
reviews
Copyright GameBench Limited
Q&A
Who are the people behind GameBench
•Sri Kannan Iyer: CEO & Founder
(London)•Over 15 years experience in the mobile •ARM, MediaTek, Analog Devices and Intel•MSEE (UT Austin), MBA (Oxford)•4 patents, 1 book chapter, speaker at several conferences
•Karthik Hariharakrishnan: CTO & Founder (Bristol)•Over 10 years in mobile engineering•ARM, Motorola, National Instruments•MSEE (University of Illinois), 3 patents
Six employees based in Bristol & London
Get in touch!
DemonstrationGamebench with Real Devices from
Testdroid Cloud
THANK YOU!More information about frameworks,
devices and testing at www.testdroid.com
