Mobile Apps – Challenges to Mobile App Testing
Transcript of Mobile Apps – Challenges to Mobile App Testing
Understanding the “Testing Challenges" in Mobile Application
Presentation Objectives
Understanding the "Testing Challenges" in Mobile Applications by
• differentiating between the different types of Mobile Applications,
• the HCI Guidelines,
• User Expectations,
• Testing Methodologies and some practical examples.
Setting Up the Testing Environment
FACTS
ACCORDING TO SDTIMES SURVEY 6 IN EVERY 7 COMPANIES ARE PLANNING TO DEVELOP A MOBILE APP WITHIN THE NEXT 18 MONTHS
SMART PHONES ARE RAPIDLY BECOMING THE PRIMARY METHOD OF INTERACTION BETWEEN BUSINESS AND CONSUMERS
So What is at Stake?
Types of Mobile Apps1. Mobile Native Apps
Types of Mobile Apps2. Mobile Web Apps
Types of Mobile Apps3. Hybrid Mobile App
Pros and Cons of Mobile Apps
Apple’s HUMAN COMPUTER INTERFACE GUIDELINES
Users have one view at a time
Re-organize how information is to be displayed from the desktop
Minimize number of user controls and input elements
Do not display UI elements that are not absolutely necessary
Provide tap-able items with a minimum dimension of 44 x 44 points
Make it easier for the end user to interact with the system
Apple’s HUMAN COMPUTER INTERFACE GUIDELINES
User standard controls with which users have had prior experience
Steps with which users can go back or retrace their steps.
Users are going to use the application under various distractions
Design process to enable users to complete their tasks as quickly
Minimize text inputs
Use dropdowns/sliders/checkbox where applicable
Android UI Design GUIDELINES
Pictures are faster than words
Consider using pictures to explain ideas
Give me tricks that work everywhere
Make it easier to learn by leveraging design patterns from other Android apps
Sprinkle Encouragement
Give feedback on actions, even if it's a subtle glow or a slight vibration.
The power of App UsersUSERS CAN AND WILL PROVIDE REVIEWS OF YOUR APPLICATION PUBLIC ON APP STORE.
Users Control ReputationFROM THE LIST BELOW WHICH APP IS THE MOST APPEALING TO THE USER?
WHAT FACTORS MAKE THE USERS WANT TO DOWNLOAD THEIR APPLICATION?
USABILITY. USER INTERFACE HAVE TO BE TOP NOTCH. APP SHOULD PERFORM ITS MAIN PURPOSE SEAMLESSLY WITH GREAT ANIMATIONS.
APP SPEED . FUNCTIONAL SPEED ON DIFFERENT TYPES OF MOBILE NETWORKS. [USERS ARE IN GENERAL IMPATIENT.]
DRAINAGE. DRAIN TOO MUCH BATTERY, USERS WILL UNINSTALL THE APPLICATION.
SECURITY. USERS ARE SENSITIVE ON PRIVACY AND TRACKING.
APP AVAILABILITY. IN ALL TYPES OF HANDHELD DEVICES.
UPDATES. USERS WANT THE APP TO BE UPDATED REGULARLY FOR IMPROVEMENTS AND BUG FIXES.
What does the Users Want?
What is the Ultimate Goal?
•A COMPANY’S BRAND STRENGTH AND REPUTATION IS DIRECTLY CO-RELATIONAL TO THEIR MOBILE APPLICATIONS.
Type of App Content
Productivity Apps.
Users using app to organize data such as Personal Finance Apps requiring multiple views.
Type of App Content
Utility Apps.
Users performing single tasks like watching videos on an attractive UI.
Type of App Content
Immersive Apps.
Gaming/simulation Applications providing custom GUI.
TESTING ON MOBILE DEVICES
UI/USABILITY TESTING
The key to this is to make sure that HCI guidelines set forward by Apple/Google are imposed on the applications.
Along with many questions, here are few we should consider asking: Are the buttons too small to click on?*
Does the UI utilize good animations and transition effects?*
Is the application smooth and fast to use?*
Does the UI renders correctly on different OS versions?
Are proper error handling messages displayed*
Supporting Different Screen Resolutions
Using Adaptable Layouts for different screen sizes.
Is the Adaptable Layout justifiable?
There is more space to display more options to the user
What would be the maximum/minimum screen resolution?
The layouts are working properly on different OS versions?
Usability Issues -- 1
A user turns of the app and tries to log in. However, the internet connection is disabled.
Usability Issues -- 1
Let the user know that the internet is turned off, else he/she will think that something is wrong with the app.
Usability Issues -- 2
The user purchased over 150 Courses from the Great Courses website.
Via the API call during Log in to the app, all the Courses and the Course Details were Loaded.
Usability Issues -- 2
This user faced a big problem. It took him over 4/5 minutes to load all the courses.
On top of that, when tested on low configuration devices, the app ended up crashing.
Usability Issues -- 2
Hence in order to rectify the issue, the API call were modified to only send the Course List first.
Afterwards, on course click, each individual courses returned the Lecture list via another API call.
This improved the loading time significantly to less than 30 seconds
Trade-off: App now has to make more calls to the server to retrieve data.
UI/USABILITY TESTING -- Events
TESTING ON MOBILE DEVICES
FUNCTIONAL TESTING
• Are there too many functionalities on-screen to distract the end users?*
• Can the tasks be broken down into simpler parts?
• Are the functionalities working cross platform?*
• Are the functionalities working even when the app is resumed/restarted?
Functional Testing Issue -1
A video is being played by the user.
Then decides to click on the Back Button or Log Out Button.
The video should stop playing for both cases
Functional Testing Issue -2
Usually, during app load various JS files are downloaded from the server.
When the internet was turned off when app is launches many of the JS files were not loaded.
The biggest and most major problem was BrightcoveExperiences.js was not loaded and hence
• No Video or Audio Files were played
• User had to kill app and restart app after activating internet connection to be able to watch videos.
TESTING ON MOBILE DEVICES
INTERRUPT TESTING
How does the application handle:
• Incoming Phone Calls
• Text Messages
• Notifications
• Low Battery Alerts
TESTING ON MOBILE DEVICES
NETWORK TESTING
A decisive factor on App’s performance
• Hence, testing under different/strained networks is a must.*
o Wi-Fi Network [High Speed/low Speed]
o 3G/2G
o Different Carriers
TESTING ON MOBILE DEVICES
CROSS-OS / CROSS-Device TESTING
Does the app performs satisfactorily in
• IOS 5.0/5.1/6.0/6.1/7.0/7.1 in both iPhone & iPad?*
• Android 4.0/4.1.4.2/4.3/Company Specific Rip?*
• Other necessary OS versions on Windows Phones, Blackberry?
TESTING ON MOBILE DEVICES
PERFORMANCE AND STRESS TESTING
• Does the application perform in low configuration devices?*
• What if the user turns on this application while a large number of apps are working on the background?*
• Is there any graphics lag?*
• Does animation slow down the application?*
• Are all user information saved before crashing?
TESTING ON MOBILE DEVICES
RESOURCE ISSUES.
• Does App cache over time leads to UI lags and application crashing?**
• Test application on low configuration emulator/devices*
• Check reliability of the application*
EmulatorsADVANTAGES OF TESTING ON EMULATORS
•Cost effective solution as actual devices are expensive
•Faster internet and has less latency than real devices
•Can be easier to develop and test app in the initial development stages
•Real-time debugging available on iOS Apps and Android Kit Kat
EmulatorsDISADVANTAGES OF TESTING ON EMULATORS
• Can be unreliable as it can cause false impression
• Mobile computing resources are not emulated accurately to be able to make judgment calls
• Mobile/3G Networks affect application behavior
• Can be very slow to use for testing. Can loose chain of thought.
Actual DevicesADVANTAGES OF TESTING ON ACTUAL DEVICES
• Allows Feel and touch sensations which is a big for understanding usability
• Only true way of understanding the user experience, taking into account the CPU, memory, screen size, etc.
• Can test under different mobile Networks as it will affect the app.
• Real-time debugging available ono iOS Apps and,o All upcoming Android versions starting from Android Kit-Kat
Actual DevicesDISADVANTAGES OF TESTING ON ACTUAL DEVICES
• Physical Resources which need to be managed.
• The logistics and costs involved in procuring and managing these devices are significantly high.
Android ChallengesDEVICE SPECIFIC TESTING IS A MUST
Android ChallengesDEVICE SENSITIVE TESTING IS A MUST
• SAMSUNG/NEXUS/HTC ALL USE MODIFIED OS VERSION
• NETWORK/MEMORY CAPACITY/OTHER RESOURCES ARE DIFFERENT ON EACH DEVICE
• DIFFERENT DEVICES USES LOCAL RESOURCES DIFFERENTLY
Android ChallengesDEVICE SENSITIVE TESTING IS A MUST
• IMPRACTICAL TO TEST ON ALL DEVICES.
• HENCE, TEST ON POPULAR MOBILE PHONES –
SAMSUNG S4 / NEXUS 5 / IPHONE 5
• ALSO TEST ON POPULAR TABS –NEXUS 10/SAMSUNG GALAXY TAB/ IPAD 2
Bug Reporting Options
1. SCREENSHOTS CAN BE USED ALONG WITH FACE TO FACE COMMUNICATION WITH DEVELOPERS TO VISUALLY GENERATE THE GLITCH FOR BETTER UNDERSTANDING.
PROS AND CONS
• IT IS THE FASTEST WAY TO REPORT BUGS.
• TOUCH AND FEEL SENSATIONS CANNOT BE REPRESENTED PROPERLY.
• GENERATE ISSUE IN FRONT OF DEVELOPER TO EXPLAIN THE TOUCH AND FEEL FACTOR PROPERLY.
WHAT ARE THE OPTIONS?
Bug Reporting Options
2. USING ANDROID SCREEN RECORD APPS
A SHORT STORY OF THE SCENARIO FOLLOWED BY THE SCREEN RECORDED SCENE.
PROS AND CONS
• TIME CONSUMING AND VIDEOS ARE LARGE AND LAGGY
• CAN GENERATE REAL SCENARIO FOR DEVELOPER TO VISUALIZE WITH TOUCH GESTURE RECORDING
• HOWEVER, CAN BE CONFUSING TO INTERPRET.
PROBABLE OPTIONS
Setting Up the Test Environment1. Java Installation
• Download and Install Java JRE from
• http://www.oracle.com/technetwork/java/javase/downloads/jre7-downloads-1880261.html
• Add java JRE bin folder to the Path Variables
• Download and Install Java JDK from
https://jdk7.java.net/download.html
• Add java JDK bin folder to the Path Variables
Setting Up the Test Environment2. ANDROID-SDK
• Download SDK from
http://developer.android.com/sdk/index.html?hl=sk
• Unzip the download on C:\
Setting Up the Test Environment
• Add Android SDK tools folder to the Path Variables
Setting Up the Test Environment
3. Apache Ant
[Java-based build tool]
Download from http://ant.apache.org/bindownload.cgi
Unzip to any location. Eg. E:\Tools\Project\Build\apache-ant-x.x.x
Open environment variables (Control Panel » System » Advanced » Environment Variables) add a new system variable as displayed below,
Setting Up the Test Environment
Set the %ANT_HOME%\bin; to the path as displayed below,
Verify the installation of Ant by typing the following command in the command prompt,
Setting Up the Test Environment
4. NODEJS
[ Software platform for scalable server-side and networking applications]
Download NODEJS from http://nodejs.org/download/
Run command prompt
If nodejs is installed then npm command will be recognized
Phone-Gap /Cordova
Phone-Gap/Cordova are application framework that allows developers to use HTML, JavaScript and CSS to create apps for Mobile Market Place.
Adobe donated the code-base of Phone-Gap to Apache, which then renamed it Cordova.
This was done so that PhoneGap can reap the benefits of open source, while at the same time having the ability to be different.
Setting Up the Test Environment
5. Cordova/Phone-Gap
Open command prompt and type
Setting Up the Test Environment
Already have a MAC VM with all the necessary environments pre-installed to be able to start testing immediately
Thank You