Mobile App Testing: The Good, the Bad, and the Ugly

4/23/15

1

Mobile App Testing: The Good, The Bad, and The Ugly

Jon D. Hagar, Consultant, Grand Software Testing [email protected]

Author: Software Test Attacks to Break Mobile and Embedded Devices

Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices” 1

*  Gaming Testing Story

*  It only takes a few minutes using an App before users like or hate it

*  Worse than that. . . *  Many users will post a social media review of the app

*  You don’t want to be a BAD

Copyright 2015, Jon D. Hagar Mobile-‐Embedded Taxonomies from “So9ware Test ACacks to Break Mobile and Embedded Devices”

2

The Mobile Opportunity

4/23/15

2

*  Depth

*  Passion

*  Speed

What Does it Take to be a Great Mobile App Tester?

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices 3

*  As the names imply, these are devices—small, held in the hand, connected to communication networks, including *  Cell and smart phones – apps *  Tablets *  Medical devices

*  Typically have: *  Many of the problems of classic embedded systems *  The power of PCs/IT *  More user interface (UI) than classic embedded systems *  Fast and frequent updates

*  However, mobile devices are “evolving” with more power, resources, apps, etc.

*  Mobile is the “hot” area of computers/software *  Testing rules and concepts are still evolving *  Now starting to include IoT

You know what they are right? Mobile and Handheld?


4/23/15

3

*  Requirements verification checking *  Necessary but not sufficient

*  Risk–based testing *  Tried and true in many contexts including mobile, but we need more

Here comes the Good, Bad and Ugly

We Need Better App Testing


The Bad

You are between a Management Rock and a Hard App

Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices” 6

4/23/15

4

*  Management directed “No testing” *  Dev-‐ops without enough “thinking” of context and risk to find the big BUGS *  Stupid requirements verification checking without GOOD test activities *  Testing without thinking of *  cost *  schedule *  users

Con: Current Badness

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 7

*  Are you part of the problem?

*  Do you help management “SEE” the info they need?

*  Are you Agile?

*  Are you using your testing skills daily?

*  Bug are out there (and always will be)…………..

Pro: In the Bad


4/23/15

5

*  From Wikipedia: Taxonomy is the practice and science of classification. The word finds its

roots in the Greek τάξις, taxis (meaning 'order', 'arrangement') and νόμος, nomos ('law' or 'science'). Taxonomy uses taxonomic units, known as taxa (singular taxon). In addition, the word is also used as a count noun: a taxonomy, or taxonomic scheme, is a particular classification ("the taxonomy of ..."), arranged in a hierarchical structure.

*  Helping to “understand and know”


9

A Bad Situation

-‐ Lets look for bugs, but where?


10

Pro: Taxonomy (researched) Super Category

Aero-‐Space Med sys Mobile General Time 3 2 3 Interrupted -‐ Satura>on (over >me)

5.5 Time Boundary – failure resul>ng from incompa>ble system >me formats or values

0.5 1 Time -‐ Race Condi>ons

3 1 Time -‐ Long run usages 4 1 20 Interrupt -‐ >ming or priority inversions

0.7 3 Date(s) wrong/cause problem

0.5 1 Clocks 4 2 Computa>on -‐ Flow 6 23 19 Computa>on -‐ on data 4 1 3 1

4/23/15

6


11

Taxonomy part 2 Super Category

Aero-‐Space Med sys Mobile General Data (wrong data loaded or used) 4 5.00 2 Ini>aliza>on 6 2.00 3 5 Pointers 8 2.00 18 10 Logic and/or control law ordering

8 43 3 30 Loop control –Recursion

1 Decision point (if test structure) 0.5 1 1 Logically Impossible & dead code

0.7 Opera>ng system – (Lack of Fault tolerance , interface to OS, other) 1.5 2 6 Software - Hardware interfaces

16 13 So9ware -‐ Software Interface

5 2.00 3 So9ware -‐ Bad command- problem on server 3 5 UI -‐ User/ operator interface

4 5.00 20 10 UI -‐ Bad Alarm 0.5 3 UI -‐ Training – system fault resul>ng from improper training

3 Other 10.6 9.00 5 5

Note: one report on C/C++ indicated 70% of errors found involved pointers

*  How many of you have a Mobile App taxonomy that you use?

Question


4/23/15

7

The Ugly We need Wisdom, Tooling, and Security

13 Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”

*  Some of you lack mobile tester skills

*  Many of you suffer from group think and lack wisdom *  We listen to the loudest voices

*  Testers do not use available ideas to aide their skill base *  Attacks, techniques, tools, concepts, standards, etc.

Con: Mobile can have an Ugly Face


4/23/15

8

*  Danger of group think in Agile Mobile Teams

*  Amplification

*  Snowballing effect

*  Polarization

*  Ignoring critical minority opinions

Pro: You Need Test Wisdom


*  Stop talking and LISTEN to all sides, particularly the ones you may not agree with *  Question beliefs *  Be passionate and follow your bliss about testing *  Try to remain open minded *  Do not submit to the negatives of group think *  Consider the context of the testing and believe that context matters *  Seek the council of people you believe to be wise *  Reward your test team for being open and providing other views without fear *  Try to take a role of “devil’s advocate” in your test team *  Fight the “me too” syndrome and everyone falling in line to the loudest voice *  Work to be a knowledgeable and skilled tester (they are different) *  Be the voice of loyal opposition in the team and think outside of the group “box” *  Don’t paint a viewpoint as totally invalid, when a few ideas of the viewpoint conflict with

local ideals

Seeking Test Wisdom (Pro: try these tricks)


4/23/15

9

Categories of Automation Tooling (Open Source and Commercial)

* Capture Playback -‐ Actual devices (cabinet vs a pile) vs Emulator -‐ API vs GUI/UI

* Planning and lifecycle support * Modeling

-‐ Risks -‐ Mind-‐mapping -‐ Formal models (UTP) -‐ Test Techniques

Pro/Con? -‐ Mobile/Handheld Test Tools


*  To Automate or Not? *  When testing configurations of hw/sw (good idea) *  When testing combinations (combinatorial test tools) *  When dealing with testing qualities *  Security (very good idea) *  Reliability (necessary) *  Configuration management (can not be done without) *  Usability (important but a hard one and questionable tools)

*  When supporting Development *  Structural testing (measures coverage) *  Static code analysis (finds hard to test bugs) *  Dev-‐Ops, Continuous Integration and Agile (really good)

More on Test Tools – Now in Mobile Support has Improved


4/23/15

10

*  Your app gets on the nightly news

*  Your team sees security as someone else’s problem

Real Ugly: Security and Privacy


*  Mobile– IoT systems are highly integrated hardware–software–system solutions which: *  Must be highly trustworthy since they handle sensitive data *  Often perform critical tasks

*  Security holes and problems abound *  Coverity Scan 2010 Open Source Integrity Report -‐ Android *  Static analysis test attack found 0.47 defects per 1,000 SLOC *  359 defects in total, 88 of which were considered “high risk” in

the security domain

*  OS hole Android with Angry Birds *  Researchers Jon Oberheide and Zach Lanier

*  Robots and Drones rumored to be attacked *  Cars and medical devices being hacked

The Current Security Situation

Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – “So9ware Test ACacks to Break Mobile and Embedded Devices”

4/23/15

11

*  Fraud – Identity *  Worms, virus, etc. *  Fault injection

*  Processing on the run *  Hacks impact *  Power *  Memory *  CPU usage


Con: Mobile Security Bugs (taxonomy)

•  Eavesdropping – “yes everyone can hear you” •  Hijacking •  Click-‐jacking •  Voice/Screen

•  Physical Hacks •  File snooping •  Lost phone

*  A pattern (of testing) based on a common mode of failure seen over and over *  Part of Exploratory Testing *  May be seen as a negative, when it really is a positive *  Goes after the “bugs” that may be in the software *  May include or use classic test techniques and test concepts *  Lee Copeland’s book on test design *  Many other good books

*  A Pattern (more than a process) which must be modified for the context at hand to do the testing

*  Testers learn mental attack patterns working over the years in a specific domain

Pro: Apply Attack-‐based Testing What is an attack?


4/23/15

12

*  Apply when the device is mobile and has *  Account numbers *  User-‐ids and passwords *  Location tags *  Restricted data

*  Current authentication approaches in use on mobile devices *  Server-‐based *  Registry (user/password)

*  Location or device-‐based *  Profile-‐based

Security Attacks


*  Attack 28 Penetration Attack Test *  Attack 28.1 Penetration Sub–Attacks: Authentication — Password *  Attack 28.2 Sub–Attack Fuzz Test *  Attack 29: Information Theft—Stealing Device Data *  Attack 29.1 Sub Attack –Identity Social Engineering *  Attack 30: Spoofing Attacks *  Attack 30.1 Location and/or User Profile Spoof Sub–Attack *  Attack 30.2 GPS Spoof Sub–Attack

Security Attacks (Con: only a starting point, a checklist of things to start with)


4/23/15

13

*  What kind of App software do you work on?

*  Security concerns?

*  Privacy concerns? What is missing?

Exercise


§  Security attacks must be done with the knowledge and approval of owners of the system and software

§  Severe legal implications exist in this area §  Many of these attacks must be done in a lab (sandbox) §  In these attacks, I tell you conceptually how to “drive a car very fast

(150 miles an hour) but there are places to do this with a car legally (a race track) and places where you will get a ticket (most public streets)”

§  Be forewarned -‐ Do not attack you favorite app on your phone or any connected server without the right permissions due to legal implications

Warnings When Conducting Security Attacks


4/23/15

14

Finally, The Good – Functional and Non-‐functional

Experiments and Attacks (Exploratory testing)

Skills App testers should have

Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC –”So9ware Test ACacks to Break Mobile and Embedded Devices” 27

Attacks (from Software Test Attacks to Break Mobile and Embedded Devices)

*  Attack 1: Static Code Analysis *  Attack 2: Finding White–Box Data Computation Bugs *  Attack 3: White–Box Structural Logic Flow Coverage *  Attack 4: Finding Hardware–System Unhandled Uses in Software *  Attack 5: Hw-‐Sw and Sw-‐Hw signal Interface Bugs *  Attack 6: Long Duration Control Attack Runs *  Attack 7: Breaking Software Logic and/or Control Laws *  Attack 8: Forcing the Unusual Bug Cases *  Attack 9 Breaking Software with Hardware and System

Operations *  9.1 Sub–Attack: Breaking Battery Power *  Attack 10: Finding Bugs in Hardware–Software Communications

*  Attack 11: Breaking Software Error Recovery *  Attack 12: Interface and Integration Testing *  12.1 Sub–Attack: Configuration Integration Evaluation *  Attack 13: Finding Problems in Software–System Fault Tolerance *  Attack 14: Breaking Digital Software Communications *  Attack 15: Finding Bugs in the Data *  Attack 16: Bugs in System–Software Computation *  Attack 17: Using Simulation and Stimulation to Drive Software

Attacks *  Attack 18: Bugs in Timing Interrupts and Priority Inversion *  Attack 19: Finding Time Related Bugs

*  Attack 20: Time Related Scenarios, Stories and Tours

*  Attack 21: Performance Testing Introduction *  Attack 22: Finding Supporting (User) Documentation

Problems *  Sub–Attack 22.1: Confirming Install–ability *  Attack 23: Finding Missing or Wrong Alarms *  Attack 24: Finding Bugs in Help Files *  Attack 25: Finding Bugs in Apps *  Attack 26: Testing Mobile and Embedded Games *  Attack 27: Attacking App–Cloud Dependencies *  Attack 28 Penetration Attack Test *  Attack 28.1 Penetration Sub–Attacks: Authentication —

Password Attack *  Attack 28.2 Sub–Attack Fuzz Test *  Attack 29: Information Theft—Stealing Device Data

*  Attack 29.1 Sub Attack –Identity Social Engineering

*  Attack 30: Spoofing Attacks *  Attack 30.1 Location and/or User Profile Spoof Sub–Attack *  Attack 30.2 GPS Spoof Sub–Attack *  Attack 31: Attacking Viruses on the Run in Factories or PLCs *  Attack 32: Using Combinatorial Tests *  Attack 33: Attacking Functional Bugs


4/23/15

15

Attack 1: Static Code Analysis (testing)

*  When to apply this attack? *  After/during coding

*  What faults make this attack successful? *  Many *  Example: Issues with pointers

*  Who conducts this attack? *  Developer, tester, independent party

*  Where is this attack conducted? *  Tool/test lab

*  How to determine if the attack exposes failures? *  Review warning messages and find

true bugs

*  How to conduct this attack *  Obtain and run tool *  Find and eliminate false positive *  Identify and address real bugs *  Repeat as code evolves *  Single unit/object *  Class/Group *  Component *  Full system

29 Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC –”So9ware Test ACacks to Break Mobile and Embedded Devices”

Attack 2: Finding White–Box Data Computation Bugs

*  When to apply this attack? *  After/during coding

*  What faults make this attack successful? *  Mistakes associated with data *  Example: Wrong value of Pi

*  Who conducts this attack? *  Developer, tester, independent party

*  Where is this attack conducted? *  Development Tool/test lab

*  How to determine if the attack exposes failures? *  Structural-‐data test success criteria

not met

*  How to conduct this attack *  Obtain tool *  Determine criteria and coverage *  Create test automation with

specific values (really a programing problem) *  NOT NICE NUMBERS

*  Run automated test cases *  Resolve failures *  Peer check test cases *  Repeat as code evolves

Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC –”So9ware Test ACacks to Break Mobile and Embedded Devices”

4/23/15

16

*  When to apply this attack? …when your app/device has a user *  What faults make this attack successful? …devices are increasingly

complex *  Who conducts this attack? …see chart on Roles *  Where is this attack conducted? …throughout lifecycle and in user’s

environments

*  How to determine if the attack exposes failures? *  Unhappy “users” *  Bugs found *  See sample checklist

Attack : Testing Usability Mobile IoT Usability Tends to be “Poor”


*  Refine checklist to context scope *  Define a role *  Watch what is happening with this role *  Define a usage (many different user roles) *  Guided explorations or ad hoc *  Stress, unusual cases, explore options *  Capture understanding, risk, observations, etc. *  Checklist (watch for confusion of the tester) *  Run Exploratory Attack (s) *  Learn *  Re-‐plan-‐design *  Watch for Bias *  Switch testers *  Repeat


Usability Attack Pattern

4/23/15

17

The Good, Bad, and Ugly of Mobile App Testing

Lots of room for Growth

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”

33

How to be Better after This Section Pick One or Two to work On

Cons: Bad and Ugly

*  Taxonomy help only if you use them

*  Skill improvement *  Knowledge and Skill

*  Security Testing *  Attack, Attack, Attack

Pro: The Good

*  Better and Faster *  Functional testing *  Test strategy and planning

*  Test Attacks

*  Tools and technique maturing

Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices “

34 After Mobile comes IoT

4/23/15

18

*  There will always be Good, Bad, and Ugly *  Work with the Good *  Work to over come the Bad *  Change the Ugly into good

*  Understanding your local context and error patterns is important (one size does NOT fit all)

*  Attacks are patterns…you must still THINK and tailor

Wrap Up of this Session


*  James Whittaker (attacks) *  Elisabeth Hendrickson (simulations) *  Lee Copeland (techniques) *  Brian Merrick (testing) *  James Bach (exploratory and tours) *  Cem Kaner (test thinking) *  Jean Ann Harrison (her thinking and help)

* Many teachers *  Generations past and future *  Books, references, and so on

Notes: Thank You (ideas used from)


4/23/15

19

*  “Software Test Attacks to Break Mobile and Embedded Devices” – Jon Hagar

*  “How to Break Software” James Whittaker, 2003 *  And his other “How To Break…” books

*  “A Practitioner’s Guide to Software Test Design” Copeland, 2004 *  “A Practitioner’s Handbook for Real-‐Time Analysis” Klein et. al., 1993 *  “Computer Related Risks”, Neumann, 1995 *  “Safeware: System Safety and Computers”, Leveson, 1995 *  Honorable mentions: *  “Systems Testing with an Attitude” Petschenik 2005 *  “Software System Testing and Quality Assurance” Beizer, 1987 *  “Testing Computer Software” Kaner et. al., 1988 *  “Systematic Software Testing” Craig & Jaskiel, 2001 *  “Managing the Testing Process” Black, 2002

Book/Notes List (my favorites)


•  www.stickyminds.com – Collection of test info •  www.embedded.com – info on attacks *  www.sqaforums.com -‐ Mobile Devices, Mobile Apps -‐ Embedded Systems Testing forum

•  Association of Software Testing

–  BBST Classes http://www.testingeducation.org/BBST/

•  Your favorite search engine

More Resources

Copyright 2015, Jon D. Hagar Grand So9ware Tes>ng, LLC – So9ware Test ACacks to Break Mobile and Embedded Devices

Mobile App Testing: The Good, the Bad, and the Ugly

Software

Transcript of Mobile App Testing: The Good, the Bad, and the Ugly