Mobile App Test Attacks to Efficiently Explore Software
-
Upload
eurostar-software-testing-conference -
Category
Software
-
view
81 -
download
0
Transcript of Mobile App Test Attacks to Efficiently Explore Software
Mobile App Test Attacks to Efficiently Explore Software
Jon D. Hagar, Consultant, Grand Software Testing
[email protected]: Software Test Attacks to Break
Mobile and Embedded Devices
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices” 1
Gaming Testing Story
It only takes a few minutes using an App before users like or hate it
Worse than that. . . Many users will post a social media review of the app
You don’t want to be a BAD
Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
2
The Mobile Opportunity
Depth
Passion
Speed
What Does it Take to be a Great
Mobile App Tester?
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices
3Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices
3
As the names imply, these are devices—small, held in the hand, connected to communication networks, including Cell and smart phones – apps Tablets Medical devices
Typically have: Many of the problems of classic embedded systems The power of PCs/IT More user interface (UI) than classic embedded systems Fast and frequent updates
However, mobile devices are “evolving” with more power, resources, apps, etc.
Mobile is the “hot” area of computers/software Testing rules and concepts are still evolving Now starting to include IoT
You know what they are right?
Mobile?
Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
Requirements verification checking Necessary but not sufficient
Risk–based testing Tried and true in many contexts including mobile, but we
need more
We need to do more as testers
We Need Better App Testing
Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
Management directed “No testing”
Dev-ops without enough “thinking” of context and risk to find the BUGS that “count”
Stupid requirements verification checking without GOOD supporting test activities
Testing without thinking of cost schedule users
Current Situation in Mobile Projects
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
6
From Wikipedia:
Taxonomy is the practice and science of classification. The word finds its roots in the Greek τάξις, taxis (meaning 'order', 'arrangement') and νόμος, nomos ('law' or 'science'). Taxonomy uses taxonomic units, known as taxa (singular taxon). In addition, the word is also used as a count noun: a taxonomy, or taxonomic scheme, is a particular classification ("the taxonomy of ..."), arranged in a hierarchical structure.
The attacks of this session are based on a researched TaxonomyCopyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
7
Lets look for bugs, but where?
A pattern (of testing) based on a common mode of failure seen over and over Part of Exploratory Testing May be seen as a negative, when it really is a positive Goes after the “bugs” that may be in the software May include or use classic test techniques and test
concepts Lee Copeland’s book on test design Many other good books
A Pattern (more than a process) which must be modified for the context at hand to do the testing
Testers learn mental attack patternsworking over the years in a specific domain
Apply Attack-based TestingWhat is an attack?
Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
A Sampling of Attacks (from Software Test Attacks to Break Mobile and Embedded
Devices) Attack 1: Static Code Analysis Attack 2: Finding White–Box Data Computation Bugs
Attack 3: White–Box Structural Logic Flow Coverage Attack 4: Finding Hardware–System Unhandled Uses in
Software Attack 5: Hw-Sw and Sw-Hw signal Interface Bugs Attack 6: Long Duration Control Attack Runs Attack 7: Breaking Software Logic and/or Control Laws Attack 8: Forcing the Unusual Bug Cases Attack 9 Breaking Software with Hardware and System
Operations 9.1 Sub–Attack: Breaking Battery Power Attack 10: Finding Bugs in Hardware–Software
Communications Attack 11: Breaking Software Error Recovery Attack 12: Interface and Integration Testing 12.1 Sub–Attack: Configuration Integration Evaluation
Attack 13: Finding Problems in Software–System Fault Tolerance
Attack 14: Breaking Digital Software Communications
Attack 15: Finding Bugs in the Data Attack 16: Bugs in System–Software Computation Attack 17: Using Simulation and Stimulation to Drive
Software Attacks Attack 18: Bugs in Timing Interrupts and Priority Inversion Attack 19: Finding Time Related Bugs
Attack 20: Time Related Scenarios, Stories and Tours
Attack 21: Performance Testing Introduction Attack 22: Finding Supporting (User)
Documentation Problems Sub–Attack 22.1: Confirming Install–ability Attack 23: Finding Missing or Wrong Alarms Attack 24: Finding Bugs in Help Files Attack 25: Finding Bugs in Apps Attack 26: Testing Mobile and Embedded Games
Attack 27: Attacking App–Cloud Dependencies
Attack 28 Penetration Attack Test Attack 28.1 Penetration Sub–Attacks:
Authentication — Password Attack Attack 28.2 Sub–Attack Fuzz Test Attack 29: Information Theft—Stealing Device
Data Attack 29.1 Sub Attack –Identity Social
Engineering Attack 30: Spoofing Attacks Attack 30.1 Location and/or User Profile Spoof
Sub–Attack Attack 30.2 GPS Spoof Sub–Attack Attack 31: Attacking Viruses on the Run in
Factories or PLCs Attack 32: Using Combinatorial Tests Attack 33: Attacking Functional Bugs
Copyright 2015, Jon D. Hagar Mobile-Embedded Taxonomies from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 1: Static Code Analysis (testing)
When to apply this attack? After/during coding
What faults make this attack successful? Many Example: Issues with pointers
Who conducts this attack? Developer, tester, independent
party Where is this attack conducted?
Tool/test lab How to determine if the attack
exposes failures? Review warning messages and find
true bugs
How to conduct this attack? Obtain and run tool Find and eliminate false
positive Identify and address real
bugs Repeat as code evolves
Single unit/object Class/Group Component Full system
10
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 2: Finding White–Box Data Computation Bugs
When to apply this attack? After/during coding
What faults make this attack successful? Mistakes associated with data Example: Wrong value of Pi
Who conducts this attack? Developer, tester, independent party
Where is this attack conducted? Development Tool/test lab
How to determine if the attack exposes failures? Structural-data test success criteria
not met
How to conduct this attack? Obtain tool Determine criteria and
coverage Create test automation with
specific values (really a programing problem) NOT NICE NUMBERS
Run automated test cases Resolve failures Peer check test cases Repeat as code evolves
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 3: White–Box Structural Logic Flow Coverage
When to apply this attack? After/during coding
What faults make this attack successful? Many Example: Statement coverage
Who conducts this attack? Developer, tester, independent
Where is this attack conducted? Tool/test lab
How to determine if the attack exposes failures? Coverage not met and/or success
criteria fails
How to conduct this attack? Obtain tool Determine criteria and
coverage Create test automation with
specific values to drive logic flow within code
Run automated test cases Resolve failures Peer check test cases Repeat as code evolves
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 4: Finding Hardware–System Unhandled User Cases
When to apply this attack? Starting at system-software analysis
What faults make this attack successful? Lack of understand of the world Example: Car braking on ice
Who conducts this attack? Developer, tester, analyst
Where is this attack conducted? Environments, simulations, field
How to determine if the attack exposes failures? An unhandled condition exist Note: data explosion problem
How to conduct this attack? Knowledge Out-of-box thinking Operation Concepts Analysis Modeling Lab testing Field testing Feedback Repeat
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 22 and 24: Finding Supporting (User) Documentation and Help File
Problems
14
When to apply this attack? As soon as user documents exist
What faults make this attack successful? Incorrect information about how
to “use” the app Who conducts this attack?
Tester, independent party, stakeholders
Where is this attack conducted? Conduct on the online or
hardcopy documents How to determine if the
attack exposes failures? Follow the instructions exactly
and determine if system works
How to conduct this attack? Access the documentation Use instructions to create a user
story Play the role of different
personas Consider giving the
documentation to a independent party
Repeat as document and systems change
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 22.1: Confirming Installability
15
When to apply this attack? When installation is available
What faults make this attack successful? “Missing” part and/or incorrect
configurations Configurations of hardware and
software may not support the app (Device fragmentation)
Who conducts this attack? Tester, independent party
Where is this attack conducted? Tool/test lab, field
How to determine if the attack exposes failures? System fails to install or run
correctly after install
How to conduct this attack? Obtain “clean” device/system (s) Identify load procedures Note: if doing device configuration
operability test use of techniques such as combinatorial or market penetration identification may be needed
Define test strategy and plan Define test design Automate if needed Execute test (follow load procedures) Confirm load and use configuration Repeat as needed
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 23: Finding Missing or Wrong Alarms
When to apply this attack? Device has alarms or
information notifications to drive user interaction
What faults make this attack successful? Time or other interactions cause
notification-alarm to be missed Who conducts this attack?
Tester, independent party Where is this attack
conducted? Tool/test lab, field
How to determine if the attack exposes failures? Alarm is missed or wrong
How to conduct this attack? Define alarms and conditions Define risks of alarms in usage
and time Define strategy and test plan Define use cases Define test design within
environments including time Run tests Review for missing/wrong alarms
and cases to “force” Leap year
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
When to apply this attack? …all the time What faults make this attack successful? …apps can be quite
complex Example: Games-Entertainment ( 40-60 % of downloads)
Who conducts this attack? Test Team A-B “user” testing (crowd, Beta, early releases in continuous
integration/Deployment, etc) Where is this attack conducted? …throughout lifecycle and in
environments How to determine if the attack exposes failures?
Unhappy “users” Bugs found See checklist
Attack : Testing Usability
Credit to Jean Ann Harison2013Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
The developer(s)—see Attacks 1, 2, and 3. The app architect or director On-team tester(s) In-company “dog food” testers Independent test players Mass beta trials Not a tester—Finally, consider who should not be
playing
Note on roles: During the testing effort and as it progresses, don’t forget that there are many different user roles
Exercise: WHAT ARE THE ROLES?
Roles in Usability
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Refine checklist to context scope Define a role
Watch what is happening with this role Define a usage (many different user roles)
Guided explorations or ad hoc Stress, unusual cases, explore options Capture understanding, risk, observations,
etc. Checklist (watch for confusion)
Run Exploratory Attack(s) Run A-B statistical Test with monitoring
Learn Re-plan/design
Watch for Bias Switch testers
Repeat
Usability Attack Pattern
Copyright 2015 Jon D. Hagar excerpted from “Software Test Attacks to Break Mobile and Embedded Devices”
Apply when the device is mobile and has Account numbers User-ids and passwords Location tags Restricted data
Current authentication approaches in use on mobile devices Server-based
Registry (user/password) Location or device-based Profile-based
My Personal Pet CauseSecurity Attacks
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
Attack 28 Penetration Attack Test Attack 28.1 Penetration Sub–Attacks: Authentication —
Password Attack 28.2 Sub–Attack Fuzz Test Attack 29: Information Theft—Stealing Device Data Attack 29.1 Sub Attack –Identity Social Engineering Attack 30: Spoofing Attacks Attack 30.1 Location and/or User Profile Spoof Sub–Attack Attack 30.2 GPS Spoof Sub–Attack
Security Attacks (Con: only a starting point, a checklist of things to
start with)
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
Security attacks must be done with the knowledge and approval of owners of the system and software
Severe legal implications exist in this area Many of these attacks must be done in a lab (sandbox) In these attacks, I tell you conceptually how to “drive a car very fast
(150 miles an hour) but there are places to do this with a car legally (a race track) and places where you will get a ticket (most public streets)”
Be forewarned - Do not attack you favorite app on your phone or any connected server without the right permissions due to legal implications
Warnings When Conducting Security
Attacks
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
There will always be Good, Bad, and Ugly Work with the Good Work to over come the Bad Change the Ugly into good
Understanding your local context and error patterns is important
(one size does NOT fit all)
Attacks are patterns…you must still THINK and tailor
Wrap Up of this Session
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
James Whittaker (attacks) Elisabeth Hendrickson (simulations) Lee Copeland (techniques) Brian Merrick (testing) James Bach (exploratory and tours) Cem Kaner (test thinking) Jean Ann Harrison (her thinking and help)
Many teachers Generations past and future Books, references, and so on
Notes: Thank You (ideas used from)
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
“Software Test Attacks to Break Mobile and Embedded Devices”
– Jon Hagar
“How to Break Software” James Whittaker, 2003 And his other “How To Break…” books
“A Practitioner’s Guide to Software Test Design” Copeland, 2004 “A Practitioner’s Handbook for Real-Time Analysis” Klein et. al.,
1993 “Computer Related Risks”, Neumann, 1995 “Safeware: System Safety and Computers”, Leveson, 1995 Honorable mentions:
“Systems Testing with an Attitude” Petschenik 2005 “Software System Testing and Quality Assurance” Beizer, 1987 “Testing Computer Software” Kaner et. al., 1988 “Systematic Software Testing” Craig & Jaskiel, 2001 “Managing the Testing Process” Black, 2002
Book/Notes List (my favorites)
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – “Software Test Attacks to Break Mobile and Embedded Devices”
• www.stickyminds.com – Collection of test info• www.embedded.com – info on attacks www.sqaforums.com - Mobile Devices, Mobile
Apps - Embedded Systems Testing forum
• Association of Software Testing– BBST Classes
http://www.testingeducation.org/BBST/
• Your favorite search engine
More Resources
Copyright 2015, Jon D. Hagar Grand Software Testing, LLC – Software Test Attacks to Break Mobile and Embedded Devices