Post on 13-Apr-2017
Taming the Testing Beast
Marc J. BalcerChief ArchitectModel Compilers LLC
How do you knowwhat to test?
Test Case
TAME – Test Authoring Made Easy
Test Workshee
t TAME
Test CaseTest
CaseTest CaseTest
Case
Excel Open Office
Test Case
Employee
Add an expense to an expense report
Employee
Add an expense to an expense report
Test Case
Expense TypeExpense DateAmountNotes
InputsExpense AddedNo Date ErrorBad Date ErrorAmount Missing ErrorAmount Too Large Warning
Expected Results
“Stateless”
Test Case
Expense TypeExpense DateAmountNotes
InputsExpense AddedNo Date ErrorBad Date ErrorAmount Missing ErrorAmount Too Large WarningNo Project ErrorEmployee Error
Expected Results
Project ExistsEmployee CurrentOther Items Exist
Preconditions
“Stateful”
Test Case
TAME – Test Authoring Made Easy
Test Workshee
t TAME
Test CaseTest
CaseTest CaseTest
CaseInputsPreconditionsExpected Results
Test Suite
How do you geta good set oftest cases?
Example: A Login Page
Inputs
Generic Login Command
Input User ID FieldEmptyToo longHas invalid charactersValid Syntax
Input Password FieldEmptyToo longValid Syntax
Inputs CharacteristicValues
(“choices”)
Kinds of Input ValuesNumeric Symbolic Enumerated Composite
A number
Specified as• Range• Precision
(number of decimal places)
May include a unit of measure
A character string
Specified as• Size• Pattern
(regular expression or equivalent)
A list of values
Specified as the set of values
May be ordered or unordered
May have a corresponding numeric or symbolic value
A single value composed of other values
Examples:• Mailing
Address• Grid
Coordinate
Numeric Type ExamplesType Name Description ExamplesYear A four-digit integer between
1900 and one plus the current year
2012 is good1832 is not good342.34 is not good“huh” is not good
Money A US Dollar value between 0 and 1000.00 to a precision of 2 decimal places.
345.23 is good-60 is not good34.219 is not good
Count A nonnegative number used to indicate a count of items between 0 and 100.
46 is good245 is not good-10 is not good
Date A calendar date to the precision of one day
15-Jan-2010 is good4/3/2010 is good*June 6 is not good
* How this is interpreted depends upon whether you are in the US or Europe…
Symbolic Type ExamplesType Name Description ExamplesISBN Book Number
10 or 13 decimal digitsThe last character may be an X instead of a zero.
0-201-74804-5 is good4-87197-308-5 is good9781492217947 is good9321G4E3 is not good
ISO Country Code
2 capital letters US is goodGB is goodEUR is not good
Order Number 4 digits plus a dash plus 6 digits:[0-9]{4}\-[0-9]{6}(or \d{4}-\d{6})
2014-002432 is good1986-231321 is good503-3324-324 is not good
Title String 1 to 100 characters, including non-ASCII (foreign language) characters
Pretty much anything as long as it’s not more that 100 characters long.
Common Input Values• Errors
• Missing (empty, null, nothing, unchecked, …)
• Too short• Too long• Wrong characters (bad syntax)• Out of range
• Acceptable Values• Good syntax• Special characters
(e.g. quotes, < > characters)• Non-ASCII and non-Latin characters
Generate All Combinations
Results
Generic Login Command
Error message "Empty User ID" is displayed
User ID Too Long
User ID has invalid characters
Empty Password
Password Too Long
User ID or password incorrect
User ID or password incorrect
User logged in
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax x x x
Input Password Field
Empty xToo long xValid Syntax x x x
Preconditions
Generic Login Command
Input User ID FieldEmptyToo longHas invalid charactersValid Syntax
Input Password FieldEmptyToo longValid Syntax
Condition User
no user with the user IDuser with the user ID
Condition User Password
does not match the user IDmatches the user ID
Facts aboutthe environment(usually set up in advance)
Forming Combinations
Generic Login Command
Input User ID FieldEmptyToo longHas invalid charactersValid Syntax
Input Password FieldEmptyToo longValid Syntax
Condition User
no user with the user IDuser with the user ID
Condition User Password
does not match the user IDmatches the user ID
4 choices
3 choices
2 choices2 choices
4 × 3 × 2 × 2 = 48 test cases
All Combinations = Many Useless Test CasesRedundant (duplicate)
User ID emptyPassword too longUser: none with the user IDPassword: does not match the user ID Result: Empty User ID errorUser ID: emptyPassword: good syntaxUser: none with the user IDPassword: does not match the user ID Result: Empty User ID error
Make No Sense
User ID emptyPassword too longUser: user with the user IDPassword: does not match the user IDUser ID bad syntaxPassword too longUser: user with the user IDPassword: does not match the user ID
Define Results
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
User ID or password incorrect error
User ID or password incorrect error
User logged in
Input User ID Field
Empty Too long Has invalid characters Valid Syntax
Input Password Field
Empty Too long Valid Syntax
Condition User
no user with the user ID user with the user ID
Condition User Password
does not match the user ID matches the user ID
Expected Results
Simple Result Marking
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
User ID or password incorrect error
User ID or password incorrect error
User logged in
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax
Input Password Field
Empty xToo long xValid Syntax
Condition User
no user with the user ID user with the user ID
Condition User Password
does not match the user ID matches the user ID
Combination Results
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
User ID or password incorrect error
User ID or password incorrect error
User logged in
Input User ID Field
Empty Too long Has invalid characters Valid Syntax x
Input Password Field
Empty Too long Valid Syntax x
Condition User
no user with the user ID user with the user ID x
Condition User Password
does not match the user ID matches the user ID x
Multiple Paths to the Same Result
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
User ID or password incorrect error
User ID or password incorrect error
User logged in
Input User ID Field
Empty Too long Has invalid characters Valid Syntax x x
Input Password Field
Empty Too long Valid Syntax x x
Condition User
no user with the user ID xuser with the user ID x
Condition User Password
does not match the user ID xmatches the user ID
same
Fully Marked Worksheet
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
User ID or password incorrect error
User ID or password incorrect error
User logged in
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax x x x
Input Password Field
Empty xToo long xValid Syntax x x x
Condition User
no user with the user ID xuser with the user ID x x
Condition User Password
does not match the user ID xmatches the user ID x
Forming Test Cases
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax
Input Password Field
Empty xToo long xValid Syntax
Condition User
no user with the user ID user with the user ID
Condition User Password
does not match the user ID matches the user ID
Test Case 1Inputs: User ID Field : Empty Expected Results: Empty User ID error------------------------------------Test Case 2Inputs: User ID Field : Too long Expected Results: User ID Too Long error------------------------------------Test Case 3Inputs: User ID Field : Has invalid characters Expected Results: User ID has invalid characters error
No Mark – Don’t Care
Generic Login Command
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax
Input Password Field
Empty xToo long xValid Syntax
Condition User
no user with the user ID user with the user ID
Condition User Password
does not match the user ID matches the user ID
Test Case 1Inputs: User ID Field : Empty Expected Results: Empty User ID error
Any choice for passwordwill result in this sameerror message.
Default Choice
Select a default if you want to put that “don’t care” category into the test case.
Combine All Choices
Select all choices if you want to combine all choices into test cases.
Alternatively check the category name.
Multiple Paths to the Same Result
Generic Login Command
User ID or password incorrect error
User ID or password incorrect error
Input User ID Field
Empty Too long Has invalid characters Valid Syntax x x
Input Password Field
Empty Too long Valid Syntax x x
Condition User
no user with the user ID xuser with the user ID x
Condition User Password
does not match the user ID xmatches the user ID
Test Case 6 Preconditions: User : no user with the user ID Inputs: User ID Field : Valid Syntax Password Field : Valid Syntax Expected Results: User ID or password incorrect-----------------------------------------------Test Case 7 Preconditions: User : user with the user ID User Password : does not match the user ID Inputs: User ID Field : Valid Syntax Password Field : Valid Syntax Expected Results: User ID or password incorrect
Combination Results
Generic Login Command
User logged in
Input User ID Field
Empty Too long Has invalid characters Valid Syntax x
Input Password Field
Empty Too long Valid Syntax x
Condition User
no user with the user ID user with the user ID x
Condition User Password
does not match the user ID matches the user ID x
Test Case 8 Preconditions: User : user with the user ID User Password : matches the user ID Inputs: User ID Field : Valid Syntax Password Field : Valid Syntax Expected Results: User logged in
Multiple CombinationsGeneric Login Command (With Special
Inputs)
User logged in
Input User ID FieldEmptyToo longHas invalid charactersValid Syntax xValid with non-ASCII chars x
Input Password FieldEmptyToo longValid Syntax xValid with quotes xValid with HTML chars xValid with non-ASCII chars x
Condition Userno user with the user IDuser with the user ID x
Condition User Passworddoes not match the user IDmatches the user ID x
Form all combinationsof these inputs.
2 × 4= 8 test cases
LimitsGeneric Login Command (With Special
Inputs)LIMITS
User logged in
Input User ID Field
Empty Too long Has invalid characters Valid Syntax xValid with non-ASCII chars 1 x
Input Password Field
Empty Too long Valid Syntax xValid with quotes 1 xValid with HTML chars 1 xValid with non-ASCII chars 1 x
Condition User
no user with the user ID user with the user ID x
Condition User Password
does not match the user ID matches the user ID x
Each choice will only beused in one test case.
Generating Test Cases
Test CaseTest
Worksheet TAME
Test CaseTest
CaseTest CaseTest
Case
Excel Open Office
Test Case List
Test SummaryTable
Test ProtocolDocument
Test Case ListTest Cases for Generic Login Command (With Special Inputs)----------------------------------------------------Test Case 1 Preconditions: Inputs: User ID Field : Empty Expected Results: Empty User ID error----------------------------------------------------Test Case 2 Preconditions: Inputs: User ID Field : Too long Expected Results: User ID Too Long error----------------------------------------------------Test Case 3 Preconditions: Inputs: User ID Field : Has invalid characters Expected Results: User ID has invalid characters error
Test Summary TableUser ID Field Password Field User User Password Results...
1Empty Empty User ID error2Too long User ID Too Long error
3Has invalid characters
User ID has invalid characters error
4 Empty Empty Password error5 Too long Password Too Long error6Valid Syntax Valid Syntax no user with the user ID User ID or password incorrect
error
7Valid Syntax Valid with quotes no user with the user ID User ID or password incorrect error
8Valid Syntax Valid with HTML charsno user with the user ID User ID or password incorrect error
9Valid Syntax Valid Syntax user with the user ID does not match the user ID User ID or password incorrect error
10Valid Syntax Valid with quotes user with the user ID does not match the user ID User ID or password incorrect error
11Valid Syntax Valid with HTML charsuser with the user ID does not match the user ID User ID or password incorrect error
12Valid Syntax Valid Syntax user with the user ID matches the user ID User logged in
13Valid Syntax Valid with quotes user with the user ID matches the user ID User logged in
14Valid Syntax Valid with HTML charsuser with the user ID matches the user ID User logged in
Default Protocol Document
Creating a Custom ProtocolHow to log in:• Click a link to go to the login page• Enter the user ID• Go to the password field• Enter the password• Click the login button• Observe whether
• an error message is displayed, or• the application indicates that the user is
logged in
Creating a Custom ProtocolHow to log in:• Click a link to go to the login page• Enter the user ID• Go to the password field• Enter the password• Click the login button• Observe whether
• an error message is displayed, or• the application indicates that the user is
logged in
We want real valuesfor each of these parameters.
Defining Values
Generic Login Command (With Special Inputs)
LIMITS
Empty User ID error
User ID Too Long error
User ID has invalid characters error
Empty Password error
Password Too Long error
User ID or password incorrect error
User ID or password incorrect error
User logged in
User ID:
User ID: abcdefghijklm
User ID: 56%#@!*(a
User ID: gary*
User ID: charlie
User ID: garcía
Password:
Password: 56%#@!*(a
Password: Good*
Password: Bad
Password: G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty x xToo long x xHas invalid characters x xValid Syntax x x x x xValid with non-ASCII chars 1 x x
Input Password Field
Empty x xToo long x xValid Syntax x x x x xValid with quotes 1 x x x xValid with HTML chars 1 x x x xValid with non-ASCII chars 1 x x
Condition User
no user with the user ID x xuser with the user ID x x x x x x x
Condition User Password
does not match the user ID x xmatches the user ID x x x
Defining Values
User ID:
User ID: abcdefghijklm
User ID: 56%#@!*(a
User ID: gary*
User ID: charlie
User ID: garcía
Password:
Password: 56%#@!*(a
Password: Good*
Password: Bad
Password: G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Parameter Name : value
Custom Protocol TemplateTest Case «TestCaseNumber» Preconditions
«TableStart:Precondition»«CategoryName»
«ChoiceName»«TableEnd:Precondition»
Inputs «TableStart:Input»«CategoryName»
«ChoiceName»«TableEnd:Input»
Results «TableStart:Result»«ResultName»«TableEnd:Result»
Procedure: (Warning: If you customize this section do not use numbered lists. Word MailMerge doesn’t have the sense to reset the numbering.)
Open the website Click the Login button Enter «User ID» into the User ID field. Hit tab Enter «Password» into the password field Click the Login button
Now verify that: «TableStart:Result»
«ResultName»
«TableEnd:Result»
WordMailmergeFields
User ID and Passwordvalues filled inhere
Simple ValuesGeneric Login Command (With Special
Inputs) User ID:
User ID: abcdefghijklm
User ID: 56%#@!*(a
User ID: gary*
User ID: charlie
User ID: garcía
Password:
Password: 56%#@!*(a
Password: Good*
Password: Bad
Password: G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax x xValid with non-ASCII chars x
Input Password Field
Empty xToo long xValid Syntax x xValid with quotes xValid with HTML chars xValid with non-ASCII chars x
Condition User
no user with the user ID xuser with the user ID x x x x x
Condition User Password
does not match the user ID xmatches the user ID x x
Complex ValuesGeneric Login Command (With Special
Inputs) User ID:
User ID: abcdefghijklm
User ID: 56%#@!*(a
User ID: gary*
User ID: charlie
User ID: garcía
Password:
Password: 56%#@!*(a
Password: Good*
Password: Bad
Password: G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax x xValid with non-ASCII chars x
Input Password Field
Empty xToo long xValid Syntax x xValid with quotes xValid with HTML chars xValid with non-ASCII chars x
Condition User
no user with the user ID xuser with the user ID x x x x x
Condition User Password
does not match the user ID xmatches the user ID x x
Complex ValuesGeneric Login Command (With Special
Inputs) User ID:
User ID: abcdefghijklm
User ID: 56%#@!*(a
User ID: gary*
User ID: charlie
User ID: garcía
Password:
Password: 56%#@!*(a
Password: Good*
Password: Bad
Password: G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty xToo long xHas invalid characters xValid Syntax x xValid with non-ASCII chars x
Input Password Field
Empty xToo long xValid Syntax x xValid with quotes xValid with HTML chars xValid with non-ASCII chars x
Condition User
no user with the user ID xuser with the user ID x x x x x
Condition User Password
does not match the user ID xmatches the user ID x x
Default Values
Generic Login Command (With Special Inputs)
LIMITS
Empty User ID error
User ID Too Long error
User ID: abcdefghijklm
User ID: 56%#@!*(a
User ID: gary*
User ID: charlie
User ID: garcía
Password:
Password: 56%#@!*(a
Password: Good*
Password: Bad
Password: G"oo"d
Password:Go<o>d&
Password:¿sueÑo
Input User ID Field
Empty xToo long x xHas invalid characters xValid Syntax x xValid with non-ASCII chars 1 x
Input Password Field
Empty xToo long xValid Syntax x xValid with quotes 1 xValid with HTML chars 1 xValid with non-ASCII chars 1 x
Condition User
no user with the user ID xuser with the user ID x x x x x
Condition User Password
does not match the user ID xmatches the user ID x x
* indicates thedefault value
Values Applied to the ProtocolTest Case 7 Preconditions
User no user with the user ID
Inputs User ID Field Valid Syntax Password Field Valid with quotes
Results User ID or password incorrect error
Procedure: Open the website Click the Login button Enter charlie into the User ID field. Hit tab Enter G"oo"d into the password field Click the Login button
Now verify that: User ID or password incorrect error
Going Beyond the Basics• Create variability in procedures
• Action Choices
• Using TAME when bugs are found• Root Cause Analysis• Modify the Tests
• Scaling Up• Higher Level Tests• Results become Preconditions
Additional variabilityTest Case 7 Preconditions
User no user with the user ID
Inputs User ID Field Valid Syntax Password Field Valid with quotes
Results User ID or password incorrect error
Procedure: Open the website Click the Login button Enter charlie into the User ID field. Hit tab Enter G"oo"d into the password field Click the Login button
Now verify that: User ID or password incorrect error
- Enter a URL directly- Click a link on the company
homepage- Click a link on an unsecured
page- Tab to the next field- Click into the password
field- Click the Login button- Tab to the Login button
and press Enter
Action ChoicesTest Case 7 Preconditions
User no user with the user ID
Inputs User ID Field Valid Syntax Password Field Valid with quotes
Results User ID or password incorrect error
Procedure: Open the website Click the Login button Enter charlie into the User ID field. Hit tab Enter G"oo"d into the password field Click the Login button
Now verify that: User ID or password incorrect error
• Enter a URL directly• Click a link on the company
homepage• Click a link on an unsecured
page• Tab to the next field• Click into the password
field• Click the Login button• Tab to the Login button
and press Enter
For each of these• Create “action”
choices• Use the action
choices in the protocol
Unforseen Circumstances
A user ID that includes spaces causes a crash
Three failed logins lock the user’s account
• Add “contains spaces” as a choice
• Add a value definition that includes spaces
• Determine how to handle a user ID that includes spaces: is it allowed or is it an error?
• Add “Preceding failed logins” as a new preconditions with several choices
• Determine whether a locked account presents a different error message
• Add result selectors
Scaling Up
Employee hasexpenses to report
Log In
Create NewExpense Report
Add ExpenseItem
Submit toManager
Expense Report Submitted
OpenExpense App
already logged in
not logged in
Login is just one step in a larger process
Scaling UpEmployee hasexpenses to report
Log In
Create NewExpense Report
Add ExpenseItem
Submit toManager
Expense Report Submitted
OpenExpense App
already logged in
not logged in
Create separatetest specificationsfor each stepin the process
Scaling Up – Business Processes
Employee
Manager
Controller
Employee has expenses to report
RecordExpenses
ApproveExpenseReport
PayReimbursement
Reimbursement
RejectExpenseReport
ResubmitExpenseReport
Submitted Expense Report
Rejected Expense Report
Resubmitted Expense Report
Approved Expense Report
Review New Expense Report
Review Resubmitted Expense Report
CancelExpenseReport
Expense Reporthas been canceled
Consider whether to resubmit
Employee hasexpenses to report
Log In
Create NewExpense Report
Add ExpenseItem
Submit toManager
Expense Report Submitted
OpenExpense App
already logged in
not logged in
Generating Test Cases
Test CaseTest
Worksheet TAME
Test CaseTest
CaseTest CaseTest
Case
Excel Open Office
Test Case List
Test SummaryTable
Test ProtocolDocument
Comparative Testing Tools• Run Tests
• White-box, code-centricNUnit, JUnit
• UI-based, capture-playbackRanorex, Telerik
• Design by Contract• Write a specification, construct a single
test caseCucumber / Gherkin
Where to use TAME• Unit testing
• specify and generate test cases that can run against software modules (classes, functions).
• GUI testing• whether a traditional installed application, a
website, or a mobile app, use TAME to specify the contents of UI pages, then generate test cases to validate user-facing behavior.
• Regression testing• update the test specification and the test
cases whenever new features are deployed or bugs are fixed.
Where to use TAME• Project Planning
• create initial test specifications to estimate the number of test cases to be run and project the cost of the test effort.
• Retrospectives• use the test specifications to probe
features to better understand the root causes of failures.
TAME for All• TAME helps the analyst
to think through what needs to be tested.
• TAME gives the tester a platform for creating many test cases quickly.
• TAME gives the project manager a means for sizing and scoping the testing effort.
TAME software
Training Courses“Writing Testable
Requirementsand Effective Test Cases”
.
“Modeling Techniquesfor Business Analysts”
Thank you
Marc J. BalcerChief Architect
Model Compilers LLCmarc@modelcompilers.com
http://www.modelcompilers.com