Post on 02-Jun-2018
8/10/2019 Test Case Execution in Software Testing
1/20
Test Case Execution
Major testing activities:
Test planning and preparation Execution (testing)
Analysis and followup
Test execution:
!ant a s"oot# transition fro" one test run to anot#er
Execution planning and "anage"ent
$elated activities: i"portant part
failure identification and "easure"ent
ot#er "easure"ent
http://www.apextgi.in/8/10/2019 Test Case Execution in Software Testing
2/20
Test Execution
%eneral steps
Allocating test ti"e (& resources)
'nvoing tests (& collecting execution info)
'dentifying syste" failures
(& gat#ering info for followup actions)
Allocating test ti"e
planned* +ut needs "onitoring & adjusting
,-+ased or Coverage+ased
Alternative: +otto"up approac#
individual test cases (test ti"e)
(su"up) overall allocation
+y ,- or coverage areas
8/10/2019 Test Case Execution in Software Testing
3/20
Test Execution
'nvoing tests follow plan* provide input varia+le values over w#oleexecution duration following preplanned se.uence
'nvoing test (,-+ased)
,- /0 input varia+les (test points) 1ollow pro+a+ilistic distri+utions (could +e dyna"ically deter"ined)
2e.uence (w#at to test first3): C,T2* product* super syste"
'nvoing test (coverage+ased)
,rgani4e sensiti4ed test cases
2e.uence (coverage #ierarc#ies)
Co""on part: $etest due to 5efect fix /0 verify fix
Code or feature c#ange
%eneral regression test
8/10/2019 Test Case Execution in Software Testing
4/20
Test Execution
Identifying system failures (oracle problem):
Similar for OP-/coverage-based
naly!e test output for deviations
"etermine: deviation # failure $
%andling normal vs& failed runs non-bloc'ing failure andling
Solving oracle problem:
eoretically undecidable&
Some cases obvious: cras* ang* etc&
Practically based on euristics:
product domain 'no+ledge
cross-cec'ing +it oter products
implementation 'no+ledge , internals
limited dynamic consistency cec'ing
8/10/2019 Test Case Execution in Software Testing
5/20
Test Execution
ailure observation and measurement:
.en determining deviation # failure
stablis +en failure occurred
used in reliability and oter analysis ailure information (e&g&* O"0):
+at/+ere/+en/severity/etc&
"efect andling and test measurement:
"efect status and cange (controlled)
Information gatering during testing: e1ample template: able 2&3 (p&45)
ollo+-up activities:
fi1-verification cycle
oter possibilities (defer* invalid* etc&)
8/10/2019 Test Case Execution in Software Testing
6/20
8/10/2019 Test Case Execution in Software Testing
7/20
Testing Analysis and 1ollow up
6a7or testing activities:
est planning and preparation
1ecution (testing)
nalysis and follo+ up est analysis and follo+ up:
1ecution/oter measurement analy!ed
nalysis results as basis for follo+ up
eedbac' and follo+ up:
defect fi1ing
decision ma'ing (e1it testing$ etc&)
ad7ustment and improvement&
8/10/2019 Test Case Execution in Software Testing
8/20
Testing Analysis and 1ollowup
Input to analysis
est e1ecution information
Particularly failure cases
iming and caracteristics data
nalysis and output
8asic individual (failure) case
problem identification/reporting
repeatable problem setup Overall reliability and oter analysis$ (6odule 9)
ollo+-up activities
"efect analysis and removal (, re-test)&
"ecision ma'ing and management&
est process and uality improvement&
8/10/2019 Test Case Execution in Software Testing
9/20
8/10/2019 Test Case Execution in Software Testing
10/20
Testing Analysis and 1ollow up
nalysis and follo+-up for overall testing:
determine if goal as beenacieved/release> steps to acieve if not> id lo+ reliability areas
0overage analysis and follo+-up& surrogate for reliability> input for stopping criterion
"efect analysis and follo+-up&
defect distribution> id ig-defect areas
ocus of Part I9&
ollo+ up activities: Similar& "ecision ma'ing and management&
est process and uality improvement&
8/10/2019 Test Case Execution in Software Testing
11/20
Test Manage"ent
People?s roles/responsibilities in formal and informal testing&
In informal testing:
@plug-and-playA by novice users&
@run-and-observeA by testers (or automation tools)& Informal testing +it ad-oc 'no+ledge for easy* simple systems&
"eceptively @easyA* but not all failures or problems easy to recogni!e&
In formal testing:
8/10/2019 Test Case Execution in Software Testing
12/20
Test Manage"ent
est team organi!ation:
9ertical: Pro7ect oriented
product domain 'no+ledge*
staffing/resource management issues (personnel reassigned)&
%ori!ontal: as' oriented
teams perform one 'ind of testing on many different products
even distribution of staff/resources
lac' of internal 'no+ledge/e1pertise
6i1ed models migt +or' better&
;sers and 5rd party testers: ;ser involvement in beta-testing and oter variations
I9,9 +it 5rd party testing/B ("O" e1tensively uses tis)
Impact of ne+ tecnologies:
08S* 0OS impact
security* dependability reuirements&
8/10/2019 Test Case Execution in Software Testing
13/20
Test Auto"ation
8asic understanding:
utomation needed for large systems&
ully automated: eoretically impossible&
ocus on specific needs/areas&
Cey issues to consider:
Specific needs and potential for automation&
1isting tools available/suitable$
related: cost/training/etc&
0onstructing specific tools$
dditional cost in usage , support&
8/10/2019 Test Case Execution in Software Testing
14/20
Test Auto"ation
utomation by test activity areas:
utomated test planning , preparation&
Automated test execution.
utomated test measurement* analysis* and follo+-up&
Sligtly different grouping due to tigt coupling for measurement ,analysis&
utomation for test e1ecution&
6any debuggers: semi-automatic (testers may intervene)&
as' seuencing/sceduling tools&
=oad/test generator: script #D runs Eenerally easier to obtain test scripts&
8/10/2019 Test Case Execution in Software Testing
15/20
Test Auto"ation
utomation for test planning/preparation:
est planning: %uman intensive not muc can be done (inspection and9)&
est model construction: similar to above&
automation possible at a small scale&
est case generation: focus&
est case generation:
rom test model to test cases&
Specific to individual tecniues
e&g&* cover cec'list items* pats* etc& 9arious specific tools&
Cey: +ic specific testing tecniue supported by te specific tool$
8/10/2019 Test Case Execution in Software Testing
16/20
Test Auto"ation
est measurement* analysis* and follo+-up&
nalyses dictate measurements needed&
6ost common: reliability/coverage&
need timing data for individual test runs
"efect measurement needed in most cases:
defect trac'ing tools collect data on personnel* component
can identify problematic areas for improvement
8/10/2019 Test Case Execution in Software Testing
17/20
Test Auto"ation
Coverage+ased testing
"easuring coverage and co"pare to preset goals
usually re.uires "ore detailed infor"ation t#an relia+ility analysis
Test coverage tools:
5ifferent levels6definitions of coverage/0 different tools
Exa"ple tools:
McCa+e: execution (control flow) pat#
2TCAT: functional coverage7 callpair infor"ation
ATAC: data flow coverage
Test coverage steps:
-reparation: progra" instru"entation Measure"ent step: run and collect data
Analysis step: analysis for coverage
8/10/2019 Test Case Execution in Software Testing
18/20
8/10/2019 Test Case Execution in Software Testing
19/20
2u""ary
est activities:
Planning , preparation: focus of Part II&
1ecution , measurement: common&
nalysis , follo+ up: focus of Part I9&
est management:
"ifferent roles and responsibilities&
Eood management reuired&
est automation:
Set realistic e1pectations&
Specific areas for automation* esp& in e1ecution* measurement* and
analysis&
8/10/2019 Test Case Execution in Software Testing
20/20