Acceptance Testing [email protected] Copyright, 2004 Jerzy R. Nawrocki Requirements...

Acceptance TestingAcceptance Testing

[email protected]/jnawrocki/require/

Copyright, 2004 Jerzy R. Nawrocki

Requirements Engineering & Project ManagementLecture 11

mailto:[email protected]






J.Nawrocki, Acceptance Testing

XPrince RolesXPrince Roles

Project ManagerAnalyst Architect


IntroductionIntroduction

Test-driven development

Acceptance tests: customer

Automated accept. testing

A. testing: customer+tester

Progress measurement based on acceptance testing


Presentation ContentsPresentation Contents

Testing Life CycleThe Scribble ApplicationThe Manual Test ProcessTest AutomationUse-Case-Based Acceptance Testing


What is test automation?What is test automation?

That’s simple!

1. Buy a test execution tool.

2. Record the manual tests.

3. Play the tests back whenever you want to.

„Just as there is more to software design than knowing a programming language, there is more to automating testing than knowing a testing tool.”

-- M. Fewster & D. Graham


Test activitiesTest activities

Identify test conditions (‘what’ to test) and prioritize

vali > Value

subject to sizei Capacity

Knapsack problem

109 11 128 8 10

15 sizei

vali

Capacity

Value




Design test cases (‘how’ to test)

Build test cases (scripts, data etc.)

Execute test cases

Compare test outcomes to expected outcomes


Quality attributes of a test caseQuality attributes of a test case

• How effective in detecting defects?

• How exemplary? (the more exemplary, the less test cases needed)

• How economic?

• How evolvable? (maintenance effort)


Test automation effectsTest automation effects

Effective

Exemplary

EvolvableEconomic


An example test caseAn example test casePrerequisites:• logged into the purchase order system as a data entry clerk;• database system must contain the standard Data Set;• no other new purchase order activity on the system.Step Input Exp.out Cond.

1 Create a new order for any one standard order, setting quantity to 100

Order confirmation message displayed

VB10VB23

2 Confirm the order Purchase order printed VB10

3 Print a new orders report New orders report printed showing just this one new order

VB10VB23

4 Cancel the order Purchase order cancellation notice printed

V8



Testing Life CycleThe Scribble Application

The Manual Test ProcessTest AutomationUse-Case-Based Acceptance Testing


The manual test processThe manual test process• Unscripted manual testing:

1. Identify conditions to test

2. Think up specific inputs

3. Enter the inputs

4. Checked it worked OK


The manual test processThe manual test process• Unscripted manual testing

• Vague manual scripts:

1. Read what to do

2. Think up specific inputs

3. Enter the inputs



Vague manual scriptVague manual script

1 Run up Scribble File menu displayed

2 Open file with sorted List File contents displayed

3 Add some items to List Items added in order

4 Move an item Item moved, List now unsorted

5 Add an item Item added at end of List

6 Delete item form list Item deleted

7 Use invalid position number Error message displayed

to delete an item

8 Save changes in new file End of test


The manual test processThe manual test process• Unscripted manual testing

• Vague manual scripts

• Detailed manual scripts:

1. Read what to do

2. Enter the inputs



Detailed manual scriptDetailed manual script

1 Click on Scribble icon Scribble opened, File menu enabled2 Move mouse to File menu, click Options available: Open, New, Exit3 Move mouse to Open option, Filenames displayed including click countries.dcm4 Move mouse to countries.dcm Text plus List of 3 countries:

Belgium, Netherlands, UK, File and List menus displayed

. . . . . .



Testing Life CycleThe Scribble ApplicationThe Manual Test Process

Test AutomationUse-Case-Based Acceptance Testing


Manual or Automatic Testing?Manual or Automatic Testing?1 Move mouse to Scribble icon2 Double click3 Move mouse to File menu4 Click5 Move mouse to Open option6 Click7 Move mouse to countries.dcm8 Double click9 Move mouse to List menu10 Click11 Move mouse to Add Item12 Click13 Type Sweden14 Move mouse to OK button

The amount of effort to automate a test case:

2 – 10 times the effort required to run the test manually (!)


Introduction – What is Rational Robot?Introduction – What is Rational Robot?

Baseline Baseline

Robot


Introduction – Rational Robot and test-first codingIntroduction – Rational Robot and test-first coding

Remarks:

• When you start testing an application is ready.

• Rational Robot is for regression testing, not for test-first coding.

• The only chance for test-first coding is SQABasic.


Testing HTML ApplicationsTesting HTML Applications

StartBrowser [ URL$, ] [ WindowTag= Name$ ]

• Starts the browser,• enables HTML testing, and • loads a page if specified.

Web page address

Identifies this instance of the browser

StartBrowser "http://www.cs.put.poznan.pl/jnawrocki/models/", "WindowTag= Parent"



Browser action$, recMethod$, parameters$

• Performs an action on a Web browser

Back, Forward, NewPage, SetFrame,

CloseWin, ..

Wait= % default

time is 30s

Browser NewPage, ""

HTMLTitle=$, Name=$,

"" ..

Wait for the topmost frame



HTML action%, recMethod$, parameters$

• Performs a mouse action on an HTML tag.

Click, .. Coords= x, y

HTML Click, "Name= Autor", "Coords= 10,12"

HTMLTitle=$, Name=$,

"" ..

J.Nawrocki, Acceptance TestingTestDatastore\DefaultTestScriptDatastore\TMS_Script\vp\test7.Weryfikacja_daty.base.grd

SQABasicSQABasic

HTMLDocument Click, "Type=HTMLDocument;Index=1", "Coords=25,14 "

Result = HTMLTableVP (CompareData, "Type=HTMLTable;Index=1", "VP=Weryfikacja_daty")


An alternative to Rational RobotAn alternative to Rational Robot

httpUnit

Based on jUnit

For programmers!


Data-driven scriptsData-driven scripts

Data-driven scripting = Test inputs stored in a separate (data) file.

countries, Sweden, USA, countries2

countries, France, Germany, test2

countries, Austria, Italy, test3



Data-driven scripting = Test inputs stored in a separate (data) file.OpenFile ‘ScribbleData’For each record in ScribbleData Read INPUTFILE Read NAME1 Read NAME2 Read OUTPUTFILE Call ScribbleOpen (INPUTFILE) FocusOn ‘Scribble’ SelectOption ‘List/Add Item’ FocusOn ‘Add Item’ Type NAME1 LeftMouseClick ‘OK’ ...EndFor

countries, Sweden, USA, countries2countries, France, Germany, test2countries, Austria, Italy, test3

ScribbleData

Control script


A more sophisticated data-driven scriptA more sophisticated data-driven scriptOpenFile ‘ScribbleData’Read INPUTFILECall ScribbleOpen (INPUTFILE)Go to next record (ie row)For each record in ScribbleData Read ADDNAME If ADDNAME <> Blank Then { FocusOn ‘Scribble’ SelectOption ‘List/Add Item’ FocusOn ‘Add Item’ Type ADDNAME LeftMouseClick ‘OK} Read MOVEFROM Read MOVETO ...EndFor

ScribbleData

countries

SwedenUSA

4 1Norway

27

countries2

Control script



Software under test

Key automated regression test cases

Additional automated regression test cases


Keyword-driven scriptsKeyword-driven scripts

Keyword-driven script = A data-driven scripts augmented with keywords representing user actions.

SQABasic???

ScribbleOpen countriesAddToList Sweden USASaveAs countries2

For each TEST_ID OpenFile TEST_ID For each record in test file Read KEYWORD Call KEYWORD EndFor CloseFile TEST_IDEndFor

Test file

Control script

ScribbleOpen...

AddToList...

SaveAs...

Supporting scripts


Shared scriptsShared scripts

Shared script = A script used (shared) by more than one test case.

SQABasic

Call ScribbleOpen (‘countries’)

FocusOn ‘Scribble’

SelectOption ‘List/Add Item’

...

Call ScribbleSaveAs (‘countries2’)


Automating test executionAutomating test execution

Automated scripts are not like manual scripts

Don’t automate testing by simply recording

Automated execution with manual verification?

How much shoul be compared (whole, minimum, something between)?

Dynamic comparison and post-execution comparison

Automated comparison messages have to be manually checked

While tests fail the second time?



Testing Life CycleThe Scribble ApplicationThe Manual Test ProcessTest Automation

Use-Case-Based Acceptance Testing




Design test cases (‘how’ to test)

Build test cases (scripts, data etc.)

Execute test cases




Identify important scenarios (criticality/frequency) and prioritize

Instantiate the selected scenarios (1 -> *)

Implement (scripts, data etc.)

Execute test cases



Use case goal levelsUse case goal levels

Book trip

Book hotelBook flight

User Goal Level

Book trip Summary Level

Book trip

Book hotelBook flight

Find flightReserve

seat Find hotelReserve

room

Subfunction Level



Identify important scenarios (criticality/frequency) and prioritize

Instantiate the selected scenarios (1 -> *)

Implement (scripts, data etc.)

Execute test cases



Test coverageTest coverage

GetSoftware G1:0

SelectAProject G1:0

GetInfoAboutProjects G1:0

RegisterACustomer G1:1 Z1:1

SubmitAProposal G1:0 R1:0 U1:0

ScenarioTotal calls


Use-Case ExampleUse-Case Example

Success GuaranteesSuccess Guarantees: Requestor has goods, correct budet ready do be debited.

TriggerTrigger: Requestor decides to buy something.Main Success ScenarioMain Success Scenario1.1. RequestorRequestor: Initiate a request.2.2. ApproverApprover: Check money in the budget, check price of goods,

complete request for submission.3.3. BuyerBuyer: Check contents of storage, find best vendor for goods.4.4. AuthorizerAuthorizer: Validate Approver’s signature.. . .ExtensionsExtensions1a. Requestor does not know vendor or price: leave those parts blank

and continue.


Test scenarioTest scenarioAktor: Reprezentant klientaCel: Zarejestrowanie organizacji klienta w bazie danych SDSPochodzi z: przypadku użycia RegisterACustomer [RegCust]Autor: J.Nawrocki; Zmodyfikowano: 28.11.2003Scenariusz: Główny scenariusz – G1Scenariusze pomocniczych przypadków: -Warunki wstępne: Firma X nie jest zarejestrowana jako reprezentant klienta.Punkt wejścia: Przeglądarka internetowa (Internet Explorer 5.0)

Krok Akcja aktora Spodziewany rezultat

Pro1 Otwórz stronę SDS-u. Strona SDS jest otwarta.

1 Otwórz formularz rejestracji. Formularz otwarty [ReqCust-Form].

2 Wprowadź dane Firmy X [CustInfo] i wyślij je.

Zostaje wyświetlony komunikat “Registration confirmed” [ReqCust-Confirm].


Test scenario – Screen shotsTest scenario – Screen shots

RegCust-Form RegCust-Confirm


SummarySummary

SDS test development:SDS test development:1.1. Use casesUse cases2.2. Manual test Manual test scenariosscenarios3.3. Automated test Automated test scriptsscripts4.4. Test casesTest cases sharing the scripts sharing the scriptshttpUnit or Rational Robot?httpUnit or Rational Robot?


QuestionsQuestions??

Acceptance Testing [email protected] Copyright, 2004 Jerzy R. Nawrocki Requirements...

Documents

Transcript of Acceptance Testing [email protected] Copyright, 2004 Jerzy R. Nawrocki Requirements...