Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP...

Hypothesis Based Testing - Application and

Adaptation for testing Enterprise Software Satvik Kini, Suite Test Center, SAP Labs India Pvt Ltd., Bangalore

December 19, 2013 Public

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 2 Public

Agenda

Hypothesis Based Tests – Overview

Traditional Test Approach vs. HBT

Stages in HBT Approach – (Detailed Explanation with a case study)

Challenges and Benefits

Key factors for success

Summary

Q&A

Hypothesis Based Tests - Overview



Hypothesis :

Tentative explanation for a certain behavior, phenomenon, events that have occurred or will occur

• Null Hypothesis – H0 – Eg: Heavy Smoking is the cause of Lung Cancer

• Alternate Hypothesis – H1 – Eg: Heavy Smoking is not the cause of Lung Cancer

Hypothesis Based test:*

Goal focused approach, test methodology based on solid scientific principles built on the core theme of

hypothesizing potential defects.

* HBT is the IP of STAG Software.



The Scientific Method

Form a Question

Research

Hypothesis

Experiment

Analyze Results

Draw Conclusions

Does my software work ?

Understand Context

Hypothesize Defects

Develop & Execute Tests

Pass or Fail ?

Does my software work ?

Traditional Test Approach Vs. HBT


Traditional Test Approach Vs. HBT

Activities

Manual Test Cases

Automated Test Cases

----------------------------

----------------------------

----------------------------

Defects

Goal

Risks/

Potential

Defects

Activities

Test Levels

Test Design

Test Execution

----------------------------

----------------------------

Determines

Drives

Clean

Software

Hopefully results in

Focus: Coverage, Automation,

Test Scripts Focus: Defect Elimination based

approach for cleanliness

Defect Elimination


Traditional Test Approach - Limitations

Limitations of Traditional Approach

• Often Software is passed with X% OK Rate

• Assumption – existing test cases cover desired parts of the software

• Hopeful encounter of bugs

• Focus on Quantitative reporting

Stages in HBT


HBT – An Example

Example 1 : Clean Water

Goal: Having Clean Water

Potential Defects in water: color, odour, suspended particles, bacteria

Cleaning Process:

• Conduct activities to first determine if each of the potential defects exists

• If a potential defect exists, then take steps to clean that defect

• Once all defects are removed, re-test the water for all potential defects and certify that the water is

clean

Result: Clean Water (goal is achieved)


HBT - Stages

Perform Testing Analyze and

Conclude

Define

Cleanliness

Criteria

Define

Potential

Defect Types

(PDT)

Understand End

user expectation,

System Behavior

Consider All

possible different

defect types and

identify each with

test quality levels

Use of different

test techniques

and tools to test

each of the PDTs

Document and

Analyze the

results and get

an overall picture

on Quality


Stages in HBT – Case Study

Scenario – Student Course Registration and Completion

Record External

Achievements

Equivalency

Determination Course Registration

Attendance Tracking

Examination Grading Produce Transcript

User Types

• Student

• Admin

• Academic

Advisor/Instructor

Market Places

• US Region

• Europe Region

• APJ Region


HBT – Cleanliness Criteria (Goal)

Example 2 : End to End (E2E) Scenario tests

Goal : The E2E Scenario Student Course Registration and Completion works fine


Conclude

Define

Cleanliness

Criteria

Define

Potential

Defect Types


HBT – Define Potential Defect Types


Conclude

Define

Cleanliness

Criteria

Define

Potential

Defect Types


HBT – Test Levels

Level 5

Expert Level E2E Tests

Level 4

Advanced E2E Scenario

Tests

workflows; role based

tests;Accessibility

Integration across

scenarios

Level 3

Integration tests

Multiple Users;

Timezone based;

Less Used features

Level 2

Primary E2E Use Cases

Integration defects;

Reporting

usability

Level 1

Transactional Coverage

Key E2E Requirements

Browser compatibility

Performance

Obvious Errors

Runtime Errors

Commonly used E2E

Use cases

• Each level corresponds to a set of potential Detect Types (PDTs)

• A purposeful detection of PDTs

• Only once all PDTs are tested for, a level is called to be reached


HBT – Test Matrix

Test levels Potential Defect Types

Level 1 Runtime Errors Obvious Functional Errors Commonly used E2E usecases

Level 2 Broswer Compatibility Calculations Performance

Level 3 Integration Reports Usability

Level 4 Currency conversion Less used features Timezone based

Level 5 Workflows Multiple Users Accessibility Integration to other scenarios

Creation of Report Card - Derive Test Matrix


HBT – Perform Testing

I. Use of Existing Test Techniques/Tools : 1. Test case driven testing or scripted testing

a. Automated test cases (only scripted tests)

b. Manual test cases (also giving the tester liberty of ad-hoc tests)

2. All pairs testing

3. Exploratory testing

4. Usage of Specific Test tools

II. Preparation of Charters


Conclude

Define

Cleanliness

Criteria

Define

Potential

Defect Types


HBT – Analyze and Conclude

• Document the results using charters/protocols

• Check the test results for each of the PDTs

• Update Report Card/Test Matrix - get a overall picture on Product Quality


Conclude

Define

Cleanliness

Criteria

Define

Potential

Defect Types


HBT – Test Matrix

Update Report Card

Test levels Potential Defects Types

Level 1 Runtime Errors Obvious Functional

Errors

Commonly used E2E

usecases

Level 2 Broswer Compatibility Calculations Performance

Level 3 Integration Reports Usability

Level 4 Currency conversion Less used features Timezone based

Level 5 Workflows Multiple Users Accessibility Integration to

other scenarios


Summary - HBT Realization

Prepare PDT List

Derive Test Matrix

Prepare Charters

Test Using Charters

Update Report Card

Protocol Logging

Define Cleanliness Criteria

Challenges, Benefits and Key Factors for

Success


Challenges

Ads

Challenges

• Understanding different user contexts –

market places

• Existence of custom code on top of

standard

• Highly configurable software –

customization and Business Functions


Benefits

Ads

• Confidence on what is tested

• Exploration of new scenarios as we do

the tests

• Reduction in customer messages

• Test Matrix gives an overall picture of

quality

• Higher Probability of finding defects as

the goal is to check existence of PDT

• Improvement - Code Coverage

Benefits


Key Factors for Success

Ads

Key Factors for Success

• Success mainly depends on

Understanding internal and external

factors and list of PDTs

• Identifying each PDT and its

corresponding level

• For each of the PDT, choosing right set

of scenarios/risks

• Choosing right set of tools and test

techniques to test each PDT

Summary


Summary

Goal focused approach, test methodology based on solid scientific principles built

on the core theme of hypothesizing potential defects.

© 2013 SAP AG or an SAP affiliate company. All rights reserved.

Thank you

Contact information:

Satvik Kini

Associate Quality Manager

SAP Labs India Pvt. Ltd, Bangalore

+91 98863 89654

[email protected]


Stages in HBT - Realization

Example of Charter :

Scope PDT Risks To Manage How Test Techniques Tester Messages Additional Comments

All Business Processes

Common Used E2E use cases

Check the straight forward E2E scenario is working

Run Automated Test cases

Automated test cases Garbage Collector Tour

Mr. A

Attendance Tracking UI

Browser Compatibility

Check Attendance Tracking UI is compatible with IE9, Chrome Version 30, Firefox, Safari

Check All the Tabs, Popups, F4 Helps, Result List, Status Icons, Calendar Display

Exploratory Tours : Super Model Tour, Garbage Collector Tour

Mr. B

Equivalency Determination

Obvious Functional Errors

Mapping of External Achievements to Internal subjects

Try different achievements with different status and validity dates

Exploratory Tour : Intellectual Tour All pair Testing

Mr. C

BACK


Stages in HBT - Realization

Update each Charter after testing each of the PDTs

Scope PDT Risks To Manage How Test Techniques Tester Messages Additional Comments

All Business Processes

Common Used E2E use cases

Check the straight forward E2E scenario is working

Run Automated Test cases

Automated test cases Landmark Tour

Mr. A None All Business processes are working fine

Attendance Tracking UI

Browser Compatibility

Check Attendance Tracking UI is compatible with IE9, Chrome Version 30, Firefox, Safari

Check All the Tabs, Popups, F4 Helps, Result List, Status Icons, Calendar Display

Exploratory Tours : Super Model Tour, Garbage Collector

Mr. B 23348743 32476764

Checked browser compatible for the IE9,chrome,Safari . Yet to check for Firefox

Equivalency Determination

Obvious Functional Errors

Mapping of External Achievements to Internal subjects

Try different achievements with different status and validity dates

Exploratory Tour : Intellectual Tour All pair Testing

Mr. C 43234234 Runtime Error while clicking on the Sort Button Blocked

BACK

Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP...

Technology

Transcript of Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP...