Bug Hunting in practice - Nordic Testing Days · Bug Hunting in practice ... management positions...

06/06/20161 © Nokia 2014 1.0 Kuisma_Szell - DocID

Public

Bug Hunting in

practiceINTERNATIONAL TESTING CONFERENCE

Nordic Testing Days

JUNE 1 - 3, 2016 IN TALLINN, ESTONIA

• Szilard Szell, (Marko Kuisma)


Public

Introduction

Marko Kuisma

•2005-2015 working in system testing and

performance testing area in Nokia and NSN

•Owner for ~1000 fault reports during testing

specialist career for six years in Nokia

•Several project manager positions in testing area

•Line manager of different test teams for four years

•Bachelor of Engineering in Telecommunications,

University of Applied Science

•Advocate for manual testing with human errors

Szilárd Széll

•2000 – various Testing Related engineering and

management positions in Nokia

•2015 – Test Coach in Central Org.

•2014 – President of Hungarian Testing Board

Studies

•MSc in IT Management, Central European

University

•MSc in Programming and Mathematics, University

of Szeged

•Certified Scrum Master, ISTQB CT-AL Test

Manager, CT-FL Agile Tester, IREB CPRE, Lean

Six Sigma Green Belt

3 © Nokia 2016

Nokia internal use

Highlights

• Core on Cloud: Open MSS, vTAS, TAS-E

• SDM Solutions

• GSM-R

• Cloud Infrastructure

• Serve at Once Intelligence, Device Mgmt, Cloud Application Mgmt

• All Core Customer Support

• T&I: SDN, SON, CEM, Big Data and Network Mgmt

Ecosystem

•Symbiotic cooperation with major ICT Universities in the region

•Strategic partnership with Hungarian Government

•European ICT hub: Ericsson, Epam Systems, Morgan Stanley, NNG, Lufthansa, SAP, Evosoft, Bosch, Siemens, IBM, Prezi, LogMeIn, BalaBit, T-Systems

•Partnering with and supporting Hungarian Testing Board

Technology Expertise

• Cloud computing and architecture - Core Virtualization

• (SDN) Software Defined Networking

• 2G/ 3G/4G Technology Expertise

• Network Function Virtualization

• Self Organizing Networks

• Big Data Management

Budapest – Core on Cloud Center

4 © Nokia 2016

Nokia internal use

Nokia as a house of Testing

• Well defined Testing Processes on Company level

• Well defined, standardized Requirements as input

• ISO9001 and TL9000 compliant processes

• Lean Six Sigma

• Complex, Highly available (99.999%), Robust Products on IT Cloud and

Telco HW

Cornerstones of System Verification


Public

Test Case Design TechniquesAs used during Product Development

Defect-based techniques

• Taxonomies

Specification-based or Black-box

techniques

• EP

• BVA

• Decision table testing

• State transition testing

• Orthogonal arrays / all pairs tables

• Use case testing

Structure-based or White-box techniques

• Statement testing

• Decision testing

• Condition / Multiple condition / Condition determination testing

• Path testing

• LCSAJ

Experience-based techniques

• Error guessing

• Checklist-based testing

• Exploratory testing

• Attacks


Public

• Exploratory Testing can be described as a goal oriented wandering

• There is a mission/goal described in a charter, but there is no planned route

you need to take

• Create a charter describing what and how and which way you want to test

• Describe duration of your test

Exploratory Testing(One) Definition

”Exploratory Testing is an interactive process of

concurrent product exploration, test design and test

execution. To the extent that the next test we do is

influenced by the result of the last we did, we are doing

exploratory testing.”

James Bach, Satisfice


Public

Exploring BarcelonaET is a goal oriented wandering


Public


Public

Explaining Exploratory testStory Cubes – the game

“Once upon a time there was a flash, into who a fish has

moved in (to live). The fish has asked the flash - Why is there a

pyramid behind you? Why are you striking it? He (the flash)

said - because an airplane is just going to fly by 200.000

millimeters below (us). Then, the passengers in the plane has

asked the pilot - Why are we turning all around (left and

right?)? The pilot said - Because I am reading a book. Then

they (the passengers) asked the waitresses - Why don’t we

get something to eat? Because (the waitresses said) a

tortoise jumped into the soup. And then the tortoise said:

Look, the sheep is parachuting out there!”

Szonja, 7 years


Public

1. Gather Developers, Testers, Customers, Business people to the same table

2. Show them a Story Cube pack

3. Start collecting 9x6 areas of the system that is important

4. Make Cubes for your own

5. Play the game, roll the dice and tell a story by reading

the pictograms

6. You just created an exploratory test case

Explaining Exploratory testStory Cubes – in testing


Public

Cube 1 Cube 2 Cube 3 Cube 4 Cube 5 Cube 6 Cube 7 Cube 8 Cube 9

Theme Unit State

Change

Load

Balancer

“CRUD” Output Transacti

on

Analysis Attribute

Anal.

Inter-

face

Side1 STU WO RE H.248

MSS

Create /

Add

Log LocUp Barring Routing A/Iu

Side2 CHU SP RE Read /

Interrogate

Alarm HandOve

r

User

plane

CHA Mc

Side3 VLRU CSWO SIP Update /

Modify

ErrorLog MO Digit EOS MAP

Side4 CM FSWO M3UA Delete /

Remove

ClearCod

e

MT EOS Service Nc,

BICC/SI

P

Side5 SU DISK H.248

MGW

Measure

ment

Forward Function AIF ISUP

Side6 IPDU POWER CHA

Ticket

SS Pre./Ext. IN CAMEL

Explaining Exploratory testStory Cubes – cubes “created” by experts


Public

• Rolled cubes: CMM -> Faulty SWO -> ClearCode -> A/Iu interaface

• Test Case scope: CMM faulty switchover during traffic

• Pre-requisite: MSS is running with traffic. WO/SP-CMM are running

smoothly.

• Execution: Issue faulty CMM switchover for WO-CMM (simulate hw

failure)

• ExpRes: CMM is 2N redundant unit, CMM is the heart of the system (main

DB copies are stored in its memory). If WO-CMM goes faulty switch over

should happen without any traffic outage. SP-CMM should take over the

traffic and the tasks smoothly

Explaining Exploratory testStory Cubes – a tale by an expert (Balazs Tenyi)


Public

Explaining Exploratory testfrom scripted testing

15

Test setup for

SUT

TEST CASE

Test setup for

SUT

Abnormal setup

to SUT

misconfigure

feature X

Use of abnormal

conditions in

variables

Test executionSimultaneous

test event

Distracting the

SUT

Modify

configuration

Test execution

OK result NOK result

Test execution

OK result

minor

faultmajor

fault

critical

fault

traditional pathexploratory path

14 © Nokia 2016

Nokia internal use

Bug Hunting


Public

• When a new release of software is ready for test, a Bug Hunt will very

clearly read the temperature ~ quality of the software.

• As an entry-criteria for new phases. View it as a ”smoke test” executed by

people, instead of automated test, if you don´t have any of these.

• As team-motivation, when test execution becomes day to day work, and

the auto-pilot is taking over, a Bug Hunt can be what you need to add extra

adrenalin to your test.

Bug HuntingWhen can we apply Bug Hunting - Based on “Go on Bug Hunt” –

from Klaus Olsen, FiSTB Testing Assembly, 2013


Public

• 4 dissimilar test systems

• 4x test pairs

• 4 x 45 minutes testing rounds

• Exploratory Testing

• Scope of Hunting

• Chat tool

• Referee, Coach

• Prize

Bug HuntingIngredients

SUT1

SUT2

SUT3

SUT4


Public

Bug HuntingSteps

1) Define

Pre-conditions for bug

hunting: scope,

environments, duration,

limitations, known issues

2) Go for Hunting

Use all the possible

methods and scenarios.

Do fault attacks, error

guessing, pair working,

parallel testing..

4) Judge

Referee makes notes of all

issues and informs if issue

is duplicate.

5) Analyze

Categorize issues.

Investigate more for

unclear issues

3) Inform

Briefly report all findings to

referee. Don’t pause

hunting for fault

investigation

6) Report

All clear faults reported


Public

Performance Test / Robustness Test teams (ESPOO):

• ORANGE Team: (SUT86); 2 person – I, I, I, I, I

• BLACK Team: (SUT33); 2 person – I, I, I

• MAGENTA Team: (SUT64); 4 person – I, I, I, I, I, I, I, I, I, I, I

Performance Test / Robustness Test teams (BUDAPEST):

• RED Team: (SUT76): 3 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I

• BLUE Team: (SUT87): 4 person – I, I, I, I, I, I, I

E2E product teams (BUDAPEST):

• GREEN Team: (SUT5/6):3 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I

• Yellow Team: (SUT7-8): 4 person – I, I, I, I, I, I, I

• PURPLE Team: (SUT71-72) :4 person – I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I, I

• Azure Team: (SUT53-54): 4 person – I, I, I, I, I, I, I

Bug HuntingGo for Hunting - Groups and Environments

~ 95 issues found

3 Major, 30 Minor Faults


Public

Bug hunting Metrics

Different issues detected

with 3 hours of bug huntingFault reports created

by 6 Product Lines so

far with 8 bug hunting

events

Fault reports created after 3

hours testing by whole team

90+

Issues detected by most

effective pair during 3

hours of bug hunting

20+Fault reports done daily by

system testing team with

traditional way

1-333 100+


Public

Planning

• Plan all Sessions up-front as part of project

• Estimate time for the session, preparation and analysis

• Communicate to stakeholders

Schedule

• As part of Sprint Review

• Before Delivering to next stage (or customer)

• As quick entry and exit quality check

Administration

• Each Session is 1 Test Case in Test Management tool, with fix time (estimate)

• Report faults

• Create new test case for opened faults as needed

Bug HuntingPlanning, scheduling and administration


Public

• Negative scoring as a punishment for not following rules

• “Hunting session” takes a month

• Using only Pre-Automated test cases

• Connect Real Incentive to results

• Collecting Bugs beforehand and keep them in your pocket until the Bug

Hunting Event

Bug HuntingMisuse – to be avoided

22 © Nokia 2016

Nokia internal use

Lessons Learned

• helps tester to think out of the box

• improves team spirit and expertise

• need to be planned up-front in Project Plan, not an ad-hoc event

• the event need preparation and time to analyze issues found

• complements requirement based testing

…is FUNA Bug Hunting event …


Public

Bug Hunting in practice - Nordic Testing Days · Bug Hunting in practice ... management positions...

Documents

Transcript of Bug Hunting in practice - Nordic Testing Days · Bug Hunting in practice ... management positions...