Test Design for Everyone

Alan PageMicrosoft

Here, Test This!

“Test Design” is what Testers Do

What is test design?

Simple:

Come up with some tests

Better(?):

Come up with a well thought out and broad set of tests for the given context (based on

application, schedule, etc.)

Another Take:

Create an optimal set of tests that finds the maximum amount of customer facing issues possible with a minimum of over-testing and

under-testing

“Over Testing” and

“Under Testing”

Application

Application Testing Effort

Application

(potential)

Over Testing

Under Testing

(stuff you forgot to

test

or didn’t know to test)

Testing Effort

bool Validate(string sideName) try { sideVal = Convert.ToUInt16(inputVal); if (sideVal < 1) { MessageBox.Show(sidename + “must be greater than 0”, ...); return false; } } catch (OverflowException) { MessageBox.Show(sideName + "must be less than 65536", ...); return false; } catch (FormatException) { MessageBox.Show(sideName + "must be an integer value, ...); return false; }

}

bool Validate(string sideName) try { sideVal = Convert.ToUInt16(inputVal); if (sideVal < 1) { MessageBox.Show(sidename + “must be greater than 0”, ...); return false; } } catch (OverflowException) { MessageBox.Show(sideName + "must be less than 65536", ...); return false; } catch (FormatException) { MessageBox.Show(sideName + "must be an integer value, ...); return false; }

}

In English

Three different types of errors:

1. Zero or less2. Bigger than 655363. Non-numbers

Any other input is testing the same thing

Application

Reduce

Over Testing

(don’t test the same

thing multiple times)Reduce Under Testing

(miss less)

Testing Effort

How do you become a better Test Designer?

You need ideas!

Ideas you have used beforeNew ideas to tryIdeas suited for this productIdeas suited to your release schedule

…and you need to know what to look for

Why Is it so Hard?

You Don’t Know What You Don’t Know

Read Philip Armour’s The Five Orders of Ignorance

All you can use is what you know

Hammers are useful…

…but not everything is a nail

What you’ve already heard about test design

http://www.satisfice.com/glossary.shtml

The “what” has been covered

Let’s focus on the “how”

You need a Toolbox

Bigger is Better

What kinds of tests do you write

New Information

False Negative

False Positive

No Information

Useful Tests?

Useful Test Provide High Levels of “New Information”

“Usefulness” depends on context

Some general guidelines:

The first time you run a test it (almost) always provides new information

If you run the same test the same way again, it may or may not provide new information

Some general guidelines (continued):

Varying your tests gives them a higher chance of providing new information (randomization, data driven tests, “smart monkeys”, model-based testing, etc. all help here)

Examples / Practice

Test This:Add or Remove Contacts

Mind Mapping

What about adding a contact with a blank name?

12

3 4

A/V and Web Conferencing Workload

Edge Servers

Externalfirewall

Internalfirewall

HTTPS:443

Reverse proxy

HTTPS:4443

SIP/MTLS:5061

SIP

/TL

S:5

06

1

A/V Edge - STUN/TCP:443, STUN/UDP:3478

A/V Edge – SRTP:443,3478,50,000-59,999

SR

TP

/UD

P:4

91

52

-65

53

5

Range of ports is configurable.

PS

OM

/TL

S:8

05

7

HT

TP

S:4

43

HTTPS:443 is used to download conferencing

content.

Traffic goes directly to Web Conferencing Service WITHOUT going through the pool’s hardware load balancer

Two inbound and two outbound unidirectional streams. Media codec varies on workload:- RTAudio for audio- RTVideo for video

Traffic goes directly to A/V Conferencing Service WITHOUT going through the pool’s hardware load balancer.

MRAS traffic.

SIP/MTLS:5062Web Conf Edge - PSOM/TLS:443

Access Edge - SIP/TLS:443

PSOM/MTLS:8057

STUN/TCP:443, STUN/UDP:3478

Directors

Monitoring

Server

SIP/MTLS

SIP/MTLS:5061

Port number to service traffic assignment:

5063 - A/V Conferencing Service

MSMQ

Codec varies per workload:- G.722 or Siren for audio- RTVideo for video

Enterprise

pool

Meeting content + metadata + compliance file share.

HTTPS traffic

SIP traffic: signaling

RTP/SRTP traffic: A/V Conferencing

PSOM traffic: Web Conferencing

If client connects on port 80, it gets redirected to port 443

Only traffic on ports 443, 4443 and 5061 must be load

balanced by HLB.

How to get better at Test Design

Practice, practice, practice

Practice testing – be curious

Collaborate

http://testing-challenges.org

Weekend Testing (http://weekendtesters.com)

Learn

How does your application work?

Find new testing ideas

Study critical thinking

Read…anything

Make your tests useful…and notice when they’re not useful

The Best Test Tool of All Time

Your Brain (of course)

Be A Great Test Designer

(and be a great tester too)

http://angryweasel.com/bloghttp://twitter.com/alanpage

Credits

• Opening slide pattern - some rights reserved by km6xo• “Scrap metal” – some rights reserved by

MAG (Mines Advisory Group)• Ideas - Some rights reserved by orkboi• Impossible triangle - some rights reserved by yui.kubo• Hammer – some rights reserved by Noel C. Hankamer• Nail - some rights reserved by crd!• Hammer / screw - some rights reserved by justinbaeder• Tomatoes - some rights reserved by lucianvenutian• Toolbox - some rights reserved by Austin ampersand Zak• Big Toolbox - some rights reserved by Fuschia Foot• Brain - Some rights reserved by biologycorner