1

Test & Evaluationas a

“Knowledge factory”

Outline

2

Knowledge-centered perspective on T&E

Shifting focus to the people

Some observations

Glimpse to theory and its applications

System Debugging Loop

3

Discovery

T&E, Reqs

Systems

Knowledge

ActionSE

T&E ~ Empirical Research

4

Make predictions based on theory/design

Verify/Falsify predictions by experimenting

Search for unpredicted behavior

Reporting findings formally

5

Should test engineers

be trained

as empirical scientists?

Should T&E be structured

as scientific discovery

process?

T&E ≠ Empirical ResearchIt requires:

6

Maximal coverage of phenomena

Scheduled delivery of knowledge

Immediate usefulness of knowledge

7

Test & Evaluation asa personal and group quest for knowledge

8

Project / Organization

Test Team

Test Engineer

9

Some observationson how the Test

Engineers think…

Knowledge iscreated by thinking

10

Using all available sources

Creating a mental model

In order to be able

to “tell a story” about the system

Without documentati

on

11

Need to Learn arises when

The production of required

documentsis impeded

The evidence

contradicts the mental

model

12

Test & Evaluation require effective and efficient

knowledge creation and learning

Test & Evaluation is reallya “Knowledge Factory”

13

How to leverage the science and practice of

working with knowledgeto better perform T&E?

14

What is Knowledge?

Knowledge is experience. Everything else is just information.Albert Einstein

15

Planning T&E

Plans

Designing Tests

Test Procedures

Executing Tests Test Protocols

Evaluating Test Reports

Planner Experience

Designer Experience

Tester Experience

Analyst Experience

UndocumentedTacit

Knowledge

DocumentedKnowledge

16

SECI Model (I. Nonaka)

Socialization in T&E

17

i iDevelopers on-site

Customer on-site

Testing in teams

Learn by shared

experience

Externalization in T&E

18

Talking with developers

Talking with customers/users

Meetings and discussions

Writing reports

i

i

ii

i

i

g

Learn by articulating in language

Combination in T&E

Complex Adaptive Integrator19

g

g

g

g oAnalyzing requirements specs

Analyzing design documents

Designing test plans

Designing test procedures

Learn by analysis and synthesis of information

Internalization in T&E

20

i

go Building “mental models” of the

system

“Playing” with the system simulations

Unstructured and exploratory testing

“Playing around” planned tests

Learn by exercise

and practice

“Soft” T&E Practices

21

Actively question the developers, customers and users to externalize their tacit knowledge

Encourage active participation of developers, customers and fellow testers in running tests

Take meetings and non-formal discussions seriously and use them to learn and share knowledge

22

Learning PreferencesEveryone learns better

when the learning matches

their preferences

Teachers should engage multiple “learning muscles” in their audience

Learners should direct their learning to sources that match their preferences

“Soft” T&E Practices

23

Know your learning preferences and use them while studying the SUTs

Augment reports with multimedia

Organize presentations of the results for relevant audiences

Conclusions

24

Test & Evaluation is a knowledge creating and sharing process

Test Engineers are professional learners

T&E should use the insights and tools of knowledge and learning theory and practice

Questions for further research

25

How the knowledge is created and shared by test engineers and test teams?

How do the knowledge processes integrate with testing activities?

What are the knowledge-related factors that contribute to or impede better testing?

What factors facilitate more effective and efficient learning of Test Engineers?

26

Any answers?