ISERN Open Issues, Grand Challenges or Have we made any progress and where are going? Vic Basili...

ISERNOpen Issues, Grand Challenges

or

Have we made any progress and where are going?

Vic Basili

2001

ISERN Areas of Contribution and Influence

What have been our contributions to Software Engineering:

ExperimentationDesign and ImplementationReplication

Research KnowledgeLevels of UnderstandingDepth of Hypotheses and Questions

Software Engineering PracticeDomains of KnowledgePracticality of Information

And what has been our influence on:the ISERN communitySE researchSE Practice

An example: Reading Experiments

Before ISERN (1986): Code Testing vs. Reading Experiment (UMD)…

First ISERN 1992: First discussion of the requirements reading (DBR)(UMD)

ICSE 1996: Basili presented a talk on the state of experimentation and discussed reading experiments to that point in time.

Set some goals for the maturing of an experimental discipline

The next 6 slides come from that talk

The Maturing of the Experimental Discipline

How is experimentation maturing?Sign of maturity in a field:

level of sophistication of the goals of an experimentunderstanding interesting things about the discipline

For software engineering that might mean:

Can we build models that allow use to measure and differentiate processes and products?

Can we measure the effect of a change in a particular process variable on the product variable?

Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context?

Can we control for product effects, based upon goals, given a particular set of context variables?


How is experimentation maturing?

Sign of maturity in a field: a pattern of knowledge built from a series of experiments

Does the discipline build on prior (knowledge, models, experiments).

Was the study an isolated event?

Did it lead to other studies that made use of this information Have studies been replicated under similar or differing conditions?

Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work?

For example, inspections, in general, are well studied experimentally

However, there has been very little combining of results, replication, analysis of the differentiating variables


How is experimentation maturing?

There is some evidence that researchers appear to be- asking more sophisticated questions- studying relationships between processes/product characteristics - using more studies in the field than in the controlled laboratory

- combining various experimental classes to build knowledge

On such example of evolving knowledge over time, - based upon experimentation and learning is - the evolution of the SEL knowledge - of the effectiveness of reading techniques and methods

Software Engineering Laboratory is a consortium (established in 1975)

- NASA/Goddard Space Flight Center - University of Maryland - Computer Sciences Corporation

Evolution of Knowledge over Time Reading Technology Experiments

This example - shows the combination of multiple experimental designs - provides insight into the effects of different variables on reading - demonstrates replication by other researchers

The experiments start with the early reading vs. testing experiments to various Cleanroom experiments to the development of new reading techniques currently under study to replications at other groups

The experiments are based upon the ideas that Reading is a key technical activity for improving the analysis of all kinds of software documents and we need to better understand its effect

Early experiments (Hetzel, Meyers) showed very little difference between reading and testing

But reading was simply reading, without a technological base


How is experimentation maturing?Several of these experiments have been replicated - under the same and differing contexts

The original analysis technique comparison has been replicated University of Kaiserslautern

Scenario-based reading study variations University of Bari, Italy University of New South Wales, Australia Bell Laboratories, USA University of Trondheim, Norway Bosch, Germanyto better understand the reading variable

ISERN organized explicitly to share knowledge and experiments has membership in the U.S., Europe, Asia, and Australia represents both industry and academia supports the publication of artifacts and laboratory manuals

Its goal is to evolve software engineering knowledge over time, based upon experimentation and learning

What will our future look like?

Experimentation can provide us with - better scientific and engineering basis for the software engineering - better models of - software processes and products - various environmental factors, e.g. the people, the organization - better understanding of the interactions of these models

Practitioners will be provided with - the ability to control and manipulate project solutions - based upon the environment and goals set for the project - knowledge based upon empirical and experimental evidence - of what works and does not work and under what conditions

Researchers will be provided laboratories for experimentation This will require a research plan that will take place over many years - coordinating experiments - evolving with new knowledge

Continuing: Reading Experiments

Since then: new variations and replications

Code Reading vs. TestingUMD (USA), UKL (Germany), U Strathclyde (Scotland), U Sao Paolo (Brazil),

Defect Based Reading UMD, Lucent, U Bari (Italy),

Perspective Based ReadingUMD, IESE, FC-MD, U Bari, UMBC (USA), U Sao Paulo, NTNU (Norway), USC (USA), COPPE/UFRJ (Brazil), Lund U (Sweden)

Object Oriented ReadingUMD, NTNU, U Rio deJaniero, USC, IESE

Other ReadingUMD, UNSW (Australia), U Strathclyde, TU Vienna (Austria), IESE

Reading Experiments: Contribution and Influence

ExperimentationDesign and Implementation

Several designs form full fractional factorial to constant comparison approaches

Are learning the limits of controlled experimentsReplication

Several at universities, often with support fro original experimentersCombining Results

Identifying common factors (Goal Template)

Need to build a stronger body of experimental designbe able to combine the results of experiments of all kindsmake “controlled” type experiments cheaper

And what has been our influence on:the ISERN community

work on the combined experiment for inspectionsSE research

two books on experimentation from ISERN participantsSE Practice

not much so far but some nation wide programs have begun, e.g., CeBASE, ViSEK, …


Research KnowledgeLevels of Understanding

evolution of process based upon empirical studyuse of experiments as learning rather than validation

Depth of Hypotheses and Questionsasking questions about meta issues, such as context variables

Questions from ICSE:Can we build models that allow use to measure and differentiate processes and

products? Yes

Can we measure the effect of a change in a particular process variable on the product variable? Yes

Can we predict the characteristics of a product (values of product variable) based upon the model of the process (values of the process variables), within a particular context? Some, but not really



Questions from ICSE:Does the discipline build on prior (knowledge, models, experiments)?

More than before

Was the study an isolated event? Few isolated events by ISERN personnel

Did it lead to other studies that made use of this information? YesHave studies been replicated under similar or differing conditions? Yes

Does the building of knowledge exist in one research group or environment, or has it spread to others - researchers building on each other's experimental work? More building together



And what has been our influence on:the ISERN community

More working together, sharing experimentsSE research

More experimentalists joining ISERNSE Practice

Some interest in national projects like CeBASE, ViSEK, ESERNET,..

What results do we have so far? For example …

Combining Results from multiple Reading Experiments

We have generated useful empirical results for technique definition guidance

For a reviewer with an average experience level, a procedural approach to defect detection is more effective than a less procedural one.

Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized.

A set of readers of a software artifact are more effective in uncovering defects when each uses a different and specific focus.


Software Engineering PracticeDomains of Knowledge

have been primarily limited to inspection areahave opened up to areas like COTS have attracted empirical modelers which include areas such as cost, quality, and schedule estimation

Practicality of Informationnational experience base activities are trying to reach out to the development communitynot too many members other than researchers

And what has been our influence on:the ISERN communitySE researchSE Practice

What can be packaged for practice, research, education?

Examples of Using Empirical Results for development, research, education

Technique Tailoring

Is tailoring the reading process associated with an inspection worth the effort?

Procedural inspections, based upon specific goals, will find defects related to those goals, so inspections can be customized. (UMD)

Implications for empirically based software development process: The better you can articulate your goals, the more effectively you can choose and tailor process.

Implications for software engineering research:It is important to empirically study the effects of processes on product

Implications for software engineering education:Don’t teach that there is a one size fits all process; teach how to tailor processes


Technique Selection Guidance

When should you use a procedural approach to code reviewing? For a reviewer with an average experience level, a procedural approach to defect detection is

more effective than a less procedural one (UMD, USC)

Implications for empirically based software development process: Experts might be more effective working on their own but most people should apply a procedural

approach. Novices need training.

Implications for software engineering research:How can we improve document reading procedures based upon how experts analyze documents?

Implications for software engineering education:Effective procedures that can be taught for reviewing documents


Technique Selection Guidance

When are peer reviews more effective than functional testing? Peer reviews are more effective than functional testing for faults of omission and incorrect

specification (UMD, USC)

Implications for empirically based software development process: If, for a given project set, there is an expectation of a larger number of faults of omission or incorrect

facts than use peer reviews.

Implications for software engineering research:How can peer reviews be improved with better reading techniques for faults of omission and incorrect

fact?

Implications for software engineering education:Teach how to experiment with and choose the appropriate analytic techniques

ISERN Areas Challenges

ExperimentationHow do we define more effective and efficient experimental techniques?How do we combine results from experiments?How do we deal with (define, study) meta data? How doe we package lab manuals, experience bases?

Research KnowledgeHow do we define the next levels of research questions?What other processes should we study? Other analytic (testing), creative (design)? What product properties should we be studying? Mostly have looked at the relationship between process and product characteristics. What else should we be studying?In our grand inspection experiment, should we be studying ourselves?How do we get more researchers interested in empirically studying their techniques?

Software Engineering PracticeHow do we ,package our results for practice?How do we get more industrial members involved?

Example of Specific Experimental Issues

In looking at the relationship between process and product, there are many questions that need to be covered:

Process variable (Independent variable) issues:How do we define/specify the process? How do we account for process conformance?

Effectiveness of Product (Dependent variable) issues:How do we select good criteria for effectiveness?

Context Variables (meta data) Issues:What subjects are performing the process? What characterizes their experience, environment,…?

Questions associated with the variables need to be further specified and documented for replication

Varying the values of these variables allow us to vary the detailed hypothesessupport validity of study results

ISERN Open Issues, Grand Challenges or Have we made any progress and where are going? Vic Basili...

Documents

Transcript of ISERN Open Issues, Grand Challenges or Have we made any progress and where are going? Vic Basili...