From myths and fashions to evidence-based software

engineering!Magne Jørgensen

Most of the methods below have once been (some still are) fashionable ...

•  The Waterfall model, the sashimi model, agile development, rapid application development (RAD), unified process (UP), lean development, modified waterfall model, spiral model development, iterative and incremental development, evolutionary development (EVO), feature driven development (FDD), design to cost, 4 cycle of control (4CC) framework, design to tools, re-used based development, rapid prototyping, timebox development, joint application development (JAD), adaptive software development, dynamic systems development method (DSDM), extreme programming (XP), pragmatic programming, scrum, test driven development (TDD), model-driven development, agile unified process, behavior driven development, code and fix, design driven development, V-model-based development, solution delivery, cleanroom development, ....."

•  Did we go from RUP (previous fashion) to agile (current fashion) based on good evidence?"

The paper clip was invented by a Norwegian

Short men are more aggressive (The Napoleon complex)

Most (93%) of our communication is non-verbal

There were/is a software crisis!

(page 13 of their 1994-report): “We then called and mailed a number of confidential surveys to a random sample of top IT executives, asking them to share failure stories.”

0%"

20%"

40%"

60%"

80%"

100%"

120%"

140%"

160%"

180%"

200%"

1994" 1996" 1998" 2000" 2002" 2004" 2006" 2008" 2010" 2012"

Average percent cost overrun!

Difficult to remove a myth!"

"

"

"

"

"

Years of critique may, however, have had an small effect (from the CHAOS Report – 2013):"

45% of features of “traditional projects” are never used(source: The Standish Group, XP 2002)!

No-one seems to know (and the Standish Group does not tell) anything about this study!

Why do so many believe (and use) this non-interpretable, non-validated claim?

They benefit from it (agile community) + confirmation bias (we all know at least one instance that fit the claim)

14% Waterfall and 42% of Agile projects are successful(source: The Standish Group, The Chaos Manifesto 2012)!

Successful = “On cost, on schedule and with specified functionality” Can you spot a serious error of this comparison?

There is an increase in cost of removing errors in later phases!

http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf

The ease of creating myths:!Are risk-willing or risk-averse developers better?

Study design: Research evidence + Self-generated argument.

Question: Based on your experience, do you think that risk-willing programmers are better than risk-averse programmers?

1 (totally agree) – 5 (No difference) - 10 (totally disagree) Neutral group: Average 5.0

Group A:

Group B: Initially

Average 3.3 Debriefing Average 2: 3.5 2 weeks later Average 3: 3.5

Initially Average 5.4 Debriefing Average 2: 5.0 2 weeks later Average 3: 4.9

When making a decision or choice, the world is no more the same (Dan Gilbert)!

ted.com/talks/lang/eng/dan_gilbert_asks_why_are_we_happy.html

“I see it when I believe it” vs “I believe it when I see it”!

•  Design: !–  Data sets with randomly set performance data comparing

“traditional” and “agile” methods. !–  Survey of each developer’s belief in agile methods!

•  Question: How much do you, based on the data set, agree in: “Use of agile methods has caused a better performance when looking at the combination of productivity and user satisfaction.”!

•  Result: "–  Previous belief in agile determined

what they saw in the random data"

Very satisfiedSatisfiedDissatisfied

9

8

7

6

5

4

3

2

1

0

Very satisfiedSatisfiedDissatisfied

Agile

User Satisfaction

Pro

du

ctiv

ity

(Fu

nct

ion

Po

ints

/W

ork

-da

y)

Traditional

Individual Value Plot of Productivity

Panel variable: Development Method

HOW TO MAKE SOFTWARE ENGINEERING MORE EVIDENCE-BASED?!

Evidence-based software engineering (EBSE)!The main steps of EBSE are as follows:"

1.  Convert a relevant problem or need for information into an answerable question."

2.  Search the literature and practice-based experience for the best available evidence to answer the question. (+ create own local evidence, if needed)"

3.  Critically appraise the evidence for its validity, impact, and applicability."

4.  Integrate the appraised evidence with practical experience and the client's values and circumstances to make decisions about practice."

5.  Evaluate performance in comparison with previous performance and seek ways to improve it."

"

Learn to formulate questions meaningful for your context/challenge/problem!

The question “Is Agile better than Traditional methods?” is NOT answerable."

•  What is agile?"

•  What is traditional?"

•  What is better?"

•  What is the context?"

Be critical (myth busting) when claims are made!

1.  Find out what is meant by the claim."–  Is it possible to falsify the claim? If not, what is the function of the

claim?"

2.  Put yourself in a ”critical mode”"–  Raise the awareness of the tendency to accept claims, even without

valid evidence, when you agree/it seems intuitively correct."–  Reflect on what you would consider as valid evidence to support the

claim."–  Vested interests? "–  Do you agree because of the source?"

3.  Collect and evaluate evidence"–  Research-based, practice-base, and “own” evidence"

4.  Synthesize evidence and conclude (if possible)"

Learn to question what statements and claims means

Know how to evaluate argumentation"

Data Claim

Backing

Warrant

Qualifier Reservation

Learn to use google scholar(search for research-based evidence)!

Learn to collect and evaluate practice-based experience!

•  Methods similar to evaluation of research-based evidence and claims"

•  Be aware of “organizational over-learning”"

Create local evidence!

•  Experimentation is simpler than you think"– Pilot studies"– Trial-sourcing"– Controlled experiments"

Is it realistic to achieve an evidence-based software engineering

profession?!

•  Moderate optimist based on my experience with teaching this to students and professionals"

•  Challenges:"– Not much research"– High number of different contexts"

•  Opportunities:"– Better use of practice-based evidence"– More common to generate local evidence"

Coffee dehydrates your body?