1 Evidence-Based Software Engineering and Systematic Reviews Barbara Kitchenham.
-
date post
22-Dec-2015 -
Category
Documents
-
view
223 -
download
0
Transcript of 1 Evidence-Based Software Engineering and Systematic Reviews Barbara Kitchenham.
1
Evidence-Based Software Engineeringand Systematic ReviewsBarbara Kitchenham
2
Agenda
The evidence-based paradigm Evidence-Based Software
Engineering (EBSE) Systematic Reviews
3
The Evidence-Based Paradigm
Evidence-based medicine has changed research practices Medical researchers found
• Failure to organise existing medical research cost lives• Clinical judgement of experts worse than systematic
reviews Evidence-based paradigm adopted by many
other disciplines providing service to public Social policy Education Psychiatry
4
Goal of EBSE
EBM: Integration of best research evidence with clinical expertise and patient values
EBSE: Adapted from Evidence-Based Medicine To provide the means by which current best evidence from
research can be integrated with practical experience and human values in the decision making process regarding the development and maintenance of software
Anticipated benefits Common goals for research groups Help for practitioners adopting new technologies Means to improve dependability Increase acceptability of software-intensive systems Input to certification process
5
Practicing EBSE
1. Convert information need into answerable question
2. Track down best evidence3. Critically appraise evidence4. Integrate critical appraisal with SE
expertise and stakeholder requirements
5. Evaluate and improve above steps
6
Systematic Reviews - 1/2
A systematic review is An overview of research studies that uses
explicit and reproducible methods Systematic reviews aim to synthesise
existing research Fairly (without bias) Rigorously (according to a defined
procedure) Openly (ensuring that the review procedure
is visible to other researchers)
7
Systematic Reviews – 2/2
Support Evidence-based paradigm Start from a well-defined question
• Step 1
Define a repeatable strategy for searching the literature
• Step 2
Critically assess relevant literature• Step 3
Synthesise literature• Step 4 (but only partially)
8
Advantages Provide information about effects of a
phenomenon across wide range of settings Essential for SE where we have sampling
problems Consistent results provide evidence that
phenomena are• Robust• Transferable
Inconsistent results• Allow sources of variation to be studied
Meta-analysis possible for quantitative studies
9
Anticipated Benefits
Create a firm foundation for future research• Position your own research in the context of existing
research Close areas where no further research is
necessary Uncover areas where research is necessary Help the development of new theories
Identify common underlying trends Identify explanations for conflicting results
Should be a standard research methodology
10
Disadvantages
Require more effort than informal reviews
Difficult for lone researchersStandards require two researchers
• Minimising individual bias
Incompatible with requirements for short papers
11
Value of Systematic Reviews Can contradict “common knowledge”
Jørgensen and Moløkken reviewed surveys of project overruns
• Standish CHAOS report is out of step with other research
• May have used inappropriate methodology
Jørgensen reviewed evidence about expert opinion estimates
• No consistent support for view that models are better than human estimators
12
Systematic Review ProcessDevelop Review Protocol
Validate Review ProtocolPlan Review
Conduct Review
Document Review
Synthesise Data
Write Review Report
Validate Report
Identify Relevant Research
Select Primary Studies
Extract Required Data
Assess Study Quality
13
Developing the Protocol
Review protocol Specifies methods to be used for a
systematic review Predefined protocol
• Reduces researcher bias by reducing opportunity for
• Selection of papers driven by researcher expectations
• Changing the research question to fit the results of the searches
Good practice for any empirical study
14
Protocol Contents -1/2
BackgroundRationale for survey
Research questionCritical to define this before starting
the researchStrategy used to search for primary
sources• Individual studies of the phenomenon of
interest
15
Protocol Contents – 2/2 Strategy to find primary studies
Search terms, resources, databases, journals, conferences
Procedures for storing references How publication bias will be handled
• Grey literature• Direct approach to active researchers
How completeness will be determined• Useful to have the baseline paper to set start date
Selection Strategy Inclusion/exclusion criteria
• Handling multiple papers on one experiment
16
Protocol Contents- 2/3
Quality assessment criteria Criteria used to evaluate quality of primary sources
Data extraction What data will be extracted from each primary source How to handle missing information How data reliability will be addressed
• Usually multiple reviewers Where data will be stored
Procedures for data synthesis Formats for summarising data Measures and analysis if meta-analysis is proposed
Should tested during protocol construction
17
Research Question – 1/2 Question types for EBSE
Assessing the effect of an SE technology Assessing the frequency or rate of a project
development factor• E.g. Rate of project failures
Identifying cost and risk factors Identifying impact of technology on reliability,
performance, cost Possible to have more general questions for
other purposes Review of research in software engineering
(Glass, et al., 2002)
18
Research Question – 2/2 Question structure
Population• People, projects types, applications types affected by
the intervention Intervention
• Software method, tool, procedure Outcomes
• Impact of technology in terms relevant to practitioners• Cost, quality, time to market
Experimental designs• Any constraints on type of primary studies to be
included
19
Next steps are easy!?
Conduct the reviewEnact the protocol
• Expect further iterations of • Search strategy• Selection criteria• Data extraction
Record any deviations from protocol Document the Review
Using procedures defined in protocol
20
Conclusions
Evidence-based approachRevolutionised medicineMay be relevant to SE
Systematic reviewsSupport the evidence-based approachValuable as a research tool
• Even if we don’t accept EBSE
21
References
Australian National Health and Medical Research Council. How to review the evidence: systematic identification and review of the scientific literature, 2000. IBSN 186-4960329 .
Australian National Health and Medical Research Council. How to use the evidence: assessment and application of scientific evidence. February 2000, ISBN 0 642 43295 2.
Cochrane Collaboration. Cochrane Reviewers’ Handbook. Version 4.2.1. December 2003.
Glass, R.L., Vessey, I., Ramesh, V. Research in software engineering: an analysis of the literature. IST 44, 2002, pp491-506
Magne Jørgensen and Kjetil Moløkken. How large are Software Cost Overruns? Critical Comments on the Standish Group’s CHAOS Reports, http://www.simula.no/publication_one.php?publication_id=711, 2004.
Magne Jørgensen. A Review of Studies on Expert Estimation of Software Development Effort. Journal Systems and Software, Vol 70, Issues 1-2, 2004, pp 37-60.
22
ReferencesKhan, Khalid, S., ter Riet, Gerben., Glanville, Julia., Sowden, Amanda, J. and Kleijnen, Jo. (eds) Undertaking Systematic Review of Research on Effectiveness. CRD’s Guidance for those Carrying Out or Commissioning Reviews. CRD Report Number 4 (2nd Edition), NHS Centre for Reviews and Dissemination, University of York, IBSN 1 900640 20 1, March 2001.
Kitchenham, Barbara. Procedures for Performing Systematic Reviews, Joint Technical Rreport, Keele University TR/SE-0401 and NICTA 0400011T.1, July 2004.
Pai, Madhukar, McCullovch, Michael, Gorman, Jennifer D., Pai, Nitika, Enanoria, Wayne, Kennedy, Gail, Tharyan, Prathap, Colford, John M. Jnr. Systematic reviews and meta-analysis: An illustrated, step-by-step guide. The National medical Journal of India, 17(2) 2004, pp 86-95.
Sackett, D.L., Straus, S.E., Richardson, W.S., Rosenberg, W., and Haynes, R.B. Evidence-Based Medicine: How to Practice and Teach EBM, Second Edition, Churchill Livingstone: Edinburgh, 2000.