Finding common denominators for agile software development ...830452/FULLTEXT01.pdf · Software...

Master Thesis Software Engineering Thesis no: MSE-2011-55 Month Year

School of Computing Blekinge Institute of Technology SE-371 79 Karlskrona Sweden

Finding common denominators for agile software development: a

systematic literature review

Saripalli Phani Shashank David Hem Paul Darse

II

This thesis is submitted to the School of Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 20 weeks of full time studies.

Contact Information: Author(s): Saripalli Phani shashank E-mail: [email protected] David Hem Paul Darse E-mail: [email protected]

University advisor: Dr Tony Gorschek Software Engineering

School of Computing Blekinge Institute of Technology SE-371 79 Karlskrona Sweden

Internet : www.bth.se/com Phone : +46 455 38 50 00 Fax : +46 455 38 50 57

IV

ABSTRACT Context: In the last decade, Agile software development methods were proposed as a response to plan-driven methods. The main aim for this paradigm shift was to cope up with constant changes. Core values that are central to agile methods are described in Agile Manifesto. Agile practices define how agile methods are implemented in practice. Objectives: This study aims to investigate what practices were considered and adopted as part of agile philosophy, and identify evidence on the use of agile practices in reference to what defines agile. Methods: We conducted a systematic literature review. The review includes 186 primary studies published between 2000 and 2010. Quality assessment based on rigor and relevance as identified in the selected papers has been conducted. Non-empirical papers were classified based on themes they addressed. Empirical papers were analyzed based on two factors:

1. if they described the use of agile practices for a software project/product in general, 2. if they described the use of agile practices for a specific purpose/activity

Application type, team size and experience of subjects in the primary studies were extracted. The alignment between practices reported in the studies with the core agile values is analyzed. Results: A total of 119 studies were conducted in industry and 67 in academia. Nearly half the papers published by researchers are non-empirical and present analysis of agile practices in various topics. Over half of the empirical papers were focused on evaluation/assessment of a specific aspect of agile. ‘Pair programming’ received most attention in this direction. Among the empirical studies that described agile practices for academic projects, ‘Pair programming’ and ‘Test driven development’ were the most considered practices. Among the 119 studies conducted in industry, 93 studies described the use of agile practices for the overall project/product development while the remaining studies described experiences of single practices like Pair Programming or the modification/adoption of agile for non-software aspects in software projects. Practices adopted have been ranked based on team size, practitioners’ experience and application type. A method for agile assessment based on core agile values is developed and exemplified. Conclusions: Practices that are considered agile vary with context although ‘Pair programming’, ‘Continuous integration’, ‘Refactoring and Testing continuous throughout the project are the most common practices used. ‘Test driven development’ was not highly adopted by practitioners. But it was not clear if test cases were written before code construction in projects where continuous testing was performed. However, this was completely opposite in academic projects. Practices ‘On-site customer’ and ‘Stand-ups’ are frequently modified. In many of the studies inspected, practices adopted were not aligned with agile values. Although practices related to technical aspects of software development are in place, agile practices that focus aspects like ‘working together’ and ‘motivated individuals’ are often not used in practice. Moreover, many of the studies were not reported to an extent that it was possible to draw any inferences on the usability/applicability, benefits and limitations of the agile practices. To this end, a major implication is to improve the quality of the studies and a need for a common framework to report the results of agile practices’ adoption.

Keywords: Agile practices, agile software development, Systematic literature review

V

ACKNOWLEDGMENTS We would like to extend our gratitude to Dr. Tony Gorschek for giving the opportunity to work

on our Master thesis under his guidance. His guidance, feedback and support throughout the thesis has been immensely valuable.

We would like to thank our parents for their continuous support, motivation and patience during our course of time in Sweden. We are thankful to Abhinaya, Rajesh and Revanth for their kind support all the time.

1

List of tables .......................................................................................................................................................................................... 2!List of figures ....................................................................................................................................................................................... 3!1! Introduction ................................................................................................................................................................................... 4!2! Background and related work ................................................................................................................................................... 5!

2.1 Agile software development .................................................................................................................................................... 5!2.2 Related work .............................................................................................................................................................................. 6!

3! Research methodology ................................................................................................................................................................ 7!3.1 Search strategy ........................................................................................................................................................................... 8!3.2 Study selection criteria .............................................................................................................................................................. 8!3.3! Primary study selection ...................................................................................................................................................... 10!3.4! Data extraction & study quality assessment ................................................................................................................... 10!3.5! Procedure for assessing research rigor and relevance ................................................................................................... 11!3.6! Validity threats .................................................................................................................................................................... 12!

3.6.1 Publication bias ................................................................................................................................................................. 12!3.6.2 Selection of primary studies ........................................................................................................................................... 12!3.6.3 Data extraction .................................................................................................................................................................. 12!

4 Results and analysis ................................................................................................................................................................... 13!4.1! Overview of the studies ...................................................................................................................................................... 13!

4.1.1 Publication year ................................................................................................................................................................ 13!4.1.2 Research method .............................................................................................................................................................. 13!4.1.3 Context ............................................................................................................................................................................... 13!4.1.4 Study design ...................................................................................................................................................................... 13!4.1.5 Validity evaluations ......................................................................................................................................................... 13!4.1.6 Rigor ................................................................................................................................................................................... 13!4.1.7 Relevance ........................................................................................................................................................................... 14!4.1.8 Practice classifications ...................................................................................................................................................... 15!

4.2 What agile practices are most cited in literature? (RQ1) ................................................................................................... 17!4.3! What agile practices are most adopted in practice? (RQ1.1) ......................................................................................... 17!

4.3.1 Research studies ............................................................................................................................................................... 20!4.4 What are the context specific aspects of agile practices reported? (RQ1.2) .................................................................... 22!

4.4.1Team size ............................................................................................................................................................................ 22!4.4.2 Application type ............................................................................................................................................................... 24!4.4.3 Context specific practices ................................................................................................................................................ 26!4.4.4 HRR .................................................................................................................................................................................... 29!

4.5 What modifications to agile practices have been reported? (RQ 1.3) .............................................................................. 29!4.6 What is the strength of evidence available on the use of agile practices in reference to agile philosophy? (RQ2) .. 31!

5 A method for agile assessment based on core agile values (MAA-CAV) .............................................................................. 34!5.1 Step 3 - Calculating measures ................................................................................................................................................ 34!5.2 Step 4 – Measure representation ........................................................................................................................................... 34!

6 Conclusion ....................................................................................................................................................................................... 37!References ........................................................................................................................................................................................... 37!Systematic literature review references ......................................................................................................................................... 39!Appendix A Overview of studies: extended material ................................................................................................................. 44!Appendix B Other practices ............................................................................................................................................................. 45!Appendix C Other empirical studies in academia ....................................................................................................................... 49!Appendix D Practice modifications/adaptations ......................................................................................................................... 51!Appendix E MAA - CAV examplification ..................................................................................................................................... 54!

2

LIST OF TABLES

Table 1 Differences between plan-driven and agile 5 Table 2 Agile principles 5 Table 3 Research Questions 8 Table 4 Search String 8 Table 5 Data Extraction 11 Table 6 Research Rigor 12 Table 7 Relevance 12 Table 8 Company Size Reported 14 Table 9 Study Rigor 14 Table 10 Relevance Across Studied 15 Table 11 Practice Classifications 15 Table 12 Citation Count of Practices 17 Table 13 Practices Found in Empirical Studies 17 Table 14 Research Studies on Software Development in General 20 Table 15 Customer Practices Proposed by Researchers 21 Table 16 Small and Large Teams 23 Table 17 Top Ranked Practices for Small Teams 24 Table 18 Top Ranked Practices for Large Teams 24 Table 19 Application Type 24 Table 20 Practice Ranking and Application Type 26 Table 21 Agile Practices Supporting Non-Technical Activities 26 Table 22 Empirical Studies Focusing on PP 27 Table 23 Empirical Studies with Focus on Software Development Process Model 28 Table 24 Empirical Studies with Focus on TDD 28 Table 25 Communication Practices 29 Table 26 Customer Collaboration Adaptations 29 Table 27 Adaptations for Stand-ups 30 Table 28 Adaptations for Project Progress 30 Table 29 Adaptations for Conducting Retrospectives 31 Table 30 Continuous Testing in Agile 31 Table 31 Mapping Values to Agile Principles 32 Table 32 Value Alignment Among Studies with Rigor>4 32 Table 33 Value Alignment Among Studies with Rigor<3 33 Table 34 Value Alignment Among Studies with Rigor=3 33 Table 35 Measure Definitions for Values and Principles 34 Table 36 Measure Illustration 35 Table 37 Template for Agile Assessment 36

SARIPALLI AND DARSE: FINDING COMMON DENOMINATORS FOR AGILE SOFTWARE DEVELOPMENT: A SYSTEMATIC LITERATURE REVIEW 3

LIST OF FIGURES

Fig. 1 Review Process 7 Fig. 2 Search Strategy 8 Fig. 3 Primary Studies Selection 9 Fig. 4 Distribution of Publications Across Years 14 Fig. 5 Research Methods 14 Fig. 6 HRR 15 Fig. 7 Team Size and Practitioner’s Experience 22 Fig. 8 The Four Steps of MAA-CAV 34 Fig. 9 Levels in Agile Assessment 36 Fig. 10 Agile Assessment Visualization 36

4

Finding common denominators for agile software development: a systematic literature

review Phani Shashank Saripalli and David Hem Paul Darse

—————————— ——————————

1 INTRODUCTIONOFTWARE is an important part of our daily life and an important factor for all industries [3]. Software de-velopment is a human centered and knowledge inten-

sive activity [7]. In today’s world software development is fast paced because of frequently changing customer needs [1]. Failing to keep the development pace may reduce the probability of market dominance [2]. Meeting customers’ needs is also equally important. Changing customer needs, i.e. product requirements (functionality and quali-ty), crossing allotted budget are also common to software industry. Therefore, software industry should respond to the challenge of requirements volatility and in turn should be flexible enough to accommodate late changes to the product [8]. There has been an increase in the atten-tion on software development methodologies by re-searchers and practitioners [4][5], to bring fastness and methodological improvements. This was aimed to ad-dress the challenges and bottlenecks in the early devel-opment methodologies, which often have been criticized for being heavy and involving large amounts of docu-mentation [9] [10] [11]. Plan-driven software development is driven by rigorous planning and is document centric. These methods contain sequential steps (phases). A prom-inent plan-driven development method is the waterfall model [1] suggested by Royce [13]. Other examples of plan-driven software development are the Rational Uni-fied Process (RUP) [14] and the V-Model [15]. Plan-driven methods have been criticized on account of late delivery of the software, limited budget constraints and lack of flexibility in welcoming late requirements [8] [16] [18].

To sustain a competitive advantage, software industry should continuously improve their processes and practic-es and also bring change by adopting new development practices [52]. A new trend of software development has emerged to address the challenges in plan-driven devel-opment. This is known as agile software development. Agile development consists of several methods, also known as agile methods, proposed in the last decade with the aim of making software development lighter and fast-er [9] [10]. These were presented in the form of “Agile Manifesto” [19]. Agile methods were developed based on the four core agile values and twelve principles listed in The Agile Manifesto. The most referenced methods are Extreme Programming (XP) [20], Scrum [21], Lean [22], Crystal Methodologies [23], Feature-driven Development

[24], and the Dynamic software development method (DSDM) [25]. These methods marked a new trend in the way software was developed [9]. The main contrasting differences between plan-based methods and agile in-clude fundamental assumptions in software develop-ment, management style, knowledge management, com-munication, organizational structure and quality control [54]. Agile methods provide a compromise between no planning at all, and too much planning with ability for information systems evolution [27]. Agile practices define how agile principles are implemented in practice [28]. Agile practices are the various ways of working put into practice to gain agility, i.e. the ability to increase devel-opment pace and respond to changes.

Publications show an increase in the evidence on the adoption/use of agile methods. For example, surveys [29] [30] gave a bigger picture of the adoption of agile meth-ods. We can find a variety of contexts in which agile adoptions have taken place: small and medium sized companies [29], large and distributed environments [31] [32]. Evidence shows that agile methods are being used in a variety of domains like embedded systems, and telecom [29] [33]. Initially agile methods were considered suitable for only small teams developing simple applications [44] [45]. Lately it has been found that agile methods can be suitable for large teams developing complex applications [45] [46]. Experiences show certain agile practices are adopted and used to complement the existing develop-ment process and sometimes methods are modified to fit the development context [34] [35]. In reality, adoption of agile methods and practices depend upon the develop-ment context, i.e. the type of software itself and other characteristics like, development effort, team size etc. [35]. It is observed that companies looking for implementing agile methods and practices do not have the same level of consideration towards the existing agile practices [36]. Krznaik et al. [37] coined the phrase “working set” which refers to the set of agile practices that bring out the best impact in a project. Over the years, the concept of select-ing agile practices to suit the projects in hand has in-creased [37] [38]. However, evidence of what agile prac-tices work(ed) well together, and in what contexts, is missing. Also, there has not been much research in defin-ing what practices can be considered within the broad umbrella term ‘agile’ [29]. The Forrester report [29] states

S


that confusion around what constitutes ‘agile’ increase as the awareness of agile development grows. Practices are transient in nature, i.e. one can find introduction of new practices or de-commitment of practices used in previous projects [37]. It is also found that project success depends on the choice of agile practices [43]. A recent review on agile methods and their implementation by Dybå and Dingsøyr [39] revealed that agile methods XP and Scrum received the most attention in practice and other methods did not receive much attention. Their review did not fo-cus on practices that were implemented as part of these methods and the context variables (for e.g. company size). The evidence of how agile development is carried in prac-tice, practice selection and the impact is relatively nascent and weak [40].

Inspired by the phrase “working set” [37] and the state of empirical evidence on agile development [39] [40] we conducted a Systematic Literature review (SLR) [42] to investigate practices that were implemented as part of agile philosophy and methodology, identifying relevance issues for implementation based on the context variables found in the review, and finally to identify how compa-nies attaint the core agile values described in The Agile manifesto in their endeavors to be ‘agile’. We follow the idea of Evidence-Based Software Engineering (EBSE) par-adigm [41] to assess industrial relevance. The remainder of the paper is organized as follows: Back-ground and related work is given in Section 2, research methodology in Section 3. Section 4 is dedicated to results and analysis of the study. In Section 5, we described our proposed method for Agile assessment based on the four core agile values (MAA-CAV). The method is illustrated using a case example. In section 6 a short discussion on our study is given. Conclusions are given in Section 7. Extended material to our study is given in Appendices A-E

2 BACKGROUND AND RELATED WORK 2.1 Agile software development In the last decade several software development methods were proposed as a response to plan based methods to make software development faster and lighter [9]. “Agile development” was coined in the year 2001 [47]. This was presented in the form of what is known as The Agile Manifesto [19]. The term “paradigm” was used by Kuhn [48] to describe a change in coherent tradition of scientific research that includes research agenda, techniques and knowledge. A shift in paradigm took place in order to tackle the problem of requirements volatility, and to de-velop software in iterations and new requirements can be accumulated [49].!The Agile Manifesto defines four values that form the core of agile software development. These are the follow-ing: (1) Individuals and interactions over processes and tools, (2) Working software over comprehensive documentation, (3) Customer collaboration over contract negotiation and (4) Re-sponding to change over following a plan. These four values describe the contrasting differences between plan-driven software development and agile software development.

RUP is not considered agile as this process heavily de-pends on planning and documentation [1]. Contrasting differences between plan-driven and agile software de-velopment are given in Table 1.

TABLE 1 DIFFERENCES BETWEEN PLAN-DRIVEN AND AGILE

Dimension Plan-driven Agile Assump-tion/Planning

Output well defined and planned from the beginning

Output not known concretely until the solution is developed. A high-level plan-ning is done and detailed planning in iterations.

Requirements Detailed re-quirements specification and agreement in the form of contract.

Welcome new & late requirements. Regular contact with customer for product features.

De-sign/Architecture

Detailed and accomplished in a specific phase.

Not initially de-tailed and contin-uous architectural evaluation.

Development Accomplished in one phase and is specifica-tion-driven.

Throughout the project and in various iterations. Code changes based on new requirements from customers.

Verification & Validation

Performed after the coding phase. Formal roles (testers, inspectors) are available.

Testing done throughout the project. No specif-ic roles. In some cases, testing is performed in par-allel with devel-opment.

Agile Manifesto states twelve principles that describe

how agile values can be attained [28]. These are given in Table 2.

TABLE 2

AGILE PRINCIPLES ID: Agile Principle (AP) PRINCIPLE

AP01 Customer satisfaction through early and continuous delivery of software.

AP02 Welcome changing requirements, even late in development.

AP03 Deliver working software frequently. AP04 Business and developers work together.

AP05 Build projects around motivated indi-viduals.

6

AP06 Face-to-face conversation. AP07 Working software.

AP08 Maintaining constant pace continuous-ly.

AP09 Technical excellence. AP10 Simplicity. AP 11 Self-organizing teams. AP 12 Continuous reflection.

In the context of Software Development, agility can be

defined as “agility means to strip away as much of the heavi-ness, commonly associated with the traditional software devel-opment methodologies, as possible to promote quick response to changing environments, changes in user requirements, acceler-ated project deadlines and the like” [50]. Agile practices de-fine how agile principles are implemented in practice [9]. In the context of software business, practices can be de-fined “to relate to the different well established ways that dif-ferent development companies have cultivated to cope with re-quirements posed by customers, business environment, market section, and software products” [51]. Each agile method has its own set of practices so that software development is aligned with agile values and principles listed in Agile Manifesto. For Example, XP has twelve practices. Practic-es are defined to implement agile principles. Agile meth-ods have their own set of practices describing various ways of implementing agile principles.

Agile practices have transient nature [37]. There is no definite set of practices that are used, instead practices are selected and modified based on the situations (context variables) at hand [36] [37] [38]. The phrase “working set” [37] is context dependent. Practices should be selected in such a way that they bring the highest impact on project while maintaining the goals of agile.

2.2 Related work Ivarsson and Gorschek [52] performed a review of Re-quirements Engineering journal where they assessed em-pirical evaluations pertaining to Requirements Engineer-ing (RE). They assessed empirical evaluations and identi-fied research rigor and relevance as a means to provide decision support for technology transfer in RE. Although their work falls under RE research, they highlighted the need to provide decision support to practitioners, a well as researchers, as part of EBSE.

There is a plethora of empirical studies and a few SLRs’ on agile development. [9] [39] [53] [54] [40]. The first review on agile was by Abrahamsson et al. [9] in 2002. They discussed the concept of agile software devel-opment, methods and their practices. They found that Scrum majorly covers project management dimensions of software development and also stated that only DSDM gives full coverage of all phases of software development. They also observed that evidence for agile methods is anecdotal and empirical validations supporting the claims were scarce.

Cohen et al. [53] published a review in 2004 that found that agile had roots in other disciplines such as manufacturing industry. Furthermore, they discussed the relations between agile and the Capability Maturity Mod-

el (CMM). No strong evidence or support for the intro-duction of agile methods was found at that time. They also stated that agile methods would not rule out tradi-tional plan-driven methods. Cohen et al. also observed that context variables, or factors such as team size and application domain would mainly affect the adoption of agile methods and selection.

Erickson et al. [54] conducted a review on XP. They found that there was only a small number of empirical studies on the success of XP. They identified a need to study XP practices in order to find what practices work in practice.

A recent review of agile methods by Diva and Dingsøyr [39] in 2008 identified 33 primary studies. They focused on empirical studies, and how agile development was conducted. They classified the studies into four cate-gories: introduction and adoption, human and social fac-tors, perceptions of agile methods, and comparative stud-ies. XP was found to be the most prominent agile method in terms of citations and also adoption. Further, Scrum was found to be a useful project management method in industry. State of the art and practice suffers from low evidence as the majority of the studies suffer from low research rigor and also, the majority of studies were not performed with experienced agile teams.

Laanti et al. [55] conducted a survey at Nokia, a large telecom company to explore employees’ opinions on agile adaption. Their survey received positive feedback on ag-ile methods, namely higher job satisfaction, increased quality and earlier detection of happiness. A majority of survey respondents stated that they would not return to their old way of working, i.e. plan-driven development. The population of the survey consists of 1000 respondents in seven different countries. The study reports that indus-trial adoption of agile is positive and efforts on research are a valid one.

Williams [56] presented a review of agile methods and practices. She observed that software development teams select a subset of agile practices and create their own hy-brid development method instead of adhering to all prac-tices prescribed in a given agile method.

Hossain et al. [57] conducted a review on the use of Scrum practices in the context of Global Software Devel-opment (GSE). They found that many of the scrum prac-tices were modified or extended in order to fit the needs of distributed development. The study presented in this paper a SLR on agile practices. The aim of the study is based on the extension of the studies that report imple-mentation of agile practices, in the context of both indus-try and academia. Our focus is to investigate what agile practices have been used given a certain set of context variables and, not methods per se. Further focus of our study is to identify what practices constitute agile philos-ophy and identify how practices have been modified to suit situations at hand. Further contribution of our study is a method to measure the state of “being agile” based on the four core agile values, principles and practices put into practice.


3 RESEARCH METHODOLOGY We describe our Research methodology in this section. This includes design and execution of our SLR. The basic outline of our review process is given in Fig. 1. This sec-tion is concluded with a discussion on threats to the va-lidity and credibility of our review. Step 1 of our review, i.e. need for a systematic review, is motivated in the related work section. Furthermore, to check if similar research was done we did a systematic search in the databases Inspec and Compendex. A Main reason for searching in these two databases is that often research papers are indexed in Inspec and Compendex and also, these are the primary search venues for SLRs in software engineering. Biolchini et al [58] have defined various synonyms for SLR. We formulated a search string using these synonyms and within Title, Abstract and Keywords. The following search string was used:

(agile OR “extreme programming” OR xp OR scrum) AND (SLR OR "systematic review" OR “literature review” OR "re-search review" OR "research synthesis" OR "research integra-tion" OR "systematic overview" OR "systematic research syn-thesis" OR "integrative research review" OR "integrative re-view")

We found that none of the search results addressed the overall aim of our study. Our review covers a wider per-spective rather than a method per se, and also our re-search questions (step 2) are different from the ones found in the search results. Our research questions are given in Table 3 with rationale. For examples, RQ1 at-tempts to identify practices that are more cited, hence more popular in terms of adoption and implementation. Furthermore, we looked for modifications to agile prac-tices that have been made in various contexts. This is an- swered in a sub-question of RQ1 (RQ1.3). Mapping of context variables and most cited practices is the focus of RQ1.2. The focus of our RQ2 is on finding the strength of evidence found in the articles reporting the use of agile practices. The main objective here is to assess the articles in terms of research rigor and relevance. This kind of assessment will increase the trust placed in the evidence itself [59]. Based on the answers to RQ1 and 2, a method to identify the state of “being agile” is developed and exemplified.

We developed our review protocol with the aim to re-duce research bias [42] and provide scope for replication of our study in the future. We developed protocol (step 3) and evaluated (step 4) in a series of iterations. A research-er who had prior experience in SLRs evaluated the review protocol. By taking his feedback and examining the SLR [39], we improved our research design. This was mainly helpful in identifying keywords for search and various databases to look in. Our final review protocol is de-scribed in the following sections.

Fig. 1. Review process

Stage 1: Need for a systematic review

Stage 2: Defining research questions

Stage 3: Develop the review protocol

Stage 4: Evaluate the review protocol

Stage:5 Pilot selection and extraction

Stage 6: Primary study selection

Stage 7: Data extraction and study quality assessment

Stage 8: Analysis / Synthesis

Stage 9: Draw conclusions

Stage 10: Documentation

Stop

Start

PLANNING

EXECUTION

ANALYSIS

8

TABLE 3

RESEARCH QUESTIONS ID Research question Rationale RQ1 What agile practic-

es are most cited in literature?

To provide an inventory of practices reported in aca-demia and implemented in practice.

RQ1.1 What practices are most adopted in practice?

To create a list of common agile practices reported in peer-reviewed literature, i.e. a sort of common de-nominator(s) for agile.

RQ1.2 What are the con-text specific aspects of agile practices reported?

To provide a mapping of context variables and the most cited practices.

RQ1.3 What modifications to agile practices have been report-ed?

To identify reported modi-fications to agile practices in their implementation.

RQ2 What is the strength of evi-dence on the use of agile practices in reference to agile philosophy?

To investigate the evidence on the use of agile practices and evaluate the state of “being agile”. To accom-plish this, research papers reporting the use of agile are evaluated.

3.1 Search strategy We derived search terms (keywords) from our research questions and also by examining [39] to identify termi-nology relevant to ‘agile’. Our search string comprises of Population AND Intervention [42] as shown in Table 4. Our search strategy to identify the initial set of studies for study inclusion is shown in Fig. 2. We verified the strength of our search string by checking our search catch against known primary studies. We initially had a set of 25 studies relevant to our study. Based on our search catch we made modifications to our search string. How-ever, we incorporated keywords for Intervention (for e.g. keyword ‘practices’). Without intervention the search resulted in nearly 80000 results, which would have been impractical and impossible to process. Also, most of the search hits were found to be highly irrelevant. Therefore, we added Intervention keywords. Although there is a chance that some relevant articles would be missed this way, this was a necessary tradeoff.

Our refined search string (see Table 4) resulted in 11032 publications. We applied the search string only on Title and Abstract. We used EndnoteX4 as reference man-agement tool for removing duplicates, creating libraries etc.

3.2 Study selection criteria

Our strategy for study selection is inclusion heavy [59].

To start with, we applied our selection criteria on both Title and Abstract. That is, no publication is excluded by reading the title solely.

TABLE 4 SEARCH STRING

Search string = Population AND Intervention Population

((agile OR (software AND agility)) OR "extreme programming" OR xp OR xp2 OR scrum OR (crystal AND software AND (clear OR orange OR red OR blue OR family OR method)) OR dsdm OR "dynamic software development method" OR fdd OR (feature AND driven AND development AND software) OR lean OR "ag-ile modeling" OR ”agile modelling” OR (streamline AND devel-opment AND software) OR (iterative AND development AND software) OR (internetAND speed AND development AND soft-ware) OR (iteration AND development AND software) OR (adap-tive AND development AND software) OR (light AND weight AND development AND software) OR (kanban AND software) OR (incremental AND software AND development))

Intervention (practices OR (ways AND working))

Fig. 2. Search strategy

The following basic criteria are applied on the obtained publications.

• Articles should be in English • Articles should be published between 2000 and

2010 • Articles should be in available in full text.


The following questions are considered to decide if a study is to be included as primary study:

1. Is the study relevant to software development in any form, i.e. embedded system, IT, safety critical etc.?

2. Does the sudy present any material on im-plementation of agile practices?

The following definition of agile is used while going

through articles: Agile refers to methodology to bring fastness in development, welcome changes to require-ments and cope up with constant changes. By ‘agile prac-tices’ we mean any ways of working implemented as part of agile methodology. If either of the questions above re-ceives a No, the study is excluded. We also excluded books, editorial notes, comments etc. Fig. 3 shows the databases we selected and stages involved in study re-trieval.

IEEE Xplore

ACM Digi-tal Library

SpringerLink ISI Web of Science

Inspec Compendex Scopus Wiley Inter-science

612 182 141 562 3456 4247 148 1608

Fig. 3. Primary studies selection

10

3.3 Primary study selection We piloted our study selection process to find if there is a homogenous understanding of selection criteria and also to find if we are not missing articles relevant to the re-view. We extracted 30 articles for the pilot from Inspec library and accomplished study selection (step 5) togeth-er. We did not find major problems with the selection criteria expect that we found articles on agile and lean covering non-software like manufacturing industry. We decided not to include such articles in our primary stud-ies. We decided to store such articles in a separate data-base. The step-by-step procedure for obtaining primary studies from the initial pool of 11032 publications is shown in Fig. 3. We first removed the duplicates (automa-tized in EndnoteX4), removed articles published in non-English languages. We discarded a total of 6041 articles based on title and abstracts and for 159 articles we could not retrieve full texts. A total of 1286 articles were left for full text review. Finally, 186 papers were left after filtering articles that did not match our selection criteria. For the 1286 articles that remained for full-text review, precision rate of 16% is observed. A reason for such a low number is that many articles that had statements on agile and practices did not have any text on implementation of agile practices while some articles described models or frame-works on various concepts (for e.g. maturity, adoption, human factors). Although these are important papers for the agile topic but we excluded them as they did not match our study focus. 3.4 Data extraction & study quality assessment Similar to study selection we did a pilot data extrac-tion by selecting 25 articles from our primary studies ran-domly. We started with an initial discussion meeting to identify properties to extract from the primary studies. We defined a list of properties to extract and data diction-ary (see Table 5). Table 5 shows the rationale behind ex-tracting the properties and mapping to RQs. We first ex-tracted data from ten articles individually. We used Fleiss’ kappa [60] as a statistical measure to evaluate ho-mogeneity between the two of us. We calculated Kappa coefficient for the data properties P1 to P10 and we found a moderate agreement value (0.6). We had a consensus meeting where we improved our understanding on our data dictionary. In the next iteration, we selected 15 more studies (plus the ten from the previous iteration) and pi-loted data extraction. We found a strong agreement be-tween us (0.85), which is considered good according to Landish and Koch [61]. We divided the primary studies according to year of publication. Data from the articles published between 2000 and 2005 are extracted by the second author, and the rest were extracted by the first author. Study quality assessment is not performed as a separate step. It is performed as part of data extraction (step 7). The first three data properties (See Data extraction form, Table 5) are combined to identify research rigor [52] [59]. Study quality is performed on the basis of identifying how the studies are reported. The properties P1 to P7 concern the credibility of studies for the evidence availa-ble, i.e. how the evidence is produced and presented [52].

The aim here is to provide trustworthy and realistic evi-dence supporting technology transfer by examining when and where evidence has been produced [52] [59]. In pilot data extraction (step 7) we encountered a few problems in extracting the property ‘research method’ (P7). We de-fined the following data dictionary for P7:

• Case study: if the study mentions it explicitly, or defines a goal and applies it using a case study (for e.g. using an application) [63]

• Experiment: if the study conducts an experiment and defines its design [63]

• Survey: if the study collects data (qualitative or quantitative) using methods like questionnaire or interviews [64].

• Action research: if this is specifically mentioned in the study.

• Lessons learned: if the study discusses reporting of implementation of practices in industry, i.e. ex-periences [52] [63].

• Pure research paper: if the study discusses con-cepts covering agile practices, implementation and do not contain any empirical part.

• Unclear: if the study does not state any research method and in cases where the research method cannot be classified undet the above categories.

For study context (P5) we classified studies into two types. Industry context studies contain studies performed in industry or in collaboration with industry (for e.g. re-searchers performing in industry). Academia context studies contain studies performed in academia or the ones that cannot be classifies as industry context. For studies performed in industry, we extracted the property P11, company size using European Recommendation 2003/361/EC [63]. In cases where we could not extract company size, we visited the websites of the companies and noted the values. Furthermore, we noted the values for properties project duration (P12) and team size (P9) for studies performed in industry for studies performed in both industry and academia. We noted subjects (P6), and scale of application used (P7). For studies performed using industrial applications, P7 is given value ‘industrial application’ and in rest of the cases it is marked as ‘other’, which include toy examples developed for student pro-jects conducted in academia etc. We have noted applica-tion type (P10) for empirical students. We classified ap-plication type into: ‘embedded’, ‘non-embedded’ (IT for banking, game development etc.), ‘unclear’ in cases where we could not classify them as embedded, ‘not mentioned’. We could not classify them as embedded, ‘not mentioned’ (where no details of product is mentioned). We could not classify them as embedded, ‘not mentioned’ (where no details of product is mentioned). For studies that did not have any empirical part, i.e. ‘pure research papers’, prop-erties P6-P12 are given the value ‘inappropriate’ instead of ‘not mentioned’.

For studies that did not have any empirical part, i.e. ‘pure research papers’, properties P6-P12 are given the value ‘inappropriate’ instead of ‘not mentioned’.


TABLE 5 DATA EXTRACTION

3.5 Procedure for assessing research rigor and

relevance Gap between what is proposed in academia and what is practiced in industry has been observed in various in-

stances, for e.g. [52] [59] [65]. A potential factor for trans-ferring research results into use in industry, or for indus-try in general is how empirical evaluations (of any sort, for e.g. using industrial applications, or toy examples) are

ID Property Data values Description Mapping to RQs

P1 Context Strong Medium Weak

To specify to what extent context of the study is described RQ2

P2 Study design Strong Medium Weak

To specify to what extent design of the study is described RQ2

P3 Validity evalations Strong Medium Weak

To specify to what extent validity of the study is discussed RQ2

P4 Research method From the reviewed articles Unclear

To capture research method used in the study RQ2, RQ3

P5 Study context Industry Academia To capture context where the study took place RQ2

P6 Subjects

Practitioners Researchers Students Inappropriate

To capture the subjects in the study RQ2

P7 Scale of application used

Industrial application Other Inappropriate

To specify the scale of the application used in the study RQ2

P8 Agile experience

Beginner Experienced Not mentioned Inappropriate

To capture agile experience of the subjects used in the study. Marked as ‘Beginner’ when the subjects reported were new to agile, i.e. implementing agile practices for the first time or were not part of an agile team.

RQ1

P9 Team size

From the reviewed articles Not mentioned Inappropriate

To capture the size of the team used in the study. RQ1

P10 Application type

Embedded Non-embedded Unclear Not mentioned Inappropriate

Classified as Embedded or non-embedded. When it is not clear from the study, application domain is classified as ‘Unclear’.

RQ1

P11 Company size

Micro Small Medium Large Not mentioned Inappropriate

Marked according to European Recommendation 2003/361/EC [63].

RQ1

P12 Project duration

From the reviewed articles Not mentioned Inappropriate

To capture project duration of the agile project described in the reported study RQ1

P13 Agile practices From the reviewed articles To capture agile practices reported in the study. RQ1,

RQ2

12

conducted and reported [59]. Furthermore, it is important for researchers and industry professionals (practitioners) to gauge results, their credibility and validity in practice [59]. Ivarsson and Gorschek [59] developed a model to identify and assess rigor and relevance to evaluate evi-dence and find decision support material to try out re-search results. Research rigor refers to how empirical evaluations have been performed and reported [59]. To assess rigor we noted values for the properties P1-P3. Calculation of rigor is given in Table 6.

Relevance refers to the potential for impact in tech-nology transfer [59]. For example research methods like case studies and lessons learned provide stronger evi-dence to practitioners than experiments conducted using student subjects [59]. Studies conducted using laboratory experiments (for e.g. experiments conducted using stu-dents in academia, conceptual analysis (for e.g., pure re-search papers without any empirical section) do not con-tribute to realism [59]. Central to technology transfer is realism involved in evaluations. For examples, studies conducted using industrial applications provide higher realism of state of practice than studies using toy exam-ples invented for evaluations. Similarly, studies using industry professionals as subjects provide a richer picture of realism over studies with student subjects [59]. Rele-vance is calculated using the properties P4-P7. Table 7 shows how relevance is calculated.

TABLE 6

RESEARCH RIGOR Research rigor = C + S + V

P1: Context (C) P2: Study design (S) P3: Validity evaluations (V) Values for each property: Strong = 2, medium = 1, weak = 0

TABLE 7 RELEVANCE Relevance = C + RM + Su + Sc

Context (C) Research method (RM) Subjects (Su) Scale of application used (Sc) Properties contributing to relevance are given the value 1 and 0 otherwise Contributing to relevance

• Studies conducted in industry context (i.e., context = industry)

• Studies using practitioners as subjects (i.e., sub-jects = practitioners)

• Studies using industrial applications (i.e., Scale of application used = industrial application)

• Studies conducted using research methods ‘case study’ OR ‘lessons learned’ OR ‘action re-search’ OR ‘exploratory survey’ OR ‘inter-views’ OR ‘experiment’ (non-laboratory)

3.6 Validity threats There are three potential threats to the validity of our SLR and its results namely publication bias, selection of primary studies and data extraction.

3.6.1 Publication bias Publication bias refers to the problem that negative re-search outcomes are less pronounced or published than positive outcomes [42]. For example, a threat can be pos-sibility of not publishing of negative outcome of agile implementation or excluding such publications in our review. However it should be noted that research ques-tions for our review are not defined from the perspective of performance or success of agile, rather to investigate what agile practices have been implemented. To reduce the threat of publication bias, we included publications that discussed implementation of agile practices. Fur-thermore, we did not restrict our search to a particular set of conference proceedings or journals, instead a wide range of digital libraries are considered to cover the breadth of the field. To gather reliable information, we did not include grey literature, technical reports, non-peer reviewed or unpublished literature. 3.6.2 Selection of primary studies We defined our search string to retrieve as many publica-tions as possible. However, considering the large number of initial results we incorporated ‘intervention’ in our search string. To ensure that searching for relevant publi-cations is unbiased we ran pilot searches, modified search strings. However, a general problem with software engineering is its inconsistency in the use of terminology. Terminolo-gy is not standardized [39]. SLR by Dybå and Dingsøyr [39] observed similar problem in identification of primary studies. If we had we omitted the use of ‘intervention’ we would have obtained a large number of relevant studies. But this trade-off was necessary considering the initial results without the use of ‘intervention’ (nearly 80000 hits). Although we reviewed our search protocol and pi-loted article search, the problem of missing a few primary studies still exists. For example, we could not retrieve full texts for more than 150 articles. To check if we missed any articles, we did manual crosscheck verification in Agle conferences. 3.6.3 Data extraction When we piloted our data extraction process, we found that many of the articles lacked details such as system characteristics used in the empirical study. We refined our data extraction form to extract data that give overall description of mainly: context in which the study was conducted, design of the study and validities addressed. This process involves subjective judgment. To ensure homogeneity, we ran a pilot study, had consensus meet-ings to get a common understanding of the data, refined data extraction based on the checklist defined by Ivarsson and Gorschek [59]. Data extraction process was hindered by the way studies were reported in several articles. For example, many of articles did not address factors that


could affect the outcome of the use of agile. To minimize the misunderstanding of the data, we improved our con-sensus iteratively. This was done as part of pilot data ex-traction process. Our Fliess’ kappa coefficient improved from 0.6 to 0.8, which shows a strong agreement between the reviewers [60].

4 RESULTS AND ANALYSIS A total of 186 primary studies were identified. These are broadly classified into two: studies of empirical nature (empirical studies) and studies that are not of empirical nature (pure research papers). The latter category mainly discussed agile concepts by illustrating agile practices. Before answering our research questions and presenting our analysis, we provide a brief overview of the studies identified.

4.1 Overview of the studies 4.1.1 Publication year A general overview of publication years in our study is given in Fig. 4. A total of 115 (52%) primary studies were published in or after 2005. Publications on agile seem to be widespread equally before and after 2005. This further may imply that there has always been substantial interest in implementing agile philosophy. Spikes can be ob-served for the years 2004 and 2009. 4.1.2 Research method A general overview of research methods used in the primany studies is shown in Fig. 5. Research methods were extracted according to the classification discussed in Section 3.4. A very high majority of studies used “lessons learned” (79, 36%) and “case studies” (52, 24%). Experi-ments have been reported in 24 (11%) studies. We could not extract research method for one study as it lacked a description on the study was conducted and research method used. 4.1.3 Context The study settings were classified into industry and academia (non-industry). Among the studies conducted in industry, 114 were empirical studies and the rest 5 are non-empirical studies. For 114 studies (63%), context was identified as Industry indicating that a high percentage of studies were performed in realistic settings. We extracted company sizes from the reviewed studies using European Recommendation 2003/361/EC [63]. Company size was not mentioned for 39 (34%) studies. In other studies re-porting company size, large can be found in 31 (28%), small and medium size in 31 (28%) studies. Interestingly, 27 (24%) of the studies that reported company size are large companies. This can further mean that agile practic-es are found to be useful in the context of large companies where it is complex to deploy or integrate new processes and techniques with existing ones [31]. Experience also suggests that agile practices match the need of large com-panies [31]. However, not discussing context in the stud-ies hinders in judging the evidence as being relevant to a

particular context [69]. For example, Of these 114 studies 46 (40%) studies did not address team size and 57 (50%) studies did not address duration of the projects. This fur-ther makes evidence based judgments weaker as the in-formation on how large or small are the projects cannot be obtained. Research, however a show that agile is suita-ble for small and medium projects [6] [67]. 48 (42%) stud-ies conducted in industry were identified as mature teams. That is, subjects in these studies were at least one project old in terms or agile or have been using agile for at least one year. Experience of the practitioners in usng agile practices was not reported in 26 (23%) of studies conducted in in-dustry. A high number of studies conducted on agile ma-ture teams (48, 42%) also suggest that using agile is prom-ising and there is substantial interest towards applying it in industry. A high percentage of studies (81%) did not address context fully in industry. Only 18% of the studies in industry context described context fully. An overview of the extent to which context is described in studies re-ported in given in Appendix A. 4.1.4 Study design Of the all the studies reviewed, 23 (12%) did not describe the property study design (P2). In 49 (26%) studies study design has been described well whereas 114 (61%) studies described the property partially (See Appendix A). It can be observed that Study design was well described in studies conducted in industry when compared to those in academia. This further increases the suitability of results presented in industry context as being more relevant for practice. 4.1.5 Validity evaluations

Of all the studies reviewed, 102 (55%) studies did not address the property validity evaluations (P3) at all. Study validity is addressed partially in 67 (36%) and in detail in 20 (11%) studies (See Appendix A). A high num-ber of studies conducted in industry (65, 54%) did not address validity and credibility of the results at all. This hinders drawing conclusions based on evidence available on the use and implementation of agile practices and ap-plying to a different context. A majority of studies con-ducted in Industry are ‘lessons learned’ (or experience reports). To make evidence more convincing and adapta-ble, practitioners should address factors that could have affected the outcome of agile implementation.

4.1.6 Rigor Rigor is obtained by summing up the properties Context (P1), Study design (P2) and Validity evaluations (P3). Ri-gor found in the reviewed studies in shown in Table 9. Of all the studies reviewed, we found a total of 61 (31%) studies that are reported well (rigor greater than or equal to 4). Of all the studies conducted in industry, 40 (35%) studies are well reported whereas in academia a total of 21 (25%) studies are well reported. These studies further can give decision support material on how agile was im-plemented.

14

Fig. 5. Research methods TABLE 8 COMPANY SIZE REPORTED TABLE 9 STUDY RIGOR

Rigor Industry Academia 0 1 3 1 11 9 2 37 18 3 31 20 4 26 9 5 12 4 6 1 4

4.1.7 Relevance

Relevance is obtained by summing up the properties Context (P5), Research method (P4), Subjects (P6) and Scale of application used (P7). An overview of relevance found in the studies is shown in Table 9. Relevance gives an understanding of to what extent studies are conducted in realistic settings. It can be observed that majority of

studies (54%) were performed in realistic studies, thereby increasing the validity of our SLR results for state of prac-tice. We combined Rogor and Relevance found across studies. Of the 198 studies, 42 (19%) studies had High Rigor and Relevance (HRR), as seen in Fig. 6.

Company size Publications – Nr. (%) Micro 10 (8) Small 15 (12)

Medium 12 (10) Large 31 (26) N/A 46 (39)

Inappropriate 5 (4)

Fig. 4. Distribution of publication across years

1"48"

15"3"

68"36"

10"3"1"1"

0" 10" 20" 30" 40" 50" 60" 70" 80"

Action"research"Case"study"Experiment"

Grounded"theory"Lessons"learned"Research"paper"

Survey"Interview"

N/A"SLR"

#"Publications"

3"8" 9"

20"

36"

19"23"

15"19"

29"

5"

2000" 2001" 2002" 2003" 2004" 2005" 2006" 2007" 2008" 2009" 2010"

Publications vs Year!Publications"


TABLE 10 Relevance across studies

Relevance Industry Academia 0 0 40 1 6 23 2 1 3 3 41 1 4 108 -

4.1.8 Practice classifications We classificed the practices found in our primary studies based on the definitions found in literature. These prac-

tices were claasified into thirtytwo different categories based on the most suitable category they fall into. The descriptions of the practices as well as their acrynms used are given in Table 11. For e.g pair programming and pair-ing are the same practice actice but have been modified by the situation. In this scenario we placed them under the same cateogery of pair programming, it is the same case with sustainable pace and 40-hour week. Practice clasisfications are based on literature by original propo-nents of Agile and the work by Petersen and Wohlin [9]. These practice classifications were used in [9] to analyze how practices relate to Agile principles. For example, Sprints and Iterations can be grouped into a single prac-tice as they share the same philosophy of developing software in a series of mini-projects [9].

Fig. 6. HRR

TABLE 11 Practice Classifications

Nr. ID Practice Definition

1 PP Pair Programming Two people write the code at one computer [56].

2 Te Continuous Testing Testing refers to any of the practices used for enabling testing thorughout the. Unit tests, Acceptance tests etc. are run in every iteration [243].

3 Re Refactoring Restructures the system by removing duplicates as well as improve communication [56].

16

4 PG Planning Game/Sprint planning

Close interaction between customer and programmer: the programmer estimates the effort needed to implement a customer stories, and the customer then decides about the scope and timing of release [9].

5 CC Customer Collaboration

A close cooperation between the customer and the developers exists for rapid feedback and clarify the requirements. This avoids implementation of features not avoided by the customers [1]. Customer collaboration in this context means, collaboration exists irrespective of the customer being on-site or off-site.

5 CO Collective ownership Anyone can change any part of the code anytime [9].

6 CI Continuous Integration A new piece of code is integrated into the code base regularly, thus the system is integrated and built many times a day. All tests are run and they have to be passed for the changes in the code to be accepted [9].

7 CS Coding standards Coding rules exists and they are followed by programmers. Communication through code should be emphasized [9].

8 Su Stand-ups Stand-ups are short meetings, approximately 15 minutes daily, to keep track of project progress [56].

9 Me Metaphors/Stories/Features

Stories are the short statements of the functionality desired by a system user or customer. Stories are customer visible, customer valued functionality that can be completed in a single iteration [9].

10 SR Short Releases The team delivers code to customers in the form of supported releases more often; the purpose of having short releases is to get the product out to customers as quickly as possible [9].

11 Rt Retrospectives/Sprint review

At the end of each iteration, team gather for a retrospective meetings. In this meeting data is presented on how the iteration went, such as how much working software was produced. The team evaluates the good that they can carry on for the next iteration as well as the loop holes that they want to discard for the next iteration. Specific action items are taken out of retrospectives and made as stories for the next iteration. This helps the team to allocate necessary resources towards process improvement [56].

12 SD Simple Design The emphasis is on designing the simplest possible solution that is implementable at the moment. Unnecessary complexity and extra code are removed immediately [9].

13 40h 40-hour week A maximum of 40 hour working week. No two overtime weeks in a row are allowed [9].

14 It Iterations/ Sprints

Sprints are used in the context of Scrum method and Scrum teams. each scrum is is a self contained .Every iteration is a self contained mini project composed of activities such as requirements, design programming and test. Each iteration is time boxed (stipulated time). If the team is aware that they can’t meet the objectives, rather than moving out of the time box the scope of the iteration is reduced. At the end of a iteration the team obtains feedback on the code delivered from the manager [56].

15 TDD Test-driven

Development Software development is a test driven. Unit tests are implemented before the code and are run contimously [56].

16 COM Communication Different channels for communication that would support communication among different.

17 Of Office Practices with central focus on Office structure or environment to sup-port adoption of use of agile software development ate classified as Office.

18 Tm Team Agile teams that utilize the team practice whih consider the cross functional team of all those necessary for the product to succeed as one team [9].


19 Pr Progress Progress is tracked my milestones based on software deliverables and major decisions rather than written documents [9].

20 CR Coach Role This practice refers to the use of an agile coach in the projects. Coach plays a major role in driving the other members in the team [56].

21 Do Documentation Practices are grouped into Documentation if their goal was to reduce documentation or documentation in agile projects.

4.2 What agile practices are most cited in literature? (RQ1)

The purpose of this research question was to identify and provide an inventory of agile practices that were consid-ered agile. A major aim of this question is to give a view-point of importance given to each of the agile practice. An overall view of practices cited in our study is given in Table 12. As we can see from Table 12, the most cited ag-ile practices are PP, Re, CC, CC, CS, PG, SD, CO, and practices related to Te. Apart from practices shown in Table 11, we identified other practices that we grouped into ‘Other practices’. The motivation for showing them as separate practices are, they are mainly single cited and were used to support other practices. These are given in Appendix B. A general observation looking at the Table is that practices like PP, Re etc. seem to be more popular. PP received the most attention (citation count: 97) among agile practices found in our study. Looking at the citation count, one can find that PP is the most popular agile prac-tice. Practices like CS, CO, SR, and Rt have almost the same citation count. We identified several other practices, which we mapped to the practices shown in Table 11. For example, Task Kanban [149] and A/B/C List [112] are put in the category ‘Features’. We identified practice modifica-tions/adaptations. These are discussed in section 4.4 (RQ1.3).

TABLE 12 Citation count of practices

Practice

Industry (Studies =

119

Academia (Studies = 69)

Nr. Nr. It 93 12 Su 39 6 SR 34 15 CO 33 20 CS 37 17 Re 43 24 CI 50 19 CC 50 16 PP 59 38

TDD 27 21 Rt 38 8 PG 48 14

SD 26 16 Pr 15 6 Me 50 26 Te 49 25 CR 6 2

COM 7 2 Of 14 0 DO 6 4 CM 2 2 40h 23 14 Tm 14 6

4.3 What agile practices are most adopted in

practice? (RQ1.1) The purpose of this RQ was to distinguish studies be-tween industry and academia and show what practices are widely adopted. Further aim of this RQ was to identi-fy coupling between agile practices in practice. To this end, we have identified what practices are used together in combination. Contrast between the use of practices in industry and academia is shown in Table 13. However, it should not be noted that all the empirical studies (both in industry and academia) are not included for analysis in Table 13. The main reason behind this that some studies discussed only implementation of a specific practice like PP, or discussed the use of non-software activities in software projects. For example, PP is cited in 65% of the studies reported in industry and 58% of the studies in academia. Results from these studies are discussed in Section 4.4.3.

TABLE 13 Practices found in empirical studies

Practice Industry (Studies

= 93) Academia (Stud-

ies = 15)

Nr. % Nr. % It 88 95 6 40

Su 33 35 5 33 SR 30 32 4 27 CO 32 34 6 40 CS 38 41 4 27 Re 44 47 9 60 CI 42 45 6 40 CC 45 48 5 33

18

PP 60 65 13 87 TDD 29 31 8 53

Rt 30 32 2 13 PG 50 54 6 40 SD 28 30 4 27 Pr 13 14 2 13

Me 58 62 4 27 Te 44 47 7 47 CR 6 6 2 14

COM 6 6 2 14 Of 11 12 0 0 DO 1 1 2 14 CM 2 2 2 14 40h 23 25 2 14 Tm 10 11 0 0

Contrasting differences between academia and indus-

try can be observed for most of the practices. PP is a dom-inant practice both in industry and academia but has a slightly higher percentage in academia. PP has been most tried out by practitioners and appears to be popular. A success factor for using PP can be credited to its ease of use, improved code quality and knowledge transfer [134]. PP was found to be a good way to review code [261]. A common reason for its industrial prominence is it is wide-ly studied and thus tending to be more popular [12]. It is suggested to build code patterns in pairs and use these patterns to build sub-modules solo [151]. PP is found to increase the level of satisfaction and confidence [83] but it is preferred that team learns the essence of PP and takes time for the results to be visible. Other advantages of PP include increase in code quality and reduction in code inspection time [96] [111] and reduction of cycle time by 40 to 50 percent [233]. Pairing is sometimes done only for complex code when team size is small and is practically difficult to accommodate PP as a regular task [106] [132] [133]. The use of PP was found to be useful for develop-ing user stories [160] [200]. Overall, PP is widely studied [12] and adopted. It was found to be a fun practice to adopt and use and can raise the level of innovation and knowledge transfer between the developers [115] [126].

TDD appears to be less popular among practitioners. TDD is found to aid team in building high quality prod-ucts. Although there can be reduction of productivity in terms of time, it is found to be compensated by improved quality and decrease in code complexity [131]. A de-motivation for TDD adoption can be that more time needs to be invested for development [165]. However, from the product-lifecycle perspective reduction in maintenance costs and customer satisfaction were observed [146] and has potential for increasing the level of testing [165]. In one study developers felt writing test cases before actual development felt unnatural and laborious although it was useful for critical tasks [286]. It is thus evident that prac-tices that were embraced or used most in academia are PP

and TDD. Of related practices like ‘small offices’ [98] and honey-

comb cubicles’ [204] were more or less invented by practi-tioners owing to practicalities. St pasted on wall accord-ing to their prioritization in an Open workspace can im-prove communication [248]. A possible reason for not adopting Open office or not considering it agile is because of investment that needs to be put in infrastructure change [158]. Moreover, it can be difficult for the devel-opers to focus working in an open environment [261]. Agile is a social process which developers seldom count [266] which may limit disciplined use of Of and Tm prac-tices. The use of shared wikis and communication chan-nels like instant messengers’ are used to reduce distance between sites [264] [255].

Scrum practices like time boxing, i.e. Sp, PB are better utilized by practitioners. On the flipside, challenges in practicing PB and planning Sprints include lack of overall and long term focus [261]. Su were considered as one of the important communication channels [168] and also used to set up the tone of the day and remove any back-logging problems [254]. For example, a company Su meet-ing may involve about 20 people often discussing status updates [254] or involve sub-teams [168]. Su were com-plimented by Pr practices like Burn-down charts [247] [122] [172].

Industrial adoption is a more inclined towards CI than other practices like CO. CI enables the team to identify issues where a change to one component impacted anoth-er [250] and also enables continuous testing as there will be always something to test [261]. However, adoption of the practice is less pronounced within large teams (team size greater than 30). Industrial adoption shows it is hard-er to practice in projects with large teams [188]. For suc-cessful use of Continuous integration, it is recommended to combine with Te [249] [110] enabling implementation of late changes [187]. It provides the team with valuable feedback because the code is in continuous and integrated condition [188] and protects against loss of configuration control [233]. Although TDD seems to be less pro-nounced, industry seems to maintain the essence of con-tinuous testing throughout the project (See Te, Table 12). Unit tests and acceptance tests are carried in every itera-tion.

On the other side, adoption of CO is associated with developers’ mindset. For example, in one study develop-ers were reluctant to adapt code written by others and continued to speak of “their code” [260]. Management and developers’ interest affect adoption of CS. For exam-ple, in one study we found developers and management were reluctant to enforce a common standard for coding. This explains why CO is little less common in terms of adoption. CS was not put into place until the code became so complex that it was unmanageable and required tight-er control [260]. Peer pressure can also impact adoption of CS [213]. Observed benefits of coding standards were readability of the code, communication and strong sup-port when using PP [187] or CS [222]. CS is understood as a simple practice to adopt [187] and can produce con-sistency in coding and code reviews [156]. Company


standard coding guidelines can be used to extend the practice [102].

XP’s design related practice, SD is considered as a dif-ficult practice to adopt and yet very useful [101]. Agile methods have been criticized for their lack of emphasis on architecture and design [85]. Architecture related practic-es like Architecture analysis, Architecture evaluation etc. were proposed by Babar [85]. Creating simplest possible design is a difficult task when the team has to plan when new code would interact with coming functionality [107]. Design planning can be performed based on available knowledge from similar projects and not just based on user stories in iteration and not in order to reduce the amount of Re [248]. For successful use of SD developers need to be equipped with knowledge of how to design effectively [116]. Non-functional attributes like security and safety are suggested to be designed through architec-tural practices while the ‘keep it simple’ rule of SD is ad-vocated to reserve to individual sub-systems [258].

Re was found to have strong impact in terms of en-hancing reusability, eliminating complexity etc. [123]. The popularity of Re can be credited to design improvement [164] [168]. Automating Re using tools like Eclipse was found very successful [168] [116].

Me is perceived as not-so easy practice to adopt [248]. Me is found to strongly interact with practices PG and Re in steering the project and creating a simple and stable structure (as a guidance scheme for naming classes, methods etc.) [123]. Me provides a high level of abstrac-tion level and are ideally suitable for systems governed by a high degree of complexity [170] and are used to communicate with the customer about the system [105]. St are brief informal descriptions of user requirements, often written by customer [141]. It is not very clear from the studies reviewed if Me are developed by systems’ customer or the team. For example, when the software is intended to a mass market, Me are to be developed and analyzed by the team [133]. Story size should be small enough such that it is possible to construct individual tasks for development and thus the team is committed to code within the deadline of iterations [266].

The citation counts for practices in the categories Of and Tm, as seen in Table 12 are less compared to other practices. Practices like Shared working room [158] were found to be less pronounced. It is suggested to have a small offices (150 to 200 people) [98] and a dedicated meeting room [247]. The use of ‘Share project space’ in-volves transformation of office infra structure of cubicles into an open room [158]. Role changing [172] and merg-ing of teams for integration [132] were among the cited practices used for team management. Re provide regular team reflections and adjustments to become more effec-tive [250]. Feedback sessions are used to demonstrate user stories for customer approval [266]. Re (or reflection) ses-sions are used as venues for iteration/sprint demo. As it can be observed from Table 12 using iterations or Scrum’s sprints are common. Su are used as a means to progress check mostly on a daily basis. A discussion on how fre-quently Su meetings were conducted is given in Section 4.4 (RQ 1.3). Coupling between practices, i.e. what prac-

tices are used in combination in industry is presented here. By coupling, we mean what practices practitioners consider as agile in combination. To identify this we de-fined three categories for coupling: (1) Strongly agile: cou-pling is categorized as ‘Strongly agile’ between practice X and practice Y if Y is cited in more than 70% papers that cited practice X, (2) Agile: coupling is categorized as ‘Ag-ile’ between practice X and practice Y if Y is cited in be-tween 50 and 70% papers that cited practice X, and (3) ‘Not so agile’: coupling is categorized as ‘Not so agile’ be-tween practice X and practice Y if Y is cited in less than 50% papers that cited practice X.

Tendency to combine Scrum practices and XP practices is prevalent, and evident. Practices like PP, CS, Re are cited in half of the studies (40%, 36%, 32% respectively) that cited Scrum based sprints. Scrum is majorly advocat-ed as a project management methodology and not code construction methodology per se [21].

Practices like CO, CS are cited more frequently than SR. Practices considered ‘Strongly agile’ with Short re-leases are: PP (74%), Re (70%) and considered ‘Agile’ (based on relative citation count) are PG (68%), Me (65%), CI (65%), CO (61%), and CS (61%).

When using Me, CC seems to be a determinant. This is evident as role of customer was reported in 81% of the studies that used Metaphors. Other practices that were found as ‘Strongly agile’ with Me include Re (85%), CO (78%), PP (89%) etc.

The tendency to use the practice Re is found to be highest when using Su. Around 60% of the studies that cited Rt cited Su. Use of St or Sp (41% each) are second highest cited when using Re. PG, CS, PP and Re are found to be ‘Strongly agile’ (94%, 91%, 88% and 79% respective-ly) with CO. Among the studies that cited CS, 77% cited PP, 72% cited Re, 77% cited CO. A possible explanation is that CS benefits to have consistency in coding [156]. PP is cited in 72% of studies using Re. But it is not clear from the studies if pairing can enhance Re.

Among studies that reported TDD, over 75% cited us-ing PP, 60% cited using PG. It is observed that TDD is used majorly in combination with PP and not other way round. Although there is an increase in development time when using TDD, there may be a considerable increase in code quality and passing more functional test-case [68]. To cut cost in development due to the use of TDD, train-ing is needed [106]. Adopting TDD without training may give mixed feelings and contradictory opinions on the efficiency of the practice thus de-motivating to use further [115] and the role of coach was suggested [137].

On a positive note, practitioners seem to embrace Rt meetings as a way to find improvement potential [254] [174]. Practice that was cited most in studies reporting Rt is Su (59%). Rt can be practiced independent of other practices [105]. Sustainable pace or working 40h, i.e 40h is an essential practice for work satisfaction and overall productivity [91]. On the flipside, working only 40h may need extra social and technical effort and companies often need to work overtime owing to schedule deadlines [123]. The emphasis on working 40 hours per week is a social concern practice [133] and teams that work overtime often

20

make mistakes [250].

4.3.1 Research studies We identified 30 research papers reported by research-

ers that discussed various topics in the context of agile software development. We classified into categories based on the topics that were central to them. These are discussed below.

Overall software development We identified five studies focusing agile for software

development in general; See Table 14.

TABLE 14 Research studies on software development in general

Article Description [103] The paper proposes how agile practices can

be used for military embedded systems. [110] The rsearch paper assesses the applicability

of XP practices to engineer high-integrity systems.

[126] The research paper presents a mapping of agile principles to best practices laid by Complex Adaptive System (CAS) theory to support response to change in developing complex software.

[143] The research paper analyzes existing Agile methodologies and maps practices to soft-ware project requirements.

[251] The paper argues to add refelective mode of thinking to each agile practice when applying to both embedded and non-embedded soft-ware development projects.

Garg [103] states the use of CM and automated testing

to support Re of embedded systems as a small change in code can affect timing behavior. Configuration control is considerer as a key ingredient in verifying software against hardware changes. She also recommends auto-mated documentation with the help of integrated tools. Paige et al. [110] suggests modifications to XP to suit the development of High-Integrity Systems (HIS). They state CO is inapplicable to HIS as it requires substantial do-main knowledge like sensors, pressure pumps and there-fore everybody involved in the project cannot have the ability to modify the code. They also state that the prac-tices PG, PP, Me, Re, 40h and CC require modifications. They suggest practices like Static analysis, explicit risk mitigation strategies and techniques, and integration of system design models (for example, Simulink/MATLAB) with XP practices. Meso and Jain [126] present a mapping between CAS theory to agile prinicples in developing complex software. They state best practices based on principles given in CAS theory. For example, to support frequent releases and CI they suggest starting with a min-imalist development team and scale up/configure team’s structute to suit project gwoing size and complexity.

Software development process model We categorized three papers into this category as they

proposed process model for agile software development. Yinghua [257] proposed UniX process adapting concepts and practices from Unified Process (UP) and XP. UniX process drops the XP practice SD and recommends work-ing out system architecture as early as possible and sub-system modules suing UML. UniX also considers the use of PP as time consuming and instead suggests using for critical code and complex parts of the systems. UniX states ‘anyone can change any code’ practice (CO) is not always practical and recommends assigning authorities with respect to competencies in particular technologies to improve code. Visconti et al. [181] propose an ideal pro-cess model for agile methods and identify practices that XP and Scrum fail to provide. They state that XP lacks practices to provide ‘Proper environment and support’ and Scrum lacks practices to allow ‘Business people and developers to work together’ and ‘Continuous attention to technical excellence and good design’. Nawrocki et al. [248] proposed a modified XP method and practices com-bining XP and ISO 9000 to provide guidance in applying XP in ISO 9000:2001 certified firms. Their method defines key process areas with maturity levels and practices asso-ciated at each level. These levels are: Customer relation-ship management (Level 1), Product quality assurance (Level 2), PP (Level 3) and Project performance (Level 4). Kirk and Tempero [153] present KiTe framework as a ba-sis for identifying risk conditions inherent in XP projects. A risk condition inherent in the practice CI, for example, is the inability of the developers to submit the developed stories on time to the build, which may result in slow progress and increased defect rate. Paulk in his report [233] summarizes how XP practices can help organiza-tions realize SW-CMM goals and identifies its key process areas that are not addresses by XP. These process areas Software subcontract management (Level 2), Training program and Integrated software management (Level 3), Quantitative process management and Software quality management (Level 4), and Technology change manage-ment and Process change management (Level 5). In simi-lar lines, Nawrocki et al. [232] propose a maturity model for XP. The model has four maturity levels and describes practices that can be used at each level.

Siebra et al. [140] investigates how XP can be used in the management of innovation process. Moreover, they define the cycle of innovation consisting of five different iterations: investigation, identification, aggregation, con-solidation and post-validation. They suggest allotting 10 days to the first four iterations and 10 days to the last it-eration. Levine and Walker [152] examine the XP practic-es and investigate how they apply to the task of assigning grades to students’ work in computer science courses. Hazzan and Dubinsky [275] present two chasms: abstrac-tion and satisfaction inherent in software development and desctibe how XP practices can bridge these chasms. Abstraction chasm deals with the cognitive abilities of the programmer to think at various levels of abstraction while the Satisfaction chasm deals with satisfying the cur-


rent needs and teammates. Bergin and Grossman [160] present an excercise to teach XP and its practices to teach novices (students, managers, customers, programmers and non-programmers). Duersen [153] studies the possi-bilities of program comprehensions, i.e. the task of build-ing mental models of software systems at various abstrac-tion levels in the context of five practices namely PP, Te, Re, Evolutionary design and Collaborative planning. He identifies comprehension risks and benefits and suggests a need for Reverse engineering perspectives on agile prac-tices.

Customer practices We identified two studies on customer practices. Wang

et al. [135] proposed a set of practices in XP projects that emphasize collaboration between developers and cus-tomers. Similarly, Martin et al. [150] present customer specific practices that enable agility in software projects. These practices are given in Table 15.

TABLE 15 Customer practices proposed by research-

ers

Article Practices [135] Creating customer team, Building up devel-

opment environment, Discussing and re-viewing project plan, Feedback meetings, Serving as consultant, Taking part in usage testing and trial operation, Tracing devel-opment process, and Checking and accept-ing final software product.

[150] Customers’ apprentice, Programmer holi-day, Story standards, Show & Tell, Custom-er pairing, Look before you leap, and Three month counselor.

Quality assurance Huo et al. [193] analyze agile practices’ quality assur-

ance abilities and state that in agile methods, quality as-surance activities are inside the development practices like PP, TDD, passing 100% uni tests, architectural spikes and CI and occur at higher frequency and earlier than in the waterfall process. Hayes et al. [256] discuss opportu-nities, challenges and plans for the synthesis of TDD and traceability. They state that TDD is a well-suited practice to achieve traceability and recommend research on the data that can be collected and analyzed to aid develpers in the TDD practice and propose a methodology to collect and maintain traceability information in the TDD prac-tice.

Succi et al. [235] propose an experimental framework to quantify benefits and costs of PP and compare design aspects of the resulting software artefacts and their defect behavior. They use a set of six Object-oriented design metrics from the CK metrics suite to describe design deci-sions. Erdogmus and Williams [269] perform an economic evaluation of PP. They used quantitative data based on

experiements conducted in a different setting 11at Uni-versity of Utah. They found that PP is an economically viable option when compared to solo programming.

Architectural practices Juric [228] presents an overview of XP practices and

raises concerns regarding their role in conceptual model-ing and code generation. Jensen et al. [155] introduce a new practice to XP called Developer stories. Developer stories are written by developers and provide opportuni-ty to consider the architecture of the application. “Devel-oper stories describe changes to units of developer-visible properties of the software” [155]. They state that Devel-oper stories act as a tool to plan and express demands for Re and facilitate opportunity to reflect upon the architec-ture and design of the system.

Software security Ge et al. [162] propose two new security practices,

Fundemental security architecture and Security training, for applying XP. They recommend providing security training for both developers and customers and use this knowledge in writing security stories and perform securi-ty testing. They suggest defining Fundemental security architecture before the commencement of It. Wäyrynen et al. [179] report an analysis of XP practices from the per-spective of the Systems Security Engineering Capability Maturity Model (SSE-CMM) and Common Criteria (CC) standards. They identify that XP requires modifications as it lacks security-engineering support in many areas con-cerning specifying security needs and assurance. They recommend the role of a security engineer who works on identifying risks along with the on-site customer. Moreo-ver, they advocate the participation of security engineer in PP activities and document a security review.

Distributed development Dou et al. [88] provide a review of tools supporting

distributed PP and propose a framework for it. The framework aims to support the use of same editor be-tween pairs and also different environments and editors between pairs. The tools they reviewed include Sangam, RIPPLE, COLLECE and COPPER. Batra [91] in his report states that agile practices need to be modified to suit the needs of distributed development. He also states that the practices 40h and CC are particularly not feasible. To this end, Hossain et al. [94] provide a review on the use of Scrum in the context of distributed development. They identified a need for empirically founded knowledge on the use of Scrum and agile practices to mitigate the chal-lenges of distributed development. They identified sever-al practices like Synchronous work hours, Local scrum team, multiple and informal communication modes or channels Distributed scrum of scrum, integrated scrum team etc. to mitigate distributed development challenges. We also identified these practices in our review thus strengthening the case of distributed development. To

22

this end they also proposed a framework for risks identi-fication and mitigation practices for distributed develop-ment using Scrum [101]. A process-support environment called MILOS to support teams in a distributed setting is described in [241] [244]. MILOS supports PG and defining user stories activities, developing and maintaining sched-ule of releases, It and also facilitate distributed PP ses-sions with Microsoft NetMeeting.

In addition to the studies described above, we identi-fied four research papers reported by practiotioners based on their experienced. We did not classify them as ‘lessons learned’ reports as they clearly lacked a research method and also there was no description of projects and data based on which the study has been reported. However, major focus of these studies was on Software develop-ment process model or a partucluar practice like PP or improving quality through CI. These studies are de-scribed as follows.

Vanderburg [105] discusses how various XP practices are tangled with eachother. He states that the practices Re, CS, SD and 40h are ‘noise filters’ improving the over-all quality and allow other practices to be effective. Miller [247] proposes an approach to resolve conflicts between pairs when there are discrepencies about design deci-sions. The approach is based on giving scores on a scale of 1-3. The highest score refers to what the next step would consist of. He also defines rules for what the pairs should do when the scores are tied at 1, 2 and 3. Rogers [177] re-ports on the importance of ‘Acceptance Testing’ per-formed by customers and gives advice on how to perform them. He also gives a variety of startegies for practicing CI [189].

4.4 What are the context specific aspects of agile

practices reported? (RQ1.2) The purpose of this research question was to identify

contexts and map to practices that were used. In regard to context, we mapped practices used by mature teams and teams using agile for the first time. Second type of context identified for analysis is the type of product. To this end,

we classified products into ‘embedded’ and ‘non-embedded’ software. For studies where the nature of the product was not clear enough, we put them in the catego-ry ‘unclear’.

4.4.1Team size

An overview of number of studies performed in indus-trial context using mature teams and teams that are new to agile is shown in Fig. 7. Among studies reporting teams that are new to agile, 12 (32%) did not mention team size. For the rest of the 25 teams, size of majority of teams is small. For 20 (80%) studies, team size is less than 20. There seems to be a clear indication of team size being small. An important question thus raises is ‘How small are agile teams?. Of the 25 studies reporting team size, 19 (76%) studies were found to have teams of less than 15 members. Overall 49 studies were found with teams that are experienced with agile.

We could extract team sizes for 26 (49%) studies. Of these 26 studies, in 19 (73%) studies team size was found to be less than 20. This can imply that experienced teams also are small in size. In 17 (89%) of the studies whose sizes are less than 20, team size was found to be less than 15. It can be thus understood that agile teams prefer team size of less than 15. Working in small teams enable work-ing closely, share knowledge and monitor the progress of the project [72]. This was also observed by [39].

An overview of practices adopted by small teams is given in Table 16. Apart from It, practices PP, PG and Su were found to be highly cited. However, it can be ob-served that there is a slight decrease in the adoption lev-els for PP and Su by Experienced practitioners when compared to practitioners new to Agile. PP can be subject to difficulties owing to problems relating to staff, attitude and is a team oriented practice requiring willingness to work together [123]. Limited resources can be a factor for dropping PP and divide individual tasks although it has benefits to offer [123]. Moreover, PP can be cost inefficient for very small teams [122]. On the other hand, Su can be considered as overkill [73]. Moreoever, as size of these

Fig. 7. Team size and practitioners’ experience


teams are between 1 and 5, or 6 and 10, it may be normal to have close coordination and discuss progress and prob-lems informally. A generic explaination to low adoption for practices such as CS and CO among small teams is that practiotioners might have been using these practices but did know the right name [122]. For examples, compa-nies can have their own established coding guidelines [102] or develepers follow generally prescribed coding

principles (for example, Object Oriented coding guide-lines). Lack of experience can hinder adopting CS, CI and also PG. Limiting factors for adopting CI can be the need for infrastructure to automate tests and add to builds. Lack of automation can also hinder CI as they are tightly coupled [123]. Sfetsos et al. [123] observed that type of projects can be a limiting factor for both small and large

TABLE 16 Small and large teams

Practice Team size = SMALL Team size = LARGE

Total = 36 Beginner = 16

Experienced = 13

Total = 13 Beginner = 5 Experienced = 6

It 75 75 75 85 100 100 Su 44 44 44 38 40 33 SR 33 31 31 31 20 50 CO 22 19 19 31 40 33 CS 25 13 13 31 40 33 Re 25 19 19 69 80 67 CI 31 25 25 31 20 50 CC 36 44 44 46 40 67 PP 56 50 50 69 80 50

TDD 14 25 25 8 0 0 Rt 31 38 38 15 20 17 PG 53 44 44 54 40 50 SD 8 13 13 31 40 33 Pr 11 31 31 0 0 0 Me 19 6 6 69 60 67 Te 47 31 31 69 60 83 CR 3 0 0 8 0 17

COM 0 0 0 0 0 0 Of 14 25 25 15 20 17 DO 2 0 0 0 0 0 Pl 0 0 0 0 0 0

CM 2 0 0 0 0 0 40h 16 0 0 23 20 33 Tm 14 19 19 8 20 0

teams. A possible explaination is the relation between project and the kins of automation infrastructure needed. For example, web based products may need skill and ex-perience in tools like Selinium while other types need competence in tools like Maven, Ant etc. Moreover, it can be a time consuming activities for small teams facing with limited resources [107]. Expertise in using tools for con-tinuous tools can be an enabling factor for CI, TDD and also to enable continuous testing, i.e. Te, as can be ob-served for experienced teams in Table 13. Likewise, to nable TDD, unit tests can be added to the build and run

multiple times a day and measure the progress of the tests.

An interesting point mentioned by Stetsos et al. [123] is the lack of guidelines for Re in the XP theory. This can be reflected in our data (See Table 16). Refactoring tools and expertise in Re for duplicate code removal, code reuse and application of design patterns or heuristics are thus necessary [123].

In regard to large teams a higer percentage of adoption for many of the practices can be observed. However, it should not that the sample size for large teams in our SLR

24

is relatively much smaller. Nevertheless, we believe our data can be used for further work. As can be seen in Table 13, practices PP, Re, Me and Te are highly adopted. Inter-estingly, we did not find any citations for TDD. Mastering writing tests early in the project can be tricky as it is diffi-cult to determine the number of tests [123]. This process can further be complex owing to system complexity and also can be a time consuming activity [165].

PP maybe adopted to increase team competence [123], as an informal way of training and to increase collective knowledge. Once this is accomplished, developers may have dropped its usage to work individually. This can be a probable reason for the decrease in the citation percent-age for experienced teams when compared to beginner teams. There is an increase in the adoption levels for prac-tices SR and Me relatively for Experienced teams. Master-ing the practice of releasing software frequently and in small cycles require team capabilities and be competent in estimating and using user stories [123]. Large systems can face integration problems owing to system complexity, large amount of code. Furthermore, integration problems are exacerbated if integrations are delayed [96]. To this end, CI was found to be efficient in revealing integration problems. Early integration builds can reduce the risk of software release by enabling continuous feedback from stakeholders and validate requirements [96]. In regard to CC, limiting factors include lack of a specific customer [106], for example in a market-driven context and organi-zational & resourcing constraints [95]. Also, lack of cus-tomers’ knowledge on importance of agile practices [123] and particularly his/her involvement right from the start of the project can affect the use of CC. Customer in-volvement is considered an important ingredient for suc-cess, however customer can find close coordination of developers sharing same room and specification pair programmers noisy [136]. Also, in large projects involving large teams, application domain can be beyond the expe-rience of the customers [123]. To this end, use of customer proxy [125] or surrogate customer can be used [111] [193]. On the positive side, customer involvement can bring rapid feedback, better use of valuable resources leading to a better product [123].

Design practice SD seems to be more pronounced with large teams based on the data. Lack of theoretical guid-ance can be a major setback in SD adoption [123]. It can be particularly difficult in large projects where the team has to plan for how the new code would interact with coming functionality [101]. SD can be considered very abstract and thus many teams do big upfront design to develop full design [72] [82]. However, having continuous unit testing can harness SD because of coordination between requirements development and code production [123]. The practice 40h is more pronounced among experienced practitioners both for small and large teams. An enabling factor for 40h is strict implementation of working tasks such as iterations and effective management [123]. How-ever, large teams may face stress situations owing to deadlines and small companies may not adhere to the standard 8-hour work schedule on a daily basis [123]. Moreover, management may have negative perception on

the cost effectiveness of 40h [151] and sometime because of distributed development teams have to work on week-ends due to time zone differences [106].

To sum, an overview of the top ranked practices (based on citations) for small and large are given in Tables 17 and 18. However, we believe the data and ranking for large cannot be over emphasized because of the small sample size. If there were more studies reported (in the order of 30s or 40s), results might have been different.

TABLE 17 Top ranked practices for small teams

Total Beginner Experienced Rank = 1 It It It Rank = 2 PP PP PP Rank = 3 PG PG, Su, CC PG, Su, CC Rank = 4 Te Rt Rt Rank = 5 Su Te, SR Te, SR

TABLE 18 Top ranked practices for large teams

Total Beginner Experienced Rank = 1 It It It

Rank = 2 Re, PP, Me, Te

Re, PP Te

Rank = 3 PG Me, Te CC, Me, Re

Rank = 4 CC PG, CC, Su, Co, CS, SD

SR, CI, PG, PP

Rank = 5 Su SR, CI, 40h, Rt, Of, Tm

40h, Su, CO, CS, SD

4.4.2 Application type

In regard to application type (P10), we extracted two categories: Embedded software and Non-embedded software. Among studies performed in industry, we iden-tified 19 and 43 studies where application types were em-bedded and non-embedded software respectively. Among studies performed in academia, non-embedded software was reported in 13 studies and we did not iden-tify any study on embedded software. An overview of adoption of agile practices is given in Table 16.

TABLE 19 Application type

Practice

Industry Academia

Embedded (19)

Non-embedded

(43)

Non-embedded

(13) % % %

It 84 96 38 Su 42 93 30 SR 42 31 30 CO 53 27 38 CS 53 33 23


Re 47 42 54 CI 53 47 46 CC 42 40 38 PP 58 58 92

TDD 26 24 61 Rt 32 38 15 PG 32 40 46 SD 32 24 23 Me 58 49 30 Te 47 47 38

PP is found to be one of the most commonly adopted

practice both in academia and industry and also for both embedded and non-embedded. However, both PP and TDD obtained more emphasis in academia. Auvinen [127] in their pilot project recommend that PP should not be enforced rather should be recommended to the team. A recommendation was that PP gives room to the pro-grammers to use their rhythm and experience and in turn was found to increase the competence level and knowledge sharing of the whole team. PP was also used in the context of distributed development for products in the telecom sector, where remote pairing was conducted using teleconferencing and common IDE [281]. On the other hand PP may requires patience and can be strenu-ous [127]. Also, pairing can be perceived as a tedious process and can be limited to detailed design and com-plex code [213].

Re appears to be one of the highly adopted practices for the embedded software. Re helps in improving the quality of code by reducing the kludginess, making the design simple, thus making the code more maintainable and facilitating room for improvement in the low level implementation [213]. On the flipside, Re can be a very different practice to learn for practitioners that are new to agile [116]. It was reported that there is no formal mecha-nism to evaluate the quality of the refactoring’s [132] and also no automation support to verify refactoring intro-duced bugs and therefore can be tedious and error prone [222]. This can be a problem for both embedded and non-embedded software, for example in a mission-critical sys-tem and also for software supporting banking activities. It was also found that Re strongly supports Object oriented code [85] and thus object orientation can be a factor in adopting Re. Successful implementation of Re needs a strong support for automated regression testing to man-age large amount of code [107].

Salo and Abrahmsson [282] conducted a survey in the context of European embedded software. Their results show that 40h and ‘Open office’ are the most widely adopted practices. However, they stated that they might have been adopted and used independent of agile philos-ophy. Results for other practices are similar. Similarly practices like CS and CO can also be implemented irre-spective of agile philosophy. For example, Schooender-woert [121] in his industrial experience of applying agile practices on an embedded software product from scratch

observed that CS and CO have been well established be-fore agile was incorporated. It is also possible that coding guidelines are informal and not written down in print [107]. It can also be possible that for this reason experi-enced practitioners do not need CS as they accumulate good coding practices through experience. A reason for not practicing CO can be that person with highest knowledge often inspects the code and make changes [107]. Developers can be reluctant to adapt to code writ-ten by others [261].

Adoption levels of CC are almost the same for both embedded and non-embedded software. However, the role of customer may not be always necessary for devel-oping embedded systems. For example, for certain sys-tems the hardware-software architecture and interaction is predefined and therefore limits the possibility of nego-tiating with the customer [213]. This can also be true to telecommunication systems where the individual systems are typically large, evolve over the years and released to a mass market. Moreover, regular communication is neces-sary between hardware and software teams so that the changes proposed by hardware design teams (for exam-ple, processor design) are incorporated [213]. Likewise, for many non-embedded applications customers are not always present. For example, for a banking software real customers are banks and ideal customers are not readily available [218]. For small projects, CC can be expensive and demanding [207]. Customers can find the working style of agile offbeat and some practices such as PP can be perceived as noisy [207]. However, an inside customer who can write stories and performs acceptance tests is very beneficial to the team [116]. A committed customer can participate in team’s daily meetings, i.e. Su and also in prioritization of user stories thus being an enabling factor for Me adoption [116]. This can be true for both embedded and non-embedded systems. However, ac-cording to the data in Table 16, Me has a low level of adoption in the context of non-embedded systems. A pos-sible reason is that Me and also SD are abstract in nature and may be problematic to use [74]. Moreover, there is no guidance to apply these practices [75]. This may be a problem especially to teams that are new to agile as can be seen from Table 14, where we identified a low level of adoption for Me in the context of small teams, and small teams that are new to agile. Modifiability, maintainability and dependability are very important aspects of embed-ded systems, which can be described through Me [76]. Practitioners seem to better describe these issues through detailed description by the use of Me. This can be inter-preted based on data in Table 19.

In regard to other practices, adoption levels for CI are similar for both embedded and non-embedded, more im-portance for continuous testing is given for embedded systems while TDD is considered as a more important agile practice for non-embedded applications. Use of CI allows teams to identify issues where a change to one component affects others. Moreover, CI enables frequent integration of hardware and software components so that at the end of the iteration the embedded product is in completely tested and in usable state [249]. Similarly, it

26

may be difficult to continuously integrate due to lack of coordination between hardware and software teams op-erating at different sites [255]. This is an infrastructure problem for practitioners at different regions in a distrib-uted development and as consequence practitioners face long delays during regular continuous builds [281]. Alt-hough CI has been adopted considerably for non-embedded systems, a negative factor can be a lack of unit test framework [107]. This can be problem to beginner teams or small teams. Moreover, CI can be perceived as complicated and time consuming [107]. TDD can be less suitable for embedded software because of the hardware that usually takes long time from finish to production [77]. Moreover, software at many times can be tested on real test environment, which often is not available imme-diately [77]. However, it is not clear from the studies if unit tests were written before code construction although studies mention testing as part of iterations irrespective of the type of application. Based on the data, we infer that practitioners prioritize to perform continuous testing by having the code tested as and when it is developed in each iteration. In regard to SD, we found it is strongly supported by code refactorings, and requires expertise [116]. SD may be difficult to practice because of the com-plexity involved in embedded software such as timing and memory constraints. In such cases maintaining sim-plest possible design solution may be difficult [78]. How-ever, sometimes assembly code in some of the embedded systems can be difficult to maintain and thus, simple de-sign is useful [78] [79]. However, this may not always be possible because teams have to focus on future milestones and explore dependencies that could have drastic out-comes in mission critical software [80]. For this reason, teams may extend designing solutions beyond the current milestone or iterations thus not exactly sticking to the SD practice. Moreover, SD is not highly adopted for other type of applications. It is considered abstract [82] and also there is a lack of guidance on how to keep the design pro-cess [41].

Based on the data presented one should not be under the assumption that most of the studies focused on XP exclusively. In many of the studies it was reported that Scrum was used in combination with XP. For example, in 18% of the studies reporting application type as non-embedded used a combination of scrum and XP and as low as 9% of the studies used Scrum exclusively. These figures are similar for embedded systems also.

In summary, we ranked the most adopted practices

with respect to the type of applications as shown in Table 20.

TABLE 20 Practice ranking and application type

Embedded Non-embedded Rank = 1 It It Rank = 2 PP, Me PP Rank = 3 CO, CS, CI Me Rank = 4 Re, Te CI, Te Rank = 5 Su, SR, CC Su

4.4.3 Context specific practices

In our analysis in previous sections, we included stud-ies that had focus on overall software development. We also identified studies that reported the use of agile prac-tice(s) for a specific cause/purpose. We classified such studies into various categories. For example, agile has been adopted in software companies for the purposes of managing non-software activities, innovation manage-ment etc. Some of the studies in industry focused on sin-gle practices, like evaluating the efficiency of PP or TDD, identifying practices relevant to architectural issues etc. We did not include such studies in our earlier analysis. In this section we summarized the findings from the studies and practices adopted. First, a summary of practices in-vented and adopted for non-software activities in soft-ware projects in industry is summarized in Table 21.

It can be observed that Studies on TDD reported mixed results as far as defect detection and productivity are concerned (See Table 24). This largely explains the low level of adopted of TDD in software product devel-opment irrespective of the application type (See Sections 4.3, 4.4.1, 4.4.2). On the other hand, studies on PP sug-gested modifications to the practice (See Table 22). This observation can also be traced to studies reporting on PP and TDD in experiments and other empirical research studies conducted with students in academia (See Ap-pendix C). Among these studies include support for PP in distributed context [93] [216] [253] [256].

Table 21

Agile practices supporting non-technical activities Article Description Category Practices

[254] Reports experiences of how com-pany values have been used to personalize agile practices.

Value creation

• Study abroad - spending time with another team (for QA, content development)

• Ask-a-dev (anyone is free to discuss with a designated person or pair)

• Now-or-never bugs (every morning bugs are triggered by the customers and deve-lopers. “never” bugs are worked on im-


mediately, “never” bugs are recorded and discarded

[98]

An experience report that shows how culture is infected with agile thinking and how agile is applied to non-software development acti-vities like training sessions, recruitment staffing, office re-forms, strategic decisions making.

Agile for non-software

• Staffing wall • Recruitment wall • Learning matrix • Feedback in training sessions • Rotation between roles • Small offices (150 to 200 people)

[88] Article reports customer practices used by agile teams.

Customer practic-es

• Customers’ apprentice • Programmer on-site • Programmer holiday • Roadshow • Customer pairing • Customer bootcamp • Big picture up-front • ReCalibration

[89] Investigated how the integration of agile methods and user-centered (UCD) is carried out in practice.

User-centered de-sign

• Paper prototyping with key customers • Evaluating HTML mockups with cus-

tomers to get more requirements • Integrating usability results in iterations

[129]

A lessons learned report on mis-takes in implementing XP practic-es in teams involving junior devel-opers.

Mistakes in XP implementation

• Not switching pairs frequently • Letting the architecture emerge • Not working in a team room • Letting team members specialize • Using long iterations • Not including junior developers

[145] The paper reports case study of integration of User experience de-sign and agile development.

User-centered de-sign/Innovation

management

• Two week sprint prior to start of devel-opment for user behavior analysis

• First week of each sprint spent on re-searching and user testing

• Personas

Table 22 Empirical studies focusing on PP

Article Description Results

[134] The paper reports experiences of using PP in an industrial project (application type: embedded)

• Frequent pair rotation improves knowledge transfer • PP improves quality – low defect count • More suitable for complex tasks • Development effort found to be lower than solo pro-

gramming

[151] The paper describes Software Process Fusion (SPF) that unites pair and solo programming.

• The model proposes to build design code patterns in pairs and use these patterns to write code in solo

• Results suggest that the longer programmers work alone, more code patterns they develop for reuse in pairs.

[191] The paper presents a field study on developer pairs and impact on productivity (application type: embedded)

• Two person teams working independently are more productive than pairs working concurrently

[219]

The paper proposes and presents the results of a framework for the study of PP through ob-servations and analysis of verbalisations, inter-actions and production of artefacts (application type: non-embedded)

• The framework found to be a mechanism for under-standing aspects of XP, assessment of their suitability on the basis of cognitive aspects

• Novice pairs were found to be overconfident believing they would develop software in their own way thus condradicting their peers

• Experienced pairs had fewer interactions as they had better understanding of their role, expertise and of their peers

28

Table 23 Empirical studies with focus on software development process model


[157]

The paper introduces a development practice ‘Divide After You Conquer’ for large agile projects Conquer phase: A stable working example of the sys-tem architecture and design is build iteratively by the core expert team. Use-cases are developed in Test Driven manner. Divide phase: The divide phase starts by Dividing the project into sub-teams based on business areas followed by pairing between developers, busininess analysts and quality analysts. Rotate team members from the newly staffed members thus allowing them to the practice CO

• Successful implementation of the practice needs efficient staffing at the beginning of the divide phase

• A need for communication between the sub-teams other than frequent members’ rotation was observed

[250]

The paper presents a development model that maps seven dimensions of agile maturity, defined into five maturity levels, to agile practices. Practices: TDD, Storing project builds and CM files, Re, CI,

• Model adopted showed a 60% drop in defects • Velocity of the team, i.e. productivity improved

between 8-20% • Many of the component teams were provided

by offshore suppliers and this was perceived as a threat to their business model

• As the model was highly engineering focused it was not initially treated positively by the busi-ness people

Table 24

Empirical studies with focus on TDD


[131] The paper reports a post hoc analysis of an IBM team on the practice of TDD over a period of 5 years

• Writing tests before code construction resulted in higer quality (lesser defect density) products

• A decrease in 15% in defect density and a productivity loss of 15% was found and it was perceived as econom-ically justified

• A very little increase in Cyclomatic code complexity over the course of 10 releases was observed

[146] The paper reports case studies at Microsoft and IBM on the utility of Test Driven development (TDD)

• Pre-release defect density decreased by 40-60% when compared to similar projects that did not use TDD

• An increase of 15-35% in development time was ob-served

• It was recommended not to adopt TDD in the middle when the design is decided

• Add tests to automatic build on a daily basis

[179] The article presents an experiment on TDD conducted with practitioners.

• It was found that test after coding reduces time for writing the tests and modifying the code

• TDD was found to increase testing accuracy and im-prove code quality

[273] The paper presents an experiement comparing code development using TDD and traditional waterfall-like approach

• TDD approach resulted in superior code quality • 78% of the programmers involved in TDD found that

that the practice increases productivity • TDD facilitates simpler design • Programmers following TDD took more time to com-

plete the task


4.4.4 HRR The purpose of this section was to examine the

strength of evidence of the most adopted practices. To this end, we extracted practices from studies with HRR. As shown in Fig. 6, there were 36 studies with HRR. Among these, 12 studies reported development of em-bedded systems and 17 studies on non-embedded. There is a marginal variation of adoption levels for many of the practices when HRR is considered. However, adoption levels for PP, Me and Te are found to be similar in com-parision to all the studies irrespective of rigor and rele-vance. For example, TDD adoption is found to be consid-erably less when compared to other practices and this observation is same for studies with HRR. On the other hand, contrasting results can be noticed in case of practic-es SR, Re, CI and Rt. That is, there is a marginal difference in citation percentage for these practices. In the context of embedded sytems, there is not much variation in terms of citation percentage for all the studies and studies with HRR, for practices PP, TDD, Me, Te, Re and CC. Howev-er, among these practices PP (58%) was the most adopted practice followed by Me (53%), Te (47%), Re and CC (42%). Among studies reporting non-embedded systems and the ones with high HRR, low variation in citation percentage can be found for the practices CC, Re, CO, Rt, Me and Te. Among these practices, the most adopted practice is Me (53%) followed by Te (47%), Re, CC and Rt (41%) and CO (29%).

Based on the data it can be understood that the most adopted practices are PP, Me, Te, Re and CC for both ap-plication types. That is, evidence suggests that these prac-tices are most adopted. However, results drawn from studies with HRR and all the selected primary studies reported do not agree with eachother. Moreover, among studies with HRR there is a not a broad discussion on how agile practices were implemented and the resulting consequences. Therefore, we find that strength of evi-dence on what are the most important agile practices, or what practices are considered the most agile in nature is not high.

In regard to studies reporting context specific practic-es, many practices have been reported. However, we can-not claim the applicability of these practices in different settings. For example, very few studies reported customer practices. Therefore, directness of evidence for these prac-tices in different settings is uncertain because these prac-tices. However, a practical implication is that practition-ers can identify these practices to their own project char-acteristics and examine what are suitable.

4.5 What modifications to agile practices have

been reported? (RQ 1.3) The purpose of this research questions was to identify

modifications/adaptions of various agile practices as re-ported in the studies. To this end, we also identified prac-tices that were used to support agile incorporation. Most interesting adaptations belong to the Communications category. Practitioners have adopted various communica-tion practices. These range from Chat or Instant messen-gers [172] [223] to using Coffee/Ice cream meetings [172].

Using communication channels like Wikis, Information radiators were found effective in communicating and also as a medium to keep track of progress [176]. CRC cards were found efficient in explaining responsibilities and inter module collaboration [133]. Other forms of commu-nication practices are summed up in Table 25.

TABLE 25 Communication practices

Communication

Practices Adapted Citation Chat [223] Coffe/Ice cream meeting [172] Communicating through Ims [172] Communication channels - Wikis, Infor-mation radiators

[176] [253]

Conferencing Cultural ambassadors to Overcome cul-tural barriers among remote teams

[137]

Extensive communication [205] Face to face communication [181] Quick feedback [205] Rapid interaction [270] Show and tell meetings [220] Tools like wiki, cvs, im, discussion boards [190] [192] White board [223] Use of collaborative tools [94]

The role of customer was found in requirements elici-

tation, by using HTML mock-ups to gather more re-quirements and paper prototyping [89] and inviting cus-tomers for development process [158]. Participation of customer spanned user stories development [260], re-views and feedback [227] and performing acceptance tests [87] [111] as can be seen in Table 26.

TABLE 26 Customer collaboration adaptations

Customer collaboration Practices Adapted Citations Active stakeholder participation [191]

Ambassador from onsite [192] Co-located customer [204] Customer accepted interfaces [238]

Customer accustomed tests [178]

Customer apprentice [82] Customer Boot Camp [193] Customer demos [108] [175]

[37] [104] Customer feedback [114] [175]

Customer involvemnt [264] Customer pairing [82] [193]

30

Customer proxies [219] Customer role [90] Customer tests [87] [111] Customers’ apprentice [193] Evaluating an HTML mockup ver-sion with customers to get more requirements

[89]

Expert user involvement [112] Kickoff (2-3 hours): discuss stories with customer, agree on require-ments, create list of tasks

[260]

Paper prototyping with customers [89]

Programmer on-site [193] Proxy client [192] Surrogate customer engagement [193] [111]

User involvment in development process and feedback

[227]

The length of It was not uniform in the studies re-viewed. It lengths varied from 2-3 days [257] to 4 weeks [104]. It specifications [261] were used to provide a de-tailed account of requirements, features and information to test features in the It. Similarly, length of sprint was kept to 2 weeks in many of the studies. Sprint breaks after 5 to 6 sprints were made mandatory in one study [173]. This was to ensure that team members do not burn out in the long run by doing sprints continuously. Other men-tions of sprint lengths include 4 weeks, three and a half weeks. Web conferences and other forms of informal channels were used to conduct Su. Su were held majorly on daily basis but other variations like Su 3 times a week [136] and shared Su once in a week [87] were evident. An overview of modifications/adaptations of Su is given in Table 27.

TABLE 27 Adaptions for stand-ups

Stand-up

Practices Adapted Citation

Daily communication through in-formal channels

[102]

Daily scrum meetings held by local sub-teams

[128]

Daily scrums reduced to three times a week

[136]

Daily stand up meetings: These are not shared by all team members, in-stead rotating role of proxy is created

[87]

Open communications and daily stand-up

[248] [113]

Scrum masters meeting in scrum of scrums

[128]

Scrum meetings shared by whole team using Videoconferencing

[87] [172]

Scrum teleconference across sites [128] Shared stand-up once in a week [87] Standup for 20 minutes [238] Weekly scrum of scrum [144]

Meeting practices include Weekly release meetings

[266], meetings to form pairs for PP [134] and Dirty laun-dry meetings [116]. Dirty laundry meetings are a unique practice that was used to slove any personal conflicts in the team that could hinder productivity.

Practices related to Office were regarding creating an open workspace [100] [210]. Open workspace facilitates communication among team members [20]. PP was found to be more efficient for critical or complex tasks and therefore was thus limited to such tasks [216] [219]. In one instance where the teams where physically distributed, web conference was used to practice PP to get new team members speed up in understanding the features of the project [172].User research was put into place to prioritize features in product backlog [145]. We found a variety of tools/artifacts that were used to track Progress. These include Kanban visibility board [250], Burn down charts [108] [122] as can be seen from Table 28.

TABLE 28 Adaptations for project progress

Progress Practices Adapted Citation Burn down charts [221],[ 253] Customized share Point site for team contact infor-mation, status updates, an-nouncements, discussions and documents

[136]

Daily updates using Wikis [87]

Darts [136] Kanban visibility board [253] Measure progress by work-ing software

[250]

Productlog burn-down [108] Project burn-down charts

[108] [122] [172]

Project dashboard [112] Project dashboard and met-rics

[96]

Project data sheet [147] Project velocity [248] Project visibility [96] Sequencing activities using Design Structure Matrix

[138]

Structured time boxing [264] Tracing development project [138] Velocity assessments [290]


Effort estimation [187]

In the context of Re, automating Re was found to make Re inexpensive and reduce human error when when modifications were frequently made to the system [96]. Running multiple build per day [249] [167] was pointed out for CI. Builds were run multiple times [249] and also overnight [87]. We identified many variations to the prac-tice Rt. Rt sessions were conducted, for example, by post-ing comments on Wikis [94], using videoconferencing [87] as can be seen in Table 29.

TABLE 29 Adaptations for conducting retrospectives

Retrospectives

Practices Adapted Citation Asynchronous retrospective meetings (posting comments and results on Wikis, e-mailing the minutes of local scrum meeting to the onshore team)

[94]

Collective and individual feedback [219] Company retrospectives - 2 to 3 times a year, 90 minutes

[247]

Continuous feedback [133] Daily feedback [167] Demo and retrospective using videocon-ferencing

[87]

Discussing and reviewing project plan [135] Estimate retrospective [149] Feedback in training sessions [98] Iteration demo and retrospective [158] KPT (Keep/Problem/Try) [149] Monthly review [113] Quality review phase [143] ReCalibration [82] [93] Regular team reflection and adjustments to become more effective

[250]

Retrospective sessions at each location and comments updated on SharePoint

[136]

Review (1-2 hours): demonstrate com-pleted stories for customer approval and feedback

[260]

Review feature list [143] Review of iterations with the core team [122] Rolling waves at the end of each itera-tion

[96]

Sprint demo [144] Weekly retrospectives [247] As discussed earlier, Te is one of the most adopted

practices. In many studies, we could not identify if tests were written prior to code construction. In regard to the practice Te, we identified several variations and types like unit tests within each It, customer defined acceptance tests etc. These are given in Table 30. Modifications for other practices are less pronounced. A list of modifica-tions to other practices is given in Appendix D.

TABLE 30 Continuous testing in agile

Testing Practices Adapted Citation

Acceptance testing [143] [105], [195] [207] [208],

Automate acceptance tests using a "“subcutaneous” test approach with a unit testing approach

[249]

Automate as close to 100% of the acceptance tests [249]

Automated regression testing [250]

Automated tests [220] Code inspection [143] Define and execute "just enough acceptance tests" [250]

Develop unit tests for all new code during a sprint [250]

Frequent testing and feedback [205] Functional verification tests [131] [168] Integrated testing [253] [270] Interface integration testing [250] Just enough testing [220] QA in sync [108] Quality agreement [112] Regression testing [102]

Run all acceptance tests in the re-gression test suite with the build (daily at a minimum)

[250]

Run all unit tests with every build [250] Story testing [147]

Support regression tests [258]

Unit testing

[248] [102] [105] [128] [131] [143] [229] [163] [168] [176] [179] [201] [210] [213] [220] [243] [245] [245] [248] [249] [261] [263] [264] [270] [255]

4.6 What is the strength of evidence available on

the use of agile practices in reference to agile philosophy? (RQ2)

However, rigor found in these studies significantly var-iesThe purpose of this research question was to evaluate the strength of evidence on the use of agile practices. To this end, studies reporting agile practices have been as-sessed on the grounds of rigor (See Table 6) and relevance (See Table 7). Rigor is calculated on a scale of 0 to 6 and

32

relevance is calculated on a scale 0 to 4 (a value of 4 im-plying highest relevance). Based on rigor identified, stud-ies are classified into three types: (1) Studies with a high rigor (rigor ≥ 4), (2) Studies with medium rigor (rigor = 3), and (3) Studies with a low rigor (rigor ≤ 3). An overview of rigor and relevance found in the studies is shown in Fig. 6. 108 studies (58%) were found to have a relevance of 4. That is, these studies were conducted in industry and thus results would be more appropriate for industry practice and agile implementation in industry. Of these studies, only 36 studies (33%) are found to have rigor greater than 3, 29 studies with rigor 3 (27%) and the re-maining 43 studies are described rather poorly. This is disappointing from a technology transfer perspective as it is difficult to synthesize how agile practices were imple-mented and on what contexts agile had been put into practice. Overall, 36 (19%) studies offer decision support for agile practices in the sense that these studies were conducted in industry (realistic settings) and were well reported. Decision support, in this context implies that context is described to an extent practitioners can com-pare to their own settings, study design is described in a way practitioners can identify a procedure to obtain simi-lar results and finally validity of the study is described so as to identify factors affecting the study results. In the first step we identified in which the practices implemented were aligned and not aligned to the four core agile values. The four core agile values are the goals of agile philosophy and the 12 principles are the rules that should be followed (See Section 2.1). A mapping of the 12 agile principles and agile values has been defined defined by Petersen [17]. He also identified linkage between vari-ous agile practices and agile principles, thus establishing traceability between agile practices and the ultimate aim of agile philosophy, i.e. agile values. Agile values and the corresponding principles that should be followed are giv-en in Table 31.

TABLE 31

Mapping values to agile principles

Value Agile Principle(s)

V1: Individuals and inter-actions over processes and tools.

AP05: Motivated individuals AP06: Face-to-face conver sation AP08: Sustainable pace AP11: Self-organizing teams

V2: Working software over comprehensive doc-umentation

AP03: Frequent deliveries AP04: Work together AP07: Working software AP09: Technical excellence

V3: Customer collabora-tion over contract negoti-ation

AP01: Customer satisfaction

V4: Responding to change over following a plan

AP02: Welcome change AP10: Simplicity AP12: Continuous reflection

Based on the practices and agile principles that they relate to, we identified which of the core agile values were aligned and not aligned to. In regard to studies with high rigor, practices implemented were aligned to attain-ing the core agile values. On the negative side, we identi-fied many studies where practices implemented were not aligned to values V1, or V4. This phenomenon is similar to studies with low and medium rigor but we identified comparatively less percentage of studies where practices were not aligned to reach either value V1 or V4. Over-view of studies and agile values that were not aligned when using agile practices are given in Tables 32 – 34. We could not identify any distinguishing relation with either company size or agile experience of practitioners in regard to alignment between practices and agile values. It can be thus understood values V1 and V4 are often ne-glected or difficult to achieve. V1 is more related to team, societal and communication aspects than technical as-pects like producing code and design. This is in line with the observation with our results discussed in Section 4.3. For example, practices like 40h week are less cited and thus meaningless used than practices like PP and Contin-uous integration. Next, we selected the studies with high rigor to analyze if practices reported were aligned to the core agile values (See Table 32)

Table 32

Value alignment among studies with Rigor > 4

Values not aligned

Ref Nr.

Company size

Practi-tioners experi-ence

None

[284] 1 Micro

Beginner

[178] [261] 2 Small

[248] [214] 2 Medium

[121] [127] [218]

3 Large

[106] 1 Not men-tioned

V1 [173] 1 Medium [113] 1 Large

V4 [90] 1 Not men-tioned

None

[207] [208] [210]

3 Micro Experi-enced

[253] 1 Large


[87] [222] [226] [255]

4 Not men-tioned

V1 [144] 1 Medium [102] [177] 2 Large

V1, V4 [220] 1 Medium

Table 33 Value alignment among studies with rigor<3

Values not aligned

Ref Nr

Company size

Practi-tioners experi-ence

None [122] [212]

2 Large Beginner

[266] [149] [170]

3 Not men-tioned

V1, V4 [133] 1 Large [129] [163] [250]

3 Not men-tioned

None [254] [245] [285]

3 Small Experi-enced

[108] 1 Large [96] [172] [174] [200]

4 Not men-tioned

V1 [249] [109]

2 Large

[173] 1 Not men-tioned

V1, V2 [157] 1 Not men-tioned

V2, V4 [175] 1 Not men-tioned

None [114] [281]

2 Small Not men-tioned [132] 1 Medium

[117] [137] [138]

2 Large

[250] [253] [170] [192] [198] [241] [262]

6 Not men-tioned

V1 [95] 1 Medium

[136] [176]

2 Not men-tioned

Table 34

Value alignment among studies with rigor=3

Values not aligned

Ref Nr Company size

Practitioners experience

None [116] [213]

2 Small Beginner

[158] 1 Medium [167] 1 Not men-

tioned V1 [234] 1 Medium

[139] 1 Large [247] [264]

2 Not men-tioned

None [175] [176]

2 Micro Experienced

[82] [204]

2 Small

[189] 1 Medium [264] 1 Large [111] [187]

2 Not men-tioned

V1 [206] 1 Small [205] 1 Medium [211] 1 Not men-

tioned V4 [193] 1 Small None [230] 1 Micro Not men-

tioned [111] [115] [128]

2 Large

V1 [95] 1 Medium V1, V4 [244] 1 Not men-

tioned Among the studies with Beginner teams, the instances where principles AP04 and AP11 were none and there was only one study that reported practices which corre-spond to following the principle AP08. This further means, it was not a rule to work together (AP04), main-tain the pace of working for 40 hours a week (AP08) and have the teams to self-organizing teams (AP11). Same phenomenon was observed in the case of teams with ex-perienced practitioners agile wise with the expectation that there were a very few more studies that followed AP08. From the studies reported in the review, we mapped a few practices to principles. These are: ‘Chang-ing role’ (AP11) [172], ‘Frequent rotation between on-site and offsite teams’ (AP11) [138], and ‘Splitting teams across sites’ (AP11) [128].

34

Although traceability between practices and princi-ples was identified in our analysis, it is difficult to say how rigorously a principle is followed actually. This is because it is not mentioned in the reviewed studies which principles were followed and seldom there was a note on agile principles. This kind of traceability was based on the analysis by Petersen [17]. Furthermore, based on the data from the studies reviewed it seems that importance to agile principles and core agile values is not much. It is also not clear how the practices were considered for adoption in the first place and secondly, what kind of evidence (opinion or measurement based initiates) is con-sidered as viable information for agile adoption. Adopt-ing agile practices on account of their popularity [81] and job satisfaction [249] is evident.

5 A METHOD FOR AGILE ASSESSMENT BASED ON CORE AGILE VALUES (MAA-CAV)

The purpose of this section is to introduce and exemplify the method proposed to measure how agile a company is from the perspective of practices implemented. Traceabil-ity between the core agile values and principles is given in Table 31. Agile values define the main goals of agile philosophy and provide rationale why companies should adopt agile [1]. Principles on the other hand constitute the rules that should be followed to achieve the four agile goals [1]. In our analysis we found that although many companies achieve the four agile goals by implementing various sets of agile practices, they do not adhere to fol-lowing the 12 agile principles. For example, in the sample of 20 studies with high region (rigor ≥ 4, See Table 32), 18 studies were found to have practices towards achieving the four agile values. However, we did not find any in-stance where all 12 principles were followed. Practices corresponding to Principles ‘AP04: Work together’ and ‘AP11: Self-organizing teams’ were not found in many studies. The same is the observation with the principle ‘AP08: Sustainable pace’ which was found only in three studies only. A general observation, thus, is achieving the four agile values is imperative to agile in practice and not following principles. Also, by having a right set of prac-tices, all four agile values can be attained without follow-ing all 12 agile principles as a rule. With certain number of practices put into use, it is interesting to answer a few questions that define agile. These are:

1. What principles are followed? 2. How much of each agile core value is achieved? 3. How agile is the company in practice?

Based on these questions we proposed the method Agile assessment based on core agile values (MAA-CAV). The purpose of the method is multifold. Firstly, the method enables mapping between agile practices implemented and agile values. Secondly, based on the principles fol-lowed we can measure how much of each value is creat-ed. Thirdly, the method enables to know how agile a company is and finally provides decision support in iden-tifying improvement potential aspects related to agile.

The method includes four main steps and is shown in Fig. 8. Step 1 of the method is covered in Table 31. Step 2 of the method is extensively covered by Petersen [17] and not discussed here. Steps 3 and 4 of the method are in the following sections.

Fig. 8. The four steps of MAA-CAV

5.1 Step 3 - Calculating measures Measures are calculated for each of the agile values. Table 35 illustrates measures for each agile value based on the mapping between principles and values. 5.2 Step 4 – Measure representation

Representation of core agile value assessment is done at two levels. Firstly, measures for each of the core agile values are represented individually and next, these are combined to identify how agile a company is. The nota-tion to represent agile value assessment is in the form of a fraction where the numerator shows the sum of all the measures of the principles corresponding to an agile val-ue, and the denominator shows if an agile value is ad-dressed at all. That is, if an agile value is addressed, the denominator takes the value 1 and 0 if it is not addressed at all. This kind of calculation is performed for all the four agile values. Thus we get four fractions (referred hereafter as value fractions), each representing an agile value. The rationale for this kind of work up is to know if an agile value is addressed and if yes it facilities to know how much of value is created (in the form of quantitative rep-resentation between 0 and 1).

TABLE 35 Measure definitions for values and princi-ples

Value Principle Measure

V1 Maximum value: 1 Minimum value: 0

AP05: Motivated individuals AP06: Face-to-face conversation AP08: Sustainable pace AP11: Self-organizing teams

For each of the principles repre-sented, measures defined are as follows : If followed and achieved: 0.25 : If followed and result correspond-


ing to this principle is not satisfactory: 0.125 ≈ 0.12 : If not followed: 0


AP03: Frequent deliveries AP04: Work to-gether AP07: Working software AP09: Technical excellence

For each of the principles repre-sented, measures defined are as follows :If followed and achieved: 0.25 :If followed and result correspond-ing to this principle is not satisfactory: 0.125 ≈ 0.12 :If not followed: 0


AP01: Customer satisfaction

:If followed and achieved: 0.33 :If followed and result correspond-ing to this principle is not satisfactory: 0.165 ≈ 0.16 :If not followed: 0


AP02: Welcome change AP10: Simplicity AP12: Continuous reflection

For each of the prin-ciples represented, measures defined are as follows :If followed and achieved: 0.33 :If followed and result correspond-ing to this principle is not satisfactory: 0.165 ≈ 0.16 :If not followed: 0

TABLE 36 Measure illustration

Value Principles followed Measures

Agile assess-ment repre-

sentation

V1

AP06 0.25

0.75/1 AP08 0.25

AP11 0.25

AP05 -

V2

AP03 0.25

1/1 AP04 0.25

AP07 0.25

AP09 0.25

V3 - - 0/0

V4

AP02 0.33

1/1 AP10 0.33

AP12 0.33

Combining V1, 2,3 and 4 2.75/3 Level 1 – Par-tial: Partial

Once measures for each of the agile values are collect-

ed in the form of fractions, they are combined to show a quantitative representation of how agile a company is. The resultant representation is a fraction where the nu-merator is the sum of all the numerators of the value frac-tions and denominator is the sum of denominators of the value fractions. The rationale here is that if a company says it is agile, it ought to address the four core agile val-ues given in agile manifesto.

We defined four levels to represent core agile assess-ment. This is shown in Fig. 9. For example, Level 4 means that all the four core agile values were addressed and were complete. That is in essence, all the principles were actually followed in practice. An example to illustrate the method described is given here (See Table 36). Lets’ con-sider among principles contributing to V1, AP05 is not followed at all. According to the measures given in Table 35, AP06, AP08 and AP11 get 0.25 points each. Next for V2, let us consider all the principles are followed with successful results; principles contributing to V3 are not followed while principles contributing to V4 are followed. The next step, calculating measures and defining core value assessment using the scale is given in Table 35.

Based on the scale defined, agile assessment level of the example company is Level 1 which means that prac-tices implemented by the company do not address all the four core agile values. Agile assessment value 2.75/3 shows that only 3 agile values were addressed (i.e. de-nominator) and the 3 values were partially achieved (i.e. numerator). A visualization chart can be used to know the area of improvement potential. An example of such visualization is shown in Fig. 10. In the graph, it can be noted that a small area is agile and a large area of what can be agile is untouched. A template (Table 37) is de-signed to aid the process of agile assessment for calculat-ing measures (Steps 2 and 3). This template can be used as a tool for charting out practices used and issues found thus pointing out to what worked and what did not. We illustrated the use of the method proposed using studies (rigor > = 4) found in our review. These are given in Ap-pendix D.

36

Fig. 9. Levels in agile assessment

TABLE 37 Template for agile assessment

• Project name • Team size • Team agile experience • Project duration!

Nr. Practice Purpose Actors Outcome Other support-ing practices used

Notes

Fig. 10. Agile assessment visualization


6 CONCLUSION We conducted a SLR that consists of 186 primary studies published between 2000-10 that investigates what agile practices have received most attention in terms of adop-tion. To this end we identified the most adopted practices in regard to team size, practitioners’ experience agile wise and application type. In addition we identified many em-pirical and non-empirical papers (research papers) that focused on specific aspects of agile software development and not software project development in general. These studies broadly fell into the following themes/categories:

• Overall software development: These studies dis-

cussed the implementation of agile practices in the overall development of a software project/product and not on very narrowed aspects.

• Non-technical activities: Studies that pro-posed/used new practice to support non-technical aspects of agile projects like customer collaboration, innovation management, Staffing etc.

• Pair-programming/ Test-driven development: Stud-ies whose focus was solely on the effort to evaluate, propose modifications, or better understand Pair-programming or Test-driven development.

• Architectural practices: These studies analyzed lack of attention on software architecture in agile and proposed/evaluated a set of architectural practices.

• Quality assurance/security or risk assessment: The-se studies focused on proposing/evaluating quality assurance aspects of agile, for example evaluating re-factoring, defining quality indicators in code refac-toring, proposing security practices to support exist-ing agile development.

The review provides a list of findings according to the above themes/categories which practiotioners compare with their settings and identify relevance to their needs. In regard to the resear papers presented, we identified a divergence rather than concurrence. That is, research pa-pers presented discussions on a variety of topics and thus our study is limited by the directness of the evidence in extracting combinatorial results. Our findings also show how practices were modified/adapted. In this regard, practices related to the role of customer, Stand-ups are most often modified.

As far as practice adoption is considered, XP practices or Scrum in combination with XP are being favored. However, we find a high rate of adoption for Pair-programming, Refactoring, Continuous Integration and continuous testing, i.e. testing throughout the project. However, it is not clear if the tests are written before code construction. However, we did not identify how success and benefits of the use of agile practices were measured. In many studies, we did not find discussions on outcomes of agile implementation and measures taken into consid-eration. To this end, a framework to measure the outcome of agile practices is needed.

We also identified that the choice of agile practices in

projects is not often aligned with the core agile values of agile philosophy. Most often non-technical aspects of ag-ile are not in place. To this end, we proposed a pre-limenary method for agile assessment based on core agile values (MAA-CAA). In our future work, we intend to validate and propose modifications. To this end, it is in-teresting to investigate the phenomenon or process be-hind choosing a set of practices in projects.

REFERENCES [1] K. Petersen, Implementing Lean and Agile Software Develop-

ment in Industry, PhD Dissertation No. 2010:04, ISBN 978-91-7295-180-8, Blekinge institute of technology, Karlskrona, 2010.

[2] George Stalk Jr. Time - the next source of competitive ad-vantage. Harvard Business Review, 66(4), 1988.

[3] B. Kitchenham and S.L. Pfleeger, “Software quality: the elusive target [special issues section],” Software, IEEE, vol. 13, Jan. 1996, pp. 12-21)

[4] K. Elissa, “An Overview of Decision Theory,"unpublished. (Unplublished manuscript)

[5] R. van Solingen, D. F. Rico, and M. Zelkowitz, #Calculating software process improvement‘s return on investment,‖ in Advances in Computers, 2006, vol. 66, pp. 1–41

[6] Lan Cao, K. Mohan, PengXu, and B. Ramesh, “How extreme does extreme programming have to be? Adapting XP practices to large-scale projects,” in System Sciences, 2004. Proceedings of the 37th Annual Hawaii International Conference on, 2004, p. 10 pp.

[7] M. Scho¨nstro¨m, Software development as knowledge work – a conceptual framework, in: J. Hedman, T. Kalling, D. Khakar, O. Steen (Eds.), Lund onInformatics, CBS Press, Copenhagen, 2005, pp.132–155.

[8] G. Kotonya and I. Sommerville, Requirements Engineering: Processes and Techniques, Wiley, 1998.

[9] Warsta, J., Salo, O., Ronkainen, J. and Abrahamsson, P., “Agile software development methods. Review and analysis”, VTT Electronic Espoo, pp. 1-107, 2002.

[10] M. Fowler, “The new methodology,” Wuhan University Journal of Natural Sciences, vol. 6, 2001, pp. 12-24.

[11] B. Boehm, “Get Ready for Agile Methods, with Care,” Comput-er, vol. 35, Jan. 2002, p. 64–69.

[12] H. Gallis, E. Arisholm, and T. Dybå, “An Initial Framework for Research on Pair Programming,” in Empirical Software Engineer-ing, International Symposium on, Los Alamitos, CA, USA, 2003, vol. 0, p. 132.

[13] W.W. Royce, “Managing the development of large software systems: concepts and techniques,” Proceedings of the 9th in-ternational conference on Software Engineering, Los Alamitos, CA, USA: IEEE Computer Society Press, 1987, p. 328–338.

[14] P. Kruchten, The Rational Unified Process: An Introduction, Addison-Wesley Professional, 2000.

[15] V. Schuppan and W. Russwurm, “A CMM-Based Evaluation of the V-Model 97,” 2000

[16] T. Hall, A. Rainer, N. Baddoo, and S. Beecham, "An empirical study of maintenance issues within process improvement pro-grammes in the software industry," in Proceedings of the IEEE International Conference on Software Maintenance, 2001, pp. 422-430.

[17] K. Petersen, “Is Lean Agile and Agile Lean? A Comparison Between Two Development Paradigms”; in Modern Software Engineering Concepts and Practices: Advanced Approaches; Ali Dogru and VeliBicer (Eds.)

[18] Truex, D. P., Baskerville, R. and Travis, J,” A methodical sys-tems development: The deferred meaning of systems develop-ment systems,”. Accounting, Management and Information Technology 10, 53-79, 2000.

[19] “Manifesto for Agile Software Development.” [Online]. Availa-ble: http://agilemanifesto.org/. [Accessed: 10-May-2011].

[20] K. Beck, Extreme Programming Explained: Embrace Change, Addison-Wesley, 2000.

[21] K. Schwaber, M. Beedle, “Agile Software Development with Scrum,” Prentice Hall, Upper Saddle River, 2001.

38

[22] M. Poppendieck, T. Poppendieck, Lean Software Development – An Agile Toolkit for Software Development Managers, Addi-son-Wesley, Boston, 2003.

[23] A. Cockburn, Crystal Clear: A Human-Powered Methodology for Small Teams, Addison-Wesley, 2004.

[24] S.R. Palmer, J.M. Felsing, A Practical Guide to Feature-driven Development, Prentice Hall, Upper Saddle River, NJ, 2002.

[25] J. Stapleton, DSDM: Business Focused Development, second ed., Pearson Education, 2003.

[26] M. Fowler, Is design dead?, Addison-Wesley Longman Publish-ing Co., Inc. Bostonp. 3–17, 2001.

[27] J. Sutherland and W.-J. van den Heuvel, “Enterprise application integration and complex adaptive systems,” Communications of the ACM, vol. 45, Oct. 2002, p. 59–64.

[28] PekkaAbrahamsson, “Agile Software Development Methods: A Minitutorial”, VTT publications, 2002.

[29] Schwaber, C., and Fichera, R.: ‘Corporate IT leads the second wave of agile adoption’ (Forrester Research, Inc., 2005).

[30] Sillitti, A., Ceschi, M., Russo, B., et al.: ‘Managing uncertainty in requirements: a survey in documentation-driven and agilecom-panies’. Proc. Software Metrics 2005, 11th IEEE Int. Symp.,2005, pp. 17–27

[31] Lindvall, M., Muthig, D., Dagnino, A., et al.: ‘Agile software development in large organizations’, Comput., 2004, 37, pp. 26–34

[32] Eckstein, J.: ‘Agile software development in large-diving into the deep’ (Dorset House Publishing, 2004)

[33] O. Salo and P. Abrahamsson, “Agile methods in European em-bedded software development organisations: a survey on the actual use and usefulness of Extreme Programming and Scrum,” Software, IET, vol. 2, Feb. 2008, pp. 58-64.

[34] Smigelschi, D.: ‘Combining predictability with extreme pro-gramming in embedded multimedia project’. Proc. 3rd Int. Conf. Extreme Programming and Agile Processes in Software Engineering (XP2002), Sardinia, Italy, 2002, pp. 191–192

[35] B. Boehm, “Get Ready for Agile Methods, with Care,” Comput-er, vol. 35, 2002, pp. 64-69.

[36] B. Ramesh, with Lan Cao, “Agile Requirements Engineering Practices: An Empirical Study,” Software, IEEE, vol. 25, Feb. 2008, pp. 60-67.

[37] L. Krzanik, P. Rodriguez, J. Simila, P. Kuvaja, and A. Rohunen, “Exploring the Transient Nature of Agile Project Management Practices,” Jan. 2010, pp. 1-8.

[38] Glazer, H., J. Dalton, D. Anderson, M. Konrad, and S. Shrum, “CMMI® or Agile: Why Not Embrace Both!”, CMU/SEI-2008-TN-003, Pittsburgh, 2008.

[39] T. Dybå and T. Dingsøyr, “Empirical studies of agile software development: A systematic review,” Information and Software Technology, vol. 50, Aug. 2008, p. 833–859.

[40] A Preliminary Roadmap for Empirical Research onAgile Soft-ware Development.

[41] B. A. Kitchenham, T. Dyba, and M. Jorgensen, “Evidence-Based Software Engineering,” Proceedings of the 26th International Con-ference on Software Engineering, p. 273–281, 2004.

[42] B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Soft-ware Engineering Group, Keele University and Department of ComputerScience, University of Durham, UK, Technical Report EBSE-2007- 01, Jul. 2007.

[43] A.C.C. França, F.Q.B. da Silva, and L.M.R. de Sousa Mariz, “An empirical study on the relationship between the use of agile practices and the success of Scrum projects,” Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Soft-ware Engineering and Measurement, New York, NY, USA: ACM, 2010, p. 37:1–37:4.

[44] A. Sidky and J. Arthur, “Determining the Applicability of Agile Practices to Mission and Life-Critical Systems,” Proceedings of the 31st IEEE Software Engineering Workshop, Washington, DC, USA: IEEE Computer Society, 2007, p. 3–12.

[45] J. Highsmith, Agile Software Development Ecosystems, Addi-son-Wesley Professional, 2002.

[46] K. Schwaber and M. Beedle, Agile Software Development with Scrum, Prentice Hall, 2001.

[47] C. Hansson, Y. Dittrich, B. Gustafsson, and S. Zarnak, “How agile are industrial software development practices?,” Journal of Systems and Software, vol. 79, Sep. 2006, p.

[48] T.S. Kuhn, The Structure of Scientific Revolutions, University Of Chicago Press, 1996.

[49] V. Rajlich, “Changing the paradigm of software engineering,” Communications of the ACM, vol. 49, Aug. 2006, p. 67–70.

[50] Erickson, K. Lyytinen, and K. Siau, “Agile Modeling, Agile Software Development, and Extreme Programming,” Journal of Database Management, vol. 16, 34. 2005, pp.

[51] M. Striebeck, “Ssh! We Are Adding a Process...,” AGILE Con-ference, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 185-193.

[52] M. Ivarsson and T. Gorschek, “Technology transfer decision support in requirements engineering research: a systematic re-view of REj,” Requirements Engineering, vol. 14,

[53] Cohen, D., Lindvall, M., and Costa, P., “An Introduction to Agile Methods,” in Advances in Computers, Advances in Soft-ware Engineering, vol. 62, M. V. Zelkowitz, Ed. Amsterdam: Elsevier, 2004.

[54] J. Erickson, K. Lyytinen, and K. Siau, “Agile Modeling, Agile Software Development, and Extreme Programming,” Journal of Database Management, vol. 16, 34. 2005, pp. 88-100.

[55] M. Laanti, O. Salo, and P. Abrahamsson, “Agile methods rapid-ly replacing traditional methods at Nokia: A survey of opinions on agile transformation,” Information and Software Technolo-gy, vol. 53, Mar. 2011, pp. 276-290.

[56] K. Sureshchandra and J. Shrinivasavadhani, “Adopting Agile in Distributed Development,” Global Software Engineering, In-ternational Conference on, Los Alamitos, CA, USA: IEEE Com-puter Society, 2008, pp. 217-221.

[57] E. Hossain, M.A. Babar, H.-young Paik, and J. Verner, “Risk Identification and Mitigation Processes for Using Scrum in Global Software Development: A Conceptual Framework,” Asia-Pacific Software Engineering Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 457-464.

[58] J. C. de Almeida Biolchini, P. G Milan, A. C. C Natali, T. U. Conte, and G. H. Travassos, “Scientific research ontology to support systematic review in software engineering, “ Advanced Engineering Informatic, vol. 21, no. 2, pp. 131-151, Apr. 2007.

[59] M. Ivarsson and T. Gorschek, “A method for evaluating rigor and industrial relevance of technology evaluations,” Empirical Software Engineering, vol. 16, no. 3, pp. 365-395, Oct. 2010.

[60] J. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin, vol. 76, no. 5, pp. 378–382,1971.

[61] J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, Mar. 1977.

[62] “Enterprise - SME definition,” Aug. 2009. [Online]. Available: http://ec.europa.eu/enterprise/enterprise_policy/ sme_definition/index_en.htm

[63] M. V. Zelkowitz and D. Wallace, “Experimental validation in software engineering,” Information and Software Technolo-gy,vol. 39, no. 11, pp. 735–743, 1997.

[64] Robert K. Yin. Case study research: design and methods. Sage Publications, 3rdedition, 2003.

[65] Neill CJ, Laplante PA (2003) Requirements engineering: the state of the practice. IEEE Softw 20(6):40–45.

[66] Dybå T, Dingsøyr T et al (2007) Applying systematic reviews to diverse study types: an experience report. First International Symposium on Empirical Software Engineering and Measure-ment (ESEM).

[67] A. Qumer and B. Henderson-Sellers, “An evaluation of the degree of agility in six agile methods and its applicability for method engineering,” Information and Software Technology, vol. 50, no. 4, pp. 280-295, Mar. 2008.

[68] B. George and L. Williams, “A structured experiment of test-driven development,” Information and Software Technology, vol. 46, no. 5, pp. 337-342, Apr. 2004.

[69] K. Petersen and C. Wohlin, “Context in industrial software engineering research,” in Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measure-ment, Orlando, USA, 2009, pp. 401–404.

[70] Peeters J. Agile Security Requirements Engineering. Presented at the Symposium on Requirements Engineering for Infor-mation Security, 2005.

[71] P. Schuh, “Recovery, redemption, and extreme programming,” Software, IEEE, vol. 18, no. 6, pp. 34-41, 2001.

[72] N. B. Moe, T. Dingsøyr, and T. Dybå, “Understanding self-organizing teams in agile software development,” in 19th Aus-tralian Conference on Software Engineering, 2008, pp. 76–85.

[73] T. Dingsøyr, G. Hanssen, T. Dybå, G. Anker, and J. Nygaard, “Developing software with scrum in a small cross-


organizational project,” Software Process Improvement, pp. 5–15, 2006.

[74] K. Conboy and B. Fitzgerald, “Method and developer charac-teristics for effective agile method tailoring: a study of XP ex-pert opinion,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 20, no. 1, p. 2, 2010.

[75] . Rumpe and A. Schröder, “Quantitative survey on extreme programming projects,” in Third International Conference on Ex-treme Programming and Flexible Processes in Software Engineering, XP2002, May, pp. 26–30.

[76] J. Srinivasan, R. Dobrin, and K. Lundqvist, “‘State of the Art’in Using Agile Methods for Embedded Systems Development,” in 2009 33rd Annual IEEE International Computer Software and Appli-cations Conference, 2009, pp. 522–527.

[77] J. Grenning and E. Class, “Test driven development for embed-ded software,” ESC, vol. 241, 2007.

[78] T. Punkka, “Agile Methods and Firmware Development.”. [79] M. Pikkarainen, J.M Sagredo and S Estela, "An Agile Practice

Tool Box." [Online]. Available: http://www.agile-itea.org/public/deliverables.php. [Accessed: 09-Oct-2011].

[80] J. Bowers, J. May, E. Melander, M. Baarman, and A. Ayoob, “Tailoring XP for large system mission critical software devel-opment,” Extreme Programming and Agile Methods—XP/Agile Universe 2002, pp. 269–301, 2002.

SYSTEMATIC LITERATURE REVIEW REFERENCES

[81] P.K. Vejandla and L.B. Sherrell, “Why an AI research team adopted XP practices,” Proceedings of the 47th Annual South-east Regional Conference, New York, NY, USA: ACM, 2009, p. 72:1–72:4.

[82] A. Martin, R. Biddle, and J. Noble, “XP Customer Practices: A Grounded Theory,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 33-40.

[83] R. Mahapatra, S.P. Nerur, K. Price, and V. Balijepally, “Are Two Heads Better than One for Software Development? The Productivity Paradox of Pair Programming,” Management In-formation Systems Quarterly, vol. 33, Mar. 2009, pp. 91-118.

[84] Nawrocki, J.R., Jasinski, M., Bartosz, W., Wojciechowski, A.: Combining Extreme Programming with ISO 9000. In: Proceed-ings of EurAsia-ICT 2002, Iran, October 29-31, 2002. Lecture Notes in Computer Science, Vol.2510. Springer-Verlag, Berlin Heidelberg New York (2002) 786–794.

[85] M.A. Babar, “An exploratory study of architectural practices and challenges in using agile software development approach-es,” Sep. 2009, pp. 81-90.

[86] P. Middleton, A. Flaxel, and A. Cookson, “Lean Software Man-agement Case Study: Timberline Inc.,” in Extreme Programming and Agile Processes in Software Engineering, vol. 3556, H. Baumeister, M. Marchesi, and M. Holcombe, Eds. Berlin, Hei-delberg: Springer Berlin Heidelberg, 2005, pp. 1-9.

[87] J. Sutherland, G. Schoonheim, N. Kumar, V. Pandey, and S. Vishal, “Fully Distributed Scrum: Linear Scalability of Produc-tion between San Francisco and India,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 277-282.

[88] Wanfeng Dou, Kui Hong, , and , Xiaoyong Zhang, “A Frame-work of Distributed Pair Programming System,” Dec. 2009, pp. 1-4.

[89] Z Hussain, W Slany, A Holzinger (2009) Investigating Agile User-Centered Design in Practice: A Grounded Theory Perspec-tive. In Andreas 5th Annual Usability Symposium 2009, Linz, pp. 279-289.

[90] S. Shahzad, “Learning from Experience: The Analysis of an Extreme Programming Process,” Information Technology: New Generations, Third International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 1405-1410.

[91] D. Batra, “Modified agile practices for outsourced software projects,” communications of the ACM, vol. 52, Sep. 2009, p. 143–148.

[92] J. Sutherland and I. Altman, “Take No Prisoners: How a Ven-ture Capital Group Does Scrum,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 350-355.

[93] T Schummer, S Lukosch (2009) Understanding Tools and Prac-tices for Distributed Pair Programming. Journal of Universal Computer Science 15(16): 3101-3125.

[94] E. Hossain, M.A. Babar, and H.-young Paik, “Using Scrum in Global Software Development: A Systematic Literature Re-view,” Global Software Engineering, International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 175-184.

[95] P Clutterbuck, T Rowlands, O Seamons (2009) A case study of SME Web application development via agile methods. In Pro-ceedings of the 2nd European Conference on Information Man-agement and Evaluation, Academic Publishing Limited, Read-ing, UK, pp. 77-87.

[96] H Koehnemann, M Coats (2009) Experiences Applying Agile Practices to Large Systems. In Agile Conference, 2009. AGILE '09, Chicago, pp. 295-300.

[97] P. Hamill, D. Alexander, and S. Shasharina, “Web Service Vali-dation Enabling Test-Driven Development of Service-Oriented Applications,” Services, IEEE Congress on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 467-470.

[98] C. Doshi and D. Doshi, “A Peek into an Agile Infected Culture,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 84-89.

[99] M. Zeng, C. Wang, and Q.-yun Long, “Practices of Extreme Programming for ERP Based on Two-dimensional Dynamic Time Scheduling Interface Method,” Software Engineering, World Congress on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 236-240.

[100] K Usha, N Poonguzhali, E Kavitha (2009) A quantitative ap-proach for evaluating the effectiveness of refactoring in soft-ware development process. In Proceeding of International Con-ference on Methods and Models in Computer Science, ICM2CS 2009, Delhi, pp. 1-7.

[101] E. Hossain, M.A. Babar, H.-young Paik, and J. Verner, “Risk Identification and Mitigation Processes for Using Scrum in Global Software Development: A Conceptual Framework,” Asia-Pacific Software Engineering Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 457-464.

[102] A.I. Nawaz and I.A. Zualkernan, “The role of agile practices in disaster management and recovery: a case study,” Proceedings of the 2009 Conference of the Center for Advanced Studies on Collaborative Research, New York, NY, USA: ACM, 2009, p. 164–173.

[103] A Garg (2009) Agile Software Development. DRDO Science Spectrum, pp. 55-59.

[104] C.A. Wellington, “Managing a project course using Extreme Programming,” Oct. 2005, p. T3G-1.

[105] G. Vanderburg, “A simple model of agile software processes – or – extreme programming annealed,” ACM SIGPLAN Notices, vol. 40, Oct. 2005, p. 539–545.

[106] B. Upender, “Staying Agile in Government Software Projects,” Agile Development Conference/Australasian Database Confer-ence, Los Alamitos, CA, USA: IEEE Computer Society, 2005, pp. 153-159.

[107] H. Svensson and M. Höst, “Introducing an Agile Process in a Software Maintenance and Evolution Organization,” Software Maintenance and Reengineering, European Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2005, pp. 256-264.

[108] J W Spence There has to be a better way! [software develop-ment]. In Proceedings of Agile Conference, AGILE 2005, Den-ver,2005, pp. 272-278.

[109] N Pikkarainen, U Passoja (2005) An Approach for Assessing Suitability of Agile Solutions: A Case Study. In 6th International Conference on Extreme Programming and Agile Processes in Software Engineering, pp. 171-179.

[110] R.F. Paige, H. Chivers, J.A. McDermid, and Z.R. Stephenson, “High-integrity extreme programming,” Proceedings of the 2005 ACM symposium on Applied computing, New York, NY, USA: ACM, 2005, p. 1518–1523.

[111] V B Miic Perceptions of extreme programming: A pilot study, In Engineering Management Conference 2005, Newfoundland, 2005, pp. 307-312.

[112] T. Little, “Context-adaptive agility: managing complexity and uncertainty,” Software, IEEE, vol. 22, Jun. 2005, pp. 28- 35.

[113] A. Law and S. Learn, “Waltzing with changes [agile software development],” Jul. 2005, pp. 279- 285.

[114] M RHilkka, T Tuure, R Matti Is extreme programming just old wine in new bottles: A comparison of two cases. Journal of Da-tabase Management 16(4), 2005, pp. 41-61.

[115] Y Dubinsky, O Hazzan, A Keren” Introducing extreme pro-gramming into a software project at the Israeli Air Force”. In

40

6th International Conference on Extreme Programming and Ag-ile Processes in Software Engineering, Sheffield,2005, pp. 19-27.

[116] A F Da Silva, F Kon, C Torteli (2005) XP South of the Equator: An eXPerience Implementing XP in Brazil. In Proceedings of the 6th International Conference on Extreme Programming and Agile Processes in Software Engineering, Sheffield, pp. 10-18.

[117] C Olivier, C David, G M George G (2005) A software methodol-ogy for applied research: eXtreme Researching. Software: Prac-tice and Experience 35(15): 1441-1454.

[118] G Canfora, A Cimitile, A A Visaggio (2005) Empirical study on the productivity of the pair programming. In 6th International Conference, XP 2005, pp. 92-99.

[119] E. Bellini, G. Canfora, F. García, M. Piattini, and C.A. Visaggio, “Pair designing as practice for enforcing and diffusing design knowledge,” Journal of Software Maintenance and Evolution: Research and Practice, vol. 17, Nov. 2005, pp. 401-423.

[120] S. Xu and V. Rajlich, “Empirical Validation of Test-Driven Pair Programming in Game Development,” Computer and Infor-mation Science, 5th IEEE/ACIS International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 500-505.

[121] N.V. Schooenderwoert, “Embedded Agile Project by the Num-bers With Newbies,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 351-366.

[122] M. Striebeck, “Ssh! We Are Adding a Process...,” AGILE Con-ference, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 185-193.

[123] P. Sfetsos, L. Angelis, and I. Stamelos, “Investigating the ex-treme programming system–An empirical study,” Empirical Software Engineering, vol. 11, 2006, pp. 269-301.

[124] J.-G. Schneider and R. Vasa, “Agile Practices in Software De-velopment - Experiences from Student Projects,” Software En-gineering Conference, Australian, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 401-410.

[125] R Moser, A Sillitti, P Abrahamsson, G Succi (2006) Does Refac-toring Improve Reusability? In 9th International Conference on Software Reuse, ICSR 2006, Torino pp. 287-97.

[126] P. Meso and R. Jain, “Agile Software Development: Adaptive Systems Principles and Best Practices,” Information Systems Management, vol. 23, 2006, p. 19.

[127] J. Auvinen, R. Back, J. Heidenberg, P. Hirkman, and L. Mi-lovanov, “Software Process Improvement with Agile Prac-tices in a Large Telecom Company,” in Product-Focused Software Process Improvement, vol. 4034, J. Münch and M. Vierimaa, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 79-93.

[128] J. Sutherland, A. Viktorov, J. Blount, and N. Puntikov, “Distrib-uted Scrum: Agile Project Management with Outsourced De-velopment Teams,” Hawaii International Conference on System Sciences, Los Alamitos, CA, USA: IEEE Computer Society, 2007, p. 274a.

[129] R. Lawrence, “XP and Junior Developers: 7 Mistakes (and how to avoid them),” Aug. 2007, pp. 234-239.

[130] L Madeyski (2007) On the effects of pair programming on thor-oughness and fault-finding effectiveness of unit tests. In 8th In-ternational Conference on Product-Focused Software Process Improvement, Riga, pp. 207-221.

[131] J.C. Sanchez, L. Williams, and E.M. Maximilien, “On the Sus-tained Use of a Test-Driven Development Practice at IBM,” Aug. 2007, pp. 5-14.

[132] R. Sison and T. Yang, “Use of Agile Methods and Practices in the Philippines,” Asia-Pacific Software Engineering Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2007, pp. 462-469.

[133] M. Sulfaro, M. Marchesi, and S. Pinna, “Agile Practices in a Large Organization: The Experience of Poste Italiane,” in Agile Processes in Software Engineering and Extreme Programming, vol. 4536, G. Concas, E. Damiani, M. Scotto, and G. Succi, Eds. Ber-lin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 219-221.

[134] Jari Vanhanen and Harri Korpi, “Experiences of Using Pair Programming in an Agile Project,” Jan. 2007, p. 274b-274b.

[135] X. Wang, Z. Wu, and M. Zhao, “The Relationship between De-velopers and Customers in Agile Methodology,” Computer Sci-ence and Information Technology, International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 566-572.

[136] M. Vax and S. Michaud, “Distributed Agile: Growing a Practice Together,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 310-314.

[137] R. Urdangarin, P. Fernandes, A. Avritzer, and D. Paulish, “Ex-periences with Agile Practices in the Global Studio Project,” Global Software Engineering, International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 77-86.

[138] K. Sureshchandra and J. Shrinivasavadhani, “Adopting Agile in Distributed Development,” Global Software Engineering, In-ternational Conference on, Los Alamitos, CA, USA: IEEE Com-puter Society, 2008, pp. 217-221.

[139] M. Summers, “Insights into an Agile Adventure with Offshore Partners,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 333-338.

[140] C.A. Siebra, M. Filho, F. Silva, and A. Santos, “Deciphering extreme programming practices for innovation process man-agement,” Sep. 2008, pp. 1292-1297.

[141] A.A. Sharifloo, A.S. Saffarian, and F. Shams, “Embedding Ar-chitectural Practices into Extreme Programming,” Software En-gineering Conference, Australian, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 310-319.

[142] S.H. Rayhan and N. Haque, “Incremental Adoption of Scrum for Successful Delivery of an IT Project in a Remote Setup,” AG-ILE Conference, Los Alamitos, CA, USA: IEEE Computer Socie-ty, 2008, pp. 351-355.

[143] M Qasaimeh, H Mehrfard, and A Hamou-Lhadj (2008) Compar-ing Agile Software Processes Based on the Software Develop-ment Project Requirements. In International Conference on Computational Intelligence for Modelling, Control and Auto-mation, Vienna, pp. 49-54.

[144] M Paasivaara, S Durasiewicz and C Lassenius (2008) Using scrum in a globally distributed project: a case study. Software Process: Improvement and Practice 13(6): 527-44.

[145] M. Najafi and L. Toyoshiba, “Two Case Studies of User Experi-ence Design and Agile Development,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 531-536.

[146] N. Nagappan, E.M. Maximilien, T. Bhat, and L. Williams, “Re-alizing quality improvement through test driven development: results and experiences of four industrial teams,” Empirical Software Engineering, vol. 13, 2008, pp. 289-302.

[147] R. Mugridge, “Managing Agile Project Requirements with Sto-rytest-Driven Development,” IEEE Software, vol. 25, 2008, pp. 68-75.

[148] L. Madeyski, “Impact of pair programming on thoroughness and fault detection effectiveness of unit test suites,” Software Process: Improvement and Practice, vol. 13, May. 2008, pp. 281-295.

[149] F. Kinoshita, “Practices of an Agile Team,” Aug. 2008, pp. 373-377.

[150] A. Martin, J. Noble, and R. Biddle, “Programmers are from Mars, customers are from Venus: a practical guide for custom-ers on XP projects,” Proceedings of the 2006 conference on Pat-tern languages of programs, New York, NY, USA: ACM, 2006.

[151] K. M. Lui and K. C. C. Chan, “Software Process Fusion: Uniting Pair Programming and Solo Programming Processes,” in Soft-ware Process Change, vol. 3966, Q. Wang, D. Pfahl, D. M. Raffo, and P. Wernick, Eds. Berlin, Heidelberg: Springer Berlin Hei-delberg, 2006, pp. 115-123.

[152] D.B. Levine and H.M. Walker, “XP practices applied to grad-ing,” ACM SIGCSE Bulletin, New York, NY, USA: ACM, 2006, p. 173–177.

[153] D. Kirk and E. Tempero, “Identifying Risks in XP Projects through Process Modelling,” Software Engineering Conference, Australian, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 411-420.

[154] K.M. Lui and K.C.C. Chan, “Pair programming productivity: Novice-novice vs. expert-expert,” International Journal of Hu-man-Computer Studies, vol. 64, Sep. 2006, pp. 915-925.

[155] R. N. Jensen, T. Møller, P. Sönder, and G. Tjørnehøj, “Archtec-ture and Design in eXtreme Programming; Introducing ‘Devel-oper Stories’,” in Extreme Programming and Agile Processes in Software Engineering, vol. 4044, P. Abrahamsson, M. Marchesi, and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidel-berg, 2006, pp. 133-142.

[156] H. Holmström, B. Fitzgerald, P.J. Ågerfalk, and E.Ó. Conchúir, “Agile Practices Reduce Distance in Global Software Develop-ment,” Information Systems Management, vol. 23, 2006, p. 7.

[157] A. Elshamy and A. Elssamadisy, “Divide After You Conquer: An Agile Software Development Practice for Large Projects,” in Extreme Programming and Agile Processes in Software Engineering, vol. 4044, P. Abrahamsson, M. Marchesi, and G. Succi, Eds. Ber-lin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 164-168.


[158] R. Coffin, “A tale of two projects [agile projects],” Jul. 2006, p. 7 pp.-164.

[159] G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C. A. Visag-gio, “Productivity of Test Driven Development: A Controlled Experiment with Professionals,” in Product-Focused Software Process Improvement, vol. 4034, J. Münch and M. Vierimaa, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 383-388.

[160] J. Bergin and F. Grossman, “Extreme Construction: Making Agile Accessible,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 384-389.

[161] R. Baskerville, B. Ramesh, L. Levine, and J. Pries-Heje, “High-Speed Software Development Practices: What Works, What Doesn’t,” IT Professional, vol. 8, Aug. 2006, pp. 29-36.

[162] X. Ge, R. F. Paige, F. Polack, and P. Brooke, “Extreme Pro-gramming Security Practices,” in Agile Processes in Software En-gineering and Extreme Programming, vol. 4536, G. Concas, E. Damiani, M. Scotto, and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 226-230.

[163] S. Berczuk, “Back to Basics: The Role of Agile Principles in Suc-cess with an Distributed Scrum Team,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2007, pp. 382-388.

[164] K Peterson (2010) An Empirical Study of Lead-Times in Incre-mental and Agile Software Development. In International Con-ference on Software Process, ICSP 2010, Paderborn, pp. 35-356.

[165] B. George and L. Williams, “An initial investigation of test driven development in industry,” Proceedings of the 2003 ACM symposium on Applied computing, New York, NY, USA: ACM, 2003, p. 1135–1139.

[166] Ramachandran V., Shukla A.; Circle of Life, Spiral of Death: are XP Teams Following the Essential Practices? in Extreme Pro-gramming and Agile Methods –XP/Agile Universe 2002; LNCS 2418, pp. 166-173; Springer, 2002.

[167] S. Andrzeevski, “Experience Report ‘Offshore XP for PDA de-velopment’,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2007, pp. 376-381.

[168] A. Elshamy and A. Elssamadisy, “Applying Agile to Large Projects: New Agile Software Development Practices for Large Projects,” in Agile Processes in Software Engineering and Extreme Programming, vol. 4536, G. Concas, E. Damiani, M. Scotto, and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007, pp. 46-53.

[169] B. Ramesh, L. Cao, K. Mohan, and P. Xu, “Can distributed software development be agile?,” Communications of the ACM, vol. 49, p. 41–46, Oct. 2006.

[170] A. Kornstädt and J. Sauer, “Mastering dual-shore development: the tools and materials approach adapted to agile offshoring,” in Proceedings of the 1st international conference on Software engi-neering approaches for offshore and outsourced development, Berlin, Heidelberg, 2007, p. 83–95.

[171] R. Benefield, “Seven Dimensions of Agile Maturity in the Glob-al Enterprise: A Case Study,” Hawaii International Conference on System Sciences, Los Alamitos, CA, USA: IEEE Computer Society, 2010, pp. 1-7.

[172] A. Danait, “Agile Offshore Techniques - A Case Study,” Agile Development Conference/Australasian Database Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2005, pp. 214-217.

[173] M.-W. Chung, S. Nugroho, and J.F. ?JF? Unson, “Tidal Wave: The Games Transformation,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2008, pp. 102-105.

[174] T. Smith, “Product Innovation is Practical, Important, and Pos-sible,” AGILE Conference, Los Alamitos, CA, USA: IEEE Com-puter Society, 2008, pp. 561-565.

[175] R. Wijnands and I. Van Dijk, “Multi-tasking agile projects: the pressure tank,” Proceedings of the 8th international conference on Agile processes in software engineering and extreme pro-gramming, Berlin, Heidelberg: Springer-Verlag, 2007, p. 231–234.

[176] M. Yap, “Value Based Extreme Programming,” AGILE Confer-ence, Los Alamitos, CA, USA: IEEE Computer Society, 2006, pp. 175-184.

[177] B Fitzgerald, G Hartnett, K Conboy (2006) Customising agile methods to software practices at Intel Shannon. European Jounal of Information Systems 15(2):200-213.

[178] L. Williams, W. Krebs, L. Layman, A. Antón, P. Abrahamsson, Toward a Framework for Evaluating Extreme Programming, presented at Empirical Assessment in Software Eng. (EASE) 2004, Edinburgh, Scot., 2004.

[179] J. Wäyrynen, M. Bodén, and G. Boström, "Security Engineering and eXtreme Programming: an Impossible marriage?," in Ex-treme programming and agile methodsXP/Agile Universe 2004, C. Zannier, H. Erdogmus, and L. Lindstrom, Eds. LNSC3134, Berlin: Springer-Verlag, 2004.

[180] S. Lee and H.-S. Yong, “Distributed agile: project management in a global environment,” Empirical Software Engineering, vol. 15, 2009, pp. 204-217.

[181] Visconti, M. & Cook, C. R. (2004). An Ideal Process Model for Agile Methods. Proceedings of 5th International Conference on Product Focused Software Process Improvement PROFES 2004, Lecture Notes in Computer Science, 3009, 431-441.

[182] MARY BETH SMRTIC, GEORGES GRINSTEINA Case Study in the Use of ExtremeProgramming in an Academic Environment Extreme Programming and Agile Methods - XP/Agile Universe 2004, Pages175-182.

[183] H. Sharp and H. Robinson, “An Ethnographic Study of XP Practice,” Empirical Software Engineering, vol. 9, 2004, pp. 353-375.

[184] Middleton PJ, “Lean software process,” Jun. 2010. [185] O. Salo, K. Kolehmainen, P. Kyllönen, J. Löthman, S. Salmijärvi,

and P. Abrahamsson, "Self-Adaptability of Agile Software Pro-cesses: A Case Study on PostIteration Workshops," 5th Interna-tional Conference on Extreme Programming and Agile Process-es in Software Engineering (XP 2004), GarmischPartenkirchen, Germany, 2004.

[186] Rogers, O. “Acceptance Testing vs. Unit Testing: A Developer’s Perspective”, Lecture Notes in Computer Science, Vol. 3134, Springer-Verlag: 22 – 31, 2004.

[187] O. Salo, “Improving Software Process in Agile Software Devel-opment Projects: Results from Two XP Case Studies,” EU-ROMICRO Conference, Los Alamitos, CA, USA: IEEE Comput-er Society, 2004, pp. 310-317.

[188] Rogers, O., Scaling Continuous Integration, presented in XP 2004, (2004).

[189] H. Robinson, H. Sharp, The characteristics of XP teams, in: Pro-ceedings of XP2004 Germany, 2004, pp. 139–147.

[190] C. Poole, "Distributed product development using extreme programming," in XP2004, GarmischPartenkirchen, Germany. pp. 60-67.

[191] A. Parrish, R. Smith, D. Hale, and J. Hale, “A field study of developer pairs: productivity impacts and implications,” Soft-ware, IEEE, vol. 21, Oct. 2004, pp. 76- 79.

[192] M.F. Nisar and T. Hameed, “Agile methods handling offshore software development issues,” Dec. 2004, pp. 417- 422.

[193] M. Huo, J. Verner, L. Zhu, and M.A. Babar, “Software Quality and Agile Methods,” Computer Software and Applications Conference, Annual International, Los Alamitos, CA, USA: IEEE Computer Society, 2004, pp. 520-525.

[194] P. Baheti, E. Gehringer, and D. Scotts. Exploring the efficacy of Distributed Pair Programming. In Proceedings of Extreme Pro-gramming and Agile Methods . XP/Agile Universe 2002, pages 208.220, Chicago, Illinois, USA, 2002.

[195] Melnik, G., & Maurer, F. (2004). Introducing Agile Methods: Three Years of Experience. Paper presented at the 30th EU-ROMICRO Conference, Rennes, France.

[196] Jokela and P. Abrahamsson, Usability assessment of an extreme programming project: close co-operation with the customer does not equal to good usability, Profes 2004 (2004).

[197] D. McKinney and L.F. Denton, “Affective assessment of team skills in agile CS1 labs: the good, the bad, and the ugly,” ACM SIGCSE Bulletin, New York, NY, USA: ACM, 2005, p. 465–469.

[198] K. Mannaro, M. Melis and M. Marchesi, Empirical Analysis on the Satisfaction of IT Employees Comparing XP Practices with Other Software Development Methodologies, Lecture Notes in Computer Science (2004), pp. 166–174.

[199] W. Ambu and F. Gianneschi, “Extreme Programming at Work,” in Extreme Programming and Agile Processes in Software Engineer-ing, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidel-berg: Springer Berlin Heidelberg, 2003, pp. 347-350.

[200] G. Luck, “Subclassing XP: breaking its rules the right way,” Jun. 2004, pp. 114- 119.

[201] I. Lokpo, M. Babri, and G. Padiou, “Assistance for Supporting XP Test Practices in a Distributed CSCW Environment,” in Ex-treme Programming and Agile Processes in Software Engineering, vol. 3092, J. Eckstein and H. Baumeister, Eds. Berlin, Heidel-berg: Springer Berlin Heidelberg, 2004, pp. 262-265.

[202] P. Becker-Pechau, H. Breitling, M. Lippert, and A. Schmolitzky, “Teaching Team Work: An Extreme Week for First-Year Pro-

42

grammers,” in Extreme Programming and Agile Processes in Soft-ware Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Ber-lin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 386-393.

[203] B.S. Boelsterli, “Iteration Advocate/Iteration Transition Meet-ing: Small Sampling of New agile Techniques Used at a Major Telecommunications Firm,” Agile Development Confer-ence/Australasian Database Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2003, p. 109.

[204] A. Law and A. Ho, “A study case: evolution of co-location and planning strategy,” Jun. 2004, pp. 56- 62.

[205] L. Cao, K. Mohan, P. Xu, and B. Ramesh, “How Extreme Does Extreme Programming Have to Be? Adapting XP Practices to Large-Scale Projects,” Hawaii International Conference on Sys-tem Sciences, Los Alamitos, CA, USA: IEEE Computer Society, 2004, p. 30083c.

[206] Kuranuki Y., Hiranabe K.: Antipractices: AntiPatterns for XP Practices. Agile Development Conference (2004).

[207] Koskela, J. and Abrahamsson, P. On-Site Customer in an XP Project: Empirical Results from a Case Study. In Proceedings 1th European Conference on Software Process Improvements (EuroSPI 2004) (Trondheim, Norway, November 10-12, 2004), 1-11.

[208] Korkala, M., Abrahamsson, P.: Extreme Programming: Reas-sessing the Requirements Management Process for an Offsite Customer. In: Proceedings of the European Software Process Improvement Conference EUROSPI 2004, Springer Verlag LNCS Series (2004).

[209] T. Bozheva, “Practical Aspects of XP Practices,” in Extreme Pro-gramming and Agile Processes in Software Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 360-362.

[210] J. Koskela, P. Abrahamsson, and J. Takalo, “Improving Re-quirements Management in Extreme Programming with Tool Support - An Improvement Attempt that Failed,” EUROMI-CRO Conference, Los Alamitos, CA, USA: IEEE Computer So-ciety, 2004, pp. 342-351.

[211] P. Hodgetts, “Refactoring the development process: experiences with the incremental adoption of agile practices,” Jun. 2004, pp. 106- 113.

[212] C. Ho, K. Slaten, L. Williams, and S. Berenson, “Work in pro-gress-unexpected student outcome from collaborative agile software development practices and paired programming in a software engineering course,” Oct. 2004, pp. F2C- 15-16 Vol. 2.

[213] B. Greene, “Agile Methods Applied to Embedded Firmware Development,” Agile Development Conference/Australasian Database Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2004, pp. 71-77.

[214] R. W. Miller, “When Pairs Disagree, 1-2-3,” in Extreme Pro-gramming and Agile Methods — XP/Agile Universe 2002, vol. 2418, D. Wells and L. Williams, Eds. Berlin, Heidelberg: Springer Ber-lin Heidelberg, 2002, pp. 231-236.

[215] J. Vanhanen, J. Jartti, and T. Kähkönen, “Practical Experiences of Agility in the Telecom Industry,” in Extreme Programming and Agile Processes in Software Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidel-berg, 2003, pp. 279-287.

[216] FAVELA, J., et al., 2004, “Empirical Evaluation of Collaborative Support for Distributed Pair Programming”, In: International Workshop on Groupware, pp.215-222, Lecture Notes in Com-puter Science, San Jose, Costa Rica, set.

[217] B. Tessem, “Experiences in Learning XP Practices: A Qualitative Study,” in Extreme Programming and Agile Processes in Software Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 131-137.

[218] B. Ramesh, with Lan Cao, “Agile Requirements Engineering Practices: An Empirical Study,” Software, IEEE, vol. 25, Feb. 2008, pp. 60-67.

[219] S. Bryant, “Double Trouble: Mixing Qualitative and Quantita-tive Methods in the Study of eXtreme Programmers,” Visual Languages and Human-Centric Computing, IEEE Symposium on, Los Alamitos, CA, USA: IEEE Computer Society, 2004, pp. 55-61.

[220] G. Broza, “Adapting Extreme Programming to Research, De-velopment and Production Environments,” in Extreme Pro-gramming and Agile Methods - XP/Agile Universe 2004, vol. 3134, C. Zannier, H. Erdogmus, and L. Lindstrom, Eds. Berlin, Hei-delberg: Springer Berlin Heidelberg, 2004, pp. 139-146.

[221] E. Bos and C. Vriens, "An agile CMM," In proceedings of 4th Conference on Extreme Programming and Agile Methods XP/ Agile Universe, pp. 129138, 2004.

[222] Aveling, B., 2004. XP lite considered harmful? In: Proceedings of the Fifth International Conference of Extreme Programming and Agile Processes in Software Engineering, LNCS 3092, Springer, pp. 94–103.

[223] S. Atsuta and S. Matsuura, “eXtreme Programming support tool in distributed environment,” vol. 2, pp. 32- 33 vol.2, Sep. 2004.

[224] T. Dingsøyr and G. K. Hanssen, “Extending Agile Methods: Postmortem Reviews as Extended Feedback,” in Advances in Learning Software Organizations, vol. 2640, S. Henninger and F. Maurer, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 4-12.

[225] Y. Dubinsky and O. Hazzan, “Using Metaphors in eXtreme Programming Projects,” in Extreme Programming and Agile Pro-cesses in Software Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 420-421.

[226] P. Amey and R. Chapman, “Static verification and extreme programming,” ACM SIGAda Ada Letters, New York, NY, USA: ACM, 2003, p. 4–9.

[227] P. Abrahamsson and J. Koskela, “Extreme Programming: A Survey of Empirical Data from a Controlled Case Study,” Em-pirical Software Engineering, International Symposium on, Los Alamitos, CA, USA: IEEE Computer Society, 2004, pp. 73-82.

[228] R. Juric, “Extreme programming and its development practic-es,” Jun. 2000, pp. 97- 104.

[229] J. Kivi, D. Haydon, J. Hayes, R. Schneider, and G. Succi, “Ex-treme programming: a university team design experience,” vol. 2, 2000, pp. 816-820 vol.2.

[230] J. Grenning, “Launching extreme programming at a process-intensive company,” Software, IEEE, vol. 18, no. 6, pp. 27-33, Dec. 2001.

[231] M.M. Müller and W.F. Tichy, “Case study: extreme program-ming in a university environment,” Proceedings of the 23rd In-ternational Conference on Software Engineering, Washington, DC, USA: IEEE Computer Society, 2001, p. 537–544.

[232] J. Nawrocki, B. Walter, and A. Wojciechowski, “Toward ma-turity model for extreme programming,” pp. 233-239, 2001.

[233] M.C. Paulk, “Extreme Programming from a CMM Perspec-tive,” IEEE Software, vol. 18, 2001, pp. 19-26.

[234] C.J. Poole, T. Murphy, J.W. Huisman, and A. Higgins, “Extreme maintenance,” 2001, pp. 301-309.

[235] Succi, G., M. Stefanovic, and W. Pedrycz. Quantitative Assess-ment of Extreme Programming Practices. in Canadian Confer-ence on Electrical and Computer Engineering. 2001. Toronto, Ontario, Canada.

[236] A.V. Deursen, “Program Comprehension Risks and Opportuni-ties in Extreme Programming,” Reverse Engineering, Working Conference on, Los Alamitos, CA, USA: IEEE Computer Socie-ty, 2001, p. 176.

[237] H. Erdogmus and L. Williams, “The Economics of Software Development by Pair Programmers,” The Engineering Econo-mist: A Journal Devoted to the Problems of Capital Investment, vol. 48, 2003, p. 283.

[238] J. B. Fenwick, “Adapting XP to an Academic Environment by Phasing-In Practices,” in Extreme Programming and Agile Methods - XP/Agile Universe 2003, vol. 2753, F. Maurer and D. Wells, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 162-171.

[239] B. George and L. Williams, “An initial investigation of test driven development in industry,” Proceedings of the 2003 ACM symposium on Applied computing, New York, NY, USA: ACM, 2003, p. 1135–1139.

[240] A. Greca, V. Jovanovic, and J. Harris, “Enhancing learning suc-cess in the introductory programming course,” vol. 1, Nov. 2003, pp. T4C- 15-21 Vol.1.

[241] S. Bowen and F. Maurer, “Process support and knowledge management for virtual teams doing agile software develop-ment,” 2002, pp. 1118- 1120.

[242] O. Hazzan and Y. Dubinsky, “Bridging Cognitive and Social Chasms in Software Development Using Extreme Program-ming,” in Extreme Programming and Agile Processes in Software Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 47-53.

[243] D. Karlström and P. Runeson, “Decision support for extreme programming introduction and practice selection,” Proceedings


of the 14th international conference on Software engineering and knowledge engineering, New York, NY, USA: ACM, 2002, p. 835–841.

[244] F. Maurer, “Supporting Distributed Extreme Programming,” Proceedings of the Second XP Universe and First Agile Universe Conference on Extreme Programming and Agile Methods - XP/Agile Universe 2002, p. 13–22, 2002.

[245] F. Maurer, S. Martel, Extreme programming: rapid develop-ment for web-based applications, IEEE Internet Computing 6 2002) 86–90.

[246] G. Melnik and F. Maurer, “Perceptions of Agile Practices: A Student Survey,” Extreme Programming/Agile Universe, Chi-cago, IL, 2002.

[247] K.R. Howard, “The Covert Agilist,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 131-134.

[248] S.J. Khalaf and K.A. Maria, “An Empirical Study of XP: The Case of Jordan,” Information and Multimedia Technology, International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 380-383.

[249] S. Stolberg, “Enabling Agile Testing through Continuous Inte-gration,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 369-374.

[250] S.H. VanderLeest and A. Buter, “Escape the waterfall: Agile for aerospace,” Oct. 2009, pp. 6.D.3-1-6.D.3-16.

[251] O. Hazzan and J. Tomayko, “The Reflective Practitioner Per-spective in eXtreme Programming,” in Extreme Programming and Agile Methods - XP/Agile Universe 2003, vol. 2753, F. Maurer and D. Wells, Eds. Berlin, Heidelberg: Springer Berlin Heidel-berg, 2003, pp. 51-61.

[252] G. Hedin, L. Bendix, and B. Magnusson, “Introducing Software Engineering by means of Extreme Programming,” Software Engineering, International Conference on, Los Alamitos, CA, USA: IEEE Computer Society, 2003, p. 586.

[253] A.M. Figueiredo, “An Executive Scrum Team,” AGILE Confer-ence, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 147-150.

[254] P. Ingalls and T. Frever, “Growing an Agile Culture from Value Seeds,” AGILE Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 119-124.

[255] B. Jensen and A. Zilmer, “Cross-Continent Development Using Scrum and XP,” in Extreme Programming and Agile Processes in Software Engineering, vol. 2675, M. Marchesi and G. Succi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2003, pp. 146-153.

[256] J.H. Hayes, A. Dekhtyar, and D.S. Janzen, “Towards traceable test-driven development,” Traceability in Emerging Forms of Software Engineering, ICSE Workshop on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 26-30.

[257] Y. Zhou, “UniX Process, Merging Unified Process and Extreme Programming to Benefit Software Development Practice,” Edu-cation Technology and Computer Science, International Work-shop on, Los Alamitos, CA, USA: IEEE Computer Society, 2009, pp. 699-702.

[258] M. Yap, “Follow the Sun: Distributed Extreme Programming Development,” Agile Development Conference/Australasian Database Conference, Los Alamitos, CA, USA: IEEE Computer Society, 2005, pp. 218-224.

[259] O. Salo and P. Abrahamsson, “Agile methods in European em-bedded software development organisations: a survey on the actual use and usefulness of Extreme Programming and Scrum,” Software, IET, vol. 2, Feb. 2008, pp. 58-64.

[260] P. Tingling and A. Saeed, “Extreme programming in action: a longitudinal case study,” in Proceedings of the 12th international conference on Human-computer interaction: interaction design and usability, Berlin, Heidelberg, 2007, p. 242–251.

[261] M. Pikkarainen, J. Haikara, O. Salo, P. Abrahamsson, and J. Still, “The impact of agile practices on communication in soft-ware development,” Empirical Software Engineering, vol. 13, no. 3, pp. 303-337, May. 2008.

[262] J. Rasmussen, “Introducing XP into Greenfield Projects: lessons learned,” Software, IEEE, vol. 20, no. 3, pp. 21- 28, Jun. 2003.

[263] J. Vanhanen, J. Itkonen, and P. Sulonen, “Improving the Inter-face Between Business and Product Development Using Agile Practices and the Cycles of Control Framework,” in Agile Devel-opment Conference/Australasian Database Conference, Los Alami-tos, CA, USA, 2003, vol. 0, p. 71.

[264] A. N. Bowers, R. S. Sangwan, and C. J. Neill, “Adoption of XP practices in the industry—A survey,” Software Pro-cess: Improvement and Practice, vol. 12, no. 3, pp. 283-294, May. 2007.

[265] O. Murru, R. Deias, and G. Mugheddue, “Assessing XP at a European Internet company,” Software, IEEE, vol. 20, Jun. 2003, pp. 37- 43.

[266] B.'Victor'and'N.'Jacobson,'“We'Didn’t'Quite'Get'It,”'Aug.'2009,'pp.'271G274.'

'

''

''''

44

APPENDIX A OVERVIEW OF STUDIES: EXTENDED MATERIAL Total primary studies: 186

Context: Industry Context: Academia Empirical 114 35

Non-empirical 5 32 Empirical papers in Industry Agile Experience

Agile experience Nr. % Beginner 38 33

Experienced (Mature) 48 42 N/A 28 24

Company size

Company size Nr. % Micro 10 9 Small 15 13

Medium 12 11 Large 31 27 N/A 46 40

Factors contributing to rigor for the 119 in Industry Context described Design described Validity discussed 0 3 11 65 1 95 72 42 2 21 36 12 Factors contributing to rigor for the 67 in Academia Context described Design described Validity discussed 0 7 12 37 1 41 42 22 2 19 13 8 Factors contributing to rigor for all the primary studies Context described Design described Validity discussed 0 10 23 102 1 136 114 67 2 40 49 20


APPENDIX B OTHER PRACTICES

Practice Category Ref Definition

Big Picture Up-Front Design [82] A short period of envisioning amongst the business stake-holders and project team to set the direction of the project.

Programmer On-site

Developer

[82]

Schedule site visits for programmers so that programmers can understand more about the end-users of the software.

Re-Calibration Management [82]

Plan to adjust commitments and resources regularly based on what both customers and developers learn during the iterations.

Road show

Customer collaboration [82]

Demonstrate the software to end users and other stake-holders so that the team can obtain feedback on the direc-tion of the project.

Program Weak Points

Developer [100]

Program Weak Points identified in terms of coupling and cohesion factors. The locations in the source code where coupling can be decreased and cohesion can be increased are also identified. The output of this phase will be program points to be refactored in the object oriented code that has to be refactored.

Culture of trust: Developer [102] Building the right teams is another centripetal force that in-

volves building trust, communication and personal bridges.

Just in time design Design [102]

As mentioned earlier, “thinking light” about the design was one of the hardest things that the team had to confront. There were strong and passionate opinions about the opti-mal way of designing the application. Often the disagree-ments are due to the variety of perspectives that developers brought to the team (e.g. data model, security, and usability perspectives).

User Backlog .


First, we prioritized the user groups based on their immedi-ate needs and internal politics. The evolving system would be rolled out to one user group at a time. Generally each user group would get one or two releases. In essence, we developed a “user backlog”.

Dirty laundry meet-ings

Management [116]

The customer decided to hold a meeting in which everyone was supposed to resolve their conflicts. This meeting was called “dirty laundry meeting” because it was a chance for everyone to say what was on their mind about others and walk away with a clean slate.

Specialists and study time

Developer [116]

The specialists brought some fresh air into the team and reduced the burden of everyone having to study all new technologies. They did not have special rights to stories in their areas. In fact they were discouraged from taking these stories at all. They were available to pair program when someone had trouble in their areas of research and also conducted seminars to teach the rest of the team what they were learning.

Early Prototyp-ing/Spiking Developer [124]

Prototyping/spiking to be an effective practice for reducing risk. However, we noticed that this requires that the teams be provided with guidelines and templates on the process of identifying the cause of risks and focus the goals of the pro-totypes/spikes to minimize them. As a consequence, the teams were much more confident that they will succeed in the project, hence substantially increasing their motivation and level of participation.

Risk management Developer [129] We started to focus the team’s attention to both identifying

the risks as well as categorizing the risks based on causality

46

(e.g. knowledge gap, skill gap, unstable library etc.). Once the risks were identified, prioritized, and potential causes for them established, the teams developed spikes, each of which aimed at minimizing a set of risks.

Accepting working software.

Customer colalboration [135]

After several iterations, the working software is almost done, and then developers can submit it to customers for usage test or trial operation. Customer accept this current release if it accords with the agreement in release plan.

Requirement man-agement.


Customer represents new requirements at the beginning of eachiteration, and confirms priority levels of them. In ar-ranged meeting, customer represents requirements or amends requirements. And, customers themselves decide whether they should submit some important changes.

Serving as consult-ant.


There are feedbacks in release plans, iteration plans, usage test, as a complementary, customer should answer any de-veloper’s questions in design, coding, and unit test phases.

Taking part in usage testing and trial op-eration.


Customer must provide test cases based on real environ-ment, join the integration test and operate current released software. To expose more software errors, released soft-ware should be run in a limited scope by some customer departments, and operation report should be provided by customer coach as quick as possible.

Checking and ac-cepting final soft-ware product.


Customer team inspects whether the final software product to meet customer’s requirement for check and accept, and then confirm the accept time with developers.

Using Nearshore Resources

.

Management [136]

Outsourcing in general and on an agile project in particular requires a lot of trust between client and vendor, near shore resources onsite (or at least someone who can be contacted in the same time zone) that could meet regularly, face to face, were crucial to building trust quickly. A technical lead and Scrum Master were used to maintain both technical and organizational lines of communication open

Direct user involve-ment


Rely on frequent on-site customer visits and direct user in-tervention techniques to identify any possible changes trig-gered by customers.

Object models Design [143]

Scrum’s main documentation consists of product and spring backlog lists. FDD and Crystal methodologies use UML dia-grams such as use cases, class diagrams, and object mod-els to document the design. Test cases have also been used by XP and Crystal methodologies as documentation artifacts.

Personas Developer [143]

User scenarios are developed for features in subsequent Sprints. Designs for features engineering plans to implement in the first Sprint are also created and tested

Customer implanta-tion

.


High-speed development teams often enlist customers or users either full time or temporarily, assigning them as Part of the team, a practice known as customer implantation. As developers construct the software from informally specified requirements and customer-developer discussions, custom-ers prioritize requirements and redirect development, re-lease-by-release.

Production prototyp-ing

Developer [161] Production prototypes are efficient enough to support live

operation.

Programmer Holiday

Customer collaboration [82] [150]

An iteration of technical tasks so that the customer can have some time to think-ahead. A “programmer holiday” is not often an actual holiday (although there maybe times when


that is indeed appropriate), but mostly the customer will choose to prioritise technical debt, lower priority bugs, tech-nical system upgrades and such above stories.

Common Source Developer [190]

Although mentioned last this is one of the first things that a distributed development team should get right. It includes not only the idea that everyone regardless of location has access to a common source base but that they have access to a common set of automation tools and processes. If you can’t build your entire product down to the packaging from a common source base and use a common set of automated testing and production tools then you are going to have a much harder time getting anything out the door.

Identifying and managing develop-ers.

Management [206] [218]

Hiring process is to be highly related to the success of its agile practices. Developers who can recognize the im-portance of pattern-based development and providing standard interfaces across the system are seen as key to successfully projects. developers’ knowledge about archi-tectural design and design patterns is seen as critical. Ap-proximately only 15% of the applicants are considered to be qualified for this project environment. Upholding developer morale is also seen as key to successful project outcomes

Architect practice .

Design [220]

The group follows an “architect” practice While the develop-ers take part in every product’s entire life-cycle, some tasks are centralized in the hands of the architect. he is responsi-ble for the vision and overall design of Informatics products, tracking design decisions, constraints and prospective de-velopment. He evaluates feature and change requests in light of existing and suggested designs

Use Cases

Design [229]

Use cases are a very useful way to capture customer re-quirements to ensure that both the customer and theteam understand the functionality. From the information gathered in that meeting and using UML, we were able to write out a set of use cases. These use cases were then further refined through several customer meetings to ensure that all the requirements were captured. At the start of each mini deliv-erable, (see incremental) we then sat down and wrote spe-cific cases for the one section.

Queue Visibility Across Distributed Sites

Progress [234] The use of an on-line queue that is available globally across our development and customer services sites. Ex: webcams.

Integration log: Progress [248]

To see how frequently new pieces of code are integrated with the system (XP suggests to have several integrations per day.

Version manage-ment system.

Developer [248] It supports collective code ownership and continuous inte-

gration.

Trainings Management [251]

To face the challenging environment offered by fast chang-ing technological scenarios, it is necessary for an Organiza-tion to have its people up to date with the latest tools and technologies. Especially for organization practicing agile method of development training of resources is essential for keeping the development process up to the mark, satisfying customers and business expectations. Training of profes-sionals always has a positive and significant impact on productivity.

Domain Presenta-tion Design [261] Systems Analysts, User Experience and Architecture team

representatives would provide the Development and Sys-

48

tems Test teams the iteration Domain Specification as well as a quick summary of the artifact they were to implement in the next iteration. Before the meeting, the User Experience resource would post the wireframes on the wall.

Heart Beat Progress [261]

A weekly ‘heart beat’ was instituted. just because heartbeats were weekly,this did not imply all project teams conducted weekly iterations. Rather, the heartbeat offered project teams a point in time and a place (ITM) from which they were able to push their deliverables up the chain whether this is specifications or code.

SIP updates Progress [261]

This was an opportunity for any representative of the cross-functional team to propose updates to the SIP as executed by the release manager.

Time out . Developer [252]

For any kind of problem that occurs in the team, if the coach cannot solve it easily and directly, a good way of tackling the problem is to take a TimeOut, i.e., to bring the team together for a brief stand-up meeting, and let the team solve the problem together. This works for technical problems, where someone on the team might have skills in a certain area, or where the team can brainstorm possible solutions or ideas for experimental spikes for solving the problem. A TimeOut is also useful for process problems where the team needs to agree on what practices to follow

Estimates:

Progress [255] During the sprint (Scrum name for an iteration), estimates are updated daily, thus allowing assessment of whether the team is on track or not

Proactive resource management

Management [94] [101] Resource management” helps ensure that a Scrum team has the necessary tools and skills to support their Scrum Practices in distributed settings.

Synchronized work hours:

Management [94] [101]

This practice is widely used by Scrum teams to ensure syn-chronous communication among distributed sites can be arranged. This is done by adjusting working hours, working from home, working long hours Some Scrum teams used strategies to avoid the need of increased overlap time. For example, a Scrum team used strict timeboxed meeting to avoid late night meeting at some sites. To make the meet-ings short and effective, team members post their three daily Scrum questions or develop backlog (feature list) before attending the distributed meetings


APPENDIX C OTHER EMPIRICAL STUDIES IN ACADEMIA Article Description [97] Web service testing to support TDD in SOA applications [165] This report reports the results of experiment conducted on using TDD taking practitioners as sub-

jects. Although TDD practice produced high quality code, passing more test cases, it took 16% more time for development.

Article Description [83] Results indicate that pairs performed at the level above the second best performers and best per-

formers of nominal pairs in each pair. There is no sufficient analysis to claim that the performance of collaborating pairs exceeds the performance of its best member working alone.

[93] Presents XPairtise, a plug-in for Eclipse that supports PP in a distributed setting. [118] The report presents an experiment to compare productivity of a solo programmer and pair pro-

grammer. Results show that PP decreases the effort of the single programmer. [119] The report describes the experiment report of extending PP to software design called 'pair design-

ing'. Results show that ‘pair designing’ is an efficient means to enforce design knowledge. [148] The paper reports an experiment to provide empirical evidence on the impact of pair program-

ming on both, thoroughness and e%ectiveness of test suites. However, the results do not show a positive impact on testing to make it more thorough (Branch coverage and mutation core indica-tor).

[154] The paper presents a controlled experiment called repeat-programming which can facilitate the understandings of relationships between human experience and programming productivity in the context of PP. Results show that PP is more efficient when two individuals are new to pro-gramming tasks, dealing with complex and callenging code. Also, in working on sloutions that individuals have already experienced PP is less efficient than solo programming. Results also show that novice-novice pairs against novice programmers are much more efficient than expert-expert pairs against expert solos.

[212] The focus of this report was in the context of females and PP. Female students felt that PP can decrease pressure while coding and increase confidence and courage. On the negative side, bad experiences in PP gave negative feelings on the practice of PP. It was mainly when they were paired with male students from whom they did not get major help/support.

[216] The paper reports an empirical evaluation of the COPPER collaborative editior to support distrib-uted PP. Results who that distributed pairs could identify same mount of errors as their collocat-ed counterparts, however no evidence found that pairs using collaborative services has better code comprehension.

[223] The paper reports evaluation of support for distributed PP. The tool aims to collect work related information when using PP which in turn can be used for improvement in project management and developers’ work conditions.

[256] The paper reports an experiemet to evaluate PP in the context of distributed development. The results indicate that PP in a distributed context is comparable to collocated pairing in terms of quality and productivity. It was found that PP in a distributed setting fosters teamwork and communication.

Article Description [100] A method to use Re practice is proposed and evaluated. The evaluation of Re is based on quantita-

tive metrics – Cyclometric complexity, LOC, CBO, LCOM, MHF and AHF. Findings show that resulting code is simpler, increase in system comprehensibility, decrease in dependencies classes, thus making the code more reusable.

[125] The report investigates if Re improves reusability of object-oriented classes. The results show that Re enhances both quality and reusability of the code.

[141] The paper introduces two practices: Continuous Architectural Refactoring (CAR) and Real Archi-tecture Qualification (RAQ) in order to empower XP's development process toward improving systems' Architecture.

[147] The paper reports storytest-driven approach as a complimentary practice to TDD, to manage agile project requirements.

[168] The paper provides students’ experience in using Me. It was found that Me leads to a more confi-

50

dent understanding of the system design. ''''''''


APPENDIX D PRACTICE MODIFICATIONS/ADAPTATIONS '

Documentation Practices Adapted Citation

Data and Documentation [219] Data integration [219] Documentation including architec-tural description

[227]

Just right documentation [192] Reduced documentation [251] Reflection and documentation [252]

Right documentation [263] Streamlined documentation [219]

''

Iterations Practices Adapted Citations Cycle of innovation consisting of 5 itera-tions: Investigation, Identification, Aggre-gation, Consolidation and Post-validation

[140]

First iteration [252]

Incremental deliverables [229] Incremental development

[279] Iteration plans: each iteration - 1 month, each release - 3 iterations

[73]

Iteration retrospectives [264] Iteration review [129] iteration specification [261] Iteration transision meetings [261] Iterations - 4 week length, planning in the 1st week and demonstration in the 4th week

[104]

Iterations (3 weeks) [87] periodic performance reviews [219] Pre-project iteration [134] Release and iteration Planning: release length - 2 months, 3 months, 2 week iterations

[90]

Review of iterations with the core team [122]

Small iterations [238] Small releases: iterations 2-3 days to 2-3 weeks

[79]

Small version of iterative, evolutionary development approach

[99]

Software iteration plans [261] Using long iterations [129] Iteration planning [90] [129]

[210] [231]

Iterations - 2 weeks [108] [266]

Iterations - 3 weeks [158] [136]

Iterations - 1 week [75] [174]

Small releases: iterations of 30-45 days [113] [132]

''

Office&Practices'Adapted& Citation'

Community'workstations' [234]'Honey'comb'cubicles' [204]'Management'support'and'organizaGtional'culture'

[205]'

Open'office'layout'' [98]'Open'workspace' [210]'Personal'vs'team'space' [234]'Shared'working'space'' [158]',[253]'

Single'room'' [72]'Small'offices'(150'to'200'people)'' [98]'

War'room''

[204]'

'Pair&programming&Practices'Adapted& Citation'

Daily'meetings'for'forming'pairs'' [134]'

Flexible'pair'programming' [218]'

Pair'courage' [212]'Pair'pressure' [212]'Pair'review' [212]'Pairing' [234]'Pairing'in'all'high'risk'activities' [216]'

Pairing'prison' [210]'PP' (practiced' in' the' beginning' and'later'abandoned)''

[132]'

52

PP'via'web'conference'' [172]'Programming'speed' [248]'Role'switching'2G3'times'a'day'' [134]'

work' in'pairs' in'all'high'risk'activiGties'

[219]'

Teaching'–'additional'roleG'spectator''

[93]'

'Planning&(&&Planing&game)&

Practices'Adapted& Citation'

Adaptive'cycle'planning'' [143]'Adaptive'planning'' [96]'Aggregate'product'plan'' [112]'Bi'weekly'planning' [204]'Big'picture'UpGFront'' [193]'Designing'upfront' [205]'['218]'

Group'planning'meeting'' [108]'Impact' of' upfront' design' on' ' other'practices'

[205]'

Milestone'and'iteration'planning' [264]'

Monthly'planning' [204]'Planning'cycle' [204]'Planning'meeting'breakouts'' [108]'

Planning'meetings'' [108]'[139]'[122]'

Programming/' pre' release' evaluaGtion'

[195]'

Project'Outline'' [143]'Release'planning' [210]'

Release' planning' (1' hour,' as' needGed):'create'long'term'product'plan''

[266]'

Release'system''

[238]'

Sizing'&'Planning'''

[92]'[281]'[266]'

''Practices Adapted

Citation

Controlled software spikes [117]

Design/code reviews [261] Flexible design [251] Just in time design [106] Technical excellence and Good de-sign

[75]

'Sprint

Practices Adapted Citation

2 week sprint (changed from 2 to 1) [92]

20-30 days sprint [95] 2-week sprint prior to start of develop-ment

[145]

2-week sprints [144]

2-week sprints in maintenance team [144]

30 day sprint [76] Develop unit tests for all new code dur-ing a sprint

[74]

Scrum pre-sprint planning [95] [282]

Sprint 2-week [128] [173] [281]

Sprint backlog with the tool Base camp and Excel

[142]

Sprint break after 5/6 sprints [173]

Sprint demo [143]

Sprint management [106]

Sprint planning [72] [76] [85] [95] [106] [128] [138] [144] [282]

Sprint planning is prepared separately with both sides of the team by the Product Owner. Sprint planning 2, cre-ating tasks, is performed jointly after both sides have received explanation of the user stories

[87]

Sprint planning meeting [181] [213] [253]

Sprint review [106] [213] [253]

Synchronized 4-week sprint [144]

Three and a half weeks sprint [102]

Time boxing with sprint

[164]

''''

Stories Practices Adapted Citation


Collect scenarios [238] Enhanced user stories [208] Splitting stories [234] Story cards [234]

[242] [267]

Synoptic picture [242] Task cards [234] Tracking stories [234] User stories were divided into tasks [227]

''

Team Practices Adapted Citation Build team around motivated individuals with the necessary environment, support, and trust to get job done.

[75]

Business leaders in monthly re-view meeting

[213]

Business owner given control [72] Changing role [172] Creating customer team [135]

Frequent rotation between onsite and offshore teams

[138]

Generally located management team

[94]

Generally located management team

[94]

Gradual team distribution

[94]

54

APPENDIX E MAA - CAV EXAMPLIFICATION

AP1 AP2 AP3 AP4 AP5 AP6 AP7 AP8 AP9 AP10 AP11 AP12 Agile as-sessment

value [173] Y Y Y Y Y 2.08/3 [178] Y Y Y Y Y Y Y 2.58/4 [190] Y Y Y Y Y Y Y Y 3/4 [73] Y Y Y Y Y Y Y Y 3/4

[121] Y Y Y Y Y Y Y Y Y Y 3.4/4 [127] Y Y Y Y Y Y Y 2.66/4 [218] Y Y Y Y Y Y Y 2.66/4 [213] 3.5/4 [106] Y Y Y Y Y Y Y Y 2.66/4 [207] Y Y Y Y Y Y 2.41/4 [208] Y Y Y Y Y Y 2.41/4 [210] Y Y Y Y Y Y Y Y Y Y 3/4 [102] Y Y Y Y Y Y Y 3/4 [253] Y Y Y Y Y Y Y Y 3/4 [222] Y Y Y Y Y Y Y Y Y Y Y 3.75/4 [226] Y Y Y Y Y Y Y 2.17/4 [255] Y Y Y Y Y Y 2.33/4 [87] Y Y Y Y Y Y Y Y Y 3.25/4

Finding common denominators for agile software development ...830452/FULLTEXT01.pdf · Software...

Documents

Transcript of Finding common denominators for agile software development ...830452/FULLTEXT01.pdf · Software...