arXiv:1907.06157v1 [cs.SE] 14 Jul 2019 · continuous integration and continuous delivery. However,...

Noname manuscript No.(will be inserted by the editor)

Software Development with Feature Toggles:Practices used by Practitioners

Rezvan Mahdavi-Hezaveh · JacobDremann · Laurie Williams

Received: date / Accepted: date

Abstract Background: Using feature toggles is a technique that allows de-velopers to either turn a feature on or off with a variable in a conditionalstatement. Feature toggles are increasingly used by software companies to fa-cilitate continuous integration and continuous delivery. However, using featuretoggles inappropriately may cause problems which can have a severe impact,such as code complexity, dead code, and system failure. For example, the er-roneous repurposing of an old feature toggle caused Knight Capital Group, anAmerican global financial services firm, to go bankrupt due to the implicationsof the resultant incorrect system behavior.Aim: The goal of this research project is to aid software practitioners in theuse of practices to support software development with feature toggles throughan empirical study of feature toggle practice usage by practitioners.Method: We conducted a qualitative analysis of 99 artifacts from the grey lit-erature and 10 peer-reviewed papers about feature toggles. We conducted asurvey of practitioners from 38 companies.Results: We identified 17 practices in 4 categories: Management practices, Ini-tialization practices, Implementation practices, and Clean-up practices. Weobserved that all of the survey respondents use a dedicated tool to create andmanage feature toggles in their code. Documenting feature toggle’s metadata,setting up the default value for feature toggles, and logging the changes madeon feature toggles are also frequently-observed practices.

Rezvan Mahdavi-HezavehDepartment of Computer Science, North Carolina State University, Raleigh, NC, USAE-mail: [email protected]

Jacob DremannDepartment of Computer Science, North Carolina State University, Raleigh, NC, USAE-mail: [email protected]

Laurie WilliamsDepartment of Computer Science, North Carolina State University, Raleigh, NC, USAE-mail: laurie [email protected]

arX

iv:1

907.

0615

7v2

[cs

.SE

] 1

1 M

ar 2

020

2 Rezvan Mahdavi-Hezaveh et al.

Conclusions: The feature toggle development practices discovered and enumer-ated in this work can help practitioners more effectively use feature toggles.This work can enable future mining of code repositories to automatically iden-tify feature toggle practices.

Keywords Continuous Integration · Continuous Delivery · Feature toggle ·Practice

1 Introduction

In 2012, developers in Knight Capital Group, an American global financialservices firm, updated their automated, high-speed, algorithmic router whichinadvertently repurposed a feature toggle1, activating functionality which hadbeen unused for 8 years. Within 2 minutes, developers realized the deployedcode behaved incorrectly but took 45 minutes to stop the system. During thattime, Knight Capital lost nearly 400 million dollars, which caused the groupto go bankrupt [2]. As illustrated, using feature toggles without following goodpractices can be detrimental to an organization.

Developers guard blocks of code with a variable as a feature toggle inconditional statements, and by changing the value of the variable, enable ordisable that part of the code in the system’s execution. The value of thevariable could be changed either in the code or remotely on a configurationserver. Feature toggles have similarities to configuration options. However, aswill be discussed in Section 3, feature toggles also have significant differencesto configuration options [3]. The use of feature toggles is a technique oftenused in continuous integration (CI) and continuous delivery (CD) contextsthat allows teams to incrementally integrate and test a new feature even whenthe feature is not ready to be released [4] [5]. Developers also use featuretoggles for other purposes, such as gradual roll out and experiments. However,feature toggles can turn into technical debt [6]. Using feature toggles addsmore decision points to the code which adds more complexity. This increasedcomplexity drives the need to remove toggles when their purpose is complete.

A software development practice is an activity or step carried out to achievea goal during the development of software. For example, unit testing is apractice for white-box testing of implementation code. The identification andcategorization of feature toggle practices used in industry may help softwarepractitioners to use toggles more efficiently and to control the accumulation oftechnical debt. The goal of this research project is to aid software practitionersin the use of practices to support software development with feature togglesthrough an empirical study of feature toggle practice usage by practitioners.Software practitioners prefer to learn through the experiences of other softwarepractitioners [7]. As such, our study obtains practice usage from practitioners.

We state the following research questions:

1 Feature toggles are also called feature flags, feature bits, feature flippers and featureswitches [1].

Software Development with Feature Toggles: Practices used by Practitioners 3

RQ1 (Identification): What are the feature toggle practices that software prac-titioners use?

RQ2 (Frequency): How frequently are feature toggle practices used?

To answer the first research question, using a keyword search we collectedpeer-reviewed papers and artifacts from the grey literature about feature tog-gles. We used an open coding technique [8] to perform qualitative analysison these artifacts to identify practices. We do not call the identified practices“best” practices because we did not obtain enough evidence to select any prac-tice as a “best” practice. To answer the second research question, we analyzedcompany-specific peer-reviewed papers and artifacts from the grey literatureand conducted a survey to find the frequency of usage of the identified prac-tices.

We summarize the contribution of this paper:

1. A list of 17 practices in four categories used to support software develop-ment with feature toggle; and

2. An analysis of the the frequency of usage of feature toggle practices inindustry.

The rest of the paper is organized as follows: in Section 2, we describethe background of the research area. In Section 3, we briefly describe prioracademic work related to our paper. In Section 4, we explain our researchmethodology. In Section 5, we report our findings. In Section 6, we discuss ourfindings. We enumerate the limitations of our study in Section 7. We concludeand describe future work on feature toggles in Section 8.

2 Background

In this section, we first provide briefly the definitions of CI and CD. Next, weexplain the feature toggle concept and it’s types. At the end, we describe thegrey literature and its importance for our paper.

2.1 Continuous Integration(CI) and Continuous Delivery(CD)

Companies must deliver valuable software rapidly to be competitive. This ex-pectation leads companies to use CI and CD to make development cyclesshorter. CI is a practice of integrating and automatically building and test-ing software changes to the source repository after each commit [9]. CD is apractice for keeping the software in a state such that it can be released toa production environment at any time [10]. CI/CD refers to a combinationof these two practices and enables delivering code changes frequently. Usingfeature toggles is one of the techniques that is used by numerous softwarecompanies who practice CI/CD [5].


f unc t i on Search ( ){var useNewAlgorithm = f a l s e ;i f ( useNewAlgorithm ){

r e turn newSearchAlgorithm ( ) ;} e l s e {

r e turn oldSearchAlgor ithm ( ) ;}

}

Fig. 1 An example of a feature toggle.

2.2 Feature Toggles

Programming languages have long provided the language constructs to im-plement feature toggles. However, the first use of this language construct tosupport CI/CD was at Flickr in 2009 [11]. Figure 1 is an example of a featuretoggle. In this example, the dynamic choice of a search algorithm depends onthe value of the useNewAlgorithm toggle. If the value of this toggle is true, thenthe new search algorithm is used, otherwise the Search method calls the oldsearch algorithm.

Feature toggles have been categorized into five types in software systemsin [4] and [12]:

– Release toggles: Toggles used to add new features in a trunk-based develop-ment context. In trunk-based development, all developers commit changesto one shared branch. Using release toggles in trunk-based developmentsupports CI/CD for partially-completed features [12] [4].

– Experiment toggles: Toggles used to perform experimentation on the soft-ware, such as is done by Microsoft [13] [14], to evaluate new features changesand their influence on user-observable behavior [12].

– Ops toggles: Toggles used to control the operational aspect of the systembehavior. When a new feature is deployed, system operators can disablethe feature quickly if it performs unexpectedly [12].

– Permission toggles: Toggles used to provide the appropriate functionalityto a user, e.g. special features for premium or paid users [12]. Permissiontoggles also called long-term business toggles in [4].

– Development Toggles: Toggles used for enabling or disabling certain fea-tures to test and debug code [4].

Permission toggles, ops toggles and development toggles are long-lived tog-gles based on their usage purpose in the code. Release toggles and experimenttoggles are short-lived toggles [12] [4].

2.3 Grey Literature


Grey literature is defined as “... literature that is not formally publishedin sources, such as books or journal articles” [15]. Software practitioners mayshare their experiences as grey literature which can be considered as a valu-able resource for researchers [16] and other practitioners. Academic publi-cations reflect the state-of-the-art and grey literature provide insight to thestate-of-the-practice in any research area. In practical research areas such assoftware engineering, combining the state-of-the-art and state-of-the-practiceis important to provide valuable results [17]. In the area of feature toggles,a large number of grey literature artifacts exist but only a small number ofpeer-reviewed papers have been published. As we will discuss in Step Two ofthe methodology in Section 4, we use the quality assessment checklist of greyliterature for software engineering provided in [18] to evaluate the quality ofour grey literature artifacts.3 Related Work

Rahman et al. [19] performed a qualitative grey literature study and conductedfollow-up inquiries to study continuous deployment practices. They reported 11continuous deployment practices used by 19 software companies. Using featuretoggles is one of these 11 practices that is used by 13 of the 19 companies. Inaddition, at the Continuous Deployment Summit [5] 2015, researchers andpractitioners from 10 companies shared their best practices and challenges.Parnin et al. [5] disseminated 10 best practices from the Summit, includingthe use of feature toggles to implement Dark Launches2.

To understand the drawbacks, strengths, and cost of using feature togglesin practice, Rahman et al. [4] performed a thematic analysis of videos andblog posts created by release engineers. They provided six themes foundedin analyzed videos and blog posts, such as technical debt and combinatorialfeature testing. To identify feature toggle practices, we used videos and blogposts from [4] and additional peer-reviewed papers and grey literature arti-facts including more videos and blog posts we found. Rahman et al. [4] alsoperformed a quantitative analysis of feature toggle usage across 39 releases ofGoogle Chrome from 2010 to 2015 and mined a spreadsheet used by Google de-velopers for feature toggle maintenance. Release toggles should be short-livedtoggles but Rahman et al. observed that 53% of the release toggles exist formore than 10 releases in Chrome. They classified unused but existing releasetoggles in the code as technical debt. The goal of our study is to identify fea-ture toggle practices and their usage in industry while the goal of their paperwas to understand the drawbacks, strengths, and cost of using feature toggles.

Rahman et al. [20] extracted four architectural representations of GoogleChrome: 1) conceptual architecture; 2) concrete architecture; 3) browser ref-erence architecture; and 4) feature toggle architecture. Using the extractedfeature toggle architecture, developers can find out which feature effects whichmodule and which module is affected by which feature. The goal of their study

2 Dark launching is a practice in which code is incrementally deployed into productionbut remains invisible to users [5].


was to show how developers can get a new viewpoint into the feature archi-tecture of the system using the extracted feature toggle architecture. Theirresult raise awareness of the impact of using feature toggles on the modulararchitecture of the system. In our paper, we focus on the practices of usingfeature toggles which is not in the scope of their study.

Meinicke et al. [3] explored the differences and commonalities between con-figuration options and feature toggles by conducting nine semi-structured in-terviews with feature toggle experts. Configuration options are key-value pairsused by end users to include or exclude functionality in a software system. Dur-ing the interviews, the authors asked practitioners about the existing literatureon feature toggles. Then, they discussed with the interviewees the configura-tion options topics and asked them if they saw common challenges and solu-tions. The researcher found that although feature toggles and configurationoptions are similar concepts, they have distinguishing characteristics and re-quirements. The goal of the usage and challenges of each of the techniques aredistinct. The researcher identified 10 themes for the differences, such as theirusers (The feature toggles are used by developers but configuration optionsare used by end users) and their lifetime (The feature toggles will be removedfrom code ideally but configuration options can exist permanently).

Sayagh et al. [21] aimed to understand the process required by practition-ers to aggregate configuration options in the software system, the challengesthey face, and best practices that they could follow. To achieve their goal, theauthors did 14 interviews with software engineering experts, conducted a sur-vey on Java software engineers, and did a literature review on the academicpapers in the area of configuration options. They identified 9 configurationmanagement activities, 21 configuration challenges, and 24 expert recommen-dations. One of the reported challenges, the increasing complexity of the codeby adding configuration options, is the same as the challenges of using featuretoggles. In addition, eight of the 17 practices we identified, as will be discussedin Section 5.1, have partial overlap with their reported activities, challengesrecommendations, such as using naming conventions. However, because of thedifferences identified between feature toggles and configuration options in [3],ten practices out of 17 identified practices are not mentioned in [21]. As anexample, configuration options are intended to be “permanent” in the codebut feature toggles intended to be “temporary”. This difference could explainthe absence of our identified practices in the clean-up category in Sayegh etal. [21]’s paper. As we provide each practice in Section 5.1, we will provideinformation on whether the feature toggle practice is in common with config-uration options.

None of this related work on feature toggles focus on identifying the featuretoggle practices broadly used in industry and their usage frequency. We fill thisgap in this paper.


1. Searching Initial Artifacts

2. Identification of Practices and Categories

3. Searching Company-specific

Artifacts

4. Extraction of Practice Usage

5. Follow-up Survey

6. Analysis of Usage Frequency of

Practices

Company-specific Yes

New artifactsYes

Fig. 2 The research methodology.

4 Research Methodology

We describe the steps of our methodology to answer the research questions.Our methodology has six steps, as shown in Figure 2. We started by searchingfor initial set of peer-reviewed papers and artifacts from the grey literaturerelated to our study scope, in Step One. In Step Two, we identified and cate-gorized practices found in this literature, analyze the quality of grey literatureartifacts and calculate the level of confidence for identified practices. In StepsThree and Four, we iteratively searched for grey literature related to specificcompanies and extract the usage of identified practices. Then, we sent a sur-vey to practitioners, in Step Five. Finally in Step Six, we analyzed the usagefrequency of practices and compare practices, if possible. Each of these stepsfrom Figure 2 will be explained in detail in the following sub-sections.

4.1 Step One: Searching Initial Artifacts

The first step in our research methodology in Figure 2 is to use a keywordsearch in the Google search engine to identify grey literature and in GoogleScholar to find peer-reviewed papers. We used the following search terms:‘feature toggle’; ‘feature flag’; ‘feature switch’; ‘feature flipper’; and ‘featurebit’. These search terms were obtained from Fowler’s blog post [1].

Selecting the related peer-reviewed papers and grey literature artifacts isdone by first and second author of the paper. For grey literature, reviewing allthe results of each searches in Google search engine was infeasible. The mostrelevant links are provided earlier in the Google search engine. We reviewedthe first 10 pages of the results of each search to select the most relevant links.To determine if a link is related to the scope of our research, we read thearticle by looking for the search term and read 2–3 sentences before and afterthe search term in the text. For videos, we kept all the links and analyzedthem in Step Two of the methodology. In the relevant grey literature artifacts,we used a snowballing approach [22]. We clicked on links and the references toother feature toggle resources found in the artifact, and we read the papers,articles or watched the videos. For peer-reviewed papers, we checked the titles


... We can also apply a Lean approach to reducing inventory, placing a limit on the number of feature flags a system is allowed to have at any one time. Once that limit is reached...

...They drastically reduced and limited the number of feature toggles that are allowed to be active at the same time...

Limit the number of feature toggles

...It is a good idea is to use special branches to manage the cleanup of flags. So right when you introduce a new feature flag, you create a cleanup branch that removes all the flags and submit a pull request for it...

Create a cleanup branch

Clean-up practices

Text Codes

Placing a limit on the number of feature flags

Limited the number of feature toggles

Create a cleanup branch

Practices Similarities Categories

Cleaning up feature toggles

Fig. 3 An example of using the open coding technique.

of search result and kept those which were related to our topic. We used thesnowballing approach for peer-reviewed papers. We checked the references andselected the related ones. In the rest of the paper, we use the term “artifacts”to refer to the set of “peer-reviewed papers and grey literature artifacts”. Thetime of the publishing of all collected artifacts is before June 2019. Somecollected artifacts in Step One were company-specific artifacts that were oftenwritten by a release manager or developer, referencing feature toggle usageat a specified company. We used these company-specific artifacts in Step Twoand Step Four.

4.2 Step Two: Identification of Practices and Categories

The grey literature and peer-reviewed papers found in Step One were usedto identify feature toggle practices. We analyzed the artifacts using an opencoding technique, a technique to analyze textual data by coding (i.e. labeling)concepts and identifying categories based on similarity and dissimilarity ofcodes [8]. First, we took notes from the videos. Then, we coded the suggestedrecommendations, experiences, and implementation details about using featuretoggles mentioned by practitioners in the textual artifacts and in the notes ofvideos. The coding of the artifacts was done by the first and second authorsof the paper.

After the identification of practices, we observed similarities and dissimi-larities between practices. We put practices with similarities into one categorybased on an open coding technique and found four categories. We give anexample of using open coding with a sample of our data in Figure 3. In thisfigure, three paragraphs from three artifacts are shown and codes are assignedto them. The codes of the two first paragraphs pointed to the same conceptso we grouped them as “Limit the number of feature toggles”. The last code ischanged to “Create a cleanup branch” practice. The similarity between thesetwo extracted practices is pointing to cleaning up feature toggles, so the two


practices are grouped as “Clean-up practices”. The result of this step is theanswer to the first research question (RQ1-Identification). Step One and StepTwo of the methodology were done iteratively. The snowball approach wasterminated when we did not identify any new practices and did not find anynew artifact.

After identification of practices from artifacts, we specified a “Level ofconfidence” for each practice which was used to quantify our confidence inthe quality, correctness and importance of the identified practices. We usedthe quality assessment checklist of grey literature for software engineeringprovided in [18] and shown in Table 1. Following their example, each of the20 question is assigned a score of 1, 0.5 or 0, so the highest score would be 20.We assigned a score of 20 to peer-reviewed papers.

Table 1: The quality assessment checklist of grey literature for soft-ware engineering adapted from [18].

Criteria Questions Notes

Authority ofthe producer

Is the publishing organiza-tion reputable?

The authorship is at-tributed to an organization.

Is an individual author as-sociated with a reputable

The authorship is at-tributed to an individualauthor(s).

Has the author publishedother work in the field?

Having other grey litera-ture in the area.

Does the author have ex-pertise in the area?

Having experience in thearea.

Methodology

Does the source have aclearly stated aim?

Having clear related sub-ject.

Does the source have astated methodology?

Having an structured flowfor discussion.

Is the source supportedby authoritative, contem-porary references?

Having any references.

Are any limits clearlystated?

Pointing to at least one re-lated limitation.

Does the work cover a spe-cific question?

Covering the concept of fea-ture toggles.

Does the work refer toa particular population orcase?

Related to feature toggles.

Objectivity

Does the work seem to bebalanced in presentation?

Discussing the subject fromdifferent views.

Is the statement in thesources as objective as pos-sible?

Including enough evidence.


Is there vested interest? Unbiased to any organiza-tion’s tool.

Are the conclusions sup-ported by the data?

Having reasonable conclu-sion.

Date Does the item have a clearlystated date?

Including date.

Positionw.r.t. relatedsources

Have key related grey lit-erature or formal sourcesbeen linked to/discussed?

Having at least one of thekey grey artifacts of thearea.

Novelty Does it enrich or add some-thing unique to the re-search?

Including any valuable datafor the research.

Does it strengthen or refutea current position?

Supporting any valuabledata for the research.

Impact Normalize Number of back-links, Number of social me-dia shares into a single met-ric

For backlinks: https://www.seoreviewtools.com/valuable-backlinks-checker/ , Forsocial media share: http://www.sharedcount.com/

Outlet type– 1st tier grey literature

(measure=1): Highcredibility: Books,magazines, theses, gov-ernment reports, whitepapers

– 2nd tier grey litera-ture(measure=0.5):Moderate credibility:Annual reports, newsarticles, presentations,videos, Q/A sites (suchas StackOverflow), Wikiarticles

– 3rd tier grey liter-ature(measure=0):Low credibility: Blogs,emails, tweets

Assign score based on typeof the artifact.

To specify the level of confidence, we combined two factors for each practiceas shown in Table 2: (1) The average quality of the artifacts that the practiceis mentioned in; and (2) The number of artifacts that point to the practice. Wedefined four level of confidence: High, Moderate-High quantity, Moderate-High

https://www.seoreviewtools.com/valuable-backlinks-checker/




http://www.sharedcount.com/

http://www.sharedcount.com/


Table 2 The level of confidence.Average quality score of artifacts

Number of artifacts [10,20] [0,10)[33,66] High (1) Moderate-High quantity (0)[0,33) Moderate-High quality (16) Low (0)

quality and Low. In Table 2, the range for average quality score of artifacts isbetween 0 and 20. We divided this range to 2 equal width ranges. The numberof artifacts analyzed in this step is 66, and we divide the range of 0 to 66to 2 equal width ranges. The numbers in front of each level of confidence isthe number of practices fell into the category. Moderate-High quantity is usedwhen the practice is mentioned in more than a half of the artifacts but theaverage quality of the artifacts is lower than the selected threshold which is10. Moderate-High quality is used when the practice is mentioned in less thana half of the artifacts but the average quality of the artifacts is more than thethreshold. We will refer to this table in Section 5.1.

4.3 Step Three: Searching Company-specific Grey Literature

Some artifacts collected in Step One were company-specific artifacts. Addi-tionally, some artifacts contained a list of companies that use feature toggles.From these artifacts, we obtained a list of companies which use feature togglesin their development cycle. Additional searches were conducted to collect moreartifacts related to feature toggles from these specific companies. We used thesearch strings in the following format: “[company name] [feature toggle term]”where company name represents the name of the company; and feature toggleterm is a search term for “feature toggle,” as defined in Step One. For eachcombination of company name and feature toggle term, a search string wasapplied to collect as many artifacts as possible. These strings were searched byusing both the Google search engine and search feature found within a com-pany’s blog. We looked at the first 10 pages of the Google search engine resultand all of the results in the company’s blogs. If a company uses a feature togglemanagement system named by an artifact, we also used that system’s nameinstead of “feature toggle term” in a search string. For example, Facebook usesGatekeeper for feature toggle management [23]. We used Gatekeeper insteadof “feature toggle term” as well as search terms for feature toggle in the searchfor Facebook.

4.4 Step Four: Extraction of Practice Usage from Company-specific Artifacts

We analyzed the company-specific grey literature artifacts collected in StepOne and Step Three to determine which practices identified in Step Two areused by the companies, as mentioned in the artifacts. If a practice was not


clearly mentioned, a second person analyzed the artifact and then we made adecision if the company used the practice or not.

Step Three and Step Four in Figure 2 were performed iteratively and re-peatedly if new artifacts for a company were found in Step Four.

4.5 Step Five: Conducting Survey

After extracting practice usage by the companies, we observed that our resultswere not complete. For instance, some of the identified feature toggle practiceswere not mentioned in any of the company-specific artifacts. In addition, wewere interested to know about the status of usage of feature toggles in industry.So, we conducted a survey to obtain more information about feature togglepractice usage.

Contact information of company employees was gathered by collecting so-cial media accounts and email addresses of named individuals associated withcompany-specific artifacts found in Steps One and Three. We also found con-tact information of managers/developers in companies that we knew were us-ing feature toggles based upon Step One, even though we did not find company-specific artifacts for them in Step Three. We requested each practitioner tocomplete the survey. We contacted the practitioners by email where email wasavailable and by social media if email addresses were not found. We sent thesurvey to 45 companies and got 20 responses for a response rate to the surveyof 44%.

The survey has 11 questions and is presented in the Appendix. On average,each practitioner needed approximately 5 minutes to answer all questions. Weused Likert scale options [24] for the 12 practices for which Likert scale optionscan be used. We provided five options in the survey for each practice to specifyhow much the survey respondents use the practice: Always, Mostly, About halfof the time, Rarely, and Never. For the remaining 5 practices, we providedpractice-specific answer options.

4.6 Step Six: Analysis of Usage Frequency of Practices

We analyzed the information from Step Four (analyzing company-specific arti-facts) and Step Five (survey) to find the frequency of usage of each identifiedpractice in the industry to answer RQ2. We integrated the result of StepFour and Step Five and report the frequency of usage of feature toggle prac-tices. In addition, we reviewed all the artifacts (including initial artifacts andcompany-specific artifacts) and record any comparison made between practicesin artifacts in this step.


Design Implementation Existence Clean-upDecision

Developer

No

YES

Fig. 4 The lifecycle of a feature toggle, adapted from [25].

5 Results

Based on [25], we propose the lifecycle of a feature toggle as shown in Figure 4.The first phase is Decision, when the development team decides if the usageof a feature toggle is necessary for their situation. When the developmentteam decides to use a feature toggle, the second phase is Design in whichthe details of the feature toggle is determined, such as type of the toggle, thepossible values of the toggle, and/or the name of the toggle. The third phaseis Implementation which includes adding the designed feature toggle to thecode. The fourth phase is Existence, in which the toggle exist in the codeand/or is updated. The fifth phase is Clean-up in which the toggle is removedfrom the code. For each identified practice, in Section 5.1, we will specify thelifecycle phases covered by the practice.

In the rest of this section, we present the results of the research method-ology. Section 5.1 provides the answer to RQ1-Identification, and Section 5.2provides the answer to RQ2-Frequency.

5.1 Feature Toggles Practices

We found 66 artifacts including 10 peer-reviewed papers, 41 blog posts andonline articles, and 15 videos in Step One. From these artifacts, we identifiedand categorized 17 practices in Step Two. We found four categories of prac-tices: Management practices (6), Initialization practices (3), Implementationpractices (3), and Clean-up practices (5). We describe the 17 practices in theircategories in the rest of this section.

The level of confidence of each identified practice is calculated as we de-scribed in Step Two of the methodology in Section 4. The range of scores ofquality assessment of grey literature artifacts is from 9.5 to 17. The numberof practices that fell into each level of confidence is shown in Table 2. “Usemanagement systems” is the only practice in High level, all the rest practicesare in Moderate-High quality level.

Some of the practices are related to other practices. For example, followingone of the practices may help following another practice. We determined therelation between practices based on the effects of using them. Related practicesare summarized in Table 5.1.

For each practice, we explain the practice using the following structure:


1. Description: The explanation of the practice.2. Goal: The goal of following the practice.3. Examples: The example(s) from the 38 companies listed in Table 5 and

practitioner’s experiences about following or not following the practice.4. Covered phases in lifecycle: The phase(s) in which the practice would be

used, as shown in Figure 4.5. Comparative practices: The list of practices which could be compared with

the practice (because of the same goal). Practices in Clean-up category arethe only practices which could be compared to each other. So, practices inother categories do not have this bullet in their structure.

6. Generalizability: A categorization of whether the practice is general to soft-ware engineering (SE); common with configuration options (C) as identifiedin [21] and discussed in Section 3; or feature toggle-specific (F).

Management Practices (6)

Management practices are practices that are performed by developmentteam members to make decisions about how to use feature toggles.

M1: Use management systems:Description: Management systems help companies to create, use, andchange the value of feature toggles. Using feature toggle management sys-tems helps to overcome technical debt and manage the added complex-ity [26]. Feature toggle management systems can have a dashboard thathelps team members to see the list of feature toggles and their current val-ues. Team members can add new feature toggles or change the values of thetoggles if they have the permission. Management systems are connected tothe code, and the changes impact the running system immediately. Man-agement systems can be open-source or closed-source. Organizations maycreate their own feature toggle management system.Goal: To manage creating and updating feature toggles in a centralizedsystem.Examples: Facebook uses the Gatekeeper toggle management system [23].Alternatively, companies can use third-party management systems, suchas LaunchDarkly3 and Split4. As an example, Behalf5 and CircleCI6 usethe LaunchDarkly feature toggle management system. Envoy7 uses Split’sfeature toggle management system.Covered phases in lifecycle: Design, Implementation, Existence, andClean-up.Generalizability: (C). Sayagh et al. [21] recommend the adoption of ex-isting configuration frameworks.

3 https://launchdarkly.com/4 https://www.split.io/5 https://www.behalf.com/6 https://circleci.com/7 https://envoy.com/

https://launchdarkly.com/

https://www.split.io/

https://www.behalf.com/

https://circleci.com/

https://envoy.com/


M2: Document feature toggle’s metadata:Description: Through the documentation of feature toggles, practitionersrecord the feature toggle’s information e.g. the owner of the feature toggle;the current status (to remove untriaged, keep, removed); the time of itscreation; and any notes.Goal: To enable practitioners to have access to the feature toggles’ meta-data at any time.Examples: Google has a spreadsheet with a list of feature toggles, theowner, toggles’ status, and notes about toggles which used in the Chromeproject [27]. However, this spreadsheet has only changed three times in2018 and three times in 2019. The developers may have moved on to use anew tool for documenting feature toggles’ metadata.Covered phases in lifecycle: Implementation, Existence, and Clean-up.Generalizability: (SE [28]/C). Sayagh et al. [21] listed “Comprehensionof Options and Values” and “Configuration Knowledge Sharing” as twoconfiguration option activities . The concept behind these two activities issimilar to documenting feature toggle’s metadata.

M3: Log changes:Description: Tracking changes that are made on feature toggles. By log-ging, the information of who changes which toggle and when is recorded[29].Goal: To document traceability of actions for creating, updating, anddeleting toggles and their values.Examples: Split’s feature toggle management system has the ability tolog changes of the feature toggles [30].Covered phases in lifecycle: Implementation, Existence, and Clean-up.Level of confidence: Moderate.Generalizability: (SE [28]/C). In [21] having “Right Granularity of Exe-cution Logs” is one of the recommendations, so logging changes is recom-mended for configuration options.

M4: Determine applicability of feature toggle:Description: Before the design and implementation of a feature toggle,the development team should determine if a feature toggle should be used.Using feature toggles adds more decision points to the code which addsmore complexity to the code and requires attention to remove toggles whenthe initial use is completed.Goal: To make an explicit decision on the creation of a toggle which mayreduce the number of feature toggles in a code base.Examples: Different companies have different approaches to making deci-sions. For example, all new features in GoPro have feature toggles8. How-ever, practitioners in Finn.no, a largest online marketplace in Norway, avoidusing feature toggles if they do not need the toggle [31].

8 https://bit.ly/2ISi1ye


Covered phases in lifecycle: Decision.Generalizability: (F).

M5: Give access to team members:Description: Through this practice, permission to change values of featuretoggles is granted to team members in addition to developers using thefeature toggle management system.Goal: To prevent feature toggle management bottleneck.Examples: If all team members, such as Q&A team members, have accessto feature toggles, they can change a toggle status in case of a problem [29].For instance, Instagram gives access to their feature management systemto the product managers and sales team [32].Covered phases in lifecycle: Design, Implementation, and Existence.Generalizability: (SE).

M6: Group the feature toggles:Description: Grouping similar feature toggles.Goal: To enable the assignment of access to groups of feature togglesto teams or team members [29]; to simplify management of dependenttoggles [25]; or to turn on or off feature toggles related to part of thesystem at the same time.Examples: Practitioners in GoPro have a two-level toggle hierarchy: sim-ple feature toggles and higher level feature toggles [33].Covered phases in lifecycle: Design, Implementation, and Existence.Generalizability: (SE).

Initialization Practices (3)

Initialization practices are used to make decisions about the design of thefeature toggle before their creation.

I1: Set up the default values:Description: Default values for each feature toggle are set in case if as-signed values are not found or do not exist.Goal: To mitigate unwanted behavior of the feature toggle.Examples: At Lyris, feature toggles without values are automaticallyturned off [34].Covered phases in lifecycle: Design and Implementation.Generalizability: (SE/C). Sayagh et al. [21] mentioned that the selectionof the “right” default value for configuration options is a challenge.

I2: Use naming convention:Description: Having predefined naming rules for feature toggles.Goal: To establish naming conventions, particularly to make the intentionof the toggle self-documented.Examples: Having the naming convention has several benefits. First, un-derstanding the purpose of using the toggle is useful [34], i.e. if the owner


of the code is changed, the new owner can understand the usage of thetoggle easily if the name of the toggle reflects its usage. An example ofLyris toggles is “ct.enable flex cache inspector’ and the purpose of usingthe toggle is clear based on its name [34]. Second, the use of a naming con-vention reduces the likelihood of multiple toggles with same names evenby different teams by following naming conventions [29]. Third, adding thetype of the toggle as a prefix in its name can help with the management ofthe toggles [35]. For instance, if the feature toggle is a short-lived toggle,like release toggles, the developer will get a signal from the name of thetoggle that the first intention of using the toggle was a short-term use andwill plan to remove it. For instance, In InVision, long-lived toggles have“OPERATIONS-” prefix [36]. Developers in this company also add theJIRA ticket number to the name of the feature toggle to make the purposeof using the toggle and responsible team to remove the toggle clear. If a“RAIN-123-release-the-kraken” is a name of the toggle, it is clear that thetoggle is related to JIRA ticket RAIN-123 and the responsible team toclean-up the feature toggle is the Rainbow team [37].Covered phases in lifecycle: Design and Implementation.Generalizability: (SE [38]/C). In [21] one of the challenges of using con-figuration options is “meaningless Option Names” .The authors listed “Ex-plicit Option Naming Convention” as a recommendation to overcome thechallenge which is similar to use naming convention practice.

I3: Determine the type of the toggle:Description: With this practice, the type of the toggle is specified usingthe toggle types mentioned in Section 2. The implementation and manage-ment of each type of the four toggle types are different.Goal: To aid in quality management of a toggle’s implementation and toenable the plan to remove the toggle on time based on the type of thetoggle.Examples: The author in [39] points to naming short-lived toggles withthe prefix of “temp-” in their name. The identification of short-lived togglescan be useful in limiting the number of toggles [40].Covered phases in lifecycle: Design and Implementation.Generalizability: (F).

Implementation Practices (3)

Implementation practices are related to implementation details of featuretoggles.

Im1: Type of assigned values:Description: Companies use three different ways to assign values to tog-gles. One way is to assign a string to feature toggles. The second way toassign values is to assign Boolean values to feature toggles. When the valueis true, the toggle is enabled [41]. The third way is to assign multivariatevalues, such as when the toggle capture user experiences.


Goal: To help practitioners to choose an appropriate type of the valuesfor feature toggles in their system.Examples: For instance, one of the feature toggles in Google Chromeproject is kDisableFlash3d[] = “disable-flash-3d”. If the value of the fea-ture toggle is set then the toggle is enabled [4]. As another example, Roll-out9 provides multivariate toggles, for instance a toggle can accept “Red”,“Blue” and “Yellow” as its value.Covered phases in lifecycle: Design and Implementation.Generalizability: (C). Sayagh et al. [21] provide the “Using Simple Op-tion Types” recommendation which mentioned string and Boolean typesfor configuration options. This recommendation has partially overlappedwith the current practice.

Im2: Ways of accessing the values:Description: We identified three ways development teams access the val-ues of feature toggles. First, the feature toggles could be primitive vari-ables, hard-coded into the program. Second, toggles could be objects andthe object has a method to determine the value of the toggle (e.g. myTog-gle.isActive()). Third, toggles could be accessed through a manager object.Managers map key/value pairs to return the value.Goal: To help practitioners to choose an appropriate way of accessing thevalues in their system.Examples: Figure 1 is an example of directly access to the values. Wefound implemented libraries in GitHub which use a method from toggleobject, such as rollout10. LaunchDarkly is an example of using managerobjects to access feature toggle values.Based on the experience of the practitioner in [42], having class of togglesand check the value of the toggle using isEnable() function of the class isbetter than checking the primitive variable of a string name of the toggle.Having feature toggle objects helps to refactor toggles same as the otherparts of the code and find every usage of it easily. The practitioner in [43]also points to the fact that using objects of feature toggles is better thanstrings because of getting compile error on all places a feature toggle isused after removing the toggle. The same comparison is mentioned in [44]:“Toggles should be real things (objects) not just a loosely typed string.This helps with removing the toggle after use: 1) Can perform a ‘find uses’of the Toggle class to see where it’s used, and 2) Can just delete the Toggleclass and see where build fails.”Covered phases in lifecycle: Design and Implementation.Generalizability: (SE [38]).

Im3: Store type:

9 https://rollout.io/10 https://github.com/fetlife/rollout


Description: The list of feature toggles and their values can be stored inone of two ways: file storage and database storage. In file storage, the val-ues of feature toggles are stored in one or multiple configuration files [4]. Indatabase storage, the values of feature toggles are stored in databases, suchas Redis [41] or SQL [44]. In addition, some companies “use a third partyservice” to fetch values of the feature toggle. If they use a feature manage-ment system, they fetch the values of feature toggles from the managementsystem.Goals: To store the values of feature toggle in an appropriate way, basedon the advantages and disadvantages of each store type.Examples: Based on the article [12], when practitioners use configurationfiles to save the feature toggle values, they may need to redeploy the appli-cation after each value update to get the reflection of the updated value.In addition, when a system is in large scale, it is hard to manage featuretoggles using configuration files and it is hard to make sure the consistencyof configuration files on different servers. So the practitioner recommendto use some sort of database to store the value of feature toggles.The practitioner in [41] is also make the same comparison in his article: “A more dynamic approach is to store the feature configuration in either anephemeral or permanent storage, like Redis or your database, respectively.Assuming your code continuously checks feature flags at runtime, all ittakes is changing a value in the central configuration service and it canhave an immediate affect on the running application without requiring arestart.”The article [45] points to the same comparison: “ Release and Experimenttoggles are likely to be set at deployment time, so from a running appli-cation perspective they are static settings... However, Ops and Permissiontoggles are dynamic and need to be configurable at run time, so you mightwant to store them in a database of some sort.” The same concept is men-tioned in [25] and [43] as well.Dropbox use configuration files and database together [46]. They explainthe reason of using both ways. Because of having a large number of pro-duction servers, they prefer using a database instead of using configura-tion files. However, this may create a huge number of fetches against thedatabase and the database will be the single point of the failure even ifthey have caching system. So, they come up with a combination of bothways. A JSON file is shared between all the production servers and containsthe value of feature toggles. If this JSON file is not accessible for any rea-son, the feature toggle management system has the ability to access to thedatabase directly. This example shows the advantages and disadvantagesof using each way of storing the values of feature toggles.Covered phases in lifecycle: Design and Implementation.Generalizability: (C). “Managing Storage Medium” activity form [21] issimilar to this practice. Moreover, one of the challenges in [21] is “Storagemedia Impact Performance” which is really similar to the practitioners’experiences we provided for this practice.


Clean-up Practices (5)

Following the clean-up practices helps practitioners to remove their featuretoggles on time and manage the complexity of using feature toggles.

C1: Add expiration date: This practice is followed using one of the fol-lowing three processes:– C1.1: Time bombs:

Description: If the toggle exists after its expiration date, a test failsor the application does not start, which causes a developer to removethe toggle [12], [47], [40]. The expiration date is the latest possible datewhich the developers should remove the toggle from the code. Usingthis practice forces practitioners to remove a toggle by the determinedexpiration date.Goal: To remove unused toggles.Examples: We did not find any specific examples in company-specificartifacts. However, the practitioner in [40] says: “(Time bomb) is veryextreme and I wouldn’t recommend doing (it). I think a lot of organi-zations would not be comfortable with doing that, it does force a lot ofother things to be good.”Covered phases in lifecycle: Design, Implementation, and Clean-up. The expiration date is identified during the design phase, the timebomb is added in the Implementation phase and the feature toggle isremoved in the Clean-up phase.Comparative practices: All the practices in this category (C1-C5).Generalizability: (F).

– C1.2: Automatic reminders:Description: Add automatic reminders to remind developers the dead-line for removing feature toggles [47]. Using this practice helps practi-tioners to remember to remove a toggle by the determined deadline.Goal: To remove unused toggles.Examples: Slack has an archival system. When developers want toadd a new feature toggle, they have to specify the date they plan todelete the toggle. If the toggle is not deleted by the specified date, thedeveloper will get an alert11.Covered phases in lifecycle: Design, Implementation, and Clean-up.Comparative practices: All the practices in this category (C1-C5).Generalizability: (SE).

– C1.3: Use cards/tasks/stories for removing toggles:Description: Add tasks/stories/cards for removing toggles to a Kan-ban board (or any other tool that the team uses) [47] or to developer’stask backlog [12], [40]. Using this practice reminds practitioners the

11 https://bit.ly/2W4hQUk


task of removing a toggle at the expiration date when the purpose ofusing the toggle is done.Effects: To remove unused toggles.Examples: Developers at Lyris create user stories for removing toggles[34]. However, the practitioner in [40] says about his experience of usingthis practice:“ I think it can help but it’s kind of the bare minimum,.. I have spent a lot of time with clients where the clean-up ticket isjust at the very top of the next Sprint’s backlog for like six months. It’salways like yeah we really should do that the next week and it will bedone next week, but it’s always next week.” Practitioner in [37] has thesame experience with this technique: “ Teams are supposed to createadditional “clean-up” tasks in JIRA for their feature flags such that wedon’t lose track of them. The reality, however, is far less sanitary. Ourfeature flags tend to pile up and we have to occasionally have a “purge”of flags that no longer seem relevant.”Covered phases in lifecycle: Design, Implementation, and Clean-up.Comparative practices: All the practices in this category (C1-C5).Generalizability: (SE [48]).

C2: Track unused toggles:Description: With this practice, dead code and unused feature toggles areremoved. Based on the logging system or using documentation, the statusof toggles could be monitored. Developers can use this data to find whenthe toggle is safe to remove [29]. When a toggle is always on or always off,it should be removed.Goal: To remove unused toggles.Examples: DropBox has a static analyzer tool with a service specificallyfor feature toggles. The static analyzer tool sends emails to feature toggleowners about removing the toggles which are not in use anymore [46].Covered phases in lifecycle: Implementation and Clean-up.Comparative practices: All the practices in this category (C1-C5).Generalizability: (F).

C3: Limit the number of feature toggles:Description: Using this practice the number of alive feature toggles at atime are limited to control the number of toggles. An alive feature toggle isa toggle which exists in the code whether it is on or off. By this limitation,practitioners have to remove an unused toggle to be able to add a newtoggle if the number of existing toggles meets the limitation [12], [40], [49].Goal: To remove unused toggles.Examples: We did not find any specific example of using this practice incompany-specific artifacts. The practitioner in [40] says this practice is hisfavorite practice for removing feature toggles.Covered phases in lifecycle: Implementation and Clean-up.Comparative practices: All the practices in this category (C1-C5).Generalizability: (F). Sayagh et al. [21] points to minimizing the numberof configuration options in the system as one the recommendations. This


recommendation is partially covered the practice of limiting the number offeature toggles. In limiting the feature toggles, developers have to removea toggle if the limitation is reached but removing is not mentioned in therecommendation of configuration options. So, we specify the practice asfeature toggle specific practice.

C4: Create a cleanup branch:Description: This is the practice of creating a branch to delete the toggleand submitting a pull request for the branch at the same time as addinga new feature toggle [35], [39]. Using this practice prevents to forget thedeletion of the feature toggle.Goal: To remove unused toggles.Examples: We did not find any example of using this practice in company-specific artifacts. The author of the [39] says that this practice works prettywell in their team: “The advantage to managing cleanup this way is thatyou do the work to remove the flag when all of the context is fresh inyour mind. At this point, you know all the pieces that get touched by thechange, and it is easier to be sure you don’t forget something .. This iscertainly not the only way to handle this issue, but it seems to work prettywell for our team.”.Covered phases in lifecycle: Implementation and Clean-up.Comparative practices: All the practices in this category (C1-C5).Generalizability: (SE).

C5: Change a feature toggle to a configuration setting:Description: This is the practice of keeping feature toggles in the codewith changed functionality. The feature toggle can be changed to admin oruser configuration settings. This technique is used when development teamdecide to keep more than one variant of the feature toggle in the code.Goal: To remove unused feature toggles.Examples: Suppose a feature toggle is used for running experiments to seewhich color is better for the “buy” button in an e-commerce application.The experimental results show that the users are happiest when they cancontrol the color of the button. Instead of deleting the feature toggle, itwill be changed to a user configuration setting [50].Covered phases in lifecycle: Implementation and Clean-up (C1-C5).Comparative practices: All the practices in this category.Generalizability: (F).

5.2 Usage of Practices in Industry

In Step One, 26 artifacts were company-specific artifacts. In Step Three andStep Four, we found 43 additional company-specific artifacts. In total, 69company-specific artifacts from 38 companies were collected. The overlap be-tween initial artifacts and company-specific artifacts is shown in Figure 5. In


Table 3 Related practices.

Management Initialization Implementation Clean-up

Use

man

agem

ent

syst

ems

Doc

umen

tfe

atur

eto

ggle

’sm

etad

ata

Log

chan

ges

Det

erm

ine

appl

icab

ility

offe

atur

eto

ggle

Giv

eac

cess

tote

amm

embe

rsG

roup

the

feat

ure

togg

les

Set

upth

ede

faul

tva

lues

Use

nam

ing

conv

enti

on

Det

erm

ine

the

type

ofth

eto

ggle

Typ

eof

assi

gned

valu

es

Way

sof

acce

ssin

gth

eva

lues

Stor

ety

pe

Add

expi

rati

onda

te

Tra

ckun

used

togg

les

Lim

itth

enu

mbe

rof

feat

ure

togg

les

Cre

ate

acl

eanu

pbr

anch

Cha

nge

afe

atur

eto

ggle

toa

confi

gura

tion

sett

ing

Man

agem

ent Use management

systems– X X X X X X X X X X X X

Document featuretoggle’s metadata

X –

Log changes X – XDetermine applica-bility of feature tog-gle

–

Give access to teammembers

X –

Group the featuretoggles

X –

Init

ializ

atio

n Set up the default

values

X –

Use naming conven-

tion

X –

Determine the typeof the toggle

X – X

Impl

emen

tati

on Type of assigned

values

X –

Ways of accessing

the values

X –

Store type X –

Cle

an-u

p Add expiration date X –Track unused tog-gles

X X X –

Limit the number offeature toggles

X –

Create a cleanupbranch

–

Change a featuretoggle to a configu-ration setting

–


Company-specific Artifacts from Step One (26)

Initial Artifacts (66) Company-specific Artifacts (69)

Company-specific Artifacts from Step Three and

Four (43)

General Artifacts from Step One (40)

Fig. 5 Number of artifacts found in in Steps 1, 3 and 4

Step Four, we analyzed these 69 company-specific artifacts to find which com-panies use the identified practices. The practices used by each company areshown in Table 5 in the Appendix. This table is useful for practitioners be-cause software practitioners prefer to learn through the experiences of othersoftware practitioners [7], as we mentioned before.

In Step Five, we conducted a survey to gather additional information aboutthe usage of feature toggles practices in industry. We had company-specificartifacts of 38 companies. Of these 38 companies, we sent out the survey to36 companies for which we had the contact information for release engineersand/or developers. In addition to these companies, we identified a list of 20companies which use feature toggles in their companies. These companies arementioned in artifacts as the example of companies which are using featuretoggles but we cannot find company-specific artifacts related to their practiceusage. We found contact information for release engineers and/or developersin 9 of these companies and sent the link of the survey to them. In total, wesent the survey to 45 companies. We got 20 responses for a response rate tothe survey of 44%. These 20 responses are from at least 17 companies. Wecannot compute the exact number because three respondents did not identifytheir company name.

As mentioned in Section 4, we used a Likert scale [24] with five optionsfor 12 of the 17 practices for which a Likert scale options can be used. In ouranalysis, we grouped Always, Mostly and About half of the time responses andassumed the companies that selected these options use the practice. We alsogrouped Rarely and Never and assumed the company does not use the practiceif the respondents selected one of these two options. The detailed result of thesurvey responses on the 12 questions with Likert scale options is shown inFigure 6. For the 5 remaining practices which includes all 3 implementationpractices and 2 management practices, survey respondents chose from a listof provided answers or to add text to an open-ended “other” response. Forexample, for the “Use management systems” (M1) respondents could choseamong open or closed source systems, chose they did not use a management


Table 4 Feature toggle practices and their usage in industry

Category (mean of fre-quencies based on artifacts,mean of frequencies basedon survey)

Practice Frequencyfrom arti-facts (38companies)

Frequencyfrom survey(20 compa-nies)

Management (42%, 63%)

Use managementsystems

32 (84%) 20 (100%)

Document featuretoggle’s metadata

25 (66%) 7 (35%)

Log changes 21 (55%) 12 (60%)Determine appli-cability of featuretoggle

8 (21%) 16 (80%)

Give access to teammembers

7 (18%) 14 (70%)

Group the featuretoggles

2 (5%) 7 (35%)

Initialization (25%, 72%)Set up the defaultvalues

22 (58%) 17 (85%)

Use naming con-vention

5 (13%) 14 (70%)

Determine the typeof the toggle

1 (3%) 12 (60%)

Implementation (66%, 100%)Type of assignedvalues (string,boolean, multi-variate, more thanone)

32 (1 (3%),7 (18%), 5(13%), 19(50%))

20 (1 (5%),6 (30%), 2(10%), 11(55%))

Ways of access-ing the values(primitive variable,objects, managers,more than one)

28 (0 (0%),0 (0%), 28(74%), 0(0%))

20 (3 (15%),5 (25%),3 (15%), 9(45%))

Store type (file,database, both,third party ser-vice)

15 (9 (24%),4 (11%), 2(5%), - )

20 (3 (15%),6 (30%),6 (30%), 4(20%))

Clean-up (3%, 39%)

Add expirationdate (Time bombs,Automatic re-minders, Usecards/tasks/sto-ries for removingtoggles)

6 (0 (0%),1 (3%), 5(13%))

14 (1 (5%),4 (20%), 9(45%))

Track unused tog-gles

1 (2%) 9 (45%)

Limit the numberof feature toggles

0 (0%) 10 (50%)

Create a cleanupbranch

0 (0%) 4 (20%)

Change a featuretoggle to a config-uration setting

0 (0%) 2 (10%)


system or add any answer to “other” response. The survey questions are listedin the Appendix of the paper.

We use results from analyzing company-specific artifacts and survey re-sponses to answer RQ2. The result of analysis of company-specific artifactsis shown in third column and the survey result is shown in the last columnof Table 4. The frequency of usage of each practice in both company-specificartifacts and survey result is shown in this table.

Based on the survey responses, the companies which the survey respondentswork have been using feature toggles for an average of 4.8 years. Among 20respondents, 19 respondents use toggles to have gradual roll out. Nineteenrespondents use toggles to support CI of partially-completed features, and17 respondents use toggles to perform A/B testing. Fifteen respondents usetoggles to have dark launches.

In the following subsection, we go through each of the categories and high-light the main findings on usage of practices. The analysis is based on thesurvey responses and company-specific artifacts for each category of practices.

5.2.1 Management Practices

The most used practice is “Use management systems” (M1) based on bothcompany-specific artifacts and survey responses. However, in comparison toconfiguration options, Sayagh et al. [21] show that developers do not tend touse existing configuration frameworks.

For the “Determine the applicability of feature toggle” (M4) practice, foursurvey respondents stated that the feature toggle is always added when a newfeature is added or any feature is changed. They do not have any decisionmaking process for using feature toggles. In companies where a feature toggleis added for each new feature, there will eventually be a large number offeature toggles so management and deletion of the toggles are more critical toprevent increased code complexity and dead code. The “Log changes” (M3)practice enables practitioners to follow “Track unused toggles” (C2) practicefrom clean-up category as shown in Table 5.1. If a company logs every changemade on feature toggles, tracking unnecessary toggles will be easy.

5.2.2 Initialization Practices

The most used practice in the initialization category based on both company-specific artifacts and survey responses is “Set up the default values” (I1) basedon Table 4. The “Use naming conventions” (I2) and “Determine the type ofthe toggle” (I3) are next in the rankings. The usage ranking of the practices inthis category is same in both company-specific artifacts and survey responses.

“Determine the type of the toggle” (I3) is a practice which helps prac-titioners to use “Limit the number of feature toggles” (C3) practice in theclean-up category more efficient, as shown in Table 5.1. If the type of the tog-gles are pre-determined, the practitioners have a list of short-lived toggles as


a suggested list of toggles to remove. Instead of checking all of the toggles, theshort-lived toggles could be checked for removal.

5.2.3 Implementation Practices

As shown in the first column of Table 4, the mean of usage frequencies of im-plementation practices is 66% based on company-specific artifacts and 100%based on the survey. This category of practices is the most used practices inindustry based on our result. When a company uses feature toggles, the de-velopment team implements the code of feature toggle including a mechanismto store the values of the toggle, select the type of the assigned value, anddetermine how to access the value.

For “Ways of accessing the values” (Im2), the experiences from practition-ers in [42], [43] and [44] as we mentioned in Examples of the practice inSection 5.1 reflect the popularity of using objects among other ways. This canjustify the survey’s result, the most used way is objects (25%). All the sur-vey’s respondents which use more than one way (45%) use objects as one ofthe ways.

For “Store type” (Im3), using a configuration file is more popular thanusing databases in the company-specific artifacts; but the survey’s responsesindicate that databases and a combination of configuration files and databasesare most used. We allowed respondents to add their own answer for this ques-tion, and three respondents mentioned using a third party service, such asget values from LaunchDarkly servers. However, we did not identify this op-tion in analyzing company-specific artifacts. We added this new store typeto Table 4 for the survey responses. As we mentioned the comparisons madebetween using configuration files and databases in [12], [41], [45], [25] and [43]in the Examples part of “Store Type” (Im3) practice, using databases givedevelopment team more flexibility but using configuration files is faster. Thesecomparisons from practitioners’ point of view can justify different results forthe practice based on company-specific artifacts and survey. Companies maystarted with configuration files and after realizing the disadvantages of it,switch to using database or a combination of configuration files and databaseto have fast updating of values.

The difference between usage frequencies of practices based on the arti-facts and based on the survey’s responses shows that companies may changethe implementation details of feature toggles over time and based on their ex-periences. Company-specific artifacts mentioned the feature toggles implemen-tation details in a time of publishing the artifacts, but the survey’s responsesreflects the current implementation details.

5.2.4 Clean-up Practices

Based on the company-specific artifacts and survey’s responses, the practicesof the clean-up category are the least used category of practices. The mean


of usage frequencies of clean-up practices are 3% based on company-specificartifacts and 39% based on the survey which is the lowest frequency category.

We did not find any comparison between “track unused toggles” (C2) and“Change a feature toggle to a configuration setting” (C5) with the rest of thepractices in this category. Based on the comparisons done by practitioner forthe rest of the practices as we mentioned in Examples of each practice inSection 5.1, “Use cards/tasks/stories for removing toggles” (C1.3) seems to benot very useful and “Time bombs” (C1.1) are extreme. However, “Limit thenumber of feature toggles” (C3) and “Create a cleanup branch” (C4) are goodto follow based on the practitioners’ experiences. Companies may use morethan one of these practices for cleaning up the feature toggles, as we observedin survey responses.

20 15 10 5 0

Change feature toggle to configuration setting

Create a clean-up branch

Limit the number of existing toggles

Track unused toggles

Use tasks/stories/cards for removing

Automatic reminders

Time bomb

Determine type of the toggles

Use naming convention

Set up a default value

Group toggles

Give access to all team members

Log changes

Document feature toggles metadata

Always Mostly About half of the time Rarely Never

0 5 10 15 20

Fig. 6 Frequency of using the subset of feature toggle practices with Likert scale based onthe survey

Among the survey responses, twelve companies are from 38 companieswhich we have their company-specific artifacts and their feature toggle prac-tices are shown in Table 5. We are able to compare what we found in company-specific artifacts with their responses to the survey for these twelve companies.For eight companies, there is discrepancies in survey responses and company-specific artifacts about the type of assigned values to feature toggles and waysof accessing these values in Implementation practices. There are no discrepan-cies found for Clean-up practices, the survey responses and observation fromcompany-specific artifacts are aligned. Three of the companies do not havedocumentation for their feature toggles anymore, based on survey responses.It could be because of the management systems they have which provide docu-mentation facilities. One of the companies followed “Determine the applicabil-


ity of feature toggles” (M4) based on the company-specific artifacts. However,they add feature toggle for any new feature now, based on their response tothe survey.

6 Discussion

Since each of the Management and Initialization category practices has positiveeffects on feature toggles management, we suggest practitioners to use them intheir feature toggle management process, specially “Use management systems”(M1). Our survey indicated that all respondents use the M1 practice becausemanaging the added complexity and technical code of adding feature togglesis easier when the management of feature toggles is carried out in a separatesystem.

Each of the Implementation category practices provides practitioners morethan one option to choose. We suggest practitioners select an appropriateoption for type of toggles, ways of accessing those toggles and store type. Inthe examples under Examples of the practices in Section 5.1, the advantagesand disadvantages of different options are provided which could guide thepractitioners on selecting appropriate options.

The common goal of all practices in the Clean-up category practices is toremove feature toggles when the purpose of using the toggles is accomplished.Based on practitioners’ experiences, “Use cards/tasks/stories for removing tog-gles” (C1.3) is not very useful and “Time bombs” (C1.1) are extreme. However,“Limit the number of feature toggles” (C3) and “Create a cleanup branch” (C4)are good to follow. We suggest, in their feature toggle development process,practitioners consider these advantages and disadvantages of clean-up prac-tices. A practitioner can choose to use multiple of these Clean-up practices(C1-C5). They are all used to remove unused toggles with different strategies.

7 Limitations

In this section, the limitations of research are discussed.

7.1 Finding artifacts

In Step One, we used a keyword search based upon five keywords to find greyliterature and peer-reviewed papers and selected artifacts that were relatedto the use of feature toggles in software development. We also followed linksand references to other artifacts in selected artifacts. We may have missedartifacts.

In Step Three, we searched for company-specific artifacts based the com-panies found in the initial artifacts. Data from companies who have not sharedtheir results on the Internet are not included in our study.


7.2 Identification of Practices and Categories

In Step Two, we did not use any automatic technique or tools to identifypractices. We may have missed some practices which were mentioned implicitlyin artifacts.

Another limitation is the lack of specific examples by companies of usingfeature toggle practices. Practitioners mentioned most of the practices withno concrete example.

In addition, testing practices are not identified and mentioned in the list ofpractices. Testing of the system which has feature toggles has different aspects,such as unit testing of feature toggles, testing all combination of feature tog-gles enabling and disabling, and testing dependent feature toggles. Anotherstudy should be conducted to cover testing concerns and practices when adevelopment team use feature toggles.

7.3 Extraction of Practice Usage from Company-specific Artifacts

In Step Four, we reviewed company-specific artifacts to extract feature togglepractices usage. If the practice was not mentioned in the artifacts, we cannotconclude that the company does not use the practice. To overcome this lim-itation, we conducted the survey to gather more information about usage offeature toggle practices in companies.

7.4 Conducting Survey

In Step Five, we found contact information of individuals associated withcompany-specific artifacts or who were release managers or developers of thecompanies. The contact information for some of the individuals could not befound or was old and out of date. To overcome this limitation, we found contactinformation of current development team members, such as release manageror developers of the companies, using company website or social media pages,such as LinkedIn. Additionally, the small sample size of the survey was alimitation.

8 Conclusion and Future Work

Feature toggles are a technique often used by companies who practice CI/CDto integrate partially-completed features into the code, conduct a gradual rollout, and/or to perform experiments. However, the development practices usedby these organizations have not been enumerated in prior research. We per-formed qualitative analysis of 99 artifacts from grey literature and 10 peer-reviewed papers. We identified 17 feature toggle practices in four categories:Management practices (6), Initialization practices (3), Implementation prac-tices (3), and Clean-up practices (5). We also quantified the frequency of usage


of these identified practices in industry by analyzing company-specific artifactsand conducting a survey.

The most popular practice in each category is consistent across the company-specific artifacts and survey responses. We observed that all of the survey’srespondents “Use a management system” (M1) to create and manage fea-ture toggles in their code. “Document feature toggle’s metadata” (M2), “Logchanges” (M3), and “Set up the default values” (I1) are three additional highly-used practices in industry based on company-specific artifacts. The least usedcategory of practices is Clean-up practices, even though cleaning-up the featuretoggles helps with managing the added complexity to the code and removingdead code. Inattention to removing feature toggles can cause severe problems,such as what happened to Knight Capital Group.

The feature toggle development practices discovered and enumerated inthis work could raise practitioners’ awareness of feature toggle practices andtheir usage in industry. Using the result of this paper could help practitionersto better use feature toggles in their projects, which was the goal of doing thiswork. We got feedback from some of the respondents to the survey that theclean-up practices seem interesting, and they decided to use identified practicesin their companies after participating in the survey.

The identified feature toggle practices discovered through this work can en-able future quantitative analysis to automatically identify practice use in coderepositories. Also a future study can be conducted on the empirical analysisof the effectiveness of the identified practices in repositories which practicesare use. Additional future work involves the automatic identification of featuretoggle bad smells in the code, such as unused feature toggles, nested featuretoggles, and development of a tool to automatically refactor the code when badsmells are identified. Also, the quality of parts of the code which is activate ordeactivate by feature toggles is one of the concerns mentioned by practition-ers [34]. Studying the impact of using feature toggles on code quality, such ashigh cohesion and low coupling, could also be a future work.

References

1. Martin Fowler. Feature toggle. [Online]. Available: https://martinfowler.com/bliki/FeatureToggle.html Accessed 24 April 2019.

2. Knightmare: A devops cautionary tale. [Online]. Available: https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ Accessed 24 April 2019.

3. Jens Meinicke, Chu-Pan Wong, Bogdan Vasilescu, and Christian Kaestner. Exploringdifferences and commonalities between feature flags and configuration options. In Tobe appear in Proc. of the 42nd International Conference on Software Engineering -Software Engineering in Practice (ICSE-SEIP), 2020.

4. Md Tajmilur Rahman, Louis-Philippe Querel, Peter C Rigby, and Bram Adams. Featuretoggles: practitioner practices and a case study. In Proc. of the 13th InternationalConference on Mining Software Repositories, pages 201–211. ACM, 2016.

5. Chris Parnin, Eric Helms, Chris Atlee, Harley Boughton, Mark Ghattas, Andy Glover,James Holman, John Micco, Brendan Murphy, Tony Savor, et al. The top 10 adages incontinuous deployment. IEEE Software, 34(3):86–95, 2017.

https://martinfowler.com/bliki/FeatureToggle.html

https://martinfowler.com/bliki/FeatureToggle.html

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/

https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/


6. Jim Bird. Feature toggles are one of the worst kinds of technical debt. [On-line]. Available: https://dzone.com/articles/feature-toggles-are-one-worst Ac-cessed 24 April 2019.

7. Geoffrey A Moore. Crossing the chasm: Marketing and selling technology project.Harper Collins, 2009.

8. Johnny Saldana. The coding manual for qualitative researchers. Sage, 2015.9. Jez Humble and David Farley. Continuous Delivery: Reliable Software Releases through

Build, Test, and Deployment Automation (Adobe Reader). Pearson Education, 2010.10. Martin Fowler. Continuousdelivery. [Online]. Available: https://martinfowler.com/

bliki/ContinuousDelivery.html Accessed 24 April 2019.11. Ross Harmes. Flipping out. [Online]. Available: http://code.flickr.net/2009/12/02/

flipping-out/ Accessed 24 April 2019.12. Pete Hodgson. Feature toggles (aka feature flags). [Online]. Available: https://

martinfowler.com/articles/feature-toggles.html Accessed 24 April 2019.13. Progressive experimentation with feature flags. [Online]. Available: https://docs.

microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/progressive-experimentation-feature-flags Accessed 24 April 2019.

14. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. Controlledexperiments on the web: survey and practical guide. Data mining and knowledge dis-covery, 18(1):140–181, 2009.

15. Carol Lefebvre, Eric Manheimer, and Julie Glanville. Searching for studies. Cochranehandbook for systematic reviews of interventions: Cochrane book series, pages 95–150,2008.

16. Vahid Garousi, Michael Felderer, Mika V Mantyla, and Austen Rainer. Benefitting fromthe grey literature in software engineering research. arXiv preprint arXiv:1911.12038,2019.

17. Vahid Garousi, Michael Felderer, and Mika V Mantyla. The need for multivocal litera-ture reviews in software engineering: complementing systematic literature reviews withgrey literature. In Proceedings of the 20th international conference on evaluation andassessment in software engineering, pages 1–6, 2016.

18. Vahid Garousi, Michael Felderer, and Mika V Mantyla. Guidelines for including greyliterature and conducting multivocal literature reviews in software engineering. Infor-mation and Software Technology, 106:101–121, 2019.

19. Akond Ashfaque Ur Rahman, Eric Helms, Laurie Williams, and Chris Parnin. Synthe-sizing continuous deployment practices used in software development. In Proc. IEEEAgile Conf., pages 1–10, 2015.

20. Md Tajmilur Rahman, Peter C Rigby, and Emad Shihab. The modular and featuretoggle architectures of google chrome. Empirical Software Engineering, pages 1–28,2018.

21. Mohammed Sayagh, Noureddine Kerzazi, Bram Adams, and Fabio Petrillo. Softwareconfiguration engineering in practice: Interviews, survey, and systematic literature re-view. IEEE Transactions on Software Engineering, 2018.

22. Claes Wohlin. Guidelines for snowballing in systematic literature studies and a repli-cation in software engineering. In Proceedings of the 18th international conference onevaluation and assessment in software engineering, page 38. Citeseer, 2014.

23. Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. Development and deploymentat facebook. IEEE Internet Computing, 17(4):8–17, 2013.

24. Rensis Likert. A technique for the measurement of attitudes. Archives of psychology,1932.

25. Abhishek Tiwari. Decoupling deployment and release- feature toggles. [Online].Available: https://www.abhishek-tiwari.com/decoupling-deployment-and-release-feature-toggles/ Accessed 24 April 2019.

26. Feature flag management systems save you from technical debt. [Online]. Available:https://rollout.io/blog/feature-flag-management-technical-debt/ Accessed 18June 2019.

27. Chromium flags. [Online]. Available: https://docs.google.com/spreadsheets/d/1XwGEpBidtPKS_xFzW8bfQ4kbuVe0OuPlpMsSd8wTmmo/edit Accessed 24 April 2019.

28. Rod Stephens. Beginning software engineering. John Wiley & Sons, 2015.

https://dzone.com/articles/feature-toggles-are-one-worst

https://martinfowler.com/bliki/ContinuousDelivery.html

https://martinfowler.com/bliki/ContinuousDelivery.html

http://code.flickr.net/2009/12/02/flipping-out/

http://code.flickr.net/2009/12/02/flipping-out/

https://martinfowler.com/articles/feature-toggles.html

https://martinfowler.com/articles/feature-toggles.html

https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/progressive-experimentation-feature-flags



https://www.abhishek-tiwari.com/decoupling-deployment-and-release-feature-toggles/

https://www.abhishek-tiwari.com/decoupling-deployment-and-release-feature-toggles/

https://rollout.io/blog/feature-flag-management-technical-debt/

https://docs.google.com/spreadsheets/d/1XwGEpBidtPKS_xFzW8bfQ4kbuVe0OuPlpMsSd8wTmmo/edit

https://docs.google.com/spreadsheets/d/1XwGEpBidtPKS_xFzW8bfQ4kbuVe0OuPlpMsSd8wTmmo/edit


29. Launchdarkly best practices. [Online]. Available: https://github.com/launchdarkly/featureflags/blob/master/5%20-%20Best%20Practices.md Accessed 24 April 2019.

30. Audit logs. [Online]. Available: https://docs.split.io/docs/streaming-audit-logsAccessed 24 April 2019.

31. Continuous deployment: Unleash your features gradually! [Online]. Available: http://bytes.schibsted.com/unleash-features-gradually/ Accessed 24 April 2019.

32. Flexible feature control at instagram. [Online]. Available: https://instagram-engineering.com/flexible-feature-control-at-instagram-a7d3417658df Accessed24 April 2019.

33. Removing risk from product launches: a webinar with launchdarkly, circleci and go-pro. [Online]. Available: https://circleci.com/blog/removing-risk-from-product-launches-a-webinar-with-launchdarkly-circleci-and-gopro/ Accessed 27 January2020.

34. Rob Loh Erik Sowa. Feature bits: Enabling flow within and across teams. [Online]. Avail-able: https://www.infoq.com/presentations/Feature-Bits Accessed 24 April 2019.

35. Tejaswini UL. There is no devops without feature flags! [Online]. Avail-able: https://www.ibm.com/developerworks/community/blogs/c914709e-8097-4537-92ef-8982fc416138/entry/THERE_IS_NO_DEVOPS_WITHOUT_FEATURE_FLAGS?lang=en Ac-cessed 24 April 2019.

36. Using the launchdarkly dashboard and json types to create light-weight applicationadministrative features. [Online]. Available: https://www.bennadel.com/blog/3465-using-the-launchdarkly-dashboard-and-json-types-to-create-light-weight-application-administrative-features.htm Accessed 27 January 2020.

37. Launchdarkly lunch-and-learn panel discussion: New york city. [Online]. Avail-able: https://www.bennadel.com/blog/3464-launchdarkly-lunch-and-learn-panel-discussion-new-york-city.htm Accessed 18 June 2019.

38. Nell B Dale, Chip Weems, and Mark R Headington. Introduction to Java and SoftwareDesign: Swing Update. Jones & Bartlett Learning, 2003.

39. Feature flags, toggles, controls - cleaning up. [Online]. Available: http://featureflags.io/feature-flags-cleaning-up/ Accessed 13 January 2020.

40. Pete Hodgson. Lean product development: Managing feature flags at scale. [Online].Available: https://www.youtube.com/watch?v=uFW4SSRtkUU Accessed 24 April 2019.

41. Mathias Meyer. Using feature flags to ship changes with confidence. [On-line]. Available: https://blog.travis-ci.com/2014-03-04-use-feature-flags-to-ship-changes-with-confidence/ Accessed 24 April 2019.

42. It’s more than feature toggles. [Online]. Available: https://www.youtube.com/watch?v=TCwMxTuDLwI Accessed 27 January 2020.

43. Get good at devops: Feature flag deployments with asp.net, webapi, & javascript. [On-line]. Available: https://channel9.msdn.com/Events/Visual-Studio/Visual-Studio-Live-Redmond-2016/W07 Accessed 27 January 2020.

44. Jason Roberts. Featuretoggle in .net. [Online]. Available: http://jason-roberts.github.io/FeatureToggle.Docs/ Accessed 24 April 2019.

45. Feature toggles in .net: tips and tricks. [Online]. Available: https://surfingthecode.com/feature-toggles-in-.net-tips-and-tricks/ Accessed 13 January 2020.

46. Introducing stormcrow. [Online]. Available: https://blogs.dropbox.com/tech/2017/03/introducing-stormcrow/ Accessed 24 April 2019.

47. Leena SN. Merge hells!! feature toggles to the rescue - pipeline conference 2017. [Online].Available: https://www.youtube.com/watch?v=R9EYY0Uu250 Accessed 24 April 2019.

48. Marcus Hammarberg and Joakim Sunden. Kanban in action. Manning PublicationsCo., 2014.

49. Gerald Schermann, Jurgen Cito, Philipp Leitner, Uwe Zdun, and Harald C Gall.WeâĂŹre doing it live: A multi-method empirical study on continuous experimenta-tion. Information and Software Technology, 99:41–57, 2018.

50. Your feature flag management needs to include retirement. [Online]. Available: https://rollout.io/blog/feature-flag-retirement/ Accessed 24 April 2019.

https://github.com/launchdarkly/featureflags/blob/master/5%20-%20Best%20Practices.md

https://github.com/launchdarkly/featureflags/blob/master/5%20-%20Best%20Practices.md

https://docs.split.io/docs/streaming-audit-logs

http://bytes.schibsted.com/unleash-features-gradually/

http://bytes.schibsted.com/unleash-features-gradually/

https://instagram-engineering.com/flexible-feature-control-at-instagram-a7d3417658df

https://instagram-engineering.com/flexible-feature-control-at-instagram-a7d3417658df

https://circleci.com/blog/removing-risk-from-product-launches-a-webinar-with-launchdarkly-circleci-and-gopro/

https://circleci.com/blog/removing-risk-from-product-launches-a-webinar-with-launchdarkly-circleci-and-gopro/

https://www.infoq.com/presentations/Feature-Bits

https://www.ibm.com/developerworks/community/blogs/c914709e-8097-4537-92ef-8982fc416138/entry/THERE_IS_NO_DEVOPS_WITHOUT_FEATURE_FLAGS?lang=en

https://www.ibm.com/developerworks/community/blogs/c914709e-8097-4537-92ef-8982fc416138/entry/THERE_IS_NO_DEVOPS_WITHOUT_FEATURE_FLAGS?lang=en

https://www.bennadel.com/blog/3465-using-the-launchdarkly-dashboard-and-json-types-to-create-light-weight-application-administrative-features.htm



https://www.bennadel.com/blog/3464-launchdarkly-lunch-and-learn-panel-discussion-new-york-city.htm

https://www.bennadel.com/blog/3464-launchdarkly-lunch-and-learn-panel-discussion-new-york-city.htm

http://featureflags.io/feature-flags-cleaning-up/

http://featureflags.io/feature-flags-cleaning-up/

https://www.youtube.com/watch?v=uFW4SSRtkUU

https://blog.travis-ci.com/2014-03-04-use-feature-flags-to-ship-changes-with-confidence/

https://blog.travis-ci.com/2014-03-04-use-feature-flags-to-ship-changes-with-confidence/

https://www.youtube.com/watch?v=TCwMxTuDLwI

https://www.youtube.com/watch?v=TCwMxTuDLwI

https://channel9.msdn.com/Events/Visual-Studio/Visual-Studio-Live-Redmond-2016/W07

https://channel9.msdn.com/Events/Visual-Studio/Visual-Studio-Live-Redmond-2016/W07

http://jason-roberts.github.io/FeatureToggle.Docs/

http://jason-roberts.github.io/FeatureToggle.Docs/

https://surfingthecode.com/feature-toggles-in-.net-tips-and-tricks/

https://surfingthecode.com/feature-toggles-in-.net-tips-and-tricks/

https://blogs.dropbox.com/tech/2017/03/introducing-stormcrow/

https://blogs.dropbox.com/tech/2017/03/introducing-stormcrow/

https://www.youtube.com/watch?v=R9EYY0Uu250

https://rollout.io/blog/feature-flag-retirement/

https://rollout.io/blog/feature-flag-retirement/


Appendix

The survey questions are as follows:1. What is your company name?2. How long has your team used feature toggles?3. What feature toggle management system is used by your team? (Check

all that apply). Options: Closed source custom system maintained by thecompany; Open source custom system maintained by the company; Thirdparty (e.g. LaunchDarkly), Open source but not maintained by the com-pany; None; Other.

4. For what purpose(s) does your team use feature toggles? (Check all that ap-ply). Options: Support CI of partially-completed features; Dark launches;A/B testing; Gradual rollout; Other.

5. Does your team make decision about using feature toggles for each feature?Options: Yes. The team checks to find if using a feature toggle is necessaryfor the new feature; No. The feature toggle is always added when a newfeature is added; Other.

6. How often does your team do the following management practices? Options:Always, Mostly, About half of the time, Rarely, and Never.– Document feature toggle’s metadata (spreadsheet, etc) to manage data

about feature toggles. (i.e. the owner of the toggle, the current value (on,off), the current status (to remove, keep) and the time of its creation).

– Logging changes to toggle values/configurations (e.g. who changes whichtoggle and when, etc.).

– Grouping toggles together in any way to simplify management or givingpermissions (i.e. related toggles, other).

– Allowing all team members (i.e. Q&A team) to have access to featuretoggles and can make changes.

7. How often does your team do the following initialization practices? Options:Always, Mostly, About half of the time, Rarely and Never.– Determining the type (permission toggle, ops toggle, release toggle, ex-

periment toggle, short-lived toggle, long-lived toggle) of the toggle atdesign step. (More information about types of toggles: https://goo.gl/4okG5Y)

– Using naming conventions for toggles (similar to variable and functionnaming conventions).

– Setting up a default value for toggle if toggle value is not found (i.e.toggle is off if its value is not found in the code).

8. How are the values of the toggles stored? (Check all that apply) Options:Configuration files; Databases; Other.

9. How are the values are assigned to the toggles in the system? (Check all thatapply) Options: Assigned boolean values (True, False); Assigned multivari-ate values (e.g. Red, Yellow, Blue); Assigned string values (e.g. ”disable-flash-3d”, ”enabled-flash-3d”); Other.

10. How does a developer access the toggle value in the code? (Check all thatapply) Options: Value is accessed by checking a primitive data type (e.g.

https://goo.gl/4okG5Y

https://goo.gl/4okG5Y


enableMyFeature == true),; Value is accessed through an object repre-senting a toggle (e.g. MyFeature.isActive()); Value is accessed through atoggle manager/mapping from key to value (e.g. Dictionary); Other.

11. How often does your team do the following clean-up practices? Options:Always, Mostly, About half of the time, Rarely and Never.– Limiting the number of existing toggles in the code.– Build or test failing if a toggle is not deleted by a specified date (Time

bomb).– Automatic reminders near date to delete the toggle.– Using tasks/stories/cards for removing toggles.– Creating a clean-up branch for removing toggle points at the time of

creation of the toggle.– Tracking unused toggles for removal.– Changing feature toggle to configuration setting to keep it in the code.


Table 5 38 Companies and their usage of identified practices from company-specific arti-facts

Management Initialization Implementation Clean-up

Company

Use

man

agem

ent

syst

ems

Doc

umen

tfe

atur

eto

ggle

’sm

etad

ata

Log

chan

ges

Giv

eac

cess

tote

amm

embe

rsD

eter

min

eap

plic

abili

tyof

feat

ure

togg

leG

roup

the

feat

ure

togg

les

Set

upth

ede

faul

tva

lues

Use

nam

ing

conv

enti

on

Det

erm

ine

the

type

ofth

eto

ggle

Typ

eof

assi

gned

valu

es

Way

sof

acce

ssin

gth

eva

lues

Stor

ety

pe

Trac

kun

used

togg

les

Add

expi

rati

onda

teC

hang

ea

feat

ure

togg

leto

aco

nfigu

rati

onse

ttin

gLi

mit

the

num

ber

offe

atur

eto

ggle

sC

reat

ea

clea

nup

bran

ch

Airbnb X X X X XApiary X X X X X X X

AppDirect X X X X X XBehalf X X X X X X

CircleCI X X X X X X XCheckr X X X X X

commercetools X X X X X XDomain X X X X

DropBox X X X X X XEnvoy X X X X X X X

Etsy X X X X X XFacebook X X X X XFINN.no X X X X X X X

Flickr X X X XGoPro X X X X X X X

Google Chrome X X X X XIBM X X X

Instagram X X X X X XInVision X X X X X X X X X XLibrato X X X X

Lyris X X X X XMain Street Hub X X X X X X

Microsoft X X X X X X XOutbrain X XPinterest X X X X X X

Rally Software X XReddit X X X X X X

Slack X XSoluto X X X

Surfline X X X X X X XThoughtWorks X X X X X X

thredUP X X X X X X XTravis-CI X X X

Twilio X X X X X XUpserve X X X X X X X

Visma X XWePay X X X X X X

Wix X X X X X XTotal (38) 32 25 21 7 8 2 22 5 1 32 28 15 2 6 0 0 0

arXiv:1907.06157v1 [cs.SE] 14 Jul 2019 · continuous integration and continuous delivery. However,...

Documents

Transcript of arXiv:1907.06157v1 [cs.SE] 14 Jul 2019 · continuous integration and continuous delivery. However,...