Automated Diagnosis of Feature Model Configurations

J. White∗,a, D. Benavidesb, D.C. Schmidta, P. Trinidadb, B. Doughertya, A. Ruiz-Cortesb

aDept. of Electrical Engineering and Computer Science, Vanderbilt University, Nashville, TN 37204, USAbDept. of Computer Languages and Systems University of Seville, Avda. de la Reina Mercedes, s/n B-41012 Seville,

Spain

Abstract

Software product-lines (SPLs) are software platforms that can be readily reconfigured for differentproject requirements. A key part of an SPL is a model that captures the rules for reconfiguringthe software. SPLs commonly use feature models to capture SPL configuration rules. Each SPLconfiguration is represented as a selection of features from the feature model. Invalid SPL configurationscan be created due to feature conflicts introduced via staged or parallel configuration or changes tothe constraints in a feature model. When invalid configurations are created, a method is needed toautomate the diagnosis of the errors and repair the feature selections.

This paper provides two contributions to research on automated configuration of SPLs. First, itshows how configurations and feature models can be transformed into constraint satisfaction problemsto automatically diagnose errors and repair invalid feature selections. Second, it presents empiricalresults from diagnosing configuration errors in feature models ranging in size from 100 to 5,000 features.The results of our experiments show that our CSP-based diagnostic technique can scale up to modelswith thousands of features.

Key words: software product-lines, configuration, diagnosis, constraint satisfiaction, optimization

1. Introduction

Background and challenges. Softwareproduct-lines (SPLs) are software platforms thatcan be reconfigured for different requirementsets (Clements and Northrop, 2002). SPLs aredesigned to facilitate the reuse of software as-sets across multiple projects and amortize devel-opment cost. For example, in the automotivedomain, an SPL can be created that allows acar’s software to provide Anti-lock Braking Sys-tem (ABS) capabilities or simply standard brak-ing, thus allowing it to be used in cars aimed at

∗Corresponding authorEmail addresses: jules@dre.vanderbilt.edu (J.

White), benavides@us.es (D. Benavides),schmidt@dre.vanderbilt.edu (D.C. Schmidt),ptrinidad@us.es (P. Trinidad),briand@dre.vanderbilt.edu (B. Dougherty),aruiz@us.es (A. Ruiz-Cortes)

different price points. Each configuration of anSPL is called a variant.

A core component of an SPL is a model of thepoints of variability within the software architec-ture. These variability points determine how thesoftware can be reconfigured to meet new require-ment sets. Points of variability are typically notindependent of one another, however, and thus anexplicit set of rules must be developed to dictatelegal combinations of configuration settings e.g., acar cannot have both ABS and standard brakingsoftware controllers.

Feature modeling (Kang et al., 1998) is a tech-nique for documenting the points of variability inan SPL, how the points of variability affect oneanother, and what constitutes a complete config-uration of the SPL. A feature model leverages fea-tures as the abstraction for documenting config-uration rules. Each feature represents an incre-

Preprint submitted to Software and Systems January 29, 2010

ment in product functionality. A feature modelcan capture different types of variability, rangingfrom ranging from requirements variability to im-plementation variability (Metzger et al., 2007).

Unique configurations of an SPL are capturedas selections of features from a feature model.SPL variants can be specified as a selection orconfiguration of features. Feature models of SPLsare arranged in a tree-like structure where eachsuccessively deeper level in the tree correspondsto a more fine-grained configuration option forthe product-line variant, as shown by the featuremodel in Figure 1. The parent-child and cross-tree relationships capture the constraints thatmust be adhered to when selecting a group of fea-tures for a variant.

Existing SPL research has focused on ensur-ing that features chosen from feature models arecorrect and consistent with the SPL and vari-ant requirements. For example, boolean cir-cuit satisfiability techniques (Mannion, 2002) andConstraint Satisfaction Problems (CSPs) (Be-navides et al., 2005; White et al., 2007) havebeen used to automate the derivation of a fea-ture set that meets a requirement set. Numer-ous tools have also been developed, such as BigLever Software Gears (Buhrdorf et al., 2003),Pure::variants (Beuche, 2003), FeAture ModelAnalyser (FAMA) (Benavides et al., 2007), andthe Feature Model Plug-in (Czarnecki et al.,2005a), to support the construction of featuremodels and correct selection of feature configu-rations.

Although SPL developers are equipped witha number of tools and techniques, configurationproblems may occur due to human errors, suchas miscommunication between engineers, or otherfactors, such as requirement changes. For exam-ple, configurations of large feature models may beproduced using staged configuration (Czarneckiet al., 2004, 2005b), where features are iterativelyselected by a group of participants. At each stage,one participant chooses features before passingthe partial configuration on to the participant inthe next stage. In these staged configuration pro-cesses, a participant may select a feature that con-flicts with a critical feature needed by a partici-

pant in a later stage. Due to the complexity offeature model constraints it may be hard to forseethese conflicts. It is also hard to debug a config-uration to figure out how to change decisions inprevious stages to make the critical feature se-lectable (Batory et al., 2006).

In many cases—particularly when SPLs areproduced from supply-chains—the configurationdecisions of multiple developers working in par-allel must be integrated. Since developers maywork in different departments, organizations, orgeographic locations where they are not in con-tinuous direct communication or have conflictinggoals, configurations may be produced with con-flicting feature selections. For example, hardwaredevelopers of one subcontractor for an automobilemay desire a lower cost set of Electronic ControlUnits (ECUs) that cannot support the embed-ded controller code features needed by the soft-ware developers of another subcontractor. Eventhough the two teams may share the same toolset,because the tools cannot prevent errors in a dis-tributed environment where the teams are not incontinuous direct communication, conflicts mayarise. Methods are therefore needed to (1) evalu-ate and debug conflicts between participants and(2) recommend modifications to the participantsfeature selections to make them compatible.

A further challenge to maintaining error freeSPL configurations is that external influences(such as supply-chain disruption, changes in mar-ket demand, or safety regulations) may changeover time and thus invalidate previously correctconfigurations. For example, if a new governmentregulation is passed that mandates higher emis-sions standards, it may necessitate changing exist-ing software configurations. A critical question isthus often determining the lowest cost method ofupdating existing software configurations to meetthe new external regulations and requirements,which is a process termed software configurationevolution.

Although prior research has shown how to iden-tify flawed configurations (Batory, 2005; Man-nion, 2002), conventional debugging mechanismscannot pinpoint configuration errors and identi-fying corrective actions. More specifically, tech-

2

niques are needed that can take an arbitraryflawed configuration and produce the minimal setof feature selections and deselections to bring theconfiguration to a state that satisfies all of thefeature model rules. This paper focuses on ad-dressing these gaps in existing research. The so-lutions that the paper presents to these gaps aredescribed in Section 2.

Solution overview and contributions. Ourapproach to debugging feature model configu-rations and automating configuration evolutiontransforms an invalid feature model configu-ration into a Constraint Satisfaction Problem(CSP) (Van Hentenryck, 1989) and then usesa constraint solver to derive the minimal setof feature selection modifications that will bringthe configuration to a valid state. We call thisconstraint-based diagnostic approach the Con-figuration Understanding and REmedy (CURE).This paper shows how CURE provides the follow-ing contributions to work on debugging errors infeature model configurations:

1. We provide a CSP-based diagnostic tech-nique that can pinpoint conflicts and con-straint violations in feature models

2. We provide a simplified diagnostic CSP thatis not as flexible as the full diagnostic CSPbut reduces diagnosis time

3. We show how CURE can remedy a config-uration error by automatically deriving theminimal set of features to select and deselect

4. We provide mechanisms for using CURE tomediate conflicting configuration participantfeature selection desires via cost optimizationtechniques

5. We show how CURE allows stakeholders todebug a configuration error or conflict fromdifferent viewpoints

6. We show how CURE can be used to automatefeature selection evolution

7. We provide empirical results showing thatCURE’s scalability can support industrialSPL feature models containing over 5,000features.

8. We present empirical results showing the im-provement in solving time that the simplifiedCSP provides

Our prior work on automated feature configu-ration (Trinidad et al., 2007) focused on detect-ing the most common types of errors in the con-straints of a feature model, such as void features,using pre-defined configurations. In further re-search (White et al., 2008), we developed newtechniques for diagnosing the configurations offeature models rather than the feature model it-self. In this paper, we also investigate featureexpansion planning techniques and adapting tosupply-chain disruption in the context of featuremodels. Furthermore, the prior work did not ac-count for how manual configuration decisions thatled to the inability of satisfying project require-ments were handled. Batory et al. (Batory et al.,2006) have described this is an important chal-lenge in the field automated software product-lineconfiguration. In this work, we address this gapin our prior work by showing how automated re-mediation can be performed to eliminate config-uration conflicts that arise from human error orconcurrent configuration processes.

This paper extends our previous work on auto-mated diagnosis of product-line configuration er-rors (White et al., 2008) in three ways. First,we provide a new methodology for applying ourCURE diagnosis technique to automating the evo-lution of feature model configurations, which al-lows SPL configurations to evolved automaticallyover time to meet changing external requirementsets. Second, we have significantly expanded thediscussion of CURE’s relationship with relatedwork on SPL configuration.

Third, in some scenarios, developers may desireto tradeoff diagnosis flexibility for improved solv-ing performance. For example, developers maywish to eliminate diagnostic capabilities that theydo not use in exchange for faster diagnostics forlarge feature models. We provide a new simplifieddiagnostic CSP formulation that can significantlyreduce solving time for large feature models andpresent new empirical results demonstrating theimprovement in solving time provided by the newCSP formulation. For all the techniques proposedin this paper, we make the implicit assumptionthat the constraints in the feature model itselfare correct. The process described by Trinidad

3

et al. (Trinidad et al., 2007) can be used to deter-mine if the feature model is error free and suitablefor the techniques described in this paper.

Paper organization. The remainder of thepaper is organized as follows: Section 2 presentsthe CURE CSP-based technique for diagnosingconfiguration errors and conflicts; Section 3 de-scribes CURE CSP modifications that can bemade to decrease solving time; Section 4 showshow CURE can be used to perform conflict media-tion, multi-viewpoint debugging, and staged con-figuration debugging; Section 5 shows how CUREcan be used to automate software feature configu-ration evolution decisions; Section 6 presents em-pirical results demonstrating the ability of CUREto scale to feature models with thousands of fea-tures; Section 7 compares CURE with relatedresearch; and Section 8 presents concluding re-marks.

2. Configuration Error Diagnosis with

CURE

Configuration Understanding and REmedy(CURE) is a constraint-based technique for di-agnosing errors and mediating conflicts in featuremodel configurations. Developers can use CUREto identify the minimal set of features needed toselect or deselect to transform an invalid inputconfiguration into a valid configuration. More-over, depending on the input provided to CURE,a flawed configuration can be debugged from dif-ferent viewpoints or conflicts between multiplestakeholder decisions in a configuration processcan be mediated.

The key component of CURE is the applica-tion of a CSP-based error diagnostic technique.In prior work, Benavides et al. (Benavides et al.,2005) have shown how feature models can betransformed into CSPs to automate feature se-lection with a constraint solver (Jaffar and Ma-her, 1994). Trinidad et al. (Trinidad et al., 2007)subsequently described how to extend this CSPtechnique to identify full mandatory features, voidfeatures, and dead feature models using Reiter’stheory of diagnosis (Reiter, 1987). This sectionpresents an alternative CSP diagnostic model, not

related to Reiter’s theory of diagnosis, for deriv-ing the minimum set of features that should beselected or deselected to eliminate a conflict in afeature configuration.

CURE can diagnose feature models that usestandard feature modeling notations, such as re-quired features, XOR groups, and optional fea-tures. It is possible to also encode cardinalitygroups into the diagnostic CSPs used by CURE,but we do not cover this in the paper. Featurequality attributes can also be taken into accountin the diagnostic process, and we discuss cost-based quality attributes in Section 4.2.

2.1. Background: Feature Models and Configura-tions as CSPs

A CSP is a set of variables and a set of con-straints over those variables. For example, A +B ≤ 3 is a CSP involving the integer variables A

and B. A constraint solver finds a valid labeling(set of variable values) that simultaneously satis-fies all constraints in the CSP. A valid labeling ofthe CSP would be (A = 1, B = 2).

To build a CSP to model a feature model config-uration problem, we construct a set of variables,F , representing the features in the feature model.Each configuration of the feature model is a setof values for these variables, where a value of 1indicates the feature is present in the configura-tion and a value of 0 indicates it is not present.More formally, a configuration is a labeling of F ,such that for each variable fi ∈ F , fi = 1 in-dicates that the ith feature in the feature modelis selected in the configuration. Correspondingly,fi = 0 implies that the feature is not selected.

Given an arbitrary configuration of a featuremodel as a labeling of the F variables, devel-opers need the ability to ensure the correctnessof the configuration with respect to the featuremodel rules. To achieve this constraint checkingability, each variable fi is associated with one ormore constraints corresponding to the configura-tion rules in the feature model. For example, if fj

is a required subfeature of fi, then the CSP wouldcontain the constraint: fi = 1 ⇔ fj = 1.

Configuration rules from the feature model arecaptured in the constraint set C. For any given

4

feature model configuration described by a label-ing of F , the correctness of the configuration canbe determined by seeing if the labeling satisfiesall constraints in C. More detailed descriptionsof the steps for transforming a feature model to aCSP appear in (Benavides et al., 2005).

2.2. Configuration Diagnostic CSP

This section describes CURE’s diagnostic CSP.First, we introduce the diagnostic recommenda-tion variables that CURE uses to indicate whichfeatures should be selected or deselected to fix aconfiguration. Next, we describe how the diagno-sis of a flawed configuration is modeled as a CSPusing these variables. Finally, we explain how theconstraints on the diagnostic variables produce acorrect diagnosis of an invalid configuration.

2.2.1. Diagnostic Recommendation Variables

As a motivating example, we use a featuremodel for an automobile. Figure 1 shows asimple feature model for an automobile. As-sume that an invalid configuration with thefeatures Automobile, Brake Control Software,Non-ABS Controller, Brake ECU, and 1 Mbit/s

CAN Bus has been created.

Figure 1: Simple Feature Model for an Automobile

To diagnosis this flawed configuration, develop-ers need a list of features that should be selectedor deselected to make the invalid configuration avalid configuration. CURE takes an invalid con-figuration and a set of constraints describing afeature model as input and produces a valid con-figuration along with the features to select anddeselect to reach the valid output configurationfrom the input configuration.

The invalid input configuration is provided toCURE as a set of labeled variables, O. In turn,

CURE produces an output configuration as a setof labeled variables F , that describe a valid config-uration that can be reached by modifying the in-put configuration O. Since the goal is to producea valid output configuration, the feature modelconstraints are input into CURE as constraintson the output variables F . That is, the featuremodel constraints define the rules that the correctoutput configuration must conform to. Finally,CURE also outputs the features that should beselected, S, and deselected, D, to modify the in-valid input configuration, O, to match the validoutput configuration, F .

The overall diagnostic process for CURE isshown in Figure 2.

Figure 2: Diagnostic Technique Architecture for CURE

In Step 1 of Figure 2, the rules of the featuremodel and the invalid configuration are trans-formed into a CSP. For example, o1 = 1 becausethe Automobile feature is selected in the invalidinput configuration. In Step 2, the solver derivesa labeling of the diagnostic CSP. Step 3 takes theoutput of the CSP labeling and transforms it intoa series of recommendations of features to selector deselect to turn the invalid configuration intoa valid configuration. Finally, in Step 4, the rec-ommendations are applied to the invalid configu-

5

ration to create a valid configuration where eachvariable fi equals 1 if the corresponding featureis selected in the new and valid output configu-ration. For example, f7 = 1 means that the 250Kbit/s CAN Bus is selected in the new valid con-figuration.

The variables, oi ∈ O, are used to input thecurrent invalid configuration of the feature modelinto CURE. If the ith feature is currently selectedin the invalid configuration, oi = 1. If the featureis not selected, oi = 0. An SPL engineer pro-vides these values to CURE as the input for thealgorithm.

To enable the constraint solver to recommendfeatures to select and deselect, two sets of out-put recommendation variables, S and D, are in-troduced to capture the features that need to beselected and deselected, respectively, to reach avalid configuration. For example, a value of 1for variable si ∈ S indicates that the ith featureshould be added as a selected feature to the in-valid input configuration captured in O. Simi-larly, di = 1 implies that the feature ith should nolonger be a selected feature in the invalid inputconfiguration.

The other output of CURE is the actual featureselection that will be reached by applying the rec-ommended feature selection changes to the invalidinput configuration. The variables, fi ∈ F , spec-ify the new feature selection that is reached byapplying the S and D recommendations to thevariables in O. That is, the F variables containthe new feature selection that CURE recommendsthe SPL engineers should use in place of the in-valid configuration. Table 1 lists the completeset of diagnostic variables that would be used byCURE for the feature model in Figure 1.

2.2.2. Diagnostic CSP Constraints

To diagnose the CSP, we want to find an alter-nate but valid configuration of the feature modeland suggest a series of changes to the invalid in-put configuration to reach the valid configuration.A valid output configuration is a labeling of thevariables in F (a configuration) such that all of

Variables

Variable Ex-planations

fi ∈ F : feature variables for thevalid configuration that will beoutput by CURE;oi ∈ O: the features selected inthe invalid input configuration;si ∈ S: features to select to reachthe valid configuration;di ∈ D: features to deselect toreach the valid configuration

Inputs

CurrentConfig.

o1 = 1, o2 = 1, o3 = 0, o4 =1, o5 = 1, o6 = 1, o7 = 0

Feature f1 = 1 ⇔ (f2 = 1),Model f1 = 1 ⇔ (f5 = 1),Rules: f2 = 1 ⇒ (f3 = 1) ⊕ (f4 = 1),

f5 = 1 ⇒ (f6 = 1) ⊕ (f7 = 1),(f6 = 1) ∨ (f7 = 1) ⇒ (f5 = 1),(f3 = 1) ∨ (f4 = 1) ⇒ (f2 = 1),f3 = 1 ⇒ (f6 = 1),f4 = 1 ⇒ (f7 = 1)

Diagnostic ∀fi ∈ F

Rules: {(fi = 1) ⇒ (oi = 1 ⊕ si = 1)∧

(di = 0)(fi = 0) ⇒ (oi = 0 ⊕ di = 1)∧

(si = 0)}

Outputs

Features toSelect:

s1 = 0, s2 = 0, s3 = 0, s4 =0, s5 = 0, s6 = 0, s7 = 1

Features toDeselect:

d1 = 0, d2 = 0, d3 = 0, d4 =0, d5 = 0, d6 = 1 , d7 = 0

New ValidConf.:

f1 = 1, f2 = 1, f3 = 0, f4 =1, f5 = 1, f6 = 0, f7 = 1

Table 1: Diagnostic CSP Construction

the feature model constraints are satisfied. Foreach variable fi, the value should be 1 if the fea-ture is present in the new valid configuration thatwill be transitioned to. If a feature is not in theoutput configuration, fi should equal 0.

We always require f1 = 1 to ensure that theroot feature is always selected in the output con-figuration. For void feature models, it will be im-

6

possible for the root feature to be selected in theoutput configuration and there will be no validsolution. In these cases, the solver will respondthat no solution was found. CURE could also beused to detect void feature models but it wouldbe more appropriate to use a technique designedfor this purpose (Trinidad et al., 2007).

Once a valid labeling of F is found, the goalis to determine how to modify the labeling of O

to match the valid feature selection denoted bythe labeling of F . First, a constraint must beintroduced to model when a feature in the in-valid input configuration needs to be deselectedto reach the correct output configuration. If theithfeature is included in the invalid input config-uration (oi = 1), but is not in the output con-figuration (fi = 0), we want the solver to recom-mend that it be deselected (di = 1). For everyfeature, we introduce the following constraint todetermine if the ith feature in O needs to be des-elected1:

(fi = 0) ⇒ (oi = 0 ⊕ di = 1) ∧ (si = 0)

If fi is not selected in the output configuration(fi = 0), then either the feature was also not se-lected in the invalid input configuration (oi = 0),or the feature needs to be deselected (di = 1).Furthermore, if a feature is not needed in the out-put configuration (fi = 0) then clearly it shouldnot be a recommended selection (si = 0).

The solver must also recommend features to se-lect. If the ith feature is selected in the outputconfiguration fi = 1, and not selected in the in-valid input configuration (oi = 0), then it needsto be selected (si = 1). For each feature, we in-troduce the constraint:

(fi = 1) ⇒ (oi = 1 ⊕ si = 1) ∧ (di = 0)

If a feature is needed by the output configuration(fi = 1), then either the feature was present inthe invalid configuration (oi = 1) or the featurewas not present in the invalid configuration andneeds to be selected (si = 1). Clearly, a featureshould not be deselected if fi = 1 and thus di = 0.

1The symbol ”⊕” denotes exclusive or

The state of each feature, oi, in the invalid in-put configuration is compared against the correctstate of the feature, fi, in the output feature con-figuration. The behavior of each comparison canfall into four cases:

1. A feature is selected and does not need

to be deselected. If the ith feature is inthe invalid input configuration (oi = 1), andalso in the output configuration (fi = 1), nochanges need be made to it (si = 0, di = 0)

2. A feature is selected and needs to be

deselected. If the ith feature is in the in-valid input configuration (oi = 1) but not inthe output configuration (fi = 0), it must bedeselected (di = 1)

3. A feature is not selected and does not

need to be selected. If the ith feature isnot in the invalid input configuration (oi = 0)and is also not needed in the output config-uration (fi = 0) it should remain unchanged(si = 0, di = 0)

4. A feature is not selected and needs to

be selected. If the ith feature is not selectedin the invalid input configuration (oi = 0) butis present in the output configuration (fi =1), it must be selected (si = 1)

2.3. Optimal Diagnosis Method

The next step in the CURE diagnosis processis to use the solver to label the variables and pro-duce a series of recommendations. For any givenconfiguration with a conflict, there may be mul-tiple possible ways to eliminate the problem. Wemust therefore tell the solver how to select whichof the (many) possible corrective solutions to sug-gest to developers.

The most basic suggestion selection criteria de-velopers can use to guide the solver’s diagnosisis to tell it to minimize the number of changesto make to the current configuration, i.e., prefersuggestions that require changing as few thingsas possible in the invalid input configuration. Toimplement this approach, we solve for a CSP la-beling that minimizes:

n∑

0

si +n∑

0

di

7

which is the total number of changes that the solu-tion requires the developer to make. By minizingthis sum we therefore minimize the total numberof required changes.

Each labeling of the diagnostic CSP will pro-duce two sets of features corresponding to the fea-tures that should be selected (S) and deselected(D) to reach the new valid configuration. Devel-opers can ask the solver to cycle through the dif-ferent potential labelings of the diagnostic CSPto evaluate potential remedies. Moreover, eachnew labeling (new diagnosis) causes the solverto backtrack and create new values for F , whichallows developers to evaluate not only the sug-gested modifications but the configuration thatthe remedy will produce. Another way to furtherrefine the guidance for the diagnosis is to con-strain the new state captured in the labeling ofF . This technique is utilized by the extensions inSections 4.1 and 4.2.

Table 1 shows a complete set of inputs andoutput suggestions for diagnosing the automotivesoftware example from Section 2.2.1. If there aremultiple labelings of the CSP, initially only onewill be returned. After the first solution has beenfound, however, the solver can much more effi-ciently cycle through the other equally ranked setsof corrective suggestions.

3. CURE Performance Enhancements

This section describes a number of CSP modifi-cations that can be used to decrease the diagnostictime of CURE.

3.1. Simplified CSP Formulation

To improve the diagnosis speed of the process,the diagnostic CSP can be modified to includefewer variables, which decreases the flexibility ofthe types diagnostic constraints and goals thatcan be utilized, but increases performance. Aspart of the diagnosis process, the solver must cal-culate values for the D and S variables, whichdetermine which features to select and deselect.The process of determining whether to select ordeselect a feature can easily be performed in codeoutside of the solver. For example, after the solver

has determined the closest valid configuration,Java code can be used to compare it to the invalidconfiguration and recommend which features toselect or deselect via a simple difference of theoi and fi variables. Moving these variables out ofthe diagnostic CSP can improve performance, buteliminates the ability to define constraints or gaolsbased on the selection and deselection of features.

To simplify the CURE CSP, the D and S vari-ables can be replaced with a single set of vari-ables, N , noting the feature selections that differbetween the invalid and closest valid configura-tion. The N variables do not indicate whether toselect or deselect a feature, but merely that its se-lection state differs in the two configurations. Inthe simplified CSP formulation, for each featurein the feature model, a corresponding ni variableis created. Each variable ni ∈ N can have valuezero or one. Moreover,

(ni = 1) ⇒ (oi 6= fi)

, i.e., ni will only have value 1 if the selection stateof the corresponding feature differs in the invalidand closest valid configurations.

Removing the S and D variables has an impacton the expressiveness of the function that can beused to define what constitutes the closest validconfiguration. For example, if we define the opti-mization goal, G, as the function:

G =n∑

0

si +n∑

0

2di

the solver can be instructed to preference selectingrather than deselecting features to fix the config-uration. That is, the solver will attempt to selectnew features wherever possible rather than des-elect features that are already in the configura-tion. Likewise, the function tells the solver thatadding 3 features is more expensive than dese-lecting 1. Developers can therefore exercise fine-grained control over the weighting of the selectionand deselection of features when diagnosing an in-valid configuration.

By removing the D and S variables, the simpli-fied CSP formulation eliminates the ability to put

8

weights on the relative importance of feature se-lection and deselection in the diagnosis. As shownby the results in Section 6.5, this tradeoff can yielda significant improvement in diagnosis speed. Thesimplified CSP formulation is preferable, there-fore, unless fine-grained weighting of feature se-lection and deselection is needed.

3.2. Bounding Diagnostic Method

Due to time constraints, it may not be pos-sible to find the optimal number of changes forextremely large feature models. In these cases, amore scalable approach is to attempt to find anysuggestion that requires fewer than K changes orwith a cost less than K. Rather than directly ask-ing for an optimal answer, we add the constraint:

n∑

i=1

si + di ≤ K

to the CSP and ask the solver for any solution.

The sum of all variables si ∈ S and di ∈ D

represents the total number of feature selectionsand deselections that need to be made to reachthe new valid configuration. The sum of both ofthese sets is thus the total number of modifica-tions that must be made to the original invalidconfiguration. The new constraint, ensures thatthe solver only accepts diagnosis solutions that re-quire the developer to make K or fewer changesto the invalid solution.

The solver is asked for any answer that meetsthe new constraints. In return, the solver will pro-vide a solution that is not necessarily optimal, butwhich fits our tolerance for change. If no solutionis found, we can increment K by a factor and rein-voke the solver or reassess our requirements. Theresults in Section 6.4 show that it is significantlyfaster to search for a bounded solution rather thanan optimal solution.

If the solver cannot find a diagnosis that makesfewer than K modifications, it will state thatthere is no valid solution that fits a K changebudget.

4. Applying CURE to Industrial Feature

Configuration Diagnostic Problems

4.1. Debugging from Different Viewpoints

The feature labeled as the source of an error ina feature model configuration may vary depend-ing on the viewpoint used to debug it. For exam-ple, if a configuration in the feature model shownin Figure 1 is created to include both Non-ABSController and 1 Mbit/s CAN Bus, either featurecan be viewed as the feature that is the source ofthe error.

If we debug the configuration from the view-point that software features trump ECU hardwaredecisions, then the 1 Mbit/s CAN Bus feature isthe error. If we assume that ECU decisions pre-cede software features, however, then the Non-ABS Controller feature is the error. A featuremodel may therefore require debugging from mul-tiple viewpoints since diagnosing the feature thatcauses an error in a feature model depends on theviewpoint used to debug it.

CURE can be provided with aditional con-straints to allow for viewpoint-based diagnostics.Each viewpoint represents a set of features thatthe solver should avoid suggesting to add or re-move from the current configuration. For exam-ple, using the automobile scenario shown in Fig-ure 1, the solver can debug the problem from thepoint of view that hardware decisions trump soft-ware features by telling the solver not to suggestselecting or deselecting any hardware features.

Debugging from a viewpoint in CURE works bypre-assigning values for a subset of the variablesin F and O. For example, to force the feature fi

currently in the configuration to remain unalteredby the diagnosis, the values fi = 1 and oi = 1 areprovided to the solver. Since (fi = 1) ⇒ (oi =1 ⊕ si = 1) ∧ (di = 0), pre-assigning these valueswill force the solver to label si = 0 and di = 0.

To debug from a given point of view, for eachfeature fv, in that viewpoint, we first add the con-straints, fv = 1, ov = 1, sv = 0, and dv = 0.Thesolver then derives a diagnosis that recommendsalterations to other features in the configurationand maintains the state of each feature fv.

9

4.2. Cost Optimal Conflict Resolution

Conflicts can occur when multiple stakehold-ers in a configuration process pull the solution indifferent directions. Debugging tools are there-fore needed to mediate the conflict in a cost con-scious manner. For example, when a car’s soft-ware configuration is incompatible with the legacyECU configuration, it is (probably) cheaper tochange the software configuration than to changethe ECU configuration and the assembly processof the car. The solver should therefore try to min-imize the overall cost of the changes.

We can extend the CSP model to perform cost-based feature selection and deselection optimiza-tion. First, we extend the CURE model to asso-ciate a cost variable, bi ∈ B, with each feature inthe feature model. Each cost variable representshow expensive (or conversely how beneficial) it isfor the solver to recommend the state of that fea-ture be changed. Before each invocation of the de-bugger, the stakeholders provide these cost vari-ables to guide the solver in its recommendationsof features to select or deselect.

Next, we construct the superset of the featuresthat the various stakeholders desire, as shown inFigure 3. The superset represents the ideal, al-though incorrect, configuration that the stake-holders would like to have. The goal is to finda way to reach a correct configuration from thissuperset of features that involves the lowest to-tal cost for changes. The superset is input to thesolver as values for the variables in O.

Finally, we alter our original optimization goalso that the solver will attempt to minimize (ormaximize) the cost of the features it suggests se-lecting or deselecting. We define a global costvariable G and let G capture the sum of the costsof the changes that the solver suggests, as follows:

G =n∑

i=1

(di ∗ bi) + (si ∗ bi)

G is thus equal to the sum of the costs of all fea-tures that the solver either recommends to selector deselect. Rather than instructing the solver tominimize the sum of S ∪D, we ask it to minimizeor maximize G.

The result of the labeling is a series of changesneeded to reach a valid configuration that opti-mally integrates the desires and decisions of thevarious stakeholders. Of course, one particularstakeholder may incur more cost than another inthe interest of reaching a globally better solution.Further constraints, such as limiting the maxi-mum difference between the cost incurred by anytwo stakeholders, could also be added. The medi-ation process can be tuned to provide numeroustypes of behavior by providing different optimiza-tion goals.

4.3. Staged Configuration Conflict Resolution

Another type of conflict can occur in stagedconfiguration (Czarnecki et al., 2004). Stagedconfiguration is a configuration process where de-velopers iteratively select features to reduce thevariability in a feature model until a variant isconstructed. The need for staged configurationhas been demonstrated (Czarnecki et al., 2004,2005b) in the context of software supply chainsfor embedded automotive software. In the firststage, software vendors provide software compo-nents that can be provided in different config-urations to actuate brakes, control infotainmentsystems, etc. In the second stage, hardware ven-dors that the software runs on must provide ECUswith the correct features and configuration to sup-port the software components selected in the firststage.

The challenge with staged configuration is thatfeature selection decisions made at some point intime T may conflict with decisions that a stake-holder desires to make at a point in time T ′ > T .For example, it is possible for software vendors tochoose a set of software component features forwhich there are no valid ECU configurations inthe second configuration stage. Identifying thefewest number of configuration modifications toremedy the error is hard because there can besignificant distance between T and T ′.

Pre-assigning values for variables in F and O

can also be used to debug staged configurationerrors. With staged configuration errors, at somepoint in time T ′, developers need to select a fea-ture that is in conflict with one or more fea-

10

Figure 3: Constructing the Feature Selection Superset for Conflict Mediation

tures selected at time T < T ′. To debug thistype of conflict, developers pre-assign the desired(but currently unselectable) feature at time T ′ thevalue of 1 for its oi and fi variables. Developerscan also pre-assign values for one or more otherfeatures decisions from previous stages of the con-figuration that must not be altered. The solver isthen invoked to find a configuration that includesthe desired feature at T ′ and minimizes the num-ber of changes to feature configuration decisionsthat were made at all points in time T < T ′.

5. Using CURE for Software Configuration

Evolution

Software feature configuration evolution is acritical part of SPL management that involvesmodifying an existing software configuration tomeet new requirements. Although building a vari-ant for a new requirement set is a standard fea-ture model configuration problem, it can not al-ways be performed without taking into considera-tion existing configurations. For example, if a keysupply-chain vendor for an automobile goes out ofbusiness, automotive manufacturers must accountfor the configuration of existing model year carswhen determining how to produce new configu-rations that eliminate the parts produced by thedefunct vendor. Simply producing new configu-rations with no regard for the existing manufac-turing, procurement, and other processes invest-ments would be costly and error-prone.

The goal of feature configuration evolution isto transform a current configuration A that sat-isfies a requirement set R into a new configura-

tion A′ that satisfies an evolved requirement setR′. Moreover, because there may be a substantialexisting investment in A, a goal of the evolutionprocess is to find A′, such that it minimizes ormaximizes a function, ∆(A, A′). For example, de-velopers may wish to choose evolution paths thatrequire adding as little additional cost to A aspossible and thus ∆(A, A′) computes the differ-ence in cost between the legacy and evolved con-figurations.

A key difference between configuration diagno-sis and configuration evolution is that both A andA′ are valid feature model configurations. Config-uration diagnosis takes an invalid configurationO and finds a valid configuration F that is asclose as possible. Although CURE was originallydeveloped for diagnosing errors in configurations,this section shows that it can also be applied tosoftware configuration evolution problems wherethe goal is to find A′ while optimizing a function∆(A, A′). This section also presents methods forusing CURE to evolve feature model configura-tions to handle a number of SPL scenarios (suchas supply-chain disruption and feature expansion)that necessitate configuration evolution.

5.1. Supply-chain Disruption

The features that are present in an SPL’s fea-ture model are dependent on an organization’sability to procure specific sets of software andhardware components. For example, the produc-tion of an automobile may depend on vendorsproviding the ECU and associated software for ahigh-end infotainment system. If any vendors goout of business, decide to no longer produce one

11

of the requisite parts, or cannot produce sufficientquantities of a part, SPL developers must deter-mine how to evolve the automobile’s configurationto eliminate the unavailable part.

When a part becomes unavailable, developersfirst need to determine which existing features orcombinations of features in the feature model areno longer viable selections due to the unavailableparts. In a simple scenario, a single feature mayno longer be available. For example, in the featuremodel shown in Figure 5, if the ECU requiredfor the Advanced Infotainment feature becomesunavailable, this feature will be invalid in a featureselection.

In more complicated situations, alternativeparts may be available from other vendors, butthese new parts may have different requirementsthan the old parts. For example, in the scenariodepicted in Figure 4 the ECU used for an au-tomobile’s infotainment system becomes unavail-able and an alternative ECU must be used. The

Figure 5: Supply-chain Disruption Leading to Feature Un-availability

new ECU has more substantial peak power re-quirements, however, which requires adding a newrelationship to the feature model. When supply-chain disruption occurs, developers must deter-mine cost-effective ways of evolving legacy config-urations to meet the new requirements.

CURE can be used to automate the process offixing configurations that have become invalid due

to supply-chain disruptions. CURE’s automateddiagnosis abilities allow developers to analyze evo-lution alternatives that optimize specific proper-ties, such as cost. The key to CURE’s abilityto automate evolution is that the legacy softwareconfiguration, A, can be modeled by the invalidconfiguration O and the target evolved configu-ration, A′, can be modeled as the closest validconfiguration F .

Cure’s goal function, G, which is typically afunction of O and F , can also be used to model∆(A, A′) by defining ∆(A, A′) in terms of the O,F , S, and D variables. Thus, we model the evo-lution process as:

C = Feature Model Constraints

O = A

F = A′

G = ∆(A, A′)

R = C

R′ = C ∧ Additional CSP Constraints

For example, the Advanced Infotainment feature,f5, in Figure 5 is no longer a valid feature selec-tion because a required ECU is not available. Toautomate the evolution of each configuration thathas this feature selected, CURE can be used toderive the closest valid configuration for which:

R′ = C ∧ (f5 = 0)

That is, find a feature model configuration whichsatisfies the feature model constraints, C, and alsodoes not include f5 in the feature selection f5 = 0.Moreover, designers can use the cost coefficientsintroduced in Section 4.2 to represent the rela-tive prices of different evolution options and useCURE to perform cost optimal evolution automa-tion by making G calculate the difference in costbetween the legacy and evolved configuration.

The more complex evolution scenario outlinedin Figure 4 can also be automated with CURE. Inthis case, developers need to evolve any configu-rations that include both the Advanced Infotain-ment and 50W features. Using CURE, develop-ers add the following additional constraint to the

12

Figure 4: Supply-chain Disruption Leading to Feature Model Rule Changes

diagnostic CSP:

R′ = C ∧ ((f5 = 1) ⇒ (f10 = 1))

This additional constraint forces the CSP solverto ensure that any evolved configurations that itsuggests adhere to the new feature model rule.Again, developers can apply cost coefficients orconstraints requiring that Advanced Infotainmentbe selected in evolved configurations (f5 = 1).

5.2. Feature Expansion Planning

Another important evolution problem thatshould have automation support is planning fea-ture expansion. Feature expansion planning is de-termining how to incorporate new features to sat-isfy new requirements, such as changing marketdemands, into existing configurations. For exam-ple, automotive designers may want to incorpo-rate automated parking assistance into the nextyear’s model of a car. As with supply-chain dis-ruption, developers cannot simply ignore the ex-isting configuration of the current year’s model.Instead, developers need to determine the mini-mal or lowest cost set of modifications that canbe applied to the current configuration to enablethe newly required features.

To automate feature expansion planning withCURE, the following new requirement set

R′ = C ∧ (∀fi ∈ New, fi = 1)

is produced that requires that each desired newfeature, fi ∈ New, is selected, i.e., the fi variablefor each desired feature, regardless if it is optional,

required, or alternative, must have value 1, mean-ing that the feature is selected. If there are hardrequirements on feature selections that can not bealtered, these are also encoded into R′ as:

R′ = C∧(∀fi ∈ New, fi = 1)∧(Hard Requirements)

For example,

R′ = C ∧ (f8 = 1) ∧ (s2 = 0 ∧ d2 = 0)

finds a configuration in Figure 5 that adds theRear Infotainment feature without modifying theForward Infotainment feature. The constraint(f8 = 1) requires that the Rear Infotainment fea-ture be selected by the CSP solver and the con-straint (s2 = 0 ∧ d2 = 0) ensures that no changesare recommended for the Forward Infotainmentfeature. As with previous examples, cost coeffi-cients and the goal function G can be defined topreference the lowest cost evolved configuration.

6. Runtime Evaluation with Randomized

Models

Effective automated diagnostic methods shouldscale to handle feature models of production sys-tems. This section presents empirical resultsfrom experiments we performed to evaluate thescalability of CURE. We compare the scalabilityof both CURE’s optimal and bounding methodsfrom Sections 2.3 and 3.2. Although we assumethat the technique is exponential in time complex-ity, the experiments are designed to demonstratethat the technique is still scalable enough to sup-port industrial feature models with 5,000 features.

13

6.1. Experimental Platform

To perform our experiments, we used the im-plementation of CURE provided by the ModelIntelligence libraries from the Eclipse Founda-tion’s Generic Eclipse Modeling System (GEMS)project (eclipse.org/gmt/gems). Internally,the GEMS Model Intelligence implementationof CURE uses the Java Choco ConstraintSolver (http://choco.sourceforge.net) to de-rive labelings of the diagnostic CSP. The experi-ments were performed on a computer with an In-tel Core DUO 2.4GHZ CPU, 2 gigabytes of mem-ory, Windows XP, and a version 1.6 Java VirtualMachine (JVM). The JVM was run in client modeusing a heap size of 40 megabytes (-Xms40m)and a maximum memory size of 256 megabytes(-Xmx256m).

A challenging aspect of scalability analysis isthat CSP-based techniques can vary in solvingtime based on individual problem characteristics.In theory, CSP’s have exponential worst-case timecomplexity, but are often much faster in practice.To evaluate CURE it was therefore necessary toapply it to as many models as possible. The keychallenge with this approach is that hundreds orthousands of real feature models are not readilyavailable and manually constructing them is im-practical.

To provide the large numbers of feature mod-els needed for our experiments we built a fea-ture model generator that randomly creates fea-ture models with the desired branching and con-straint characteristics. We also imbued the gen-erator with the capability to generate feature se-lections from a feature model and probabilisti-cally insert a bounded number of errors/conflictsinto the configuration. The feature model gener-ator and code for these experiments is availablein open-source form from (code.google.com/p/ascent-design-studio).

From our preliminary feasibility experiments,we observed that the branching factor of the treehad little effect on the algorithm’s solving time.We also compared diagnosis time using modelswith 0%, 10%, and 50% cross-tree constraints andobserved that each increment in the percentage ofcross-tree constraints improved performance. For

example, with the optimal method and 1,000 fea-ture models, the average diagnosis time graduallydecreased from 47 seconds with 0% cross-tree con-straints to 36 seconds with 50% cross-tree con-straints. The key indicator of the solving com-plexity was the number of XOR- or cardinality-based feature groups in a model.2

Our tests limited the branching factor to atmost five subfeatures per feature. We also setthe probability of XOR- or cardinality-based fea-ture groups being generated to 1/3 at each fea-ture with children. We chose 1/3 since most fea-ture models we have encountered contain more re-quired and optional relationships than XOR- andcardinality-based feature groups. The total num-ber of cross-tree constraints was set at 10%. Wealso eliminated all diagnosis results from void fea-ture models, since void feature models producedfaster diagnostic times and would have skewed theresults towards smaller solving times.

To generate feature selections with errors, weused a probability of 1/50 that any particular fea-ture would be configured incorrectly. For eachmodel, we bounded the total errors at 5. In ourinitial experiments, the solving time was not af-fected by the number of errors in a given fea-ture model. Again, the prevalence of XOR- orcardinality-based feature groups was the key de-terminer of solving time.

6.2. Experiment 1: Bounding Method Scalability

Hypothesis. We hypothesized that CURE’sCSP-based diagnostic technique could be used todiagnose configurations with 1,000s of featureswhen the bounding method was employed.

Experiment design. To test the diagnos-tic capabilities of the bounding method, we gen-erated large numbers of feature models and aflawed configuration for each model. The CUREbounded diagnostic method was used to diagnoseeach of the flawed configurations. The speed ofthe bounding technique allowed us to test 2,000feature models at each data point (2,000 different

2XOR and cardinality-based feature groups are featuresthat require the set of their selected children to satisfy acardinality constraint (the constraint is 1..1 for XOR).

14

variations of each size feature model) and test thebounding method’s scalability for feature modelsup to 500 features.

With models above 500 features, we had to re-duce the number of samples at each size to 200models due to time constraints. Although thesesamples are small, they demonstrate the generalperformance of our technique. Moreover, the re-sults of our experiments with feature models upto 500 features were nearly identical with samplesizes between 100 and 2,000 models.

Analysis of results. Figure 6 shows the timerequired to diagnose feature models ranging insize from 50 to 500 features using the boundedmethod. The figure captures the worst and aver-

Figure 6: Diagnosis Time for Both Methods for Large Fea-ture Models

age solving time in the experiments. As seen fromthe results, our technique could diagnose modelswith 500 features in an average of ≈300ms.

The upper bound used for this experiment wasa maximum of 10% feature selection changes.When the feature bound was too tight for thediagnosis (i.e., more were needed to reach a cor-rect state) the solver quickly declared there wasno valid solution. We therefore discarded all in-stances where the bound was too tight to avoidskewing the results towards shorter solving times.

Figure 6 shows the results of testing the solvingtime of the bounding method on feature modelsranging in size from 500 to 5,000 features. Modelsof this size were sufficient to demonstrate scalabil-ity for common production systems. The resultsshow that for a 5,000 feature model, the averagediagnosis time was ≈ 50 seconds.

Another key variable we tested was how thetightness of the bound on the maximum num-

ber of feature changes affected the solving timeof the technique. We took a set of 200 featuremodels and applied varying bounds to see howthe bound tightness affected solution time. Fig-ure 7 shows that tighter bounds produced fastersolution times. These results indicate that tighter

Figure 7: 500 Feature Diagnosis Time with BoundingMethod and Varying Bounds

bounds allow the solver to discard infeasible solu-tions quickly and thus arrive at a solution faster.

6.3. Experiment 2: Optimal Method Scalability

Hypothesis. We hypothesized that the op-timal CURE diagnostic method could also scaleto feature models with 1,000s of features. We ex-pected that the diagnostic time would be substan-tially longer than with the bounding method.

Experiment design. We tested the scalabil-ity of the optimal diagnosis method using 2,000samples below 500 features and 200 samples forall larger models. For each feature model, an in-valid configuration was generated and diagnosedwith CURE. We tracked the best, average, andworst diagnosis time for each size model.

Analysis of results. Figure 6 shows the re-sults from feature models up to 500 features. At500 features, the optimal method required an av-erage of ∼1.5 seconds to produce a diagnosis. Fig-ure 6 also shows the tests from larger models rang-ing in size up to 5,000 features. For a model with5,000 features, the solver required an average of∼3 minutes per diagnosis.

6.4. Comparative Analysis of Bounding and Op-timal Methods

Finally, we compared the scalability and qual-ity of results produced with the two methods.Figure 6 shows the bounding method performs

15

and scales significantly better than the optimalmethod. For feature models of up to 1,000 fea-tures, however, both techniques take less than5 seconds and the optimal method is the bet-ter choice. This result raises the question of howmuch of a tradeoff in solution quality for speed ismade when the bounding method is used over theoptimal method for larger models.

The bound that is chosen determines the qual-ity of the solution that is produced by the solver.The optimality of a diagnosis given by the bound-ing method is the number of changes suggested bythe bounding method, Bounded(S ∪ D), dividedby the optimal number of changes, Opt(S ∪ D),

which yields Bounded(S∪D)Opt(S∪D)

. Since the bounding

method uses the constraint (S ∪ D) ≤ K to en-sure that at most K changes are suggested, wecan state the worst case optimality of the boundedmethod as K

Opt(S∪D). The closer our bound, K, is

to the true optimal number of changes to make,the better the diagnosis.

Since tighter bounds produce faster solvingtimes and better results, debuggers should startwith very small bounds and iteratively increasethem upward as needed. One approach is to layeran adaptive algorithm on top of the diagnosis al-gorithm to move the bound by varying amountseach time the bound proves too tight. Anotherapproach is to employ binary search to home inon the ideal bound. We will investigate both tech-niques in our future work.

6.5. Experiment 3: Simplified CSP Formulation

Hypothesis. We hypothesized that the sim-plified CSP formulation would provide a constantreduction in solving time compared to the stan-dard complex CSP formulation.

Experiment design. We performed experi-ments to compare the diagnosis speed of the stan-dard complex CSP formulation against the simpli-fied CSP formulation, as described in Section 3.1.We used our feature model generation and con-figuration infrastructure to produce feature mod-els and flawed configurations of varying sizes andthen diagnosed them with both techniques. Theresults in this section present the differences indiagnosis times that we observed. We generated

and solved feature models ranging in size from 500to 5,000 features. For each size feature model, wesolved 20 instances and tracked the average solv-ing time of the two CSP formulations.

Analysis of results. The results are shown inFigure 8. The results in this figure show the sim-

Figure 8: Comparison of Diagnosis Times for the Complexand Simplified CSP Formulations on Feature Models with500 to 5,000 Features

plified CSP technique had a much faster averagediagnosis time. For example, with 5,000 features,the simplified CSP formulation had an averagediagnosis time of less than half of the standardcomplex CSP formulation.

We took the data shown in Figure 8 and ana-lyzed it to calculate the percentage of reductionin average solving time provided by the simpli-fied CSP formulation over the complex formula-tion. Figure 9 shows the results of this analysis.For smaller feature models with 500 features, the

Figure 9: Simplified CSP Formulation Reduction in Diag-nosis Time Versus Feature Model Size

simplified technique provided a 25% reduction indiagnostic time. As the size of the feature modelsgrew, moreover, the percentage of reduction in di-agnosis time did not remain constant but grew asa function of the size of the feature models. For

16

example, at 2,000 features, the simplified formu-lation provided a roughly 43% decrease in averagesolving time. At 5,000 features, the simplified for-mulation provided a roughly 64% decrease in av-erage solving time.

The results indicate that for small feature mod-els, the simplified technique can provide a modestimprovement in the overall average diagnosis timeacross a number of feature models. The improve-ment comes at a reduction in the expressiveness ofthe goal functions that can be created. For largefeature models, however, the simplified CSP for-mulation provides a significant decrease in averagesolving time. Moreover, the results indicate thatthe improvement provided by the simplified CSPformulation is proportional to the size of the fea-ture model, i.e., the simplified CSP formulationsaves the most time on the most time-consumingmodels to diagnose.

6.6. Debugging ScenariosStaged configuration and viewpoint debugging

( Sections 4.3& 4.1) are special cases of the tech-nique where the solver is not allowed to modifythe selection state of one or more features (i.e.,the viewpoint or the feature at time T ′). Boththese special cases of debugging actually reducethe search space by fixing values for one or moreof the CSP variables. For example, performingstaged configuration debugging (which fixes thevalue for one CSP variable on a model with 1,000features) reduced the optimal method’s averagesolving time by ≈ 2.5 seconds and the boundingmethod by ≈ .1 seconds.

Cost-based conflict mediation ( Section 4.2)performs identically to the standard diagnosistechnique. Cost-based mediation merely intro-duces a series of coefficients, bi ⊂ B into theoptimization goal. These coefficients do not in-crease solving time. Moreover, initiating the di-agnosis method with the superset of the configu-ration participants’ desired feature selections alsodid not impact performance.

7. Related Work

This section compares our work on CURE withrelated research projects in the areas of auto-

mated configuration, autonomic healing, and con-figuration quality evaluation.

7.1. Constraint Relaxation and Explanation

Junker (Junker, 2004) has investigated the no-tion of preferred explanations in the contextof explaining conflicts in over-constrained CSPs.Junker’s work uses a series of user-provided pref-erences, described as a partial ordering of the con-straints, to help compute a set of constraints torelax to make the CSP tractable. CURE is similarto this work in that it uses user-provided informa-tion to produce relaxations of the CSP that bet-ter fit with the goals of the user. CURE, however,allows developers to specify preferences in termsof the delta, such as the change in cost, betweenthe invalid configuration and the new configura-tion that is derived. Junker’s work does not pro-vide the ability to specify constraint preferencesin terms of the difference between an input andoutput feature model. Junker’s research also doesnot specify how these techniques can be appliedto feature model configuration. CURE, however,provides concrete methods for applying constraintrelaxation to staged configuration, configurationconflict resolution, and other critical feature mod-eling topics.

Other constraint relaxation and explanationtechniques, such as those developed by Garciaet al. (de la Banda et al., 2003), include meth-ods for finding a minimal set of unsatisfiable con-straints. Garcia’s concept of minimality searchesfor the minimal number of constraints. This con-cept of minimality is valid for some diagnosis casesbut not for cost-based diagnostics. This work issomewhat similar to CURE’s ability to find theminimal set of changes that can be applied to aconfiguration to reach a new and valid configura-tion. Finding a minimal set of unsatisfiable con-straints, however, is not specific to feature mod-eling and does not provide mechanisms for per-forming cost-based feature diagnostics or otherimportant industrial feature modeling analyses,such as staged configuration debugging. Also, aswith with Junker’s techniques and constraint re-laxation in general, Garcia’s diagnosis techniquesonly propose ways of retracting configuration de-

17

cisions and not ways of both adding and remov-ing configuration decisions as CURE does. Fur-thermore, as with other techniques, this approachdoes not provide the ability to define preferenceson the diagnostics that are related to the changesthat must be made to the original configuration.

Another series of constraint explanation ap-proaches are based on forms of truth mainte-nance (Hagg et al., 2006). These approachesrecord a justification with each constraint that isadded to the CSP. When a conflict occurs, usersmust retract their justifications, which causes allof their associated constraints to be removed, un-til a satisfiable CSP is reached. CURE allows de-velopers to specify feature selections that can andcannot be changed, which is similar to choosingwhich justifications to remove. At the same time,CURE allows developers to specify diagnosis pref-erences that are a function of the delta betweenthe input configuration and output configuration.Truth maintenance systems do not provide thiscapability, which is crucial for performing opti-mal cost-based conflict resolution, staged config-uration debugging, and software feature evolutionanalysis.

7.2. Automated Configuration

In prior work, Trinidad et al. (Trinidad et al.,2007) showed how feature models can be trans-formed into diagnosis CSPs and used to identifyfull mandatory features, void features, and deadfeature models. Developers can use this diagnos-tic capability to identify feature models that donot accurately describe their products and to un-derstand why not. The technique we describedin this paper builds on this idea of using a CSPfor automated diagnosis. Whereas Trinidad et al.focus on diagnosing feature models that do notdescribe their products, we build an alternate di-agnosis model to identify conflicts in feature con-figurations. Moreover, we provide specific recom-mendations as to the minimal set of features thatcan be selected or deselected to eliminate the er-ror.

Debugging techniques for feature models wereinvestigated by Batory et al. in Batory (2005).Their techniques focus on translating feature

models into propositional logic and using SATsolvers to automate configuration and verify cor-rectness of configurations. In general, their worktouches on debugging feature models rather thanindividual configurations. Our approach focuseson another dimension of debugging, the ability topinpoint errors in individual configurations andto specify the minimal set of feature selectionsand deselections to remove the error. Moreover,propositional logic-based approaches do not typ-ically provide maximization or minimization ofnumeric formulas as primitive functions providedby the solver. Since, CURE uses a CSP-basedapproach, minimization/maximization diagnosisfunctionality is built-in.

For example, developers may want to determinethe cheapest way of correcting a flawed featureselection, where the definition of cost is definedby a complex numerical formula. To performthis type of cost-based correction of invalid fea-ture selections requires the use of numerical max-imization/minimization functions. SAT solversare geared towards boolean logic and not numer-ical optimization.

Pure::variants (Beuche, 2003), Feature Model-ing Plugin (FMP) (Czarnecki et al., 2005a), FeA-ture Model Analyser (FAMA) (Benavides et al.,2007), and Big Lever Software Gears (Buhrdorfet al., 2003) are tools developed to help devel-opers create correct configurations of SPL fea-ture models. These tools enforce constraints onmodelers as the features are selected. None ofthese tools, however, addresses cases where fea-ture models with incorrect configurations are cre-ated and require debugging. The technique de-scribed in this paper provides this missing capa-bility. These tools and our approach are comple-mentary since the tools help to ensure that cor-rect configurations are created and our techniquediagnoses incorrect configurations that are built.

Propositional logic has been used to repre-sent software product lines by Mannion Mannion(2002). These techniques are useful for represent-ing product line models, but do nothing to cor-rect flawed feature models. Moreover, they donot take intelligent conflict resolution into con-sideration. CURE differs in that it can be used

18

to diagnose and remedy existing, invalid featureselections. The two approaches are complemen-tary, Mannion’s techniques can flag flawed con-figurations and CURE can be used to repair theconfiguration flaws.

7.3. Autonomic Healing

Dudley et al. (2004) present systems that canautonomically identify and repair system prob-lems. In the event that a system fails, theseself-healing solutions are capable of determiningthe element or elements that are responsible forthe failure. Once the nature of a failure is de-termined, a replacement element is automaticallychosen from a set of potential replacement ele-ments, thus allowing the system to recover. LikeCURE, this technique is corrective and capable oftaking an invalid system configuration, discover-ing the element or elements that make the config-uration invalid, and replacing them with elementsautonomically to create a valid system. UnlikeCURE, however, this technique does not makeuse of feature models and is subject to runtimeconstraints, making it hard to use for SPLs ordiagnosis optimization.

7.4. Configuration Quality Evaluation

Other techniques provide mechanisms for rea-soning about feature model configurations whenthe exact non-functional properties of each fea-ture are not known. Zhang et al. (2003) introduceBayesian Belief Networks, which predict the im-pact of feature selection on quality attributes byconsidering the effects of choosing similar variantson the quality of past projects. Jarzabek et al.(2006) present a goal-oriented analysis that placesemphasis on the satisfaction of quality attributeduring feature selection. These techniques explorea different area of configuration than CURE, theare focused on finding configurations when thereis uncertainty about the effects of configurationdecisions. In contrast, CURE examines existingconfigurations, in which the ramifications of se-lecting a particular feature are already known.

Other modeling approaches, such as COVA-MOF (Sinnema et al., 2004) and QRF (Niemelaand Immonen, 2007), examine the ramifications

of feature selection on the overall quality of thesystem. CURE also examines the effect of select-ing a feature on the overall quality of the system.Unlike CURE, however, these techniques do notexamine how to correct flawed system configura-tions. CURE is a configuration diagnosis methodand not a configuration methodology.

The use of various quality attributes to assesspotential system configurations have been investi-gated. For example, Immonen (Immonen, 2005).introduces a tool for the prediction of availabilityand reliability for a system configuration. Olu-mofin et al. (Olumofin and Misic, 2005) provides amethod that examines the availability, reliability,performance, and modifiability of potential sys-tem configurations. Like CURE, these quality at-tributes are used to determine which feature toselect at each point of variability. Unlike CURE,however, these methods are used in the designphase to construct an initial system configuration,whereas CURE is used to correct flawed featureselections.

Etxeberria and Sagardui (2008) present a man-ual technique for using domain-relevant qualityattributes to evaluate variants in product line ar-chitectures. An additional quality feature tree canbe added as a branch to extended feature models.These quality features have a predefined qualita-tive and quantitative impact on the overall sys-tem if selected. Feature selection can thereforedetermine the overall quality of a system configu-ration and assure that the system possesses nec-essary functionality. Again, CURE is a correctivetechnique for restoring validity to flawed featureselections, whereas the method in Etxeberria andSagardui (2008) is used during the design phaseto determine valid initial configurations.

8. Concluding Remarks

Production SPLs rely on large feature modelsthat make it hard to always produce correct con-figurations. When errors due occur, the lack ofconfiguration debugging tools forces developers tomanually pinpoint errors in configurations thatcan span thousands of features. Moreover, dueto the size of many feature models and other fac-

19

tors (such as supply-chains), SPL configurationmay be performed in stages or in parallel (Batoryet al., 2006). In these situations, the feature se-lections of individual developers can conflict andintroduce errors in feature models that are evenmore challenging to identify and correct.

Another challenge of using an SPL is main-taining configurations over time. For example,an SPL that encompasses components obtainedthrough a supply-chain may have features that be-come unavailable due to vendors exiting the mar-ket or discontinuing support. In these scenarios,developers need a way of automating the evolu-tion of existing configurations to new and correctfeature selections that take these changes into ac-count.

This paper introduced a technique called Con-figuration Understanding and REmedy (CURE)that transforms invalid configurations into CSPsand automates (1) the diagnosis of invalid configu-rations and (2) the evolution of existing configura-tions to meet changing requirement sets. CUREderives the closest valid configuration to an ex-isting configuration that meets a new set of re-quirements. Moreover, developers can customizeCURE to provide optimal cost-based error andevolution diagnostics.

CURE’s CSP-based diagnosis model is extensi-ble and can be modified to perform conflict media-tion, run faster by using a simplified CSP formu-lation, or debug from different viewpoints. Ourempirical results show CURE can scale to produc-tion feature models with 5,000 features and stillprovide a diagnosis in between 45 seconds and 4minutes. These time bounds should be sufficientfor the design-time use of this algorithm in manyproduction SPLs.

The following are lessons learned from our ef-forts with CURE thus far:

• Simplifying the diagnostic CSP can

yield substantial improvements in diag-

nosis time. A simplified version of CURE’sCSP can be used to provide significant im-provements in solving speed with a limitedreduction in diagnostic flexibility. The re-duced flexibility does not allow developers to

place limits on only feature selection or dese-lection. If developers do not need to preciselypinpoint one type of change, the simplifieddiagnostic CSP should be used.

• CURE can scale to 1,000s of fea-

tures. We learned from our experimentswith CURE that it can diagnose models withroughly 5,000 features. The results fromtechniques, such as bounding or CSP simplifi-cation, indicate that potential to scale CUREup to larger feature model sizes. The tech-niques improve CURE’s speed by reducingthe number of variables that are present inthe CSP and placing a precise bound on thesearch space.

• It is hard to predict CURE’s diagnosis

time. CURE’s diagnosis time is dependenton the structure of the constraints in the fea-ture model. For some models, CURE pro-vides very fast diagnosis. In future work, weplan to investigate heuristic diagnostic tech-niques that can speed CURE’s diagnosis timefor models it does not perform well on.

• A good K bound for CURE is hard to

ascertain. The optimality of the diagno-sis provided by the bounding method is de-termined by how close K is set to the trueminimum number of features that need to bechanged to reach a valid state. Setting anaccurate bound for K is not easy. In fu-ture work, we plan to investigate differentmethods of honing the boundary used in thebounding method.

• Configuration evolution problems can

be modeled as configuration diagnosis

problems. Although configuration evolu-tion problems do not necessarily start withan invalid feature selection, we learned thatCURE’s diagnosis techniques can still be ap-plied successfully to them. When a fea-ture model’s constraints are changed or aconfiguration’s feature selection needs to bemodified, CURE can be used to automati-cally modify legacy configurations by model-

20

ing them as invalid configurations. Automat-ing configuration evolution can provide costoptimal modification planning, which is hardto achieve with manual evolution techniques.

The CURE diagnosis technique has beenimplemented as part of the ASCENT De-sign Studio Intelligence project and is availablein open-source format from code.google.com/

p/ascent-design-studio and www.isa.us.es/

fama.

9. Acknowledgements

We would like to thank Jose Galindo for hishard work implementing CURE in FAMA.

References

Batory, D., 2005. Feature Models, Grammars, and Prepo-sitional Formulas. Software Product Lines: 9th Interna-tional Conference, SPLC 2005, Rennes, France, Septem-ber 26-29, 2005: Proceedings.

Batory, D., Benavides, D., Ruiz-Cortes, A., 2006. Auto-mated analysis of feature models: Challenges ahead.Communications of the ACM December.

Benavides, D., Segura, S., Trinidad, P., Ruiz-Cortes,A., 2007. FAMA: Tooling a framework for the auto-mated analysis of feature models. In: Proceeding of theFirst International Workshop on Variability Modellingof Software-intensive Systems (VAMOS).

Benavides, D., Trinidad, P., Ruiz-Cortes, A., 2005. Au-tomated Reasoning on Feature Models. 17th Confer-ence on Advanced Information Systems Engineering(CAiSE05, Proceedings), LNCS 3520, 491–503.

Beuche, D., 2003. Variant Management withPure:: variants. Tech. rep., Pure-Systems GmbH,http://www.pure-systems.com.

Buhrdorf, R., Churchett, D., Krueger, C., November 2003.Salion’s Experience with a Reactive Software ProductLine Approach. In: Proceedings of the 5th InternationalWorkshop on Product Family Engineering. Siena, Italy.

Clements, P., Northrop, L., 2002. Software Product Lines:Practices and Patterns. Addison-Wesley, Boston.

Czarnecki, K., Antkiewicz, M., Kim, C., Lau, S., Piet-roszek, K., October 2005a. In: FMP and FMP2RSM:Eclipse Plug-ins for Modeling Features Using ModelTemplates. ACM Press New York, NY, USA, pp. 200–201.

Czarnecki, K., Helsen, S., Eisenecker, U., 2004. StagedConfiguration Using Feature Models. Software Prod-uct Lines: Third International Conference, SPLC 2004,Boston, MA, USA, August 30-September 2, 2004: Pro-ceedings.

Czarnecki, K., Helsen, S., Eisenecker, U., 2005b. Stagedconfiguration through specialization and multi-levelconfiguration of feature models. Software Process Im-provement and Practice 10 (2), 143–169.

de la Banda, M. J. G., Stuckey, P. J., Wazny, J., 2003.Finding all minimal unsatisfiable subsets. In: PPDP.ACM, pp. 32–43.

Dudley, G., Joshi, N., Ogle, D., Subramanian, B., Topol,B., 2004. Autonomic Self-Healing Systems in a Cross-Product IT Environment. Proceedings of the Interna-tional Conference on Autonomic Computing, 312–313.

Etxeberria, L., Sagardui, G., 2008. Variability DrivenQuality Evaluation in Software Product Lines. In: Soft-ware Product Line Conference, 2008. SPLC’08. 12th In-ternational. pp. 243–252.

Hagg, A., Junker, U., O’Sullivan, B., August 2006. A sur-vey of explanation techniques for configurators. In: Pro-ceedings of ECAI-2006 Workshop on Configuration.

Immonen, A., 2005. A method for predicting reliability andavailability at the architectural level. Research Issues inSoftware Product-Lines-Engineering and Management,T. Kakola and JC Duenas, Editors.

Jaffar, J., Maher, M., 1994. Constraint Logic Program-ming: A Survey. Journal of Logic Programming 19 (20),503–581.

Jarzabek, S., Yang, B., Yoeun, S., 2006. Addressing qual-ity attributes in domain analysis for product lines. In:Software, IEE Proceedings-. Vol. 153. pp. 61–73.

Junker, U., 2004. Quickxplain: Preferred explanations andrelaxations for over-constrained problems. AAAI Press/ The MIT Press, pp. 167–172.

Kang, K. C., Kim, S., Lee, J., Kim, K., Shin, E., Huh,M., January 1998. FORM: A Feature-Oriented ReuseMethod with Domain-specific Reference Architectures.Annals of Software Engineering 5 (0), 143–168.

Mannion, M., 2002. Using first-order logic for product linemodel validation. Proceedings of the Second Interna-tional Conference on Software Product Lines 2379, 176–187.

Metzger, A., Pohl, K., Heymans, P., Schobbens, P.-Y.,Saval, G., 2007. Disambiguating the documentation ofvariability in software product lines: A separation ofconcerns, formalization and automated analysis. In: Re-quirements Engineering Conference, 2007. RE ’07. 15thIEEE International. pp. 243–253.

Niemela, E., Immonen, A., 2007. Capturing quality re-quirements of product family architecture. Informationand Software Technology 49 (11-12), 1107–1120.

Olumofin, F., Misic, V., 2005. Extending the ATAM Ar-chitecture Evaluation to Product Line Architectures. In:IEEE/IFIP Working Conference on Software Architec-ture, WICSA.

Reiter, R., 1987. A theory of diagnosis from first principles.Artificial Intelligence 32 (1), 57–95.

Sinnema, M., Deelstra, S., Nijhuis, J., Bosch, J., 2004. CO-VAMOF: A Framework for Modeling Variability in Soft-

21

ware Product Families. LECTURE NOTES IN COM-PUTER SCIENCE, 197–213.

Trinidad, P., Benavides, D., Duran, A., Ruiz-Cortes, A.,Toro, M., 2007. Automated error analysis for the ag-ilization of feature modeling. Journal of Systems andSoftware, in press.

Van Hentenryck, P., 1989. Constraint Satisfaction in LogicProgramming. MIT Press Cambridge, MA, USA.

White, J., Czarnecki, K., Schmidt, D. C., Lenz, G., Wien-ands, C., Wuchner, E., Fiege, L., October 2007. Au-tomated Model-based Configuration of Enterprise JavaApplications. In: The Enterprise Computing Confer-ence, EDOC. Annapolis, Maryland USA.

White, J., Schmidt, D. C., Benavides, D., Trinidad, P.,Ruiz-Cortez, A., Sep. 2008. Automated Diagnosis ofProduct-line Configuration Errors in Feature Models.In: Proceedings of the Software Product Lines Confer-ence (SPLC). Limerick, Ireland.

Zhang, H., Jarzabek, S., Yang, B., 2003. Quality Pre-diction and Assessment for Product Lines. LECTURENOTES IN COMPUTER SCIENCE, 681–695.

22