IEEE TRANSACTIONS ON SOFTWARE …nikolaos/publications/...Assessing the Refactorability of Software...

Assessing the Refactorabilityof Software Clones

Nikolaos Tsantalis,Member, IEEE, Davood Mazinanian, and Giri Panamoottil Krishnan

AbstractThe presence of duplicated code in software systems is significant and several studies have shown that clones can be

potentially harmful with respect to the maintainability and evolution of the source code. Despite the significance of the problem, there is

still limited support for eliminating software clones through refactoring, because the unification and merging of duplicated code is a very

challenging problem, especially when software clones have gone through several modifications after their initial introduction. In this

work, we propose an approach for automatically assessing whether a pair of clones can be safely refactored without changing the

behavior of the program. In particular, our approach examines if the differences present between the clones can be safely

parameterized without causing any side-effects. The evaluation results have shown that the clones assessed as refactorable by our

approach can be indeed refactored without causing any compile errors or test failures. Additionally, the computational cost of the

proposed approach is negligible (less than a second) in the vast majority of the examined cases. Finally, we perform a large-scale

empirical study on over a million clone pairs detected by four different clone detection tools in nine open-source projects to investigate

how refactorability is affected by different clone properties and tool configuration options. Among the highlights of our conclusions, we

found that a) clones in production code tend to be more refactorable than clones in test code, b) clones with a close relative location

(i.e., same method, type, or file) tend to be more refactorable than clones in distant locations (i.e., same hierarchy, or unrelated types),

c) Type-1 clones tend to be more refactorable than the other clone types, and d) clones with a small size tend to be more refactorable

than clones with a larger size.

Index TermsCode duplication, software clone management, clone refactoring, refactorability assessment, empirical study

1 INTRODUCTION

CODE duplication has been recognized as a potentiallyserious problem having a negative impact on the main-tainability, comprehensibility, and evolution of softwaresystems. Over the years, the software clone research com-munity has developed several techniques for the detectionand analysis of duplicated code [1], and more recently hasfocused on clone management activities [2], such as tracingclones in the history of a project, analyzing the consistencyof modifications to clones [3], updating incrementally clonegroups as the project evolves [4], and prioritizing the refac-toring of clones [5], [6].

In addition to the development of tools and techniquesfor the detection and management of software clones,several researchers investigated empirically the effect ofduplicated code on maintenance effort and cost [7], error-proneness due to inconsistent updates [8], [9], softwaredefects [10], change-proneness [11], and change propaga-tion [12]. However, to the best of our knowledge, there is nostudy investigating the refactorability of software clones.What portion of the clones detected by tools can be actually refac-tored? Additionally, there is a lack of tools that can automat-ically analyze software clones to determine whether they

can be safely refactored without changing the programbehavior. Refactorability analysis is an important missing fea-ture from clone management, since it could be used to filterclones that can be directly refactored, when the developersare interested in finding refactoring opportunities for dupli-cated code. In this way, maintainers can focus their effort onparts of the code that can immediately benefit from refactor-ing, and thus expedite maintainability improvement.

In this paper, we present an approach that takes as inputtwo clone fragments detected from any tool and appliesthree steps to determine whether they can be safely refac-tored (i.e., without any side effects). First, our approachfinds code fragments with identical nesting structureswithin the input clones that could serve as potential refactor-ing opportunities. We consider that two code fragments canbe unified, and therefore refactored, if they share a commonnesting structure. In the second step, our approach finds amapping between the statements of the code fragmentsthat maximizes the number of mapped statements and min-imizes the number of differences between the mapped state-ments by exploring the search space of alternative mappingsolutions. This is generally an NP-hard problem [13], andsince exhaustive search is impractical, our solution relies onheuristics to reduce the search space. From the refactoringpoint of view, we support that a mapping solution with asmaller number of differences between the mapped state-ments has a higher refactorability compared to an alternativemapping solution with a larger number of differences. Thereason is that some differences cannot be safely parameter-ized, and thus a larger number of differences increases theprobability of side effects from the parameterization ofdifferences. Finally, in the last step, the differences between

The authors are with the Department of Computer Science and SoftwareEngineering, Concordia University, Montreal, Quebec, Canada H3G 1M8.E-mail: {nikolaos.tsantalis, giri.krishnan}@concordia.ca,[email protected].

Manuscript received 8 Sept. 2014; revised 5 May 2015; accepted 16 June 2015.Date of publication 21 June 2015; date of current version 13 Nov. 2015.Recommended for acceptance by A. Hassan.For information on obtaining reprints of this article, please send e-mail to:[email protected], and reference the Digital Object Identifier below.Digital Object Identifier no. 10.1109/TSE.2015.2448531

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 11, NOVEMBER 2015 1055

0098-5589 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

the mapped statements detected in the previous step areexamined against a set of preconditions to determinewhether they can be parameterized without changingthe program behavior. Additionally, the statements thathave not been mapped in the previous step are examinedagainst a set of preconditions to determine whether theycan be safely moved before or after the execution of theextracted method containing the mapped statements.

This paper is an extension over our previous work [14],which contains the following improvements and additions:

1) We replace the Control Dependence Tree with theProgram Structure Tree (PST) [15] for representing thenesting structure of the source code. The reason forthis replacement is that control dependencies canform graphs in the case of unstructured control orexception flow, while the PST is defined for arbitraryflow graphs, even irreducible ones.

2) We support additional types of differences in theAST matching of statements (Section 3.1) to makemore flexible the unification of statements with non-trivial differences and improve the quality of therefactoring solution.

3) We introduce conditions in our divide-and-conqueralgorithm to guarantee that the resulting sub-solutions can be safely combined into a valid globalsolution respecting the initial nesting structure of theclone fragments (Section 3.3.2).

4) We introduce an additional decomposition phase ofthe statement mapping problem (Section 3.3.4) todeal better with the problem of combinatorialexplosion.

5) We extend the list of examined preconditions withfour new preconditions (Section 3.4). Additionally,we provide formal ways to detect preconditionviolations.

6) We provide tool support for the automatic refactor-ability analysis of software clones, the visual inspec-tion of the differences and precondition violationsfound in a pair of clone fragments, and the refactor-ing of method-level clone fragments (Section 4).

7) We evaluate the correctness and efficiency of ourapproach (Sections 5.2 and 5.3).

Our technique supports the analysis of clones detected inJava programs, and therefore the defined preconditions areadjusted to the features/limitations of the Java program-ming language. For instance, the fact that Java does notsupport parameter passing by reference, makes necessary thedefinition of some preconditions that could be overcome bylanguages supporting this feature. The defined precondi-tions cover Type-1 clones (i.e., identical code fragmentsexcept for variations in whitespace, layout, and comments[1]), Type-2 clones (i.e., structurally/syntactically identicalfragments except for variations in identifiers, literals andtypes in addition to Type-1 differences [1]), and Type-3clones (i.e., copied fragments with statements changed,added or removed in addition to Type-2 differences [1]).

Clone detection tools apply many different approaches forthe detection of duplicated code fragments, including text-based, token-based, tree-based, metrics-based, and graph-based techniques [1] and have a variety of configuration

options. Both of these factors (i.e., detection approach andconfiguration options) affect the quality of the detected cloneswith respect to refactorability. For instance, text-based andtoken-based approaches are more likely to return clone frag-ments having incomplete statements (i.e., partially matchedstatements) or different nesting structures compared to tree-based and graph-based approaches [16]. Therefore, it is veryinteresting to investigate the performance of different clonedetection techniques with respect to the refactorability of theclones they detect, and how different configuration optionsmay affect the quality of their results.

This work makes two main contributions in the area ofsoftware clone management:

1) We present a reliable and efficient approach to auto-matically assess the refactorability of softwareclones. To the best of our knowledge, this is the firstapproach to provide such an in-depth solution tothis problem.

2) We conduct a large-scale empirical study (Sec-tion 5.4) on 1,150,967 clone pairs to investigate therefactorability of software clones taking into accountdifferent dimensions: Source code type (production versus test code) Clone location (same file versus different files) Clone type (Type-1 versus Type-2 versus Type-3) Clone size Precondition violation types.

2 PRELIMINARIES

In this section, we will briefly describe two core programstructures that are used in our approach.

2.1 Program Structure Tree

The Program Structure Tree was introduced by Johnsonet al. [15] as a hierarchical representation of program struc-ture based on single-entry single-exit (SESE) regions of thecontrol flow graph. Johnson et al. extended the notion ofdominance and postdominance to control flow edges anddefined SESE region as an ordered edge pair (a, b) of distinctcontrol flow edges a (entry edge) and b (exit edge) where:

The PST essentially captures the nesting relationship ofSESE regions, as well as chains of sequentially composedSESE regions. Fig. 1c depicts the PST for the code exampleshown in Fig. 1a. Fig. 1b shows the control flow graph withits SESE regions marked in dotted rectangles. The nestingstructure of the SESE regions in the control flow graph isused to generate the PST. Chains of sequentially composedSESE regions, such as regions A and D, are grouped withina dotted rectangle in the PST.

As it will be explained later in Section 3.2, the first step ofour refactorability analysis approach involves the detectionof matching nesting structures within two clone fragments.The nesting structure of a program is essentially captured bythe control predicate nodes of the PST. For this purpose, we

1) a dominates b2) b postdominates a3) every cycle containing a also contains b and vice

versa.

1056 IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 41, NO. 11, NOVEMBER 2015

define the Nesting Structure Tree (NST), as the tree that con-tains only the control predicate nodes (e.g., if, for state-ments) of the original PST. For example, the NST for the PSTof Fig. 1c contains only nodes root,A,D, and F.

2.2 Program Dependence Graph

The Program Dependence Graph (PDG) [17] is a directedgraph with multiple edge types, in which the nodesrepresent the statements of a function or method, and theedges represent control and data flow dependenciesbetween statements. More specifically, we distinguish twokinds of statements, namely control predicate statements(i.e., statements with a body e.g., if, for) and non-predicate statements (i.e., leaf statements without a body).In the case of a control predicate statement, we consideronly its conditional expression(s) (i.e., we ignore the state-ments inside its body) when computing dependenciesfrom/to it. In the case of a leaf statement, we consider theentire statement (i.e., all expressions inside it) when com-puting dependencies from/to it.

A control dependence edge denotes that the execution ofthe statement at the end point of the edge depends on thecontrol conditions of the control predicate statement at thestart point of the edge. A data dependence edge is alwayslabeled with a variable v and denotes that the statement atthe end point of the edge is using the value of v, which hasbeen previously modified by the statement at the start pointof the edge. If the data dependence is carried through a loopnode l, then it is considered as a loop-carried dependence.

The PDG representation used in this paper is extended intwo ways. First, we introduce composite variables [18] repre-senting the state of the objects being referenced within thebody of a method, and create additional data dependenciesfor these variables by analyzingmethod calls thatmaymodifyor use the state of the referenced objects. Second, we add twomore types of edges in the PDG, which are used in the exami-nation of preconditions (Section 3.4). These edges are anti-dependencies and output-dependencies. An anti-dependenceedge due to variable v denotes that the statement at the endpoint of the edge is modifying the value of v, which has beenused by the statement at the start point of the edge (i.e., theopposite of a data dependence). An output-dependence edgedue to variable v denotes that both statements at the start andend points of the edgemodify the value of v.

3 APPROACH

Our approach is designed to process two different forms ofinput:

1) Two code fragments within the body of the samemethod, or different methods, reported as clones bya clone detection tool.

2) Two method declarations considered to be dupli-cated (i.e., method-level clones), or containing dupli-cate code fragments somewhere inside their bodies.

In a nutshell, our approach for assessing the refactorabil-ity of two clone fragments comprises three major steps, asshown in Fig. 2:

Fig. 2. An overview of the proposed refactorability analysis approach.

Fig. 1. Generating the program structure tree from a control flow graph with SESE regions.

TSANTALIS ET AL.: ASSESSING THE REFACTORABILITY OF SOFTWARE CLONES 1057

1) Nesting structure matching. The nesting structure ofthe input clone fragments is analyzed to find maxi-mal isomorphic subtrees. The assumption is that twocode fragments can be unified only if they have anidentical nesting structure. Each pair of matchedsubtrees will be further investigated as a separateclone refactoring opportunity in the next steps.

2) Statement mapping. The statements within the subtreepairs extracted from the previous step are mappedin a divide-and-conquer fashion. Taking advantageof the identical nesting structure between the iso-morphic subtrees, the global mapping problem isdivided into smaller sub-problems (by mapping thestatements nested under control predicate nodes atthe same level of the subtrees). For each sub-problemthe corresponding Program Dependence subgraphsare mapped by applying a maximum common sub-graph (MCS) algorithm. At the end, the sub-solutionsare combined to give the global mapping solution.

3) Precondition examination. Based on the differencesbetween the mapped statements in the global solu-tion, as well as the statements that may have not beenmapped, a set of preconditions regarding the preser-vation of program behavior is examined. If none ofthe preconditions is violated, the clone fragmentscorresponding to the mapped statements can besafely refactored, and thus are considered refactorable.

Our ideawas inspired by Johnson et al. [15]who supportedthat any global analysis algorithm can be applied unchangedto each SESE region, and the partial results can be combinedusing the PST to give the global result. This (i.e., the PST) letsus apply analysis algorithms in a divide-and-conquer fashionto the program, which can be a win if the combining of partialresults is not overly expensive. In our case, the statementmapping process is applied in a divide-and-conquer fashion,based on the nesting structure of the clone fragments ascaptured by their PSTs. The partial results (i.e., the mappingsub-solutions) can be combined as long as the global result(i.e., the final mapping solution) complies with the originalnesting structure of the clone fragments.

3.1 Abstract Syntax Tree Compatibility

The first and second steps of our approach rely heavily onthe matching of statements between the two examined clonefragments based on the analysis of their abstract syntax tree(AST) structure. In the first step (Section 3.2) the controlpredicate nodes (e.g., if, for statements) inside the clonefragments are matched by comparing the AST structure oftheir conditional expressions (i.e., the bodies of the controlpredicates are excluded from the comparison). In the secondstep (Section 3.3) the statements nested under the controlpredicate nodes (i.e., leaf statements without a body) arematched by comparing their entire AST structure.

In our approach, we consider two statements as compati-ble, if they correspond to the same AST statement type andhave a homomorphic AST structure [19]. This means thatwe allow a subtree expression in the first AST (e.g., amethod call) to be mapped to a leaf expression (e.g., a vari-able identifier) or another kind of subtree expression (e.g., aclass instance creation) in the second AST, as long as thismapping respects the core structure of the ASTs. The only

restriction in the mapping of sub-expressions within thetwo statements is that the mapped expressions should beevaluated to the same class/primitive type or types beingsubclasses of a common superclass. This restriction allowsto extract differences between the clone fragments that canbe potentially parameterized by introducing a parameter ofthe same type.

We provide a high degree of freedom in the mapping ofexpressions within the statements in order to make moreflexible the unification of duplicated code with non-trivialdifferences. Table 1 contains the complete list of expressiontypes that can be parameterized if found different betweentwo given statements.

Our AST matching algorithm has been implemented byextending the ASTMatcher superclass provided in EclipseJDT framework. The default implementation matches twoASTs only if they are structurally isomorphic (i.e., they havean identical tree structure and exactly the same node types/values). Our implementation adds a relaxation in the match-ing of AST nodes thatmay have different types or values, andadditionally returns a list of the differences detected betweenthemapped statements that is used in the examination of pre-conditions (Section 3.4). Table 2 shows the difference typeswhich are reported by our AST matching implementation.The last two difference types, namely operator and variabletypemismatches (with the exception of generic type parameters,e.g.,) cannot be parameterized. In the cases where a dif-ference refers to a property of a primary expression (e.g., twomethod calls having a different name or a different numberof arguments), the entire primary expression (e.g., methodinvocation) ismarked for parameterization.

3.2 Nesting Structure Matching

In the first step of the proposed approach, our goal is to findmaximal isomorphic subtrees within the nesting structures(i.e., the NSTs) of the clone fragments given as input, sincethere is no guarantee that the input code fragments will

TABLE 1Supported Expression Types In AST Matching

Expression Type Example

Method Invocation expr.method(arg0, ...)Super Method Invocation super.method(arg0, ...)String Literal stringCharacter Literal cBoolean Literal true or falseNumber Literal 5.6Null Literal nullType Literal Type.classClass Instance Creation new Type(arg0, ...)Array Creation new Type[expr]Array Access array[index]Field Access this.identifierSuper Field Access super.identifierParenthesized Expression (expr)Simple Name identifierQualified Name Type.identifierCast Expression (Type)exprThis Expression thisPrefix Expression -exprInfix Expression expr1 + expr2


have an identical nesting structure. To this end, we devel-oped an algorithm that takes as input two NSTs and findsall non-overlapping largest common subtrees [20]. Each result-ing subtree match will be further investigated as a separateclone refactoring opportunity.

Initially, we collect from the two NSTs all leaf nodes,which either do not have siblings, or all of their siblingsare also leaf nodes. We select these nodes in order to startthe detection of isomorphic subtrees from the deepest lev-els of the NSTs in a bottom-up fashion. Next, we extractall matching (i.e., AST compatible) pairs between the col-lected leaf nodes from the two NSTs. In the case a leafnode from the first NST can be matched with multipleleaf nodes from the second NST, we keep only the bestmatching pair (i.e., the pair with the minimum number ofdifferences). Each leaf node pair is given as input toAlgorithm 1, which performs a combination of bottom-upand top-down tree matching techniques [20]. Accordingto Valiente [20], for two unordered trees T1 and T2 withn1 and n2 nodes, respectively, the algorithm for bottom-up maximum common subtree isomorphism runs in the

order of (n1 n2)2.In a nutshell, the algorithm first compares the sibling

nodes of the node pair given as input to find matchingsibling pairs. For each matching sibling pair it performs atop-down tree match (line 11) and examines if the result-ing subtree match is exactly paired (line 12). Two sub-trees are considered as exactly paired if there is a one-to-one correspondence between their nodes (i.e., a bijection). Inset theory, there is a bijection from set X to set Y when

every element of X is paired with exactly one element ofY, and every element of Y is paired with exactly one ele-ment of X. The top-down tree match function is essen-tially a fail-fast mechanism that stops the mainalgorithm from exploring non-exactly paired subtreematches at an early stage. If all matching sibling pairslead to exactly paired top-down subtree matches, thenthe parent nodes of the node pair given as input are vis-ited. Finally, if the parent nodes match, then the functiondescribed in Algorithm 1 is recursively executed with thenew parent node pair as input. The proposed algorithmreturns the largest exactly paired subtree match startingfrom the given input node pair. We designed the algo-rithm to return only exactly paired subtree matches inorder to avoid inconsistencies or gaps in the nestingstructure of the matched subtrees.

Algorithm 1. Recursive Function Returning a MaximalExactly Paired Subtree Match.

Input: a pair of matching NST nodes (nodei, nodej)Output: a set of matching NST node pairs1: function BOTTOMUPMATCH(nodePair, solution)2: solution = solution [ nodePair3: siblingsi = nodePair.nodei.siblings4: siblingsj = nodePair.nodej.siblings5: mSiblings = ? "matched siblings6: mPairs = ? "matched node pairs7: for each siblingi 2 siblingsi do8: for each siblingj 2 siblingsj do9: if compatibleAST(siblingi, siblingj) and not

alreadyMatched(siblingj) then

10: pair = (siblingi, siblingj)11: pairs = TOPDOWNMATCH(pair)12: if exactlyPairedSubtrees(pairs) then13: mSiblings =mSiblings [ pair14: mPairs =mPairs [ pairs15: break " first-match16: end if17: end if18: end for19: end for20: if mSiblingsj j = siblingsij j = siblingsj

then

21: solution = solution [mPairs22: parenti = nodePair.nodei.parent23: parentj = nodePair.nodej.parent24: if compatibleAST(parenti, parentj) then25: pair = (parenti, parentj)26: BOTTOMUPMATCH(pair, solution)27: end if28: end if29: end function

Algorithm 1 applies two heuristics to avoid the explora-tion of all possible matching pairs of NST nodes. At the leaflevel of the NSTs, the algorithm selects only the best match-ing pairs, while in the other levels it always selects the firstmatching pair. We introduced these two heuristics to makemore efficient the matching of long if-else-if chains, whereall if statements in the chain have similar conditionalexpressions and can be matched with each other, thus lead-ing to the problem of combinatorial explosion.

TABLE 2Detected Differences between Matched Nodes

Difference Type Example

Variable Identifier int x = y; int x = z;Literal Value int x = 0; int x = 1;Method Name foo(arg); bar(arg);Argument Number foo(); foo(arg0);Caller Expression expr.foo(); foo();Array Dimension x = a[i]; x = a[i] [j];Array Initializer a[] = {0, 1}; a[] = {1};Infix Ext. Operandsy x = 4 a; x = 3 b 2;Infix Left Operand 4 a = = 6 c; 4 a + 7 = = 6 d;Infix Right Operand a + b = = 3 c-d; a + c = = 3 d;Subclass Type ArrayList x Vector xAST Compatible int x = foo(); int x = 5;Field Access$

this.field getField()Getter callField Assignment$

this.field = a; setField(b);Setter call

Operator x = y + z; x = y z;Variable Type int x = 5; double x = 5;

y Extended infix operands is the way Eclipse JDT represents deeply nested infixexpressions of the form L op R op R2 op R3... where the same operator appearsbetween all the operands (the most common case being lengthy stringconcatenation expressions).* A subclass type difference denotes that two variables have different types,which are subclasses of a common superclass (e.g., AbstractList in thecase of ArrayList and Vector). On the other hand, a variable type differ-ence represents all other cases of variables having different types. We made thisdistinction, because the statements containing variables with different subclasstypes can be potentially unified by generalizing the variable types to thecommon superclass type.


3.3 Statement Mapping

In the previous step of our approach, we described an algo-rithm that extracts isomorphic subtrees from the NSTs ofthe clone fragments given as input. In this step, we presentan approach for finding a locally optimal mapping (seeSection 3.3.2) between the statements nested under the con-trol predicate nodes of the NST subtrees. Statement mappingis an injective function that associates each statement fromthe first clone fragment with at most one statement from thesecond clone fragment. The statements that cannot be asso-ciated with any statement from the other clone fragment(due to AST incompatibility) are considered as unmapped.

To facilitate the refactoring of duplicated code, an opti-mal mapping should not only contain a maximal number ofmapped statements, but also a minimal number of differen-ces between them. The minimization of differences is of keyimportance for the refactoring of clones, since it directlyaffects the number of parameters that have to be introducedin the extracted method containing the common functional-ity, as well as the feasibility of the refactoring transforma-tion. A large number of parameters makes more difficultthe use/reuse of the extracted method, since calling such amethod would require passing several arguments. Addi-tionally, a large number of differences implies a higherprobability for a precondition violation, since the parame-terization of some differences could cause a change in theprogram behavior (as explained in Section 3.4).

3.3.1 Motivation for Optimizing Statement Mapping

Fig. 3 illustrates two alternative mappings for two codefragments found in methods drawDomainMarker anddrawRangeMarker, respectively, within the classAbstractXYItemRenderer of the JFreeChart open-sourceproject (version 1.0.14). These methods contain over90 duplicated statements covering their entire body. How-ever, for the sake of simplicity, we have included only asmall portion of the duplicated code. The number next toeach statement indicates the index of this statement in theordered list of method statements.

Fig. 3a depicts the actual nesting structure of the twocode fragments on the left and right hand side, along with a

statement mapping as obtained from a matching approachselecting the first or best match in a top-down fashion. Twostatements positioned on the same line, next to each other,are considered as mapped (e.g., statement 61 on the lefthand side is mapped to statement 62 on the right hand sideof Fig. 3a). A matching approach that does not explore theentire search space always selects the first or best match inthe case of multiple possible matches (e.g., statement 61 onthe left hand side can be mapped to either statement 62 or74 on the right hand side). As a result, the mapping (61, 62)is the first match encountered in a top-down traversal, butalso the best match in terms of similarity, since statement 61on the left hand side is exactly the same with statement 62on the right hand side. By matching statement 61 with 62,and 73 with 74, we finally obtain the mapping solutionshown in Fig. 3a. This solution is maximal, since all 25 state-ments from each code fragment have been successfullymapped; however, it contains a large number of differencesbetween the mapped statements.

Fig. 3b depicts an optimal statement mapping, which isagainmaximal in terms of the number ofmapped statements,but also has the minimum number of differences betweenthe mapped statements. By examining carefully the codefragments, one can observe that the code inside the body ofstatement 61 on the left hand side is exactly the samewith thecode inside the body of statement 74 on the right hand side,and the same holds for statement 73 on the left hand sidewith statement 62 on the right hand side. Consequently, bydetecting the symmetrical structure of the two code frag-ments and parameterizing the differences in the conditionalexpressions of the respective if statements, we can obtainthe optimal mapping shown in Fig. 3b. This alternative map-pingmakes feasible the refactoring of the clones and introdu-ces significantly less parameters to the extractedmethod.

3.3.2 Decomposition of the Mapping Problem

The core of our statement mapping technique is a divide-and-conquer algorithm that breaks the initial mappingproblem into smaller sub-problems based on the nestingstructure of the isomorphic NST subtrees extracted in theprevious step.

Fig. 3. Example motivating the need for optimal mapping.


LetNSTi be the first subtree andNSTj the second one. Ina nutshell, Algorithm 2 performs a bottom-up processing ofevery level in the subtrees. For each node of NSTi at a givenlevel, it explores all possible pairs of matching control predi-cate nodes at the same level of NSTj. Each pair of matching

control predicate nodes is used as a starting point for theapplication of a graph matching algorithm (Section 3.3.3),which matches the Program Dependence subgraphs con-taining only the non-predicate nodes nested under the pred-icate nodes of the starting point. After the examination of allpossible matching combinations the best sub-solution (i.e.,the solution with the largest number of mapped nodes andthe smallest number of differences between them) isappended to the final solution. Algorithm 2 is essentially agreedy algorithm [21] that makes locally optimal choices ateach level of the NST subtrees with the hope of finding aglobally optimal solution.

Algorithm 2. ADivide-and-Conquer Statement MappingProcess Based onNesting Structure.

Input: two isomorphic NSTsOutput: the final mapping solution1: function PDGMAPPING (NSTi, NSTj)2: level = NSTi.maxLevel = NSTj.maxLevel3: solution = ?4: while level 0 do5: cpNodesi = nodes at level of NSTi6: cpNodesj = nodes at level of NSTj7: for each cpi 2 cpNodesi do8: states = ? "MCS states9: for each cpj 2 cpNodesj do10: if validNesting(cpi, cpj) then11: mapping = (cpi, cpj)12: root = createState(mapping)13: SEARCH(root,mapping)14: states = states [ findMCS(root)15: end if16: end for17: solution = solution [ best(states)18: end for19: decrement level20: end while21: end function

The examination of all possible pairs of matching controlpredicate nodes at every level of the subtrees, makes possi-ble the matching of symmetrical structures, as the onesshown in Fig. 3, as well as control predicates placed in a dif-ferent order within the clone fragments (i.e., Type-3 clonedifferences).

Function validNesting (line 10) ensures that the resultingsub-solutions can be combined to form a valid global solu-tion (i.e., a solution that complies with the nesting structureof the NST subtrees). This function takes as input two con-trol predicate nodes cpi and cpj in the NST subtrees andexamines three conditions:

1) Preservation of nesting structure: all nodes in the pathof cpi to the root of NSTi should be compatiblewith the corresponding nodes in the path of cpj to

the root of NSTj. This condition ensures that the

current mapping can lead to a final solution thatcovers the entire trees (i.e., there will always becompatible parents to be mapped until we reach theroots of the trees).

2) Preservation of sibling relationships: for all control pred-icate node mappings (ni, nj) created at the currentlyexamined level of the subtrees as part of a best sub-solution, if cpi is a sibling of ni inNSTi (i.e., cpi and nihave the same parent node in NSTi), cpj should be asibling of nj in NSTj and vice versa. This conditionensures that for all the predicate node mappings cre-ated in the current level, their actual siblings in thetrees will bemapped in the current level.

3) Preservation of parent-child relationships: for all controlpredicate node mappings (ni, nj) created at the previ-ously examined level of the subtrees as part of a bestsub-solution, if cpi is the parent of ni in NSTi, cpjshould be the parent of nj in NSTj and vice versa.This condition ensures that for all the predicate nodemappings created in the previous level, only theiractual parents in the trees will be mapped in the cur-rent level.

If these three conditions hold for every pair of controlpredicate nodes leading to a best sub-solution, then allresulting sub-solutions can be safely combined into a validglobal solution.

Function createState (line 12) creates an initial state ofthe search space containing only the node mappingpassed as argument. Function findMCS (line 14) returnsthe states corresponding to the maximum common sub-graphs in the search tree. These states are the leaf nodesin the deepest level of the search tree. Finally, functionbest takes as input a set of states and applies a three-stepelimination process to return the state with the largestnumber of mapped statements and the smallest numberof differences between the mapped statements. In the firststep, we keep only the states with the largest number ofmapped statements (max) and eliminate the states havinga number of mapped statements lower than max. In thesecond step, we keep only the states with the smallestnumber of distinct differences (min) and eliminate thestates having a number of distinct differences greaterthan min. The reason we decided to compare the numberof distinct differences (instead of the number of all differ-ences) is to avoid penalizing states that include the samevariable rename in many different syntactic positions. Asa result, all Variable Identifier differences corresponding tothe same pair of identifiers are considered as one distinctdifference regardless of the number of times they arerepeated in the clone fragments. In the third and finalstep, we select the state with the smallest number of non-distinct differences.

3.3.3 Program Dependence Subgraph Mapping

As explained in Section 3.3.2, the original statementmappingproblem is decomposed into smaller sub-problems, i.e.,mapping the sets of non-predicate statements ncpNodesi andncpNodesj nested under two control predicate nodes from

NSTi and NSTj, respectively. Each statement mapping sub-problem is expressed as a graph matching problem by


extracting the Program Dependence subgraphs containingonly the nodes in sets ncpNodesi and ncpNodesj, respec-

tively. The graph matching problem is solved by applying amaximum common subgraph algorithm.

The reason we expressed the statement mapping prob-lem as a PDG matching problem is to better support themapping of Type-3 clone fragments. As explained byKomondoor and Horwitz [22], the PDG is the ideal structureto find non-contiguous clones (i.e., clones whose state-ments do not occur as contiguous text in the program), andclones in which matching statements have been reordered.Therefore, a PDG mapping approach can reduce the ambi-guity of statement matching, since the similarity of twostatements can be assessed not only based on their textualor AST-structure similarity, but also based on the corre-spondence of incoming/outgoing data dependencies from/to other statements.

The detection of the maximum common subgraph is a wellknown NP-complete problem for which several optimaland suboptimal algorithms have been proposed in the liter-ature. Conte et al. [23] compared the performance of thethree most representative optimal algorithms, which arebased on depth-first tree search:

1) the McGregor algorithm [24] that searches for themaximum common subgraph by finding all commonsubgraphs of the two given graphs and choosing thelargest one.

2) the Durand et al. algorithm [25] that builds the associ-ation graph between the two given graphs and thensearches for the maximum clique of the latter graph.

3) the Balas & Yu algorithm [26] that also searches forthe maximum clique, but uses more sophisticatedgraph theory concepts for determining upper andlower bounds during the search process.

All three algorithms have a factorial worst case time com-plexity with respect to the number of nodes in the graphs, inthe order of N21!N2N11!, where N1 and N2 are the numbersof nodes in graphs G1 and G2, respectively [23]. The differ-ences among the three algorithms actually lie only in theinformation used to represent each state of the search space,and in the kind of the heuristic adopted for pruning searchpaths [23]. Conte et al. [23] concluded that the McGregoralgorithm is more suitable for the applications that use reg-ular graphs (i.e., graphs where each vertex has the samenumber of neighbors).

For the implementation of our MCS search technique(Algorithm 3), we have adopted the McGregor algorithm[24], because it is simpler to implement and has a lowerspace complexity, in the order of ON1, since only thestates associated to the nodes of the currently exploredpath need to be stored in memory [23]. The othertwo algorithms require the construction of the associationgraph between the two given graphs, which in the worstcase can be a complete graph with a space complexityin the order of ON1 N2 [23]. Given two PDG subgraphs,namely PDGi and PDGj, Algorithm 3 applies the follow-ing constraints:

1) An edge of PDGi is traversed only once in each pathof the search tree (line 6).

2) A node from PDGi is mapped to at most one nodefrom PDGj (and vice versa) in each path of thesearch tree (line 12).

3) Two edges edgei and edgej are considered compati-ble (line 9) if they connect nodes which are compati-ble (i.e., the nodes in the starting and ending pointsof the edges, respectively, should be compatible witheach other) and they have the same dependence type(i.e., they are both control or data flow dependen-cies). In the case of control dependencies, bothshould have the same control attribute (i.e., True orFalse). In the case of data dependencies, the dataattributes should correspond to variables havingthe same name, or to variables detected as renamedduring the AST compatibility analysis of theattached nodes. Finally, if both data dependenciesare loop-carried, then the loop nodes through whichthey are carried should be compatible too.

Algorithm 3. Recursive Function Building a Search Tree.

Input: a parent state in the search tree, a pair of mapped PDGnodes (nodei, nodej)

Output: a search tree1: function SEARCH(pState, nodeMapping)2: edgesi = nodeMapping.nodei.edges3: edgesj = nodeMapping.nodej.edges4: for each edgei 2 edgesi do5: visited = pState.visitedEdges6: if edgei 62 visited then7: visited = visited [ edgei8: for each edgej 2 edgesj do9: if compatible(edgei, edgej) then10: vNi = edgei.otherEndPoint11: vNj = edgej.otherEndPoint12: if not mapped(vNi) and not mapped(vNj)

then13: mapping = (vNi, vNj)14: state = createState(mapping)15: if not prune (state) then

16: stateadd! pState.children

17: SEARCH(state,mapping)18: end if19: end if20: end if21: end for22: end if23: end for24: end function

Algorithm 3 builds recursively a search tree by visitingthe pairs of mapped PDG nodes in depth-first order. Eachnode in the search tree is created when a new pair ofPDG nodes is mapped and represents a state of the searchspace. Each state keeps track of all visited edges andmapped PDG nodes in its path starting from the rootstate. Function createState (line 14) copies the visited edgesand mapped nodes from the parent state to the childstate. Function prune (line 15) examines the existence ofother leaf states in the search tree that already contain thenode mappings of the newly created state. In such a case,the branch starting from the newly created state is pruned


(i.e., not further explored). The reason we added this con-dition is because we realized that in several casesthe search algorithm was building branches containingexactly the same node mappings, but in different order.The leaf states in the deepest level of the search tree cor-respond to the maximum common subgraphs.

3.3.4 The Problem of Combinatorial Explosion

The reason we proposed the divide-and-conquer approach(Section 3.3.2) is that a direct application of Algorithm 3 onthe original problem (i.e., the complete PDGs) is likely tocause a combinatorial explosion. As the number of possiblematches for the nodes increases, the width of the search treeconstructed by the MCS algorithm grows rapidly as a resultof the numerous combinatorial considerations to beexplored. In order to reduce the risk of combinatorial explo-sion, we decided to take advantage of the nesting structureof the clone fragments and break the original mappingproblem into smaller sub-problems. However, there are stillsome cases that could cause a combinatorial explosion.

Fig. 4 shows two code fragments found in methodscreateHorizontalBlock (left hand side) andcreateVerticalBlock (right hand side), respectively,within the class StackedBarRenderer3D of the JFree-Chart open-source project (version 1.0.14). As it can beobserved all GeneralPath object creations (statements10, 16, 22, 28, 34, 40) can be mapped with each other,since the only difference is the name of the variable (bot-tom, top, back, front, left, and right). Additionally,all statements following the object creations (i.e., methodcalls moveTo, lineTo, and closePath) can be mappedwith each other due to the degree of freedom we allow inthe matching of expressions within the statements (e.g.,differences in method call names, number of arguments,and variable identifiers). This situation leads to a very

large number of matching statement combinations (i.e., asearch space explosion) that deteriorates dramatically theperformance of the MCS algorithm.

In order to tackle this problem, we perform an addi-tional decomposition of the mapping problem into sub-problems by finding subsets of related statements withinthe original set of statements whenever it is possible. Weconsider a subset of statements as related to each other, ifthey modify the state of an object referenced by the samevariable. For example, statements 10-15 (on the left handside of Fig. 4) modify the state of the object referenced bybottom, statements 16-21 modify the state of the objectreferenced by top and so on. The detection of statementsmodifying the state of an object referenced by ref is per-formed by examining the presence of composite variables(Section 2.2) in the form of ref.field inside the set ofDefined variables of each statement. After extracting thesubsets of statements in both code fragments, we applyAlgorithm 3 on the Program Dependence subgraphs cor-responding to each subset of statements. Again, each sub-graph from the first code fragment is matched with allsubgraphs from the second code fragment and the bestsub-solution (i.e., the solution with the largest number ofmapped nodes and the smallest number of differencesbetween them) is appended to the final solution.

Fig. 4 shows in dashed rectangles the subsets that weredetected in the two code fragments, as well as the mappingsolution resulting after the decomposition of statementsinto subsets. In this example, we can see that the developersessentially renamed variables bottom to right, top toleft, left to top, and right to bottom. It should beemphasized that the same mapping solution would beachieved even if the developers had reordered the subsetsof statements in the second code fragment, in addition tothe renaming of variables.

3.4 Preconditions

After the completion of the statement mapping process, weneed to determine whether the clone fragments can besafely extracted into a common method by parameterizingall existing differences between the mapped statements andmoving the unmapped statements before or after the execu-tion of the common statements.

According to Opdyke [27], each refactoring should beaccompanied with a set of preconditions, which ensure thatthe behavior of a program is preserved by the refactoring. Ifany of the preconditions is violated, then the refactoring isnot applicable, or its application would cause a change inthe program behavior. In this section, we define a set of pre-conditions that should be examined before the refactoringof duplicated code.

3.4.1 Preconditions Handling Differences between

Mapped Statements

In order to extract the duplicated code into a commonmethod, the differences between the mapped statementsshould be parameterized. Essentially, this means that theexpressions being different should be passed as argumentsto the extracted method call, and therefore these expressionswill be evaluated (or executed) before the execution of the

Fig. 4. Decomposing code fragments into subsets of statements to facethe problem of combinatorial explosion in the MCS algorithm.


extracted common statements. Obviously, a change in theevaluation or execution order of the parameterized expres-sions could cause a change in the program behavior.

In Fig. 5a, methods m1 and m2 in class B contain exactlythe same code with the exception of a difference in methodcalls a.foo() and a.bar(). In the first statement, bothmethods call a.getX() to read attribute x from object ref-erence a. In the next statement, the value of attribute x ismodified through method calls a.foo() and a.bar(),respectively. As a result, there exists an anti-dependence dueto variable a.x from the first to the second statement ofmethods m1 and m2, respectively. In order to mergethe duplicated code, the common statements are extractedin method ext(), as shown in Fig. 5b, and expressionsa.foo() and a.bar() are passed as arguments in the callsof the extracted method. This transformation breaks the pre-viously existing anti-dependence, since after the refactoring,variable a.x is first modified and then used. As a matter offact, a new inter-procedural data-dependence due to vari-able a.x is introduced after refactoring. The breaking of theoriginal anti-dependence is causing a change in the behav-ior of the program. In the original version in Fig. 5a, theexecution of method test results in m1 printing 0 and m2printing 1, while in the refactored version in Fig. 5b theexecution of method test results in m1 printing 1 andm2 printing 6. In a similar manner, the breaking of data-dependencies or output-dependencies could also cause achange in the program behavior.

Therefore, we propose the following precondition tohandle the differences existing between mapped state-ments. This precondition excludes the differences corre-sponding to variables that have been renamed betweenthe two clone fragments, because such differences do notrequire parameterization.

Precondition 1. The parameterization of the differ-ences between the mapped statements should notbreak any existing control, data, anti, and outputdependencies.

In order to detect such a precondition violation for theexpressions ei and ej being different between two mappedstatements si and sj, we first have to find the sets of varia-bles Vi and Vj (including also composite variables) that aremodified or used by ei and ej. Let Mi and Mj be the sets ofstatements that have been mapped from the first and thesecond clone fragment in methods mi and mj, respectively.Precondition #1 is violated if either statement si or sj has anincoming dependence from a mapped statement due to avariable in Vi and Vj, respectively. Formally,

fpi v!si 2 Dmi j pi 2 Mi ^ si 2 Mi ^ v 2 Vig 6 ? orfpj v!sj 2 Dmj j pj 2 Mj ^ sj 2 Mj ^ v 2 Vjg 6 ? ,

where pv!s denotes a control, data, anti, or output depen-

dence from statement p to statement s due to variable v, andDm denotes the set of dependencies in the PDG of methodm. If statement si or sj has a self-loop dependence (i.e., adependence starting from and ending to the same state-ment) due to a variable belonging to sets Vi and Vj, respec-tively, which is carried through a loop statement l, thenPrecondition #1 is violated as long as l belongs to themapped statements.

In our approach, we consider a more strict version ofthe original control dependence definition, which we calluse-constrained control dependence. Statement s is con-trol dependent to predicate statement p due to variable v,if s is nested (directly or indirectly) under p, p examinesthe value of v in its condition, and s uses the value of v.We consider that a variable is examined in a predicatestatement, if the variable appears somewhere in its condi-tion; for example, in the left or right operand of a rela-tional operator (, !, >, ,

under a try block p, and s contains a method call, or athrow statement throwing an exception handled by p.

In the example shown in Fig. 5a, we would find thatvariable a.x is modified through method calls a.foo() anda.bar() (i.e., the expressions being different between themapped statements). Therefore, Vi a : xf g and Vj a : xf g.The parameterization of expressions a.foo() and a.bar()would eliminate the original anti-dependencies due to vari-able a.x from themapped statements int x a.getX();.

It should be noted that we exclude the dependenciesfrom unmapped statements, because these statements willbe moved in the original methods either before or after theexecution of the extracted method containing the mappedstatements (assuming that their move does not violate Pre-condition #5 that will be explained in Section 3.4.2), andtherefore any existing dependencies to/from the parameter-ized expressions will be preserved.

In the case where some mapped statements in the clonefragments contain variables having different subclass typesof a common superclass, our statement matching approachreports a Subclass Type difference (Section 3.1, Table 2). Inorder to unify properly the statements containing such dif-ferences into a single statement, we should generalize thedifferent variable types to the common superclass type [28].Practically, this can be achieved by finding the declarationof the variable in the unified code or in the parameters ofthe extracted method and setting the type to the commonsuperclass type. This unification mechanism can be appliedas long as the mapped statements in the clone fragments arenot calling methods declared in the subclasses (excludingcalls to overridden methods) through these variables.

Therefore, we propose the following precondition to han-dle the Subclass Type differences existing between mappedstatements.

Precondition 2. Matched variables having differentsubclass types should call only methods that aredeclared in the common superclass or are beingoverridden in the respective subclasses.

Let si and sj be a pair of mapped statements that use vari-ables vi and vj having the subclass types ti and tj, whereti 6 tj. In order to detect such a precondition violation, wefirst have to find the sets of methodsMCi in class ti andMCjin class tj called through vi and vj, respectively. Formally,

MCi fmc:declaration j mc:class ti ^mc:invoker vigMCj fmc:declaration j mc:class tj ^mc:invoker vjgwheremc denotes a method call inside statement s.

Next, for each pair of method declarations in sets MCiand MCj having the same signature, we examine whether

the common superclass of ti and tj declares or inherits a

method with the same signature (i.e., the subclasses over-ride a method of the common superclass). If no suchmethod is found in the common superclass, then Precondi-tion #2 is violated. From our analysis, we exclude themethod declaration signatures that are not common in setsMCi and MCj, because the corresponding method calls in

the mapped statements will have to be parameterized, since

they refer to different methods. Therefore, these cases willbe examined with Precondition #1.

Another category of differences whose parameterizationwould cause a change in the behavior of the program isrelated to fields (instance variables) being modified in theclone fragments. Fig. 7 shows a case of two clone fragments,found in class Plot of the JFreeChart project, that modifythe value of fields backgroundPaint and outline-Paint, respectively, in lines 3 and 8. These two clone frag-ments can be unified by introducing a parameter for thefields being different in the extracted method. However, theextracted method will update only the value of the localparameter and will not update the values of the originalfields passed as arguments, since all parameters are passedby value in Java.

Therefore, we propose the following precondition to han-dle differences in fields being modified by assignment state-ments or increment/decrement operators. We restrict theeffect of this precondition only to modified fields and notlocal variables, because the extracted method should returnthe values of the local variables being modified within clonefragments (Precondition #6 that will be explained inSection 3.4.3 handles the case of multiple returned variables).

Precondition 3. The parameterization of fieldsbelonging to differences between the mappedstatements is possible only if they are not modified.

The final precondition is related to the return type of themethod calls found in the differences between the mappedstatements. The corresponding method declarations shouldnot return the void type, since it is not possible to introducea parameter of void type in the extracted method.

Precondition 4. The parameterization of methodcalls belonging to differences between the mappedstatements is possible only if they do not return avoid type.

3.4.2 Preconditions Handling Unmapped Statements

The statement mapping process may result in unmappedstatements. These statements could not be mapped withany statement from the other clone fragment becauseeither there is no corresponding statement (i.e., state-ments that exist in only one of the clone fragments), orthere exists a corresponding statement, but it has been soextensively modified that its AST structure is no longercompatible (Section 3.1) (i.e., statements that cannot be

Fig. 7. Example of clone fragments modifying fields.


unified due to incompatible AST structure). Assumingthat the mapping process resulted in a set of unmappedstatements from the first and the second clone fragmentin methods mi and mj, respectively, these statementsshould be moved either before or after the call of theextracted method containing the mapped statementsinside mi and mj, respectively. As in the case of theparameterization of differences, the move of theunmapped statements could break existing data-, anti-,and output-dependencies from/to mapped statements.Therefore, we propose the following precondition to han-dle the unmapped statements.

Precondition 5. The unmapped statements should bemovable before or after the mapped statementswithout breaking existing control, data, anti, andoutput dependencies.

In order to detect such a precondition violation, we devel-oped two rules that examine whether moving an unmappedstatement before the first or after the last mapped statement,respectively, would break existing dependencies. LetMi andMj be the sets of statements that have beenmapped from the

first and the second clone fragment in methods mi and mj,

respectively. Let Ui and Uj be the sets of statements that have

not been mapped from the first and the second clone frag-ment inmethodsmi andmj, respectively.

Moving a statement before the mapped statements. Anunmapped statement belonging to Ui or Uj cannot bemoved before the mapped statements if it has an incom-ing dependence from a statement in Mi and Mj, respec-

tively. Formally,

fpi v!si 2 Dmi j pi 2 Mi ^ si 2 Uig 6 ? orfpj v!sj 2 Dmj j pj 2 Mj ^ sj 2 Ujg 6 ?

where pv!s denotes a control, data, anti, or output depen-

dence from statement p to statement s due to variable v,and Dm denotes the set of dependencies in the PDG ofmethod m. If statement si or sj has a self-loop dependence

(i.e., a dependence starting from and ending to the samestatement), which is carried through a loop statement l,then it can still not be moved before the mapped state-ments as long as l belongs to the mapped statements.

Moving a statement after the mapped statements. Anunmapped statement belonging to Ui or Uj cannot be movedafter the mapped statements if it has an outgoing depen-dence to a statement inMi andMj, respectively. Formally,

fpi v!si 2 Dmi j pi 2 Ui ^ si 2 Mig 6 ? orfpj v!sj 2 Dmj j pj 2 Uj ^ sj 2 Mjg 6 ?

where pv!s denotes a data-, anti-, or output-dependence

from statement p to statement s due to variable v, andDm denotes the set of dependencies in the PDG ofmethod m. Additionally, if an unmapped statement isusing a local variable that is modified by the mappedstatements, then the extracted method should return thevalue of this variable.

3.4.3 Preconditions Related to Method Extraction

Murphy-Hill and Black [29] have recorded the most com-mon preconditions for the Extract Method refactoring thatwere encountered during a formative study, in which theyobserved 11 programmers performing a number of ExtractMethod refactoring operations using the Eclipse refactoringtool. Their list [29] includes the following preconditions:

In this section, we will adjust these preconditions to theExtract Clone refactoring (i.e., extracting two duplicatedcode fragments into a separate method).

Regarding the first precondition in the list of Murphy-Hill and Black, our approach guarantees that the mappedstatements will always form a complete syntactic unitin two ways. First, in the implementation of Algorithm 1(i.e., the algorithm that finds exactly paired subtrees withinthe nesting structures of the clone fragments), we take spe-cial care in the matching of if/else and switch casestructures. Two if statements can be matched only if theyhave matching if-else-if chain structures. Additionally, twoswitch statements can be matched only if the case state-ments nested inside them are exactly paired. In this way,we make sure that the list of mapped statements will notcontain any partially matched control predicate structures.Our approach for matching nested control structureshas several similarities with the way that the clone detectionalgorithm proposed by Koschke et al. [30], which isbased on abstract syntax suffix trees, handles nested sequen-ces (i.e., blocks). Second, our AST comparison mechanism(Section 3.1) examines the entire AST structure of the non-predicate statements, and therefore by definition there areno partially matched non-predicate statements. As a result,the mapped statements to be extracted constitute a list ofsyntactically complete statements.

The second precondition in the list of Murphy-Hill andBlack can be adjusted as follows:

Precondition 6. The mapped statements within theclone fragments should return at most one variableof the same type to the original methods fromwhich they are extracted.

In order to detect such a precondition violation, we firsthave to determine the sets of variables RVi and RVj thatshould be returned by the mapped statements to the origi-nal methods mi and mj, respectively. We take into account

1) The selected code must be a list of statements.2) Within the selection, there must be no assignments to

variables that might be used later in the flow of exe-cution. For Java, this can be relaxed to allow assign-ment to one variable, the value of which can bereturned from the new method.

3) Within the selection, there must be no conditionalreturns. In other words, the code in the selectionmust either always return, or always flow beginningto end.

4) Within the selection, there must be no branches tocode outside of the selection. For Java, this means nobreak or continue statements, unless the selectionalso contains their corresponding targets.


only local variables that are declared within the bodies ofmiand mj, as well as the parameters of mi and mj. We excludeinstance variables (i.e., class fields) that may be modifiedwithin the clone fragments, because they have a class scope,and thus it is redundant to be returned from the extractedmethod. Let Mi and Mj be the sets of statements that havebeen mapped from the first and the second clone fragment,respectively. Let Ri and Rj be the sets of statements thatwill remain in mi and mj, respectively, after the extractionof the mapped statements. Formally, Ri Ai nMi andRj Aj nMj, where sets Ai and Aj include all statementswithin the bodies of mi and mj, respectively. In general,when extracting a code fragment from a method, the varia-bles that should be returned to the original method arethose for which a data dependence exists from the set ofextracted statements to the set of remaining statements. Inthe case of Extract Clone refactoring, we formally define

RVi fv 2 V mi j p v!q 2 Dmi ^ p 2 Mi ^ q 2 RigRVj fv 2 V mj j p v!q 2 Dmj ^ p 2 Mj ^ q 2 Rjg

where pv!q denotes a data-dependence from statement p to

statement q due to variable v, V m denotes the set of varia-bles which are declared within the body of method m(including the parameters of m), and Dm denotes the setof dependencies in the PDG of methodm.

Precondition #6 is violated if RVij j > 1, or RVj

> 1, or

RVij j 6 RVj

. If RVij j RVj

1, then the variable in RVi

should have the same type with the variable in RVj.One way to overcome the problem of multiple returned

variables would be to make the extracted method return abean object in which the returned variable values areassigned to appropriate fields. Although this is a feasiblesolution, it requires the introduction of a new class, whichwill be instantiated only by the extracted method (i.e., aclass instantiated only once in the entire system). The beanobject solution would make more sense, if there were sev-eral clone fragments in different clone groups returning thesame set of variables. In that case, each extracted methodwould instantiate a bean object, thus making the newlyintroduced class more reusable.

The third precondition in the list of Murphy-Hill andBlack can be adjusted as follows:

Precondition 7. The mapped statements within theclone fragments should not contain any conditionalreturn statements.

A conditional return statement, as the one shown below,can be used to branch out of a control flow block anddirectly exit a method.

public void originalMethod(){

...

if (condition)

return;

...

}

Extracting a piece of code containing a conditionalreturn statement, and then simply calling the extracted

method, would make the original method not to exit inthe same way as it did before. One way to overcome thisproblem would be to make the extracted method return aboolean flag that is set to true when the conditionalreturn statement is reached in the extracted code, and isset to false otherwise.

private boolean extractedMethod() {

boolean flag = false;

if (condition){

flag = true;

return flag;

}

...

return flag;

}

If the value of the returned flag is true, then the originalmethod should exit directly after the execution of theextracted method as shown below.

public void originalMethod(){

...

boolean flag = extractedMethod();

if (flag)

return;

...

}

However, this solution requires to insert additional con-ditional code in both the original and the extracted methods,thus increasing the initial complexity of the code.

The fourth precondition in the list of Murphy-Hill andBlack can be adjusted as follows:

Precondition 8. The mapped branching statements(break, continue) should be accompanied withthe corresponding mapped loop statements.

In Java the unlabeled break statement is used to termi-nate the innermost for, while, or do-while loop, or theenclosing switch statements. The unlabeled continuestatement is used to skip the current iteration of the inner-most for, while, or do-while loop. As a result, whentwo branching statements are mapped the correspondingloops should be also mapped. Otherwise the extraction of abranching statement without the corresponding loop wouldcause a compilation error. In the same manner, the labeledbreak statement (i.e., break label;) terminates the outerloop marked with the specified label, and the labeledcontinue statement (i.e., continue label;) skips thecurrent iteration of the outer loop marked with the specifiedlabel. When two labeled branching statements are mapped,the corresponding loops marked with the specified labelsshould be also mapped. It should be noted that ourapproach supports consistently renamed labels between theclone fragments (i.e., when the label used in the branchingstatements is renamed in the same way with the label mark-ing the outer loop).

3.5 Limitations

A major limitation of the proposed approach is that it doesnot support the analysis and refactoring of clone groups


(also known as clone classes) containing more than twoclone fragments. In order to support the analysis of clonegroups, the proposed approach should be extended to findlargest common subtrees in the nesting structures of multi-ple clone fragments. Then, the differences among the clonefragments of the group could be extracted by applying ourstatement mapping approach on every possible combina-tion of clone pairs in the group, and summarizing the dif-ferences corresponding to the same syntactic positions (i.e.,a single parameter should be introduced in the extractedmethod for all the differences found in the same syntacticposition). Finally, by examining the same set of precondi-tions on all differences extracted in the previous step, wecould determine whether the entire clone group is refactor-able (i.e., a clone group is refactorable if there are no pre-condition violations). Obviously, adapting the currentclone-pair-based approach to support the analysis of clonegroups might cause serious scalability problems, especiallywhen the size of a clone group is large.

Recently, Lin et al. [31] proposed an approach for detect-ing differences across multiple clone instances. In theirapproach, MCIDiff (Multi-Clone-Instances Differencing),each clone instance in a clone group is parsed into asequence of tokens. Then, MCIDiff computes the LongestCommon Subsequence (LCS) of all clone instances in theclone group, and analyzes the LCS to determine differentialranges across the clone instances. Finally, it produces as out-put a list of differential multisets of tokens that summarizedifferences that can be found in parameterized and gappedclones. Finding the global optimal alignment for N sequen-ces is an NP-complete problem. Therefore,MCIDiff adopts aprogressive alignment approach to compute an approxi-mate solution. The preconditions defined in this paper canbe certainly examined on the differences extracted byMCIDiff to assess whether a clone group can be safely refac-tored. However, this would first require to map the tokensreturned by MCIDiff to appropriate AST nodes in order todetermine the expressions to be parameterized (e.g., if atoken corresponds to a partial AST expression, then theentire AST expression should be parameterized). This is ageneral limitation of token-based differencing approaches,because they do not consider the syntactic structure of theprogram [31]. Next, the PDGs of all the methods involvedin the clone group will have to be generated in order extractthe dependencies required for the examination of the pre-conditions. This additional cost of AST and PDG generationcould perhaps justify to build a solution that operatesdirectly on ASTs and PDGs, like our approach.

4 TOOL SUPPORT

The proposed technique for assessing the refactorability ofsoftware clones has been implemented as an Eclipse plug-in, which is part of the JDeodorant1 code smell detectionand refactoring suite, and can be used in two ways:

1) In the GUI mode the user interacts directly with theEclipse IDE by selecting pairs of methods containingduplicated code fragments to be analyzed for poten-tial refactoring opportunities.

2) In the batch processing mode the user specifies as inputfiles containing results from clone detection tools,and our Eclipse plug-in is executed in headlessmode analyzing all clone pairs found in the inputfiles and generating a report.

In the following sections, we present the features offeredby each execution mode.

4.1 GUI Mode

In this mode, the user has to select two methods containingduplicated code fragments either in the Package Explorer ortheOutline view of Eclipse. The Package Explorer view allowsthe selection of methods belonging to different Java files,while the Outline view allows the selection of methods onlyfrom the same Java file. By right-clicking on the selectedmethods and selecting Refactor Duplicated Code... fromthe context menu, the dialog shown in Fig. 8 appears afterthe analysis of the methods. In this dialog the user caninspect the refactoring opportunities detected in the selectedmethods. If more than one refactoring opportunity is pres-ent (i.e., multiple isomorphic subtrees have been detected inthe nesting structures of the methods), the user can inspecteach one of them separately by selecting the correspondingoption in the Select Refactoring Opportunity combo box(Fig. 8, point 1).

4.1.1 Clone Visualization

The mapped and unmapped statements corresponding toeach refactoring opportunity selected by the user are visual-ized in two side-by-side tree structures representing thenesting structures of the duplicated code fragments, asshown in Fig. 8. Each node in the tree structures representsa mapped or unmapped statement. The unmapped state-ments are highlighted in red color, while the differencesbetween the mapped statements are highlighted in greencolor. The two tree structures are synchronized in the sensethat collapsing/expanding a node in the first tree will auto-matically collapse/expand the corresponding node in thesecond tree and vice versa. Additionally, the vertical andhorizontal scrollbars surrounding the tree structures aresynchronized, so that the same code area of the clone frag-ments is always displayed when scrolling.

When the user hovers over the mapped/unmappedstatements a tooltip appears with the following information:

1) Semantic differences. As it has been shown in Table 2(Section 3.1), our approach can detect various typesof differences between the mapped statements. As inthe case of CloneDifferentiator [32], our approach isaware of the program elements in which the differ-ences occur, and thus it can provide a more mean-ingful explanation of the differences (Fig. 8, point 2)compared to text differencing techniques that ignoresemantic information. In addition, the detectedsemantic differences are used to improve the qualityof the applied refactoring transformation by avoid-ing redundant parameterizations. For example, a dif-ference regarding a field access that is replaced withthe corresponding getter method call should not beparameterized, since the involved expressions are1. http://www.jdeodorant.com


semantically equivalent and thus one of them can beused in the unified code.

2) Precondition violations. Our approach examines allpreconditions described in Section 3.4 and presentson each (mapped or unmapped) statement the corre-sponding precondition violations (Fig. 8, points 3and 4).

3) Suggestions. In some cases of precondition violations,our approach makes suggestions that could makethe examined clone fragments refactorable. Forexample, if there is a difference involving a privatemethod call that cannot be parameterized due to aprecondition violation, our tool suggests to inline thecalled method. Additionally, if there are statementsthat cannot be unified, because they access variableshaving different class types, our tool suggests tomake these class types extend a common superclass.

Finally, our approach detects the variables that havebeen consistently renamed in the clone fragments andpresents them to the user (Fig. 8, point 5). Differencesinvolving renamed variables should not be parameterized,and therefore are not examined against preconditions. Thealgorithm for the detection of consistently renamed varia-bles works as follows:

Let set D include all distinct Variable Identifier differences(see Table 2 in Section 3.1) that were detected in the examinedclone fragments. For each d 2 D, where d is a pair of variableidentifiers in the form vi; vj, we examine two conditions.

1) If D contains a pair of variable identifiers in the formvi; y, where y 6 vj or x; vj, where x 6 vi, then d isnot considered as a consistent variable rename. Inthat case, either vi or vj has been replaced with two ormore different identifiers in the other clone fragment.

2) Let si and sj be a pair of mapped statements and Dsthe set of Variable Identifier differences detectedbetween these two statements. If si uses variable vior sj uses variable vj, and vi; vj =2 Ds, then d is notconsidered as a consistent variable rename. In thatcase, either vi or vj has not been replaced with anyidentifier in the other clone statement.

The second condition is examined on every pair ofmapped statements. The proposed algorithm for thedetection of renamed variables is greatly affected by thequality of the statement mapping solution produced byour approach.

4.1.2 Clone Refactoring

If there are no precondition violations the user can proceedto refactor the clone fragments. By clicking on thePreview button, the user can have a detailed preview ofall the changes that will take place in the code after theapplication of the refactoring. Our tool supports the follow-ing refactoring scenarios at the moment:

1) Extract Method is applied when the clone fragmentsare located in methods that belong to the same class.

Fig. 8. A visualization of the differences and precondition violations detected in method isRebuildRequired of the classes WeblogicDeploymentTool (left) and WebsphereDeploymentTool (right) in Apache Ant 1.9.0 project.


In this scenario the unified code is extracted in a newprivate method within the same class.

2) Extract and pull up method is applied when the clonefragments are located in methods that belong to dif-ferent subclasses of the same superclass. If the super-class is extended by only these two subclasses, thenthe unified code is placed in a new protected methodwithin the superclass. If there are more subclassesextending the superclass, then the unified code isplaced in a protected method within a new interme-diate class extending the common superclass andbeing inherited by the two subclasses. Fieldsand methods declared in the subclasses that arecommonly accessed in the clone fragments and havean identical structure (i.e., fields having the sametype and name, and methods being Type-1 clonesor Type-2 clones with only local variable renames asdifferences) are also pulled up.

3) Introduce template method is a special case of the previ-ous refactoring. If the methods being commonlyaccessed in the clone fragments do not belong to theaforementioned clone types, but have an identicalsignature and the same return type, then an abstractmethod with the same signature is created in thesuperclass where the unified code is pulled up. Afterthe application of the refactoring, the unified codewill call the newly introduced abstract method thatis overridden in the two subclasses. Therefore, thisrefactoring introduces an instance of the TemplateMethod design pattern [33].

4) Introduce utility method is applied when the clonefragments are located in methods of unrelatedclasses (not being part of the same inheritancehierarchy), and the fragments do not access anyinstance variables or methods. Then, the unifiedcode can be extracted into a static method placedin a utility class.

4.2 Batch Processing Mode

This mode is suitable for large-scale refactorability analy-sis of the clones detected in an entire Java project. Toenable this kind of analysis, we created a separate Eclipsecommand-line application that executes the JDeodorantplug-in in headless mode for each clone pair reportedby a clone detection tool. The user has to provide thefollowing input:

1) The path to the file/folder containing the clonedetection results.

2) The name of the clone detection tool. Currently,CCFinder, Deckard, and NiCad clone detection toolsare supported.

3) The name of the Eclipse Java project in which theclones were detected. This project should be open inthe Eclipse workspace.

4) The path to the output file of the refactorability anal-ysis report.

The tool performs the analysis in two steps. In the firststep, it parses the clone detection results and generates aspreadsheet containing some basic information about thedetected clones (Fig. 9). Each row in the spreadsheet corre-sponds to a clone instance and contains information such asthe clone group id of the clone instance, the class and methodthat the clone belongs to, the start/end line and offset of theclone in the Java file it belongs to. Additionally, based onthe recorded location and start/end offset of the cloneinstances, the tool determines whether some instances havea sub-clone relationship and records this information in acolumn of the spreadsheet. Clone y is a sub-clone of x, if xand y belong to the same class and method, and start-offset(y) start-offset(x) and end-offset(y) end-offset(x).

In the second step, the tool parses the source code of thespecified Java project in which the clones were detected.Next, it processes the clone instances of each clone group byexamining all possible combinations of clone pairs in thegroup. For each clone instance (i.e., row in the spreadsheet)in the examined group, the tool locates the method in whichthe clone fragment belongs to and generates the methodsPDG. At this point, the corresponding row in the spread-sheet is updated with some additional source code analysisinformation extracted from the PDG, such as the total num-ber of statements in the PDG and the clone fragment,respectively. If a clone instance extends beyond the bound-aries of a method (i.e., class-level clone), then the entireclone group is excluded from the analysis.

For each examined clone pair the tool applies the pro-posed refactorability analysis approach, which results intwo pieces of information. The first piece is the statementmapping information, which includes the statements thathave been mapped, the differences between the mappedstatements, and the unmapped statements from each clonefragment. The second piece is the precondition violationinformation, which includes all examined preconditionsthat failed. Both pieces of information are combined into anHTML report as the one shown in Fig. 10. This report makesuse of advanced JavaScript and cascading style sheets (CSS)features to give an experience to the user similar to the GUImode (Section 4.1) experience during the inspection of the

Fig. 9. A spreadsheet containing the refactorability analysis results for a clone group.


refactorability analysis results. In the same manner, the usercan dynamically interact with the report by collapsing/expanding nodes in the side-by-side tree clone structurevisualization, and hovering over mapped and unmappedstatements to get more information about the differencesand violated preconditions in the form of tooltips. Fig. 10shows an example of two clone fragments having multiple(often combined) advanced differences, such as the replace-ment of field accesses with getter method calls, the replace-ment of field assignments with setter method calls, and thereplacement of this reference with a variable (i.e., task).

After the generation of the HTML report, a cell with ahyperlink to the report is inserted into the spreadsheet asshown in the Details area on the right hand side of Fig. 9.The id of the cell and the corresponding HTML report filename has the format a-b-c, where a is the clone group id, b isthe position of the first clone instance in the group, and c isthe position of the second clone instance in the group. Thecell gets a green background color if the clone pair is refactor-able (i.e., the number of precondition violations is equal tozero), and a red background color in the opposite case. Thecoloring of the cells allows to easily distinguish the refactora-ble from the non-refactorable clone pairs, and also discoversome interesting patterns among the examined clone pairs.For instance, Fig. 9 shows a clone group consisting of sixclone instances detected by Deckard in Apache Ant 1.7.0project. The Details area presents the results from the anal-ysis of all possible clone pair combinations (15 in total). Wecan easily observe that the sixth (last) clone instance wasassessed as non-refactorable in all clone pair combinations in

which it is involved. This is a clear indication that the lastclone instance has a relatively weak similarity with the rest ofthe instances in the group, and therefore should be excludedfrom this particular clone group in the refactoring process.

The reason we selected the combination of a spreadsheetwith links to HTML documents for reporting the refactor-ability analysis results is twofold. First, spreadsheets allowthe application of various column-filters in order to filterout undesired clone groups, such as groups without anyrefactorable clone pairs, and groups containing class-levelclone instances, or sub-clone instances. Second, these docu-ment types are supported in all operating systems by stan-dard applications (e.g., spreadsheet processors and webbrowsers), and thus there is no need to install additionalsoftware in order to inspect the results.

5 EVALUATION

The evaluation section is organized into four parts. In thefirst part (Section 5.1), we describe the process that we fol-lowed for collecting our experimental data.

In the second part (Section 5.2), we evaluate the correct-ness of the proposed refactorability analysis approach. Toachieve this goal, we refactored 610 clone pairs that havebeen assessed as refactorable and were completely orpartially covered by unit tests. We consider a positive refac-torability assessment as correct, if the corresponding refac-toring is applicable in practice without introducing compileerrors and there are no unit test failures after the applicationof the refactoring.

Fig. 10. A dynamic HTML report with the refactoring opportunities detected in method transferFiles of the classes FTP (left) andFTPTaskMirrorImpl (right) in Apache Ant 1.9.0 project.


In the third part (Section 5.3), we evaluate the perfor-mance of our approach. For each examined clone pair wecollected the execution times corresponding to all threephases of our technique (i.e., the detection of common nest-ing structures within the clone fragments, the mappingof the statements within the common nesting structures,and the examination of preconditions). In addition, we col-lected the total number of distinct statement comparisonsperformed by our technique for each clone pair, and made acomparison with a hypothetical exhaustive search approachthat does not take into account the nesting structure of theclone fragments.

In the fourth part (Section 5.4), we perform a large-scaleempirical study on the clones detected by four differentstate-of-the-art clone detection tools in nine Java open-source systems to investigate whether and how the refactor-ability of software clones is affected from various cloneproperties, such as the clone source code nature (productionversus test code), the relative clone location, the clone type,and the clone size.

5.1 Experiment Setup

In this section, we provide information about the selectionof the subject systems and clone detection tools used in thestudy, as well as the process we followed for collecting theexperimental data.2

5.1.1 Subject Selection

In order to avoid bias in the selection of projects, weadopted the systems used in the study conducted byTairas and Gray [34]. As shown in Table 3, the listincludes nine Java open-source projects coming from dif-ferent application domains and having a different devel-opment history, ranging from 2 to 8 years. These twovariation points certainly affect the characteristics of thedetected clones with respect to their domain-specificityand the maturity of the involved code, thus allowing formore generalizable results. Additionally, the projects varyin size ranging from 50 to 200 KLoC.

5.1.2 Clone Detector Selection

As it is evident from the qualitative study performed by Royet al. [1], there is a large number of available clone detectiontools (over 40), which makes more difficult the selectionof tools for the context of our study. Roy et al. [1] catego-rized the clone detection approaches into five categories,namely text-based, token-based, tree-based, metrics-based,and graph-based. According to this categorization the text-based, token-based, and tree-based techniques are the mostdominant (i.e., these three categories have a comparativelylarger number of available tools).

Therefore, for our experiment we considered the threemost dominant categories of clone detection techniques (i.e.,text-based, token-b

IEEE TRANSACTIONS ON SOFTWARE …nikolaos/publications/...Assessing the Refactorability of Software...

Documents

Transcript of IEEE TRANSACTIONS ON SOFTWARE …nikolaos/publications/...Assessing the Refactorability of Software...