Transfer Learning for Improving Model Predictionsin Highly Configurable Software

Pooyan Jamshidi, Miguel Velez, Christian KastnerCarnegie Mellon University, USA

{pjamshid,mvelezce,kaestner}@cs.cmu.edu

Norbert SiegmundBauhaus-University Weimar, Germany

norbert.siegmund@uni-weimar.de

Prasad KawthekarStanford University, USApkawthek@stanford.edu

Abstract—Modern software systems are built to be used indynamic environments using configuration capabilities to adapt tochanges and external uncertainties. In a self-adaptation context,we are often interested in reasoning about the performance ofthe systems under different configurations. Usually, we learna black-box model based on real measurements to predictthe performance of the system given a specific configuration.However, as modern systems become more complex, there aremany configuration parameters that may interact and we end uplearning an exponentially large configuration space. Naturally,this does not scale when relying on real measurements in theactual changing environment. We propose a different solution:Instead of taking the measurements from the real system, welearn the model using samples from other sources, such assimulators that approximate performance of the real system atlow cost. We define a cost model that transform the traditionalview of model learning into a multi-objective problem that notonly takes into account model accuracy but also measurementseffort as well. We evaluate our cost-aware transfer learningsolution using real-world configurable software including (i) arobotic system, (ii) 3 different stream processing applications,and (iii) a NoSQL database system. The experimental resultsdemonstrate that our approach can achieve (a) a high predictionaccuracy, as well as (b) a high model reliability.

Index Terms—highly configurable software, machine learning,model learning, model prediction, transfer learning

I. INTRODUCTION

Most software systems today are configurable, which givesend users, developers, and administrators the chance to cus-tomize the system to achieve a different functionality ortune its performance. In such systems, hundreds or eventhousands of configuration parameters can be tweaked, makingthe system highly configurable [36]. The exponentially grow-ing configuration space, complex interactions, and unknownconstraints among configuration options make it difficult tounderstand the performance of the system. As a consequence,many users rely on default configurations or they change onlyindividual options in an ad-hoc way.

In this work, we deal with the type of configurable systemsthat operate in dynamic and uncertain environments (e.g.,robotic systems). Therefore, it is desirable to react to environ-mental changes by tuning the configuration of the system whenwe anticipate that the performance will drop to an undesirablelevel. To do so, we use black-box performance models thatdescribe how configuration options and their interactions influ-ence the performance of a system (e.g., execution time). Black-box performance models are meant to ease understanding,debugging, and optimization of configurable systems [36]. Forexample, a reasoning algorithm may use the learned model in

Fig. 1: Transfer learning for performance model learning.

order to identify the best performing configuration for a robotwhen it goes from indoor to an outdoor environment.

Typically, we learn a performance model for a given con-figurable system by measuring from a set of configurationsselected by some sampling strategy. That is, we measure theperformance of a given system multiple times in differentconfigurations and learn how the configuration options andtheir interactions affect performance. However, such a wayof learning from real systems, whether it is a robot or asoftware application, is a difficult task for several reasons:(i) environmental changes (e.g., people wandering aroundrobots), (ii) high costs or risks of failure (e.g., a crashed robot),(iii) the large amount of time required for measurements (e.g.,we have to repeat the measurements several times to get areliable value), and (iv) changing system dynamics (e.g., robotmotion). Moreover, it is often not possible to create potentiallyimportant scenarios in the real environment.

In this paper, as depicted in Figure 1, we propose a differentsolution: instead of taking the measurements from the realsystem, we reuse prior information (that we can get fromother sources at a lower cost) in order to learn a performancemodel for the real system faster and cheaper. The conceptof reusing information from other sources is the idea behindtransfer learning [38], [32]. Similar to human beings that canlearn from previous experience and transfer the learning toaccomplish new tasks easier, quicker, and in a better way, inthis work, we use other sources to provide cheaper samplesfor accelerating model learning. Instead of taking the mea-surements from the real system (we refer to as the target), wemeasure the system performance using a proxy of the system(we refer to as the source, e.g., a simulator). We then usea regression model that automatically learns the relationshipbetween source and target to learn an accurate and reliableperformance model using only a few samples taken from thereal system, leading to much lower cost and faster learning

period. We define a cost model that turns model learning into amulti-objective problem taking into account model accuracy aswell as measurement cost. We demonstrate that our cost-awaretransfer learning will enable accurate performance predictionsby using only a few measurements of the target system. Weevaluate our approach on (i) a robotic system, (ii) 3 streamprocessing applications, and (iii) a NoSQL database system.

In summary, our contributions are the following:• A cost-aware transfer learning method that learns accu-

rate performance models for configurable software.• An implementation, analysis and experimental evaluation

of our cost-aware transfer learning compared to a notransfer learning method.

II. MODEL LEARNING FOR PERFORMANCE REASONING INHIGHLY CONFIGURABLE SOFTWARE

We motivate the model learning problem by reviewingspecific challenges in the robotics domain. However, ourmethod can be applied in other configurable systems as well.

A. A motivating exampleThe software embedded in a robot implements algorithms

that enable the robot to accomplish the missions assigned to it,such as path planning [24]. Often, robotics software exposesmany different configuration parameters that can be tuned andtypically affect the robot’s performance in accomplishing itsmissions. For instance, we would like to tune the localiza-tion parameters of an autonomous robot while performing anavigation task as fast and safe as possible while consumingminimal energy (corresponding to longer battery life).

Navigation is the most common activity that a robot doesin order to get from point A to point B in an environment asefficient as possible without hitting obstacles, walls, or people.Let us concentrate on the localization algorithm that enablesnavigation tasks and previous studies shown that configurationsetting is influential to its performance [24]. Localization is thetechnique that enables the robot to find (i) its current positionand (ii) its current orientation at any point in its navigationtask [37]. Configurations that work well for one robot inone environment may not work well or at all in anotherenvironment for multiple reasons: (i) The environment inwhich robots operate may change at runtime. For instance,the robot may start in a dark environment and then reaches amuch brighter area. As a result, the quality of localization maybe affected as the range sensors for estimating the distanceto the wall provide measurements with a higher error rateand the false readings increase then. (ii) The robot itselfmay move to an environment where the motion of therobot will change. For example, when a wheeled robot movesto a slippery floor, then the motion dynamics of the robotwill change and the robot cannot accelerate the speed withthe same power level. (iii) The physics of the environmentmay change. For example, imagine the situation where someitems are put on a service robot while doing a mission. Asa result, the robot has to sustain more weight and consumesmore energy for doing the same mission. Therefore, it maybe desirable to sacrifice the accuracy by adjusting localizationparameters in order to perform the mission before running outof battery. (iv) A new mission may be assigned to the robot.

While a service robot is doing a mission, a new mission maybe assigned, for example, a new delivery spot may be defined.In cases where the battery level is too low to finish the newmission, we would change to a configuration that sacrificesthe localization accuracy in favor of energy usage.

In all of these situations, it is desirable to change the config-uration regarding the localization of the robot, in an automatedfashion. For doing so, we need a model to reason about thesystem performance especially when we face contradictingaspects that require a trade-off at runtime. We can learn aperformance model empirically using observations which aretaken from the robot performing missions in a real environ-ment under different configurations. But, the observations inreal-world are typically costly, error-prone, and sometimesinfeasible. There are, however, several simulation platformsthat provide a simulated environment, in which we can learnabout the performance of the robot in different conditionsgiven a specific configuration (e.g., Gazebo [34]). Although theobservations from a simulated environment are far less costlythan the measurements taken from the real robot and imposeno risks when a failure happens, they may not necessarilyreflect real behavior, since not every physical aspect can beaccurately modeled and simulated. Fortunately, simulations areoften highly correlated with the real behavior. The key insightbehind our approach is to learn the robot performance modelby using only a few expensive observations on a real system,but several cheap samples from all sources.

B. ChallengesAlthough we can take relatively cheap samples from simula-

tion, it is impractical and naive to exhaustively run simulationenvironments for all possible configurations:• Exponentially growing space. The configuration space

of just 20 parameters with binary options for each com-prises of 220 ≈ 1m possible configurations. Even for thissmall configuration space, if we spend just one minutefor collecting each sample, it will take about 2 yearsto perform an exhaustive sampling. Therefore, we cansample only a subset of this space, and we need to dothe sampling purposefully.

• Negative transfer. Not all samples from a simulatorreflect the real behavior and, as a result, they wouldnot be useful for learning a performance model. Morespecifically, the measurements from simulators typicallycontain noise and for some configurations, the data maynot have any relationship with the data we may observeon the real system. Therefore, if we learn a model basedon these data, it becomes inaccurate and far from realsystem behavior. Also, any reasoning based on the modelpredictions becomes misleading or ineffective.

• Limited budget. Often, different sources of data exist(e.g., different simulators, different versions of a system)that we can learn from and each may impose a differentcost. Often, we are given a limited budget that we canspend for either source and target sample measurements.

C. Problem formulation: Black-box model learningIn order to introduce the concepts in our approach concisely,

we define the model learning problem using mathematical

notations. Let Xi indicate the i-th configuration parameter,which ranges in a finite domain Dom(Xi). In general, Xi

may either indicate (i) an integer variable (e.g., the numberof iterative refinements in a localization algorithm) or (ii) acategorical variable (e.g., sensor names or binary options (e.g.,local vs global localization method). The configuration spaceis mathematically a Cartesian product of all of the domains ofthe parameters of interest X = Dom(X1)× · · · ×Dom(Xd).

A black-box response function f : X → R is usedto build a performance model given some observations ofthe system performance under different settings. In practice,though, the observation data may contain noise, i.e., yi =f(xi) + εi,xi ∈ X where εi ∼ N (0, σi) and we onlypartially know the response function through the observationsD = {(xi, yi)}di=1, |D| � |X|. In other words, a responsefunction is simply a mapping from the configuration space toa measurable performance metric that produces interval-scaleddata (here we assume it produces real numbers). Note thatwe can learn a model for all measurable attributes (includingaccuracy, safety if suitably operationalized), but here wemostly train predictive models on performance attributes (e.g.,response time, throughput, CPU utilization).

Our main goal is to learn a reliable regression model, f(·),that can predict the performance of the system, f(·), given alimited number of observations D. More specifically, we aimto minimize the prediction error over the configuration space:

argminx∈X

pe = |f(x)− f(x)| (1)

In order to solve the problem above, we assume f(·) isreasonably smooth, but otherwise little is known about theresponse function. Intuitively, we expect that for “near-by”input points x and x′ their corresponding output points y andy′ to be “near-by” as well.

D. State-of-the-art

In literature, model learning has been approached from twostandpoints: (i) sampling strategies and (ii) learning methods.

1) Sampling: Random sampling has been used to collectunbiased observations in computer-based experiments. How-ever, random sampling may require a large number of samplesto build an accurate model [36]. More intelligent samplingstrategies (such as Box-Behnken and Plackett-Burman) havebeen developed in the statistics and machine learning commu-nities, under the umbrella of experimental design, to ensurecertain statistical properties [28]. The aim of these differentexperimental designs is to ensure that we gain a high levelof information from sparse sampling (partial design) in highdimensional spaces. Relevant to the software engineering com-munity, several approaches tried different designs for highlyconfigurable software [18] and some even consider cost as anexplicit factor to determine optimal sampling [35].

For finding optimal configurations, researchers have triednovel ways of sampling with a feedback embedded inside theprocess where new samples are derived based on informationgained from the previous set of samples. A recent solution [8]uses active learning based on a random forest to find a gooddesign. Recursive Random Sampling (RRS) [42] integrates arestarting mechanism into the random sampling to achieve

high search efficiency. Smart Hill Climbing (SHC) [41] in-tegrates importance sampling with Latin Hypercube Design(LHD). An approach based on direct search [45] forms asimplex in the configuration space, and iteratively updates thesimplex through a number of operations to guide the samplegeneration. Quick Optimization via Guessing (QOG) [31]speeds up the optimization process exploiting some heuristicsto filter out sub-optimal configurations. Recently, transferlearning has been explored for configuration optimizations byexploiting the dependencies between configurations parame-ters [9] and measurements for previous versions of big datasystems using an approach called TL4CO in DevOps [3].

2) Learning: Also, standard machine-learning techniques,such as support-vector machines, decision trees, and evolution-ary algorithms have been tried [21], [43]. These approachestrade simplicity and understandability of the learned modelsfor predictive power. For example, some recent work [44]exploited a characteristic of the response surface of the con-figurable software to learn Fourier sparse functions by only asmall sample size. Another approach [36] also exploited thisfact, but iteratively construct a regression model representingperformance influences in an active learning process.

3) Positioning in the self-adaptive community: Performancereasoning is a key activity for decision making at runtime.Time series techniques [12] shown to be effective in predictingresponse time and uncovering performance anomalies aheadof time. FUSION [14], [13] exploited inter-feature relation-ships (e.g., feature dependencies) to reduce the dimensions ofconfiguration space, making runtime performance reasoningfeasible. Different classification models have been evaluatedfor the purpose of time series predictions in [2]. Performancepredictions also have been applied for resource allocationsat runtime [20], [22]. Note that the approaches above arereferred to as black-box models. However, another categoryof models known as white-box is built early in the lifecycle, by studying the underlying architecture of the system[17], [19] using Queuing networks, Petri Nets, and StochasticProcess Algebras [4]. Performance prediction and reasoninghave also been used extensively in other communities such ascomponent-based [5] and control theory [15].

4) Novelty: Previous work attempted to improve the pre-diction power of the model by exploiting the information thathas been gained from the target system either by improvingthe sampling process or by adopting a learning method.Our approach is orthogonal to both sampling and learning,proposing a new way to enhance model accuracy by exploitingthe knowledge we can gain from other relevant and possiblycheaper sources to accelerate the learning of a performancemodel through transfer learning.

Transfer learning has been applied previously for regres-sion and classification problems in machine learning [32], insoftware engineering for defect predictions [27], [29], [30]and effort estimation [26] and in systems for configurationoptimization [9], [3]. However, in this paper, we enable ageneric form of transfer learning for the purpose of sensitivityanalysis over the entire configuration space. Our approach canenable (i) performance debugging, (ii) performance tuning,(iii) design-time evolution, or (iv) runtime adaptation in thetarget environment by exploiting any source of knowledge

1. Model Learning

2. Predictive Model

4. Transfer Learning

3. Cost Model

ImprovedPredictive

ModelConfigurationParameters

Fig. 2: Overview of cost-aware transfer learning methodology.

from the source environment including (i) different workloads,(ii) different deployments, (iii) different versions of the system,or (iv) different environmental conditions. The performance ofthe system varies when one or more of these changes happen,however, as we will show later in our experimental results,the performance responses are correlated. This correlation isan indicator that the source and target are related and there isa potential to learn across the environments. To the best of ourknowledge, our approach is the first attempt towards learningperformance models using cost-aware transfer learning forconfigurable software.

III. COST-AWARE TRANSFER LEARNING

In this section, we explain our cost-aware transfer learningsolution to improve learning an accurate performance model.

A. Solution overviewAn overview of our model learning methodology is de-

picted in Figure 2. We use the observations that we canobtain cheaply, with the cost of cs for each sample, froman alternative measurable response function, g(·), that yieldsdifferent but related response to the target response, Ds ={(xs, ys)|ys = g(xs) + εs,xs ∈ X}, and transfer these obser-vations to learn a performance model for the real system usingonly few observations from that system, Dt = {(xt, yt)|yt =f(xt)+εt,xt ∈ X}. The cost of obtaining each observation onthe target system is ct, and we assume that cs � ct and thatthe source response function, g(·), is related (correlated) to thetarget function, and this relatedness contributes to the learningof the target function, using only sparse samples from thereal system, i.e., |Dt| � |Ds|. Intuitively, rather than startingfrom scratch, we transfer the learned knowledge from the datawe have collected using cheap methods such as simulatorsor previous system versions, to learn a more accurate modelabout the target configurable system and use this model toreason about its performance at runtime.

In Figure 3, we demonstrate the idea of transfer learningwith a synthetic example. Only three observations are takenfrom a synthetic target function, f(·), and we try to learna model, f(x), which provides predictions for unobservedresponse values at some input locations. Figure 3(b) depictsthe predictions that are provided by the model trained usingonly the 3 samples without considering the transfer learningfrom any source. The predictions are accurate around thethree observations, while highly inaccurate elsewhere. Figure3(c) depicts the model trained over the 3 observations and

1 3 5 9 10 11 12 13 140

50

100

150

200source samplestarget samples

(a) (b)

(c) (d)

Fig. 3: (a) 9 samples have been taken from a source (re-spectively 3 from a target); (b) [without transfer learning] aregression model is constructed using only the target sam-ples. (c) [with transfer learning] another regression model isconstructed using the target and the source samples. The newmodel is based on only 3 target measurements while providingaccurate predictions. (d) [negative transfer] a regression modelis constructed with a transfer from a misleading responsefunction and therefore the model prediction is inaccurate.

9 other observations from a different but related source.The predictions, thanks to transfer learning, are here closelyfollowing the true function with a narrow confidence interval(i.e., high confidence in predictions provided by the model).Interestingly, the confidence interval around points x = 2, 4in the model with transfer learning have disappeared, demon-strating more confident predictions with transfer learning.

Given a target environment, the effectiveness of any transferlearning depends on the source environment and how it isrelated to the target [38]. If the relationship is strong andthe transfer learning method can take advantage of it, theperformance in the target predictions can significantly improvevia transfer. However, if the source response is not sufficientlyrelated or if the transfer method does not exploit the relation-ship, the performance may not improve or even may decrease.We show this case via the synthetic example in Figure 3(d)where the prediction model uses 9 samples from a misleadingfunction that is unrelated to the target function (the responseremains constant as we increase x); as a consequence, thepredictions are very inaccurate and do not follow the growingtrend of the true function.

For the choice of the number of samples from the sourceand target, we use a simple cost model (cf. Figure 2) that isbased on the number of source and target samples as follows:

C(Ds,Dt) = cs · |Ds|+ ct · |Dt|+ CTr(|Ds|, |Dt|), (2)C(Ds,Dt) ≤ Cmax, (3)

where Cmax is the experimental budget and CTr(|Ds|, |Dt|)is the cost of model training, which is a function of source

and target sample sizes. Note that this cost model makes theproblem we have formulated in Eq. (1) into a multi-objectiveproblem, where not only accuracy is considered, but also costcan be considered in the transfer learning process.

B. Model learning: Technical details

We use Gaussian Process (GP) models to learn a reliableperformance model f(·) that can predict unobserved responsevalues. The main motivation to choose GP here is that itoffers a framework in which performance reasoning can bedone using mean estimates as well as a confidence intervalfor each estimation. In other words, where the model isconfident with its estimation, it provides a small confidenceinterval; on the other hand, where it is uncertain about itsestimations it gives a large confidence interval meaning thatthe estimations should be used with precautions. The otherreason is that all the computations are based on linear algebrawhich is cheap to compute. This is especially useful in thedomain of self-adaptive systems where automated performancereasoning in the feedback loop is typically time constrainedand should be robust against any sort of uncertainty includingprediction errors. Also, for building GP models, we do notneed to know any internal details about the system; thelearning process can be applied in a black-box fashion usingthe sampled performance measurements. In the GP framework,it is also possible to incorporate domain knowledge as prior,if available, which can enhance the model accuracy [21].

In order to describe the technical details of our transferlearning methodology, let us briefly describe an overview ofGP model regression; a more detailed description can be foundelsewhere [40]. GP models assume that the function f(x) canbe interpreted as a probability distribution over functions:

y = f(x) ∼ GP(µ(x), k(x,x′)), (4)

where µ : X → R is the mean function and k : X × X → Ris the covariance function (kernel function) which describesthe relationship between response values, y, according to thedistance of the input values x,x′. The mean and variance ofthe GP model predictions can be derived analytically [40]:

µt(x) = µ(x) + k(x)ᵀ(K + σ2I)−1(y − µ), (5)σ2t (x) = k(x,x) + σ2I − k(x)ᵀ(K + σ2I)−1k(x), (6)

where k(x)ᵀ = [k(x,x1) k(x,x2) . . . k(x,xt)], I isidentity matrix and

K :=

k(x1,x1) . . . k(x1,xt)...

. . ....

k(xt,x1) . . . k(xt,xt)

(7)

GP models have shown to be effective for performancepredictions in data scarce domains [21]. However, as wehave demonstrated in Figure 3, it may become inaccuratewhen the samples do not cover the space uniformly. Forhighly configurable systems, we require a large number ofobservations to cover the space uniformly, making GP modelsineffective in such situations.

C. Model prediction using transfer learning

In transfer learning, the key question is how to make accu-rate predictions for the target environment using observationsfrom other sources, Ds. We need a measure of relatedness notonly between input configurations but between the sourcesas well. The relationships between input configurations wascaptured in the GP models using the covariance matrix thatwas defined based on the kernel function in Eq. (7). Morespecifically, a kernel is a function that computes a dot product(a measure of “similarity”) between two input configurations.So, the kernel helps to get accurate predictions for similarconfigurations. We now need to exploit the relationship be-tween the source and target functions, g, f , using the currentobservations Ds,Dt to build the predictive model f . To capturethe relationship, we define the following kernel function:

k(f, g,x,x′) = kt(f, g)× kxx(x,x′), (8)

where the kernels kt represent the correlation between sourceand target function, while kxx is the covariance function forinputs. Typically, kxx is parameterized and its parameters arelearned by maximizing the marginal likelihood of the modelgiven the observations from source and target D = Ds ∪ Dt.Note that the process of maximizing the marginal likelihoodis a standard method [40]. After learning the parameters ofkxx, we construct the covariance matrix exactly the same wayas in Eq. 7 and derive the mean and variance of predictionsusing Eq. (5), (6) with the new K. The main essence oftransfer learning is, therefore, the kernel that captures thesource and target relationship and provides more accuratepredictions using the additional knowledge we can gain viathe relationship between source and target.

D. Transfer learning in a self-adaptation loop

Now that we have described the idea of transfer learning forproviding more accurate predictions, the question is whethersuch an idea can be applied at runtime and how the self-adaptive systems can benefit from it. More specifically, wenow describe the idea of model learning and transfer learningin the context of self-optimization, where the system adaptsits configuration to meet performance requirements at runtime.The difference to traditional configurable systems is that welearn the performance model online in a feedback loop undertime and resource constraints. Such performance reasoning isdone more frequently for self-adaptation purposes.

An overview of a self-optimization solution is depicted inFigure 4 following the MAPE-K framework [11], [25]. Weconsider the regression model that we learn via transfer learn-ing in this work as the Knowledge component of the MAPE-K that acts as an interface to which other components canquery the performance. For deciding how many observationsand from what source to transfer, we use the cost model thatwe have introduced earlier. At runtime, the managed systemis Monitored by pulling the end-to-end performance metrics(e.g., latency, throughput) from the corresponding sensors.Then, the retrieved performance data needs to be Analysed.Next, the model needs to be updated taking into account thenew performance observations. Having updated the model,a new configuration may be Planned to replace the current

Environment

R.1 Performance Monitoring performance

repository

GP modelM

A P

E

Highly Configurable System

R.2 Data Analysis

R.4 Select the most promising

configuration to test

R.5 Update System Configuration

R.3 Model Update

K

AS

Autonomic Manager

simulation data

repository

D.1 Change config.

D.2 Perform sim.

D.3 Extract data

R.3.1 Transfer Learning

Simulator

RuntimeDesign-time

Real System

Fig. 4: Integrating transfer learning with MAPE-K loop, whereknowledge update is realized with transfer learning.

configuration. Finally, the new configuration will be enactedby Executing platform specific operations. This enables model-based knowledge evolution using machine learning [22].

The underlying model can be updated not only when a newobservation is available but also by transferring the learningfrom other related sources. So at each adaptation cycle, wecan update our belief about the correct response given datafrom the managed system and other related sources. Thiscan be particularly useful when the adaptive systems need tomake a decision when the internal knowledge utilized in theMAPE-K loop is not accurate enough, e.g., in early stagesof learning when the system needs to react to environmentalchanges the internal knowledge is not accurate and as aresult the adaptation decisions are sub-optimal [23]. In thissituation, even the measurements from a noisy source (e.g.,simulator) could be beneficial in order to boost the initialperformance achievable using only the transferred knowledge,before any further learning is done. Also, another challengethat has been reported in the past is the slow convergent ofthe learning process [23], in this situation, transfer learningcan accelerate the learning process in the target environmentgiven the transferred knowledge compared to the amount oftime to learn it from scratch.

IV. EXPERIMENTAL RESULTS

We evaluate the effectiveness and applicability of ourtransfer learning approach for learning models for highly-configurable systems, in particular, compared to conventionalnon-transfer learning. Specifically, we aim to answer thefollowing three research questions:RQ1: How much does transfer learning improve the predictionaccuracy?RQ2: What are the trade-offs between learning with differentnumbers of samples from source and target in terms ofprediction accuracy?RQ3: Is our transfer learning approach applicable in thecontext of self-adaptive systems in terms of model trainingand evaluation time?

We will proceed as follows. First, we will return to our mo-tivating example of an autonomous service robot to explore thebenefits and limitations of transfer learning for this single case.

Subsequently, we explore the research questions quantitatively,performing experiments on 5 different configurable systems.We have implemented our cost-aware transfer learning ap-proach in Matlab 2016b and, depending on each experiment,we have developed different scripts for data collections andautomated configuration changes. The source code and dataare available in an online appendix [1].

A. Case study: Robotics software

We start by demonstrating our approach with a smallcase study of service robots, which we already motivated inSection II-A. Our long-term goal is to allow the service robotsto adapt effectively to (possibly unanticipated) changes in theenvironment, in its goals, or in other parts of the system,among others, by reconfiguring parameters. Specifically, wework with the CoBot robotic platform [39].

To enable a more focused analysis and presentation, wefocus on a single but important subsystem and two parameters:The particle-filter-based autonomous localization componentof the CoBot software [7] determines the current locationof the robot, which is essential for navigation. Among themany parameters of the component, we analyze two thatstrongly influence the accuracy of the localization and requiredcomputation effort, (1) the number of particle estimates and(2) the number of gradient descent refinement steps appliedto each of these particles when we receive a new update fromsensors. At runtime, the robot could decide to reconfigureits localization component to trade off location accuracy withenergy consumption, based on the model we learn.

We have extensively measured the performance of thelocalization component with different parameters and with dif-ferent simulated environmental conditions in prior work [24].For the two parameters of interest (number of particles andrefinements), we used 25 and 27 different values each, evenlydistributed on the log scale within their ranges ([5− 10, 000]and [1 − 10, 000]), i.e., |X| = 25 × 27 = 675. Specifically,we have repeatedly executed a specific mission to navigatealong a corridor in the simulator and measured performancein terms of CPU usage (a proxy for energy consumption),time to completion, and location accuracy. Each measurementtakes about 30 seconds. In Figure 5a, we illustrate how CPUusage depends on those parameters (the white area representsconfigurations in which the mission fails to complete).

As a transfer scenario, we consider simulation results givendifferent environment conditions. As a target, we considermeasurements that we have performed in the default configura-tion. As a source, we consider more challenging environmentconditions, in which the odometry sensor of the robot isless reliable (e.g., due to an unfamiliar or slippery surface);we simulate that the odometry sensor is both miscalibrated(30%) and noisy (±45%). We assume that we have collectedthese measurements in the past or offline in a simulator andthat those measurements were therefore relatively cheap. Asshown in Figure 5b, localization becomes more computa-tionally expensive in the target environment. Our goal is touse (already available) data from the noisy environment forlearning predictions in the non-noisy target environment withonly a few measurements in that environment.

Observations: For both source and target environments,we can learn fairly accurate models if we take large numbersof samples (say |D| > 80% × |X|) in that environment,but predictions are poor when we can sample only a fewconfigurations (say |D| < 1% × |X|). That is, we need atleast a moderate number of measurements to produce usefulmodels for making self-adaptation decisions at runtime. Also,using the model learned exclusively from measurements in thesource environment to predict the performance of the robot inthe more noisy environment leads to a significant predictionerror throughout the entire configuration space (cf. Figure 5c).However, transfer learning can effectively calibrate a modellearned from cheap measurements in the source environmentto the target environment with only a few additional measure-ments from that environment. For example, Figure 5d showsthe predictions provided by a model that has been learnedby transfer learning using 18 samples from the source andonly 4 samples from the target environment. As a result, themodel learned the structure of the response properly in termsof changing of values over the configuration space.

In Figure 6a, we illustrate how well we can predict thetarget environment with a different number of measurementsin the source and the target environment. More specifically,we randomly sampled configurations from both source andtarget between [0 − 100%] and [0 − 10%] respectively andwe measured the accuracy as the relative error comparingpredicted to actual performance in all 675 configurations. Note|Ds| = 0 corresponds to model learning without transferlearning. We can see that, in this case, predictions can besimilarly accurate when we replace expensive measurementsfrom the target environment by more, but cheaper samples.Moreover, we have noticed that the models learned withouttransfer learning are highly sensitive to the sample selection(cf. Figure 6b). The variance becomes smaller when weincrease the number of source samples (along with the y-axis)leading to more reliable models. In this case study, all modellearning and evaluation times (both for normal learning andtransfer learning) are negligible (< 5s), cf. Figure 6c,d.

Relatedness: In this case study, we can also observethat the relatedness between the source and target mattersby changing the environments more or less aggressively. Forthat purpose, we simulated six alternative source environmentsby adding noise with different power levels to the source weconsidered previously (i.e., odometry miscalibration of 30%and noise of ±45%, cf., Figure 7). Typically, the stronger thetarget environment is changed from the source (default) envi-ronment, the larger the difference in performance is observedfor the same configuration between the two environments. Thisrelationship is visible in correlation measures in Figure 7).The prediction error reported in Figure 7 indicates that trans-fer learning becomes more effective (i.e., produces accuratemodels with a few samples from the target environment) ifmeasurements in the source and target are strongly correlated.This also demonstrates that even transfer from a response witha small correlation helps to learn a better model comparingwith no transfer learning.

Based on this observation, we need to decide (i) from whichalternative source and (ii) how many samples we transfer toderive an accurate model. In general, we can: either (a) select

5 10 15 20 25number of particles

5

10

15

20

25

nu

mb

er

of

refin

em

en

ts

0

5

10

15

20

25

5 10 15 20 25number of particles

5

10

15

20

25

nu

mb

er

of

refin

em

en

ts

0

5

10

15

20

25

5 10 15 20 25number of particles

5

10

15

20

25

nu

mb

er

of

refin

em

en

ts

0

5

10

15

20

25CPU usage [%] CPU usage [%]

(a) (b)

(c) (d)

Prediction without transfer learning

5 10 15 20 25

5

10

15

20

25

10

15

20

25

Prediction with transfer learning

Fig. 5: Illustration of transfer learning: (a) a source responsefunction from which we can take samples; (b) a target responsefunction which we are interested to learn; (c) a prediction with-out transfer learning; (d) a prediction with transfer learning.

2

2

2

2

2

2

2

2

2

44

4

4

6

6

68

8 1012

14 16

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

5

5

5

55

5

5

55

1010

10

10

1515

15

20

20

25303540

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

(a) (b)

0.5 0.5 0.51 1 1

1.5 1.5 1.5

2

2 2

2.5 2.52.5

33

3

3

3.5

3.5

3.5

3.5

4

4 4

4

4

4.5 4.5 4.5

5

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

0.005 0.005 0.0050.01 0.01 0.01

0.015 0.015 0.0150.02 0.02 0.02 0

.02

0.0

25

0.025

0.025 0.025

0.025

0.02

50.025

0.0

30.03

0.03

0.0

3

0.03

0.0

30.03

0.035

0.0

35 0.0

35

0.04

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

(c) (d)

Fig. 6: (a) Prediction error of the trained model, (b) varianceof predictions (transfer learning contributes to lower the un-certainty regarding model predictions, therefore, more reliablelearning), (c) training and (d) evaluation overhead.

the most related source and sample only from that source,or (b) select fewer samples from unrelated sources and morefrom the more related ones to transfer. Note that our transferlearning method supports both strategies. For this purpose, weuse the concept of relatedness (correlation) between sourceand target. The former is simpler, but we need to know, apriori, the level of relatedness of different sources. The latteris more sophisticated using a probabilistic interpretation ofrelatedness to learn from different sources using a concept ofdistance from the target. Our method supports this through the

10

20

30

40

50

60

Ab

so

lute

Pe

rce

nta

ge

Err

or

[%]

Sources s s1 s2 s3 s4 s5 s6

noise-level 0 5 10 15 20 25 30corr. coeff. 0.98 0.95 0.89 0.75 0.54 0.34 0.19µ(pe) 15.34 14.14 17.09 18.71 33.06 40.93 46.75

Fig. 7: Prediction accuracy of the model learned with samplesfrom different sources of different relatedness to the target.GP is the model without transfer learning.

kernel function in Eq. 8 that embeds the knowledge of sourcecorrelations and exploits that knowledge in the model learningprocess. The selection of a specific strategy depends on severalfactors including: (i) the cost of samples from each specificsource (samples from some sources may come for free, i.e.cs = 0 and we may want to use more samples from this sourcedespite the fact that it may be less relevant), (ii) the boundon runtime overhead (more samples leads to a higher learningoverhead), and (iii) domain for which the model is used (somedomain are critical and some adversaries may manipulate theinput data exploiting specific vulnerabilities of model learningto compromise the whole system security [33]). We leave theinvestigation of strategy selection as a future work.

B. Experiment: Prediction accuracy and model reliabilityWith our case study, we demonstrated our approach on a

specific small example. To increase both internal and externalvalidity, we perform a more systematic exploration on multipledifferent configurable systems.

Subject systems: First, we again use the localizationcomponent of the CoBot system, but (since we are no longerconstrained by plotting results in two dimensions) explorea larger space with 4 parameters. In addition, we selectedhighly-configurable systems as subjects that have been ex-plored for configuration optimization in prior work [21]: threestream processing applications on Apache Storm (WordCount,RollingSort, SOL) and a NoSQL database benchmark systemon Apache Cassandra. WordCount is a popular benchmark [16]featuring a three-layer architecture that counts the numberof words in the incoming stream and it is essentially aCPU intensive application. RollingSort is a memory intensivesystem that performs rolling counts of incoming messagesfor identifying trending topics. SOL is a network intensivesystem, where the incoming messages will be routed througha multi-layer network. From the practical perspective, thesebenchmark applications are based on popular big data engines(e.g., Storm) and understanding the performance influence

of configuration parameters can have a large impact on themaintenance of modern systems that are based on these highly-configurable engines. The Cassandra measurements was doneusing scripts that runs YCSB [10] and was originally devel-oped for a prior work [3].

The notion of source and target set depends on the subjectsystem. In CoBot, we again simulate the same navigation mis-sion in the default environment (source) and in a more difficultnoisy environment (target). For the three stream processingapplications, source and target represent different workloads,such that we transfer measurements from one workload forlearning a model for another workload. More specifically, wecontrol the workload using the maximum number of messageswhich we allow to enter the stream processing architecture.For the NoSQL application, we analyze two different transfers:First, we use as a source a query on a database with 10 millionrecords and as target the same query on a database with 20million records, representing a more expensive environment tosample from. Second, we use as a source a query on 20 millionrecords on one cluster and as target a query on the same datasetrun on a different cluster, representing hardware changes.Overall, our subjects cover different kinds of applicationsand different kinds of transfer scenarios (changes in theenvironment, changes in the workload, changes in the dataset,and changes in the hardware).

Experimental setup: As independent variables, we sys-tematically vary the size of the learning sets from both sourceand target environment in each subject system. We samplebetween 0 and 100 % of all configurations in the sourceenvironment and between 1 and 10 % in the target.

As a dependent variable, we measure learning time andprediction accuracy of the learned model. For each subjectsystem, we measure a large number of random configurationsas the evaluation set, independently from configurations sam-pled for learning, and compare the predictions of the learnedmodel f to the actual measurements of the configurations inthe evaluation set Do. We compute the absolute percentageerror (APE) for each configuration x in the evaluation set|f(x)−f(x)|

f(x) × 100 and report the average to characterize theaccuracy of the prediction model. Ideally, we would use thewhole configuration space as evaluation set (Do = X), but themeasurement effort would be prohibitively high for most real-world systems [21], [36]; hence we use large random samples(cf. size column in Table I).

The measured and predicted metric depends on the subjectsystem: For the CoBot system, we measure average CPU usageduring the same mission of navigating along a corridor as inour case study; we use the average of three simulation runs foreach configuration. For the Storm and NoSQL experiments, wemeasure average latency over a window of 8 and 10 minutesrespectively. Also, after each sample collection, the experi-mental testbed was cleaned which required several minutesfor the Storm measurements and around 1 hour (for offloadingand cleaning the database) for the Cassandra measurements.We sample a given number of configurations in the sourceand target randomly and report average results and standarddeviations of accuracy and learning time over 3 repetitions.

TABLE I: Overview of our experimental datasets. “Size” col-umn indicates the number of measurements in the datasets and“Testbed” refer to the infrastructure where the measurementsare taken and their details are in the appendix.

Dataset Parameters Size Testbed

1 CoBot(4D)

1-odom miscalibration,2-odom noise,3-num particles,4-num refinement

56585 C9

2 wc(6D)1-spouts, 2-max spout,3-spout wait, 4-splitters,5-counters, 6-netty min wait

2880 C1

3 sol(6D)1-spouts, 2-max spout,3-top level, 4-netty min wait,5-message size, 6-bolts

2866 C2

4 rs(6D)1-spouts, 2-max spout,3-sorters, 4-emit freq,5-chunk size, 6-message size

3840 C3

5

6

cass-10

cass-20

1-trickle fsync, 2-auto snapshot,3-con. reads, 4-con. writes5-file cache size in MB6-con. compactors

1024 C6x,C6y

Results: We show results of our experiments in Fig-ure 8. The 2D plot shows average errors across all subjectsystems. The results in which the set of source samples Ds isempty represents the baseline case without transfer learning.Although the results differ significantly among subject systems(not surprising, given different relatedness of the source andtarget) the overall trends are consistent.

First, our results show that transfer learning can achieve highprediction accuracy with only a few samples from the targetenvironment; that is, transfer learning is clearly beneficial ifsamples from the source environment are much cheaper thansamples from the target environment. Given the same numberof target samples, the prediction accuracy becomes better upto even 3 levels of magnitudes as we include more samplesfrom the source.

Second, using transfer learning with larger samples fromthe source environment reduces the variance of the predictionerror. That is, models learned with only a few samples fromthe target environment are more affected by the randomnessin sampling from the configuration space. Conversely, transferlearning improves the quality and reliability of the learnedmodels, by reducing sensitivity to the sample selection.

Third, the model training time increases monotonicallywhen we increase the sample size for target or source (cf.Figure 6c for the CoBot case study and for the other subjectsystems in the appendix). At the same time, even for largesamples, it rarely exceeds 100 seconds for our subject systems,which is reasonable for online use in a self-adaptive system.Learning time is negligible compared to measurement timesin all our subject systems.

Finally, the time for evaluating the models (i.e., using themodel to make a prediction for a given configuration), doesnot change much with models learned from different samplesizes (cf. Figure 6d) and is negligible overall (< 300ms).

Summary: we see improved accuracy and reliability whenusing transfer learning with additional source samples (RQ1);we see clear trade-offs among using more source or targetsamples (RQ2); and we see that training and evaluation timesare acceptable to be applied for self-adaptive systems (RQ3).

C. Discussion: Trade-offs and cost modelsOur experimental results clearly show that we can achieve

accurate models both with a larger number of samples fromsource or target environment or both: In Figure 8, we can seehow many models for different numbers of source and targetsamples yield models with similar accuracy. Those differentpoints with the same accuracy can be interpreted as indiffer-ence curves (a common tool in economics [6]). We can investdifferent measurement costs to achieve models with equivalentutility. In Figure 8a, we show those indifference curves moreexplicitly for the CoBot system. Using the indifference curvesenables us to decide how many samples from either a sourceor a target we should use for the model learning process that ismost beneficial in increasing predictive accuracy over unseenregions of the configuration space while satisfying the budgetconstraint. Given concrete costs and budgets, say, samplesfrom the target environment are 3 times more expensiveto gather than samples from the source environment, weplotted an additional line representing our budget and find thecombination of source and target samples that produces thebest prediction model for our budget (see intersection of thebudget line with the highest indifference curve in Figure 8).Although we will not know the indifference lines until we runextensive experiments, we expect that the general trade-offs aresimilar across many subject systems and transfer scenarios andthat such a cost model (cf. Eq. 2) can help to justify decisionson how many samples to take from each environment. Weleave a more detailed treatment to future work.

In this context, we can fix a cost model (i.e., fixing specificvalues for cs, ct) and transform the indifference diagrams intoa multi-objective goal and derive the Pareto front solutions asshown in Figure 9. This helps us to locate the feasible Paretofront solutions within the sweet spot.

D. Threats to validity and limitations1) Internal validity: In order to ensure internal validity, we

repeated the execution of the benchmark systems and measureperformance for a large number of configurations. For doingso, we have invested several months for gathering the mea-surements, which resulted in a substantial dataset. Moreover,we used standard benchmarks so that we are confident in thatwe have measured a realistic scenario.

2) External validity: In order to ensure external validity, weuse three classes of systems in our experiments including: (i) arobotic system, (ii) 3 different stream processing applications,and (iii) a NoSQL database system. These systems havedifferent numbers of configuration parameters and are fromdifferent application domains.

3) Limitations: Our learning approach relies on severalassumptions. First, we assume that the target response functionis smooth. If a configuration parameter has an unsteadyperformance behavior, we cannot learn a reliable model, butonly approximate its performance close to the observations.Furthermore, we assume that the source and target responsesare related. That is they are correlated to a certain extent. Themore related, the faster and better we can learn. Also, we needthe configurable system to have a deterministic performancebehavior. If we replicate the performance measurements forthe same system, the observed performance should be similar.

55

5

5

5

5 5

5

5

10

10

10

15

15

20253035

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

10

10

10

10

20

20

20

20

20

30

30

30

4040

40

5050

50

6060

60

70

708090

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

20

20

20

20

20

20

4040

40

606060

808080

100100

120

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

20

20

20

20

4040

40

6060

60

8080

100120

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

20

20

20

4040

40

606060

808080100

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

20

2020

20

20

4040

40

606060

808080100

1 2 3 4 5 6 7 8 9 100

10

20

30

40

50

60

70

80

90

100

(a) (b) (c)

(d) (e) (f)

Fig. 8: Prediction accuracy for (a) CoBot, (b) WC, (c) SOL, (d) RS, (e) cass (hardware change), (f) cass (DB size change).

0 200 400 600 800 1000 1200Measurement cost ($)

0

10

20

30

40

50

60

70

80

accuracy threshold

budg

et

thre

shol

d

sweet spot

Fig. 9: A two-objective optimization goal. We are interestedin the solutions in the targeted sweet spot.

V. CONCLUSIONS

Today most software systems are configurable and perfor-mance reasoning is typically used to adapt the configuration inorder to respond to environmental changes. Machine learningand sampling techniques have been previously proposed tobuild models in order to predict the performance of unseenconfigurations. However, the models are either expensive tolearn, or they become extremely unreliable if trained on sparsesamples. Our cost-aware transfer learning method is orthog-onal to the both previously proposed directions promoting tolearn from other cheaper sources. Our approach requires only

very few samples from the target response function and canlearn an accurate and reliable model based on sampling fromother relevant sources. We have done extensive experimentswith 5 highly configurable systems demonstrating that our ap-proach (i) improves the model accuracy up to several orders ofmagnitude, (ii) is able to trade-off between different number ofsamples from source and target, and (iii) imposes an acceptablemodel building and evaluation cost making appropriate forapplication in the self-adaptive community.Future directions. Beside performance reasoning, our cost-aware transfer learning can contribute to reason about othernon-functional properties and quality attributes, e.g., energyconsumption. Also, our approach could be extended to supportconfiguration optimization. For example, in our previous work,we have used GP models with Bayesian optimization to focuson interesting zones of the response functions to find optimumconfiguration quickly for big data systems [21]. In general,our notion of cost-aware transfer learning is independent of aparticular black-box model and complementary to white-boxapproaches in performance modeling such as queuing theory.A more intelligent way (e.g., active learning) of sampling thesource and the target environment to gain more information isalso another fruitful future avenue.

ACKNOWLEDGMENT

This work has been supported by AFRL and DARPA(FA8750-16-2-0042). Kaestner’s work is also supported byNSF awards 1318808 and 1552944 and the Science of SecurityLablet (H9823014C0140). Siegmund’s work is supported bythe DFG under the contract SI 2171/2.

REFERENCES

[1] Online appendix: https://github.com/pooyanjamshidi/transferlearning.[2] I. D. P. Anaya, V. Simko, J. Bourcier, N. Plouzeau, and J.-M. Jezequel.

A prediction-driven adaptation approach for self-adaptive sensor net-works. In Proceedings of the 9th International Symposium on SoftwareEngineering for Adaptive and Self-Managing Systems (SEAMS), pages145–154. ACM, 2014.

[3] M. Artac, editor. Deliverable 5.2: DICE delivery tools-Intermediateversion. 2017. http://www.dice-h2020.eu/.

[4] S. Balsamo, A. Di Marco, P. Inverardi, and M. Simeoni. Model-based performance prediction in software development: A survey. IEEETransactions on Software Engineering (TSE), 30(5):295–310, 2004.

[5] S. Becker, L. Grunske, R. Mirandola, and S. Overhage. Performanceprediction of component-based systems. In Architecting Systems withTrustworthy Components, pages 169–192. Springer, 2006.

[6] B. R. Binger and E. Hoffman. Microeconomics with calculus. W. W.Norton and Company, 1988.

[7] J. Biswas and M. M. Veloso. Localization and navigation of the CoBotsover long-term deployments. The International Journal of RoboticsResearch (IJRR), 32(14):1679–1694, 2013.

[8] B. Bodin, L. Nardi, M. Z. Zia, H. Wagstaff, G. Sreekar Shenoy,M. Emani, J. Mawer, C. Kotselidis, A. Nisbet, M. Lujan, B. Franke,P. H. Kelly, and M. O’Boyle. Integrating algorithmic parameters intobenchmarking and design space exploration in 3D scene understanding.In Proceedings of the International Conference on Parallel Architecturesand Compilation (PACT), pages 57–69. ACM, 2016.

[9] H. Chen, W. Zhang, and G. Jiang. Experience transfer for the configura-tion tuning in large scale computing systems. SIGMETRICS, 37(2):51–52, 2009.

[10] B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears.Benchmarking cloud serving systems with YCSB. In Proc. of Sympo-sium on Cloud Computing (SOCC). ACM, 2010.

[11] R. de Lemos, H. Giese, H. A. Muller, M. Shaw, J. Andersson, M. Litoiu,B. Schmerl, G. Tamura, N. M. Villegas, T. Vogel, D. Weyns, L. Baresi,B. Becker, N. Bencomo, Y. Brun, B. Cukic, R. Desmarais, S. Dustdar,G. Engels, K. Geihs, K. M. Goschka, A. Gorla, V. Grassi, P. Inverardi,G. Karsai, J. Kramer, A. Lopes, J. Magee, S. Malek, S. Mankovskii,R. Mirandola, J. Mylopoulos, O. Nierstrasz, M. Pezze, C. Prehofer,W. Schafer, R. Schlichting, D. B. Smith, J. P. Sousa, L. Tahvildari,K. Wong, and J. Wuttke. Software engineering for self-adaptive systems:A second research roadmap. In Software Engineering for Self-AdaptiveSystems II, pages 1–32. Springer, 2013.

[12] J. Ehlers, A. van Hoorn, J. Waller, and W. Hasselbring. Self-adaptivesoftware system monitoring for performance anomaly localization. InProceedings of the 8th ACM International Conference on AutonomicComputing (ICAC), pages 197–200. ACM, 2011.

[13] A. Elkhodary, N. Esfahani, and S. Malek. Fusion: A framework for en-gineering self-tuning self-adaptive software systems. In Proceedings ofthe eighteenth ACM SIGSOFT International Symposium on Foundationsof Software Engineering (FSE), pages 7–16. ACM, 2010.

[14] N. Esfahani, A. Elkhodary, and S. Malek. A learning-based frameworkfor engineering feature-oriented self-adaptive software systems. IEEETransactions on Software Engineering (TSE), 39(11):1467–1493, 2013.

[15] A. Filieri, H. Hoffmann, and M. Maggio. Automated multi-objectivecontrol for self-adaptive software design. In 10th Joint Meeting onFoundations of Software Engineering (FSE), pages 13–24. ACM, 2015.

[16] A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, and H.-A.Jacobsen. BigBench: Towards an industry standard benchmark for bigdata analytics. In Proceedings of the 2013 ACM SIGMOD InternationalConference on Management of Data, pages 1197–1208. ACM, 2013.

[17] H. Gomaa and M. Hussein. Model-based software design and adapta-tion. In Proceedings of the 2007 International Workshop on SoftwareEngineering for Adaptive and Self-Managing Systems (SEAMS), page 7.IEEE Computer Society, 2007.

[18] J. Guo, K. Czarnecki, S. Apel, N. Siegmund, and A. Wasowski.Variability-aware performance prediction: A statistical learning ap-proach. In Proceedings of the IEEE/ACM 28th International Conferenceon Automated Software Engineering (ASE), pages 301–311, 2013.

[19] J. Happe, H. Koziolek, and R. Reussner. Facilitating performancepredictions using software components. IEEE Software, 28(3):27, 2011.

[20] N. Huber, F. Brosig, S. Spinner, S. Kounev, and M. Bahr. Model-basedself-aware performance and resource management using the Descartesmodeling language. IEEE Transactions on Software Engineering, 2017.

[21] P. Jamshidi and G. Casale. An uncertainty-aware approach to optimalconfiguration of stream processing systems. In IEEE 24th InternationalSymposium on Modeling, Analysis and Simulation of Computer andTelecommunication Systems (MASCOTS), pages 39–48, 2016.

[22] P. Jamshidi, C. Pahl, and N. C. Mendonca. Managing uncertainty inautonomic cloud elasticity controllers. IEEE Cloud Computing, 3(3):50–60, 2016.

[23] P. Jamshidi, A. Sharifloo, C. Pahl, H. Arabnejad, A. Metzger, andG. Estrada. Fuzzy self-learning controllers for elasticity managementin dynamic cloud architectures. In 12th International ACM SIGSOFTConference on Quality of Software Architectures (QoSA), pages 70–79.IEEE, 2016.

[24] P. Kawthekar and C. Kastner. Sensitivity analysis for building evolving& adaptive robotic software. In IJCAI Workshop on Autonomous MobileService Robots, 2016.

[25] J. O. Kephart and D. M. Chess. The vision of autonomic computing.Computer, 36(1):41–50, 2003.

[26] E. Kocaguneli, T. Menzies, and E. Mendes. Transfer learning in effortestimation. Empirical Software Engineering, 20(3):813–843, 2015.

[27] R. Krishna, T. Menzies, and W. Fu. Too much automation? The bell-wether effect and its implications for transfer learning. In Proceedingsof the 31st IEEE/ACM International Conference on Automated SoftwareEngineering (ASE), pages 122–131. ACM, 2016.

[28] D. C. Montgomery. Design and analysis of experiments. John Wiley &Sons, 2008.

[29] J. Nam and S. Kim. Heterogeneous defect prediction. In Proceedings ofthe Joint Meeting on Foundations of Software Engineering (FSE), pages508–519. ACM, 2015.

[30] J. Nam, S. J. Pan, and S. Kim. Transfer defect learning. In Proceedingsof the International Conference on Software Engineering (ICSE), pages382–391. IEEE Press, 2013.

[31] T. Osogami and S. Kato. Optimizing system configurations quicklyby guessing at the performance. In Proceedings of the 2007 ACMSIGMETRICS International Conference on Measurement and Modelingof Computer Systems, volume 35, 2007.

[32] S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactionson Knowledge and Data Engineering, 22(10):1345–1359, 2010.

[33] N. Papernot, P. McDaniel, A. Sinha, and M. Wellman. Towards thescience of security and privacy in machine learning. arXiv preprintarXiv:1611.03814, 2016.

[34] M. Reckhaus, N. Hochgeschwender, J. Paulus, A. Shakhimardanov, andG. K. Kraetzschmar. An overview about simulation and emulation inrobotics. IEEE International Conference on Simulation, Modeling, andProgramming for Autonomous Robots, pages 365–374, 2010.

[35] A. Sarkar, J. Guo, N. Siegmund, S. Apel, and K. Czarnecki. Cost-efficient sampling for performance prediction of configurable systems.In 30th IEEE/ACM International Conference on Automated SoftwareEngineering (ASE), pages 342–352. IEEE, 2015.

[36] N. Siegmund, A. Grebhahn, S. Apel, and C. Kastner. Performance-influence models for highly configurable systems. In Proceedings ofthe 2015 10th Joint Meeting on Foundations of Software Engineering(FSE), pages 284–294. ACM, 2015.

[37] S. Thrun, W. Burgard, and D. Fox. Probabilistic robotics. MIT press,2005.

[38] L. Torrey and J. Shavlik. Transfer learning. Handbook of Research onMachine Learning Applications and Trends: Algorithms, Methods, andTechniques, 1:242–264, 2009.

[39] M. Veloso, J. Biswas, B. Coltin, and S. Rosenthal. CoBots: Robustsymbiotic autonomous mobile service robots. In International JointConference on Artificial Intelligence (IJCAI), volume 4423, pages 4423–4429, 2015.

[40] C. K. Williams and C. E. Rasmussen. Gaussian processes for machinelearning. MIT Press, 2006.

[41] B. Xi, Z. Liu, M. Raghavachari, C. H. Xia, and L. Zhang. A smarthill-climbing algorithm for application server configuration. In 13thInternational Conference on World Wide Web (WWW), pages 287–296.ACM, 2004.

[42] T. Ye and S. Kalyanaraman. A recursive random search algorithmfor large-scale network parameter configuration. ACM SIGMETRICSInternational Conference on Measurement and Modeling of ComputerSystems, 31(1):196–205, 2003.

[43] N. Yigitbasi, T. L. Willke, G. Liao, and D. Epema. Towards machinelearning-based auto-tuning of MapReduce. In IEEE InternationalSymposium on Modelling, Analysis and Simulation of Computer andTelecommunication Systems (MASCOTS), 2013.

[44] Y. Zhang, J. Guo, E. Blais, and K. Czarnecki. Performance predictionof configurable software systems by Fourier learning. In InternationalConference on Automated Software Engineering, pages 365–373, 2015.

[45] W. Zheng, R. Bianchini, and T. D. Nguyen. Automatic configuration ofinternet services. ACM SIGOPS Operating Systems Review, 41(3):219,2007.