On the Value of User Preferences in Search-Based Software Engineering:

On#the#Value#of#User#Preferences#in#Search4Based#So7ware#Engineering:##

A#Case#Study#in#So7ware#Product#Lines###

Tim#Menzies,##Abdel#Salam#Sayyad,#

Hany#Ammar##

WVU,#USA#Nov’12##

#

Sound#bites#

•  The#new#age#of#the#app##

•  Stop#Nnkering#with#small#stuff##

•  Enough#with#the#usual#suspects:##–  NSGA4II,#SPEA2,#etc#

#

•  If#preferences#maSer#–  Then#the##best#opNmizer#understands#preferences#the#best#

2#

Roadmap#

①   Feature(based,SE,②   Algorithms,③   IBEA,④   Tree,muta<on,⑤   Conclusion,#

WELCOME,TO,,THE,NEW,WORLDE,,,

5#

The#Nmes,#they#are#a#changing#Olde,worlde:,,

product(based,SE,e.g.#Microso7#office#

New,worlde:,,app(based,SE,,

•  E.g.#Apple#app#store#

6#

The#Nmes,#they#are#a#changing#Olde,worlde:,,

product(based,SE,

•  Vendors#tried#to#retain#their#user#base#via#some#complete#ecologies##

•  One#so7ware#soluNon#for#all#user#needs###(e.g.#Microso7#Office).###

•  Large,#complex,#so7ware#pla\orms,##–  very#slow#to#change.##

New,worlde:,,app(based,SE,,

•  Smart#phones#and#tablet4based#so7ware##

•  Users#choosing#many#numbers#of#small#apps#from#different#vendors,##–  each#performing#a#specific#

small#task.##

•  Vendors#must#quickly#and#conNnually#reconfigure#apps##–  To#retain#and#extend#their#

customer#base.#7#

Feature–oriented#domain#analysis#•  Feature#maps#=#a#

lightweight#method#for#defining#a#space#of##opNons#

•  Product4line#configuraNons#•  Defacto#standard#for#

modeling#variability##

8#

Original#FODA#paper#:#2700+#citaNons#

9#

Half#since#2007#

hSp://www.splot4research.org/#

10#200+#models,##plus#an#instance#generator#

LINUX#kernel#=#6000+#features#

11#

86%#declare#constraints#of#some#sort,##Most#features#refer#to#244#other#features.###

Need#for#beSer#automaNon#•  Such#complexity#needs#automated#support#

–  especially##feature#models##combined#with#•  #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.##

12#

Bad##scalability#

Search#for#valid#products:##9#state#of#the#art#theorem#provers##[Pohl,#ASE’11]##

And#these#were#“simple”#models#

Diving##deeper#•  Much#prior#work#explored##

Nny#objecNve#spaces##–  Two#or#three#objecNves#–  Or,#higher#(but#only#for#small#models)#

##

#•  So7ware#engineering#=#navigaNng#compeNng#concerns#

1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)#2.  That#offers#most#features#3.  Build#“stuff”#In#least#Nme#4.  That#we#have#used#most#before#5.  Using#features#with#least#known#defects# 13#

The#usual#

suspects#

Roadmap#


MOEA=#MulN4objecNve###evoluNonary#algorithms##############################

•  Repeat#Nll#happy#or#exhausted#–  SelecNon#(cull#the#herd)#–  Cross4over#(the#rude#bit)#–  MutaNon#(stochasNc#jiggle)#

15#

Some#MOEA#ApplicaNon#Domain Application Types

Control gas pipeline, pole balancing, missile evasion, pursuit

Design semiconductor layout, aircraft design, keyboard configuration, communication networks

Scheduling manufacturing, facility scheduling, resource allocation

Robotics trajectory planning

Machine Learning designing neural networks, improving classification algorithms, classifier systems

Signal Processing filter design

Game Playing poker, checkers, prisoner’s dilemma

Combinatorial Optimization

set covering, travelling salesman, routing, bin packing, graph colouring and partitioning

MOEA#for#Search4based#SE#

TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams#Requirements## #Bagnall,#Mansouri,#Zhang#Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd##Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe#Heap#allocaNon #Cohen,#Kooi,#Srisa4an##Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer##SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani##Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS#Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,##

# # # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,## # # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins#

Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7#Model#checking #Alba,#Chicano,#Godefroid#Probing # # #Cohen,#Elbaum##UIOs# # # #Derderian,#Guo,#Hierons#Comprehension #Gold,#Li,#Mahdavi#Protocols# # #Alba,#Clark,#Jacob,#Troya#Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo#Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis#

17#

So#study#FODA4#to#learn#how#to#improve#these#

tasks.#

Much#increased#interest##in#Search4based#SE#

18#

The#Pareto#FronNer#•  Mutants#=#<D,O>#=#<decisions,#objecNves>#

–  E.g.#car#•  Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels#•  ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)#

–  E.g.#learning#formula#•  Decisions:#what#variables#and#constants#to#use#•  ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may#complete)#

•  Pareto#fronNer:#select#the#non4dominated#mutants#–  X#dominates#Y##

•  if#for#all#objecNves,##X#is#never#any#worse##than#Y#•  If#for#one#objecNve,#X#beSer#than#Y#

19#

20#

Issues:#1)  Spread#2)  Hypervolume#3)  ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)#

Once#you#know#fronNer#•#Select#from#here#•#Ignore#here#

The#usual#suspects:###=#NSGA4II####=#SPEA2#

21#

Combines#N#objecNves##to#one##with#some##weighNng#scheme#

In#this#case,#NSGA4II#gets#more#spread#

Some#details#on#the#usual#suspects#

NSGA(II,,•  Is#a#geneNc#algorithm#

#•  Non4dominated#sort#

–  HeurisNc#way#to#fast#group#mutants#into#bands##

•  Crowd#pruning##via#approximate#hypercube#around#each#mutant:#O(Onlogn)#

SPEA2,,•  Is#a#geneNc#algorithm#

#•  Changes#the#definiNon#of#

“dominaNon”#–  SPEA#(version#1)#scored#mutants#by#

how#many#others#they#dominated#–  Got#confused#by#overlaps#in#the#

dominaNon#sets##

•  SPEA#(version#2):#–  Adds#a#“local#density#factor”#to#the#

dominaNon#weight#–  Mutants#in#dense#areas#valued##

more###

•  SPEA2#beSer#than#SPEA1#

22#

1#2#

3#

2#

Any#number#of#opNmizaNons##to#tradiNonal#GAs#

•  The#history#of#MOEAs#in#the#last#15#years#is#–  OpNmize#via#hybrid#GA#+#other#

search#method##

•  Local#search:##–  before#select,#do#a#liSle#simulated#

annealing#on#X%#of#the#populaNon##

•  ScaSer#search#(Glover’s#next#generaNon#tabu#search)#–  Includes#a#liSle#local#search#

#•  ParNcle#swam#opNmizaNon#

–  May#do#as#well#as#scaSer#search##(Yin#and#Glover#2007)#

•  DifferenNal#evoluNon#(Storn#1996)#–  Mutate#by#interpolaNons#between#

exisNng#mutants#–  For#x#in#mutants##

####y#=#any1#+#extrapolate(any3#–#any2)#####if#y#dominates#x#then#x#=#y#

#

•  Cellular#automata,#•  #Ant#colony#opNmizaNon,####•  Bayesian#staNsNcs#to#bias#the#

mutaNon,##•  Etc##•  etc#

23#

Roadmap#


Three#groups#of#Algorithms#

25#

Spea2#

Nsga4II#

DE# ScaSer##search#

PSO#

SA#

mocell#Z3#

IBEA#

Indicator4based##evoluNonary#algorithms#

SMT#solvers#

DominaNon#Is#a#binary#concept#

DominaNon#Is#a#conNnuous#concept#

Aggressive#exploraNon#of#preference##space#

IBEA#

26#

•  Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences#•  I(x1,x2):#

–  Least#adjust#objecNve#scores#such#that#x1#dominates#x2#•  Repeat#Nll#just#a#few#le7#

–  Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else#######

–  Sort#all#instances#by#F#–  Delete#worst#

•  Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors###

K=#0.05#

Case#studies#

27#

Cross4tree#constraints#

Data#from#hSp://www.splot4research.org/#Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/##

4#studies:#Bi,#tri,#quad,#five4#objecNves#

28#

So7ware#engineering#=#navigaNng#compeNng#concerns#1.  That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#

100%)#2.  That#offers#most#features#3.  Build#“stuff”#In#least#Nme#4.  That#we#have#used#most#before#5.  Using#features#with#least##

known#defects###Binary#objecNves#=#1,2#Tri4objecNve#########=#1,2,3#Quad4objecNve####=#1,2,3,4#Five4objecNve######=#1,2,3,4,5#

#HV#############=#hypervolume#of#dominated#region#Spread######=#coverage#of#fronNer#%#correct#=#%constraints#saNsfied#

29#


30#Comment,1:,all#about#the#same#for#the#24objecNve#problem#


31#Comment,2:,E4shop#is#a#nasty#problem:#needs#50M#evals#


32#Comment,3:,IBEA#has#no#spread#operators,#but#gets#best#spread#


33#Comment,4:,IBEA#has#no#HV#operators,#but#usually#gets#best#HV#


34#Comment,5:,All#the#non4IBEA#algorithms#are#very#similar#


35#Comment,6:,IBEA#does#much,#much##beSer#on#constraints#

Why#is#this#interesNng?#

Other,MOEAs,•  The#usual#suspects#are#widely,#

uncriNcally#used#in#many#MOEA#applicaNons#–  E.g.#especially#NSGA4II#and#SPEA2#

•  Focused#on#internal#algorithmic#tricks#–  Techniques#for#

•  #improving#spread##•  Improving#HV#•  Avoid#overlaps#in#cross4over#of#

dominated#space#•  etc#

•  And#the#net#effect#of#all#those#differences?#–  Not#much#

IBEA,•  Rather#stupid#on#those#

internal#tricks#–  Just#does#a#ye#olde#crossover#

mutate#GA#–  Plus:#aggressive#exploraNon#

of#the#preference#space#•  And#the#net#effect#of#all#

those#differences#–  BeSer#spreads#–  BeSer#HV#–  Fewer#constraint#violaNons#

36#

Conclusion:#preference#is#power#

Roadmap#


What#about#non4MOEA#soluNons?#

38#

Spea2#

Nsga4II#

DE# ScaSer##search#

PSO#

SA#

mocell#Z3#

IBEA#

Indicator4based##evoluNonary#algorithms#

SMT#solvers#

DominaNon#Is#a#binary#concept#

DominaNon#Is#a#conNnuous#concept#

Aggressive#exploraNon#of#preference##space#

Ethan’s#complaint#

•  So7ware#engineer##designs#are#o7en##nested#hierarchical##constraints#

•  Ethan#Jackson,#Microso7,##advocate#for#the#Z3#SMT#solver:#– Why#mutate#at#random,#then#check#for#constraint?#

–  BeSer#to#drive#the#mutaNons#by#the#constraints?#

39#

Dump#MOEAs?##Move#to#more#logical#forms?#

Pro:,move,to,,say,,SMT,solvers,

•  Next#generaNon#of#algorithm#–  The#next#big#thing#

•  BeSer#to#drive#the#mutaNons#by#the#constraints?#

Con:,stay,with,MOEA,

•  ImplementaNon#complexity#less#

•  More#tools#•  Easier#modificaNon#and#

experimentaNon#•  Models#can#be#expressed#

anyway#you#like#

40#

Tree#mutaNon#•  Work#in#progress#•  Simple#adaptaNon#of#current#MOEAs#for#systems#

of#hierarchical#constraints##

if#rand(0,1)<#mutaNon_probability:############Don't#mutate#if#you're#violaNng#one#of#the#rules:#

# #1)#if#deselecNng#root#feature## #2)#if#selecNng#feature#whose##

##########################parent#is#not#selected## #3)#if#deselecNng#feature#that#

#########################another#selected#feature#requires## #4)#if#group#cardinality#violaNon##else:## #flip#this#bit# # # ###### # #if#selecNng#(turning#on)#a#feature#then## # # #turn#on#children#### # #else#if#deselecNng#(turning#off)#

#######################################feature#then:## # ######### #########turn#off#all#children# 41#

IBEA#stabilizes#70#Nmes#faster#

Tree#mutaNon#preserves##domain#constraints#

42#

So#what##case#for#SMT?#

Roadmap#


Sound#bites#•  The#new#age#of#the#app#

–  In,this,new,worlde:,,use,FODA,(feature(oriented,domain,analysis),

•  Stop#Nnkering#with#small#stuff##–  Many,MOEAs,have,strikingly,,

similar,performance,

•  Enough#with#the#usual#suspects:##–  NSGA4II,#SPEA2,#etc#–  Too,much,uncri<cal,applica<on,of,these,algorithms,

•  If#preferences#maSer#–  Then#the##best#opNmizer#understands##preferences#the#best#–  IBEA:,aggressive,preference,explora<on,–  Tree,muta<on:,respect,your,domain,

44#

On the Value of User Preferences in Search-Based Software Engineering:

Education

Transcript of On the Value of User Preferences in Search-Based Software Engineering: