Controller Parameter Tuning of Delta Robot Based on Servo ...
On Parameter Tuning in Search-Based Software Engineering: A Replicated Empirical Study
-
Upload
abdel-salam-sayyad -
Category
Technology
-
view
182 -
download
0
description
Transcript of On Parameter Tuning in Search-Based Software Engineering: A Replicated Empirical Study
On Parameter Tuning in Search-Based Software Engineering:
A Replicated Empirical Study
Abdel Salam Sayyad Katerina Goseva-Popstojanova
Tim MenziesHany Ammar
West Virginia University, USA
International Workshop on Replication in Software Engineering Research (RESER)
Oct 9, 2013
2
Sound bitesSearch-based Software Engineering
Is here… to stay.A helper… Not an alternative to human SE
Randomness… is an essential part of Search Algorithms
… hence the need for statistical examination (A lot to learn from Empirical SE)
Parameter TuningA real problem…Default values (rules of thumb) do exist… and (sadly?) they are being followed
Default parameter values fail to optimize performance… … As seen in the original study, and in this replication…
No Free Lunch Theorems for Optimization [Wolpert and Macready ‘97]the same parameter values don’t optimize all algorithms for all
problems.
Roadmap
① Randomness of Search② The original study③ The replication④ Conclusion
Roadmap
① Randomness of Search② The original study③ The replication④ Conclusion
5
Searching for what?
• Correct solutions…– Conform to system relationships and constraints.
• Optimal solutions…– Achieve user objectives/preferences…
• Complex problems have big Search spaces– Exhaustive search not a practical idea.
6
Genetic Algorithm
• Start with a large population of candidate solutions… (How large?)
• Evaluate the fitness of your solutions.• Let your candidate solutions crossover –
exchange genes… (How often?)• Mutate a small portion of your solutions.
(How small?)• How do those choices affect performance?
7
Multi-objective Optimization
Higher-level Decision Making
The Pareto Front
The Chosen Solution
8
Survival of the fittest(according to NSGA-II [Deb et al. 2002])
Boolean dominance (x Dominates y, or does not):- In no objective is x worse than y- In at least one objective, x is better than y Crowd
pruning
9
Indicator-Based Evolutionary Algorithm (IBEA) [Zitzler and Kunzli ‘04]
1) For {old generation + new generation} do– Add up every individual’s amount of dominance with
respect to everyone else
– Sort all instances by F– Delete worst, recalculate, delete worst, recalculate, …
2) Then, standard GA (cross-over, mutation) on the survivors Create a new generation Back to 1.
10
NSGA-II… the default algorithm• Much prior work in SBSE (*)
--------------------------(*) Sayyad and Ammar, RAISE’13
Used NSGA-II
Didn’t state why!
Roadmap
① Randomness of Search② The original study③ The replication④ Conclusion
12
The Original Study
• A. Arcuri and G. Fraser, "On Parameter Tuning in Search Based Software Engineering," in Proc. SSBSE, 2011, pp. 33-47.
• A. Arcuri and G. Fraser, "Parameter Tuning or Default Values? An Empirical Investigation in Search-Based Software Engineering," Empirical Software Engineering, Feb 2013.
• Problem: generating test vectors for object-oriented software.
• Fitness function: percentage of test coverage.
13
Results of original study
• Different parameter settings cause very large variance in the performance.
• Default parameter settings perform relatively well, but are far from optimal on individual problem instances.
Roadmap
① Randomness of Search② The original study③ The replication④ Conclusion
15
Feature–oriented domain analysis [Kang 1990]
• Feature models = a lightweight method for defining a space of options
• De facto standard for modeling variability, e.g. Software Product Lines
Cross-Tree Constraints
Cross-Tree Constraints
16
What are the user preferences?• Suppose each feature had the following metrics:
1. Boolean USED_BEFORE?2. Integer DEFECTS3. Real COST
• Show me the space of “best options” according to the objectives:1. That satisfies most domain constraints (0 ≤ #violations ≤ 100%)2. That offers most features3. Maximize overall feature that were used before. (promote re-use)4. Minimize overall known defects.5. Minimize cost.
17
Previous Work [Sayyad et al. ICSE’13]
• IBEA (continuous dominance criterion) beats NSGA-II and a host of other algorithms based on Boolean dominance criterion.
• Especially with a high number of objectives.• Quality indicators:– Percentage of conforming (useable) solutions
• We’re interested in 100% conforming solutions.
– Hypervolume (how close to optimal?)– Spread (how diverse?)
18
Setup
19
What are “default settings”?
• Population size = 100
• Crossover rate = 80%– 60% < Crossover rate < 90%• [A. E. Eiben and J. E. Smith, Introduction to Evolutionary
Computing.: Springer, 2003.]
• Mutation rate = 1/Features • [one bit out of the whole string]
20
Research Questions
21
Results [10 sec / algorithm / FM]
22
Answer to RQ1
• RQ1: How Large is the Potential Impact of a Wrong Choice of Parameter Settings?
• We confirm Arcuri and Fraser’s conclusion: “Different parameter settings cause very large variance in the performance.”
23
Answer to RQ2
• RQ2: How Does a “Default” Setting Compare to the Best and Worst Achievable Performance?
• Arcuri and Fraser concluded that: “Default parameter settings perform relatively well, but are far from optimal on individual problem instances.”
• We make a stronger conclusion: “Default parameter settings perform generally poorly, but might perform relatively well on individual problem instances.”
24
Answer to RQ3
• RQ3: How does the performance of IBEA’s best tuning compare to NSGA-II’s best tuning?
• Our results show that “IBEA’s best tuning performs generally much better than NSGA-II’s best tuning.”
25
RQ4: Parameter Training• Find best tuning for a group of problem instances, apply it to a
new problem instance, would it be best tuning for the new problem?
• Arcuri and Fraser concluded that: “Tuning should be done on a very large sample of problem instances. Otherwise, the obtained parameter settings are likely to be worse than arbitrary default values.”
• Our conclusion: “Tuning on a sample of problem instances does not, in general, result in the best parameter values for a new problem instance, but the obtained setting are generally better than the defaults settings.”
Roadmap
① Randomness of Search② The original study③ The replication④ Conclusion
27
Conclusion
• And, sadly, many SBSE researchers choose “default” algorithms (e.g. NSGA-II) along with “default” parameters.
• Default parameter values fail to optimize performance…
Acknowledgment
This research work was funded by the
Qatar National Research Fund under the National Priorities
Research Program • Alternatives?– A long way to go!
• Parameter control• Adaptive parameter control