Automated Software Engineering

Cs791Fall 2015

tjmenzie@ncsu.eduNov 21, 2014

Career advice

Learn Modeling

(it’s the next big thing in SE)

2

This kind of modeling? No

3

This kind of modeling? Nope

4

(though it does look pretty cool, heh?)

Executable Software Models

5

Software models: are everywhere

• If you call an ambulance in London or New York,

– those ambulances are controlled by emergency response models.

• If you cross the border Arizona to Mexico,

– A models determines if you are taken away for extra security measures.

• If you default on your car loans,

– A model determines when (or if) someone to repossess your car.

• If the stock market crashes,

– it might be that some model caused the crash.

6

• Karplus and Levitt – 2013 Nobel prize in chemistry– development of multi-scale

models for complex chemical systems

– Explored complex chemical reactions (e.g. split-second changes of photosynthesis).

7

• Models are now a central tool in scientific research.

– in physics, biology and other fields of science

– complex simulations using supercomputers.

• E.g. genomic map required analyzing 80 trillion bytes

• E.g.. Other computational modeling projects– the rise and fall of native

cultures, – subnuclear particles – the Big Bang.

Software models: are everywhere

Question: How to best reason

about models?

Answer: automated software engineering

8

The Human Condition:a balancing act between goals

• The human condition – a constant balancing act between vaguely understood, often

competing, goals.

• For example: – Lets build the software faster...

– ... with few bugs ...

– ... using less budget.

• For the consequences of "better, faster, cheaper” :– NASA's loss of the Mars Polar

Orbiter

– $300 million , wasted9

Example

• Choices explored by satellite designers at NASA's Jet Propulsion Laboratory.

– Dozens of experts in propulsion,communications,

guidance and control etc– Week-long sessions – Thrash out possibilities

for NASA's next deep space mission.

• Groups sitting “here” can make decisions that impact “there”, and they just do not realize.

10

Keeping track of all those decisions:Lightweight requirements

notations• E.g. Martin Feather’s DDP

language– Mitigations …– … that retire risks….– …. That damage

requirements

• Multi-objective optimization– Find the cheapest

mitigations that enable the most requirements

11

Keeping track of all those decisions:Lightweight requirements

notations• My optimizers out-performing

state of the art tools

– Faster

– Simpler policies

12

Data Mining for Very Busy People, Tim Menzies, Ying Hu, IEEE Computer, October 2003

Keeping track of all those decisions:Lightweight requirements

notations• E.g. John

Mylopoulos‘soft goals

• Competingrequirements

• Simulations toexplore allthe what-ifs

13Chiang, Eliza, Menzies, Tim, Simulations for very early lifecycle quality evaluations, Software Process: Improvement and Practice 7(3-4), 2003,

• E.g. Kang’s product lines[Kang’90]

• Add in known constraints – E.g. “if we use a camera

then we need a high resolution screen”.

• Extract products – Find subsets of the

product lines that satisfy constraints.

– If no constraints, linear time

– Otherwise, can defeat state-of-the-art optimizers [Pohl et at, ASE’11] [Sayyad, Menzies ICSE’13].

Cross-TreeConstraints

More lightweight requirements notations

14

A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.

Automated software engineering and optimization

• Many SE activities are like optimization problems [Harman,Jones’01].

• Due to computational complexity, exact optimization methods can be impractical for large SBSE problems

• So researchers and practitioners use metaheuristic search to find near optimal or good-enough solutions.– E.g. simulated annealing [Rosenbluth et al.’53]

– E.g. genetic algorithms [Goldberg’79]

– E.g. tabu search [Glover86]

15

• Repeat till happy or exhausted– Selection (cull the herd)

– Cross-over (the rude bit)

– Mutation (stochastic jiggle)

Optimization andevolutionary algorithms

12

3

5

4

6

7

8

9

Pareto frontier-- better on some

criteria, worse on noneSelection:

-- generation[i+1] comes from Pareto frontier of generation[i]

16

Long tradition of software, simulation, and optimization

• Not one “best” solutions– Its all trade-offs– Satisficing = satisfy + sacrifice

• Problems have to be explored via computersimulations

– Try it out and see– And approach pioneered

by John Von Neumann,in the 1950s

17

Explosive growth of SE + optimization

Q: Why?

A: Thanks to Big Data, more access to more cpu.18

Applications of Optimization in SE1. Requirements Menzies, Feather, Bagnall, Mansouri, Zhang

2. Transformation Cooper, Ryan, Schielke, Subramanian, Fatiregun, Williams

3.Effort prediction Aguilar-Ruiz, Burgess, Dolado, Lefley, Shepperd

4. Management Alba, Antoniol, Chicano, Di Pentam Greer, Ruhe

5. Heap allocation Cohen, Kooi, Srisa-an

6. Regression test Li, Yoo, Elbaum, Rothermel, Walcott, Soffa, Kampfhamer

7. SOA Canfora, Di Penta, Esposito, Villani

8. Refactoring Antoniol, Briand, Cinneide, O’Keeffe, Merlo, Seng, Tratt

9. Test Generation Alba, Binkley, Bottaci, Briand, Chicano, Clark, Cohen, Gutjahr, Harrold, Holcombe, Jones,

Korel, Pargass, Reformat, Roper, McMinn, Michael, Sthamer, Tracy, Tonella,Xanthakis, Xiao,

Wegener, Wilkins

10. Maintenance Antoniol, Lutz, Di Penta, Madhavi, Mancoridis, Mitchell, Swift

11. Model checking Alba, Chicano, Godefroid

12. Probing Cohen, Elbaum

13. UIOs Derderian, Guo, Hierons

14. Comprehension Gold, Li, Mahdavi

15. Protocols Alba, Clark, Jacob, Troya

16. Component sel Baker, Skaliotis, Steinhofel, Yoo

17. Agent Oriented Haas, Peysakhov, Sinclair, Shami, Mancoridis 19

Example 1(of four)

Automated program repair

20

Multi-objective Optimization, in the 21st century, automated repair

A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12

21

A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12

22

A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12

23

•

A Systematic Study Of Automated Program Repair: Fixing 55 Out Of 105 Bugs For $8 Each : Claire Le Goues ; Michael Dewey-vogt ;Stephanie Forrest ; Westley Weimer, ICSE’12

24

Example 2(of four)

Software Product Lines

25

• E.g. Kang’s product lines[Kang’90]

• Add in known constraints – E.g. “if we use a camera

then we need a high resolution screen”.

• Extract products – Find subsets of the

product lines that satisfy constraints.

– If no constraints, linear time

– Otherwise, can defeat state-of-the-art optimizers [Pohl et at, ASE’11] [Sayyad, Menzies ICSE’13].

Cross-TreeConstraints

More lightweight requirements notations

26

A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.

Problem: many competing goals2 or 3 or 4 or 5 goals

Software engineering = navigating the user goals:1. Satisfy the most domain constraints (0 ≤ #violations ≤ 100%)2. Offers most features3. Build “stuff” In least time4. That we have used most before5. Using features with least known defects

27

A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.

Issues of scale up• This model: 10 features, 8 rules

• [www.splot-research.org]: ESHOP: 290 Features, 421 Rules

• LINUX kernel variability projectLINUX x86 kernel6,888 Features; 344,000 Rules

Cross-Tree Constraints

28

A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.

State of the Art (as of Nov’13)Features

9

290

544

6888

SPLO

TLi

nu

x (L

VA

T)

Pohl ‘11 Lopez-Herrejon

‘11

Henard‘12

Sayyad,Menzies’

13a

Velazco ‘13

Sayyad, Menzies’13b

Johansen ‘11

Benavides ‘05

White ‘07, ‘08, 09a, 09b, Shi ‘10, Guo ‘11

Objectives

Multi-goalSingle-goal

300,000+

clauses

29A.S. Sayyad, J. Ingram, T. Menzies, and H. Ammar: Scalable Product Line Configuration: A Straw to Break the Camel's Back. In the International Conference on Automated Software Engineering (ASE’13). Palo Alto, USA. November 2013.

Example 3(of four)

Space ships!

30

Skip re-entry• My optimizers vs state of the art numeric optimizers

• My tools: ran 40 times faster

– Generated better solutions

• Powerful succinct explanation tool

31Automatically Finding the Control Variables for Complex System Behavior Gregory Gay, Tim Menzies, Misty Davies, and Karen Gundy-Burlet Journal - Automated Software Engineering, 2010 [PDF]

Example 4 (of four)

Avionics Requirements Engineering

32

33Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim Menzies, Misty Davies, IEEE Trans Human Factors, to appear

WMC: GIT’s Work Models that Compute [Kim’11]

• Cognitive models of the agents (both pilots and computers) – Late descent,

– Unpredicted rerouting,

– Different tailwind conditions

• Goal: validate operations procedures (are they safe?)

• NASA’s analysts want to explore 7000 scenarios.– With current tools (NSGA-II)

– 300 weeks to complete

• Limited access to hardware– Queue of researchers wanting

hardware access

– Hardware pulled away if in-flight incidents for manned space missions

Asiana AirlinesFlight 214

34Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim Menzies, Misty Davies, IEEE Trans Human Factors, to appear

Active Learning: Skip similar examples,focus on the most different

4 mins (GALE) vs 7 hours (rest)

Better Model-Based Analysis of Human Factors for Safe Aircraft Approach, Joseph Krall, Tim Menzies, Misty Davies, IEEE Trans Human Factors, to appear

Career advice

Learn Modeling

(it’s the next big thing in SE)36

Question: How to reason about models?

Answer: automated software engineering

Cs791, fall 2015

37