Presentation for ISCAS – 23/10/2008€¦ · Software Release Planning Problem F F F F F F F F F...

1

Presentation for ISCAS – 23/10/2008

Simulation-Based Planning, Re-Planning and Stability Analysis for Operational Release Plans

Dietmar PfahlSimula Research Laboratory & University of Oslo

2

Outline

Software Release Planning ProblemSoftware Release Planning ProblemSimulationSimulation--Based Operational Release Based Operational Release PlanningPlanningPlanning and Dynamic RePlanning and Dynamic Re--PlanningPlanningStability AnalysisStability AnalysisWorkWork--inin--Progress and Future WorkProgress and Future WorkConclusionConclusion

2

3

Evolutionary Software Development

Project 1Release 1

Project 2

Project 3

Set o

f Fea

ture

s! Technological Constraints

! Resource Constraints

! Risks! Budget Constraints

! Stakeholder Value

Release 2

Release3

Shorter Time-to-Market

Earlier Customer FeedbackEasier to Maintain

flexible approach better product better customer satisfaction

Better Resource Allocation

4

Software Release Planning Problem

F F FF F FF F F

Release 1

Fixed Schedule

Features

Release 2

Release n

. . . Budget and ResourceConstraints, Risks,Value, Stakeholder Preferences, etc.

Strategic Level “sort of” Knapsack ProblemOperational Level “sort of” Project Planning

Problem ( resource allocation / scheduling)

Features,TasksDevelopers

NP-hard problems!

3

5

Operational Release Planning Problem

Consider this 7-tuple <F, T, FT, D, E, P, Dep>:F – set of release-specific features f [1, …, k]T – set of task types t [1, …, m]FT – set of feature/task type-combinations (= tasks) ft [1, …, km]D – set of developers d [1, …, n]E – set of estimated efforts eff per task [1, …, km]P – set of estimated (relative) productivities prod

per task [1, …, nm]Dep – dependency relationships dep between subsequent task types

[1, …, m-1]

Goal: assign developers d ∈ D to tasks ft ∈ FT such that

maxi=1…k{end-time(ftim)} min

(and additional restrictions hold, i.e., one-to-one mapping of dk to ftij)

f1f2

ft11 ft12 ft13

ft21 ft22 ft23

6

Example with Start-Start Dependency

Features (F)Estimated workload (effort) per task type [e.g., Person-Weeks (PW)]

Task Types (T): design, implementation, test, etc.

dependency

Developers (D)(relative) productivity per task type [0 … 2]one task at a timeno task change once assigned

?

4

7

Example with Start-Start Dependency

[NgR05] Ngo-The, A and Ruhe, G (2005) Optimized Resource Allocation for Incremental Software Development (submitted to Transactions on SE).

Operational Release Planning Problem

Feature/Task ScheduleDeveloper Allocation

Example Case:8 Features3 Task Types6 Developers

D4

8

Outline


5

9

Operational Release Planning (ORP)

Example:

End-start dependency between subsequent tasks1-to-1 mapping between developers and tasks

Effort Estimates [person-week]

Productivity Estimates

[dimensionless]

F1 F2 F3 F4 F5 F6 F7 F8 D1 D2 D3 D4 D5 D6

T1: Design 3 8 6 3 5 7 10 6 1.5 1 2 0 0.5 2

T2: Implementation 6 3 10 3 6 5 5 8 2 1.5 1 2 1.5 1 Task Type

T3: Test 6 2 5 6 4 3 6 10 1 2 0 1.5 2 1

Allocate developers to tasks such that:

maxi=1…8{end-time(taski,3)} min

k Features

m

10

Operational Release Planning (ORP)

Example:Assignment of developers to feature/task-combinations:

0 7.5 15 22.5 30Time (Week)

F1,T1F1,T2F1,T3F2,T1F2,T2F2,T3F3,T1F3,T2F3,T3F4,T1F4,T2F4,T3F5,T1F5,T2F5,T3F6,T1F6,T2F6,T3F7,T1F7,T2F7,T3F8,T1F8,T2F8,T3

D6D4

D4D3

D4D4

D2D2

D6D1

D3D2

D3D1

D6D1

D1D1

D6D4

D2D5

D5D5

Assignment of developers to feature/task-combinations:

0 7.5 15 22.5 30Time (Week)


D6D4

D4D3

D4D4

D2D2

D6D1

D3D2

D3D1

D6D1

D1D1

D6D4

D2D5

D5D5

D4

So why simulation-based ORP ?

REPSIM-2Is about5-10% worsethanOPTIMIZERAZORP

6

11

Simulation-Based ORP

WhyWhy??More More flexibilityflexibility withwith regards to regards to

dynamicdynamic rere--planning,planning,stabilitystability analyses, and analyses, and problemproblem refinementrefinement

as as comparedcompared to (to (staticstatic) ) optimizationoptimization algorithmsalgorithmsHow?How?

UsingUsing eithereither discretediscrete--eventevent or or continuouscontinuous processprocesssimulation simulation modelsmodels (have (have bothboth))

WhatWhat??(Planning and) (Planning and) ReRe--planningplanningStabilityStability AnalysisAnalysis

12

Outline


7

13

Technique: Discrete-Event

Tool: EXTEND®

Planning – Simulation Model DynaReP

14

Planning – Model Parameters

Release r

Schedule

F4 F8

F3 F7

F5

F2 F6

F1

Features

T1 10 D6 D6 D6 D6 D6T2 6 D5 D5 D5 D5T3 6 D2 D2 D2

D2

D5

D6

prod

2

1.5

2

D2 D4 D6

D1 D3 D5

Developers

Task 1

Task 2

Task 3eff10, 6, 6

TaskDependency∈ [0%, 100%]

Period (weeks): 1 2 3 4 5 6 7 8 9 10 11 12

k

n

m

8

15

Planning – Heuristic

Assign the next available developer with the highest task-specific productivity to the next waiting feature with the largest effort (for a specific task).

Note:If only developers with very low productivity are currently idle, this mapping rule can result in assigning a developer with low productivity to a large feature will become a bottleneck!To avoid such a worst case situation, threshold variables are defined which exclude developers with a productivity that is too low.Finding a good set of threshold values can be automized(using a built-in optimization functionality)

16

Dynamic Re-PlanningPlanning

Initial Plan

9

17

Initial PlanInitial Plan

Modified PlanModified Plan

Dynamic Re-Planning (Example 1)

Developer D4 becomesunavailableafter end ofthe 7th week

18

Dynamic Re-Planning (Example 2)

Feature F9 is addedafter end ofthe 7th week

Initial PlanInitial Plan

Modified PlanModified Plan

10

19

Re-Planning – Other Possible Analyses

Drop in/out of developers Addition/deletion of featuresOverestimated or underestimated effortsOverestimated or underestimated productivitiesVarying task type dependencies

All the above:In any combinationAt any point in time (repeatedly)

20

Re-Planning – Additional Analyses after Model Enhancement

In addition or complimentary to the previous:Productivities defined per feature (not task type)Feature dependencies

Other aspects: Learning effects during developmentTime pressure effects… and many others

11

21

Outline


22

Stability Analysis – Why?

Problem:Planning parameters are estimates

Based on empirical dataBased on expert knowledge

It makes sense to assume distributions that define a “probable range” for actual parameter values

12

23

Stability AnalysisMain question in the following:

How sensitive does the initial operational plan react to changes in any of the planning parameters, i.e.

Effort estimatesProductivity estimatesTask type dependencies

Two classes of analyses currently possible:Developer allocations to tasks are

Fixed Analyze effect on work-backlogFlexible Analyze effect on duration and plan structure

24

Stability Analysis

Main Question:How sensitive does the initial operational plan react to changes in any of the planning parameters, i.e.




13

25

Technique: SystemDynamics

Tool: Vensim®

Stability Analysis – REPSIM-1

F-TF-T-inflow F-T-outflow

actual-Eff-F-T

actual-Prod-T-D

actual-Prod-F-T-D

Eff-Variation

Prod-Variation

Eff-F-T

Prod-T-D

Cum-F-T-outflow

F-T-inflow-waiting

Task-Dependency

<Alloc-F-T-D>

new - B ase line00 0

"F - T- o utflo w "[F eature ,Task ]

0 3 6 9 1 2Time (W eek )

F 1 ,T1F 1 ,T2F 1 ,T3F 2 ,T1F 2 ,T2F 2 ,T3F 3 ,T1F 3 ,T2F 3 ,T3F 4 ,T1F 4 ,T2F 4 ,T3F 5 ,T1F 5 ,T2F 5 ,T3F 6 ,T1F 6 ,T2F 6 ,T3F 7 ,T1F 7 ,T2F 7 ,T3F 8 ,T1F 8 ,T2F 8 ,T3

26

Stability Analysis – Example 1Gantt charts showing active tasks for varying task dependencies (Baseline = Task Type Dependency 0%)

new - Baseline0 0 0

"F - T- o utflo w "[F ea ture,Task ]

0 3 6 9 1 2Time (W eek )


new - Base line0 5 0


0 3 6 9 1 2Time (W eek )


new - Base line1 0 0


0 3 6 9 1 2Time (W eek )


0% 50% 100%

14

27

Stability Analysis – Example 1Work load reduction per task type for task type dependency equal to 0%: absolute values on the left, relative values on the right

Work load reduction per task type for task type dependency equal to 50%: absolute values on the left, relative values on the right

Eff-T-gap60

45

30

15

0

33

33

3 3 33

33

33 3 3 3

2 22

22

22

22

22

22

22

2

11

11

11

11

1 1 11

11

1 10 1 2 3 4 5 6 7 8 9 10 11 12

Time (Week)

"Eff-T-gap"[T1] : new-Baseline000 Week*Person1 1 1 1 1 1"Eff-T-gap"[T2] : new-Baseline000 Week*Person2 2 2 2 2 2"Eff-T-gap"[T3] : new-Baseline000 Week*Person3 3 3 3 3 3 3

Eff-T-gap-relative100

75

50

25

0

33

33

33

33

3

3

33 3 3

3

2 22

22

22

22

22

22

22

1

11

11

11

11 1

11

11

1 10 1 2 3 4 5 6 7 8 9 10 11 12

Time (Week)

"Eff-T-gap-relative"[T1] : new-Baseline000 Dmnl1 1 1 1 1 1"Eff-T-gap-relative"[T2] : new-Baseline000 Dmnl2 2 2 2 2 2 2"Eff-T-gap-relative"[T3] : new-Baseline000 Dmnl3 3 3 3 3 3

Eff-T-gap60

45

30

15

0

3 3 3 3 3 3 3 3 33

33 3 3 3

2 2 22

22

22

22

22

22

22

11

11

11

11 1 1 1

11

11 1

0 1 2 3 4 5 6 7 8 9 10 11 12Time (Week)

"Eff-T-gap"[T1] : new-Baseline050 Week*Person1 1 1 1 1 1"Eff-T-gap"[T2] : new-Baseline050 Week*Person2 2 2 2 2 2"Eff-T-gap"[T3] : new-Baseline050 Week*Person3 3 3 3 3 3 3

Eff-T-gap-relative100

75

50

25

0

3 3 3 33

3 3 3 33

33 3 3

3

2 22

22

22

22

22

22

22

1

11

11

11

11 1 1

11

11 1

0 1 2 3 4 5 6 7 8 9 10 11 12Time (Week)

"Eff-T-gap-relative"[T1] : new-Baseline050 Dmnl1 1 1 1 1 1"Eff-T-gap-relative"[T2] : new-Baseline050 Dmnl2 2 2 2 2 2 2"Eff-T-gap-relative"[T3] : new-Baseline050 Dmnl3 3 3 3 3 3

28

Stability Analysis – Example 1Task type-specific work backlog (cumulated over features) for varying task dependencies

Workload of Unfinished Tasks (relative)

0

10

20

30

40

50

60

70

80

90

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Task Completion Dependency (uniform)

Estim

ated

Tas

k Vo

lum

e B

ackl

og[%

of T

otal

Est

imat

ed T

ask

Volu

me]

Eff-T-gap-relative[T1] Eff-T-gap-relative[T2] Eff-T-gap-relative[T3]

15

29

Stability Analysis – Example 2Task type-specific work backlog (cumulated over features) for varying effort estimates


0.002.004.006.008.00

10.0012.0014.0016.0018.0020.00

0.80 0.90 1.00 1.10 1.20

Estimated Task Volume Variation (uniform)

Estim

ated

Tas

k Vo

lum

e B

ackl

og[%

of T

otal

Est

. Tas

k Vo

lum

e]

Eff-S-gap-relative[T1] Eff-S-gap-relative[T2] Eff-S-gap-relative[T3]

30

Stability Analysis – Example 3Task type-specific work backlog (cumulated over features) for varying productivity estim.


0.002.004.006.008.00

10.0012.0014.0016.0018.0020.00

0.80 0.90 1.00 1.10 1.20

Estimated Developer Productivity Variation (uniform)

Estim

ated

Tas

k Vo

lum

e B

ackl

og[%

of T

otal

Est

. Tas

k Vo

lum

e]


16

31

Stability Analysis – Example 1+2Task type-specific work backlog (cumulated over features) for varying task dependencies and varying effort estimates

Workload of Unfinished Tasks (absolute) withTask Dependency = 0.0 (uniform)

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

0.80 0.90 1.00 1.10 1.20


Estim

ated

Tas

k Vo

lum

e B

ackl

og[%

of T

otal

Est

imat

ed T

ask

Volu

me]

Eff-S-gap-relative[T1] Eff-S-gap-relative[T2] Eff-S-gap-relative[T3]Workload of Unfinished Tasks (absolute) with

Task Dependency = 0.1 (uniform)

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

0.80 0.90 1.00 1.10 1.20


Estim

ated

Tas

k Vo

lum

e B

ackl

og[%

of T

otal

Est

imat

ed T

ask

Volu

me]

Eff-S-gap-relative[T1] Eff-S-gap-relative[T2] Eff-S-gap-relative[T3]Workload of Unfinished Tasks (absolute) with

Task Dependency = 0.2 (uniform)

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

0.80 0.90 1.00 1.10 1.20


Estim

ated

Tas

k Vo

lum

e B

ackl

og[%

of T

otal

Est

imat

ed T

ask

Volu

me]


32

Stability AnalysisMain Question:

How sensitive does the initial operational plan react to changes in any of the planning parameters, i.e.




17

33


Tool: Vensim®


F-TF-T-inflow F-T-outflow

actual-Eff-F-T

actual-Prod-T-D

actual-Prod-F-T-D

Eff-Variation

Prod-Variation

Eff-F-T

Prod-T-D

Cum-F-T-outflow

F-T-inflow-waiting

Task-Dependency

<Alloc-F-T-D>Assignment of developers to feature/task-combinations:

0 7.5 15 22.5 30Time (Week)


D6

D4

D4

D3

D4

D4

D2D2

D6

D1

D3

D2

D3

D1

D6

D1

D1

D1

D6

D4

D2

D5

D5

D5

34


Tool: Vensim®


Alloc-F-T-Dassign-task remove-from-task

Idle-D-Pool Assigned-D-Poolallocate-D

<F-T>

<TIME STEP>

release-D

<TIME STEP>

Threshold

"Assigned-D-Pool"[Developer]

0 7.5 15 22.5 30Time (Week)

D1

D2

D3

D4

D5

D6

F2

F6 F4 F5 F6 F6

F3 F3 F3 F3

F5 F4

F2 F2 F1 F1

F8 F8 F8

F7 F1 F5 F3

F7

Alloc-F-T-Dassign-task remove-from-task

Idle-D-Pool Assigned-D-Poolallocate-D

<F-T>

<TIME STEP>

release-D

<TIME STEP>

Threshold


0 7.5 15 22.5 30Time (Week)

D1

D2

D3

D4

D5

D6

F2

F6 F4 F5 F6 F6

F3 F3 F3 F3

F5 F4

F2 F2 F1 F1

F8 F8 F8

F7 F1 F5 F3

F7


0 7.5 15 22.5 30Time (Week)

D1

D2

D3

D4

D5

D6

F2

F6 F4 F5 F6 F6

F3 F3 F3 F3

F5 F4

F2 F2 F1 F1

F8 F8 F8

F7 F1 F5 F3

F7

18

35

Stability Analysis – Research GoalResearch Question:Analyze impact of effort and productivity over/under-estimation on

DurationStructure of Operational Release Plan:

Developer allocation to tasksDifferences in start and end times of tasks

"Alloc-F-T-D"[Feature,Task,D1]

0 7.5 15 22.5 30Time (Week)


36

Stability Analysis – HypothesesDecrease (increase) of productivity and increase (decrease) of effort results in …

H1: significant increase (decrease) of release duration.H2: significant instability in the assignment of developers to tasks. H3: significant instability in the start and end times of tasks.

19

37

Stability Analysis – ExperimentsBaseline Case Simulations

38

Stability Analysis – ResultsEffect on Duration Distribution Fitting

Histogram (Spreadsheet1 1v*50c)

Var1 = 50*0.5*normal(x, 24.5413, 1.4342)

20.5 21.0 21.5 22.0 22.5 23.0 23.5 24.0 24.5 25.0 25.5 26.0 26.5 27.0 27.5 28.0 28.5

Var1

0

2

4

6

8

10

12

No

of o

bs

N(24.54, 1.43)

20

39

Stability Analysis – ResultsEffect on Duration Single Sample T-Test

?

alpha = 0.05

?

40

Stability Analysis – ResultsEffect on Duration (with Effect Sizes)

21

41

Stability Analysis – ResultsEffect on Developer Allocation

Total number of allocations = 8 x 3 = 24With 6 developers:

Box & Whisker Plot

Median 25%-75% Min-Max Case 1

Case 2Case 3

Case 4Case 5

Case 6Case 7

Case 8Case 9

Case 10Case 11

Case 12

0

2

4

6

8

10

12

14

16

18

20

22

24

Median: 29% change Median: 58% change

42

Stability Analysis – ResultsEffect on Task Scheduling

Formulas:

Data:

Dv_diff ∈ [0, 1]

22

43

Stability Analysis – ResultsDetails of Case 10

1816141210864

16

12

8

4

0726048362412

8

6

4

2

0

80706050403020

8

6

4

2

00.80.70.60.50.4

20

15

10

5

0

Alloc_diffFr

eque

ncy

ST_diff

ET_diff Dv_diff

Mean 12.16StDev 2.780N 50

Alloc_diff


ST_diff


ET_diff

Mean 0.61StDev 0.09871N 50

Dv_diff

Histogram of Alloc_diff, ST_diff, ET_diff, Dv_diffNormal

44

Stability Analysis – SummaryH1: Variation Impact on Duration ?

Asymmetric Variation: duration is always significantly differentSymmetric Variation: Cases 2 and 10 (and almost 8) have no significantly difference in duration (and effect size < 0.5, i.e., no “practical significance” according to Cohen)

H2: Variation Impact on Developer Allocation yesAverage change of developer allocation is in the range of 29-58% of all allocations

H3: Variation Impact on Task Scheduling yesAverage change of feature/task scheduling is in the range of 0.42-0.71 (where 1 is equivalent to zero overlap).

Note: These results have been corroborated in simulations with much larger releases (>60 features; up to 13 developers).

23

45

Additional Stability AnalysesFeature inclusion (a: 3 weeks / b: 6 weeks)

run1 11c ase0 -ba se linerun1 11c ase0 -s1arun1 11c ase0 -s1b

"F-T -out flow " [Fe ature ,Ta sk]

0 7 .5 1 5 22 .5 3 0T ime (W ee k)

F1,T 1F1,T 2F1,T 3F2,T 1F2,T 2F2,T 3F3,T 1F3,T 2F3,T 3F4,T 1F4,T 2F4,T 3F5,T 1F5,T 2F5,T 3F6,T 1F6,T 2F6,T 3F7,T 1F7,T 2F7,T 3F8,T 1F8,T 2F8,T 3F9,T 1F9,T 2F9,T 3

46

Outline


24

47

Work in Progress

Model EnhancementsFacilitate more types of analyses

e.g., feature-dependency, (partly) fixed developer allocations in re-planning

Relax assumptions e.g., 1-to-1 relationships between task types, 1-to-1 assignment of developers to tasks

Complement with Empirical Studies

48

ConclusionSimulation-based planning/re-planning of releases on operational level may support decision makers in dynamic environmentsSimulation-based stability (or uncertainty) analysis could be an input to risk managementLimitations:

Initial plans are not optimal (about 5-10%)Experimental basis still small “Risk-drivers” (internal properties) of ORPs not yet fully clear

25

49

Future WorkAnalyze properties of ORPs to better understand when duration is significantly effected by estimate uncertaintyImprove heuristics or find way to integrate with (static) optimization algorithmsIntegrate with strategic product management (multi-release perspective)

50

Thank You

26

51

ReferencesAl-Emran, A.; Khosrovian, K.; Pfahl, D.; Ruhe, G.: Studying the Impact of Uncertainty in Operational Release Planning. Submitted to ESEM 2007.Al-Emran, A.; Pfahl, D.: Operational Planning, Re-Planning and Risk Analysis for Software Releases. Accepted for publication in Proceedings of PROFES 2007.Al-Emran, A.; Pfahl, D.; Ruhe, G.: DynaReP – A Discrete Event Simulation Model for Re-planning of Software Releases. Accepted for publication in Proceedings of ICSP 2007.Pfahl, D.: ProSim/RA – Software Process Simulation in Support of Risk Assessment. In: S. Biffl, A. Aurum, B. Boehm, H. Erdogmus, P. Grünbacher (eds.), Value-based Software Engineering, Berlin: Springer, 2005, 263-286, ISBN 3-540-25993-7.Pfahl, D., Al-Emran, A., Ruhe, G.: Simulation-Based Stability Analysis for Software Release Plans. In: Wang, Q. et al. (eds.): SPW/ProSim 2006 - Proceedings. LNCS 3966, Berlin-Heidelberg: Springer, 2006, 262-273.Pfahl, D.; Al-Emran, A.; Ruhe, G.: A System Dynamics Model for Analyzing the Stability of Software Release Plans. In: Software Process Improvement and Practice (in print).

---------------------------------------------------------------------------------------------------------------Ngo-The, A., Ruhe, G.: Optimized Resource Allocation for Incremental Software Development. TR 062/2006, Laboratory for Software Engineering Decision Support, University of Calgary (2006)

52

Simulated Plan vs. Optimal Plan [NgR05]

27

53

new-Baseline100case0new-Baseline100case0-old


0 7.5 15 22.5 30Time (Week)

D1

D2

D3

D4

D5

D6

new-Baseline100case0new-Baseline100case0-old

"Idle-D-Pool"[Developer]

0 7.5 15 22.5 30Time (Week)

D1

D2

D3

D4

D5

D6

Stability Analysis – Sim. Model B-2

Presentation for ISCAS – 23/10/2008€¦ · Software Release Planning Problem F F F F F F F F F...

Documents

Transcript of Presentation for ISCAS – 23/10/2008€¦ · Software Release Planning Problem F F F F F F F F F...