Productivity measurement of agile teams (IWSM 2015)

28
Performance measurement of agile teams

Transcript of Productivity measurement of agile teams (IWSM 2015)

Page 1: Productivity measurement of agile teams (IWSM 2015)

Performance

measurement of agile

teams

Page 2: Productivity measurement of agile teams (IWSM 2015)

|

Performance measurement of agile teamscontinuous development of a product –performance measured by sprint

Cracow Poland, October 7 2015

2Performance measurement of agile teams

Harold van Heeringen, METRI

Theo Prins, Sogeti Nederland

Edwin van Gorp, Sogeti Nederland

Page 3: Productivity measurement of agile teams (IWSM 2015)

|

Sizing, Estimating & Control

Sogeti department responsible for: Functional size measurement: Nesma, IFPUG and COSMIC; Estimating new projects, releases and contracts; Project control based on software metrics; Performance measurement; Benchmarking; Performance reporting.

This presentation is about one engagement where the SEC department was asked to help solving the issues with regard to the productivity of a Sogeti agile team in a government project…. measured per sprint!

Performance measurement of agile teams 3

Page 4: Productivity measurement of agile teams (IWSM 2015)

|

Background

RfP of Dutch government agency in 2014 Agile development of a product, taken over from another supplier; Performance measurement of the team per sprint; Based on ISO/IEC norm, in this case Nesma function points; Realistic targets set, based on Sogeti historical data, external

benchmark data and historical data of previous supplier!

Sogeti offered to measure performance based on Project Delivery Rate (PDR), measured in project function points delivered per sprint;

Politics: Customer needs to prove the new supplier performs better! Customer selects Sogeti because of proved historical performance

and specialized department Sizing, Estimating & Control.

4Performance measurement of agile teams

Page 5: Productivity measurement of agile teams (IWSM 2015)

|

The problem

The application turned out to be very complex. Even the OTAP environment was very complex to handle. The product backlog consisted of many non-functional backlog items;

Functional size measurement is a powerful way to objectively size progress when it comes to functionality added, changed, or deleted, but not for measuring non-functional sprint backlog items processed by the team;

The customer (product owner) decides which functional and non-functional backlog items are put on the sprint backlog;

Traditional data and performance measurement methods turned out to be ineffective when measuring performance on sprint level when more than average non-functional backlog items are put on the sprint backlog.

5Performance measurement of agile teams

Page 6: Productivity measurement of agile teams (IWSM 2015)

|

Two types of agile projects

1. Development of a set of specific requirements, prioritized on a backlog, realized in a specific duration by a specific team in a specific amount of effort hours and cost. At one point, the project, and the product, is finished. (in traditional terms: new development or release).

2. Continued development (evolving) an existing application. No definite end goal when the project or product is finished. A year divided into X sprints of Y weeks and a fixed team working to deliver the sprint backlog items. This only ends when the organization decides maintenance is no longer needed. (in traditional terms: maintenance).

This industry paper investigates Type number 2.

6Performance measurement of agile teams

Page 7: Productivity measurement of agile teams (IWSM 2015)

|

First supplier

Big international system integrator; Application developed from scratch; About 15 sprints; Sprints with fluctuating functional sizes in Nesma FP; High complexity; Average PDR about 17 hours/FP.

Almost all backlog items of functional nature!

However, the customer was not happy and decided to select a new supplier.

7Performance measurement of agile teams

Page 8: Productivity measurement of agile teams (IWSM 2015)

|

After transition

From Q3 2014 onwards, Sogeti took over; Sprints of 3 weeks; Sogeti team, product owner supplied by customer; Product backlog contains many non-functional items:

Scrum team works hard, but hardly delivers function points; PDR (h/FP) relatively high, not in line with target PDR;

Customer contract manager blames Sogeti for not being productive; Contract under pressure, media attention, pressure and politic issue; Sogeti department Sizing, Estimating & Control asked to analyze the

performance and to propose improvements.

Performance measurement of agile teams 8

Page 9: Productivity measurement of agile teams (IWSM 2015)

|

Function points

Important to understand, using a functional size measurement method (an ISO/IEC standard) means measuring the size of the functional user requirements that are implemented in the software;

Non-functional requirements are not measured at all! More non-functional work in a sprint means that less functionality is

realized, and therefore a higher PDR (hours/FP)/lower productivity. In project estimation and benchmarking, the influence of NFR is

accounted for by the historical data used or the parametric model used, or the peer group that is constructed based on projects with similar characteristics.

Performance measurement of agile teams 9

Page 10: Productivity measurement of agile teams (IWSM 2015)

|

Story points

Usual way to estimate effort in agile teams; Team members assign story points to each backlog item, reflecting the

amount of work needed to realize the item; Subjective, not repeatable, not verifiable and not defensible, but

mainstream practice in agile teams because of ease of use; Story point-based metrics can not be compared with any measurements

or metrics outside the team;

Story point measurement is not a standard, but SP do take into account the effort spent on non-functional backlog items.

Performance measurement of agile teams 10

Page 11: Productivity measurement of agile teams (IWSM 2015)

|

Non functional backlog items are important

11Performance measurement of agile teams

Sprint X FP SP Effort hours

Backlog item 1 4 4 90

Backlog item 2 0 6 120

Backlog item 3 0 2 45

Backlog item 4 5 3 65

Backlog item 5 4 3 80

Total 13 18 400

PDR = 400/13 = 30,8 Hours/FPRatio F/NF SP backlog items: 10/8

Page 12: Productivity measurement of agile teams (IWSM 2015)

|

Sprint performance example

Performance measurement of agile teams 12

Sprint 1 2 3 4 5 6 7 8

Effort 345 389 367 412 365 375 390 401

Size (FP) 15 5 16 3 25 0 36 32

Sprint 1 2 3 4 5 6 7 8

PDR

(h/FP)

23,0 77,8 22,9 137,3 14,6 n/a 10,8 12,5

0,0

20,0

40,0

60,0

80,0

100,0

120,0

140,0

160,0

3 4 5 6 7 8 9 10

PDR (h/FP) Target PDR (h/FP)

Page 13: Productivity measurement of agile teams (IWSM 2015)

|

Issue

Team spends a lot of time on non-functional backlog items; PDR is not good enough to reach target; Stakeholders don’t understand this ‘technical issue’ and only see metrics

on the dashboard PDR significant worse than expected; But… customer product owner decided on the product backlog items to

put on the sprint backlog! So, disappointing PDR mainly caused by the number of functional and

non-functional backlog items put in the sprint by the product owner (= the customer!).

Sogeti SEC wishes to address this issue and to come up with a proposal for a more accurate performance measurement method.

13Performance measurement of agile teams

Page 14: Productivity measurement of agile teams (IWSM 2015)

|

SEC proposal

Agile Normalized Size (ANS)Functional size that could have been realized if the product owner only had put functional backlog items in the sprint backlog.

Based on this ANS, a PDR (hour/FP) can be determined that can be compared to the PDR’s in the databases with historical data.

14Performance measurement of agile teams

Page 15: Productivity measurement of agile teams (IWSM 2015)

|

Method

1. Measure the functional size of the realized functional backlog items with a standard method (Nesma/IFPUG FPA, COSMIC, …);

2. Determine whether the realized backlog items are functional or non-functional;

3. Determine the number of story points of the functional backlog items realized in the sprint;

4. Determine the total number of story points realized in the sprint; 5. Determine the agile normalized size:

(functional size / # functional story points)

* total # story points

15Performance measurement of agile teams

Page 16: Productivity measurement of agile teams (IWSM 2015)

|

The example extended

16Performance measurement of agile teams

Sprint X FP SP Hours

Backlog item 1 4 4 90

Backlog item 2 0 6 120

Backlog item 3 0 2 45

Backlog item 4 5 3 65

Backlog item 5 4 3 80

Total 13 18 400

Agile normalized size = (13 / 10) * 18 = 23,4 nFPPDR = 400/23,4 = 17,1 hours/nFP

Functional size: 13 FP

Functional SP: 10 SP

Total SP: 18 SP

Regular PDR: 400/13

= 30,7 hours/FP

Page 17: Productivity measurement of agile teams (IWSM 2015)

|

The effect in multiple sprints

Sprint Size (FP) Functional SP Non-functional SP Total SP ANS (nFP)

16 20 32 12 44 27,5

17 25 28 16 44 39,3

18 18 24 20 44 33,0

19 29 35 4 39 32,3

20 4 6 36 42 28,0

21 15 16 24 40 37,5

17Performance measurement of agile teams

Page 18: Productivity measurement of agile teams (IWSM 2015)

|

Advantages / disadvantages

Advantages Reduced influence of non-functional backlog items; The use of an ISO/IEC FSM standard – ability to benchmark.

Disadvantages Depending on accurate story point assignment (subjective); Possible for the team to tweak the performance figures;

As the product owner is present, this risk is considered to be small;

Impossible to measure ANS when the functional size delivered is 0.

Performance measurement of agile teams 18

Page 19: Productivity measurement of agile teams (IWSM 2015)

|

Productivity measurement

Sprint Size (FP) ANS (nFP) Hours Hours/FP Hours/nFP

16 20 27,5 500 25,0 18,2

17 25 39,3 480 19,2 12,2

18 18 33,0 530 29,4 16,1

19 29 32,3 468 16,1 14,5

20 4 28,0 534 133,5 19,1

21 15 37,5 522 34,8 13,9

19Performance measurement of agile teams

Page 20: Productivity measurement of agile teams (IWSM 2015)

|

The effect in multiple sprints

20Performance measurement of agile teams

Page 21: Productivity measurement of agile teams (IWSM 2015)

|

The example

Sprint Size (FP) Functional SP Non funct. SP Story Points ANS (nFP)

16 20 32 12 44 27,5

17 25 28 16 44 39,3

18 18 24 20 44 33,0

19 29 35 4 39 32,3

20 4 6 36 42 28,0

21 15 16 24 40 37,5

22 0 0 41 41 n.t.b.

23 18 24 20 44 33,0

21Performance measurement of agile teams

Page 22: Productivity measurement of agile teams (IWSM 2015)

|

Sprint 22: no productivity measurement

Sprint Size (FP) ANS (nFP) Hours Hours/FP Hours/nFP

16 20 27,5 500 25,0 18,2

17 25 39,3 480 19,2 12,2

18 18 33,0 530 29,4 16,1

19 29 32,3 468 16,1 14,5

20 4 28,0 534 133,5 19,1

21 15 37,5 522 34,8 13,9

22 0 N/A 512 N/A N/A

23 18 33,0 508 28,2 15,4

22Performance measurement of agile teams

Page 23: Productivity measurement of agile teams (IWSM 2015)

|

Issue: completely non-functional sprints

In sprint 22, zero function points were delivered; Size in FP is 0, ANS impossible to determine (dividing by zero); Impossible to determine productivity.

Solution: progressive approach.

23Performance measurement of agile teams

Page 24: Productivity measurement of agile teams (IWSM 2015)

|

Progressive approach

Size measurement and productivity measurement not per sprint, but until the last sprint;

Does not focus on sprint, but on overall performance

(∑1-n functional size / ∑1-n functional story points)

* ∑1-n total story points

24Performance measurement of agile teams

Page 25: Productivity measurement of agile teams (IWSM 2015)

|

Progressive approach

Sprint Size (FP) ANS (nFP) Hours Hours

(cumulative)

ANS

Progressive

Hours (cum) /

nFP (prog)

16 20 27,5 500 500 27,5 18,2

17 25 39,3 480 980 66,0 14,8

18 18 33,0 530 1.510 99,0 15,3

19 29 32,3 468 1.978 132,2 15,0

20 4 28,0 534 2.512 163,6 15,4

21 15 37,5 522 3.034 199,2 15,2

22 0 N/A 512 3.546 231,4 15,3

23 18 33,0 508 4.054 264,3 15,3

25Performance measurement of agile teams

Page 26: Productivity measurement of agile teams (IWSM 2015)

|

Difference between the methods

Performance measurement of agile teams 26

Page 27: Productivity measurement of agile teams (IWSM 2015)

|

Starting points

DocumentationAfter each sprint the functional documentation should be made up-to-date and it must be clear:

Which functionality was added in the sprint;

Which functionality was changed in the sprint and in which way;

Which functionality was deleted in the sprint;

This should be part of the definition of done.

Effort administration The effort hours need to be booked in the effort administration in such a

way that it is possible to clearly identify the effort hours in scope and out of scope of the performance measurement.

Performance measurement of agile teams 27

Page 28: Productivity measurement of agile teams (IWSM 2015)

|

Conclusions and recommendations

The productivity of an agile team in a contract can be measured and benchmarked while taking into account the effect of non-functional requirements;

The customer now understands that non-functional backlog items have impact on the PDR when using only Nesma/IFPUG function points in agile projects when measuring on a sprint level. Customer is able to explain that internally and politically. Pressure is less now, because targets are met.

The method can help other organizations as well!

Performance measurement of agile teams 28