[ICSE2014 SEET]Quantitative Assessment with Using Ticket Driven Development for Teaching Scrum...

Cloud Specialist Program Initiative for Reality-based Advanced Learning

Education Network for Practical Information Technologies

◯Hiroshi Igaki

Naoki Fukuyasu

Sachio Saiki

Shinsuke Matsumoto

Shinji Kusumoto

Quantitative Assessment with Using Ticket Driven Development for Teaching Scrum Framework

Hiroshi Igaki (Osaka University, Japan)

@hirocell

https://www.facebook.com/hiroshi.igaki


Our course consists of 10 days lessons and 5days PBL

© Cloud Spiral partners 2013 2

Course Setup

10days (Each day has 6 hour @Osaka)

PBL (5days)

Scrum Facilitation Skill

Mongodb Java JavaScript Application Server

Continuous Integration

Version Control System

April July August

Reviewing Testing

Specification Reading

Web Application Framework (DWR)



Scrum Framework Sprint Planning

Product Backlog Sprint Backlog

Development (w/ Daily Scrum)

Scrum Master (is not a manager as a servant leader)

Development Team

Increment

Product Owner

Customer

Sprint Review

Sprint Retrospective

Product Owner

Sprint(1~4W)

Development Team

Transparency

Inspection

Adaptation

Inspection

Adaptation


Developers must follow the process with several associated rules

© Cloud Spiral partners 2013

Disregarded Processes and Rules

4

Specifications

Review Report Done

Debugging

Reviewing

Source Code

OK

NG Coding

Unit Testing Unit Test Report

Coding rules

Dev Plan

Unit Test Plan

Review checklist

Every component should be tested and reviewed



Disregarded process

5

Hi, Teacher!! We have succeeded to save development time We have omitted unit testing and reviewing

Specifications

Review Report Done

Debugging

Reviewing

Source Code

OK

NG Coding

Unit Testing Unit Test Report

Coding rules

Dev Plan

Unit Test Plan

Review checklist


In short term project, it sometimes happens to work terrific if some important rules are disregarded by students.


Disregarded process

6

Hi, Teacher!! We have succeeded to save development time We have omitted unit testing and reviewing

Specifications

Done

Debugging Source Code

Coding Coding rules

Dev Plan

Unit Test Plan

Review checklist


In Scrum project, members pull tasks which the person will do by themselves

Students tend to select tasks which they can do already respectively


Inequality in Learning Opportunities

coding

review

coding unit test

review

integration test presentation


Such inequality on task assignment may bring projects to a successful conclusion

Such inequality may demotivate team members

Such inequality also limits each student to learn new things, and to share their knowledge


Inequality in Learning Opportunities

coding

review

coding

unit test review

integration test presentation

In order to improve team performance, someone except

me should do these tasks


We placed our PBL as a first step to learn Scrum for every student

In our Scrum project, we have decided to expect every student to experience processes and rules required in team software

development correctly

various kinds of tasks which should be conducted in the project


Our PBL Objectives


Students form a team with 5~6 members

In 2013, total number of students is 49 (from 9 universities)

Every student is in master’s course

Students have 5 days for development


PBL Setup

Sprint #1 (Tutorial)

Sprint #2 Sprint #3 Sprint #4 Sprint #5

26th July 19th Aug. 20th Aug. 21th Aug. 22th Aug.


Every team develops same Web application “Ticket Purchasing system”

All specifications are given Students can select use cases (user stories) which they

develop

Development rules are given

Project Assessment Criteria are given Assessing Quality, Assignment and Delivery


Project Input

Spec.s

Review Report Done

Debugging

Review

Source Code

OK

NG Implement

Unit Test Unit Test Report

Coding rules

Dev Plan

Unit Test Plan

Review checklist

Display Bought Ticket List

Display Event List

Login

Logout

Buy Ticket

Signup

Register Event

Display Event Detail

Signup Promoter

Display Registered Event List

Cancel Ticket Display Bought

Ticket List

Register Seat Category


Every team develops same Web application “Ticket Purchasing system”

All specifications are given Students can select use cases (user stories) which they

develop

Development rules are given

Project Assessment Criteria are given Assessing Quality, Assignment and Delivery


Project Input

Spec.s

Review Report Done

Debugging

Review

Source Code

OK

NG Implement

Unit Test Unit Test Report

Coding rules

Dev Plan

Unit Test Plan

Review checklist


写真を何枚か貼る




Typical Development Timeline Sprint Retrospective (17:30~19:00)

Drinking (19:30-24:00)

Sprint(1Day)

Teacher Feedback

Development (10:30~17:00)

Scrum Master

Development Team

Increment

Product Owner

Teacher

Sprint Review (17:00~17:30)

Development Team

Sprint Planning(9:00~10:30)


Product Owner


In TiDD, every task is recorded as a ticket.

Componentass 種類：実装着手時刻：13:00 終了時刻：14:00 見積時間：0.5H 総時間：1.0H 担当者：受講生X スプリント：UC[Login]




Component： DisplayEventList.java

type：Unit Testing Start： Complete： estimated：0.5H Total hours： Owner： Sprint：#1


Introduction of Ticket Driven Development(TiDD) to Sprint Planning Meeting and Development

15

Sprint Planning(9:00-10:30)


Display Event List coding showEventList.html

review showEventList.html

coding DisplayEventList.java

unit test DisplayEventList.java

review DisplayEventList.java

integration test Display Event List


Each member updates his tickets according to its progress of the corresponding task


Development Phase

16

Development (10:30~17:00)

Student A

Increment






type：Unit Testing Start：11:30 Complete：12:30 estimated：0.5H Total hours：1H Owner：Student A Sprint：#1

coding showEventList.html

review showEventList.html

coding DisplayEventList.java

unit test DisplayEventList.java

review DisplayEventList.java

integration test Display Event List

Such ticket data enables us to visualize when and who conducted what kinds of tasks.


We defined, Quality, Assignment, and Delivery metrics

These metrics are exhibited to students beforehand

Students can check most of the values of metrics by themselves during each Sprint


Project Assessment Criteria


Assessing process quality by checking how correctly developers followed rules

QSB, QCT, QRV, QIT


Quality Metrics

𝑄𝑆𝐵 =# 𝑜𝑓 𝑡𝑖𝑚𝑒𝑠 𝑤ℎ𝑖𝑐ℎ 𝑡ℎ𝑒 𝑠𝑜𝑢𝑟𝑐𝑒 𝑐𝑜𝑑𝑒 𝑐𝑎𝑛 𝑏𝑒 𝑏𝑢𝑖𝑙𝑡 𝑎𝑓𝑡𝑒𝑟 𝑐𝑜𝑚𝑚𝑖𝑡

# 𝑜𝑓 𝑇𝑜𝑡𝑎𝑙 𝑐𝑜𝑚𝑚𝑖𝑡𝑠

𝑄𝐶𝑇 =# 𝑜𝑓 𝑐𝑜𝑚𝑚𝑖𝑡𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑖𝑛𝑔 𝑡𝑖𝑐𝑘𝑒𝑡𝑠

# 𝑜𝑓 𝑇𝑜𝑡𝑎𝑙 𝐶𝑜𝑚𝑝𝑜𝑛𝑒𝑛𝑡𝑠


Every developer must check whether the person’s code is buildable or not when the person commits

QSB

19 †Martin Fowler, “Continuous Integration,” http://www.martinfowler.com/articles/originalContinuousIntegration.html ,Sep. 2000.

Student A

Coding


Student B

Unit Testing

Student C

Reviewing

Student D

Coding

Commit Checkout Checkout

=# 𝑜𝑓 𝑡𝑖𝑚𝑒𝑠 𝑤ℎ𝑖𝑐ℎ 𝑡ℎ𝑒 𝑠𝑜𝑢𝑟𝑐𝑒 𝑐𝑜𝑑𝑒 𝑐𝑎𝑛 𝑏𝑒 𝑏𝑢𝑖𝑙𝑡 𝑎𝑓𝑡𝑒𝑟 𝑐𝑜𝑚𝑚𝑖𝑡

# 𝑜𝑓 𝑇𝑜𝑡𝑎𝑙 𝑐𝑜𝑚𝑚𝑖𝑡𝑠

8

10

http://www.martinfowler.com/articles/originalContinuousIntegration.html

http://www.martinfowler.com/articles/originalContinuousIntegration.html


In T4’s case, they did not find that the source code in the repository could not be built for a while

Finally, in the #5, 7 teams have achieved 100%


Result of QSB

0%

20%

40%

60%

80%

100%

#1 #2 #3 #4 #5

T1

T2

T3

T4

T5

T6

T7

T8

T9

T4


Developers must commit when they completed each task

All tasks must be recorded as tickets

If a new task is added, a ticket must be added simultaneously


QCT =# 𝑜𝑓 𝑐𝑜𝑚𝑚𝑖𝑡𝑠 𝑤ℎ𝑖𝑐ℎ ℎ𝑎𝑣𝑒 𝑐𝑜𝑟𝑟𝑒𝑠𝑝𝑜𝑛𝑑𝑖𝑛𝑔 𝑡𝑖𝑐𝑘𝑒𝑡𝑠

# 𝑜𝑓 𝑇𝑜𝑡𝑎𝑙 𝐶𝑜𝑚𝑚𝑖𝑡𝑠


type：Coding Start：10:45 Complete：11:45 estimated：0.5H Total hours：1H Owner：Student A Sprint：#1


DisplayEventList.java

EventDetail.java

Commit(11:00)

Commit(11:40)

1

2

Student A


6 teams achieved over 80% in every Sprint


Result of QCT

0%

20%

40%

60%

80%

100%

#1 #2 #3 #4 #5

T1

T2

T3

T4

T5

T6

T7

T8

T9

T3

T7


Bad/Good Result of QCT T3

In T3, a member forgot to update most of tickets in #2,3

In T7, most of members updated their tickets after they completed their tasks

There were a few students who tend to forget updating tickets in almost all teams

T7


0%

20%

40%

60%

80%

100%

#1 #2 #3 #4 #5

M31

M32

M33

M34

M35

M36

0%

20%

40%

60%

80%

100%

#1 #2 #3 #4 #5

M71

M72

M73

M74

M75


Assignment metrics check equality on task assignment for each task type

ACOD, AWUT, ARV, AWIT, AEIT


Assignment Metrics

Coding showEventList.java

Coding showEvent.html

Coding DisplayEventList.java

Coding EventList.java

Coding showEventList.js

Coding showEventList.html

Coding Event.java

Coding showEventDetail.html

Coding EventDao.java




Coding showEvent.js

Coding EventDetail.java

Coding DisplayEventController.java

Coding AllEventList.java

Coding EventList.java

Coding Event.java

Student A Student B Student C


Assignment criterion has several metrics corresponding to the task type

ACOD, AWUT, ARV, AWIT, AEIT


Assignment Metrics

Coding Event.java

Total number of tasks = 18 Average number of tasks per member = 18/3 = 6 Is assigned tasks for each member in the range of 6 ± 20% ? (4.8~7.2)


Coding showEventList.html Coding

DisplayEventList.java Coding DisplayEventList.java


Coding DisplayEventList.java Coding

DisplayEventList.java Coding showEventList.html

Coding Event.java Coding

Event.java Coding DisplayEventList.java

1

3

Coding DisplayEventList.java Coding

showEventList.html Coding DisplayEventList.java Coding

DisplayEventList.java Coding

DisplayEventList.java Coding

DisplayEventList.java

6 8 4 Student B Student A Student C


All assignment metrics are calculated based on the accumulated number of assigned tasks.


Result of ACOD

0%

20%

40%

60%

80%

100%

#1 #2 #3 #4 #5

T1

T2

T3

T4

T5

T6

T7

T8

T9

T3

T7


Bad/Good Result of ACOD T7

In T7, a student conducted all coding tasks in #4 to increase the number of assigned tasks

In T3, students have tried to avoid the widening of values between max and min

T3


0

2

4

6

8

10

#1 #2 #3 #4 #5

M71

M72

M73

M74

M75 0

2

4

6

8

10

#1 #2 #3 #4 #5

M31

M32

M33

M34

M35

M36


Bad/Good Result of ACOD T7

Comparing this metrics between these 2 teams, T3 is better

In the T7, there is no free rider on coding tasks

T3


0

2

4

6

8

10

#1 #2 #3 #4 #5

M71

M72

M73

M74

M75 0

2

4

6

8

10

#1 #2 #3 #4 #5

M31

M32

M33

M34

M35

M36


About the assignment metrics, we have obtained feedback directly from 4 students after our PBL


Positive comments on assignment metrics

• I did not almost have experience about not only JavaScript but Java. However, the constraints on assignment motivated me to learn both of them

• The assignment constraints promoted our team to help each other. As a result, I could have experiences on various tasks.


Assessing how many user stories are developed by every team(DNOU)

Assessing predictability about the number of user stories which the team will develop(DPOA)


Delivery Metrics

Team DNOU DPOA

T1 7 7/7 = 100% T2 5 5/6 = 83% T3 5 5/9 = 56% T4 4 4/5 = 80% T5 4 4/5 = 80% T6 4 4/7 = 57% T7 3 3/5 = 60% T8 3 3/5 = 60% T9 3 3/6 = 50%


QSB QCT QRV QIT ACOD AWUT ARV AWIT AEIT DNOV DPOA

T1 100% 92% 84% 75% 60% 40% 40% 100% 100% 7 100%

T2 95% 82% 83% 100% 100% 80% 100% 100% 100% 5 83%

T3 95% 86% 60% 63% 100% 67% 100% 67% 83% 5 56%

T4 87% 84% 88% 83% 100% 100% 100% 100% 100% 4 80%

T5 96% 79% 87% 83% 100% 40% 80% 100% 100% 4 80%

T6 92% 82% 81% 50% 83% 100% 100% 100% 100% 4 57%

T7 96% 90% 76% 100% 40% 40% 40% 100% 100% 3 60%

T8 98% 80% 50% 40% 83% 67% 100% 67% 100% 3 60%

T9 93% 91% 72% 57% 83% 33% 67% 100% 83% 3 50% 31

All Results of QAD after All Sprints In the view point of delivery, T1 has achieved

highest score the score of the assignment metrics of T1 is low



All Results of QAD after All Sprints In the view point of the assignment metrics,

T2, T4, and T6 achieved higher score Quality metrics values of T2 and T4 are also high

QSB QCT QRV QIT ACOD AWUT ARV AWIT AEIT DNOV DPOA

T1 100% 92% 84% 75% 60% 40% 40% 100% 100% 7 100%

T2 95% 82% 83% 100% 100% 80% 100% 100% 100% 5 83%

T3 95% 86% 60% 63% 100% 67% 100% 67% 83% 5 56%

T4 87% 84% 88% 83% 100% 100% 100% 100% 100% 4 80%

T5 96% 79% 87% 83% 100% 40% 80% 100% 100% 4 80%

T6 92% 82% 81% 50% 83% 100% 100% 100% 100% 4 57%

T7 96% 90% 76% 100% 40% 40% 40% 100% 100% 3 60%

T8 98% 80% 50% 40% 83% 67% 100% 67% 100% 3 60%

T9 93% 91% 72% 57% 83% 33% 67% 100% 83% 3 50%



Conclusion

[ICSE2014 SEET]Quantitative Assessment with Using Ticket Driven Development for Teaching Scrum...

Education

Transcript of [ICSE2014 SEET]Quantitative Assessment with Using Ticket Driven Development for Teaching Scrum...