SIGDOC 2011 - Necessary and Neglected? An Empirical Study of Internal Documentation in Agile...

Leiden University. The university to discover. Leiden University. The university to discover.

Christoph J. Stettina ([email protected]) Werner Heijstek ([email protected])

Necessary and Neglected? Empirical Study of Internal Documenta?on in Agile SoAware Development Teams SIGDOC 2011, Pisa, Italy

This research has been kindly supported by the Leiden University Fund

Leiden University. The university to discover.

Introduc?on Agile Knowledge Transfer

•  Adap&ve rather then predic&ve: No heavy documenta&on created up-‐front

•  Direct communica&on rather than documenta&on •  Lean: Documenta&on → “No value for the end user” ?

All fine, but:

l  Project handover and maintenance l  Loss of undocumented knowledge (Abrahamsson et al, 2003)

l  LiDle empirical data (Dybå and Dingsøyr, 2008)


Objec?ves

Percep?ons from within teams: How do prac&&oners feel about documenta&on?

Research Ques?ons

1.  How do team members in agile soIware development projects document their work?

2.  How do they perceive the amount and importance of their internal documenta&on?


Methodology

Qualita?ve ScrumMaster interviews: Project and team environment

Quan?ta?ve ques?onnaire: Comparable Likert scale data on:

A)  Documenta&on: Perceived amount, effort, importance

B)  SoIware Tools: Usage, usability and importance (issue tracking, revision control, electronic discussion, Scrum support, document management and calendar & scheduling)


Methodology: Ques?onnaire Design

Percep?ons Regarding Documenta?on -‐ Perceived amount, effort and importance 1. How much +me do you spend on wri+ng documenta+on daily? 2. How do you feel about documenta+on at work? 3. How effec+ve do you consider finding internal documenta+on? 4. How important do you consider documenta+on for your project?

5. How important do you consider physical ar+facts like story cards or “the wall” for your project?


Methodology: Data Collec?on

Par?cipant and team iden?fica?on: l  SNS, Google Groups, SlideShare, Flickr, etc. l  Ac+vely involved in Scrum at collec+on +me


Roles

Data: 79 individuals, 13 countries, 8 teams

Experience (in years) Country


Data: Team Sample

T1 (UK) MMO Game back-‐end T2 (US) Collabora&ve SW for construc&on T3 (UK) Digital media agency T4 (NO) Smart Card key solu&ons T5 (NL) Corporate sites and web shops T6 (SE) News guide, community website T7 (IN) E-‐commerce T8 (NZ) State insurance company → Broad mul&na&onal sample


Results: Too li^le documenta?on?

"Code comments are used in an effec+ve manner. During project development any needed documenta+on is generally available. However, finding documenta+on for older projects is not always easy, and some+mes this documenta+on is missing.”

-‐-‐ Team member T6


Results: Too li^le documenta?on?

How important do you consider documentation for your project?

Amount: Distribu?on towards too li^le Importance: Important or very important

5 35 355Effort: Majority 15 mins or less daily Effec?veness: Normal distribu?on


Results: Tools as a “Backchannel”?

Adapta+on of support tools ”[W]e have a wiki that we are supposed to use”

-‐-‐ ScrumMaster T6 Virtual Teams & GSD "We have good experience using physical ar+facts for

local projects, but most of our projects are mul7 loca7on and require an electronic solu7on.”

-‐-‐ ScrumMaster T4


Results: Tools as a “Backchannel”?

!"#$%&'&$()&*+,-,$.$!"#"#$/*01&*$2-34

!"#$%&'&()*+%,-.&%/"%0

!"#$%&'(&")**'+,-.%&"'(,/#+%

563$7&)8-)9$0:$&;;*-3<$;&=8&93,$;>',$ '63$,0*>'-0),$:07$?3@-,-0)$A0)'70* $&)<$B,,>3$57&=8-)9$-)$

:3&'>73<$;0,-'-0),"$56-,$;*&=3C3)'$-,$3D;3='3<$:07$&'$*3&,'$'63$:-7,'$=&'3907+$&,$-'$-,$<-73='*+$-)@0*@3<$

4-'6-)$'63$;70=3,,$0:$=73&'-)9$,0:'4&73$13-)9$<3$:&='0$'63$,'07&93$:07$'63$6-963,'$900<$0:$&$,0:'4&73$

=0C;&)+$E13,-<3,$'63-7$3C;*0+33,F$8)04*3<93G"$B,,>3$57&=8-)9H$7&)83<$,3=0)<$4-'6$&*C0,'$IJKH$

<3;-=',$'63$,0*>'-0)$0:$&)0'637$-C;07'&)'$,0:'4&73$3)9-)337-)9$900<$)&C3*+$1>9,H$07$13''37H$'63$*-,'$

3)=*0,-)9$'63C$&)<$'6373:073$&$833;37$0:$'63$,0:'4&73$L>&*-'+"$56-,$C3&),$'6&'$'63$;70<>=3<$,0>7=3$

=0<3$&)<$'63$<-,=0@373<$<3:3=',$&73$13-)9$,>;;07'3<$1+$&>'0C&'3<$,0*>'-0),"$

B)'373,'-)9$&,$'63$&>'607$:-)<,H$6043@37H$-,$'63$:&='$'6&'$&$=&'3907+$&79>&1*+$)0'$*3,,$-C;07'&)'H$-,$

13-)9 $ *3&,' $ &;;*-3< $4-'6-) $ ,0:'4&73 $ 3)9-)337-)9 $ '3&C," $%0=>C3)' $M&)&93C3)' $ ,=073, $ '63 $ *&,'$

;0,-'-0)$136-)<$N=7>C$N>;;07'$&)<$,0*>'-0),$:07$O*3='70)-=$%-,=>,,-0)"$563$=&'3907+$3)=*0,3,$'63$

,'07&93 $0: $ 73L>-73C3)',H $<3,-9) $;&;37,H $>,3< $ -)'37:&=3,H $ =73&'3< $ :>)='-0), $&)< $ ,>1.:>)='-0), $ -)$

739>*&7$,0:'4&73$;70P3=',"

Q)3$3D;*&)&'-0)$'0$'63$*04$;37=3-@3<$-C;07'&)=3$0:$<0=>C3)'$C&)&93C3)'$=0>*<$13$'63$3C;6&,-,$

0:$&9-*3$C3'60<0*09-3,$EN=64&137H$#RRST$N=64&137H$UJJSG$0)$<-73='$=0CC>)-=&'-0)$,3''-)9$47-''3)$

<0=>C3)'&'-0) $ -)'0 $ '63 $1&=8970>)< $ &)< $C&8-)9 $ -' $ 73<>)<&)'H $ &) $ -<3& $ ,>;;07'3< $1+ $ '63 $(9-*3$

M&)-:3,'0$E=0C;&73$U"#G"$()0'637$3D;*&)&'-0)$=0>*<$13$&$;0,,-1*3$&)'-;&'6+$0:$3)9-)337,$'04&7<,$

&='-@-'-3,$)0' $<-73='*+ $ *-)83<$'0$ '63$=73&'-@3$;70=3,,$0:$47-'-)9$=0<3$E&,$C&)+$3)9-)337,$ -)<33<$

;73:37$47-'-)9$=0<3$-),'3&<$0:$<7&:'-)9$'63-7$-<3&,$-)$'3D'G"$($=0C1-)&'-0)$0:$'63,3$'40$:&='07,$=0>*<$

'63)$*3&<$-)'0$&$)39*-93)=3$0:$<0=>C3)'&'-0)$:&7$C073$'6&)$-)'3)<3<$1+$'63$C3'60<0*09-3,"$

B)'373,'-)9$:07$ '63$&>'607$-,$&*,0$'63$;37=3-@3<$-C;07'&)=3$&)<$&;;*-=&'-0)$0:$;7097&C,$<-73='*+$

&-C-)9$&' $N=7>C$,>;;07'$,=07-)9$,3=0)<$4-'6$0)*+$?3@-,-0)$A0)'70* $13-)9$;37=3-@3<$0:$6-9637$

-C;07'&)=3" $ %>3 $ '0 $ '63 $ ,&C3 $ 73&,0), $ &, $ <-,=>,,3< $ &10@3H $ )&C3*+ $ '63 $ 3C;6&,-, $ 0) $ <-73='$

=0CC>)-=&'-0)$&)<$;6+,-=&*$&7'-:&=',H$'63$&>'607$3D;3='3<$&$7&'637$*04$=-7=>*&'-0)$0:$,>;;07'-@3$

'00*, $ &C0)9 $N=7>C $;7&='-'-0)37,H $ 07 $ &' $ *3&,'H $ -', $ >,&93 $ 0)*+ $ &C0)9 $N=7>CM&,'37, $ '0 $ ,>;;07'$

RV

!""#$%&'()*+,-

.$/+"+0,%10,2'03

43$)2'0,+)%5+")#""+0,

6)'#7%6#880'2

50)#7$,2%9(,(-$7$,2

1(3$,:('%;%6)<$:#3+,-

=>==? @=>==? A==>==?

BC?

BD?

DE?

DB?

CE?

FG?

60H2I('$%J"(-$

$

!""#$%&'()*+,-

.$/+"+0,%10,2'03

43$)2'0,+)%5+")#""+0,

6)'#7%6#880'2

50)#7$,2%9(,(-$7$,2

1(3$,:('%;%6)<$:#3+,-

= G D F B A= AG AD

D>B

@>D

@>CG

D>KE

D>@D

D>KD

@>EE

F>CB

@>BG

@>BA

@>AB

@>DC

60H2I('$%J"(L+3+2M

!780'2(,)$

N(8

J"(L+3+2M

$

!""#$%&'%()*+,-.+/)01'&($)*+)2+#$34+$)2%5'&3+5(%6+(%$+13&73(834+#$'9("(%:+'*4+93"(3834+(01)&%'*73

83%

37%

Document Management: Least usage and believed usability

Scrum Support: Wide applica?on of soAware tools to support the process


Discussion

Documenta?on l  Confirm lack of undocumented knowledge l  Majority spends < 15 mins on wri&ng documenta&on daily

l  Believe documenta&on is important and too liDle

SoAware l  Pure availability of support tools not enough l  Global SoIware Development, Virtual Teams l  Traceability of decisions valued


Conclusion ?

What is an appropriate balance of explicit and tacit knowledge in agile soXware development projects?


Conclusion → Future Work Expecta+ons & Sa+sfac+on in agile seZngs → Who needs what documenta&on?

Process Alignment and Cost/Quality balance → Effects of documen&ng itera&vely

SoXware support and codifica+on of informa+on → How to code informa&on in wikis and issue trackers?

Visual methods → Collabora&ve, Agile Modeling, ICONIX, ADSD


Ques?ons?

Thank you for your aDen&on! [email protected]


Validity

Validity Considera?ons

l  Consistency of data → Likert scales l  Low amount of data → Team agreement l  Socially Desirable Responding → Anonymity


Results: Team Sample Table 2: Descriptive variables, team results (x) and agreement (σ2

)

T1 T2 T3 T4 T5 T6 T7 T8 avg.agr.

country UK US UK NO NL SE IN NZ

team size (pers.) 4 9 5 12 6 4 8 6

collected answers 4 6 5 6 5 3 8 4

avg. exp. (yrs.) 7.75 13.7 6.6 12.7 2.6 10 7 3.5

spacial distribution co-loc. co-loc. co-loc. distrib. co-loc. co-loc. distrib. co-loc.

documentation tool Wiki Con-

fluence

Wiki

Google

Docs

- - Wiki Con-

fluence

Wiki

-

perceived doc. x -0.25 -0.50 -0.40 -1.30 -1.00 -0.75 -0.13 0amount σ2 (0.19) (0.25) (1.44) (0.89) (0.40) (0.67) (0.61) (0) (.56)

perceived eff.. x 0.65 0.76 0.44 0.60 0.52 0.50 0.75 0.45finding doc σ2 (0.69) (0.47) (0.16) (1.33) (1.44) (0.89) (0.69) (0.69) (.80)

perceived x 1.00 0.70 0.76 0.57 0.72 0.75 0.7 0.85importance artif. σ2 (0) (2.25) (0.16) (0.47) (1.04) (0.67) (0.50) (0.69) (.72)

averageagreement σ2 (.29) (.99) (.59) (.90) (.96) (.74) (.60) (.46) (.69)

6.2 Software, More Than a Backchannel

The teams in our study predominantly adopt collabora-

tion tools to document and share agile artifacts such as user

stories or sprint backlogs. An interesting finding is the per-

ceived importance and application of software that directly

aim to support Scrum. This is surprising to that extend that

one could expect the sufficiency of direct communication and

physical artifacts. Convenient handling of agile artifacts and

distributed settings seems to be one reason here. “We havegood experience using physical artifacts for local projects, butmost of our projects are multi location and require an elec-tronic solution.”, says the ScrumMaster of Team T4.

The perceived usability of solutions for electronic discus-

sion showed the smallest gap among all categories, meaning

that the participants find the current solutions very usable.

According to comments from team members, the solutions

surpass the expectations of a pure“backchannel” (Team T3).

The growing instant messaging culture seems to make a con-

tribution here and Skype has been the most named tool

in this category. A quote from the ScrumMaster of Team

T7: “Communication with the team and our client workedvery well when we decided to move away from ”voice” con-versations (accents, network latencies, time wasted settingup conference phones) to text chats. Even though they cantake substantially longer, logs are permanent and we foundit easier to share documentation, make decisions and stickto them.”.

7. CONCLUSIONS

In this paper we present the results of an empirical study

on documentation in agile Scrum software development teams.

Executed in a multidimensional manner we analyze the col-

lected data sets consisting of quantitative team and global

samples as well as qualitative interview questions. Our find-

ings stem from a representative data set of agile practitioners

and international teams and warrant further study.

One of our main findings is that documentation alone is

insufficient. While agile methods recommend to make“lean”

documentation, suggesting that documentation should only

include information that is used, we found that agile soft-

ware development practitioners perceive their internal doc-

umentation as important but that they feel that too little

documentation is available. Analogously to the observations

of Clear [3], we found that documentation is rather seen as

a burden than a co-created (core) artifact and found sup-

port to the perceptions in literature that without ensuring a

proper documentation process agile methods can cause ma-

jor knowledge loss during or after system development [1,17].

We found that agile teams adopt collaboration tools to

share and work on agile artifacts and that Scrum dedicated

software is perceived as important and helpful to support the

method. We found that instant messaging is perceived as a

helpful “backchannel” and supports documenting decisions.

We can conclude that integration of software tools into the

process is crucial. Lightweight solutions such as Wikis or

Google Docs are prominent, their adoption however needs

further research (compare Tab. 2).

When interviewing practitioners the authors often found

abroad interest for agile methods. Discussing their tools

and routines it was the first time developers would address

a software development process with passion. This, however,

seems not yet true for agile documentation, and future re-

search needs to address an appropriate incorporation within

the process to make knowledge transfer truly agile and suf-

ficient.

7.1 Validity Considerations

Due to the low amount of data sets containing 79 indi-

viduals conclusions were drawn carefully. As we base our

evidence on small team data sets, we have improved the

transparency of data by adding the variance of given answers

among the team members. Throughout the whole process

of data collection, we encouraged participants to give real-

istic answers and emphasized the anonymous treatment of

data to establish a reasonable level of trust and to reduce

bias. No results other than the processed outcome for the

whole team would be distributed or given to their superiors.


References Abrahamsson, P. Warsta, J., Siponen, M. T. and Ronkainen, J. (2003) New direc&ons on agile methods: a compara&ve analysis. In Proceedings of the 25th Interna&onal Conference on SoIware Engineering, ICSE 2003, pages 244–254, Washington, DC, USA, 2003. IEEE Computer Society. Clear, T. (2003) Documenta&on and agile methods: striking a balance. SIGCSE Bulle&n, 35(2):12–13 Dyba, T., Moe, N. B. (1999) Rethinking the concept of soIware process assessment. In Proceedings of European SoIware Process Improvement Conference (EuroSPI 1999), Pori, Finland Dyba, T. and Dingsøyr, T. (2008) Empirical studies of agile soIware development: A systema&c review. Informa&on SoIware Technology, 50(9-‐10):833–859, 2008. Fægri, T.E., Dyb˚a, T., Dingsøyr, T.: Introducing knowledge redundancy prac&ce in soIware development: Experiences with job rota&on in support work. Inf. SoIw. Technol. 52, 1118–1132 (2010) Rubin, E. and Rubin, H. (2011) Suppor&ng agile soIware development through ac&ve documenta&on. Requirements Engineering, 16:117–132

SIGDOC 2011 - Necessary and Neglected? An Empirical Study of Internal Documentation in Agile...

Education

Transcript of SIGDOC 2011 - Necessary and Neglected? An Empirical Study of Internal Documentation in Agile...