Tes t in g in A La r g e sc a l e
a g il e Dev el o pm en t En v ir o n m en t
Myth or reality?
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 2
1. Introduction 2. Agile on a large scale 3. Testing in a large scale agile environment 4. Conclusion
Co n t e n t /a g e n d a
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 3
› Ericsson: – The leading vendor for mobile network infrastructure
› Tough competition on the telecommunication market: – Number of competitors – Merging of traditional Circuit Switched Technology and Packet
Switched Technology towards all IP Network
› Motivation: – Increase the feedback loops towards design – Delivery of high quality products to the customer – Being flexible to adapt faster to the market’s demand – Early and Frequent Delivery
In t r o d u c t io n
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 4
AGILE ON A LARGE SCALE
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 5
› Manifesto for Agile Software Development › We are uncovering better ways of developing
software by doing it and helping others do it. Through this work we have come to value:
– Individuals and interactions over processes and tools – Working software over comprehensive documentation – Customer collaboration over contract negotiation – Responding to change over following a plan
› That is, while there is value in the items on the right, we value the items on the left more.
Ag il e
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 8
Sc RUM IN A NUTSHELL › One team collaborating to one feature
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 9
– Chief Product Owner: 1 – Number of Product owners: 5 – Number of Scrum teams: 8 – Size of the teams: 6-8 – Number of Scrum Masters: 8 – Number of people: around 70 – Number of locations: 4 – Number of Releases: 2 – Number of manhours: ~100000 – Number of User Stories: ~700
l a r g e s c a l e
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 10
› Our legacy: The Ericsson Mobile Soft Switch
› 8085042 PLEX statements › 20212606 ASA instructions corresponding to
approximately 95000000 Intel instructions › 2196 SW blocks
LARGE SCALE
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 11
TESTING IN A LARGE SCALE AGILE
ENVIRONMENT
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 12
› Cross functional team of 7 +/- 1 SW developers: › Virtual teams › Synchronized sprints inside a feature › Common Backlog › Common ceremonies to keep everybody informed
Wa y s o f w o r k in g
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 13
Wa y s o f Wo r k in g
Team A Team B Team C Team D
Backlog Setup strictly based on functional areas
Application
Transport
Internet
Network
US1
US3 US2
US4
US5
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 14
Wa y s o f Wo r k in g Backlog Setup strictly based on functional areas
Team A Team B Team C Team D
Application
Transport
Internet
Network
US1
US3 US2
US4
US5
› Risk of dependencies between User Stories
› Difficulties to test from an “end to end” perspective
› More planning needed › Prone to generate delays
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 15
Wa y s o f Wo r k in g
Team A Team B Team C Team D
Backlog Setup with end to end focus and functional areas
Application
Transport
Internet
Network
US1
US3
US2
US4
US5
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 16
Wa y s o f Wo r k in g Backlog Setup with end to end focus and functional areas
Team A Team B Team C Team D
Application
Transport
Internet
Network
US
1
US
3
US
2
US
4
US
5
› Testing possible from “end to end” perspective as the User Story is concluded
› Less planning needed › Reduce dependencies due
to functional areas › Competence build through
training on the job and mix of competences
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 17
Wa y s o f Wo r k in g
Team A Team B Team C Team D
Virtual Test Team
System Architecture Team
LSV team
Scrum of Scrum
Setup of virtual teams
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 18
Wa y s o f Wo r k in g
Team A Team B Team C Team D
Virtual Test Team
System Architecture Team
LSV team
Scrum of Scrum
Setup of virtual teams
› Overview over the activities with higher priority – Testing, Architecture, Software Delivery, Organisational topics
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 19 © Ericsson AB 2009 | Ericsson Internal | X (X) | Date
Te s t Ov e r v ie w
Rebase mandatory
Rebase optional Note: Simplified view, details on following slides
LSV
LLV
trac
k
Feature n
LLV
trac
k
Feature 3
LLV
trac
k
Feature 2
LLV
trac
k
Feature 1
1 2 3 4 5 6 7 8 N N+1 M
LSV Delivery
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 20
Te s t in g a c t iv it ie s
Functional Tests Examples & Story Tests
Unit Tests System Module Tests
Exploratory Testing Usability Testing
(User Acceptance Tests)
Performance & Load Testing Security Testing
“ility” Testing
Q2 Q1
Q3 Q4
Unit Test
Function Test Integration
Function Test Integration
Verification
Automated
Automated & Manual Manual
Tools Technology facing
Supp
ortin
g th
e Te
am
Critique Product
1: taken from Crispin/Gregory: Agile Testing, Addison Wesley, 2009
Business facing
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 21
› Q1 Kind of tests are – Unit tests – testing small pieces of code – System Module Tests – test that units work together correctly
› Q2 kind of tests are – Functional & Integration tests – positive and deterministic negative
tests, if possible to automate – User Story Acceptance Tests
Te s t in g a c t iv it ie s
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 22
› Q3 Kind of tests – Exploratory Testing – feedback into stories – Feature Test and Regression Test with
Background Traffic – (early) Characteristics
› Q4 Kind of tests – Dimensioned Load on LLV/LSV – High Load/Overload on LSV (most likely) – Parafunctional tests (Performance, stress) – Interoperability, Security
Te s t in g a c t iv it ie s
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 24
› Detailed view on Testing Timeframe
TEST De t a il e d v ie w
Main
Feature1
Feature2
Team A
Team B
Team C
Team D
LSV
0
0
0
0
0
0
0
0
1
1
1
2
2
3
3
1
1
2
2
3
1
1
4
2 4
2
1
1
1
2
2
3
1
1
4
2 4
2
1
Unit Test
Functional Test
Integration Test
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 25
Co n t in u o u s in t e g r a t io n
“…a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (incl. test) to detect its errors as quickly as possible…”
Martin Fowler
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 26
GEOFF
CI Wo r k f l o w
F2 F1 F4
FCS Sprint 1
Sprint 3
Sprint 2
Sprint n
CI Ser v er
…
FIP
AutoRT preparation: S2 TCs
AutoRT preparation: S3 TCs
AutoLT preparation: e2e feature TCs
AutoLT preparation: e2e feature TCs
Dump building
AutoLT execution (appropriate suite)
Dump building
AutoLT execution (appropriate suite)
Dump building
AutoLT execution (appropriate suite)
Dump building
AutoLT execution (appropriate suite)
AutoRT execution S1 TCs
AutoRT execution S1+S2 TCs
AutoRT execution S1+…+S(n-1) TCs
TF
AutoRT preparation: S1 TCs
M Ax Week
M Ax Week
M Ax Week
M Ax Week
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 28
Te s t TOOLs & En v ir o n m e n t
M-MGW
MSC-S Dual Blade 60
MSC-S Blade Cluster
RNC BSC
MME MME M-MGW
BSC RNC
HLR CSCF
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 29
› Unit Test: – Ericsson proprietary tool testing the interfaces
› Function Test: – Manual: using a simulated environment simulating all nodes but with
the possibility to load the real MSC software – Automated: using a proprietary tool with an open source framework
based on TTCN-3
› Regression Test: – Based on the automated test cases from the Function Test – Running on a Continuous Integration Server (Jenkins)
› Feature load (moderate/high/overload): – Proprietary tool based on TTCN-3 generating load on the real
environment
Te s t TOOLs & En v ir o n m e n t
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 30
› Coordination › Maintenance of a high reliability › Working high quality Legacy › Documentation › Test tool & environment
CONCLUSION/WRAP-UP
Challenges
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 31
› Eliminate waste › Early indication of Software Quality › Early corrective actions possible › Reduce test time enabling earlier release of product › Enabling of Continuous Integration as first step towards a
possible Continuous Delivery › Early customer involvement
Co n c l u s io n
Benefits
Testing in large scale Agile Development | Ericsson Internal | © Ericsson AB 2013 | 2013-01-28 | Page 32
› Manifesto for Agile Software Development › We are uncovering better ways of developing
software by doing it and helping others do it. Through this work we have come to value:
– Individuals and interactions over processes and tools – Working software over comprehensive documentation – Customer collaboration over contract negotiation – Responding to change over following a plan
› That is, while there is value in the items on the right, we value the items on the left more.
Co n c l u s io n
Top Related