Defining Quality Models for Agile Projects
Transcript of Defining Quality Models for Agile Projects
FP7-ICT-2011-8. Project #: 318082
Defining Quality Models for Agile Projects
FP7-ICT2011-8. Project #: 318082
Contents
• A few words about speakers’ biographies and companies• The U-QASAR Project • What to Measure in Agile• Practical Workshop: defining a Quality Model for one Agile
company
FP7-ICT2011-8. Project #: 318082
The U-QASAR Project (1)
• U-QASAR. An EU funded research project under FP7.
• 8 organizations from Finland, Norway, Germany, Luxembourg and Spain: Experts in SW Quality, Ontology modeling and implementation, SW Engineering methods, and SW Measurement.
FP7-ICT2011-8. Project #: 318082
The U-QASAR Project (2)
• Objective: Create a solution to provide project managers, software engineers, testers and other actors with an objective measurement of quality for:
– Software Development processes– Software Products
FP7-ICT2011-8. Project #: 318082
Methodology + Platform:
The U-QASAR Project (3)
FP7-ICT2011-8. Project #: 318082
“If you can not measure it, you can not improve it…”
Lord Kelvin
What to Measure in Agile (1)
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (2)
Defect count, test success rate, test failure rate technical debt in categories, technical debt in effort
velocity, completed web pages, schedule performance index, business value delivered,
customer satisfaction, defect count after testing, violations of static code analysis, build status…
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (3)
• What Metrics are Agile teams using in practice?:– Metrics proposed by Agile literature + Custom Metrics.– Measuring Products and Processes.– Reasons for and effects of using metrics:
• Iteration and Project Planning• Iteration and Project Progress Tracking• Software Quality Measurement• Fixing Software Process Problems• Motivating People
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (4)
Iteration and Project Planning• Why
– Prioritization of tasks– Scoping the Iteration/Release– Effort prediction
• Metrics– E.g Effort estimate, actual velocity of previous
iteration, team’s available hours
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (5)
Iteration and Project Progress Tracking• Why
– Monitoring progress– Increase Visibility for all stakeholders– Achieving goals (Are we going to make it?)
• Reduce scope, add more resources
– Balance workflow (prevent overloading people)
• Metrics– Velocity / Burndown, completed work items
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (6)
Understanding and Improving Quality• Why
– Understand and increase quality of the product– Prevent defects reaching customer– Predict and Evaluate post release quality– Ensure the level of testing
• Metrics– Technical debt, maintenance effort, defect deferred, N
of change requests, test coverage, critical defects sent by customer
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (7)
Fixing Software Process Problems• Why
– Identify and predict problems– Identify phases where no value is added
• Metrics– Lead time, processing time, cost types, story
percentage completed, fix times of failed build, violation of static code analysis, queue time, check-ins per day
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (8)
Motivating People• Why
– Motivate people to react faster and improve• E.g Fix build faster, fix bugs faster, increase unit testing
• Metrics– E.g Fix build faster, fix bugs faster, increase unit
testing, defect trend, work in progress
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (9)
Entities AttributesProducts Internal External
Products running tested features, build statuscustomer satisfaction, progress as working code
Test plans number of test cases
Codetechnical debt in categories , technical debt in effort, violations of static code analysis
Featurestask's expected end date, task done, effort estimate, story complete percentage
business value delivered
Requirements requirement's cost types, percentage of stories prepared for Iteration
Defectsdefect trend indicator, predicted number of defect
N. E. Fenton and S. L. Peeger. Software metrics: a rigorous and practicalapproach. PWS Publishing Co., 1998.
Categorisation of Metrics by Fenton and Pfleeger
FP7-ICT2011-8. Project #: 318082
What to Measure in Agile (10)
Entities AttributesProcesses Internal External
Testingdefect count, test success rate, test failure rate, defects deferred, test coverage, test growth ratio
number of bounce backs, fault slips
Implementation
velocity, number of unit tests, completed web pages, cost performance index, schedule performance index, planned velocity, common tempo time, check-ins per day, fix time of failed build
story flow percentage
Reqs.engineering velocity of elaborating features
WholeDevelopment
Cycle
cycle time, lead time, processing time, queuetime, maintenance effort, work in progress, variance in handovers, throughput, queue, implemented vs wasted requirements
ResourcesTeam team effectiveness
Customer revenue per customer
N. E. Fenton and S. L. Peeger. Software metrics: a rigorous and practicalapproach. PWS Publishing Co., 1998.
Categorisation of Metrics by Fenton and Pfleeger
FP7-ICT2011-8. Project #: 318082
Practical Workshop:
Defining a Quality Model for anAgile company
FP7-ICT2011-8. Project #: 318082
Defining a Quality Model (1)
• Create a Quality Model– It can be based on existing standards or models.– It can be an organization’s self-made model.– It can deal with process, products, or both.
Creation of a model consists of identifying expected values, weights and thresholds for:
• Quality Objectives, or aggregation of indicators to evaluate the achievement of an objective.
• Quality Indicators, or aggregation of metrics used to evaluate• Quality Metrics, or raw measurements obtained automatically or
manually
FP7-ICT2011-8. Project #: 318082
Defining a Quality Model (2)
• Create Quality Objectives (QOs) or Select existing ones.
• Select or create Quality Indicators (QIs) by aggregating metrics.
CONTEXT AWARENESS & NEGOTIATION• Weights, Thresholds and Values are negotiated among the
participants of the QA process• QOs’ and QIs’ are adapted along the development LC (e.g. a feature
is identified as critical for the customer in the 3rd Iteration).
FP7-ICT2011-8. Project #: 318082
Iteration 1
Iteration 2
Iteration N
Defining a Quality Model (3)
QM 1.0
QM 1.1
QM v.r
Adapted along the development LC (e.g. a feature is identified as critical for the customer in the 3rd Iteration).
All actors negotiate QM, Weights, Thresholds and Values
FP7-ICT2011-8. Project #: 318082
Example of a Quality Model selected for an specific Project:
Defining a Quality Model (4)
FP7-ICT2011-8. Project #: 318082
Case Study (1)
• Spanish company founded in 2014• Mobile applications development• Agile methods• SME: 10 employees
• We need to manage Quality of our:- Processes- Products
FP7-ICT2011-8. Project #: 318082
Case Study (2)
Help us to define our Quality Model!!
FP7-ICT2011-8. Project #: 318082
How: Methodology
• Method 1: Quality Objectives Settings Method
• Method 2: Planning a Measurement Program
• Method 3: Defining the Metrics: Goal/Question/Metric
• Method 4: Data Collection• Method 5: Reliability and Validity of
the Measures
Iteration 1
Iteration 2
Iteration N
FP7-ICT2011-8. Project #: 318082
Method 1: QO Setting Method (1)• Inputs :
– Stakeholders & ExpertiseManagers-decision-makers, Developers,Quality assurance, Customers, Project managers…
– Quality Model:ISO 9126 ISO/IEC 25000
– Existing measurement data:Raw data of result of test cases in Excel documents
FP7-ICT2011-8. Project #: 318082
Why all the stakeholders….
managers-decision-makers => holistic vision of the Company
developers => are the requested goals measurable?
Project managers => knowledge of historical projects
Customers => we look at customer satisfaction
Method 1: QO Setting Method (2)
FP7-ICT2011-8. Project #: 318082
Everybody is committed to achieve the Quality Goals
Method 1: QO Setting Method (3)
FP7-ICT2011-8. Project #: 318082
Method 1: QOSM• Step 1: Preparation and pre-assignment
– Setting the Context (Example study case)– Setting the Roles (Roles involved in the process)– Scheduling a workshop (Now)– Pre-assignment to participants (5mins)
Test process conformance
Usability
FP7-ICT2011-8. Project #: 318082
Method 1: QOSM• Step 2: Brainstorming
– Group brainstorming (20 mins)– Forming objectives on the wall (15 mins)
FP7-ICT2011-8. Project #: 318082
Method 1: QOSM• Step 3: Prioritizing
– Voting (10mins)
FP7-ICT2011-8. Project #: 318082
Method 1: QOSM• Step 4: Elaborating
– Top-5 selection– Detailed description (1 Quality Objective):
Name Short name
Description Description on higher level
Rationale Motivates why the objective is important. Gains and risks; E.g., costs, business benefits, customer viewpoint
Related factors Tentative ideas of what contributes, or prevents achieving this objective
Votes Number of votes in prioritization
One or more quality indicators Measures that can be used indicate if this objective is achievedFor each quality indicator:- Description- Current value of the measure- Target level to achieve the objective- Deadline - Continuous or stepwise
Easiness of collecting We have it / Requires some work / Impossible
FP7-ICT2011-8. Project #: 318082
Method 1: QOSM• Step 4: QO detailed description
Name SW easy to learnDescription How easy is for the end user to learn how to use the application
Rationale End user experience. Customer satisfaction
Related factors Knowledge of the end user. Accessibility. Device
Votes 8
One or more quality indicators Effectiveness of ‘help’: easiness of finding help functions- Not measured yet- Satisfaction > 85%- 5 months- Stepwise: Per each prototype delivered to the ownerTime to learn: How long it takes to learn the application- Not measured yet- Time < 5 days- 5 months - Stepwise: Per each project finished
Easiness of collecting Requires some work (For example, usability surveys)
FP7-ICT2011-8. Project #: 318082
Method 1: QOSM
• Step 5: Post-Workshop QO– Completion of the description of the Quality
Objective– Digging out the existing data on current level– Communicate the Quality Objectives
FP7-ICT2011-8. Project #: 318082
Method 2: Planning measurement program
• Identifying Quality Indicator types:– Purpose of the Measurements– Measurability (numbers, human assessment, not
measurable)– Automation of data collection– Target of the measurement– Frequency of the Feedback– Organizational Level of Information Use
• Planning the use of data
FP7-ICT2011-8. Project #: 318082
• Quality Indicator definitionName Effectiveness of help
Purpose Evaluate the easiness of finding the help functions
Measurability The indicator it is going to be measured by both: human assessment and quantitative numbers. It could be evaluated by means of:- Satisfaction of the end user (Survey) - Time to find the help functions
Degree of automation None
Target Product
Frequency of feedback Whenever a SW project is delivered to client
Organizational level of information use
Developer Manager
Method 2: Planning measurement program
FP7-ICT2011-8. Project #: 318082
Method 3: Defining the Metrics
• Method 3: Defining the Metrics: Goal/question/metric
FP7-ICT2011-8. Project #: 318082
Next Steps: Methods 4 & 5
• Method 4: Data collection• Successful measurement activities• Manual or automatic data collection
• Method 5: Reliability and validity of the measures• Assessing measurement validity• Evaluating proposed metrics
FP7-ICT2011-8. Project #: 318082
Many thanks !!!
facebook.com/UQASAR
twitter.com/UQASAR
linkedin.com/groups/ UQASAR-Project-4682307
www.uqasar.eu
Contact us!