Transcript of Statistical Process Control & Metrics Suggestions to alter perceptions of the value of testing. Mike...
- Slide 1
- Slide 2
- Statistical Process Control & Metrics Suggestions to alter
perceptions of the value of testing. Mike Jarred & Ilca
Moussavou
- Slide 3
- Slide 2 IDBS 2012 Common challenges (perception & value)
for Test Managers to solve ChallengeObjective Value of testing not
universally accepted (the NEED for testing is often accepted)
Demonstrate and gain recognition of the value of the testing group
Make test group the agent for change in the company Testing
perceived as a bottleneckAlter perception of testing as a
bottleneck to a barometer of project health More FTE Required than
budget would accommodate Demonstrate how rework levels consuming
test resources improve quality to reduce rework. Divorce rework
from testing as a cost (guilty by association) Isolated objective
targets across departments make it hard to demonstrate value /
drive strategic initiatives Provide information to Executive Team
so they can set cross departmental targets Testing process
improvement driven by practitioners, but not necessarily aligned to
business objectives Ensure test improvements support business goals
Quality viewed subjectively, with different interpretation by
different stakeholders. Only quantitative measure is defect count.
Remove the subjectivity of quality and work towards a measurable
quality definition. Show value add of test by driving quality
initiatives, not just measuring it.
- Slide 4
- Slide 3 IDBS 2012
- Slide 5
- Slide 4 IDBS 2012 A problem shared World Quality Report 2012
Sogeti / HP (global survey of more than 1500 CIOs, CFOs, IT and QA
Directors) Less than 5% of firms regard their testers as best in
class is it really that low, or perception? Testing needs to be a
formalised step in the application lifecycle we still see instances
of the testing function being considered as an afterthought or
viewed as a roadblock preventing application deployment So, this is
what our bosses are saying about us What can we do to address this
perception this is their reality
- Slide 6
- Slide 5 IDBS 2012 No silver bullet, or standard approach
- Slide 7
- Slide 6 IDBS 2012 Demonstrate and gain recognition of the value
of testing If there is no measure of the protection / assurance
testing provides to the business this inevitably raises a question
of value Often there are only limited existing measures; e.g. time
in test per project, and defect data raised by projects.
Introduction of Test Effectiveness Metric (aka Defect Leakage, with
a positive spin - if your figures are good!) Test Effectiveness =
defects found in test as percentage of defects found by test and
customers Demonstrate per product line the Test Effectiveness, with
improvement over time A communication process for this will be
covered later in the slide deck Gain advocates for example a
Support Manager or Product Manager They feel the pain if test dont
add value They understand the impact of good testing on their
departments, the organisation and customers
- Slide 8
- Slide 7 IDBS 2012 Example Test Effectiveness timeline %
Time
- Slide 9
- Slide 8 IDBS 2012 Changing the perception of test Introduce
daily reporting that shows the trends of Quality & Progress
over time of the project constraints to test progress easily
demonstrable. Increasing Visibility & Predictability = value
add! Start reporting these metrics only on the high risk / critical
projects Once the opportunity cost (e.g. less projects being
tested) is understood by Executive, testing no longer seen as a
bottleneck, but the solution is to the left of testing in the
lifecycle Testing can be perceived as a bottleneck Capture time to
demonstrate how much effort is applied to rework (or, WASTE!)
Introduction of Initial Build Quality (IBQ) (% of tests that pass
FIRST time they are run, for new functionality and regression) IBQ
= 50 tests executed, 25 pass, 25 fail = 50% IBQ measure of
development process.
- Slide 10
- Slide 9 IDBS 2012 Daily Test Report illustrative trend
information
- Slide 11
- Slide 10 IDBS 2012 Demonstrate rework levels impact on
available test resource: illustrative slippage due to rework Effort
(Hours ) Date
- Slide 12
- Slide 11 IDBS 2012 Raising value awareness Internal Conference?
Examine possibility of promoting ideas internally, e.g. a
conference day Invite VPs, Sales, Marketing, Product Management,
Project Management, Development, Support (any key stakeholders and
advocates who provide internal independent references for testing)
Make testers feel valued, and grow profile of the test group.
Market this internally the same way an external events company
would promote a conference, add external speakers to add
credibility / gravitas This can significantly raise the profile of
testing Demonstrate increased test effectiveness with an increase
of workload (if this is a reality it often is) Demonstrate IBQ,
discuss the impact on test resourcing and waste Discuss, amongst
other things, how testing could support quality further given
understanding of customers needs / want / perception
- Slide 13
- Slide 12 IDBS 2012 Inform Executive Team so cross departmental
targets can be set Create awareness that departmental improvements
need to support other departments, and relate to strategic business
goals Propose, and take ownership of running a Metrics Governance
Board. Snr. Managers representing IT & Business a senior
stakeholder owns the meeting (but testing run it) Executive summary
used by senior management in leadership team meeting with CEO
Reporting for all Products - processes needed base lining Unit Test
Coverage & IBQ, Overall Quality (Development Process Measures)
Test Effectiveness (Test Process Measure) Support metrics support
SLA achievement Once Performance baselines are understood, targets
for improvements can be set.
- Slide 14
- Slide 13 IDBS 2012 Statistical Process Control (SPC) Simple
approach to quality control and process improvement involving
statistical data analysis SPC consists of: - Measuring of a process
(using objective data) - Eliminating variation due to assignable
causes - Regular monitoring of a process - Improving the process to
its best target value Tool Used: Control Chart Total Variation =
Common Cause Variation + Special Cause Variation
- Slide 15
- Slide 14 IDBS 2012 Statistical Process Control Day to day
example Special Cause Variation E.g.: Heavy Traffic, Accident on
the road, Woke up late DaysJourney Time Monday45 minutes Tuesday50
minutes Wednesday47 minutes Thursday120 minutes Friday49
minutes
- Slide 16
- Slide 15 IDBS 2012 Statistical Process Control Control
Chart
- Slide 17
- Slide 16 IDBS 2012 Ensure SDLC improvements support business
goals Root Cause Analysis of control limit breaches drives change
to process to reach departmental targets.
- Slide 18
- Slide 17 IDBS 2012 Statistical Process Control - Benefits a)
Continuous Process Improvement b) Process Baselines c) Early
visibility and reaction d) Quantitative Management Decision Making
e) Economical Value
- Slide 19
- Slide 18 IDBS 2012 Does this make me a bad parent? Pocket money
is earned, and lost , in 5 pence increments depending on behaviour
Sunday is count-up day - and root cause analysis!
- Slide 20
- Slide 19 IDBS 2012 Building Trust - Credibility of Metrics
Metrics provided by SPC need credibility Show a Governance Board
the following ; Approach taken in capturing data, including the
source of data Historic data, (up to 4 years old) that not only
provide the baseline but show outliers (events) in the corporate
memory they recognise, so future outliers will be trusted. The data
is usually available in Quality Centre & other management
toolsets it is simply a case of extracting it and presenting the
information.
- Slide 21
- Slide 20 IDBS 2012 Target Setting Initial targets set, across
the SDLC Targets set using historical data, with attention to the
previous 6 9 months As the targets are set using detailed
knowledge, the targets are seen as fair and reasonable Target
setting should involve the customer of the process; e.g. IBQ is a
measure of development build Test can set expectations of
acceptable quality as a development target. Support can set the
target for test effectiveness
- Slide 22
- Slide 21 IDBS 2012 The vision - transition of testing to agents
of change Test can have a credible voice at the top table, with
authority to challenge conventional thinking Process improvements
across the organisation can be driven through testing and the
information the function provides Test Group can position itself
positioned as the trusted advisor - seen by execs as the advocates
in improvements to the SDLC Metrics Governance Board can trigger
improvement initiatives to Increase predictability of project
outcomes Improve quality of software into test Move towards defect
prevention culture Our conference output has seen major investment
in performance, security & business analysis to improve
usability and quality of products.
- Slide 23
- Slide 22 IDBS 2012 Success Factors Standardised Testing
Methodology, consistently applied Standardised testing tools and
SDLC tools to capture Data surrounding your process. Ability to
extract the data from QC database. Establish responsibilities for
data capture, presentation & utilisation. Change the culture
use advocates / champions outside your group help Good
communication (top / down / sideways). Create feedback loops Access
to a mathematician if you dont have an Ilca - tools like Minitab
for SPC Ensure sponsorship work out what is important to your
sponsor and how this would help them achieve their goals. Youll
probably get permission. If not, do it anyway - you may be seen as
innovative Keep it simple Senior Execs dont have the time for
detail, or necessarily understand the detail Stress metrics are
used for process improvement not individual performance management
(there is no quicker way to derail any inertia!)
- Slide 24
- Slide 23 IDBS 2012 Closing thoughts You already have lots of
data use it to sell and tell a story Everybody has a process, even
agile teams! its about measuring what you feel is significant in
your process to see if it can be improved Ensure the information is
credible Be aware that metrics can drive behaviour (not always
favourably) Use industry information and compare blow you own
trumpet when you smell a success story (e.g. WQR only 5% companies
achieve 100% ROI on automation) build your case, no one will be
able to confidently challenge (you are the experts!)