W7 Concurrent Class
10/2/2013 1:45:00 PM
"Key Strategies to Survive the
Mega Test Program"
Presented by:
Robert Goetz
Kaiser Permanente
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Robert Goetz
Kaiser Permanente
With more than twenty years of leadership experience in all aspects of IT, Bob Goetz has
focused on leading QA organizations for the last several years. As part of this, Bob has led the
testing of several mega programs—more than $500 million. The first mega program turned
Bob’s hair grey as he gained the experience to survive and thrive. That multi-year effort
implemented new core business software while transforming business processes and
introducing new products to the market.
1
Key Strategies to Survive the Mega Test ProgramBob Goetz, Executive Director, Test Services
Kaiser Permanente - Health Plan BIO
2 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Topics
About Kaiser Permanente
What is a Mega Program?
Key Strategies, Methods & Processes
Final Takeaways
Q & A
2
Kaiser Permanente: The Largest Not-for-Profit Health Plan
• A Durable Partnership
– Kaiser Foundation Health Plans and Hospitals– Eight Permanente Medical Groups
• More than 9 million members • Serves 9 states and District of Columbia• 37 medical centers; 611 medical office buildings • 175,000+ employees• 17,000+ physicians • $50.6 billion operating revenue• Largest labor management partnership in history• Largest non-academic research organization in the U.S.• Largest private user of electronic medical records in the world
Data as of year-end 2012
July 29, 2013 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.3
The Kaiser Permanente Mission
Kaiser Permanente’s mission is to provide high-quality, affordable healthcare services and to improve the health of our members and the communities we
serve.
July 29, 2013 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.4
3
Mid-Atlantic Region Washington, DC Maryland Virginia
Georgia Region Atlanta, GA
Colorado Region Denver / Boulder, CO Colorado Springs, CO
Ohio Region• Cleveland, OH• Akron, OHNorthern
California Region
Hawaii Region
Northwest Region Portland, OR Vancouver, WA
SouthernCalifornia Region
The Markets We Serve
July 29, 2013 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.5
Kp.org and My Health Manager –World’s Leading Personal Health Record
YE 2012 –
– 116 million visits
– 32 million lab test results viewed
– 13 million secure messages through Email My Doctor
– 12 million Rx refills
4
What is a Mega Program?
7 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Or how did my hair turn grey?
What is a Mega Program?
8 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
5
What is a Mega Program?
9 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
What is a Mega Program?
10 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
• Huge Opportunity… Huge Risks
• Bets your company’s future
• Has Board of Director (BOD) Oversight
• External Review / Audit
• Your job is on the line
6
What is a Mega Program?
11 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
• Huge Opportunity… Huge Risks
• Bets your company’s future
• Has Board of Director (BOD) Oversight
• External Review / Audit
• Your job is on the line
How can you SURVIVE and even THRIVE a Mega Program?
Set the Stage
12 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
7
Set the Stage
Everything is a business problem– What is the mega program trying to solve?
– Why is it important?
Discover the situation you are facing– Who are the stakeholders?
– Who holds the decision rights?
– Who decides “quality”? What is “highest level of quality”?
– What are the Risks? Issues?
13 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Standardize Processes
14 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
8
Standardize Processes
15 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
• Do a very quick analysis of process AND compliance• Don’t assume your teams are following your “standards”
Standardize Processes
Test Strategy
Test Plan
Test Suites/Cases
Data Creation
Data Usage
Naming Standards
Component Names
Automation
Test Closeout Report
Defect Standards
16 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
9
Standardize Processes
17 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
• Do a very quick analysis of process AND compliance• Don’t assume your teams are following your “standards”
• Create New Tester Orientation Guide• Test Suite/Test Case Standards
• Defect Standards
Standardize Processes
18 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Defect Severity Severity Level Description1. Critical Complete system outage and/or is detrimental to the
majority of the development, user, and/or testing efforts and there is no known workaround. This would also be known as an emergency defect.
2. High Serious adverse impact to the user and there is not an effective work-around, however work can continue.
3. Medium Moderate adverse impact to the user but there is an effective workaround.
4. Low No immediate adverse impact to the user. Normally cosmetic.
10
19 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.
Build Allies
July 29, 2013
Build Allies
20 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Understand the history of your partners– Do they trust you?
– Find out who they trust and why – learn from that
Be transparent– Admit what you know AND (more importantly!) what you don’t know
Give them a seat at the table– Treat them as they deserve: as professionals
– Loop them in on decisions
– Find out what they need and work their problems for them
11
Don’t Forget Your “Friends”
21 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Don’t Forget Your “Friends”
22 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Truism– A divided house falls
– Never throw development (or UAT) under the bus – even as you shake off their dust!
– To the business I.T. is one org – they don’t care who does what
Establish early in the project your accountability for metrics– Own it – if at all possible
– Be transparent – share all data possible – with everyone
Get data to development early – Give them a chance to consume it and be able to respond
12
Measure Progress
23 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Measure Progress
24 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Treat Defect Data as a Precious Commodity– Be consistent in how you log defects (e.g. root cause)
Make it public and protected– Daily reports, open SharePoint Site
– Converted to PDF format
Formats
– Static “point in time” reports
– Trending Charts
13
25 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Burn Up Chart
26 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Burn Down Chart
14
27 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Backlog Chart
28 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Defect Metrics
15
29 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Root Cause
Speak in the Language of Business
30 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
16
Speak in the Language of Business
31 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Remember– This is never a technology project – it is a business project using
technology.
Speak in their terms
– Call center?
Impact to Average Speed to Answer
– Ordering System?
Impact to Average Processing Time
– Manufacturing?
Impact to Work in Process (WIP)
Speak in the Language of Business
32 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Know your audience– Do they want a short “Yes” / “No” answer or long story?
Don’t conflate your answer
Answer the question behind the question
Back it up with data – current data
If you don’t know – that is OK – Tell them you’ll find out and get back to them
Remember to tell the story behind the data
17
33 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
“Even though the overall defect backlog increased during the last month, we see the Critical defect backlog decreasing.
This is a good sign that Critical defects are getting the priority they need. We still have many older then 90 day Critical defects, but this is a good sign that work is being done to address this.”
Create a safe space
34 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
18
Create a safe space
35 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Testing / QA is an easy target– We are at the “end” of the project
– We are easily measured
Find a weekly time for the team to vent and support each other
– What is said in the room, stays in the room
Double check the pulse of the team:– What is working? What is not?
– What can we change? What should I (as test leader) change?
– What is blocking that I need to unblock?
Final Take Aways
36 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
19
Final Take Aways
37 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Top Things to Remember1. You are not alone
2. Standardize, Standardize, Standardize
3. Build Allies and Monitor your Friends
4. Measure your Progress and Treat Data as Gold
5. Tell the Story in Business Teams
6. Protect the Team
Q & A
38 | © 2013 Kaiser Foundation Health Plan, Inc. For internal use only.July 29, 2013
Top Related