The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
-
Upload
josiah-renaudin -
Category
Software
-
view
42 -
download
0
Transcript of The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
![Page 1: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/1.jpg)
W14 Test Techniques 5/4/16 15:00
The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing
Presented by:
Stephen Burlingame
Navicure
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888-‐-‐-‐268-‐-‐-‐8770 ·∙·∙ 904-‐-‐-‐278-‐-‐-‐0524 -‐ [email protected] -‐ http://www.stareast.techwell.com/
![Page 2: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/2.jpg)
Stephen Burlingame Navicure Stephen Burlingame’s introduction to software quality assurance occurred during his university senior project where he discovered critical defects in software designed to diagnosis breast cancer in digitized mammograms. Since graduation, Stephen has worked exclusively in the quality assurance field. In 2006, he joined the medical claims clearinghouse Navicure, Inc., where he is director of quality assurance, responsible for overseeing the testing for an application that processes more than one million transactions daily. Living in the Atlanta metro area with his wife, Julia, Stephen has served as president of Atlanta SPIN (Software and Systems Process Improvement Network). Â
![Page 3: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/3.jpg)
The Canary in the Coal Mine
Creating an Early Warning QA Tool to Help Your Testing
1
![Page 4: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/4.jpg)
Slides written by
Stephen Burlingame
Director Of Quality Assurance at Navicure Inc., a medical claims clearinghouse offering additional billing and analysis tools for medical professionals.
Board member of Atlanta SPIN, a non-profit organization dedicated to promoting software process improvement
2
![Page 5: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/5.jpg)
Who’s familiar with vCAC from
VMWare?
3
![Page 6: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/6.jpg)
vCAC from VMWare Example Interface
4
![Page 7: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/7.jpg)
Created CentOS Installation with Custom Software Deployed
Now our QA team can request environments on as
needed basis
Software build is deployed and configured automatically
Near “push button” operation
5
![Page 8: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/8.jpg)
Yea! We’ve created virtual
test environments! Our work
is completed and now we
can all go home!
In reality the hard work is not
complete
You’ve solved some difficult
problems but possibly created
new challenges
6
![Page 9: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/9.jpg)
Environment Chaos!
Old Setup
CERT IMP
New Setup
CERT IMP
Virtual 1
Virtual 2
Virtual N
Virtual 3
7
![Page 10: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/10.jpg)
Managing and keeping
track of environments is
getting out of hand!
QA engineers spend more time
figuring out what systems they
have and the health those
systems than ever before.
More systems = more problems
8
![Page 11: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/11.jpg)
Problem: How Can We Monitor Those
Systems and Not Spend More Time in
Maintenance than Testing?
9
![Page 12: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/12.jpg)
Solution: Create a Daily Report
This report will highlight system health and basic
performance
This report will focus on displaying key performance
indicators in graphical form that is intuitive
This report will alert on known conditions that indicate
defects
10
![Page 13: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/13.jpg)
“QA Health Checker”
11
![Page 14: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/14.jpg)
“QA Health Checker”
Data Examples
Start Simple!
o Disk space alerts
o Indicate services are up or down
o Look for directories present or missing or when last updated
12
![Page 15: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/15.jpg)
“QA Health Checker”
Data Examples
Get More Complicated!
o Write SQL queries that look for “bad” data or missing data
o Look for errors in logs or indicate that error logs have content
o Look for scenarios that indicate system is experiencing issues
13
![Page 16: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/16.jpg)
“QA Health Checker”
Data Examples
Let’s Get Crazy!
o Create a uniform CSS
o Implement basic pictographic principles
o Fit this report into retrospective discussions so you improve the metrics captured
14
![Page 17: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/17.jpg)
15
![Page 18: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/18.jpg)
16
![Page 19: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/19.jpg)
17
![Page 20: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/20.jpg)
A More
Technical Look
Test systems run nightly cron jobs
to gather data and place it on
their local hard drive
Central reporting machine knows
which IP addresses to read from
and pulls the stats into a prepared
report
Report is emailed to a distribution
list early every weekday
18
![Page 21: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/21.jpg)
A More
Technical Look
Used Python as the scripting
language
Created a deployment script and
documented the release process
Documented the report contents
and general architecture
Used Confluence as the
documentation portal
19
![Page 22: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/22.jpg)
Changing the Conversation With
Employees
20
![Page 23: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/23.jpg)
Old Conversation
QA Engineers didn’t always know what was going on or
how interpret data. They complained about “phantom
issues” and “non-reproducible bugs”. Used vague terms like
“slow”, “buggy”, or “weird”. Documentation was simply not
enough.
21
![Page 24: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/24.jpg)
New Conversation
QA Engineers know right away the types of issues they are
having or about to have in the near future. Team leads are
keeping their focus on early signs of issues leading up to
release. Management is having pro-active conversation
with employees rather than a reactive dialog.
22
![Page 25: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/25.jpg)
Closing Thought
Recommendation:
Take a slight risk with this report and experiment
Keep it adjusting and evolving!
23
![Page 26: The Canary in the Coal Mine: Create an Early Warning Tool to Help Your Testing](https://reader034.fdocuments.in/reader034/viewer/2022051709/587938c01a28ab23468b4b21/html5/thumbnails/26.jpg)
Time For Questions
Contact Information:
Email Address – [email protected]
If in Atlanta, come to an Atlanta SPIN meeting!
24