Seagate Technologies: an insider’s guide to best practices for establishing a Performance Testing...
-
Upload
hp-software-solutions -
Category
Documents
-
view
1.609 -
download
7
description
Transcript of Seagate Technologies: an insider’s guide to best practices for establishing a Performance Testing...
Reneé WilsonSQA, Seagate Technology
Seagate Technologies: an insider‟s guide to best practices for establishing a Performance Testing Center of Excellence
Establishing a PC Center of Excellence
June 2010
22
Establishing a Performance Testing CoE- Agenda
Introduction
History
Transition from LR to PC
Building a Performance Testing Center of Excellence
The Performance Testing Lifecycle
CoE Quality Elements
Scripting Tips and Tricks
Survival Guide
Establishing a PC Center of Excellence
June 2010
33
Seagate: Storage Leader
Seagate is the world’s leading provider of storage devices
Q3 FY2010*: 50.3M drives shipped; revenue of $3.05B
Provides storage devices for Enterprise, Desktop, Mobile
Computing, Consumer Electronics and Retail Markets
Builds Hard Disk Drives and Solid State Drives
31% overall market share
Broadest product offering in the industry – largest customer base
Ownership and vertical integration of critical technologies: heads
and media
52,394** employees worldwide
* For third fiscal quarter ended April 2, 2010** Includes interns, contractors, and agency temps
Establishing a PC Center of Excellence
June 2010
44
Seagate Global Presence
Oklahoma City, OK Teparuk & Korat, Thailand
Paris, France
Longmont, CO
Springtown, N. Ireland
Wuxi, China
Shanghai, China
Tokyo, JapanBeijing, China
Delhi, India
Shrewsbury, MA
Suzhou, China
Taiwan
Amsterdam, Netherlands
Shakopee, MN
Bloomington, MN
Sunnyvale, CA
Santa Clara, CA
Fremont, CA
Scotts Valley, CA
Science Park, Singapore
Johor & Penang, Malaysia
Woodlands, Singapore
Ang Mo Kio, Singapore
HQs, Admin and/or Sales
Design
Customer Support
Drive & Component Mfg
Data Centers
Establishing a PC Center of Excellence
June 2010
55
History- Our LoadRunner Roots
LoadRunner users for the past 10 years
3 Controllers at one site - 6 Load Generators at primary data center
Growing test volume made test setup and results data very hard to manage
Loss of data due to decentralized desktop operations
Customers needed more visibility to previous testing and trends over time
VU Licensing spread across the three controllers
IT growing beyond what LoadRunner capabilities
Initial motivation of upgrade from LoadRunner to Performance Center based on
need for distributed testing
Data Centers started to globalize
Needed better processes to schedule and allocate assets
Required more effective use of pool assets
Establishing a PC Center of Excellence
June 2010
66
Transition from LR to PC
Did not happen over night… Transition took six to nine months
Support for early adopters
Change management for late bloomers
Discovered new infrastructure requirements
Staging
Install &
Test
Prod
Install %
Validation
Production
Testing with
PC
Test Process
redesign
Accounts for
DC and Devel
teams
SiteScope
install and
usage
Fully
weaned
from LR
1 2 3 4 5 6 7 8 9 10 11 12Months
LoadRunner Performance Center
Built new
CoE
Processes
Process
continuous
improvement
A few “hiccups” along the way – Protocols, Script activities, Scenarios
Overall improvement to our Performance Test processes – Performance
Center helped us get better at what we do
Establishing a PC Center of Excellence
June 2010
77
Performance Testing CoE- Essential Elements
Group Charter
Connections with other CoE‟s and enterprise processes
Documented process
Leverage decentralized/global operations
Performance testing lifecycle
Quality and best practice standards
Script Best Practices
Knowledge management
Collaborating with Customers and Stakeholders
Customer education
Establishing a PC Center of Excellence
June 2010
88
Charting A Course– Developing a Group Charter
Who do you represent? Charter may be different depending your organizational reporting structure
What services do you provide? Standard fare of testing functions vs. one-off test activities
What tasks are the Performance Testing Team chartered to perform?
What tasks are NOT offered?
Who are your Customers?
What is your service SLA?
What are your hours of service?
Is there a project prioritization process?
Is the Performance Test Team sized appropriately for success?
What are your communication mediums and who are your target
audiences?
Establishing a PC Center of Excellence
June 2010
99
Making the Connections– Connecting with Other CoE
Tools and Processes
Application
Release Data
Identification
of latent
defects
Monitor
use cases
Readiness for
Production
Identification of
Performance-
based defects
Test use
cases
Readiness for
Test
Establishing a PC Center of Excellence
June 2010
1010
Connecting The Dots- Drawing the Critical Communication Lines
SDLC Where does Performance Testing fit in to you development processes
Diagnostics What other tools can be leveraged to enhance the Performance test effectively
Security Testing and Management What synergies between the two groups can be leveraged
Release Management Reduce testing “surprises” by having a better radar for upcoming service needs
Deployment Readiness Processes Where does Performance testing fit in to this process
Does the Performance test team get a vote?
Post Deployment Monitoring What “intimate” application knowledge can be leveraged from the Performance test team?
Communication How closely is the Performance test reporting aligned with other reporting mediums including general Project
Management Reporting?
Establishing a PC Center of Excellence
June 2010
1111
Documenting Your Process …and Making It Visible
High level descriptive process
Drives Performance Test Team functions
Sets Customer expectations forengagement and timeline
Key Elements of Process
Entry point for new requests
Entry point for retest
Touch points with Customers
and Developers
Quality oversight
Needs to be Accessible and Visible
We use wiki in Seagate
Pe
rfo
rma
nc
e T
es
t G
rou
pC
us
tom
er
Contact
Performance test
Team with test
schedule/scope &
Clarity info
Estimate
Test
efforts
Prioritize test
in
Performance
Test queue
Perform walkthru
of application and
use cases
Write/validate
Performance
Center
scripts
Perform Load
tests (nominal,
peak, scalability,
WAN)
Start
Publish Test
Report and
Archive
Data
Done
Perform test
review with
project team
if required
Provide
required data
sets user
accounts, etc.
Write
Performance
Test Plan
Approve
Test Plan
Provide Load
Test
Requirements
Initial Contact
and Discovery
1-Week
Test Pipeline Lead Time
2-Weeks
Test Planning
and Approval
1-Week
Scripting and
Shakedown
Testing
1-Week
Test
Performance and
Reporting
1-Week
Performance Test Timeline
Performance Testing Business Process
Establishing a PC Center of Excellence
June 2010
1212
Process & Service Improvement Tips
Engage with Development Teams early in SDLC process
Earlier engagement of Performance Test Team yields a better chance for success
Better test requirements through dialog and collaboration yields more meaningful
tests and data
More up-front commitment yields high ROI through delivery of a higher-quality test
assets
Offer a standard set of
Performance Testing Services
Follow your process
Look for improvementopportunities
Look for points of connectivity
with other CoEs
Educate your Customers
Establishing a PC Center of Excellence
June 2010
1313
Global collaboration platform for performance testing
Meeting place for test planning and execution
Available to Performance Test Team
Available to test customers and data consumers
Collaboration elements
Schedule and planning
Test assets
Real-time test data
Test results and trend analyses
Allows almost 24x7 test support and development with small staff
Operations activities performed at a lower cost
Higher-priced resources can perform more planning and liaison functions
Team presence resident at developer‟s venue
Has facilitated specialization of Performance Test Team members
Empowers and motivates team members
Maximizing CoE Assets- Leveraging Global Operations
Establishing a PC Center of Excellence
June 2010
1414
Performance Testing Lifecycle-Making Your CoE Process Visible to
Customers and Team
Preplanning
• Test calendar
• Test Services
• Test Capabilities
• Example Test Plans
• Example Test Reports
• Maintenance windows
• Blackouts
Planning
• Test Plan
• Test Schedule
• Customer engagement
• Environment avail.
• Diagnostics
• Script and test
walkthroughs
Testing
• Test alerts
• User accounts
• Test collaboration
meeting
• Diagnostics
Post test
• Test Result
Announcement
• Test Reports
• Trend analyses
Performance testing is a process rather than an event
Each phase of Performance test lifecycle includes artifacts and deliverables
Needs to be visible and available to stakeholders
Performance Test Lifecycle
Establishing a PC Center of Excellence
June 2010
1515
Critical
Artifact
Performance Testing Lifecycle- Preplanning
Mostly a Customer resource
Provides Performance Test Calendar visibility
Planning for environment availability
Includes educational material
Example artifacts and deliverables
Helps set Customer expectations
for services and lead times
We post all of this on our (wiki)
webpage
Establishing a PC Center of Excellence
June 2010
1616
Performance Testing Lifecycle- Planning
Close engagement with customer
Earlier in the SDLC the better
Development of detailed test plan
Agreement on test schedule
Needs to be coordinated with Test Calendarin line with available resources and assets
Be cognizant of rework-loops
Test Plan walkthrough with stakeholders
Plan for required diagnostics
Script quality review with test team
Posting of final test plan (document management)
• Change history
• Goals and scope
• Topology being tested
• User account info
• Special application info
• Test use cases
• Scenarios
• Acceptance criteria
What’s in a
Performance Test Plan?
Critical
Artifact
Establishing a PC Center of Excellence
June 2010
1717
Performance Testing Lifecycle- Testing
Schedule test meetings
Stakeholder can log into Performance Center and watch test in real-time
Email Load Test Alert to all Stakeholders
Queue and validate diagnostic
functionality
Perform full battery of required testing
Analyze each test before moving on to next
Often tests do not succeed on the first try
Perform Testing
Critical
Artifact
Establishing a PC Center of Excellence
June 2010
1818
Performance Testing Lifecycle- Post Test
Analyze and create report
Post report to common accessible location
Send result email with executive summary of results
Send to same stakeholders that received Load test Alert
Have standard nomenclature for results (i.e. pass, fail, conditional pass, interim
results, etc)
Provide links to reports
Include Trend data if possible
Always follow-up with Customer
Document any lessons-learned
We post 3 reports Traditional Word doc
HTML Summary
Wiki news item
Critical
Artifact
Wiki news item page
Establishing a PC Center of Excellence
June 2010
1919
Quality Touch-points- A Few Simple Activities That Can Have
a Big Impact
Performance Test Plan is your contract with Customer Formalize the plan
Tests should not be performed until this is correct/approved
Walkthroughs with Customer critical
Scripting standards and best practices Keep a list – follow it!
Script code walkthroughs
Small-scale shakedowns
Help Customers develop realistic schedules Tests rarely succeed on the first pass
Always have test review with Customer and Stakeholders
Look for ways to make CoE processes and work products visible
Look for ways to improve your processes – avoid open loops
Establishing a PC Center of Excellence
June 2010
2020
Be the Scryptkeeper ! Write „em down – Integrate into your own Best Practices
Annotate and comment! It‟s just code…
Use Actions to manage script flow
References to external sites like hitbox or doubleclick should be removed
Randomize think times – be aware of distribution
Run script as a thread (as opposed to a process)
Make use of “Snapshot on Error”
Use “Continue on Error” like garlic… sparingly
Use “simulate parameter” tool to preview parameter allocations – especially useful when using “unique” option
Avoid “Silent” Failures – always encapsulate everything in named transaction and use web_reg_find
- Our Top-Ten Scripting Best Practices
Establishing a PC Center of Excellence
June 2010
2121
Scripting Fun- A Few Scripting Techniques We Really Like
Sybil Make use of variable transaction names when you
need more measurement granularity – scripting
short cut – use sparingly!
Don't Re-invent the Wheel Use or create Correlation Libraries for hard-to-script applications. Some vendors provide
them! Recording Options/Correlation
The Sledge Hammer Sometimes the only way to
verify content is to see how
much stuff came back
Oracle Reform When recording Oracle NCA Protocol requires modification
to default.cfg file (in script folder) as well as “Names”
change on Oracle Server side
Establishing a PC Center of Excellence
June 2010
2222
Leveraging the Knowledge- Building Organizational Knowledge into Your CoE
An organization continues to advance its knowledge by preserving the lessons it learns
Fourth Generation Management, Brian Joiner
Scripting with new or multiple protocols
Tips, tricks, and time-savers
Quality and reliability improvements
Architectural knowledge
Troublesome applications
Ways to leverage existing assets
Better runtime settings
Scenario management
Transaction name nomenclature
Documentation management
Always be on the lookout for ideas and concepts
Make sure your team has forum for collecting and managing
Find a way to store and institutionalize the knowledge
Use quality reviews as a medium to find new ideas and reinforce existing ones
Areas where we’ve benefited and grown our organizational knowledge
Establishing a PC Center of Excellence
June 2010
2323
Collaboration- Making Customers and Stakeholders Active Partners
Real-time performance worth a thousand reports Encourage Stakeholders to watch the test
Keep test plans accessible and visible Document management
Quality Center
Ensure visibility of test results
Tie-in with other Stakeholder groups Quality-assurance
Operations
Monitor Team
After test failures – be part of “solution” Collaborate with teams to look for root-cause
Suggest other tests that could help drill down to root-cause
Engage in post-mortem activities
Always look at what went “right” and what went “wrong”
Feedback lessons-learned into CoE Business Processes
Establishing a PC Center of Excellence
June 2010
2424
Customer Education- Supporting Your Customers
Give them as much access to self-help as possible
Engage with Customers on all projects
Offer training Primers
Familiarity and training sessions
Account Management Group accounts
Tailor permissions and views for test participation
Write reports (or amend them) for Customer consumption Plain English vs. IT-Speak goes a long way!
Customers LIKE trend analyses
Be accessible
Schedule regular business reviews with Customers and Stakeholders Review upcoming challenges, issues, and testing
Establishing a PC Center of Excellence
June 2010
2525
Surviving the Game- Practical Survival Tips for Performance CoE
Establish an SLA for Performance CoE Services
Prioritize your projects
Establish and enforce standards
Engage Customers – they need to have “skin” in the game
Leverage collective group knowledge
Apply the appropriate level of diagnostics
Attempt to preserve scenarios for reuse
Reboot controllers and load generators on a regular basis
Make use of user notesin results summary
Perform analysis in Performance Center immediately after run
Provide interpretation of results in test reports
Establishing a PC Center of Excellence
June 2010
2626
CoE Enablers- Performance Center Capabilities
We Really Like Customer Involvement
Global access
Test calendar and scheduling
Asset pooling
Automated central repository Scripts
Scenarios
Test Data
Trending Big time saver!
Customers greatly appreciate
User privilege management
Peripheral tool access
A few features we’d really like to see…
Tighter integration with Quality Center
Leverage test plans and reference results
Defect/requirement connectivity
Document repository on a project level
Hierarchical project architecture
Projects within programs
Establishing a PC Center of Excellence
June 2010
2727
Q&A
28 ©2010 Hewlett-Packard Development Company, L.P.
To learn more on this topic, and to connect with your peers after
the conference, visit the HP Software Solutions Community:
www.hp.com/go/swcommunity
29