Post on 16-Dec-2014
description
How to Become A Testing Expert Liang Gao
Typical Interview Question
• Do you know programming• Answer: Yes, a little• Which language do you know the best• Answer: I know C a little• How many lines of code have you written• Answer : Umm.., about 300 lines• Do you know Java or .NET, • Answer : No• Do you know Linux• Answer : No• Do you know networking• Answer : I installed Windows XP on my PC….• Ok, you can work for us as testing engineer
Tester’s Dilemma
• Testing is not cool• Tester is second class when compared with
developer• Developer using me as servant• I don’t see I can make difference in our
company• Testers are keep coming, they are young and
better than me. • 我在吃青春饭, I don’t know where my career
path is• OK , I need to be a developer and write
code!
Top Management's Dilemma
你也知道, 现在招到好的测试工程师是有多难 - 华为北京研究所副所长 2008
It took Juniper 6 month plus to find a new QA director
Career Story of Ting
• Graduate at 1990, only job found is the tester at Sun
• Join Cisco as tester in 1993, then test automation engineer (design Cisco’s automation framework)
• Cisco grows from 2000 to 40000 from 1993 to 1998
• Join Netscreen as the 1st testing engineer at 1998• NetScreen went to Nasdaq on 2001• Grow with NetScreen as testing lead, testing
manager, testing senior manager and testing director
• Found Sigma at 2004, served as CEO as today
Lesson
• You may have to stay long enough to be competitive.
What is a testing expert?
• Ability to find critical bugs in given time frame (hot gun)
• Ability to build a comprehensive testing strategy in given time frame (expert)
• Ability to manage the release process (top expert)
What is a testing expert Looks like?
• Play the video of James Bach
Find Bug that could make News
Bugs in the News
• A Cisco Secure Access Control Server (ACS) that is configured to use Extensible Authentication Protocol-Transport Layer Security (EAP-TLS) to authenticate users to the network will allow access to any user that uses a cryptographically correct certificate which can be expired, or come from an untrusted Certificate Authority (CA) and still be cryptographically correct.
• CSCse58195. The WLC contains a bug when processing WLAN ACLs that causes the WLANvACL configuration to be saved with an invalid checksum. When the configuration is subsequently reloaded at boot time, the checksum fails and the WLAN ACLs are not installed.
Bugs in the News
• CSCdv24925 It is possible to read stored configuration file from the Storage Router without any authorization.
• CSCdu45417 It is possible to halt the Storage Router by sending a fragmented packet over the Gigabit interface.
• CSCdv24925 An unauthorized person may read the configuration of the Storage Router. That may lead to unauthorized access of a storage space.
Bugs in the News
• Versions of the Cisco ACE 4710 Application Control Engine appliance prior to software version A1(8a) use default administrator, web management, and device management account credentials. The appliance and module do not prompt users to modify system account passwords during the initial configurationprocess.
• Crafted SSH Packet Vulnerability • Crafted SNMPv2c Packet Vulnerability
2 Factors of a hot gun
• Technical Expertise• Thinking methodology
Hot Gun’s Bug Percentage
P4
P2
P1
P3
Cosmetic
Functionality
Major Functionality
Critical Functionality, Crash, Hang
10%
30%
40%
20%
Thinking Methodology
Thinking Methodology
• Customers• Information• Developer
revelations• Team• Equipment
& Tools• Schedule• Test Items• Deliverable
s
Structures Functions
Data
Platform
Operation
Time
Capability
Reliability
Usability
Security
Scalability
Performance
Installability
Compatiabilty
Supportability
Testabilty
Measurability
Portability
Localizations
Function testing
Domain testing
Stress testing
Flow testing
Scenario testing
Claims testing
User testing
Automatic testing
Thinking Methodology
• Customers• 客服• 售前• 售后• 市场
Thinking Methodology
发布
升级 / 安装
配置
入网
运营
维护
发布流程测试
Installation/Upgrade 测试 HA
你知道客户典型配置么?你知道客户跑什么业务?你知道客户跑什么业务?
你知道客户典型应用场景么?
客户 1-2 年运营的可能遇到情况
可维护性测试
兼容性?
Thinking Methodology
Negative Stress Boundary Features Interaction
Security
Feature Point 1
Feature Point 2
Feature Point 3
Feature Point 4
Feature Point 5
Develop Testing Strategy
• Be the review person• 知其然也知其所以然
Develop Testing Strategy
• What is your goal first?– Find more bugs?– Find more critical bugs?– Ensure product or feature has no critical
defects– Ensure customers will be ok after the release?
• The strategy– Bug oriented?– Coverage oriented?– Customer oriented?
What is Coverage Strategy?
• How to thoroughly test OSPF Hello protocol?
– Function points? (tester)– User Scenarios? (test expert)– Scalability? (test expert)– Performance?– Security? (test expert)– ………
What is Coverage Strategy?
• 7 platforms• 6 different line cards• 2 modes (main/aggressive)• AH/ESP• CA/No CA• HA/No HA• Hub Spoke/Partial Mesh/Full Mesh
7x6x2x2x2x3 = 1088
Types of Testing Covered on Different Release
Major Release
Minor Release
Patch Release
Platform Release
SFR CSP
New Feature Test Full Full TBD Full
Regression Test Full Partial Partial Partial Partial
System Test Full Full Full Full
Interoperability Test Full TBD TBD TBD
Performance Test Full TBD Full TBD
Capacity Test Full Full TBD
Security Test Full Full Full Full Full
Automation Test(partial regression)
Full Full Full Full Full Full
SFR – Special Feature Release CSP – Customer Specific Patch
Advise to the New Expert
• Practice, Practice, Practice• Don’t confuse experience with Expertise• Don’t trust the folklore – but learn it anyway• Take nothing on faith, own your methodology• Drive your own education, no one else will• Reputation = Money: Build and protect your
reputation• Relentlessly gather resources, materials and
tools. • Associate with demanding colleagues. • Write, speak
An Expert’s Vision
• An Expert’s Vision– I can test anything– Under any condition– On any given time frame.
An Expert’s Vision
Top Expert
• Release process• Testing organization management
Release process
Release Process
• Start from the MRD• End with the FCS (first customer ship)
敏捷开发与持续集成 – 迭代中的持续集成
Test Organization management
Thank you!