Automated Testing Automated Testing In MMP In MMP
DevelopmentDevelopmentControlling Cost, Risk and Controlling Cost, Risk and
ScheduleScheduleLarry MellonLarry Mellon
Austin Game ConferenceAustin Game ConferenceSept 10-11, 2003Sept 10-11, 2003
Major Functions of Major Functions of AutomationAutomation
Repeatable, Synchronized Input Repeatable, Synchronized Input
DebuggingDebugging Performance TuningPerformance Tuning RegressionRegression
Data ManagementData Management
Test ResultsTest ResultsRaw DataRaw Data
Major Weaknesses of Major Weaknesses of AutomationAutomation
JudgmentJudgment
Rigid Inputs, Rigid AnalysisRigid Inputs, Rigid Analysis
Visual EffectsVisual Effects
Semi-Automated TestingSemi-Automated Testing
(1) (2)
(3)AutomationAutomation Manual Manual
Command Steps Command Steps
Validation Steps Validation Steps
Automation: Automation: ArchitectureArchitecture
Startup&
Control
System Under Test
Test ManagerTest Selection/SetupControl N Clients
RT probes
Repeatable, Sync’edTest Inputs
System Under Test
Scripted Use CasesEmulated User Play SessionsMulti-client synchronization
Collection&
Analysis
System Under Test
Report ManagersRaw Data Collection
Aggregation / SummarizationAlarm Triggers
Automated Testing: Automated Testing: TopicsTopics
Business CaseBusiness Case
TSO Applications & ImplementationTSO Applications & Implementation
WrapupWrapup
QuestionsQuestions
Automated Testing: OverviewAutomated Testing: Overview
Customer satisfactionCustomer satisfaction
Development risk / operational costsDevelopment risk / operational costs
Development / Schedule Development / Schedule RiskRisk
How do we know when we are ready to launch?How do we know when we are ready to launch?
How do we know how fast we are going?How do we know how fast we are going?
Complex, Large-Scale SystemComplex, Large-Scale System
Large Development TeamLarge Development Team
Accelerated, Measurable Accelerated, Measurable DevelopmentDevelopment
Repeatable, Synchronized InputRepeatable, Synchronized Input
Data ManagementData Management
DevelopmentDevelopment
Instant Feedback,Rapid Triage,
History
ManagementManagement
StabilityReference Cases
Summarized, AggregatedResults
Current StatusHistoryTrends
Customer SatisfactionCustomer Satisfaction
StabilityStability
New ContentNew Content
Automated testing keeps existing code working while new code is
being added.
Automated testing keeps existing code working while new code is
being added.
Subscription: Profit & Running Subscription: Profit & Running CostsCosts
Profit…
NewContent
Regression
CustomerSupport
Operations
~ $10per
customer
Automation: Lower Running Automation: Lower Running CostCost
Profit…
Regression
CustomerSupport
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Automation: Lower Automation: Lower Running CostRunning Cost
Profit…
CustomerSupport
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Lower Testing CostLower Testing Cost
Automation: Lower Automation: Lower Running CostRunning Cost
Profit…
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Lower Testing CostLower Testing Cost
Happy Customers Don’t CallHappy Customers Don’t Call
Business Case: SummaryBusiness Case: Summary
Tool investment accelerates development, provides tangible
measures of progress
Tool investment accelerates development, provides tangible
measures of progress
Tool investment pays out over entire lifecycle of subscription-based
service
Tool investment pays out over entire lifecycle of subscription-based
service
Automated Testing: Automated Testing: TopicsTopics
Business CaseBusiness Case
The Sims OnlineApplicationsImplementation
The Sims OnlineApplicationsImplementation
WrapupWrapup
QuestionsQuestions
Automated Testing: OverviewAutomated Testing: Overview
TSO: ApplicationsTSO: Applications
Load TestingLoad Testing
QA RegressionQA Regression
StabilityStability
Operations
Development
Load Testing: GoalsLoad Testing: Goals
Expose issues that only occur at scaleExpose issues that only occur at scale Establish hardware requirementsEstablish hardware requirements Establish response is playable @ scaleEstablish response is playable @ scale Emulate true user behaviourEmulate true user behaviour
ClientMetrics
Game Traffic
ResourceMetrics
Debugging Data
Test Driver CPU
Load Control Rig
Server Cluster
Load Testing Team
System Monitors
Internal Probes
Test ClientTest
ClientTest
Client
Test Driver CPU
Test ClientTest
ClientTest
Client
Test Driver CPU
Test ClientTest
ClientTest
Client
Stability Via Monkey TestsStability Via Monkey Tests
Code Repository Compilers
Continual Repetition of Unit / Integration TestsContinual Repetition of Unit / Integration Tests
Reference Servers
Stability Via Monkey TestsStability Via Monkey Tests
Reference Servers
1. Last “known good”2. What works, right now3. Aggregate reporting
1. Last “known good”2. What works, right now3. Aggregate reporting
AgingAging
Moving partsMoving parts
Critical Path: Unit Critical Path: Unit TestingTesting
Failures on the Critical Path block access to much of the game.Worse, unreliable failures…
Failures on the Critical Path block access to much of the game.Worse, unreliable failures…
enter_house ()
Test Case: Can an Avatar Sit in a Chair?
use_object ()
buy_object ()
buy_house ()
create_avatar ()
login ()
daysBug
Introduced
Development
Checkin
Smoke
Regression
Build
Time to Fix
Cost of Detection
ImpactOn
Others
ImpactOn
Others
Working Code
Candidate Code
Pass / Fail,Diagnostics
Development
Checkin
Smoke
Sniff
RegressionPre-Checkin Regression: don’t let broken code into Mainline.
Pre-Checkin Regression: don’t let broken code into Mainline.
Automated Testing: Automated Testing: TopicsTopics
Business CaseBusiness Case
The Sims OnlineApplicationsImplementation
The Sims OnlineApplicationsImplementation
WrapupWrapup
QuestionsQuestions
Automated Testing: OverviewAutomated Testing: Overview
Test Client
Single, Data Driven Test Single, Data Driven Test ClientClient
Regression Load
Single API
ReusableScripts & Data
Testing feature correctness Testing system performance
Test Client
Data Driven ReportingData Driven Reporting
Regression Load
SingleAPI
ReusableScripts & Data
SingleAPI
ConfigurableLogs & Metrics
Key Game StatesPass/Fail
Responsiveness
Test Client Derived From Real Test Client Derived From Real Client Client
Game Client
View
Logic
Presentation LayerPresentation Layer
Test Client Test Client Game Client Game Client
Scripted Player Scripted Player ActionsActions
Script Engine
State
Game GUI
Client-Side Game Logic
Commands
State
Presentation Layer
What Level To Test At?What Level To Test At?
Game Client
MouseClicks
Presentation Layer
Regression: Too Brittle (pixel shift)Load: Too Bulky
Regression: Too Brittle (pixel shift)Load: Too Bulky
View
Logic
What Level To Test At?What Level To Test At?
Game Client
InternalEvents
Presentation Layer
Regression: Too Brittle (Churn Rate vs Logic & Data)
Regression: Too Brittle (Churn Rate vs Logic & Data)
View
Logic
Gameplay: Semantic Gameplay: Semantic AbstractionsAbstractions
NullView ClientView
LogicPresentation LayerChat Enter Lot
Use Object
…Route Avatar
Basic gameplay changes less frequently than UI or protocol implementations.Basic gameplay changes less frequently than UI or protocol implementations.
Scriptable User Play Scriptable User Play SessionsSessions
Set of Presentation Layer “primitives”Set of Presentation Layer “primitives” State probes: peek/poke arbitrary game stateState probes: peek/poke arbitrary game state Synchronization: Synchronization: wait_until, remote_commandwait_until, remote_command
RecordableRecordable…… Test ScriptsTest Scripts: :
Specific / ordered inputsSpecific / ordered inputs Specific game responsesSpecific game responses
Single user play session, multi-user sessionSingle user play session, multi-user session
Toolkit UsabilityToolkit Usability
Data ManagersData Managers
Pushbutton Use: Developers, Testers,Test Farms, Everywhere
Pushbutton Use: Developers, Testers,Test Farms, Everywhere
Continual Process ImprovementTool UI / feature setFalse positives
Continual Process ImprovementTool UI / feature setFalse positives
Automated Testing: Automated Testing: TopicsTopics
Business CaseBusiness Case
WrapupWrapup
QuestionsQuestions
Automated Testing: OverviewAutomated Testing: Overview
TSO Apps & ImplTSO Apps & Impl
WrapupWrapup
Very Useful, Very ReusableVery Useful, Very Reusable
Development AccelerationDevelopment Acceleration
Lower Operational CostsLower Operational Costs
Cautionary TalesCautionary Tales
Tabula RasaTabula Rasa
Tabula RasaTabula Rasa
PreCheckin SniffTestPreCheckin SniffTest Keep Mainline workingKeep Mainline working
Hourly Stability CheckersHourly Stability Checkers Baseline for DevelopersBaseline for Developers
Easy to Use == UsedEasy to Use == Used
Distribute Test Development & Ownership Across Full TeamDistribute Test Development & Ownership Across Full Team
Load Test: Early & OftenLoad Test: Early & Often Break it before LiveBreak it before Live
Executive SupportExecutive Support Radical shifts in ProcessRadical shifts in Process
Dedicated Tools GroupDedicated Tools Group
Cautionary TalesCautionary Tales
Flexible Design Requires Flexible TestsFlexible Design Requires Flexible Tests
Signal To Noise RatioSignal To Noise Ratio
Defects In The Testing SystemDefects In The Testing System
Top Related