AAA Automated Testing

AAA Automated Testing…For AAA Games

Francesco CarucciTechnical Director

How much does a game cost?• Development costs spiraling up• EA Exec: “Cost of games is crazy!”• Some examples:

– Killzone 2: > 40 M$– Halo 3: > 30 M$– GTA 4: > 100 M$ (rumored)

• Clearly, this is not sustainable in the long term

Development cost partitioning• To keep it simple:

– Producing assets– Writing code (game code, technology,

tools)– Debugging and maintaining code

• Code dictates cost of producing assets– Better tools means less time to create assets– More efficient code means less time spent

optimizing assets

Solutions?• Better production frameworks

– SCRUM or some variation of it• More experienced teams

– Higher senior/junior ratio actually costs less

• Better tools (3rd party engines, off the shelf packages)

• Out sourcing (yay for assets, nay for code)• Higher quality code

Code is not the King• But it’s a first class citizen and we want to keep its cost

down– Higher code quality reduces Cost Of Ownership

• Less bugs• Less over engineering

– Higher code quality means code easier to change and iterate upon

– Higher code quality means more time to be spent on polishing the game

– Higher code quality means less money spent to make a game!

• And a bigger chance to ship on time and hit that market sweet spot

Change Prevention Process• Many teams follow a change management

process whose main goal is to prevent “feature creep”

• It’s effectively a Change Prevention Process• But you can’t design “fun” up front

– Games must be developed iteratively until “fun” is found…

• So?– Embrace Change

Cost of Change– Game Design

requirements change• “The aim system is good, but I would

like it more peachy, you know a bit more of this, but slightly less of that, more rocking, less oompfh. Can you change it?”

– Traditionally Cost Of Change rises sharply over time

• “No, I can’t change it, It would break X and Y and probably Z too, I don’t really know”

– We want to keep Cost Of Change as flat as possible

• “Yes we can!”

Enter Automated Testing…• Automated Testing has been a broadly

accepted tool for several decades in the Software Industry aimed at reducing cost of developing working code

• In fact… (warning: Boring Slides Ahead)

Assessing test-driven development at IBM

• “In a software development group of IBM Retail Store Solutions, we built a non-trivial software system based on a stable standard specification using a disciplined, rigorous unit testing and build approach based on the test- driven development (TDD) practice. Using this practice, we reduced our defect rate by about 50 percent compared to a similar system that was built using an ad-hoc unit testing approach. The project completed on time with minimal development productivity impact. Additionally, the suite of automated unit test cases created via TDD is a reusable and extendable asset that will continue to improve quality over the lifetime of the software system. The test suite will be the basis for quality checks and will serve as a quality contract between all members of the team.”

http://collaboration.csc.ncsu.edu/laurie/Papers/MAXIMILIEN_WILLIAMS.PDFhttp://portal.acm.org/citation.cfm?id=776892

An initial investigation of test driven development in

industry• Simposium on Applied Computing

• “Test Driven Development (TDD) is a software development practice in which unit test cases are incrementally written prior to code implementation. In our research, we ran a set of structured experiments with 24 professional pair programmers. One group developed code using TDD while the other a waterfall-like approach. Both groups developed a small Java program. We found that the TDD developers produced higher quality code, which passed 18% more functional black box test cases. However, TDD developer pairs took 16% more time for development. A moderate correlation between time spent and the resulting quality was established upon analysis. It is conjectured that the resulting high quality of code written using the TDD practice may be due to the granularity of TDD, which may encourage more frequent and tighter verification and validation. Lastly, the programmers which followed a waterfall-like process often did not write the required automated test cases after completing their code, which might be indicative of the tendency among practitioners toward inadequate testing. This observation supports that TDD has the potential of increasing the level of testing in the industry as testing as an integral part of code development”

http://portal.acm.org/citation.cfm?id=952753&dl=GUIDE&coll=GUIDE&CFID=46811546&CFTOKEN=92849791

Google• “Standard Google practices include unit tests and

functional tests, continuous builds, last known good build, release branches with authorized bug fix check-ins, focused teams for release engineering, production engineering and QA, bug tracking and logging of production runs.”

http://agile2007.agilealliance.org/index.php%3Fpage=sub%252F&id=713.html

Professionalism and Test-Driven

Development • “Test-driven development is a discipline

that helps professional software developers ship clean, flexible code that works, on time. In this article, the author discusses how test-driven development can help software developers achieve a higher degree of professionalism”

http://ieeexplore.ieee.org/Xplore/login.jsp?url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F52%2F4163008%2F04163026.pdf&authDecision=-203

Unit Testing in Games• “In our experience, the introduction of

automated tests and continuous integration makes development teams more efficient and results in more reliable, and often quite simply, better software. Additionally, it reduces the pressure and workload on development teams by reducing the effort for manual testing, and allows bugs to be found earlier in the development process. Certainly, automated tests alone won't make your game a hit. But almost as certainly, they will make life easier for developers, artists, project leaders, producers and even players.”

http://www.gamasutra.com/features/20050329/roken_pfv.htm

Unit Testing in Games (contd…)

• “Test-driven development can be a very effective development technique. We have successfully applied it to game development in a variety of situations, and we’re convinced of the many benefits it has provided us. Right now, the idea of writing code without writing tests first feels quite alien to most of us, and we treat TDD like the scaffolding in building construction: a necessary tool that will not be shipped to the customer but that helps tremendously during development”

http://gamesfromwithin.com/?p=50

Unit Testing in Games (contd…)

• “Wir haben in den letzten fünf Jahren ausschließlich positive Erfahrungen mit automatisierten Tests und Continuous Integration gesammelt. Die automatisierten Tests sichern Stabilität auf hohem Niveau, während Continuous Integration zusätzlich eine deutliche Arbeitserleichterung darstellt. Der Aufwand für die Einrichtung und Wartung der Verfahren ist überschaubar und hat sich schnell amortisiert. Wenn man die anfängliche Skepsis überwindet, möchte man unserer Erfahrung nach diese Art der automatisierten Qualitätssicherung sehr schnell nie mehr missen.”

http://www.igda.dimajix.net/fileadmin/ressources/report-03/07_roeken_summary.pdf

What’s an Automated Unit Test• A snippet of code that exercise a Unit

Under Test (method of a class), assert its correctness and is executed automatically every time the system is built– Running unit test is effectively a

compilation step• Compiler checks if the code is syntactically

correct• Unit tests check if the code is semantically

correct

Test First vs Test Last• Writing automated tests is accepted to produce higher

quality code at lower cost– More tests == less time spent debugging

• Should tests be written after production code? (Test-Last)• Should tests be written before production code? (Test-

First)• “This observation supports that TDD has the

potential of increasing the level of testing in the industry as testing as an integral part of code development”

• Test Driven Development == Test-First

Test-Driven Development• Write a failing test before

the code is written• Write the simplest code

that could possibly make the test pass

• Refactor the code to eliminate all possible duplications and code smells

• Red Green Refactor– That’s the coding

rhythm

A Failing Test• First write a failing test:

– CRY_TEST (FireButtonPressAndRelease){

StartFire();ASSERT_IS_TRUE(IsFiring());

}

– The test is dictating the interface and the contract of the unit being written:

• When StartFire method is invoked, the weapon is firing

• … then implement the minimal code to make it pass– bool IsFiring() const

{return true;

}

Another Failing Test• Write another failing test to dictate more

behavior:– CRY_TEST (FireButtonPressAndRelease)

{ StartFire();

StopFire();ASSERT_IS_FALSE(IsFiring());

}

– When StartFire method is invoked, and then StopFire method is invoked, the weapon is not firing.

– The actual word explanation is redundant, the whole information needed is in the code itself in readable form!

Another Failing Test (contd.)• Minimal implementation:

– bool StartFiring(){

m_isFiring = true;}

bool StopFiring(){

m_isFiring = false;}

bool IsFiring() const {

return m_isFiring;}

How does it work• Tests are small, simple, straightforward and test only one

condition– Long and complicated tests must be avoided

• Tests are self documenting• Tests describe the design of the code to be written

– Contract– Invariants– Post-conditions– Calling conventions

• A TDD cycle should take few minutes (often less than a minute)– 30 minutes to write a test means a bad test– Break bigger tests in smaller steps (help test coverage)– Begin with baby steps, increase step size when

becoming more comfortable

Why TDD?• Small and clearly defined steps lead to the final implementation• No over engineering: write only the code that makes failing test

pass– Minimal maintenance– Less code for the same needed functionality (including

tests!) – Less time needed to change the code when requirements

change• Less coupling between units

– Less cost to maintain units• Tests are a by product of design and code construction

– They keep contracts tested automatically through the life of the project

– Code is easier to modify, broken contracts easier to discover– Defects and design inconsistencies are found early on

• It costs less to fix fewer bugs!

When TDD is not appropriate• Exploratory coding, throw away

prototype code to quickly explore different solutions for a problem– But the proper solution must be

implemented test-driven to reduce cost

• Code wrapping around existing libraries• GUI layer• Very low level code dealing with the

hardware

Excuses for not Testing

Excuses for not Testing (contd.)

• … writing tests takes time and this time is stolen from writing code– People forget about the time spent debugging and

maintaining code which is much larger than the time spent writing it

– Tests written test-first are part of the design process, that has to be done anyway. TDD is just a formalized way of designing code. Overall it takes less time to produce working code test-driven

– When the code base is covered, modifying code is safer and takes less time cause of regression provided by the automated tests

– By writing tests first, fewer bugs are experienced by working with generally tested code (no huge amount of time spent debugging)


• … tests can’t cover everything– Some test coverage is still better than

no test coverage at all– Legacy code shouldn’t be tested unless

it’s being refactored– Rome wasn’t built in a day, but it was

built eventually (and conquered the known world )


• … but this code is temporary and I will change it later, I don’t need to test it now– All code will be modified soon or later: making

code simpler to change is the main reason to write it test-driven: reduce over engineering to the minimum, keep it simple

– Requirements change: modify the affected tests and production code. If unit tested, the code base is more decoupled and easier to modify with less side effects


• … game development is different: automated testing might not work– Game development as a whole is different…– … but writing code is the same if not simpler– Game code is mostly algorithmic, which is easier to test

than GUI code, for example– An A* algorithm is mostly the same, be it in a game or in

a web crawler– Game code changes often: more changes means more

pressure for automated tests to protect against side effects


• … games are about fun: you can’t capture fun with automated tests– But you can capture bugs, and less time

spent debugging means more time looking for the “fun” part

– Writing a test for a bug fix ensures that bug will not resurface later on• 30% of bug fixes create a new bug caused

by side effects


• … this is AAA Game, it’s too risky to do automated testing on it– AAA game means highly polished, highly technologic,

great game that delivers great fun– In order to achieve AAA, the most efficient production

tools are needed, Automated Testing is an efficient tool– It’s too risky to NOT do automated testing on a AAA

game– Be among the first to gain the competitive advantage

and improve the Industry

A Slice of a bigger Pie• Automated tests (even test-driven) don’t solve all

problems – Chatting up a girl will still be a problem for a programmer

• Automated tests must be part of a wider agile

framework including:– Continuous integration– Team code ownership– Daily deliverables– Simple code that works– Functional and Integration testing

• These practices work together and strengthen each other with the goal of producing more functionalities in less time.

What’s next?• More automated tests beyond Unit testing• Functional testing

– Test more units functioning together at higher level– Requires more time to build a Functional Testing

Framework– Can capture bugs generated by the interaction of

different units

• Smoke testing– Load different levels after every check-in looking for

crashes– Run scripted sequences looking for crashes– Generate random inputs looking for crashes

What’s next? (contd.)• Asset testing

– Load different assets from the engine to check for crashes and correctness

• Animations, Meshes, Textures

– Example: “a Mesh must be smaller than 5000 polygons if it’s a character, 20000 polygons if it’s a building and must have a physics mesh smaller than 100 polygons associated”

• It requires metadata• Run tests after every asset check-in by content

creators! • Validating assets before they get into the build can

save an enormous amount of debugging time

Adopting Automated Testing• It doesn’t matter how much you read about it, it’s like riding a

bike: you need to try it to learn it and understand it• It needs strong leadership, clear vision, courage and a company

willing to invest in Agile Practices– Lots of money to be saved and huge competitive advantage

are at stake here– Don’t take decisions without understanding the

issues• Automated Testing often goes against perceived common

practices in the Game Industry– It needs a lot of mentoring and training (writing good tests is

difficult)– It needs strong commitment by management– The whole team must be bought into it– The rogue primadonnas unwilling to improve their skills

must be removed from the team

Questions?• Shoot, I can take it • Email:

– [email protected]– [email protected]

AAA Automated Testing

Technology

Transcript of AAA Automated Testing