Liam Curtis Unit 77

Assignment 1

Testing for games – Bugs and the Testing Process

Introduction

This assignment will cover the key elements in testing and the types of bugs or errors that

can be found and where they could occur. To begin I will explain the two categories for bugs

in terms of when they can occur, why it happens and how it can be fixed.

BUGS AND ERRORS

Runtime and Compiler errors

Runtime and Compiler errors are two states that a bug or error can appear in,

depending on which state the bug appears in will determine how difficult it will be to re

create or remove. A compiler error occurs before the game, or any coded software, has a

chance to enter a runtime state so before it finishes booting up. Fortunately a compiler

error is often easy to

solve as a programmer

will know when it is

fixed as the game will

be able to boot. FIG.A

shows a basic compiler

error where a single

misspelled word causes the entire application to cease functioning as the compiling process

was interrupted by this error. The mistake is easily corrected by changing the word to its

actual spelling after which the game is able to compile properly and start. This isn’t however

the only way a compiler error can occur, some compiler errors may occur due to a hardware

or software conflict.

Runtime errors are a little more

complicated as they do not prevent the

game from running. Runtime bugs and

errors are often caused by an intended

function not functioning in the desired

way it is easy to understand this but, to

find the error in the code is much more

difficult then with a compiler error.

Runtime errors can have very specific

requirements to actually occur and to

recreate specific circumstances that

cause the error is usually difficult and time consuming. FIG.B shows a screenshot of my

PlayStation mobile game running in debug mode on PC. At first glance nothing appears to be

wrong apart from quite a few enemy instances on screen, this is the error. Because of a fault

in the code more enemies appear on screen than intended. In this situation the runtime

FIG.A

FIG.B

Liam Curtis Unit 77

Assignment 1

error was easily fixed as the cause of the problem is already known or at least the

programmer would know where to fix the code as it is linked to the instance generating of

the enemy sprites but there are many run time errors that will occur that are not so easily

explained from defective collision physics to miscalculated scores.

When an error isn’t so obvious that it can’t be solved without too much thought,

bugs can be classified further to help narrow down the cause one of these classifications is

known as defect types.

Defect Types – Functions and Assignments

Defect types occur entirely by human error and as such are probably the most

common types of bugs to occur. There are two types of defect that create problems slightly

differently, function and assignment errors.

Function errors are almost non-errors, a function error will occur when a function is

told to do something that it

wasn’t intended to do but still

works and does what it wasn’t

intended to do. The reason that a

function error is almost not an

error is because it is still

functional but due to a mistake

by the programmer the function

is different. A function error can

be difficult to find as it is

misleading if the programmer believes that the particular function was coded correctly then

time will be wasted searching elsewhere to see if another section of the code is conflicting

with the function in question. Function errors are very common and there is a particularly

notorious one in the Civilisation series of games, FIG.C shows images from the games in

which an NPC, Ghandi, sends nukes to destroy his enemy’s territories in the first round. This

action was originally completely unintended but the game still functioned properly during

this action and afterwards. To this day this bug has not been changed or fixed partially

because of fan feedback to keep it in the game but it is also speculated that a fix just was

unable to be found.

Assignment errors are the other form of defect type, these errors are caused once

again by human error and so the game will think nothing of it and assume it is correct. The

difference between a function error and an assignment error is function errors are often

caused by code being written in the wrong way, assignment errors are the cause of

incorrect variables set at the wrong value given to an object, class or function. An

assignment error can vary slightly, it could either be that the wrong variable is sent to a

value and possibly increases or decreases the value by a unintended amount, or the variable

could change the value in the wrong way for instance instead of adding to the value as

intended the value could be subtracted from instead.

FIG.C

Liam Curtis Unit 77

Assignment 1

If a problem is occurring and it doesn’t originate from human error than it is likely a

software or hardware issue, these errors and bugs are caused by defect triggers.

Defect Triggers – Exception, start-up, stress, restart

Errors or bugs created by a defect trigger can be rather malicious and difficult to deal

with, some are hardware based others are software based and can occur in quite a few

different instances. An exception defect occurs when the game is unable to understand

what it should do when something random happens; this could be anything from a

controller being unplugged to losing connection to the internet during an online session in

game. Most exception

defects will result in a

crash as the game will not

know what to do next and

stay on its current frame

with no instructions with

what to do afterwards.

Exception defects are

usually easy to fix but the

less obvious exceptions

are hard to find, dealing

with them is as easy as creating a function that displays a message informing the player of

the issue that has occurred and then maybe returning them to the main menu depending on

the issue an example of this can be seen in FIG.D in which the player has lost connection and

has returned to the menu

Another possible defect that can occur is a start up error. Errors that occur during

initialisation of the game are often caused by a compiling error however a start up defect

error can be caused by elements outside of the games code. It is possible that another piece

of software on the system is

installed in such a way that

conflicts with the game on boot

up which prevents the game

from starting at all or perhaps a

critical piece of software or

driver is not updated that is

needed to run the game. While

unlikely, it is also possible that

a PC will not have the required

system specifications to allow

the game to function at all.

FIG.E is a screenshot from Minecraft telling the user that their video drivers are out of date

and prevented the game from starting, start up defects are usually random and cannot

FIG.D

FIG.E

Liam Curtis Unit 77

Assignment 1

always be solved by the programmer especially on PC unlike compiler errors. When it comes

to start up defects it can fall on the consumer to figure out what is wrong.

Sometimes a system won’t be able to handle everything a game can throw at it. Too

much happening on screen can put a strain on the hardware being used to run the game. It

is the job of the game developer to optimise a game to the best of their ability so that it can

run as smoothly as possible on the platform that it is being sold on. Console games are

entirely the game developer’s responsibility for the game running smoothly while a game on

PC will rely on the individual’s hardware, although good optimization will allow a game to

run on as many PCs as possible. While the hardware will always limit how much can be done

with a game it is always down to the software that will create stress and as such there are

particular things developers need to do to prevent pointless stress. A good example of too

much stress in a game could come from Dead Rising 3. FIG.F is a screenshot from Dead

Rising 3 where many zombies are swarming a car, the team behind the game were very

ambitious with their design choices insisting that they could have up to 500 zombies on

screen at once. This design choice is argued however to be too ambitious for the system

that it was sold on. The extreme number of AI (artificial intelligence) enemies on the screen

at a time can lower the targeted 30 fps (frames per second) to a mere 16 fps, further stress

added by explosions and particle

effects can slow the game to a crawl

leaving the game playable but almost

unbearable.

The last defective trigger that is

capable of causing game bugs, errors

and crashes is during a restart. Restart

refers to the game in a reset state so

not necessarily reloading the entire

game but after dying or quitting, or loading a save file as well. Restarts can trigger a variety

of bugs and errors from something as simple as not knowing what scene or menu to load to

after dying or quitting or enemies remaining present even after death. While it is simple to

fix a bug that is caused by a restart, in a game there will often be many ways to die or reach

some kind of game over so one way of restarting might not have the problems of another.

FIG.G is a screenshot from my PlayStation Vita game; this screenshot shows a game over

screen but enemy ships still moving on the

screen. This unintended bug allows the

player to start the game again with the

enemies from the previous attempt still on

screen which can result in an early unfair

death. An easy way to deal with this is to

make it so that all instances of the enemy

ships are destroyed when the player’s lives

reach 0, the screen can temporarily fade out

FIG.F

FIG.G

Liam Curtis Unit 77

Assignment 1

and in again to make the ships disappearing not seem out of place as well.

We have identified that defect triggers are quite often the fault of hardware and or

the software although there is a wider variety of complications between software and

hardware that don’t necessarily come under any of the defect triggers above.

Defect Triggers – Hardware and software configurations

Computers are very complex systems and there are certain things about them that

aren’t fully understood For instance why a piece of software would install in the same place

as another or why removing or attaching a peripheral to the computer can then cause other

drivers to stop working but there are many problems that arise linked to software and

hardware that can create strange bugs and errors in a game.

Hardware is the raw resource that a game needs to function, with consoles you

know what you are getting in every

system as they are required to have the

exact same specs of every one of that

console this also means that driver

clashes and games released on that

system are guaranteed to run, but with

PC every person has a different system

and this often creates problems that

are difficult to predict. A bug existed in

the Titanfall beta on PC that ruptured

models and textures for users running

the game on integrated graphics, FIG.H. It is likely the bug was created because of the

intergraded graphics chips or cards did not have enough memory or processing speed to run

the game in a stable state. Most games on PC are able to automatically detect a dedicated

graphics card over an integrated one but for reasons unknown the Titanfall beta could not,

the solution of course would be to include such a feature that will automatically use

dedicated graphics it is likely that the retail version of Titanfall will do this.

Of course as stated before hardware sometimes isn’t the root of all problems,

software can be as equally to blame. Software problems are in a wide variety it is possible,

for instance, that a game can run perfectly well until another piece of software is installed at

a later date that for some reason has the same file path as the already installed game,

perhaps antivirus software may completely disallow the start up of a game application or a

defective rendering software like Direct X could cause problems similar to that of FIG.H.

All bugs and errors have solutions but if you only know what the bug is it will be

much harder to fix fortunately there are 4 operating regions that a bug or error can occur in.

FIG.H

Liam Curtis Unit 77

Assignment 1

Operating Regions – Pre-game, Game Start, In-Game, Post-Game

Errors and bugs aren’t usually something that vary too much in where they occur, if

the programmer can pinpoint when a bug happens then it makes it that much easier to fix.

Operating regions are 4 states that a game can be in, by identifying which of the 4 states a

bug is reoccurring in it should give the programmer an idea of where in the code he needs

to look for the bug.

The 4 states that most games go through is Pre-game, Game Start, In-Game and

Post-Game. Pre-Game bugs or errors occur before the player has their main interaction with

the game usually at the Main Menu, these bugs aren’t too common as menus are relatively

easy to program. If we think of the menu as a webpage however, with each option there

acting as a link, if a link is broken or links to the wrong place then it will not give the desired

result this would be particularly more of a problem in a menu with lots of options fixes for

errors like this would be generally obvious

and easy to fix.

The second state is the game start

stage. This is the period of initialisation

between the menu and the gameplay.

Problems are more likely to occur here

then at the actual menu as there are

many more processes for the game to go

through. Loading into a game can be the

start of possible errors a particularly

common bug would be if the game was

unable to read a save file. During this

state the game also will be loading assets,

models, textures etc. if a file path for an

asset is broken this would be one state in which the programmer would find out as after the

loading period certain assets may be missing an example of which can be seen in FIG.I.

Fixing the file path is easy and the game start state narrows a problem like this down to this

state or the next, the in-game state.

The in-game sate will present an abundance of runtime errors and bugs as it is the

sate with the most processes to go through. As stated

previously this stage can be responsible for incorrect

textures or broken assets but a lot more bugs are

presented than that. With endless possibilities in-game

bugs can vary from failed interactions to defective

collision and physics. There is also an increased

likelihood of game crashes due to stress on the

hardware, FIG.J shows a screenshot from Minecraft

where an extreme amount of particles are created at

once causing the game to stutter and then crash. While most game would allow the

FIG.I

FIG.J

Liam Curtis Unit 77

Assignment 1

developer to cut down on particle effects open world and sand box games are notoriously

difficult for developers to control the amount of things happening in the game when it is in

the player’s control.

The final state game errors and bugs will occur in is the post-game state. This state

typically does not have to worry about rendering to many new assets but will usually take

place in a game over screen, high score table or a loading screen back to the main menu.

Like with the other states transition to the next will create possibilities of crashes or errors.

Some possible bugs that can occur here are being taken to the wrong place after the post-

game state, replaying the level instead of returning to the main menu for instance, this

could be caused by the player running out of lives but the programmer did not code the

game to recognise that when the player’s lives are at 0 they should get a game over.

Bugs and Errors: The conclusion

This has covered the essential information on how bugs work, what causes them and

how they can be found and possibly dealt with but also how they are categorized for ease of

detection. Bugs are defined in so many different categories quite simply to narrow down the

cause and create a more efficient way of tackling them. Of course in order to find bugs the

game must be tested and that isn’t as simple as sitting down and just playing a game the

next section will go into detail about the process behind testing.

TESTING IN VIDEO GAMES

Intro

Testing isn’t the most complex or even the most difficult task in game development

but it is extremely important. The standards of quality in games has risen and consumers

expect high production values in all areas of the industry a game with plentiful of bugs can

easily stir up a negative aura around it and drive people away from wanting to play it, this is

why testing is so important. Testing allows for a development team to identify problems

with their game and then fix them accordingly, this of course is done in a way that is the

most efficient and effective way to get it done and over the course of a games development

testing will go through a few different phases.

Test Phases – Alpha, Beta, Gold, Cert & Internal and External

Much like the rest of a game’s production testing is done in Phases, these phases

change appropriately the further into development the game gets.

The first stage of testing is known as Alpha testing, this phase is started when the

game reaches a playable state with most gameplay mechanics working. As well as hunting

down early errors and bugs, alpha testing is early enough into a game’s development that

most things are not feature locked, changes can be considered that will affect core

gameplay elements such as level design, how much damage enemies do or how much

health the player has without too much being set in stone that these kinds of changes

Liam Curtis Unit 77

Assignment 1

couldn’t be made. The majority of Alpha Testing is internal with only a few games doing

external alpha testing, this is usually because of the large amount of placeholder assets still

in the game, assets that that exist to temporarily replace an asset that will exist later Voice

rovers, textures etc. and so that the wrong impression isn’t given of the game before it is

complete and in its final form.

Internal testing is a term used for testing that is done inside the development studio

there is no game that will do any kind of external testing without first having done internal

testing. Internal testing will be done in all phases of testing right up until the game’s release,

it allows for a game to be fully tested without the worry of a bad impression given to the

public because of bugs or unfinished content. External testing is the opposite with usually a

portion of a game sent out to the public to test and give feedback themselves. External

testing can either be done in a closed phase, meaning that only those who are invited can

access the test, or an open phase, open allows anyone to download and join. Not only does

external testing provide the development team with bugs they may have not found on their

own but also allows for server stress tests with multiplayer titles. Alpha tests often stick to

internal testing for the purpose of

keeping the game seen in a positive

way by consumers, external beta tests

however is becoming more popular as

more people have access to the

internet.

Beta testing is the next phase

after Alpha testing. In this phase the

game is feature locked meaning that

how the game is played is set in stone, the purpose of the beta is to crush new bugs that

have appeared and test online functionality. This phase is often made external as well as

being done internally, this allows the public to help find more bugs, give feedback and stress

test servers. External beta tests usually start of Closed requiring a sign up, FIG.1A, and then

they may become open after servers have stabilised and the majority of bugs have been

dealt with. After the beta tests are complete a developer

can proclaim their game has gone gold and continue from

there.

Gold testing is where last minute changes can be

made. In its ‘Gold’ state a game is completely playable and

there should be no game breaking bugs that render it

unplayable. The only purpose of the gold test is to play

through the game completely making sure that everything

is working for the most part, there will be some leeway by

the hardware manufactures as games can be patched at a

later date to fix commonly found bugs or rebalance guns

and items at any point after the game’s release.

FIG.1A

FIG.1B

Liam Curtis Unit 77

Assignment 1

Certification, the Final Phase in testing, is done by the hardware manufacturers,

Sony, Microsoft Nintendo etc. or in the case of PC sometimes software platform owners like

valve for steam or EA for origin. In order to pass certification on a platform the game must

meet specific requirements set by the company doing the certification and then it can be

released on their platform. FIG.1B is an example of a certification process for PlayStation

Mobile, the guidelines state that any applicable feature in the game on the guidelines must

be handled in a certain way and work properly. The application will be rejected if the game

does not meet certification just like any other game that is being released on any platform

however larger game like inFamous, Metal Gear or Halo will have much stricter

certifications to pass as they will be played by a larger audience.

Testing is done in phases to ensure that a game is released in a timely manner but

there are also a few additional processes that testing will go through to ensure that a game

runs as well as it possibly can.

Processes

In the previous section I

discussed the process of play testing

which involves simply playing the

game and making notes of each bug

or problem that is come across, this

next section is going to talk about

two different processes that will

occur naturally along the testing

phases.

White box testing is the

process that is done when a game is

tested in its raw form on a PC. This

process is mostly done by coders or programmers with the intention of testing specific areas

of the game in its development state. This helps to identify bugs and errors other people

may not pick up on such as problems originating from the engine rather than the game.

White box testing makes it much quicker to make changes after identifying issues and then

re testing that section again. FIG.1C shows my PS Vita game being white box tested on a PC,

this allows me to quickly play a section of the game and then make a change quickly if I find

any issues. The term ‘white box’ comes from traditionally testing games on PC when PCs

were mainly white, Black box testing refers to consoles as traditionally most consoles were

black.

Black box testing is testing done on the final hardware that the game is intended to

be running on when released, although not all consoles are black anymore. Black box

testing is a very important step in the development of a game, the game before hand will

most likely be built on a very capable PC more so than the final hardware and so new

problems are likely to arise when running on the intended platform. Black box testing on a

FIG.1C

Liam Curtis Unit 77

Assignment 1

PC game works slightly differently, testing a game in its development kit on PC will be

different to testing it in its final

executable file format and so that is

generally how black box testing is

managed for PC with the same testing

being done as if it was being tested on

the final hardware for a console

game. Early when talking about white

box testing I showed my Ps Vita game

being tested on PC, FIG.1D shows my

Vita game being black box tested on

the PS Vita. Black box testing the

game on the final hardware allowed

me to find more bugs then I had on in the white box tests, in turn this allowed me to change

the game significantly to prevent high priority bugs.

So along with normal Play Testing, White Box and Black Box testing allows for a

thorough search of all possible areas that a game can develop issues in. Having talked about

the phases in testing and the processes that go alongside them the only thing that remains

is talk about the testing strategy, the plan and order that is followed in a repeating life cycle

for the entire testing process.

Life Cycle

Testing isn’t something that can be just

done some planning is involved before it can be

carried out, this keeps the testing phases on track

and ensures that the game will be better with

each cycle.

The testing life cycle is easily summed up

with a spiral diagram, FIG.1E, starting with the

design planning or generation. The first step in

the testing cycle is actually having something to

test which is left down to the programmers and designers to put together. A test scenario

will then be put together whether that be a level, tech demo or simulation etc. results will

then be gathered by taking notes on the test as it is played. The section being tested will

continue to be evaluated for a set period of time or until the testers are satisfied that they

have found as many high priority bugs as they possibly can. The information the testers

gathered is then distributed to the necessary individuals who will be in charge of analysing

the data that they have, from here the cycle starts again.

With the information analysed the design teams and programmers will generate new

code and or fix the code that already was there, from there a new test will be sent out to

the testers and so on and so forth. This Testing cycle remains the same all the way through

FIG.1D

FIG.1E

Liam Curtis Unit 77

Assignment 1

the games testing phases from Alpha to Gold and post-game for the release of new content

via DLC (Downloadable Content) and patches. This is the most efficient way to test a game

and makes the transition from alpha to beta and from beta to gold much quicker.

Testing in Video Games Conclusion

Following the testing cycle and making sure the game will run properly in each of the

processes are essential to moving the entire games development along the testing phase’s

pipeline and released into the market. Careful planning creates efficiency and efficiency

means more time that can be spent testing and more time spent testing a game can make

any game feel polished.