Measuring Risk in Stealth Games

Jonathan TremblaySchool of Computer ScienceMcGill University, Montréal

Québec, Canadajtremblay@cs.mcgill.ca

Pedro Andrade TorresCiência da ComputaçãoUniversidade Salvador,

SalvadorBahia, Brasil

pedro@mail.mcgill.ca

Clark VerbruggeSchool of Computer ScienceMcGill University, Montréal

Québec, Canadaclump@cs.mcgill.ca

ABSTRACTLevel design for stealth games requires the ability to exploreand understand the possible paths players may take througha given scenario and how they are impacted by different de-sign choices. Good tool support can help by demonstratingthe existence of such paths, but for rapid, interactive design,the relative difficulty of possible solutions also needs to bequantified, in a way that correlates well with human per-ception of risk. Here we propose and evaluate three differ-ent metrics for defining and quantifying the risk of stealthypaths. We validate and compare these measures througha small human study, showing that a simple path-distancemeasure correlates best with human judgement. An evalua-tion of a non-trivial stealth scenario demonstrates the prac-ticality of our approach, and shows how such measures canbe useful in understanding a level design.

KeywordsArtificial Intelligence, Video Games, Stealth

1. INTRODUCTIONStealthy behaviour, sneaking past or up to enemies, is an

interesting and popular mechanism in many First PersonShooter (FPS) and Role Playing Game (RPG) games, withthe design approach even becoming a sub-genre on its own[20]. Developing stealth scenarios, however, is non-trivial—a designer must select locations and parameters for staticand dynamic enemies, the placement of different stealth ob-stacles or enhancers (sound, light/dark), and also considerthe impact of level geometry [14]. The success of a designwith respect to ahead-of-time gameplay goals then dependson the behaviour of actual players within the scenario, some-thing which is traditionally available only as a result of ex-tensive beta-testing, well after the design process should becompleted.

Better stealth level design can be facilitated through ap-propriate interactive design tools, which can use artificial in-telligence to expose the existence of different possible player

paths through a stealth scenario [18]. This gives design-ers the ability to dynamically evaluate the relative existenceof a stealthy path solution. Qualitative evaluation of leveldifficulty or game complexity, though, depends strongly onhow players perceive possible solutions—during immersivegameplay, players will view different stealthy path choicesas more or less likely to result in exposure and thus moreor less dangerous or difficult. Design of a stealth metricthat quantifies this perception is thus an important aspectof stealth level design as such definition is non-existent inthe literature. A reliable measure of stealth difficulty wouldenable possible stealth paths to be algorithmically analyzedduring and interactive with the level design, improving theability of designers to appropriately scale game difficulty,and avoiding the long round trip otherwise required for hu-man evaluation.

We propose three different, intuitively appropriate met-rics for measuring stealth danger. We use a human studyto evaluate these metrics, showing that while our measuresall have a reasonably close correspondence to human per-ception, the (conceptually) simple measure of path-distancecorrelates best with human judgement of relative risk. Ourapproach is integrated into a non-trivial, Unity3D-based de-sign tool, illustrating technical feasibility of our metric eval-uation, and allowing us to further demonstrate applicationof a measurement-based approach to a non-trivial stealthscenario taken from a realistic computer game.

Specific contributions of our work include:

• We propose and describe 3 different metrics for mea-suring player perception of risk in a stealth game con-text. These metrics consider intuitively appropriatefactors such as distance to enemy (Dist), line of sight(LOS), and the presence of “near misses” in being seen(NM).

• Using a human study, we evaluate how well our metricscorrelate with a human ranking of the relative risk ofdifferent generated stealth paths. In this we find that(path) distance to an enemy is likely a dominant factorin evaluating risk, more important than more complexline-of-sight oriented features.

• Finally, we demonstrate application of our metrics toevaluation of the risk distribution of a realistic gamelevel from Metal Gear Solid.

2. BACKGROUNDThe stealth genre is characterized by games that empha-

size stealthy movement (avoiding detection by enemies) as a

fundamental mechanism. A number of examples exist, suchas Mark of the Ninja [8] or Dishonored [2], although the ap-proach is also popular in many combat-oriented, First Per-son Shooter (FPS) and Role-Playing Games (RPG), givingthe player an additional interesting, alternative gameplaystyle that can also be used to save resources (ammo, health,magic, etc.). In this section we will explore the definitionof a stealth game. From this definition we then introduce ahigh-level algorithm that finds undetected paths from a tob.

2.1 Stealth GamesStealth games imply presentation of a challenge to the

player of moving from one location to another, while avoid-ing detection by static and mobile enemy entities. In orderto mitigate the challenge different mechanics are providedto the player. Designs that allow players to hide at differentpoints, exploiting occlusion or shadow, and abilities such asinvisibility, teleportation, etc., help the player. Challenge isincreased by various environmental factors: snow may leavevisible movement traces, metal floors or loose objects mayproduce noise when walked on, alarming enemies in the level,and so forth. A level is created by combining these differ-ent structures together with basic enemy positioning, move-ments and detection abilities. This complex relationship be-tween components will then generate the player’s experience.Heuristically, a level can be considered to be stealth friendlyif the in-game tools that reduce detection are greater thanthe environmental challenges that increase it [14].

Stealth is also encountered in FPSs and RPGs as part ofnormal combat preparation, where players seek to scout outthe environment in order to gain knowledge about enemymovements and placements. This usage constitutes less ofa stealth game in itself, but does not change the main tech-niques involved.

2.2 Finding Undetected PathsOur work presumes the existence of some collection of

stealthy paths through a game level. Such paths may bedetermined by observing and recording successful humanplayers, or through specialized path-finding algorithms thatsearch the state space for undetected routes. For use in in-teractive design the second option is of course necessary.

Our approach to calculating stealthy paths is fully de-scribed in previous work [18], but may be sketched out asfollows. First, we make assumptions reasonable to stealthgames, that enemy movement is deterministic, and that ourpath-finding goal is to guarantee the player’s path does notintersect an enemy’s field of view (FOV). More complex sce-narios where the player clears parts of an area by enticingsome enemies away by being temporarily seen are thus dis-allowed.

Given an enemy’s deterministic path and a geometry witha start and goal position, we can formalize the problem.We begin with an overall space χ consisting of the 2D leveldesign extruded over time. Within this space we can thendefine χfree to be the space where an agent may move freely,not colliding with obstacles or enemy FOVs: χfree = χ −(χobstacles ∪ χFOV). In order to easily search this space wemap χ to a discretized environment using 4p for the stepsin precision. A slice of this space is shown in figure 1. Thisview shows the level at one point in time, using red cellsto represent obstacles (as a screenshot of the tool output

Figure 1: Level discretization: green represents walkableareas, red represents obstacles, orange the field of view ofenemies, and blue and yellow spheres the start and goallocation respectively.

there is some minor perspective), orange the enemies’ FOVs,green representing χfree, and the spheres the starting andending player locations. As this viewpoint is moved throughthe time dimension, enemies will move according to theirdeterministic motions, changing which cells are contained intheir FOV. The result is a full 3D space that represents thecomplete level dynamics.

Within this 3D space, we discover possible stealth pathsthrough a basic path-finding algorithm, beginning at thestart position and reaching the goal position, while avoidingobstacles, FOVs, and constrained by forward time-movementand feasible movement velocities. As our goal is to enablequick analysis in an interactive context, we use a heuristicsearch based on the Rapidly exploring Random Tree (RRT)algorithm [11]. While not optimal, this approach allows usto rapidly generate a large number of paths, with the fur-ther advantage that the randomization properties of RRTbetter approximates human search of the problem space.Other path-finding approaches are possible of course; themain requirement, however, is that we receive a feasiblepath, q ⊆ χfree, consisting of a sequence of points in our3D (x, y, time) space. To inspect this path, we define afunction g that takes as arguments a path q and a time t toreturn a 2D position in the plane: g(q, t) = α, where α isa 2D vector. We access a specific coordinate value of the αtuple using an appropriate subscript, e.g. αx or αy.

3. RELATED WORKOur work is intended to better understand and be able

to model player behaviour in stealth games. A significantamount of previous work has been directed at analyzing andmodelling player behaviours. Informal, early work was doneby Bartle as a categorization of player types [4], he definedfour basic kinds of players, Achievers, Socializers, Explor-ers, and Killers, with each type having different motives tointeract with a game. A game’s success can then be partlyexplained by how much it appeals to the different types, aswell as the ecology of types a game attracts. Subsequent,more formal studies have since confirmed this rough catego-rization [1].

A different approach is taken by Sweetser and Wyeth [15],who introduced the Gameflow model to evaluate player en-joyment in games regardless of type. This model borrowsfrom the flow theory in psychology originally proposed byCsikszentmihalyi, on which they added a level of abstractionspecifically oriented to games. Like Bartle’s work and otherstatistical studies, however, this does not offer a clear formalmodel at how a machine could, without the help of a humanbeing, evaluate a human player’s experience/trace.

A few precise models have been defined to measure playerexperience. In the game series Left 4 Dead 1 and 2 fromValve, an “intensity” metric is used to represent player gameexperience. The metric varies positively when a player isinjured by an opponent, is incapacitated by an opponent,pushed or pulled off a ledge by an opponent, and whennearby opponents die, and is reduced by less stressful stretch-es of gameplay. The AI Director uses this knowledge aboutthe player’s experience to create a tailored game [6]. Measur-ing player experience in this fashion has since been appliedin a number of different contexts, including an analysis ofdifficulty in World of Warcraft [3], and in more recent workby Tremblay et al., where the authors used an intensity met-ric to dynamically adapt companion behaviour [19].

Geometric models have also been applied to game analy-sis. Liapis et al., for example, translated some game designpatterns [5] into simple algorithms that describe symmetry,area control and exploration [10]. They showed how thesedifferent metrics could be used to optimally evolve differentlevels. Metrics algorithms are often closely linked to gen-erative methods for game content [7]; many generative pro-cesses follow a core structure of generating some content,e.g. a game level, followed by using a metric-based utilityfunction on the content to determine the resulting quality[17]. For example, in the work of Togelius and et al., theypresented generative methods to evolve race tracks, measur-ing the result using neural network-based agents [16]. Thisallowed them to evaluate the quality of the track, measur-ing amount of progress, variation in progress, and differencebetween maximum and average speeds.

Perhaps the closest prior work to our presented work is byShi and Crawfis, who presented a design tool that computesmetrics on the optimal path a player may find to get througha level, given obstacles and enemy distribution [13]. Theyconsidered properties such as the minimum damage cover,longest path, and standard deviation of cover points. Weare concentrating on metrics relevant to the stealth gamesgenre, but incorporation of these kinds of FPS metrics wouldbe interesting in more complex situations, where stealth andcombat combine. Both their work and ours fall under Nel-son’s state space characterization strategy [12].

4. METRICSThe amount of danger or risk inherent in a stealthy path

has a close relation to the potential for discovery by the en-emy. Relative proximity of enemies is thus important, asis the direction in which enemies face—if an enemy directlylooks at a player the risk of failure is also increased. Wefirst describe a metric that relies on path distance, followedby one that focuses on the enemy’s relative angle of sight,and finally a more complex metric that tries to measure howclose a player came to being discovered. The different met-rics presented should be looked as propositions for a formaldefinition of risk and will be validated in section 5.

4.1 Dist: Distance to enemyHeuristically, a player close to an enemy risks discovery

more than one far away. Since discovery is typically predi-cated on visual contact, however, this distance measure alsoneeds to take into account game obstacles—an enemy be-hind a thin wall represents less of a danger than one equallyclose but not occluded. Our distance to an enemy measure(Dist) is thus defined in terms of path-distance using an A∗

search within χfree rather than simple Euclidean or Man-hattan distance. In order to compute this metric we needto consider proximity to each enemy at each point in time.We thus define d∗(α, β) to be the path-distance between twoplanar position, α and β. This gives us the equation,

Dist(p) =

T∑t=1

[∑e∈E

1

d∗(g(p, t), g(e, t))3

](1)

In equation 1 p represents a player’s path, and e is an en-emy path from the set of all enemy paths E. T is defined asthe maximum t value in our path p. In the above equationwe use the reciprocal of the distance cubed as a non-linearmeans of scaling intensity as the player gets closer to an en-emy. This is based on our need to have the function weightcloser enemies as much more dangerous than distant ones,and our observations during prototyping that other expo-nents tend to either over-value or under-value proximity.

Variable movement speeds imply a further scaling factormust also be applied to produce meaningful values. Supposethere are two paths, initially identical, and deviating onlyduring the last unit of distance, where despite being wellaway from any enemies one path ends up taking takes twiceas much time as the other. Intuitively, such paths shouldhave very similar values in terms of relative danger, but theaccumulation of terms over time in equation 1 can give theslower path arbitrarily higher danger values, depending onthe relative movement speeds.

To reduce the impact of these extra factors, we normalizethe Dist value by the total length, L, of the path in threedimensions (2D×time).

Dist(p) =Dist(p)

L

A drawback to Dist is that it is relatively expensive tocalculate. Even though A∗ is an efficient search algorithmfor path planning, an A∗ search is done for every enemy in Eand repeated for every time step t. Incremental approachesand cached searches would improve this, as agents typicallyonly move small distances between time steps. Since ourfocus is on evaluating the metric itself rather than optimizingefficiency this remains future work.

4.2 LOS: Considering enemy viewThe risk of discovery is also increased when enemies look

toward a player—the more directly a player is in the line ofsight of an enemy the higher the risk. Our second metric fo-cuses on this factor, emphasizing the relative angle betweenthe enemy’s direct line-of-sight (LOS) and the player.

The first step in this computation is to know if thereexists a direct LOS from the player to any enemies. Ifso, the angle, Angle(v1, v2), between the vector formed byr = g(p, t) − g(e, t) and the direction the enemy is facing,f(e, t) is calculated. For these we define helper functions,

0.5

0.75

Player

0.75

1.0

θ

Enemy

r

f(e,t)

Figure 2: Threshold-based angular cost calculation. Thisfigure illustrates how the Cost function maps angles to[0.5, 1.0].

Vis(p, e, t) =

{1 if r ⊆ χfree

0 otherwise

θ(g(p, t), g(e, t)) = Angle(r, f(e, t))

This angle is then weighted according to the Cost function,considering how far outside the field of view of the enemythe player is, as illustrated in figure 2.

Considering just the angular proximity of the player tothe enemy is of course not enough in itself. An enemy thatis very far away, outside the range of vision, is not muchof a threat, even if they look directly at the player. Wealso have the same concern as with Dist, that the variationsin duration of the path have a significant impact on themetric value. Thus we scale the accumulated angular valuesby dividing by both the Euclidean distance (cubed), andthe total path length, L. Note that in this case Euclideandistance (d) rather than path-distance (d∗) is usable, as wehave already determined a straight line-of-sight exists.

LOS(p) =

T∑t=1

[∑e∈E

Cost(θ(g(p,t),g(e,t)))

d(g(p,t),g(e,t))3Vis(p, e, t)

]L

(2)

4.3 NM: Measuring nearly missesThe last two metrics miss some important information

about the player’s behaviour, in that they do not accountfor the player’s past or future behaviour. A situation thatinvolves a near-miss in terms of discovery, barely avoidingbeing seen, is more risky than one where the player hasample lattitude to easily avoid detection.

Our last metric, LOS, attempts to capture the presenceof such risky manoeuvres. At a given point in time, we lookat the states of the last n positions of our player, as well asthe next m positions in the future using a fixed time-stepM t. If these prior or future positions are exposed to enemyview, as shown in figure 3, the player experienced (or willexperience) a near-miss in terms of detection, suggesting arisky, stressful movement.

The full, window-based risk calculation is shown in equa-tion 3. The function Seen(α, τ) take as argument a planartuple and a specific time τ . The whole equation 3 is intendedto capture the idea that the closer a player passes (or willpass) to an enemy’s FOV, the greater the risk. Note that

PlayerPt-1

Pt-2

Pt-n

…

Pt+1

…Pt+2

Pt+m

Figure 3: A representation of the NM metric time-window.Here enemies can see both near-past and near-future playerpositions, and so even if the path is successful due to chang-ing enemy FOVs, the player is undertaking a risky move-ment.

unlike the previous two metrics NM is an unscaled value,not normalized to the total length of the path. The risky,near-miss behaviours we are trying to capture in this cal-culation imply singularly stressful events that make a pathdangerous, even if the rest of it is relatively safe.

Seen(α, τ) =

{1 if (αx, αy, τ) ∈ χFOV

0 otherwise

W−(p, t, n) =

n∑i=1

(n− i)2 · Seen (g(p, t− i), t)

W+(p, t,m) =

m∑i=1

(m− i)2 · Seen (g(p, t+ i), t)

NM(p) =

T∑t=1

(W−(p, t, n) +W+(p, t,m)

)(3)

An obvious final direction for these metrics would be con-sider a hybrid form, combining the individual metrics insome fashion. Our interests in this work, however, is to firstunderstand the efficacy of the our metrics in isolation beforeaddressing the significant complexity of tuning the weightsof each metric’s contribution in a hybrid. Efficiency is alsoexpected to be a concern, and a hybrid form would need todemonstrate a sufficient trade-off between cost and improve-ment. The experimental work we now present shows thatthe individual metrics already have good predictive power.

5. EXPERIMENTAL RESULTSOur experimental work is aimed at demonstrating and

comparing the value of our metrics with respect to measur-ing stealth difficulty or danger, which will allow us to gaindeeper knowledge of the definition of risk. In this sectionwe first discuss how the metrics behaves differently in a par-ticular scenario, and then use a human study to argue forand compare validity. Finally, we will see a simple exampleapplication of the metrics to understand an in-game stealthlevel.

5.1 Metric BehavioursOur metrics have interesting qualitative differences in how

they measure danger, and thus tend to identify risky be-haviour at different points. Consider an example taken fromlevel 14 of our human study from section 5.2, and shown infigure 5. This figure shows three key-frame moments in

1 2 3Dist

Risk

LOS

NM

Figure 4: Evolution of the presented metrics over time (s)on level 14. The numbers refer to key-frames in Figure 5.

the level. The player is represented by a small blue sphere,and proceeds along the gray trace from the the large bluesphere (bottom right of the center) to the large green sphere(bottom left of the center) positions, while the black barsindicate walls, and the orange areas the (discretized) FOVsof 3 mobile enemy guards.

Figure 4 shows the evolution over time of each of our met-ric values. Note that here these values are not normalizedby the total length of the path. This way we can clearlysee at which key moment the metrics are activated. Eachmetric indicates points of maximal danger at different timesin the player’s progress. Although the Dist metric showssome activity in key-frames 5a and 5c, the peak of the Distmetric occurs in key-frame 5b, where proximity to an enemyis greatest, and the overall accumulated path-distances fromthe player to all the enemies ends up maximized. The LOSmetric, however, considers the point of maximal danger tobe at key-frame 5a, when the player moves directly throughthe line-of-sight of an enemy, if just outside the visual range.The NM metric finds risky behaviour only at key-frame 5c,when the enemy has just turned to look at the earlier pathof the player.

Arguments can be made for each of these that they ac-curately represent the element of risk in this stealthy path.Maximal points, however, clearly differ. In the followingsection subsection we thus look at how these metrics com-pare to human rankings, in order to determine which bestcorresponds to human judgement.

5.2 Human StudyOur human study asked participants to evaluate different

player paths (produced by the algorithm presented in sec-tion 2) and determine which was safer. The degree of corre-lation between participant choices and which paths our met-rics determined safer would thus show whether our metricsmatched human perception, and thus tell us which factorsmost contributed to what players felt was risky.

Participants were presented with an animated gif imageon endless loop showing the movements of two players, in-cluding enemy movements and FOVs. Levels were basedon different, but game-realistic geometries and enemy ar-rangements, attempting to provide a cross-section of levelcomplexity and path safety. Figure 6 shows 3 examples of

the 15 levels1 presented—the small red and blue spheres rep-resented the two players, both beginning and ending at thesame points (choice of red or blue for the different pathswas randomized). After viewing a level, the participant hadto click on a button to select which path was the overallsafest, allowing us to compare the resulting ranking withour individual metric ranks. The study was designed totake about 15–20 minutes to complete (avoiding participantburn-out). The study was conducted under unsupervisedconditions (i.e., on the participant’s web browsers, at theirleisure) and consisted of 27 anonymous participants, mainlydrawn from the graduate and undergraduate population ofour department.

Table 1 summarizes the raw data from our study. Foreach of the 15 levels, and for humans and each metric, weindicate a 1 if the participants or metric selected player 1’spath as safest, 2 if they selected player 2’s path as safest,and 0 if there was no consistent choice. The latter valueonly occurred for human players, where we imposed a 75%threshold on agreement between participants to establish aranking. This count can be recuperated from the first lineof table 1.

Table 1: Human rankings vs. metrics. 1 refers to blue playerand 2 to the red player. 0 represents that no agreement wasreached by the humans.

Level 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

# voted 1 22 1 18 27 23 12 7 7 0 0 4 22 4 22 8

Human 1 2 0 1 1 0 0 0 2 2 2 1 2 1 0Dist 1 2 1 1 1 2 2 2 2 2 2 1 2 1 1LOS 1 2 1 1 2 1 2 2 2 2 2 2 2 2 2NM 0 2 2 1 1 1 1 2 2 2 2 1 2 2 1

As an initial and basic observation, we observe that allmetrics have quite good agreement with the humans. Outof the 10 levels for which human agreement reached our 75%threshold LOS agrees 7 times, and NM agrees 7. Surpris-ingly, given that it is conceptually the simplest, the Distmetric stands out as achieving perfect agreement with thehumans. Even with just 10 of our 15 human judgementsconsidered definitive this is very unlikely if due to randomchance, suggesting that path-distance might be a more im-portant factor than others.

Our expectation after examining the paths found in thelevels we had designed was that levels 2–4 and 9–13 repre-sented situations in which one path was clearly safer thanthe other, while levels 1, 5–8, 14, and 15 were more am-biguous. We thus now explore levels 1, 3, 5, 12, and 14 asexample situations where the outcome either did not matchour expectation, or where our metrics disagreed.

Level 1 - As shown in figure 6a, this level consisted ofa central occlusion, with a single guard blocking one routearound the obstacle and two paths going the other way. Bothpaths easily avoid the enemy guard, with the only significantdifference being that the red path arrived at the goal at thesame time as the enemy was rotating on the right.

The metrics for level 1 shown in table 2 indicate that Distand LOS ranked the blue path safer than the red path, al-though with a very small difference in value. We will revisitthis concept for larger values, but in this case, since we are

1The levels and study are available at http://goo.gl/fGg3pR.

(a) Key 1, t ' 200s. (b) Key 2, t ' 500s. (c) Key 3, t ' 560s.

Figure 5: Different key-frames of level 14.

(a) Level 1 (b) Level 3 (c) Level 5 (d) Level 12

Figure 6: Examples of levels used in the study.

Table 2: Level 1 Metrics

Dist LOS NMblue path (1) 0.00002 0 0red path (2) 0.00003 0.00003 0

interested in ranking the paths, this is a valid ranking. LOSmeasures some danger for the red path whereas no line ofsight exists between the blue player and the enemy. In thecase of NM, since the players’ paths do not cross the enemy’spath, they both were measured as zero. Since the humansclearly identify the blue player being the safest, this sug-gests our NM metric may be too coarse, and that a focus onunusually dangerous events is not sufficient.

Level 3 - This is a tricky level, in that there are two possi-ble ways to get to the goal, and the players proceed throughone or the other; see figure 6b. The north corridor has afast moving enemy walking north/south, and the south cor-ridor has a slow moving enemy walking from east to west.Here the blue player quickly raced through the north corri-dor, while the red player slowly sneaked behind the enemyon the south corridor and then dashed to the goal once nearthe end. Metric data for this level is shown in table 3.

Table 3: Level 3 Metrics

Dist LOS NMblue path (1) 0.00115 0.00362 428203red path (2) 0.00242 0.01243 204

This represents a corner case for NM. The red player re-ceived a small value, mostly because despite closely follow-ing the enemy, the player is far enough away that little costwas attributed as she dashed out and was walking outsidethe cost window. The distance-based weighting the othertwo metrics, however, tends to give a higher value to thered player. Since humans did not achieve good agreement

themselves, however, we can see this as a trade-off in risk—abrief, close call in the blue player’s path is roughly equiva-lent to the long, slow, moderately dangerous progress by thered player.

Level 5 - This level only has a rotating camera with along, narrow FOV; see figure 6c. The blue player walks fullyoutside of this field of view, whereas the red player waitsfor an opening in the rotation to dash directly to the goal.Metrics for this level are shown in table 4.

Table 4: Level 5 Metrics

Dist LOS NMblue path (1) 0.00002 0.01798 0red path (2) 0.00003 0.00223 6155

The humans here again prioritized the distance to enemy,even though the blue player actually spent more time withinthe enemy’s viewpoint. Our different metrics of course pri-oritize this differently: LOS gives greater weight to the an-gular proximity, while the path-intersection of the red playerresults in higher NM.

Level 12 - In this level, shown in figure 6d, the blue playergoes around the long way, but at one point, walks in front ofthe FOV of the east enemy. The red enemy takes more risksby taking the short cut through the middle of the level.

Table 5: Level 12 Metrics

Dist LOS NMblue path (1) 0.00127 28.565 276620red path (2) 0.00191 0.005677 682589

The results in table 5 show that the LOS metric ranks thered player safer than the blue player. This is mainly becausethe blue player walks in front of FOV of the east enemy,causing the metric to add a high cost to that path. Dist and

NM assess the situation better and more like the humans,attributing greater danger to the red player who both comescloser the enemy, and much closer to being seen.

Level 14 - Similar to level 12, the blue player in this levelfollows a roundabout route, trying to avoid contact withenemies, as shown in the key-frames of figure 5. The redplayer (not shown) walks through the enemies’ FOV spacequite quickly.

Table 6: Level 14 Metrics

Dist LOS NMblue path (1) 0.00025 2.36677 1196323red path (2) 0.00242 0.127234 645539

As opposed to level 12, however, the results in table 6show that both LOS and NM rank the red player as beingthe safest, since despite the proximity to enemies the pathmanages to almost never cross in front of the enemy, evenwithin a window of past and future positions. Dist agreesmore with the humans, ranking blue safer since its routetakes it well away from the enemies.

Each of our metrics tries to measure different propertiesof what might be considered risky behaviour in a stealthcontext. Comparison with human perception verifies thatthese factors are indeed important concerns for players aswell, and while confirmation of our results in a larger humanstudy is necessary, suggests there may be a useful rankingof these factors. Human results best correspond to a pure(geometric) distance measure, and our measures based onnear-misses and angular proximity match less well. Thismay also depend on context, however, and a similar explo-ration using a first-person perspective visualization (ratherthan overhead view) would be interesting—near-misses andbeing apparently within an enemy’s FOV may be more im-portant concerns if the player is more immersed in the gamecontext, and is also less able to easily determine the extentof enemy FOV.

5.3 Metrics for level analysisAn interesting application of stealth metrics is with re-

spect to level analysis. Measurement of the different pathsthrough a level gives us an overall measure of level difficulty,as well as distribution of solution difficulty, both of whichcan then be used in tuning level design.

As an initial demonstration of the technique, we thus ap-plied our metrics to the first level of Metal Gear Solid [9],a highly regarded stealth game. In this level the player hasto sneak around two moving enemies within a cargo dockin an attempt to get to the surface exit on the other side.Figure 7 shows the initial (blue) and goal (green) player po-sitions, along with the probability distribution of the playerbeing seen by 2 patrolling enemy guards. This is a complexscenario with multiple path choices. In figure 8 we showa sample of 1500 paths found by the stealth tool, cluster-ing them into 4 main groupings (colours), based on whichcorridor a path traverses (indicated by the circles in Figure8). Note that this clustering is heuristic and based on ourobservations; it is also possible to have one path associatedwith more than one cluster.

Table 7 shows the average and median metric values ofthe different clusters. From this simple data we can seethat the red cluster seems to have safer paths than the rest,

Figure 7: Metal Gear Solid’s first level with a static proba-bility distribution of getting seen by the enemy. The darkerthe orange, the higher the chance of getting seen.

especially if we look at the Dist metric. This was not obviousjust from the simple, time-flattened probabilities shown infigure 7, but does mirror our informal observations, as pathsin this cluster seemed to do quite well at avoiding enemies.

Figure 8: Clustering of feasible paths within the level.

We present both average and median since outlier pathscan easily skew these simple statistics. It is, however, impor-tant to investigate these since they may represent extremalor unusual strategies. Figure 9 shows a detailed view of thedistribution of the Dist metrics for the red and green clus-ters as the average and mean are widely different. From thedistribution, one can observe that even though it is not thetrend, it is possible to achieve the same level of safety inboth the green and red clusters. Smaller peaks in the greencluster suggest sub-clusters of higher risk, likely due to dif-ferences in timing that result in more closely encounteringenemies.

Table 7: Clustering result, showing the average and medianfor each metric.

Metrics Red Blue Green MagentaAvg Med Avg Med Avg Med Avg Med

Dist (×10−3) 0.6 0.2 3.7 0.9 1.8 1.0 0.2 0.9LOS (×10−2) 0.7 0.02 13.8 0.4 1.6 0.01 4.7 0.3

NM (×105) 2.0 1.6 2.8 2.4 2.5 1.9 1.9 0.7

Informed by this kind of data, a designer has a betterunderstanding of her level design, and can use the relativerisk value, as well as the distribution of risk values overdifferent path clusters as a guide for manipulating the level

0.0000 0.0002 0.0004 0.0006 0.0008 0.0010 0.0012 0.0014 0.0016Metric values

0

5

10

15

20

25

30

35N

um

ber

of

path

s in

that

inte

rval

Figure 9: Histogram of red and green clusters for the Distmetric.

design to better balance overall risk, or introduce interestingvariation in strategy choices for players.

6. CONCLUSIONS AND FUTURE WORKQuantitative metrics for measuring player behaviours are

an important element in improving and even automatingaspects of game design. In this work we described 3 non-trivial metrics for measuring risk in a stealth game context.We showed that our metrics correlate with human percep-tion (to varying degrees), and how the resulting data maybe used to understand a level design.

For future work we are interested in further validating andextending our metrics. An investigation of different gamecontexts and presentations would help determine whetherthe importance of the distance metric that we find in ourstudy is indeed general. Optimizing the efficiency of metriccalculation is also a concern. Our main interest, however, isin applying metrics to improve game design, providing inter-active input to the designer, using metrics to quantify designpatterns or principles, and in using dynamically computedrisk values to identify player styles for improved adaptivityin gameplay and NPC behaviours.

7. ACKNOWLEDGEMENTSThis research was supported by the Fonds de recherche du

Quebec - Nature et technologies, and the Natural Sciencesand Engineering Research Council of Canada.

8. REFERENCES[1] A. Alix. Beyond P-1: Who plays online? In

Proceedings of DiGRA 2005 Conference, 2005.

[2] Arkane Studios. Dishonored.http://www.dishonored.com/, 2012.

[3] M. Ashton and C. Verbrugge. Measuring cooperativegameplay pacing in World of Warcraft. In FDG’11:Proceedings of the Sixth International Conference onFoundations of Digital Games, pages 77–83, June2011.

[4] R. A. Bartle. Hearts, clubs, diamonds, spades: Playerswho suit muds. Journal of MUD Research, 1, 1996.

[5] S. Bjork and J. Holopainen. Patterns in Game Design(Game Development Series). Charles River Media,2004.

[6] M. Booth. AI systems of Left 4 Dead.http://www.valvesoftware.com/publications/

2009/ai_systems_of_l4d_mike_booth.pdf, 2009.

[7] K. Compton, J. C. Osborn, and M. Mateas.Generative methods. In The Fourth ProceduralContent Generation in Games workshop, PCG, 2013.

[8] Klei Entertainment. Mark of the Ninja.http://www.markoftheninja.com/, 2012.

[9] Konami Digital Entertainment. Metal Gear Solid,1998. www.metalgearsolid.com.

[10] A. Liapis, G. N. Yannakakis, and J. Togelius. Towardsa generic method of evaluating game levels. InProceedings of the AAAI Artificial Intelligence forInteractive Digital Entertainment Conference, pages30–36, 2013.

[11] S. Morgan and M. Branicky. Sampling-based planningfor discrete spaces. In Proceedings of the InternationalConference on Intelligent Robots and Systems (IROS2004), volume 2, pages 1938–1945, 2004.

[12] M. Nelson. Game metrics without players: Strategiesfor understanding game artifacts, 2011.

[13] Y. Shi and R. Crawfis. Optimal cover placementagainst static enemy positions. In Proceedings of the8th International Conference on Foundations ofDigital Games, FDG 2013, pages 109–116, 2013.

[14] R. Smith. Level-building for stealth gameplay - GameDeveloper Conference.http://www.roningamedeveloper.com/Materials/

RandySmith_GDC_2006.ppt, 2006.

[15] P. Sweetser and P. Wyeth. GameFlow: a model forevaluating player enjoyment in games. Comput.Entertain., 3(3):3–3, July 2005.

[16] J. Togelius, R. De Nardi, and S. Lucas. Towardsautomatic personalised content creation for racinggames. In Computational Intelligence and Games, CIG2007, pages 252–259, 2007.

[17] J. Togelius, G. Yannakakis, K. Stanley, andC. Browne. Search-based procedural contentgeneration: A taxonomy and survey. ComputationalIntelligence and AI in Games, IEEE Transactions on,3(3):172–186, 2011.

[18] J. Tremblay, P. A. Torres, N. Rikovitch, andC. Verbrugge. An exploration tool for predictingstealthy behaviour. In Proceedings of the 2013 AIIDEWorkshop on Artificial Intelligence in the GameDesign Process, IDP 2013, 2013.

[19] J. Tremblay and C. Verbrugge. Adaptive companionsin FPS games. In FDG’13: Proceedings of the 8thInternational Conference on Foundations of DigitalGames, pages 229–236, May 2013.

[20] Wikipedia. Stealth game.https://en.wikipedia.org/wiki/Stealth_game,2013.