Machine Learning in Computer Games

Machine Learning in Computer Games

Marc Ponsen

11/29/04

2

Game AI: The Last Frontier

“Progress in graphics and sound has slowed in recent years… Now, more than ever, good game play is at the forefront and AI is one of the most critical components (Rabin 2004).”

“… Interactive computer games provide a rich environment for incremental research on human-level AI… Populating games with realistic, human-level characters will lead to fun, challenging games with great game play (Laird 2000).”

3

Why Learning of Game AI?

The process of learning in games generally implies the adaptation of behavior for opponent players in order to improve performance

Self-correction Automatically fixing exploits

Creativity Responding intelligently to new situations

Scalability Better entertainment for strong players Better entertainment for weak players

4

Online – during gameplayAdapt to player tacticsAvoid repetition of mistakesRequirements: computationally cheap, effective,

robust, fast learning (Spronck 2004) Offline - before the game is released

Devise new tacticsDiscover exploits

Offline vs. Online Learning

5

Machine Learning Techniques

6

Overview

Reinforcement Learning (RL) Introduction to RLRL application: Dynamic ScriptingDemo NeverWinter Nights ©

Evolutionary Algorithms (EA) Introduction to EAEA application: EA in Stratagus/WargusDemo Wargus

7

Introduction to Reinforcement Learning Techniques Framework based on punishments and rewards Maximize the frequency of rewards Minimize the frequency of punishments RL is popular research area because:

RL can solve wide variety of complex problems RL will find close to optimal solution RL learns as it interacts with the environment

8

How to provide Punishments & Rewards Weight adaptation function based on fitness

score(s) Lookup tables Neural networks

9

When to provide Punishments & Rewards Reward when AI achieves objective or the opponent

finds itself in a state where it can’t achieve its objective

Reward when AI does something to increase the chance of achieving objective (guided rewards)

Punish when opponent has reached objective Punish when AI does something to decrease the

chance of achieving objective (guided negative rewards)

Using both overall and guided (negative) rewards, speeds up the learning process

10

RL application: Dynamic Scripting

Dynamic Scripting (DS) is an online learning technique inspired by RL

Original implementation of DS (Spronck 2004) in the Computer RolePlaying Game NeverWinter Nights ©

11

Dynamic Scripting

Rulebase A

Rulebase B

Script A

Script B

Combat

generate

generate script

script

scripted

scripted

control

control

human control

human control

weight updates

team controlled by human player

team controlled by computer

A

B

12

Dynamic Scripting and Requirements

Computationally Cheap - Script generation and weight updates once per encounter

Effective - Rules are manually designed Robust - Reward/penalty system Fast Learning – Experiments showed that DS

is able to adapt fast to an unchanging tactic

13

NeverWinter Nights © Demo

14

Dynamic Scripting in Wargus

Different rulebases for different game states State transition on constructing a building that

allows new units or new research

15

Rules in Rulebases

12 Build rules 9 Research rules 4 Economy rules 25 Combat rules

AiNeed(AiBarracks)

AiResearch(AiUpgradeArmor1)

AiNeed(AiWorker)

AiForce(1, {AiSoldier, 9})AiWaitForce(1)AiAttackWithForce(1)

16

Script Generation

State 1Rulebase

State nRulebase

StartRule1.1

BuildRule 1.x

Rulen.1

State 20Rulebase

Rule20.20

AttackLoop

Rule20.19

17

Tactics Two `balanced’ tactics

Small Balanced Land Attack (SBLA)Large Balanced Land Attack (LBLA)

Two `rush’ tacticsSoldier Rush (SR)Knight Rush (KR)

18

Dynamic Scripting Test

Dynamic player (using dynamic scripting) plays 100 consecutive games against static player

Randomisation Turning Point (RTP): First game that dynamic player outperforms static player with 90% probability according to a randomisation test (Cohen, 1995)

19

Dynamic Scripting RTP Results

Tactic Tests Low High Avg. Med. >100 Won

SBLA 31 18 99 50 39 0 59.3

LBLA 21 19 79 49 47 0 60.2

SR 10 10 1.2

KR 10 10 2.3

Dynamic Scripting WORKS! It adapts efficiently against SBLA and LBLADynamic Scripting is unable to adapt to the optimized tactics SR and KR

20

Improve Domain-Knowledge Reliability

Successful adaptive game AI is based invariably on domain knowledge of the game it is used in

Step 1: Use EA to evolve new tactics to deal with problematic static opponents

Step 2: Translate evolved tactics to improved domain knowledge (rulebase revision)

21

Introduction to Evolutionary AlgorithmsIdea: Biological analogy on how populations of species evolve over

generations

Step 1: start with a population (each member is a candidate solution)

…

Step 2: Create the next generation by considering evolutionary operations on the population from the previous generation (genetic operators such as mutation or crossover) and a fitness function (only the more fit get to contribute to the next generation)

…

Continue the process until a certain condition is reached

22

Chromosome

Start State 1 State 2 End

State marker Rule x.1 Rule x.2 Rule x.n

State m

Chromosome

State

Rule ID Parameter 1 Parameter p Rule Parameter 2

Start S C1 2 5 def B S E 8 R 15 B 4 3 S

State number x

1 3 4

Rule 1.1 Rule 1.2 Rule 3.1 Rule 3.2 Rule 3.3

State 1 State 3

AiForce(2, {AiSoldier, 5})AiForceRole(“defend”)

23

Genetic Operators State Crossover Rule Replace Mutation Rule Biased Mutation Randomization

Start State 1 State 2 End Parent 1 State 6 State 8 State 12 State 13 State 16 State 19 State 20

Start State 1 State 2 End Child State 6 State 8 State 12 State 13 State 16 State 19 State 20

Start State 1 State 3 End Parent 2 State 4 State 8 State 12 State 13 State 14 State 17 State 20

24

Wargus Demo

25

Evolution Results

Tactic Tests Low High Avg.

SR 10 0.73 0.85 0.78

KR 10 0.71 0.84 0.75

EA was successful in rapidly discovering tactics able to defeat both rush tactics

26

Analysis of Counter-Tactics

Counter `soldier rush’ with (enhanced) `soldier rush’

Against `knight’s rush’Expand territory when defendableQuickly develop strong unitsUse catapults

27

Rulebase Improvement

New rulesAnti-SR: create strong soldiers and attackAnti-KR: if a `knights’ state can be reached, do so

and start building knightsDefendable expansion: build defensive force then

expand Changes

Adjustment of parameters of several combat rules

28

Improved Rulebase Results

Tactic Tests Low High Avg. Med. >100 Won

SBLA 11 10 34 19 14 0 72.5

LBLA 11 10 61 24 26 0 66.4

SR 10 10 27.5

KR 10 10 10.1

18 99 50 39 59.3

19 79 49 47 60.2

1.2

2.3

Enhanced performance against ALL tactics!

29

Conclusions

Dynamic scripting can be successfully implemented in (commercial) CRPG & RTS games

Offline evolutionary learning can design counter-tactics for RTS games

Evolved counter-tactics can be used to improve domain knowledge used by adaptive game AI

30

References

Spronck, P., I. Sprinkhuizen-Kuyper, and E. Postma. 2004a. “Online Adaptation of Game Opponent AI with Dynamic Scripting.”

Sweetser, P. (2002). Current AI in Games: A review Manslow. “Learning and Adaptation” & “Using reinforcement

learning to Solve AI Control Problems” Laird, J.E. 2000. “Human-Level AI's Killer Application:

Computer Game AI.” Ponsen, M. Spronck. P. 2004 “Improving Adaptive AI with

Evolutionary Learning”