Models of Human Models of Human Performance - revisionPerformance - revision

CSCI 4800/6800CSCI 4800/6800Spring 2006Spring 2006

KraemerKraemer

ObjectivesObjectives Introduce theory-based models for Introduce theory-based models for

predicting human performancepredicting human performance Introduce competence-based models Introduce competence-based models

for assessing cognitive activityfor assessing cognitive activity Relate modelling to interactive Relate modelling to interactive

systems design and evaluationsystems design and evaluation

Some Background ReadingSome Background ReadingDix, A et al., 1998, Human-Computer Interaction Dix, A et al., 1998, Human-Computer Interaction

(chapters 6 and 7) London: Prentice Hall(chapters 6 and 7) London: Prentice HallAnderson, J.R., 1983, The Architecture of Cognition, Anderson, J.R., 1983, The Architecture of Cognition,

Harvard, MA: Harvard University PressHarvard, MA: Harvard University PressCard, S.K. et al., 1983, The Psychology of Human-Card, S.K. et al., 1983, The Psychology of Human-

Computer Interaction, Hillsdale, NJ: LEAComputer Interaction, Hillsdale, NJ: LEACarroll, J., 2003, HCI Models, Theories and Carroll, J., 2003, HCI Models, Theories and

Frameworks: towards a multidisciplinary science, Frameworks: towards a multidisciplinary science, (chapters 1, 3, 4, 5) San Francisco, CA: Morgan (chapters 1, 3, 4, 5) San Francisco, CA: Morgan KaufmanKaufman

Why Model Performance? Why Model Performance? Building models can help develop theoryBuilding models can help develop theory

– Models make assumptions explicitModels make assumptions explicit– Models force explanationModels force explanation

Surrogate user:Surrogate user:– Define ‘benchmarks’Define ‘benchmarks’– Evaluate conceptual designsEvaluate conceptual designs– Make design assumptions explicitMake design assumptions explicit

Rationale for design decisionsRationale for design decisions

Why Model Performance?Why Model Performance? Human-computer interaction as Applied Human-computer interaction as Applied

ScienceScience– Theory from cognitive sciences used as basis Theory from cognitive sciences used as basis

for designfor design

– General principles of perceptual, motor and General principles of perceptual, motor and cognitive activitycognitive activity

– Development and testing of theory through Development and testing of theory through modelsmodels

Pros and Cons of ModellingPros and Cons of Modelling PROSPROS

– Consistent description through (semi) formal Consistent description through (semi) formal representationsrepresentations

– Set of ‘typical’ examplesSet of ‘typical’ examples– Allows prediction / description of performanceAllows prediction / description of performance

CONSCONS– Selective (some things don’t fit into models)Selective (some things don’t fit into models)– Assumption of invariabilityAssumption of invariability– Misses creative, flexible, non-standard activityMisses creative, flexible, non-standard activity

Generic Model Process?Generic Model Process?

Define system: {goals, activity, tasks, Define system: {goals, activity, tasks, entities, parameters}entities, parameters}

Abstract to semantic level Abstract to semantic level Define syntax / representationDefine syntax / representation Define interactionDefine interaction Check for consistency and completenessCheck for consistency and completeness Predict / describe performancePredict / describe performance Evaluate resultsEvaluate results Modify modelModify model

User Models in DesignUser Models in Design BenchmarkingBenchmarking

Human Virtual MachinesHuman Virtual Machines

Evaluation of conceptsEvaluation of concepts

Comparison of conceptsComparison of concepts

Analytical prototypingAnalytical prototyping

BenchmarkingBenchmarking What times can users expect to take What times can users expect to take

to perform taskto perform task

– Training criteriaTraining criteria– Evaluation criteria (under ISO9241)Evaluation criteria (under ISO9241)– Product comparisonProduct comparison

Human Virtual MachineHuman Virtual Machine How might the user perform?How might the user perform?

– Make assumptions explicitMake assumptions explicit– Contrast viewsContrast views

Evaluation of ConceptsEvaluation of Concepts Which design could lead to better Which design could lead to better

performance?performance?

– Compare concepts using models prior to Compare concepts using models prior to building prototypebuilding prototype

– Use performance of existing product as Use performance of existing product as benchmarkbenchmark

Performance vs. CompetencePerformance vs. Competence Performance ModelsPerformance Models

– Make statements and predictions about Make statements and predictions about the time, effort or likelihood of error the time, effort or likelihood of error when performing specific tasks;when performing specific tasks;

Competence ModelsCompetence Models– Make statements about what a given Make statements about what a given

user knows and how this knowledge user knows and how this knowledge might be organised.might be organised.

Notions of MemoryNotions of Memory Procedural Procedural

– Knowing howKnowing how– Described in ACT by Production RulesDescribed in ACT by Production Rules

DeclarativeDeclarative– Knowing thatKnowing that– Described in ACT by ‘chunks’Described in ACT by ‘chunks’

Goal StackGoal Stack– A sort of ‘working memory’A sort of ‘working memory’– Holds chunks (goals)Holds chunks (goals)– Top goal pushed (like GOMS)Top goal pushed (like GOMS)– Writeable Writeable

Production SystemsProduction Systems

Rules = (Procedural) KnowledgeRules = (Procedural) Knowledge

Working memory = state of the worldWorking memory = state of the world

Control strategies = way of applying Control strategies = way of applying knowledgeknowledge

Production SystemsProduction Systems

Architecture of a production system:Architecture of a production system:

Rule base Working Memory

Interpreter

The Problem of ControlThe Problem of Control Rules are useless without a useful way to Rules are useless without a useful way to

apply themapply them

Need a consistent, reliable, useful way to Need a consistent, reliable, useful way to control the way rules are appliedcontrol the way rules are applied

Different architectures / systems use Different architectures / systems use different control strategies to produce different control strategies to produce different resultsdifferent results

Production RulesProduction Rules

IF conditionIF conditionTHEN actionTHEN action

e.g., e.g., IF ship is dockedIF ship is docked

And free-floating shipsAnd free-floating shipsTHEN launch shipTHEN launch shipIF dock is freeIF dock is free

And Ship matchesAnd Ship matchesTHEN dock shipTHEN dock ship

The Parsimonious Production The Parsimonious Production Systems Rule NotationSystems Rule Notation

On any cycle, any rule whose conditions are On any cycle, any rule whose conditions are currently satisfied will firecurrently satisfied will fire

Rules must be written so that a single rule will Rules must be written so that a single rule will not fire repeatedlynot fire repeatedly

Only one rule will fire on a cycleOnly one rule will fire on a cycle

All procedural knowledge is explicit in these All procedural knowledge is explicit in these rules rather than being explicit in the interpreterrules rather than being explicit in the interpreter

Why Cognitive Architecture?Why Cognitive Architecture? Computers architectures:Computers architectures:

– Specify components and their Specify components and their connectionsconnections

– Define functions and processesDefine functions and processes

Cognitive Architectures could be seen Cognitive Architectures could be seen as the logical conclusion of the as the logical conclusion of the ‘human-brain-as-computer’ ‘human-brain-as-computer’ hypothesishypothesis

General RequirementsGeneral Requirements Integration of cognition, perception, and actionIntegration of cognition, perception, and action

Robust behavior in the face of error, the Robust behavior in the face of error, the unexpected, and the unknown unexpected, and the unknown

Ability to run in real timeAbility to run in real time

Ability to LearnAbility to Learn

Prediction of human behavior and performancePrediction of human behavior and performance

Adaptive Control of Thought, Adaptive Control of Thought, Rational (ACT-R)Rational (ACT-R)

http://act.psy.cmu.eduhttp://act.psy.cmu.edu

Adaptive Control of Thought, Rational Adaptive Control of Thought, Rational (ACT-R)(ACT-R)

ACT-R symbolic aspect realised over ACT-R symbolic aspect realised over subsymbolic mechanismsubsymbolic mechanism

Symbolic aspect in two parts:Symbolic aspect in two parts:– Production memoryProduction memory– Symbolic memory (declarative memory)Symbolic memory (declarative memory)

Theory of rational analysisTheory of rational analysis

Theory of Rational AnalysisTheory of Rational Analysis Evidence-based assumptions about Evidence-based assumptions about

environment (probabilities)environment (probabilities)

Deriving optimal strategies (Bayesian)Deriving optimal strategies (Bayesian)

Assuming that optimal strategies reflect Assuming that optimal strategies reflect human cognition (either what it actually human cognition (either what it actually does or what it probably ought to do)does or what it probably ought to do)

(Very simple) ACT(Very simple) ACT Network of propositionsNetwork of propositions

Production rules selected via pattern Production rules selected via pattern matching. Production rules coordinate matching. Production rules coordinate retrieval of chunks from symbolic memory retrieval of chunks from symbolic memory and link to environment.and link to environment.

If information in working memory matches If information in working memory matches production rule condition, then fire production rule condition, then fire production ruleproduction rule

ACT*ACT*

Declarativememory

Proceduralmemory

Working memory

Retrieval Storage Match Execution

OUTSIDE WORLD

Encoding Performance

Addition-Factsix U (4); T (1); H (0)

eight

addend1 sum

addend2

Knowledge Representation

1618 +_____34_____1

Goal buffer: add numbers in right-most columnVisual buffer: 6, 8Retrieval buffer: 14

Symbolic / Subsymbolic levelsSymbolic / Subsymbolic levels Symbolic levelSymbolic level

– Information as chunks in declarative memory, Information as chunks in declarative memory, and represented as propositionsand represented as propositions

– Rules as productions in procedural memoryRules as productions in procedural memory Subsymbolic levelSubsymbolic level

– Chunks given parameters which are used to Chunks given parameters which are used to determine the probability that the chunk is determine the probability that the chunk is neededneeded

– Base-level activation (relevance)Base-level activation (relevance)– Context activation (association strengths)Context activation (association strengths)

Conflict resolutionConflict resolution Order production rules by preferenceOrder production rules by preference Select top rule in listSelect top rule in list Preference defined by:Preference defined by:

– Probability that rule will lead to goalProbability that rule will lead to goal– Time associated with ruleTime associated with rule– Likely cost of reaching goal when using Likely cost of reaching goal when using

sequence involving this rulesequence involving this rule

ConclusionsConclusions ACT use simple production systemACT use simple production system

ACT provides some quantitative ACT provides some quantitative prediction of performanceprediction of performance

Rationality = optimal adaptation to Rationality = optimal adaptation to environmentenvironment

States, Operators, And States, Operators, And Reasoning (SOAR)Reasoning (SOAR)

http://www.isi.edu/soar/soar.htmlhttp://www.isi.edu/soar/soar.html

States, Operators, And Reasoning States, Operators, And Reasoning (SOAR)(SOAR)

Sequel of General Problem Solver (Newell Sequel of General Problem Solver (Newell and Simon, 1960)and Simon, 1960)

SOAR seeks to apply SOAR seeks to apply operatorsoperators to to statesstates within a within a problem spaceproblem space to achieve a to achieve a goalgoal. .

SOAR assumes that actor uses all SOAR assumes that actor uses all available knowledge in problem-solvingavailable knowledge in problem-solving

Soar as a Unified Theory of Soar as a Unified Theory of CognitionCognition

Intelligence = problem solving + Intelligence = problem solving + learninglearning

Cognition seen as search in problem Cognition seen as search in problem spacesspaces

All knowledge is encoded as productions All knowledge is encoded as productions a single type of knowledgea single type of knowledge

All learning is done by chunking All learning is done by chunking a single type of learninga single type of learning

Young, R.M., Ritter, F., Jones, G.  1998 Young, R.M., Ritter, F., Jones, G.  1998 

"Online Psychological Soar Tutorial" "Online Psychological Soar Tutorial"

available at: available at: http://www.psychology.nottingham.ac.uk/staff/http://www.psychology.nottingham.ac.uk/staff/Frank.Ritter/pst/pst-tutorial.htmlFrank.Ritter/pst/pst-tutorial.html

SOAR ActivitySOAR Activity

Operators:Operators:  Transform a state via some action  Transform a state via some action

State:State:  A representation of possible stages of   A representation of possible stages of progress in the problemprogress in the problem

Problem space:Problem space:  States and operators that can   States and operators that can be used to achieve a goal.be used to achieve a goal.

Goal:Goal: Some desired situation. Some desired situation.

SOAR ActivitySOAR Activity Problem solving = applying an Operator to a Problem solving = applying an Operator to a

State in order to move through a Problem Space State in order to move through a Problem Space to reach a Goal. to reach a Goal. 

Impasse =   Where an Operator cannot be applied Impasse =   Where an Operator cannot be applied to a State, and so it is not possible to move to a State, and so it is not possible to move forward in the Problem Space. This becomes a forward in the Problem Space. This becomes a new problem to be solved.new problem to be solved.

Soar can Soar can learnlearn by storing solutions to past by storing solutions to past problems as problems as chunkschunks and applying them when it and applying them when it encounters the same problem againencounters the same problem again

SOAR ArchitectureSOAR ArchitectureChunkingmechanism

Production memory

Pattern ActionPattern ActionPattern Action

Decision procedure

Working memoryManager

Preferences Objects

Conflict stackWorking memory

ExplanationExplanation Working MemoryWorking Memory

– Data for current activity, organized into Data for current activity, organized into objectsobjects

Production MemoryProduction Memory– Contains production rulesContains production rules

Chunking mechanismChunking mechanism– Collapses successful sequences of Collapses successful sequences of

operators into chunks for re-useoperators into chunks for re-use

3 levels in soar3 levels in soar Symbolic – the programming levelSymbolic – the programming level

– Rules programmed into Soar that match Rules programmed into Soar that match circumstances and perform specific actionscircumstances and perform specific actions

Problem space – states & goalsProblem space – states & goals– The set of goals, states, operators, and The set of goals, states, operators, and

context.context. Knowledge – embodied in the rulesKnowledge – embodied in the rules

– The knowledge of how to act on the The knowledge of how to act on the problem/world, how to choose between problem/world, how to choose between different operators, and any learned chunks different operators, and any learned chunks from previous problem solvingfrom previous problem solving

How does it work?How does it work? A problem is encoded as a current state A problem is encoded as a current state

and a desired state (goal)and a desired state (goal) Operators are applied to move from one Operators are applied to move from one

state to anotherstate to another There is success if the desired state There is success if the desired state

matches the current statematches the current state Operators are proposed by productions, Operators are proposed by productions,

with preferences biasing choices in with preferences biasing choices in specific circumstancesspecific circumstances

Productions fire in parallelProductions fire in parallel

ImpassesImpasses If no operator is proposed, or if there is a If no operator is proposed, or if there is a

tie between operators, or if Soar does not tie between operators, or if Soar does not know what to do with an operator, there is know what to do with an operator, there is an impassean impasse

When there are impasses, Soar sets a new When there are impasses, Soar sets a new goal (resolve the impasse) and creates a goal (resolve the impasse) and creates a new statenew state

Impasses may be stackedImpasses may be stacked When one impasse is solved, Soar pops up When one impasse is solved, Soar pops up

to the previous goalto the previous goal

ConclusionsConclusions It may be too "unified"It may be too "unified"

– Single learning mechanismSingle learning mechanism– Single knowledge representationSingle knowledge representation– Uniform problem stateUniform problem state

It does not take neuropsychological evidence It does not take neuropsychological evidence into account (cf. ACT-R)into account (cf. ACT-R)

There may be non-symbolic intelligence, e.g. There may be non-symbolic intelligence, e.g. neural nets etc not abstractable to the neural nets etc not abstractable to the symbolic levelsymbolic level

Comparison of ArchitecturesComparison of Architectures

ACT-RACT-R SOARSOARTypeType HybridHybrid SymbolicSymbolic

TheoryTheory Rational Rational analysisanalysis

Problem Problem solvingsolving

BasisBasis Cog. Psy.Cog. Psy. AIAI

LTMLTM Productions; Productions; factsfacts

ProductionsProductions

WMWM Goal stackGoal stack Working Working memorymemory

LearningLearning YesYes YesYes