Toward quantifying the effect of Toward quantifying the effect of prior training on task performanceprior training on task performance

MURI Annual Review MURI Annual Review September 26-27, 2006September 26-27, 2006

Bill RaymondBill Raymond

OverviewOverview• Project goal:Project goal: Quantify the effects on Quantify the effects on performance of different training methods for performance of different training methods for complex military tasks.complex military tasks.

• Feature decomposition:Feature decomposition: 1.1.Task typeTask type2.2.Training methodTraining method3.3.Performance assessment (context & Performance assessment (context &

measures)measures)4.4.Training principlesTraining principles

• Planning matrix:Planning matrix:- Capture where we know of, and can quantify in Capture where we know of, and can quantify in terms of performance measures, effects of terms of performance measures, effects of training method and performance context on task training method and performance context on task components.components.

• Quantify principles:Quantify principles: - Derive performance functions for points in the Derive performance functions for points in the feature space using empirical data from feature space using empirical data from laboratory tasks.laboratory tasks.

- Generalize performance functions for Generalize performance functions for implementation in IMPRINT modeling tool to implementation in IMPRINT modeling tool to simulate training effects on task performance.simulate training effects on task performance.

Decomposition issuesDecomposition issues• Constraints on decompositionsConstraints on decompositions

Features must relate to experimental Features must relate to experimental designsdesigns Must be able to describe all Must be able to describe all experimental tasks.experimental tasks.

Task, training, and performance Task, training, and performance context features can be no finer than context features can be no finer than experimental manipulations.experimental manipulations.

Features may be different for research Features may be different for research and IMPRINTand IMPRINT Can’t control training in the real Can’t control training in the real world as carefully as in the world as carefully as in the laboratorylaboratory

Not all experimental results will be Not all experimental results will be major effects.major effects.

IMPRINT task categories are already IMPRINT task categories are already defined.defined.

Planning features should converge to Planning features should converge to final IMPRINT features, diverging from final IMPRINT features, diverging from research featuresresearch features

Planning matrix issuesPlanning matrix issues

• What will the matrix construction What will the matrix construction provide?provide? Current and planned research Current and planned research coverage of spacecoverage of space

May be used by us or others for May be used by us or others for future planningfuture planning

Approximation of final IMPRINT Approximation of final IMPRINT training featurestraining features

Initial step in determining the Initial step in determining the generality of performance functions generality of performance functions in the spacein the space

• Training variables - during skill Training variables - during skill learning:learning: How was the skill taught?How was the skill taught? What kind of practice did What kind of practice did learners get?learners get?

How did practice relate to the How did practice relate to the way the skill will be used?way the skill will be used?

• Performance context variables - at Performance context variables - at skill use:skill use: How does expected performance How does expected performance relate to training?relate to training?

How long has it been since How long has it been since training?training?

Did learners get refresher Did learners get refresher training?training?

Starting point:Starting point:Analyzing training and performanceAnalyzing training and performance

Pedagogy

Practice

Performance

}

}

Task, training, and performance matrixTask, training, and performance matrix

Task components

Training features Performan

ce context

Pedagogy

Practice

Visual

Numerical AnalysisInformation processingFine motor - discreteFine motor - continuousGross motor - lightGross motor - heavyCommunication (reading & writing)Communication (oral)

IMPRINT task taxons

IMPRINT task taxons

Data entry Data entry

Data entry

Data entry

Data entry

Data entry

Data entry

Data entry

Data entry

Pedagogy parametersPedagogy parameters• MethodMethod

Instruction (=default)Instruction (=default)DemonstrationDemonstrationSimulationSimulationDiscoveryDiscoveryModeling/mimickingModeling/mimickingImmersionImmersion

• Learning location Learning location (local = default, (local = default, remote/distance)remote/distance)

• Discussion/Question and answer Discussion/Question and answer (no = (no = default, yes)default, yes)

• IndividualizationIndividualization (no = default, yes) (no = default, yes)

Task by Pedagogy parametersTask by Pedagogy parameters

Task components

Pedagogy

Method

Learning

location

Discussion/Q&A?

Individualized?

Visual

Numerical Analysis

Information processingFine motor - discreteFine motor - continuousGross motor - lightGross motor - heavyCommunication (reading & writing)Communication (oral)

IMPRINT task taxons

IMPRINT task taxons

Data entry

Data entry

Data entry

(Instruction)

(Instruction)

(Instruction)Classification

Inst/Discovery

Practice parametersPractice parameters• Scheduling of trials and sessionsScheduling of trials and sessions

NumberNumberSpacing (massed = default, spaced, Spacing (massed = default, spaced, expanding/contracting) expanding/contracting)

Distribution (mixed = default, blocked)Distribution (mixed = default, blocked)• Scope of practiced task Scope of practiced task (partial, whole = default, (partial, whole = default, whole + supplemental)whole + supplemental)

• Depth of processingDepth of processing (no = default, yes)(no = default, yes)• Processing mediationProcessing mediation (no = default, yes)(no = default, yes)• Stimulus–response compatibilityStimulus–response compatibility (yes = default, (yes = default, no)no)

• Time pressureTime pressure (no = default, yes)(no = default, yes)• FeedbackFeedback - presence - presence (no = default, all trials, (no = default, all trials, periodic)periodic)

• Context of practiceContext of practiceDistractor (no = default, yes)Distractor (no = default, yes)Secondary activity (no = default, yes) Secondary activity (no = default, yes)

Task by practiceTask by practiceTask

components

PracticeSchedulin

gScope Processing

depthProcessin

g mediation

Stimulus-response

compatibility

Time pressur

e

Feedback Context

Visual

Numerical Analysis

Information processing

Item repetitio

n, #

Sessions, Spacing

Part/whole

Yes (presentat

ion format)

Yes (prior

knowledge)

No (Input-output Format)

Yes (respons

e & accuracy

)

Distractor/2ndaryactivity (vocal

activity)

Fine motor - discrete

Item repetitio

n, #

Sessions, Spacing

Part/whole

Yes (respons

e & accuracy

)

Distractor/2ndaryactivity (vocal

activity)

Fine motor - continuous

Gross motor - light

Gross motor - heavy

Communication (reading & writing)

Communication (oral)

Data entry

Data entry

Data entry

IMPRINT task taxons

IMPRINT task taxons

Performance context parametersPerformance context parameters

• TransferTransferNew context (relative to training)New context (relative to training)New task (relative to training)New task (relative to training)

• Delay intervalDelay interval (default = none, time (default = none, time period)period)

• Refresher trainingRefresher training (default = no, (default = no, schedule)schedule)

Task by performance parametersTask by performance parameters

Task componentsPerformance context

New context New task Delay

intervalRefresher training

Visual

Numerical Analysis

Information processing

Yes (typing hand, output configuratio

n)

Yes

Fine motor - discrete

Yes (typing hand, output configuratio

n)

Yes

Fine motor - continuousGross motor - light

Gross motor - heavyCommunication (reading & writing)Communication (oral)

IMPRINT task taxons

Data entry

Data entry

Data entry

Data entry

Quantifying training principlesQuantifying training principles• Data Entry used as an exampleData Entry used as an example• Consider two principlesConsider two principles

Practice Practice Learning ( Learning (Power law of Power law of practicepractice))

Skill practice - no item Skill practice - no item repetitionrepetition

Specific learning - item Specific learning - item repetitionrepetition

Prolonged work Prolonged work Diminished performance Diminished performance• Quantify effects for each taxonQuantify effects for each taxon

Cognitive (“Information processing”)Cognitive (“Information processing”)Physical (“Fine motor - discrete”)Physical (“Fine motor - discrete”)

• ……and performance contextand performance contextTransfer to new items (similarity Transfer to new items (similarity dimension)dimension)

Retention of learned skill (refresher Retention of learned skill (refresher training)training)

Skill practice: Quantifying learningSkill practice: Quantifying learningTotal RT for 1 subject

y = -0.0005x + 2.6806

11.5

22.5

33.5

44.5

5

0 100 200 300 400 500 600Item

RT (sec)

• Skill practice improves performance .5 Skill practice improves performance .5 msec/itemmsec/item

Mean decreases 300 msec in 640 (unique) Mean decreases 300 msec in 640 (unique) itemsitems

• Where does skill practice come from?Where does skill practice come from?Repetition of individual digits (and Repetition of individual digits (and pairs of digits?)pairs of digits?)

Cognitive or physical learning?Cognitive or physical learning?Individual differences?Individual differences?

Skill practice: Origin or learningSkill practice: Origin or learningPair repetition?Pair repetition?

• Subjects appear to “chunk” digits 1 & 2, Subjects appear to “chunk” digits 1 & 2, digits 3 & 4digits 3 & 4so they may be learning something about so they may be learning something about pairs of digitspairs of digits

Chunking effect

0

0.2

0.4

0.6

0.8

1

1.2

Digit 1 Digit 2 Digit 3 Digit 4 EnterKeystroke

RT (sec)

Skill practice: Origin of learningSkill practice: Origin of learningPair repetition?Pair repetition?

Pair practice for one subject

y = 0.3473x + 2.478R2 = 0.0007

11.5

22.5

33.5

44.5

5

0.06 0.16 0.26 0.36 0.46Pair repetition per item practiced

Total RT (sec)

]

• Effect of 2-digit chunk practice appears Effect of 2-digit chunk practice appears minimalminimal

Skill practice is general facility at Skill practice is general facility at number typingnumber typing

Skill practice: Type of learningSkill practice: Type of learningPhysical or cognitive?Physical or cognitive?

• Speed improvement occurs on digits 1 and 3Speed improvement occurs on digits 1 and 3Learning is cognitiveLearning is cognitive

Keystroke speed improvementfrom Block 1 to Block 5

-25-20-15-10-505

1015

Digit 1 Digit 2 Digit 3 Digit 4 Enter

Keystroke

RT change (msec)

Skill practice: Individual differences Skill practice: Individual differences

• ““ChunkersChunkers” are 15% slower than “” are 15% slower than “non-non-chunkerschunkers””Appears to be a strategy choiceAppears to be a strategy choicePedagogy - advantage for instruction Pedagogy - advantage for instruction over “discovery”?over “discovery”?

Component times for chunkers and non-chunkers

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

Keystroke and block

RT (sec)

Digit 1

EnterDigit 4

Digit 3Digit 2

Specific learning: Quantifying learningSpecific learning: Quantifying learningTotal RT for one subject

1

1.5

2

2.5

3

3.5

4

4.5

5

0 100 200 300 400 500Item

Total RT (sec)

• Repetitious practice improves performance Repetitious practice improves performance faster initiallyfaster initially

Power law of practicePower law of practice

Total RT for 1 subject

y = -0.0005x + 2.6806

11.5

22.5

33.5

44.5

5

0 100 200 300 400 500 600Item

RT (sec)

Total RT

00

Item

Total RT (sec)

General learning functionsGeneral learning functions

• Performance as a function of number of Performance as a function of number of repetitionsrepetitionsPlanned experimentPlanned experiment

.

.

.?

Total RT

00

Item

Total RT (sec)

General learning functionsGeneral learning functions

• Transfer and retentionTransfer and retentionPlanned experimentPlanned experiment

. . .

New items? Old items?

Transfer RetentionLearning

Prolonged practiceProlonged practice

• Prolonged work results in an increase in Prolonged work results in an increase in errorserrorsAccuracy rate decline of about 1% over Accuracy rate decline of about 1% over 320 items320 items

• Where does the decline in accuracy Where does the decline in accuracy originate?originate?Cognitive or physical fatigue?Cognitive or physical fatigue?

Accuracy decline with prolonged skill practice

0.08

0.085

0.09

0.095

0.1

0.105

0.11

0.115

0.12

1 2 3 4 5Block

Error rate

Prolonged practice: Prolonged practice: Type of performance decline Type of performance decline

• Two types of errors:Two types of errors:Stimulus adjacency errors: Stimulus adjacency errors: 12341234 1244 1244 Key adjacency errors: Key adjacency errors: 1234 1234 1264 1264

• 90% of errors are of these two types90% of errors are of these two types

• Origin of errorsOrigin of errorsStimulus adjacency = cognitiveStimulus adjacency = cognitiveKey adjacency =motor phase, which could Key adjacency =motor phase, which could be motor output planning (cognitive) or be motor output planning (cognitive) or motor execution (execution)motor execution (execution)

Prolonged practice:Prolonged practice:Type of performance declineType of performance decline

• Practice results in an increase in key Practice results in an increase in key adjacency errorsadjacency errors

Accuracy decline occurs during the motor Accuracy decline occurs during the motor phase (which may be both cognitive and phase (which may be both cognitive and physical)physical)

Accuracy for two errors types(no feedback)

0

2

4

6

8

10

12

14

16

0 10 20 30 40 50 60

Items (/10)

Number of errors

Stimulus adjacency errorsKey adjacency errors

Prolonged practice:Prolonged practice:Type of performance declineType of performance decline

• Feedback eliminates the speed-accuracy Feedback eliminates the speed-accuracy trade-offtrade-off

If feedback is cognitive, then the If feedback is cognitive, then the relevant processes in the motoric phase relevant processes in the motoric phase must be cognitivemust be cognitive

Accuracy for two errors types(with feedback)

0

2

4

6

8

10

12

14

16

0 10 20 30 40 50 60

Items (/10)

Number of errors

Stimulus adjacency errorskey adjacency

SummarySummaryTask

components

Training featuresPerforman

ce context

Pedagogy Practice

Informationprocessing (Cognitive)

Method:•Instruction - strategy instruction may improve speed•“Discovery” - some Ss 15% slower

Scheduling:•no reps - speed decrease linear (.5 msec/item)

•item reps - power law (parameters to be determined)

Feedback:•no feedback - accuracy decline (1%/300 items)

•typing/accuracy feedback - no decline

Transfer: Retention:(planned experiment)

Fine motor - discrete(Physical)

Transfer: Retention:(planned experiment)

IMPRINT task taxons

IMPRINT task taxons