1

A Learning Model of a Long, Non-iterative Spreadsheet Task

Frank E. Ritter, Jong W. Kim, and Jaehyon PaikCollege of IST, Penn State

Presented at the ACT-R Workshop, 5 aug 2010

• Learning, long term, large task

• Non-iterated task• Exploration of model

strategies

• Retention• Subtask learning• Individual differences

2

The Spreadsheet Task

• Using the Dismal (Emacs) spreadsheet (Ritter & Wood, 2005)• In AquaEmacs (Reitter et omnia alia, 2009).

• Recorded behavior with RUI logger (Kukreja et al., 2006)

• 14 subtasks, taking about 30 to 10 min. (Kim, 2008)

1. FILE OPEN2. SAVE AS THE FILE WITH INITIALS3. CALCULATE AND FILL IN FREQUENCY COLUMN (B6-B10)4. CALCULATE TOTAL OF FREQUENCY COLUMN IN B135. CALCULATE AND FILL IN NORMALIZATION COLUMN (C1 TO

C5)6. CALCULATE TOTAL OF NORMALIZATION COLUMN IN C137. CALCULATE LENGTH COLUMN8. CALCULATE TOTAL OF LENGTH COLUMN9. CALCULATE TYPED CHARACTERS COLUMN10. CALCULATE TOTAL OF TYPED CHARACTERS COLUMN11. INSERT TWO ROWS AT A0 CELL12. TYPE IN YOUR NAME IN A013. FILL IN CURRENT DATE USING THE DISMAL COMMAND14. SAVE AS PRINTABLE FORMAT

1. FILE OPEN1. Get ready2. Attend to file3. Move to file4. Click on file5. Attend to open file6. Move to open file7. Click on open file8. Attend to dismal file9. Move to dismal file10. Click on dismal file11. Click to choose

2. SAVE AS THE FILE WITH INITIALS1. Attend to dfile2. Move to dfile3. Click on dfile4. Attend to save-buffer-as

….

3

The Models

• 12 models• 1 novice model, 9 rules, 540 chunks + 2 types + goal• 1 expert model, 540 rules (+ 542 chunks, not used)• 10 intermediate models, 0%, 10%, …90% expert

0% had 540 rules and 542 used chunks 10% had 540 rules and 488 used chunks (last 90%)50% had 540 rules and 271 chunks used (last 50%)90% had 540 rules and 54 used chunks (last 10%)

• Could be further distributions and uses of rules and chunks• With of course different transfer, etc.

• We add: 452 keystrokes, 126 moves, and 75 handmoves = 620.6 s added

4

Novice Model

• 9 rules to walk DM task tree, 542 chunks• Production compilation starts immediately

• Learns to stop and retrieve start across trials!• We should reset chunker time when reloading model

• Learned productions are several combinations

• On trial100, 37 rules fire, some retrievals still

QuickTime™ and a decompressor

are needed to see this picture.

5

Expert Model

• 542 rules, 542 chunks, no chunks used in 100% expert

• 54 rules fired at trial100

• Need a clever graphic to show how this happens• Matt Walsh: we need a model of the model

• 5,854 rules learned over 100 trials

QuickTime™ and a decompressor

are needed to see this picture.

6

10 runs, all Models

6

• Lots of learning (mostly proceduralization)

• Looks ok

7

How long does learning go on? (100 runs)

• ~10x real-time• Runs, good speed up• Not linear in log/log space

• Learns out to trial100

• Seems to predict non-power law, due to I/O constraint

Dismal Spread Sheet Model 1.0 with 100 trials

100

1,000

10,000

1 10 100

Trials

Task Completion Time (sec.)

Model /learned rules / fired rules

Novice 5,770 50 5,8580 5,741 60 5,83210 5,576 70 5,88520 5,633 80 5,78730 5,611 90 5,80840 5,727 Expert 5,854

8

Discussion of Models

• Different types of models, • fundamentally novice (finds next subtask), • fundamentally expert (knows next subtask)

• Novice walks memory, gets faster at walking DM, learns actions to do

• Expert 0 has 542 retrievals to do, in order, linked• Expert 100 has 0 retrievals to do, in order, linked• Both learn rules and most strengthen DMs

• Both learn to do more at once, two actions on a single rule• Learns rules they shouldn’t, to do output in same rule,

because /PM not used, /PM would stop chunker [major coal, but fixable we think with /PM]

9

The Environment and Data (Kim, 2008)

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

RUI

Dismal

Vertical mouse

• 30 subjects• Did the task 4 times• Long term retention

data gathered, but not shown here

10

Comparison of Models with Data

• Group data does not match slope

• Group data does not match intercept

• Except trial 4 = expert• Novice best shaped fit• Other problem: humans

separated by day, ACT-R by 0 seconds

• Log/log does not help• People learn faster, and

model is learning too fast because of not using /PM

0200400600800

1,0001,2001,4001,6001,8002,000

0 1 2 3 4 5 6 7 8 9 10

Trials

Task Completion Time (sec.)

Novice

Expertise 50%

Expert

Human data

11

Limitations

• Does not use Perceptual-Motor• Could use PM to stop learning to do all at once

• Does not fit data in good detail• Tasks are organized as a tree• Will need learning in /PM as well

12

Future Work

• Use /PM, will slow down chunker• Use graph of tasks not tree

• Examine effect of chunk activation on rule learning• Examine structure of task and reuse/repractice of

common subtasks• Match individual learning data• Examine retention• Examine how to explain/explore such models

5,000 initial rules and 60,000 learned rules

13

Summary

• Modeling a 30 to 10 min. non-iterative, repeated task with

learning (with later retention modeling to come)

• Enjoyed the chunker and architecture’s speed and trace

• Have 30 subjects of data, logged

• Grappled with 65,000 rules

• Compared to learning data, and lost

• Have shown we can model long-ish term behavior, we did

30 min., and longer (100 min.) is very possible

This work was supported by ONR under contracts N00014-06-1-0164, N00014-09-1-1124, and DTRA under contract 1-09-1-0054.

14

Publications & References

Kim, J. (2008). Procedural skills: From learning to forgetting. Department of Industrial and Manufacturing Engineering, The Pennsylvania State University, University Park, PA.

Kim, J., & Ritter, F. E. (2007). Automatically recording keystrokes in public clusters with RUI: Issues and sample answers. In Proceedings of the 29th Annual Conference of the Cognitive Science Society, 1787. Cognitive Science Society: Austin, TX.

Kukreja, U., Stevenson, W. E., & Ritter, F. E. (2006). RUI—Recording User Input from interfaces under Windows and Mac OS X. Behavior Research Methods, 38(4), 656-659.

Ritter, F. E., & Wood, A. B. (2005). Dismal: A spreadsheet for sequential data analysis and HCI experimentation. Behavior Research Methods, 37(1), 71-81.