Modeling expert memory search, Modeling expert memory search, knowledge access, and decision knowledge access, and decision making:making: A model of crossword puzzle play A model of crossword puzzle play

Shane T. Mueller & Kejkaew Thanasuan

Department of Cognitive and Learning Sciences

Michigan Technological University

1

Why Study crossword?Why Study crossword? Naturalistic approach to

studying decision making and knowledge-based problem solving

Experts in this task have highly-developed memory encoding and retrieval skills

The knowledge space is well-characterized

Crossword ExpertiseCrossword Expertise

3

Top players are 2-5x faster than good casual players, who are 10x faster than novices

Crossword ExpertiseCrossword Expertise4

Expert Novice

GoalGoal

To understand crossword skill by developing a computational model of crossword solving.

Rely heavily on constraints from a natural corpus.

Understand how 'cues' provide both memory access and constraint

Schematic Model ofSchematic Model ofKnowledge accessKnowledge access

6

Memory search and Access Memory search and Access

7

Any feature in the clue (word stem or hint) will activate a set of answer words according to the relative probability.

Probabilities of multiple features can combine to form activation distribution.

Orthographic RepresentationOrthographic Representation

8

Each answer has associations from orthographic units in the clues that lead to that answer.

Currently, lexical units include:

letters

adjacent letter pairs

length units Based on Mueller & Thanasuan (2014,

JMP) model of word-stem completion.

9

Used (250K token/4M word) Ginsberg database (described later)

Each word coded for 26 letter-features + 27^2 letter-pair features

A bank of 10 (logarithmically-defined) length features.

Orthographic CorpusOrthographic Corpus

250K

26 729 10

Orthographic Search through Orthographic Search through activation spaceactivation space

TU-K--

TUSK TURK TUSKS TUCK TUCKS TURKS TURKEY TIMBUKTU TUCKIN TUTU

---K--

SKATE KOREA OSAKA ANKLE KNEE KOALA KNEES SKEET ASKED AKRON

0.05923629 0.04718652 0.03634697 0.02930575 0.025934 0.0205008 0.01378494 0.01108984 0.01044018 0.01013701

0.0009366936 0.0009117806 0.0009024288 0.0008888983 0.0008640101 0.0008119471 0.0007999176 0.0007830753 0.0007498722 0.0007384729

10

Semantic RepresentationSemantic Representation

11

Each answer has associations from lexical units in the clues that lead to that answer.

Currently, lexical units include:

word

word pairs Association strength increases with

each experience.

No effort made to form semantic associations based on contextual semantics (e.g., LSA) or linguistic analysis

12

Ginsberg's Crossword Clue database 4,000,000+ clues from 50K

puzzles 250K unique answers (rows) With stemming by Celex 2.5,

110K clue words (columns) 550K clue word pair units

(columns) Very sparse matrix (.000024

used)

Semantic Knowledge BaseSemantic Knowledge Base

12

250K

110K 550K word-pairs

Semantic ActivationsSemantic Activations“Thanksgiving Bird”

CARVEPIE BASTE EMU TOM TURKEY TURKEYTROT IBIS EGRET MEALPIES WOODYWOODPECKER

ROC EMUS LOON

“Flop”EDSEL BOMB DUD EDSELS ISHTAR KER SANDAL THONG FIASCO ANEGG EARED UTURN HIT SMASH NERD....TURKEY (25)

0.00760 0.00622 0.00581 0.00551 0.005390.02554 0.02339 0.01639 0.00829 0.00822 0.00539 0.00539 0.00502 0.00470 0.00451

0.2300.086 0.072 0.063 0.057 0.032 0.017 0.017 0.015 0.013 0.013 0.0130.011 0.011 0.010.....0.005713

Memory SearchMemory Search

Any clue can provide both semantic and orthographic cues.

Top options get sampled and evaluated against orthographic constraints and (for orthographic route) semantic cues.

Semantic retrievals must get 'recovered' ala SAM, based on their strength.

Parameters control number of candidates that can be checked and recovery probability,

Hypothesis—search goes on in each domain separately and independently.

Single-route models Single-route models

Crossword ExperimentCrossword Experiment One clue at a time. Stimuli: 56 answers, clued with easy/difficult

semantic and easy/difficult orthographic. Difficult: 1 letter; Easy: 3 blanks Clue difficulty selected subjectively 4-11 letter words Conducted in lab (novice) and ACPT, and

on-line (experts)16

Implemented in the Psychology Experiment Building Language (PEBL). See http://pebl.sourceforge.net

17

Semantic ModelSemantic Model Evaluation Evaluation

43 clues For 24, model and data agreed

with pre-determined difficulty

For 7, model and data agreed, opposite of pre-specified difficulty.

For 3, model agreed with pre-specified difficulty but not data

For 9, humans followed prespecified difficulty, but model did not.

Some failures of Semantic ModelSome failures of Semantic Model

ANSWER BANANA

BRAVADO

FREEZERS

JEWELS

PINECONE

SHOELACE

EASY CLUE

"Good source of potassium"

Swaggering show of courage"

"Food storage places”

"Precious stones"

"Fir tree fruit"

A child's sneaker usually lacks this"

HARD CLUE

"split ingredient”

"Swagger"

"Spoilage slowers"

"Safe deposit"

“Wreath adornment"

“It goes over the tongue"

Experiment ResultsExperiment Results

20

Comparison to noviceComparison to novice

21

Expert resultsExpert results

22

Each uses a generate-and- check scheme

Single route models produce no difficulty effect on non-cued route

Single Route ModelsSingle Route Models

23

Qualitative InterpretationQualitative Interpretation

For Experts, we see both semantic and orthographic difficulty.

Semantic difficulty diminished as more orthographic letters are given.

Consistent with cascading or parallel models search along each rout.

Additional RT data may be needed to understand parallel-serial architecture

SummarySummary Difficulty effects can be produced (usually) by relying

SOLELY on statistics of the environment (no free parameters).

'Cues' provide both memory access and constraint.

Experts have better semantic retrieval which gets them started in the puzzle.

Also, improved orthographic completion that eliminates semantic difficulty effects.

In expert memory, Cues provide both activation and constraint.

25