Supplemental Material

Appendix A: Word List

Abstract Concrete

CONSCIENCE POLITICS KEYS SANDALS

HAZARD ADHERENCE ICE* RUBY

FOLLY LOYALTY MAGAZINE RUG

TOPIC PROPORTION LEAF TOOL

METHOD COWARDICE BLOSSOM TONGUE

AFFECTION WORTH CEREAL* TOES

JUDGMENT TREATMENT BENCH VIOLIN

INSTANT DUTY TRIPOD LIBRARY

QUALITY CONFESSION LADDER* PLASMA

APTITUDE CONDITION PRINTER APRON

TALENT ADVANTAGE CHARCOAL RIBBON

LUCK INNOCENCE RADISH CIRCUS

BOREDOM SPAN WAX ELEVATOR

REGRET AWARENESS HAMMOCK STEREO

BRAVERY OUTCOME CURTAIN PENNY

BET ZEAL MUSTARD CAVE

ILLUSION FORESIGHT MISSILE* BELT

BEING PRESTIGE BEARD SQUARE

ENVY PURPOSE LIME* MUD

DOMAIN WONDER PUZZLE JAM

MANNER CAUSE CACTUS PARCEL

1

BLESSING DECEIT ARMOR SYRINGE

SPIRIT EFFORT DASHBOARD

BUNNY

PREFERENCE EXPRESSION BEETLE COMEDIAN

APPROVAL EXTREME CLOG KANGAROO

THEORY FAILURE DIAPER TROUT

PATIENCE FORFEIT FRACTURE SHACK

WILL FUTURE GLOBE BARK

ENIGMA AMBITION HINGE PUPIL

BELIEF ABUNDANCE KITE SKILLET

CONTEXT INFLUENCE MOWER SPLINTER

FANTASY LACK MOTOR GOWN

TEMPTATION MASTERY PADDLE SHERIFF

PROTOCOL RARITY PEDAL ACCORDION

HATRED NONSENSE PATIO* CAKE

DISCRETION OBSESSION ROOSTER HARBOR

FRAUD PLEASURE ERASER DENIM

MERCY RELIEF STAPLE EAGLE

ADVICE CHAOS TANGERINE RIBS

INTEREST TERROR TORTOISE LASH

Note. * Items removed from analysis because they were at ceiling on recognition accuracy.

2

Appendix B: Mediator Coding Scheme

MEDIATOR AT STUDY

Study Mediator Produced—score for each instance of a 3 repetition item.

Yes (1)—generated a verbal report of a mediator that in principle could be a basis for associative encoding. Ideally, the mediator should include both cue and target, but tokens for the words (especially for abstract items) are sufficient.

No (0)—no words linking the cue and target are provided. A repeating of the cue and/or target, even with non-descriptive other words, is not sufficient, nor is an elaboration on the concepts represented by the cue or target only. Attempts to create a mediator that indicate only one of the words may have been imaged (i.e., partial “mediators”) and word salad.

e.g. CACTUS-PARCEL: “Willy’s it’s one of my favorite restaurants”

e.g. BET-ZEAL : “two people making a bet”

MEDIATOR AT RECALL

Items coded as No (0) for Study Mediator Produced should not get scored at mediator recall. Italics below indicate special cases within a mediator category.

Omission

Yes (1) —no mediator reported at recall even though one was generated at study.

e.g. “I don’t remember my image”, “I’ve got nothing”, no response within time allotted

No (0)—anything else

Verbatim

Yes (1)—verbatim or functionally verbatim (change in tense, part of speech, articles, preposition, or pronouns, and elaboration are allowed). Flipping of subject and predicate, or parts of one of those, is allowed. Only one has to be verbatim if there were multiple generated at study.

Functionally verbatim:

e.g. BLOSSOM-TONGUE: “One of those tongue tattoos with a flower blossom” “A tongue tattoo that had a blossom flower”

Verbatim with unrecalled target:

3

e.g. MISSILE-BELT: “There was a missile in her belt” “There was a missile in her belt”

Recalled “pocket” as target.

No (0)—anything else

Gist

Yes (1)—preserves the main idea/meaning of the study mediator. Omission of non-vital descriptors allowed; change in tense, spelling, part of speech, articles, or pronoun allowed; synonyms and elaboration allowed. Only has to be gist consistent with one mediator if multiple were generated at study.

e.g. MANNERS-PLEASED: “Having good manners and it being the cause of your mother's happiness” “A mother being pleased because her child used good manners”

e.g. BEETLE-COMEDIAN: “I see this huge funny comedian that is sitting in a small beetle car” “And it was the big fat comedian in the small car”

No (0)—anything else

Partial

Yes(1)—omission or substitution of cue or target (or both) or token of either while maintaining important content of the mediator; omission of vital descriptors or of verbs crucial to understanding how the cue and target relate to each other.

e.g. FRACTURE-SHACK: “ he fractured a bone walking in to the shack” “he fractured a bone”

Gist/Partial

e.g. STAPLE-EAGLE: “the eagle couldn't fly because she had a staple in her wing” “the eagle got a staple in her wing”

No (0)—anything else

Commission

Yes (1)—mediating (linking) words are new, such that the main idea/meaning of the mediator is changed

e.g. DOMAIN-WONDER: “she wondered about the domain” “the domain was unknown”

4

Gist/Commission

e.g. FANTASY-LACK: “ a giant library where someone was taking out all of the fairy tale books and burning them” “a person in a library throwing all of the fantasy fiction books off the shelves”

Partial/Commission

e.g. AFFECTION-WORTH: “two people showing affection and it being worth it” “people pleased because they showed affection”

Gist/Partial/Commission

e.g. MISSILE-BELT: “I pictured a belt with missiles on it and the guy could easily fire it at enemies” “ I pictured a belt and it had missiles it can fire off at me”

No (0)—anything else

Intrusion(coded but not analyzed)

Yes (1)—a type of commission error that includes an intra-list study mediator not generated at study for the pair in question. Mediators or parts of mediators from those reported at recall do not count.

e.g. MUSTARD-CAVE: “I see some random cave man eating a hotdog” “This is the birthday party” (birthday party was part of the mediator generated for PATIO-CAKE)

Gist/Intrusion—none generated

Partial/Intrusion—none generated

Commission/Intrusion

e.g. HINGE-PUPIL: “ there was a hinge in her pupil” “her fracture was fixed with a hinge” (fracture was part of the mediator for “FRACTURE-SHACK)

Gist/Partial/Intrusion—none generated

Gist/Commission/Intrusion

e.g. DIAPER-TROUT: “ I imagine a trout wearing a diaper” ”I imagine a trout in a diaper with clogs on” (a kangaroo wearing clogs was the mediator for CLOG-KANGAROO)

Partial/Commission/Intrusion

5

e.g. KITE-SKILLET: “ the kite was near the skillet” ”the magazine was near the kite” (magazine was part of the mediator for MAGAZINE-RUG)

No (0)—anything else

Hybrid (coded but not analyzed)

Yes (1) –a type of commission/intrusion error that includes descriptors or phrases from multiple study mediators

e.g. STAPLE-EAGLE: “ I'm thinking of how rats are a staple part of an eagles diet” “I remember a pedal operating a stapler”

Partial/Commission/Intrusion/Hybrid

e.g. RADISH-CIRCUS: “a clown juggling radishes at a circus” “a cactus with radish blossoms on it” (cactus and blossom are both cues from other trials)

No (0)—anything else

6

Appendix C: Target Recall, Strategy Recall, FOK, and Target Recall-FOK Resolution

The study generated data on all measures for a full set of 80 items (40 concrete and 40

abstract paired-associates). As noted in the main paper, it is more informative to evaluate FOKs

for unrecalled items only, because this analysis parallels the definition of the feeling-of-knowing

as monitoring of the availability of information in memory that cannot currently be retrieved

(i.e., information that could not be recalled). However, some interesting phenomena arise when

the data on cued recall and the FOKs and strategy recall data for all items, including recalled

items, are considered. This supplemental appendix reports on the relevant variables for this

experiment for all items that had been studied, including those that generated successful recall.

Cued Recall and Strategy Recall

Table 1 reports mean cued recall as a function of concreteness and repetition. To parallel

analyses run in the main paper on recognition memory, we conducted a mixed model analysis of

item-level recall outcomes (success, failure) using a generalized mixed multi-level model

estimated in SAS PROC GLIMMIX, using a logit link function. The model specified a random

effect on intercepts (individual differences in associative recall) as well as a random residual

variance component. We do not report all details of the results here. Items varied in

recallability, as reflected in the effect of Cue, F(78, 3294) = 2.42, p< .001. As would be

expected from the literature (e.g., Paivio, 2007; Rowe & Schnore, 1971) and our recent work

(Hertzog, Fulton, Mandviwala, & Dunlosky, 2013), Concreteness had a strong effect on cued

recall,F(1, 3307) = 148.45, p< .001 that persisted over the 7-day delay, even though levels of

recall were lowered by the long retention interval. Repetition had the largest influence on cued

recall, F(1, 3335) = 581.03, p< .001. Theconcreteness X repetition interaction was not reliable in

7

the log-odds space, F< 1, even though it is clear from Table 1 that the linear difference in recall

between concrete and abstract pairs was greater for items presented three times.

We also analyzed the ordinal encoding strategy recall variable described in the main

paper, setting errors to 1, partial mediator recall to 2, gist mediator recall to 3, and verbatim

recall to 4. The analysis was conducted at the item level using SAS PROC MIXED with a

random variance component for intercepts, reflecting individual differences in mean levels of

mediator recall. Items varied in degree of strategy recall, F (78, 3372) = 2.41, p< .001.

Controlling on these item differences, concreteness affected strategy recall,F (1, 3374) = 262.18,

p< .001 with better strategy recall for concrete items, marginal M = 2.12, SE = 0.06, relative to

abstract items, M = 1.64, SE = 0.06(Hertzog, Fulton, et al., 2013). Scaled as Cohen’s d, the

effect in SD units was d = 0.51. The largest effect on recall was associated with Repetition, F(1,

3373) = 1809.10, p< .001, with items presented three times generated much better strategy recall

than items studied only once (marginal M = 2.51, SE = 0.06, versus M = 1.26,SE = 0.03,

respectively, d = 1.36). Concreteness and Repetition also interacted in producing strategy recall,

F(1, 3373) = 77.59, p< .001. Concrete items presented three times for study generated strategy

recall that was, on average, gist recall of the mediator (M = 2.88, SE = 0.06), in contrast to

abstract items presented once (M = 1.14, SE = 0.06), which on average could not be recalled.

Thus, repeated presentations amplified the concreteness effect (d = 0.23 for items presented once

versus d = 0.80 for items presented three times). A concreteness effect on strategy recall is fully

consistent with Hertzog et al. (2013), while also showing that the effect persists after a one-week

retention interval. The repetition effect is new and demonstrates that increased study

opportunities dramatically increases long-term access to studied mediators. As treated in detail in

the main paper, these effects were also observed for unrecalled items alone, setting the stage for

8

strategy recall to be a cue that influences FOKs for unrecalled items and carries some of the

effects of repetition on FOKs.

FOK Magnitude and FOK Resolution for All Items

To evaluate influences on FOK magnitude, we ran a multi-level model with Cue,

Concreteness, and Repetition as independent variables and item-level FOKs as the dependent

variable. The model included a random intercept to capture individual differences in mean

FOKs. This model paralleled the ones reported in the main paper for unrecalled items (see Table

2a for the raw cell means in the 2 X 2 factorial design). There were reliable effects of all

independent variables: Cue, F(78, 3372) = 2.27, p< .001, Concreteness, F(1, 3373) = 234.75,

p< .001, Repetition, F(1, 3372) = 1685.09, p< .001, as well as aconcreteness X repetition

interaction, F(1, 3372) = 71.31, p< .001. Concrete items received higher FOKs (marginal M =

52.34, SE = 2.33) than abstract items (marginal M = 46.52, SE = 2.37), d = 0.23. Items presented

three times generated higher FOKs (marginal M = 55.95, SE = 2.32) than items presented once

(marginal M = 42.91, SE = 2.38), d =0.50. The interaction effect reflected an amplification of

the concreteness effect for items presented three times, d = 0.35, relative to one presentation, d =

0.10, consistent with the effects of these variables on cued recall reported above.

As might be expected, then, cued recall was a potent influence on FOKs. As seen in

Table 2b and Table 2c, FOKs differed dramatically as a function of whether an item was

recalled, with much higher FOKs for items with correctly recalled targets.

A traditional method in metacognitive research for evaluating this relationship is to

compute FOK resolution with respect to cued recall as measured by Goodman-Kruskal gamma

correlations between the two variables. Table 4 shows that FOKs were highly correlated with

9

cued recall(see also Eakin & Hertzog, 2012a). Indeed, the FOKs in this respect behave much

like delayed judgments of learning(Eakin & Hertzog, 2012b; Rhodes & Tauber, 2011).

Interestingly, providing three encoding opportunities reduced the recall-FOK gamma

correlations, F(1, 43) = 12.14, p = .001, with near-perfect gammas for once presented items (M =

.95, SE = .02), but not for thrice-presented items (M = .86, SE = .02), d = 0.51. This is the

opposite pattern that one observes for gamma correlations with recognition memory accuracy

(Hertzog, Dunlosky et al., 2010; see also below). The main paper demonstrates that repetition

increases access to encoding strategy information for unrecalled targets, which in turn seems to

influence FOKs. In contrast, recall success is the dominant cue accessed for once-presented

items. It is likely then that the reduction in gamma correlations seen in Table 4 is attributable to

the emergence of strategy recall information which influences FOKs in addition to recall

outcomes, more so for item presented three times.

Clearly successful target recall is a potent influence on FOKs. At the same time, the

overall set of results, especially those accounting for FOKs for unrecalled items, show that

multiple cues influence FOKs, and that the accessibility of strategy recall in particular is

enhanced when items are studied multiple times(for further treatment of a multiple-cue

utilization perspective, see Hertzog, Hines, & Touron, 2013).

Table 1

Target Recall and Strategy Recall as a Function of Concreteness and Repetition

Target RecallAbstract Concrete

Mean SE Mean SE1 Presentation .02 .01 .11 .013 Presentations .35 .01 .67 .01

Strategy Recall

10

Abstract ConcreteMean SE Mean SE

1 Presentation 1.16 .03 1.37 .033 Presentations 2.15 .03 2.88 .03

Table 2a

Mean FOK (all items) as a Function of Concreteness and Repetition

Abstract ConcreteMean SE Mean SE

1 Presentation 26.65 1.13 31.99 1.073 Presentations 56.21 1.10 77.42 1.06

Table 2b

Mean FOK (recalled items) as a Function of Concreteness and Repetition

Abstract ConcreteMean SE Mean SE

1 Presentation 82.97 4.01 82.62 1.823 Presentations 84.03 1.14 91.52 .72

Table 2c

Mean FOK (unrecalled items) as a Function of Concreteness and Repetition

Abstract ConcreteMean SE Mean SE

1 Presentation 25.34 1.06 25.69 1.063 Presentations 40.75 1.28 47.87 1.76

Table 3

Fitted Mean Goodman-Kruskal Gamma Correlations of FOKs with Target Recall for All Items

from the Mixed Model Analysis

Abstract Concrete

11

N G SE N G SE

1 Presentation 44 .96** .04 44 .95** .02

3 Presentations 44 .85** .02 44 .86** .03

**p< .001

Note.G = Goodman-Kruskal gamma correlation. N indicates the number of persons in each cell

with computable G correlations. SE is the fitted standard error for fitted least-squares means of

the G correlations (see main text for further details).

12

Appendix D: FOK-Recognition Memory Resolution

As indicated in the main paper, we evaluated FOK-recognition memory relationships

using multi-level models. It is traditional in metacognition research to evaluate the resolution of

metacognitive judgments by analyzing within-person ordinal correlations of judgments with

validating outcomes in a two-stage process, first computing Goodman-Kruskal correlations of

judgments and outcomes for each person, and then treating these values as test statistics in a

second stage (see Dunlosky & Metcalfe, 2009 for an introductory treatment; Gonzalez & Nelson,

1996). For this reason, we provide information on gamma correlations from this experiment in

this supplemental appendix. The correlations were computed by excluding items that produced

ceiling recognition memory, thus paralleling the analyses reported in the main paper.

Gamma correlations (G) revealed reliable within-person correlations of FOKs with

recognition memory accuracy for previously unrecalled items, G = .34, SE = .05, p< .001. When

data were broken down into the 2 X 2 Concreteness X Repetition cells (see Table 4), a repeated

measures model in SAS PROC MIXED revealed a reliable effect of repetition, F (1, 42) = 5.21,

p = .03, d = 0.55. Resolution was higher for thrice-presented items (marginal mean G = .44, SE

= .11) than for once-presented items (G = .17, SE = .06). This result replicated earlier findings of

higher resolution for multiply-presented items during encoding (Hertzog, Dunlosky, & Sinclair,

2010). The main effect of Concreteness was non-significant, F (1, 42) = 1.51, p = .23, d = 0.25.

However, Table 4, below, also reveals a potential issue for interpreting these results.

Given high recognition memory performance for thrice-presented items, gamma correlations

could not be computed for a large number of participants in the cells involving thrice-presented

concrete items. In contrast to standard implementations of repeated-measures analysis using the

general linear model, which deletes cases with any missing G correlations, the mixed model

analysis we employed makes use of all available data under missing-at-random assumptions. For

13

example, if ceiling effects in recognition for thrice-repeated concrete items make G non-

computable in that cell for a given individual, the three G’s from the other cells in the 2 X 2

matrix for that person would still contribute to the estimated marginal means and significance

tests. This approach requires making a missing-at-random assumption, however.

Given the skewed marginal distributions of recognition accuracy it was not possible to

further divide items by encoding recall to use gamma correlations to test the hypothesis that

encoding recall accounted for the relationship of FOKs to recognition accuracy. These outcomes

are yet another reason why the multi-level regression approach provided superior information

about FOKs and their relationships to other cues; the use of gamma correlations was simply

inadequate for these purposes given this design and the resulting data.

Table 4

Fitted Mean Goodman-Kruskal Gamma Correlations of FOKs with Recognition Memory

Accuracy for Unrecalled Items from the Mixed Model Analysis

Abstract Concrete

N G SE N G SE

1 Presentation 41 .22* .08 36 .13 .10

3 Presentations 35 .24* .10 8 .64* .21

* p< .05

Note.G = Goodman-Kruskal gamma correlation. N indicates the number of persons in each cell

with computable G correlations. SE is the fitted standard error for fitted least-squares means of

the G correlations (see text for further details).

14

Appendix E: FOK-Confidence Judgment (CJ) Relationship

This appendix reports on FOK-CJ relationships for correctly recognized items that were

analyzed using multi-level models in the main paper. Our earlier work focused on using

traditional Goodman-Kruskal gamma correlations to evaluate these relationships. To provide a

point of reference to earlier work, we report the relevant gamma correlations here. The

correlations were computed by excluding items that produced ceiling recognition memory, thus

paralleling the analyses reported in the main paper.

Aggregate gamma correlations, ignoring the independent variables, revealed reliable

within-person correlations of FOKs with confidence judgments for unrecalled, correctly

recognized items, G = .31, SE = .04, p< .001. When data were broken down into the 2 X 2

Concreteness X Repetition cells (see Table 5), a repeated measures model in PROC MIXED

revealed no reliable main effects or interaction associated with the independent variables. A test

of the null hypothesis that the grand mean for G = 0 was rejected, t(42) = 3.04, p< .05, with the

estimated G = .15 (SE = .05). Thus, resolution was reduced by conditionalizing on Concreteness

and Repetition. This difference between the aggregate gamma correlation and the average partial

gamma correlation (entries in the cells of Table 5) can be interpreted as indicating that repetition

and concreteness influenced FOKs, and that controlling for these variables reduced FOK-CJ

associations. The multi-level models in the main paper provide a more compelling means of

evaluating these claims.

Table 5

Fitted Mean Goodman-Kruskal Gamma Correlations of FOKs with Confidence Judgments For

Unrecalled, Correctly Recognized Items from the Mixed Model Analysis

15

Abstract Concrete

N G SE N G SE

1 Presentation 40 .13 .07 39 .24* .06

3 Presentations 35 .15 .08 17 .09 .13

* p< .05

Note.G = Goodman-Kruskal gamma correlation. N indicates the number of persons in each cell

with computable G correlations. SE is the fitted standard error for fitted least-squares means of

the G correlations (see text for further details).

16

Appendix F: Data and SAS Code

Data

Raw Data: https://app.box.com/s/n4ys9oz4naqxc7cxhher

Mediator Codes: https://app.box.com/s/1yx6fk4mwo50fmy0oqhx

Codebook

https://app.box.com/s/1b48430ueuie4a09aqol

SAS code

MLM SAS: https://app.box.com/s/9ylvacmeweotahwstwi8

After downloading the data, you will need to change the path under Proc Import to indicate the location on your personal computer where you have saved the data. This needs to be done twice as there are two separate Excel files containing the data.

17

References

Dunlosky, J., & Metcalfe, J. (2009). Metacognition. Thousand Oaks, CA US: Sage Publications,

Inc.

Eakin, D. K., & Hertzog, C. (2012a). Age invariance in feeling of knowing during implicit

interference effects. The Journals of Gerontology: Series B: Psychological Sciences and

Social Sciences, 67B(5), 555-562. doi: 10.1093/geronb/gbr150

Eakin, D. K., & Hertzog, C. (2012b). Immediate judgments of learning are insensitive to implicit

interference effects at retrieval. Memory & Cognition, 40(1), 8-18. doi: 10.3758/s13421-

011-0138-4

Gonzalez, R., & Nelson, T. O. (1996). Measuring ordinal association in situations that contain

tied scores. Psychological Bulletin, 119(1), 159-165. doi: 10.1037/0033-2909.119.1.159

Hertzog, C., Dunlosky, J., & Sinclair, S. M. (2010). Episodic feeling-of-knowing resolution

derives from the quality of original encoding. Memory & Cognition, 38(6), 771-784. doi:

10.3758/MC.38.6.771

Hertzog, C., Fulton, E. K., Mandviwala, L., & Dunlosky, J. (2013). Older adults show deficits in

retrieving and decoding associative mediators generated at study. Developmental

Psychology, 49(6), 1127-1131. doi: 10.1037/a0029414

Hertzog, C., Hines, J. C., & Touron, D. R. (2013). Judgments of learning are influenced by

multiple cues in addition to memory for past test accuracy. Manuscript submitted for

publication.

Paivio, A. (2007). Mind and its evolution: A dual coding theoretical approach. Mahwah, NJ:

Lawrence Erlbaum Associates.

18

Rhodes, M. G., & Tauber, S. K. (2011). The influence of delaying judgments of learning on

metacognitive accuracy: A meta-analytic review. Psychological Bulletin, 137(1), 131-

148. doi: 10.1037/a0021705

Rowe, E. J., & Schnore, M. M. (1971). Item concreteness and reported strategies in paired-

associate learning as a function of age. Journal of Gerontology, 26(4), 470-475.

19