Developing a Tool for Measuring the Decision-Making Competence.pdf

Developing a Tool for Measuring the Decision-Making Competenceof Older Adults

Melissa L. FinucaneEast-West Center

Christina M. GullionCenter for Health Research, Kaiser Permanente Northwest

The authors evaluated the reliability and validity of a tool for measuring older adults’ decision-makingcompetence (DMC). A sample of 205 younger adults (25–45 years), 208 young-older adults (65–74years), and 198 old-older adults (75–97 years) made judgments and decisions related to health, finance,and nutrition. Reliable indices of comprehension, dimension weighting, and cognitive reflection weredeveloped. Comparison of the performance of old-older and young-older adults was possible in thisstudy, unlike previous research. As hypothesized, old-older adults performed more poorly than young-older adults; both groups of older adults performed more poorly than younger adults. Hierarchicalregression analyses showed that a large amount of variance in decision performance across age groups(including mean trends) could be accounted for by social variables, health measures, basic cognitiveskills, attitudinal measures, and numeracy. Structural equation modeling revealed significant pathwaysfrom 3 exogenous latent factors (crystallized intelligence, other cognitive abilities, and age) to theendogenous DMC latent factor. Further research is needed to validate the meaning of performance onthese tasks for real-life decision making.

Keywords: decision-making competence, aging, person–task fit framework, measurement methods

Supplemental materials: http://dx.doi.org/10.1037/a0019106.supp

Measuring older adults’ decision-making competence (DMC)reliably and validly is critical for (a) identifying individuals whosedecision-making abilities are impaired; (b) avoiding inappropriateremoval of decision-making power from competent individuals;and (c) advancing theories of when and how decision makersacross the adult life span are able to meet the demands of variousdecision environments. Even though robust assessment tools areessential for modeling and assisting successful aging, few toolsprovide a performance-based analysis of older adults’ DMC. Inthis article, we report progress in an ongoing research program thataims to measure older adults’ decision skills using behavioraldecision tasks.

Approaches to Measuring DMC

A common approach to measuring DMC is via intuitive clinicaljudgment. Typically, expert physicians are the “gold standard”: Theyjudge an individual’s DMC on the basis of their clinical experienceand their impression of the patient, given clinical and contextualinformation (Grisso, Appelbaum, & Hill-Fotouhi, 1997; Janofsky,McCarthy, & Folstein, 1992; Kim, Caine, Currier, Leibovici, & Ryan,2001). Studies have shown, however, that judgments of the sameindividual may vary across physicians (T. Kitamura & Kitamura,2000; Marson, 1994; Marson, McInturff, Hawkins, Bartolucci, &Harrell, 1997). In addition, clinicians often assess individuals inisolation, with relatively little behaviorally based information.Consequently, the reliability and external validity of their judg-ments may be limited.

In an attempt to clarify methods for measuring decision-makingabilities, researchers have developed several instruments (e.g.,Fitten, Lusky, & Hamann, 1990; Grisso et al., 1997; F. Kitamuraet al., 1998; Schmand, Gouwenberg, Smit, & Jonker, 1999; Stan-ley, Guido, Stanley, & Shortell, 1984). Most instruments assessmedical competencies (e.g., competence to consent to treatment),and many have been tested on impaired, ill, or hospitalized per-sons. Few researchers have developed instruments to evaluateday-to-day decision making. Furthermore, most existing psy-chogeriatric instruments sum a mixture of abilities (Vellinga, Smit,van Leeuwen, van Tilburg, & Jonker, 2004), even though someauthors argue convincingly that the separate scores of differentabilities should not be added (Grisso et al., 1997; Palmer, Nayak,Dunn, Appelbaum, & Jeste, 2002). Overall, questions of reliabilityand validity remain for many existing tools (Betz, 1996; Clark &Watson, 1995; Ryan, Lopez, & Sumerall, 2001).

Melissa L. Finucane, East-West Center, Honolulu, Hawaii; Christina M.Gullion, Center for Health Research, Kaiser Permanente Northwest, Port-land, Oregon.

This paper was prepared with support from the National Institute onAging (Grant R01AG021451-01A2) and the National Science Foundation(Grant BCS-0525238) awarded to Melissa L. Finucane during her tenure asSenior Research Investigator at the Center for Health Research, Hawaii.We are indebted to the individuals who agreed to participate in this study.We wish to thank Rebecca Cowan (Project Manager) and the researchassistants who helped with data collection. We also thank Jennifer Coury,Lisa Fox, Catherine Marlow, Lin Neighbors, and Daphne Plaut for helpwith materials and manuscript preparation.

Correspondence concerning this article should be addressed to MelissaL. Finucane, Senior Fellow, Research Program, East-West Center, 1601East-West Road, Honolulu, HI 96848. E-mail: [email protected]

Psychology and Aging © 2010 American Psychological Association2010, Vol. 25, No. 2, 271–288 0882-7974/10/$12.00 DOI: 10.1037/a0019106

271

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

Performance-based measures of DMC are rare. Recent excep-tions include the Adult Decision-Making Competence index(A-DMC; Bruine de Bruin, Parker, & Fischhoff, 2007) and theYouth Decision-Making Competence index (Y-DMC) (Parker &Fischhoff, 2005). These indices incorporate a variety of experi-mental tasks developed originally to describe general cognitiveprocesses. The authors have reported correlated responses acrosstasks comprised by the indices, suggesting a “positive manifold” ofdecision-making performance (Stanovich & West, 2000). How-ever, the composite nature of the tasks obscures decision-makerstrengths and weaknesses at the level of specific decision skills.Furthermore, some items in these indices may not be appropriatefor very old populations (e.g., asking the probability of getting intoa car accident when driving is not relevant for many old-olderadults who do not drive).

An alternative approach to developing performance-based mea-sures of DMC taken by Finucane and colleagues (Finucane, Mertz,Slovic, & Schmidt, 2005; Finucane et al., 2002) has emphasizedthe need for standardized measurement of the multiple specificskills that constitute competent decision processes. In developingitems applicable across the adult life span, these researchers haveattempted to capture functionally different capacities that may beserved by different cognitive abilities and information-processingmodes. However, initial research suggested that some items hadlow discriminability and indices of some skills (e.g., consistency)had low reliability. In addition, limited measurements of decision-maker characteristics in earlier studies restricted examination ofthe construct validity of the tasks. Also, samples did not includesufficient individuals representing very old adults. This group wasarguably of greatest interest and concern, given demographictrends toward an increasing proportion of old-older adults in thepopulation.

Understanding the Multidimensional Nature of DMC

Competent decision making requires several key skills, includ-ing understanding information, integrating information in an inter-nally consistent manner, identifying the relevance of informationin a decision process, and inhibiting impulsive responding. Per-formance on these skills is expected to reflect the degree ofcongruence between characteristics of the decision maker and thedemands of the task and context (Finucane et al., 2005).

DMC as Compiled Cognition

One way of looking at DMC is as a complex, “compiled” formof cognition (Appelbaum & Grisso, 1988; Park, 1992; Salthouse,1990; Willis & Schaie, 1986) dependent on basic cognitive abili-ties, such as crystallized and fluid intelligence, memory capacity,and speed of processing (Kim, Karlawish, & Caine, 2002; Schaie& Willis, 1999), that change with increasing age (MacPherson,Phillips, & Sala, 2002; Salthouse & Ferrer-Caja, 2003). Accordingto this view, age-related changes in DMC are likely to havecharacteristics similar to age-related changes in these basic cogni-tive abilities.

For instance, comprehension may be affected by an individual’sinductive and arithmetical reasoning abilities, such as problemidentification, decomposition, step sequencing, information ma-nipulation, and perceptions of consequences (Allaire & Marsiske,

1999). Comprehension will suffer as text complexity increases, orreadability decreases (Meyer, Marsiske, & Willis, 1993), andoutstrips the reader’s reasoning skills and memory capacity(Allaire & Marsiske, 1999; Finucane et al., 2002; Meyer, Russo, &Talbot, 1995; Park, Willis, Morrow, Diehl, & Gaines, 1994;Tanius, Wood, Hanoch, & Rice, 2009).

Similarly, decline in memory and processing speed and greaterreliance on simpler strategies among older adults may make themmore prone than younger adults to inconsistency in decision mak-ing across alternative framings of statistically equivalent informa-tion (Finucane et al., 2002, 2005; Weber, Goldstein, & Barlas,1995). Holding constant the relative importance of decision di-mensions (attributes) across situations that differ only superficiallyis important, because it both generates and reflects reliable pref-erences. Some research suggests that though age-related cognitivedeficits may increase susceptibility to framing effects, age-relatedstrengths (e.g., added experience) may counteract the effects whenapplicable (Ronnlund, Karlsson, Laggnas, Larsson, & Lindström,2005).

Researchers have paid more attention to comprehension andconsistency than to how the skills of identifying the relevance ofinformation and inhibiting impulsive responding are associatedwith age-related decline in cognitive abilities. Appelbaum, Grisso,Frank, O’Donnell, and Kupfer (1999) found that, in assessments ofcompetence to consent to research, better appreciation of therelevance of information to one’s personal situation was related tolower age (for women) and having prior research experience.However, the small number of participants in these studies and thesmall range of responses limit the interpretability of the results.Also, Applebaum et al.’s measures were designed only for patientpopulations and did not directly examine patients’ appreciation ofthe importance of specific dimensions of information in theirdecision processes.

The ability to inhibit impulsive responding has been examinedrecently by Frederick (2005), who found that the performance ofyounger adults on the Cognitive Reflection Test was correlatedpositively with measures of their general cognitive ability andcapacity to make “patient” (or risk-seeking) decisions. Cognitivereflection among older adults has not been studied, but analyticoversight of impulsive tendencies might be expected to declinewith age because self-monitoring depends on basic cognitive abil-ities.

Some authors have argued that good decision making dependsalso on numeracy, the ability to work with numbers and to under-stand basic probability and mathematical concepts (Schwartz,Woloshin, Black, & Welch, 1997). Hibbard, Peters, Dixon, andTusler (2007) showed via path analysis of data collected from asample of adults 18–64 years of age that numeracy contributes tocomprehension of hospital performance information, which in turncontributes to choosing a higher quality hospital. Peters and col-leagues (Peters, Dieckmann, Vastfjall, Mertz, & Slovic, 2009;Peters et al., 2006) reported that compared with less numerateindividuals, more numerate individuals are more likely to retrieveand use appropriate numerical principles and are less susceptible toframing effects in judgments about the quality of student work,violence of patients with mental illness, and attractiveness ofgambles. However, except in one study, the samples recruited byPeters et al. (2006) were restricted to mostly younger adults. Somereports suggest that numeracy tends to be lower in older than

272 FINUCANE AND GULLION

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

younger adults (Kutner, Greenberg, & Baer, 2005), but the role ofnumeracy in DMC remains an open question.

Experience and Automaticity

A second way of conceptualizing DMC is as a function ofexperience and automaticity. Experiencing the demands of dailylife helps individuals to develop expert knowledge and automaticprocesses (which become increasingly selective and domain-specific with age; Baltes & Baltes, 1990; Salthouse, 1991). Olderadults have been shown to perform better than younger adults onage-appropriate problems with which they are experienced (Artis-tico, Cervone, & Pezzuti, 2003; Brown & Park, 2003). Thus, DMCmay be preserved with age to the extent that decision tasks areroutine and predictable (cf. Baltes, 1993).

Associative and automatic processes may become more salientwith age because of a strategic avoidance of more deliberativeprocessing (Salthouse, 1996; A. D. Smith, 1996), shifts in moti-vation (Carstensen, 1995, 1998), or increased knowledge permit-ting reliance on gist (Adams, Smith, Nyquist, & Perlmutter, 1997;Hess, 1990; Reyna & Brainerd, 1995; Reyna, Lloyd, & Brainerd,2003; Shanteau, 1992). These changes may result in more misun-derstanding of information or greater vulnerability to cognitivebiases with increasing age (Johnson, 1990) when decision tasksdemand a thorough deliberation of information. However, studiesshow that both analytic and experiential or automatic processes areneeded for making sound but efficient decisions. “Dual-process”models (Damasio, 1994; Epstein, 2003; Kahneman, 2003) implythat decision makers given the same information can use qualita-tively different processing, potentially resulting in better or worsedecisions depending on the match between the mode and task. Forinstance, a person who relies primarily on a feeling about the gistof information may have difficulty with literal (factual) compre-hension problems but be entirely consistent across similar prob-lems.

Evidence suggests that more analytic information processing isrelated to better decision making, at least on the types of nonprosebehavioral decision tasks used in this study. For instance, aninformation processing style that reflects the tendency to thinkhard about problems, such as “need for cognition” (Cacioppo &Petty, 1982) or “analytic” (Epstein, Pacini, Denes-Raj, & Heier,1996), has been correlated with fewer framing errors (S. M. Smith& Levin, 1996). In-depth processing of information may helpdecision makers see through irrelevant differences in framings thatare normatively equivalent (Bruine de Bruin et al., 2007; althoughsee LeBoeuf & Shafir, 2003; Parker, Bruine de Bruin, & Fischhoff,2007).

Addressing Limitations of Previous Research

Developing a tool for measuring the DMC of older adultsrequires the identification and rigorous assessment of perfor-mance-based tasks that reliably capture distinct decision skills andcorrelate meaningfully with other variables known to influencedecision processes. The present work addressed the limitations ofprevious research on aging and DMC in several ways. Tasksmeasuring comprehension and consistency were refined; new tasksmeasuring the ability to identify the weight placed on specificdimensions of information and the ability to inhibit impulsive

responding in decision processes were developed and evaluated. Alarge, age-balanced sample was been recruited to permit compar-ison of the performance of younger, young-older, and old-olderadults. Finally, measures of individual differences were expandedto include at least two marker tests of each of several basiccognitive abilities, to ensure robust analysis of the construct va-lidity of the DMC indices.

Aims in the Current Study

There were three main aims in the current study.

Aim 1: Assemble items ranging in difficulty to measurespecific decision skills (comprehension, consistency, dimen-sion weighting, cognitive reflection) across the adult lifespan.

Aim 2: Evaluate the reliability of indices of comprehension,consistency, dimension weighting, and cognitive reflection interms of internal consistency.

Aim 3: Evaluate the convergent and discriminant constructvalidity of the DMC indices in terms of the degree to whichthey correlate as expected with each other and with otherrelevant constructs (including age, social variables, health,cognitive abilities, attitudes and self-perceptions, decisionstyle, numeracy) via (a) analyses of variance, (b) hierarchicalregression, and (c) structural equation modeling.

Because one of our main interests was to provide a way ofmeasuring differences between age groups, we tested the hypoth-esis of a negative association between age and performance onDMC indices. We also tested the hypothesis of a positive associ-ation between cognitive abilities and performance on DMC indi-ces.

Method

Participants

We recruited 611 participants from the Kaiser Permanente Ha-waii membership and the general community in Honolulu, Hawaii.Each participant was paid $40; for individuals tested outside ourresearch center (e.g., at a retirement community), an additional $5per participant was donated to the participating organization.

Eligibility criteria were evaluated in a telephone-screening in-terview. Individuals were required to have at least an eighth-gradeeducation, to be able to read English well or very well, not to bedepressed (score of less than 3 on the Patient Health Question-naire–2; Kroenke, Spitzer, & Williams, 2003), not to be anxious(score of less than 3 on a two-item anxiety questionnaire devel-oped for this study), to have no physical ailment that wouldprevent participation (e.g., blindness, severe arthritis), and to havea score of 18 or higher on the ALFI-MMSE, a mini-mental statusexamination (Roccaforte, Burke, Bayer, & Wengel, 1992).

Materials and Procedure

Overview. Each participant completed a questionnaire book-let (average completion time was about ninety minutes for older

273MEASURING DECISION-MAKING COMPETENCE

This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

adults and forty-five minutes for younger adults). The full set ofitems in the tool for assessing DMC is provided in the onlinesupplemental materials. The booklet contained multiple pen-and-paper tasks designed to measure comprehension, consistency, di-mension weighting, and cognitive reflection. (Additional fillertasks were presented but are not relevant to the present paper andare thus not reported.) The content of the tasks was relevant tomaking choices in one of three domains that older adults are likelyto encounter: (a) health (e.g., selecting a health care plan); (b)finance (e.g., selecting a mutual fund); and (c) nutrition (e.g.,choosing among food products). Other sections of the bookletcollected information about decision style, self-perceptions, anddemographics. The booklet was printed in 14-point font whereverpossible to accommodate age-related vision deficits. Finally, theparticipants completed a battery of cognitive abilities measuresadministered in person. The battery took about thirty minutes tocomplete.

Comprehension measures. The comprehension problems(labeled COM) had one of two formats. One format presented atable of options with their values on a set of dimensions (choicearray), followed by literal and/or inferential comprehension ques-tions. The second format involved a similar tabular presentationbut with instructions to choose an option that satisfies givencriteria. The correct answer was unambiguous in both formats, andthese items were scored using an item key. There were seven“simple” problems (COM-1 to COM-7) and seven “complex”problems (COM-8 to COM-14). Simple problems involved threeto five options with up to seven dimensions; each complex prob-lem described at least eight options on at least five dimensions.The score range (0–19) for the comprehension index is greaterthan the number of problems because some problems were accom-panied by two questions. A sample comprehension problem isshown in Figure 1; the full set of problems is provided in the onlinesupplemental materials. Eight problems were developed de novo;

six problems were based on problems used by Finucane et al.(2005), revised to increase their difficulty (by adding options orchanging values in the choice array). Problems COM-1, COM-4,COM-6, COM-9, COM-12, and COM-13 in the present study werebased on problems C-2, C-6, C-10, C-4, C-7, C-11, respectively, inFinucane et al. (2005).

Consistency measures. The consistency measures (labeledCON) were designed to measure the ability to hold constant therelative importance of decision dimensions across superficiallydifferent contexts. Russo, Krieser, and Miyashita’s (1975) “order-ing” paradigm was used for all consistency problems, becauseFinucane et al. (2005) reported that a task based on this paradigmin the health domain showed moderate difficulty and good dis-criminability. Participants were presented with a total of nineproblems to measure consistency: five simple problems and fourcomplex problems. An example consistency problem is shown inFigure 2; the full set of problems is provided in the online sup-plemental materials. Problems CON-1 and CON-6 were based onadaptations of problems I-1 and I-2 about health used by Finucaneet al. (2005). Other consistency problems were developed de novo.

In the present study, each simple consistency problem (CON-1to CON-5) presented individuals with a choice array of fouroptions described on three or four dimensions, whereas eachchoice array in the complex problems (CON-6 to CON-9) pre-sented 10 options. Each problem was presented twice. In onepresentation, the options were shown in an invariant random order;in the second presentation, the options were shown ordered highestto lowest on one of the dimensions (the two presentations wereseparated by several other tasks). For each presentation, partici-pants indicated their first choice. A pair of responses was scored asconsistent when a participant selected the same option in bothpresentations.

Dimension weighting. Developed de novo, the dimension-weighting measures (labeled DW) also presented a choice array ineach item but assessed whether respondents reported accurately onwhich dimension they weighted most heavily in making a choice.When respondents do not do what they say they want to do, we caninterpret that as an inability to maximize utility according to theirpreferences. This is not just inconsistency but a failure or mistakeindicating irrational behavior from a utility maximization perspec-tive. Participants were presented with a total of 12 problems tomeasure their insight into their own dimension weighting: sixsimple problems [DW-1 to DW-6] and six complex problems [DW-7to DW-12]. An example DW problem is shown in Figure 3; the fullset of problems is provided in the online supplemental materials.

In this series, each choice array presented values on severaldimensions of a set of options. The values were designed so thatthe best value on the dimension considered most important corre-sponded with a unique option as the best choice. (Pilot testingidentified the value differences necessary to create dominant op-tions.) Participants were asked to indicate (a) which plan was theirfirst choice and (b) which item of information (dimension) wasmost important to them when making that choice. Each simpleproblem described five options on five dimensions, and eachcomplex problem described eight options on eight dimensions.

Cognitive reflection. Frederick’s (2005) Cognitive ReflectionTest measures ability to suppress erroneous answers that spring tomind “impulsively.” We interpret errors as indicating an inabilityto favor analytic over intuitive processes when needed in judgment

COM-3. Consider the information presented below about several brands of yogurt, and answer the following questions. (All the information needed to answer the questions is given in the box.)

Yogurt Nutrition Facts

Brand A Brand B Brand C Brand D Brand E Serving Size 100g 250g 200g 250 g 200g Calories Per Serving 220 250 190 200 180 Amount Per Serving

Total Fat 1.5g 3.0g 2.0g 2.5 g 3.0g Cholesterol 20.0mg 15.0mg 25.0mg 10.0 mg 15.0mg Sugars 30g 25g 20g 29 g 30g Protein 14g 18g 14g 12 g 13g

Net Weight 500g 500g 1000g 1000 g 500g

(a) Which brand has the lowest amount of protein per serving? (Mark only one answer.)

(b) Which brand has the lowest number of servings per container? (Mark only one answer.)

― Brand A ― Brand A

― Brand B ― Brand B

― Brand C ― Brand C

― Brand D ― Brand D

― Brand E ― Brand E

Figure 1. Example comprehension problem with (a) literal and (b)inferential questions. The correct answer for (a) is Brand D and for (b)is Brand B.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

and decision making. Participants were presented with a total ofsix problems (labeled CR; see online supplemental materials) tomeasure cognitive reflection: three problems from Frederick’soriginal test and three new problems designed to parallel theoriginal items. An example of one of the new problems is “Soupand salad cost $5.50 in total. The soup costs a dollar more than thesalad. How much does the salad cost?”

Cognitive ability marker tests. Each participant completed abattery that included at least two marker tests of each of fourbroad, second-order dimensions of intelligence: fluid intelligence,crystallized intelligence, memory span, and perceptual speed. Thisfour-factor structure has been confirmed in older adults via con-firmatory factor analyses (Baltes, Cornelius, Spiro, Nesselroade, &Willis, 1980). The measures were selected from a large body offactor-referenced tests that have been developed and validatedover decades of research on the structure of mental abilities (Car-roll, 1974; Cattell, 1971; Ekstrom, French, Harman, & Derman,1976; Horn & Hofer, 1992; Salthouse, 1993; Thurstone, 1928;Wechsler, 1987). Tests required reading at eighth-grade level; weused enlarged print to assure legibility for older adults. Given thelarge number of tests in this battery, we used a fractional design tominimize participant burden and ensure that all tests were admin-istered to some respondents and all possible pairs of constructswere represented. Table 1 shows that the design resulted in a totalof 10 groups of participants, each providing five to seven measurestaking between 27 and 35 min total to complete. Each test wastaken by between four and eight groups. Group assignment wasrandom and was balanced over age strata. Table 2 shows thenumber of participants who actually were measured on each testassigned to their group.

Crystallized intelligence was represented by three tests of verbalcomprehension with varying difficulty. The first, drawn from thePrimary Mental Abilities collection of tests (Ekstrom et al., 1976),was Recognition Vocabulary (V-1). We used the mean of the twoV-1 part scores in analyses. The other verbal ability tests wereSalthouse’s (1993) Synonyms and Antonyms tests.

Fluid intelligence was represented by two measures of induc-tion. The first measure was drawn from Raven’s Advanced Pro-

gressive Matrices (Raven, Raven, & Court, 1998) and includedthree sample problems followed by the 18 odd-number items. Thesecond measure of fluid intelligence was the Primary MentalAbilities Locations test (Ekstrom et al., 1976); we used the meanof scores from the two sections of this test in analyses.

Memory span (Ms) was represented first by Backward DigitSpan from the Wechsler Adult Intelligence Scale—Third Edition(WAIS–III; Wechsler, 1987). The second measure was the PairedAssociates test by Salthouse and Babcock (1991); we used themean of scores from the two parts of this test in analyses.

Perceptual speed (Ps) was represented by the primary ability ofspeed of processing. One measure was the WAIS–III Digit SymbolSubstitution test. The second measure was the Letter Comparisontest (Salthouse & Babcock, 1991); we used the mean of scoresfrom the two parts of this test in analysis.

Other measures.Decision style. The first two measures of decision style were

presented during the in-person testing session. The Rational–Experiential Inventory—Short Form (REI–S) was used to captureindividual differences in two factorially independent processingmodes (Rational and Experiential), as proposed by cognitive–experiential self-theory (Denes-Raj & Epstein, 1994; Epstein,1994; Epstein et al., 1996). Our second in-person measure ofdecision style was obtained from the Risk–Benefit Rating task(Finucane, Alhakami, Slovic, & Johnson, 2000). The score fromthis task used in analyses was the correlation between risk andbenefit ratings on the 23 paired items within individual; a highernegative correlation indicated greater reliance on an experientialdecision style. The third measure, the Jellybeans task (Denes-Raj& Epstein, 1994), was included in the questionnaire booklet. Foranalysis, the score was recentered at 0 and consequently rangedfrom �3 (preferred Bowl A, a more experiential decision process)

CON-5. Below are facts about some mutual funds. For each fund, the box shows the previous year’s gross return, minimum investment required, and years that the fund has been active. Based on these data, which fund would you choose? Note that there is no right answer. Base your answer on your own preference.

Previous Year’s Gross Return

Minimum Investment

Years of Activity

Fund A 7.10% $1,500 15

Fund B 5.15% $1,000 8

Fund C 6.60% $1,500 7

Fund D 8.40% $2,500 10

Which fund would be your first choice? (Mark only one answer.)

― Fund A

― Fund B

― Fund C

― Fund D

Figure 2. Example consistency problem presenting unordered informa-tion (the second presentation shows the same information ordered by grossreturn).

DW-2. Imagine you need to choose a health plan based on the member ratings given in the box below. Look carefully at the data and tell us which one you would choose. Note that there is no right answer. Base your answer on your own preference.

Plan A

Plan B

Plan C

Plan D

Plan E

Percent of members who believe they are able to get early disease detection 62% 74% 61% 63% 60%

Percent of members very satisfied overall with plan 60% 62% 74% 61% 63%

Percent of members who said it was easy to get a referral to a specialist 61% 63% 60% 62% 74%

Percent of members satisfied with customer service

63% 60% 62% 74% 61%

Percent of members satisfied with physical therapy facilities 74% 61% 63% 60% 62%

(a) Which plan would be

your first choice? (Mark only one answer.)

(b) Which item of information was most important to you when you made your choice in Question (a)? (Mark only one answer.)

― Plan A ― Percent of members who believe they are able to get early disease detection

― Plan B ― Percent of members very satisfied overall with plan

― Plan C ― Percent of members who said it was easy to get a referral to a specialist

― Plan D ― Percent of members satisfied with customer service

― Plan E ― Percent of members satisfied with physical therapy facilities

Figure 3. Example dimension-weighting problem.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

to �3 (preferred Bowl B, a more analytic decision process). Allother measures (described below) were included in the question-naire booklet.

Attitudes and self-perceptions. To measure experience andmotivation, we asked respondents to indicate their agreement on a4-point scale (1 � strongly disagree, 2 � disagree, 3 � agree, 4 �strongly agree) with two statements: “I am very experienced withthe kinds of decisions I made during this study” and “Whencompleting the tasks in this study, I was motivated to take eachtask seriously.” Respondents were also asked to rate their skill inusing tables and charts to make decisions on a 4-point scale (1 �poor, 2 � fair, 3 � good, 4 � excellent). Self-perceived memory(compared with that of people of the same age) was rated on a5-point scale (1 � much worse, 2 � slightly worse, 3 � about thesame, 4 � slightly better, 5 � much better). Self-reported driving

frequency during the past year was measured with a 5-point scale(1 � never, 2 � rarely, 3 � off and on, 4 � frequently, 5 � almostevery day).

As a measure of perceived need for decision support, respon-dents rated on a 4-point scale (1 � none, 2 � a small amount, 3 �moderate amount, 4 � a lot) how much assistance they would seekif they had to choose a health plan, nutritional plan, or checkingaccount, in the next year. For each individual, we calculated anindex of decision support by averaging responses across the threeitems.

Perceived decision self-efficacy was measured with five itemsadapted from Lockenhoff and Carstensen (2007). A sample item is“Imagine you need medical care. How confident are you that youcan make decisions that would help you get the best care?” Eachitem was rated on a 7-point scale (1 � cannot do at all to 7 �

Table 1Fractional Design for Administration of Cognitive Ability Marker Tests

Measure Maximum time (s)

Group Total no. groups inwhich test wasadministered1 2 3 4 5 6 7 8 9 10

Crystallized intelligenceSynonyms 150 1 1 0 0 0 0 1 0 1 1 5Antonyms 150 1 1 0 0 0 0 1 0 1 1 5Vocabulary (Part 1, Part 2) 240 each 1 1 1 0 1 0 0 1 0 0 5

Fluid intelligenceLocations 360 0 1 1 1 0 1 1 0 1 0 6Raven’s Matrices 600 0 0 1 1 0 0 0 1 0 1 4

MemoryPaired Associates (List 1, List 2) 300 each 0 0 0 0 1 1 1 1 0 1 5Backward Digit Span 600 1 0 0 1 1 1 0 0 1 0 5

Perceptual speedDigit Symbol Substitution 120 1 1 0 1 0 0 1 1 0 1 6Letter Comparison (Part 1, Part 2) 30 each 1 1 1 0 1 1 1 1 1 0 8

Decision styleREI–S Rational/Experiential 180 0 0 1 0 0 1 0 1 1 1 5Risk–Benefit (Part 1, Part 2) 300 1 1 0 1 1 0 1 0 1 1 7

Total no. tests in each group 7 7 5 5 5 5 7 6 7 7Total time (min) 31 27 28 33 34 30 29 34 30 35

Note. 1 � test administered, 0 � test missing. Vocabulary � Recognition Vocabulary; Locations � Primary Mental Abilities Locations; Raven’sMatrices � Raven’s Advanced Progressive Matrices; REI–S � Rational–Experiential Inventory—Short Form; Risk–Benefit � Risk–Benefit Rating.

Table 2No. Participants Administered Each Cognitive Ability Marker Test

Test

Group

N1 2 3 4 5 6 7 8 9 10

Synonyms 27 32 0 0 0 0 24 0 25 27 135Antonyms 27 32 0 0 0 0 24 0 25 27 135Vocabulary (Part 1, Part 2) 27 32 29 0 27 0 0 27 0 0 142Locations 0 32 27 24 0 21 24 0 23 0 151Raven’s Matrices 0 0 28 28 0 0 0 26 0 26 108Paired Associates (List 1, List 2) 0 0 0 0 27 25 25 27 0 25 129Backward Digit Span 27 0 0 28 27 27 0 0 25 0 134Digit Symbol Substitution 27 31 0 27 0 0 24 27 0 23 159Letter Comparison (Part 1, Part 2) 27 32 29 0 27 27 25 27 25 0 219REI–S Rational/Experiential 0 0 29 0 0 27 0 27 25 27 135Risk–Benefit (Part 1, Part 2) 26 32 0 28 27 0 25 0 24 27 189

Note. Vocabulary � Recognition Vocabulary; Raven’s Matrices � Raven’s Advanced Progressive Matrices; Locations � Primary Mental AbilitiesLocations; REI–S � Rational–Experiential Inventory—Short Form; Risk–Benefit � Risk–Benefit Rating.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

certainly can do). For each individual, we calculated an index ofself-efficacy by summing responses across the five items.

Self-rated health. Respondents were asked to rate their phys-ical health in comparison with that of others of the same age andsex on a 5-point scale (1 � excellent, 2 � good, 3 � average, 4 �fair, 5 � poor). Respondents used the same scale to rate theirmental health. Respondents were asked to indicate whether theytook prescription medication for each of 24 conditions (e.g.,asthma, arthritis, cancer); the sum of positive responses was re-corded as “number of medications taken.” Finally, respondentswere asked to indicate the number of times they visited a doctor ornurse in the last month (excluding overnight stays and dentalvisits) and the number of nights they stayed overnight in a hospitalin the past year.

Numeracy. Our measure of numeracy is the total number ofcorrect responses to three items testing comprehension of proba-bilistic information. The items are from a scale developed bySchwartz et al. (1997). An index score was created by summingthe number of correct responses for each respondent. We chosethis three-item measure over a longer eight-item measure (Lipkus,Samsa, & Rimer, 2001) because the two measures are moderatelycorrelated (rs � .48, p � .001) but the shorter measure minimizedparticipant burden (Donelle, Hoffman-Goetz, & Arocha, 2007).

Demographics. We asked participants to report their birthdate, gender, education, income category, and ethnicity.

Statistical Analysis

We used multiple imputation (SAS PROC MI, data augmenta-tion with Markov chain Monte Carlo data generation) to fill inmissing test scores. Three old-older participants did not completeany of the in-person tests and were excluded from the imputed dataset. Because most missing data were missing by design in thein-person test battery, these data can be considered missing com-pletely at random. There were no missing data on the numeracyitems or DMC index scores; skips were scored as an incorrectresponse. Eight imputations were created, in order to ensure effi-ciency of at least 95% (relative to asymptotic results) in allvariables included in the planned analyses. The imputation modelincluded demographic features, responses to questions about otherindividual differences, and DMC index scores and other scoresfrom the booklet, in addition to the in-person tests.

All analyses were done with SAS® (Release 9.1). Analyseswere carried out identically over all imputations, and results werecombined using Rubin’s rules (Rubin, 1987). This process in-volves adjusting the degrees of freedom, in order to obtain unbi-ased p values when sample size depends in part on imputed dataand the standard error estimates include between-imputation vari-ability. Because data are complete by imputation, the sample sizedoes not vary between analyses.

The analysis first examined the psychometric performance ofthe DMC tasks, including difficulty and discrimination. Next, weverified the internal consistency of the various multi-item scores.Then we discarded poorly performing tasks and adjusted scores tomaximize psychometric quality. This work was followed by mul-tistep validation strategy of the DMC index scores, includingevaluation of both convergent and discriminant validity andwhether numeracy might be a mediator on the association between

cognitive skills and DMC, according to methods given by Baronand Kenny (1986).

Finally, structural equation modeling of the covariance matrix,done with SAS PROC CALIS, was used to test hypotheses aboutthe association of participant age, numeracy, and cognitive abili-ties with DMC performance. Modeling proceeded in two steps:first verifying the fit of the hypothesized measurement model inthe endogenous and exogenous variables and then adding andtesting the fit of the hypothesized structural model linking theendogenous and exogenous latent factors (Mueller & Hancock,2008). Observed variables were centered on their mean. For over-all models, the mean was overall participants, and for analysescarried out by age-group, the data were centered on the age-groupmean. The analyses were conducted with SAS PROC CALIS(Release 9.1).

The objective for the measurement model was to verify that theobserved variables belong to their respective hypothesized latentfactor, as indicated by the t test on the estimate of the � (slope)coefficient. The variable was dropped from the model if this test wasnot significant; if constraining the parameter to zero had a nonsignif-icant effect on model fit, �2 test on change in �2*log(likelihood), p �.05 for both tests; and if modification tests did not suggest thevariable had a stronger association with a different latent factor. Ifmodification tests suggested a variable belonged on another factor,we moved it to that factor and determined whether this improvedthe fit. Modifications were made one at a time.

The final measurement model was changed into the structuralmodel by adding an equation in which the DMC latent factor is afunction of the exogenous latent factors. The new parametersadded at this stage were examined for significance, and the modelwas simplified (by eliminating unneeded paths) where indicated.

Results

Participant Characteristics

As shown in Table 3, we were successful in enrolling approx-imately equal percentages of women in the three age groups. Thethree groups were similar in household income and education butdiffered in race/ethnicity distribution and ALFI-MMSE ( p �.0001).

Internal Consistency of Predictor Measures

Internal consistency (Cronbach’s alpha) was moderate for per-ceived need for decision support (.71 overall and .71, .65, .73, foryounger, young-older, and old-older adults, respectively) and highfor perceived decision self-efficacy (.86 overall and .83, .87, .87,for younger, young-older, and old-older adults, respectively).Cronbach’s alpha for numeracy was .53 overall (.54, .50, .48 foryounger, young-older, and old-older adults, respectively).

Aim 1: Assemble Items Ranging in Difficulty

Overall, we were able to assemble items for measuring each ofthe decision skills with a considerable range of difficulty levels.Item difficulties within age subgroups ranged from 0.25 to 0.93 forthe comprehension items, 0.30 to 0.85 for the consistency items,0.55 to 0.88 for the dimension-weighting items, and 0.16 to 0.65


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

for the cognitive-reflection items (difficulties for each item areprovided in the online supplemental materials). To enable discrim-ination across a full range of abilities, a set of test items generallyshould include items with a range of difficulties, varying fromaround .10 to .90 (Betz, 1996). This range is covered by the itemsassembled, suggesting that these items would produce a rangeof scores. This selection of test items also ensured that even theleast able respondents would be able to answer a few itemscorrectly.

Aim 2: Evaluate the Reliability of the Indices of DMC

Aim 2(a): Internal consistency. We created an index scorefor each respondent by summing the number of correct responsesacross the sets of problems in each of the four decision skills:comprehension, consistency, dimension weighting, and cognitivereflection. We obtained Cronbach’s alpha on each of these fourindices and then investigated whether we could improve coeffi-cient alpha by removing items less related to the total score,dropping these items in a stepwise fashion. We found that the mostreliable index was created by retaining all problems. The finalinternal consistency coefficients for each index for each age-groupare shown in Table 4. Cronbach’s alpha is adequate for all but theconsistency index, which was dropped from further analysis.

Aim 2(b): Correlations of items with each other. The cor-relations between the comprehension index and the dimension-weighting and cognitive-reflection indices were .46 and .60,respectively. The correlation between dimension weighting and

cognitive reflection was .30. All correlations were significant( p � .0001) and positive, indicating a positive manifold oftasks, and there was relative consistency in performance acrossindices.

A principal-components analysis of the items constituting theoverall comprehension, dimension-weighting, and cognitive-reflection indices resulted in three factors accounting for 27.5% ofthe variance (see Table 5). At least one factor loading for each ofthe variables (with the exception of COM-1, COM-2, COM-8b,and COM-11) was .30 or more, suggesting internal consistency, inthe sense that the problems capture an underlying construct foreach of the main skills needed for competent decision making. Ingeneral, the items that make up the first factor capture individuals’ability to use information (i.e., to combine or transform dimensionvalues) and include mainly comprehension items. The items thatmake up the second factor capture the ability to identify theimportance of information (i.e., to recognize salient information orto understand one’s own weighting scheme) and include mainlydimension-weighting items. A third factor, composed of only twocognitive-reflection items, may represent some subcomponent ofproblem solving.

Aim 3: Evaluate the Construct Validity of the Indicesof DMC

Aim 3(a): Analyses of variance. Figure 4 shows the meandifficulty (proportion correct) of the comprehension, dimension-weighting, and cognitive-reflection items for each age-group. As

Table 3Participant Characteristics

MeasureOverall

(N � 608)Younger

(N � 205)Young-older(N � 208)

Old-older(N � 195) Significance test

Age in years, M (SD) 61.4 (19.9) 34.9 (6.0) 69.7 (2.9) 80.3 (4.7)Age range in years 25–97 25–45 65–74 75–97Sex (% female) 62.5 66.8 58.2 62.6 �2 � 0.8, p � .3665Annual household income (% � $50,000) 47.8 47.3 47.4 48.8 �2 � 0.11, p � .7532Education (% high school or more) 83.1 86.3 83.2 79.5 �2 � 3.3, p � .0681Hispanic (% yes) 5.6 13.7 1.9 1.0 �2 � 30.7, p � .0001Race (%) �2 � 53.5, p � .0001

Caucasian 45.8 28.2 54.9 54.7Asian 31.8 32.1 29.2 34.4

Native Hawaiian/Pacific Islander 15.4 27.4 11.5 6.8Other 7.0 12.4 4.3 4.1

ALFI-MMSE, M (SD) 21.3 (1.0) 21.6 (0.8) 21.3 (0.9) 21.0 (1.1) F(2, 605) � 16.2, p � .0001

Note. ALFI-MMSE � telephone version of the Mini-Mental State Examination.

Table 4Cronbach’s Coefficient Alpha for Comprehension, Consistency, Dimension-Weighting, andCognitive-Reflection Indices for Each Age Group

Decision skill Index score range

Cronbach’s �

Overall Younger Young older Old older

Comprehension 0–19 .79 .74 .78 .75Consistency 0–9 .41 .41 .32 .44Dimension weighting 0–12 .62 .54 .66 .60Cognitive reflection 0–6 .80 .82 .80 .77


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

expected, one-way analysis of variance, with age as a between-groupsfactor, revealed a significant main effect of age for the compre-hension index, F(2, 605) � 37.6, p � .0001; the dimension-weighting index, F(2, 605) � 10.2, p � .0001; and the cognitive-reflection index, F(2, 605) � 9.2, p � .0001. Contrasts betweenage groups confirmed the differences were significant, with thesole exception of the young-old versus old-old contrast on cogni-tive reflection ( p � .0594). Compared with younger adults, young-older and old-older adults generally experienced more difficulty onevery type of task. Mean index scores (and standard deviations) foreach age group for each index (see Table 6) confirm that the agegroups are ordered consistently on age with respect to each DMCindex.

Aim 3(b): Correlations and regressions. Covariation be-tween age and other variables suggests that the differing perfor-mances of younger, young-older, and old-older adults on the DMC

indices might be accounted for by differences in these variablesacross age groups, including mean age trends. As shown in Table 7,age groups differed on measures of health, cognitive abilities, andnumeracy and on some measures of attitudes, self perceptions, anddecision style. Significant differences were not always in thedirection of decline with increasing mean age. For instance, meanson physical health and performance on crystallized intelligencemeasures were positively associated with mean age across agegroups.

Correlations of the three DMC indices with the measures ofindividual differences are shown in Table 8. Most of the variablesin Table 8 correlate significantly with the index scores, indicatingthat several factors predict decision skills. However, significantintercorrelations exist among the individual difference variables aswell (see online supplemental materials). Given that the contribu-tions of the various predictors are not necessarily independent of

Table 5Principal-Components Solution, With Portion of Variance Accounted for in Each Item(Communalities or h2)

Item

Factor

h2

A (informationcombination/

transformation)

B (informationsalience/

relevance)C

(other)

CR-3 0.73� �0.01 0.33� 0.64CR-6 0.72� �0.02 0.29 0.61COM-12b 0.69� 0.05 0.24 0.53COM-3b 0.65� 0.10 0.29 0.51CR-5 0.63� 0.13 0.09 0.42COM-13a 0.54� 0.15 0.18 0.35CR-1 0.52� 0.05 0.27 0.35COM-13b 0.51� 0.15 �0.07 0.29COM-7 0.47� 0.05 �0.13 0.24COM-14 0.44� 0.11 �0.12 0.21COM-5b 0.42� 0.30 0.09 0.28COM-10 0.41� 0.21 �0.04 0.21COM-4 0.41� 0.28 0.02 0.24COM-2 0.28 0.23 0.01 0.13COM-1 0.26 0.25 �0.04 0.13COM-11 0.25 0.13 �0.02 0.08COM-8b 0.24 0.23 0.06 0.11DW-2 0.23 0.48� �0.02 0.28DW-5 0.07 0.47� 0.14 0.25DW-1 0.13 0.44� �0.02 0.21COM-6 0.31� 0.41� 0.10 0.28DW-9 0.05 0.40� 0.16 0.19DW-7 0.13 0.40� �0.06 0.18COM-9 0.34� 0.40� �0.02 0.27DW-4 0.08 0.39� �0.02 0.16DW-10 �0.01 0.38� 0.13 0.16DW-11 0.00 0.37� 0.28 0.21COM-8a 0.31� 0.37� �0.17 0.26DW-3 �0.01 0.35� 0.12 0.14COM-3a 0.18 0.34� 0.05 0.15DW-6 �0.04 0.34� 0.29 0.20DW-12 0.03 0.33� 0.11 0.12DW-8 0.15 0.32� �0.07 0.13COM-5a 0.12 0.31� �0.05 0.11COM-12a 0.11 0.31� �0.15 0.13CR-4 0.23 0.09 0.80� 0.69CR-2 0.30 0.06 0.79� 0.71

Note. An asterisk indicates substantial loading.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

each other, we conducted stepped regression modeling to estimatethe relative influences of age and individual difference variableson the DMC indices.

We conducted a hierarchical sequence of multiple regressionson data from the combined sample for each of the three mostreliable index scores: comprehension, dimension weighting, andcognitive reflection. We were primarily interested in (a) how muchage-related variance exists in each index; (b) excluding age, howmuch variance could be explained by social variables, healthmeasures, basic cognitive abilities, attitudinal and self-perceptionmeasures, and decision style; and (c) how much age-related vari-ance is independent of the other variables in the model.

Results of the regressions are summarized in Table 9. Using thecomprehension index score as the dependent variable, we foundthat when age was entered into the regression model by itself, itaccounted for 10.6% of the variance. Then, excluding age, signif-icant changes in R2 were found after entering each of the othergroups of variables. Adding age last resulted in a nonsignificantincrement in R2 of �.001.

Using the dimension-weighting index score as the dependentvariable, we found that when age was entered into the regressionmodel by itself, it accounted for 3.0% of the variance. Then,excluding age, significant changes in R2 were found after enteringeach of the other groups of variables. Adding age last resulted ina significant increment in R2 but of only .003 (a change of 1.4%).

Using the cognitive-reflection index score as the dependentvariable, we found that when age was entered into the regressionmodel by itself, it accounted for 3.3% of the variance. Then,excluding age, significant changes in R2 were found after enteringthe social, health, and cognitive skill variables. The attitudinal anddecision-style measures did not increase R2 significantly. Addingage last also resulted in a nonsignificant increment in R2 of .003 (achange of 0.69%).

We were also interested in examining the extent to which theDMC indices could be explained (mediated) by numeracy aboveand beyond the influence of basic cognitive abilities (Baron &Kenny, 1986). Many judgment and decision tasks rely heavily onunderstanding basic mathematical concepts, so we expected aconsiderable amount of DMC variance to be accounted foruniquely by numeracy. First, we directly regressed the same set of10 cognitive skills used in the Table 9 analyses (see Analysis 2,Step 3 of Table 9) on each of the DMC indices, to confirm thatthere is an effect that might be mediated. The results of this

analysis for comprehension, dimension weighting, and cognitivereflection are shown in Table 10 (Analysis 1, Step 1). For com-prehension, cognitive skills alone account for 46.7% of the vari-ance. When we regressed the set of cognitive skills on numeracy,we obtained a model R2 of .28 ( p � .0001), which confirms thatthey are related (Baron & Kenny, 1986). When numeracy isentered in Analysis 1, Step 2, it adds just 3.9% of explainedvariance in performance on the comprehension index. Thus, afteradjustment for cognitive skills, numeracy accounted for a small butsignificant amount of comprehension variance and so appears to bea mediator. However, the magnitude of �R2 (see Analysis 2, Step2, when cognitive skills are added to a model with numeracy only)indicates that numeracy is not a total mediator: After we accountedfor numeracy, cognitive skills accounted for an additional 25.3%of variance in comprehension. A comparison of total R2 in Step 1of Analysis 1 and �R2 in Step 2 of Analysis 2 reveals thatcognitive skills’ explanation of comprehension variance decreasedfrom 46.7% to 25.3% after we controlled for numeracy. Thus,numeracy accounted for 45.8% (21.4% out of 46.7%) of thecognitive-skills-related variance in the comprehension index.

We found a similar pattern of results when conducting hierar-chical regressions to predict the dimension-weighting andcognitive-reflection indices (see Table 10). Cognitive skills aloneaccounted for 13.2% (34.8%) of the explained variance in dimen-sion weighting (cognitive reflection). Numeracy alone accountedfor 10.4% (28.0%) of the variance. After we controlled for cog-nitive skills, numeracy accounted for an additional 4.5% (7.2%) ofthe variance. Numeracy accounted for 44.7% (59.8%) ofcognitive-skills-related variance in the dimension-weighting(cognitive-reflection) indices.

Following the steps outlined by Baron and Kenny (1986), wecan conclude that numeracy does not completely mediate therelationship between cognitive skills and DMC indices. Eventhough cognitive skills are related to both numeracy and the DMCindices and numeracy affects the indices after controlling forcognitive skills, there is still a relationship between cognitive skillsand the indices after controlling for numeracy.

Aim 3(c): Structural equation modeling. We evaluated thefit between the new data collected in this study and our hypothe-sized model of DMC, starting with the measurement model (theassociations among exogenous variables only). An examination ofthe initially hypothesized measurement model indicated that risk–benefit correlation was not confirmed as an indicator for decisionstyle. We removed it completely from the data set, and thissignificantly improved fit. We tested three ways of adding nu-

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Younger Young-Older Old-Older

Age Group

Mea

n Di

fficu

lty (P

ropo

rtio

n Co

rrec

t)

Comprehension ItemsDimension-Weighting ItemsCognitive-Reflection Items

Figure 4. Mean difficulty (proportion correct) for comprehension,dimension-weighting, and cognitive-reflection items for each age group.

Table 6Mean Score (and Standard Deviation) for the OverallComprehension, Dimension-Weighting, and Cognitive-ReflectionIndices for Each Age Group

Decision skill Younger Young older Old older

Comprehension (maximumscore � 19) 14.0 (3.2) 12.4 (3.8) 10.9 (3.7)

Dimension weighting(maximum score � 12) 9.4 (2.0) 8.9 (2.4) 8.4 (2.4)

Cognitive reflection(maximum score � 6) 2.5 (2.1) 2.1 (2.0) 1.7 (1.8)


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

meracy to the model: as a single-variable factor, as a marker ondecision style, and as a marker on the DMC factor. All of theseapproaches resulted in worse fit, and numeracy was omitted fromthe measurement model. We evaluated, based on modificationindices, whether combining some of the cognitive skills latentfactors might improve fit. The final measurement model was thestarting point for the next phase: structural modeling. We removedseveral paths in response to p values greater than .05. Theseincluded correlation between fluid intelligence and crystallizedintelligence and the path from decision style to DMC. The good-ness of fit index was 0.90, and the root mean square residual was0.08. We were not able to obtain an interpretable solution in theanalysis by age-group, possibly because the sample size within agegroups is too small. The final structural equation model is shownin Figure 5.

Discussion

Our first aim was to assemble performance-based items rangingin difficulty to measure specific DMC skills across the adult lifespan. The items produced a test yielding an appropriate range ofscores and different levels of complexity that are often faced inreal life (Tanius et al., 2009). Unlike previous research, our study

developed multiple new problems for measuring DMC in thedomains of health, nutrition, and finance. It thus extended the workof Finucane et al. (2005) by developing items with a wider rangeof difficulty and measuring additional skills (dimension weightingand cognitive reflection) in multiple domains in a standardizedmanner across the adult life span.

Our second aim was to assess the reliability of indices ofspecific DMC skills. The new index of comprehension showedbetter internal consistency (� � .74) than was found by Finucaneet al. (2005), who reported their comprehension error index hadcoefficient alphas of .52 for younger adults (19–50 years) and .58for older adults (65–94 years). Good internal consistency wasfound for the new dimension-weighting index (� � .54) andcognitive-reflection index (� � .77). Unfortunately, we were un-able to improve the internal consistency of the consistency index(�.44, similar to the findings of Finucane et al.). Intercorrelationsbetween the items comprising the new indices reflected two re-lated, mutually supportive constructs, long identified as central todecision making (Edwards, 1954; Raiffa, 1968). These results alsosupport Stanovich and West’s (2000) suggestion that performanceon conventional decision-making tasks reflects a positive manifoldrather than random performance errors. The reliability of the

Table 7Means (and Standard Deviations) on Individual-Difference Measures and Tests of Significant Differences Among Age Groups

Measure Overall Younger Young older Old older Significance testa

Self-reported healthPhysical health (% excellent or good) 77.2 67.3 80.8 83.8 �2 � 15.5, p � .0001Mental health (% excellent or good) 91.3 89.3 91.7 92.9 �2 � 1.7, p � .1926No. medications 1.7 (1.0) 1.2 (0.6) 2.0 (1.1) 2.0 (1.1) F(2, 278302) � 50.2, p � .0001Doctor/nurse visits in previous month 0.7 (0.9) 0.5 (0.8) 0.8 (0.9) 0.9 (1.0) F(2, 138166) � 9.6, p � .0001

Crystallized intelligenceVocabularyb 13.5 (4.0) 11.8 (4.1) 14.1 (3.8) 14.7 (3.5) F(2, 849.2) � 29.0, p � .0001Synonymsb 6.3 (2.9) 5.2 (2.8) 6.7 (3.0) 7.1 (2.7) F(2, 360.9) � 21.1, p � .0001Antonymsc 5.8 (2.9) 5.0 (2.8) 6.2 (3.0) 6.4 (2.7) F(2, 1385) � 14.4, p � .0001

Fluid intelligenceRaven’s Matricesc 6.4 (3.2) 8.8 (2.7) 5.9 (2.8) 4.3 (2.3) F(2, 1752.0) � 144.5, p � .0001Locationsc 4.3 (2.7) 5.6 (2.9) 3.9 (2.5) 3.3 (2.0) F(2, 210.4) � 38.7, p � .0001

MemoryBackward Digit Spanb 6.8 (2.6) 7.6 (3.0) 6.4 (2.3) 6.4 (2.4) F(2, 490.0) � 12.4, p � .0001Paired Associatesb 2.1 (1.9) 3.3 (1.9) 1.5 (1.5) 1.4 (1.4) F(2, 1213.9) � 84.0, p � .0001

Perceptual speedLetter Comparisonb 9.2 (3.0) 11.7 (2.9) 8.3 (2.0) 7.6 (2.3) F(2, 8493.2) � 159.9, p � .0001Digit Symbol Substitutionb 64.5 (21.3) 82.4 (20.6) 60.1 (14.5) 50.3 (13.9) F(2, 168.6) � 153.5, p � .0001

Attitudes and self-perceptionsExperience 2.9 (0.7) 2.9 (0.7) 2.9 (0.7) 2.9 (0.6) F(2, 605) � 0.2, p � .8071Motivation 3.4 (0.6) 3.4 (0.6) 3.4 (0.5) 3.4 (0.6) F(2, 605) � 0.1, p � .8945Skill with tables 2.9 (0.7) 3.1 (0.7) 2.8 (0.7) 2.7 (0.6) F(2, 145061) � 18.9, p � .0001Decision support 1.8 (0.7) 2.0 (0.7) 1.7 (0.6) 1.8 (0.7) F(2,1.9E6) � 8.6, p � .0002Self-reported memory 3.7 (0.9) 3.6 (0.9) 3.7 (0.9) 3.9 (1.0) F(2, 145024) � 5.3, p � .0049Driving almost daily (%) 63.4 69.8 64.4 55.5 �2 � 8.7, p � .0033Decision self-efficacy 5.6 (0.9) 5.6 (0.8) 5.6 (0.9) 5.4 (0.9) F(2, 605) � 2.8, p � .0646

Decision styleRisk–benefit correlationsb �0.43 (0.37) �0.47 (0.32) �0.43 (0.38) �0.39 (0.40) F(2, 132.6) � 1.8, p � .1755REI–S, Rationalc 42.5 (7.9) 43.8 (8.0) 42.3 (7.6) 41.5 (8.2) F(2, 958.2) � 4.0, p � .0189REI–S, Experientialc 40.3 (8.6) 41.4 (8.1) 40.6 (8.2) 38.8 (9.4) F(2, 4628.2) � 4.8, p � .0085Jellybeansc 0.4 (2.1) 0.7 (2.1) 0.5 (2.1) 0.1 (2.1) F(2, 153976.3) � 3.5, p � .0290

Numeracy 1.8 (1.0) 2.0 (0.9) 1.7 (0.9) 1.5 (0.9) F(2, 605) � 16.3, p � .0001

a Degrees of freedom are adjusted according to Rubin’s rules, as part of combining imputations. The less efficient the imputed values, the more the df willbe inflated. b Indicates that the variable was transformed to normality for imputation and then back-transformed to the original units. c Indicates that thevariable was not transformed but does have imputed values.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

indices suggests potential for developing a normed psychometricDMC test.

Our third aim was to assess the construct validity of the DMCindices. We examined performance on DMC indices across a wideage span, with special focus on a previously neglected and poorlyunderstood group, namely, individuals over 74 years of age. Wefound as predicted that young-older and old-older adults showedpoorer performance on the DMC indices than younger adults did.We also found, as predicted, that better performance on the DMCindices was related primarily to better cognitive skills. Highereducation and income, being male, and being healthier were ad-ditional predictors of better DMC. The dominance of cognitiveskills as a predictor helps build the case for the utility of theseDMC indices, suggesting that they mostly reflect people’s abilityto understand and use information effectively (as opposed to theirbeing primarily a function of broader demographic factors).

The results showed also that numeracy accounted for a signif-icant amount (but not all) of the explained variance in DMCindices, above and beyond the variance accounted for by cognitiveskills. The ability to work with numbers and to understand basicprobability and mathematical concepts seems to be an appliedskill, independent of basic cognitive abilities, that is required forgood decision making.

The present results supported the discriminant validity of theDMC indices. Whereas we expect our indices to be positively

correlated with related abilities, such as memory or speed ofprocessing, those correlations cannot be too high or they wouldindicate the tool might be measuring the same construct. Thus,moderate, positive associations with related, but not identical,constructs demonstrated adequate discriminant validity.

The structural equation model portrayed in Figure 5 indicatesthat across the adult life span, individuals with lower crystallizedintelligence or poorer performance on other cognitive abilities tendto exhibit poorer performance on our DMC measures. Increasingage also signifies poorer DMC performance. Decision style, how-ever, only signifies change in other key variables. Rather thanimplying that the relationship of each exogenous factor exerts anindependent effect on DMC, the results suggest that we may in factneed fewer independent explanatory mechanisms to account forthese effects (cf. Salthouse & Ferrer-Caja, 2003). Although ourstudy demonstrates the feasibility of using structural modeling toexamine the existence and relative magnitude of statistically dis-tinct influences on DMC, the generalizability of the results shouldbe tested with data from different samples and different operation-alizations of the theoretical constructs.

The findings reported above are restricted to the DMC skills weexamined. There may be different relationships between age andother skills (e.g., involving the application of emotional wisdom).For instance, Bruine de Bruin et al. (2007) reported that olderadults performed significantly better than younger adults on a task

Table 8Correlations of Decision-Making Competence Index Scores With Demographic Variables and Individual Difference Measures

Measure

Comprehension indexDimension-weighting

index Cognitive-reflection index

r p r p r p

Age �.33 �.0001 �.17 �.0001 �.18 �.0001Gender (female � 0, male � 1) .08 .0633 .11 .0083 .20 �.0001Income .20 �.0001 .14 .0007 .21 �.0001Education .33 �.0001 .19 �.0001 .33 �.0001Physical healtha �.10 .0156 �.09 .0339 �.11 .0058Emotional healtha �.15 .0002 �.10 .0110 �.11 .0080No. medicationsa �.22 �.0001 �.15 .0003 �.12 .0036Doctor/nurse visitsa �.09 .0353 �.07 .0805 �.05 .1885Vocabularya .27 �.0001 .19 .0001 .27 �.0001Synonymsa .34 �.0001 .12 .0030 .35 �.0001Antonymsb .37 �.0001 .15 .0013 .35 �.0001Raven’s Matricesb .50 �.0001 .24 �.0001 .44 �.0001Locationsb .26 �.0001 .09 .0370 .24 �.0001Backward Digit Spana .30 �.0001 .22 �.0001 .26 �.0001Paired Associatesa .40 �.0001 .15 .0005 .32 �.0001Letter Comparisona .36 �.0001 .22 �.0001 .23 �.0001Digit Symbol Substitutiona .39 �.0001 .19 �.0001 .24 �.0001Experiencea .05 .2004 .10 .0127 .06 .1191Motivationa .11 .0055 .11 �.0076 .08 .0559Skill with tablesa .33 �.0001 .22 �.0001 .29 �.0001Decision supporta �.01 .7737 �.15 .0003 �.01 .8976Self-reported memorya �.05 .2411 .00 .9988 .00 .9450Drivinga .19 �.0001 .16 .0001 .11 .0054Decision self-efficacya .00 .9381 .05 .1899 �.01 .8308Risk–benefit correlationa �.06 .2234 �.06 .2144 �.04 .4324REI–S, Rationalb .30 �.0001 .23 �.0001 .16 �.0001REI–S Experientialb .13 .0041 .11 .0110 .02 .6232Jellybeansb .25 �.0001 .16 .0001 .21 �.0001Numeracyb .50 �.0001 .32 �.0001 .53 �.0001

a Indicates that the variable was transformed to normality for imputation and then back-transformed to the original units. b Indicates that the variable wasnot transformed but does have imputed values.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

Tab

le9

Hie

rarc

hica

lR

egre

ssio

nsW

ith

R2

and

Incr

emen

tin

R2

Fro

ma

Seri

esof

Reg

ress

ion

Mod

els

Wit

hth

eC

ompr

ehen

sion

,D

imen

sion

-Wei

ghti

ng,

and

Cog

niti

ve-R

efle

ctio

nIn

dice

sas

the

Dep

ende

ntV

aria

bles

Inde

pend

ent

vari

able

Com

preh

ensi

onD

imen

sion

wei

ghtin

gC

ogni

tive

refl

ectio

n

Cha

nge

inR

2T

otal

R2

Adj

uste

dR

2F

test

of�

R2

Cha

nge

inR

2T

otal

R2

Adj

uste

dR

2F

test

of�

R2

Cha

nge

inR

2T

otal

R2

Adj

uste

dR

2F

test

of�

R2

Ana

lysi

s1

Step

1:A

geby

itsel

f.1

06.1

05F

(1,

606)

�71

.9,

p�

.000

10.

030

0.02

9F

(1,

606)

�18

.8,

p�

.000

1—

0.03

30.

032

F(1

,60

6)�

20.8

,p

�.0

001

Ana

lysi

s2

Step

1:So

cial

vari

able

sa.1

54.1

46F

(7,

600)

�1,

142.

6,p

�.0

001

.050

.041

F(7

,60

0)�

1,44

1.9,

p�

.000

1.1

62.1

53F

(7,

600)

�12

8.0,

p�

.000

1St

ep2:

Hea

lthm

easu

resb

.057

.211

.198

F(4

,59

6)�

10.8

,p

�.0

001

.024

.074

.058

F(4

,59

6)�

3.9,

p�

.000

1.0

16.1

78.1

64F

(4,

596)

�2.

9,p

�.0

214

Step

3:B

asic

cogn

itive

skill

sm

easu

resc

.294

.505

.489

F(9

,58

7)�

38.7

,p

�.0

001

.083

.157

.130

F(9

,58

7)�

6.4,

p�

.000

1.2

43.4

21.4

02F

(9,

587)

�27

.4,

p�

.000

1St

ep4:

Atti

tudi

nal

and

self

-per

cept

ion

mea

sure

sd

.014

.519

.497

F(7

,58

0)�

2.4,

p�

.000

1.0

36.1

93.1

56F

(7,

580)

�3.

7,p

�.0

001

.005

.426

.400

F(7

,58

0)�

0.7,

p�

.653

6

Step

5:D

ecis

ion

styl

em

easu

rese

.020

.539

.515

F(4

,57

6)�

6.2,

p�

.000

1.0

27.2

20.1

79F

(4,

576)

�5.

0,p

�.0

001

.008

.434

.404

F(4

,57

6)�

2.0,

p�

.088

0St

ep6:

Add

ing

age

�.0

01.5

39.5

14F

(1,

575)

�0,

p�

1.00

00.0

03.2

23.1

81F

(1,

575)

�2.

2,p

�.0

002

.003

.437

.406

F(1

,57

5)�

3.1,

p�

.080

6Fu

llm

odel

.539

.514

F(3

2,57

5)�

403.

8,p

�.0

001

.223

.181

F(3

2,57

5)�

338.

0,p

�.0

001

.437

.406

F(3

2,57

5)�

44.1

,p

�.0

001

aE

xoge

nous

soci

alva

riab

les

incl

ude

gend

er,

inco

me,

educ

atio

n,an

dra

ce.

bH

ealth

mea

sure

sin

clud

eph

ysic

alhe

alth

,em

otio

nal

heal

th,

num

ber

ofm

edic

atio

ns,

and

num

ber

ofdo

ctor

/nur

sevi

sits

.c

Bas

icco

gniti

vesk

ills

mea

sure

sin

clud

eV

ocab

ular

y,Sy

nony

ms,

Ant

onym

s,R

aven

’sA

dvan

ced

Prog

ress

ive

Mat

rice

s,L

ocat

ions

,Bac

kwar

dD

igit

Span

,Pai

red

Ass

ocia

tes,

Let

ter

Com

pari

son,

and

Dig

itSy

mbo

lSu

bstit

utio

n.d

Atti

tudi

nal

and

self

-per

cept

ion

mea

sure

sin

clud

eex

peri

ence

,m

otiv

atio

n,sk

illw

ithta

bles

,de

cisi

onsu

ppor

t,se

lf-r

ated

mem

ory,

driv

ing,

and

deci

sion

self

-ef

fica

cy.

eD

ecis

ion-

styl

em

easu

res

incl

ude

risk

–be

nefi

tco

rrel

atio

n,R

atio

nal–

Exp

erie

ntia

lIn

vent

ory—

Shor

tFo

rm(R

EI–

S)R

atio

nal

and

Exp

erie

ntia

l,an

dJe

llybe

ans.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

called “recognizing social norms.” As suggested elsewhere (Baltes& Baltes, 1990; Denney, 1989; Finucane et al., 2002; Salthouse,1991), the knowledge and experience of older adults may make upfor declines in cognitive abilities and allow their problem solvingto be more automatized.

In contrast to the A-DMC developed by Bruine de Bruin et al.(2007), the indices in the present study permit examination ofDMC at the level of component abilities. Our approach allows arefined analysis of specific skills that may increase or decreasewith age and other variables, thereby pinpointing exactly whichpart of a decision process is going well or poorly. Nonetheless,many real-world decisions require multiple skills to be appliedsimultaneously, and the A-DMC seems to be a good tool formeasuring this type of performance. Together, the two tools pro-vide a comprehensive approach. Future research could examine theexternal validity of both tools.

Limitations

Several limitations of this study should be noted. First, someauthors, as a rule of thumb, strictly require a reliability of 0.70 orhigher before they will use an instrument. This level of reliabilitywas achieved only by the comprehension and cognitive-reflectionindices. The rule of thumb should be applied with caution, though,because the appropriate degree of reliability depends upon the useof the instrument (e.g., we could increase reliability by addingitems, but because this tool is designed to be appropriate for olderadults, we intentionally kept it short).

A second limitation of this study is that we used conveniencesamples and (for budget reasons) did not sample 45- to 64-year-olds. Consequently, the results may not generalize to the broaderpopulation. A third limitation is that our research did not addressactivities of daily living besides health, nutrition, and financialdecisions. Researchers should assess health-risk behaviors (nota-bly smoking, exercise, and driving) and retirement and deathissues (e.g., inheritance tax issues, advance directives), but thosewere beyond the scope of this study. Fourth, the content of thecognitive-reflection items is somewhat different from that of thecomprehension and dimension-weighting items. Future researchmight benefit from content reflecting more real-world, personallyrelevant choices for cognitive-reflection items.

Fifth, the cross-sectional rather than longitudinal design meansthat cohort effects may be interpreted as developmental changes.The better control offered by a longitudinal design would beadvantageous, but the cost and logistical complexity of conductinga study that would extend for more than 50 years (if 45-year-oldswere to be followed until they were 100 years old) is not warranteduntil a reliable and valid DMC measurement tool is established.The cross-sectional design also poses a challenge for interpreting“age-related variance,” because the design confounds individualdifferences in age and the average between-person differences.The covariance may be at least partly a product of age-relatedmean trends in the population (Hofer, Berg, & Era, 2003).

Finally, we must underscore that our correlational results do notimply causality. Although speculation about the cognitive pro-cesses or neural substrates responsible for variables influencingDMC is tempting, our correlational analyses are best viewed as aclassification and not an explanation of these effects. More directtests of causal relationships include prospective studies examiningT

able

10H

iera

rchi

cal

Reg

ress

ions

Exa

min

ing

Con

trib

utio

nof

Num

erac

yvs

.C

ogni

tive

Skil

lsto

Var

ianc

ein

Inde

xSc

ores

(Par

tici

pant

sW

ith

Impu

ted

Dat

a,C

onst

ant

N�

608)

Inde

pend

ent

vari

able

Com

preh

ensi

onD

imen

sion

wei

ghtin

gC

ogni

tive

refl

ectio

n

Cha

nge

inR

2T

otal

R2

Adj

uste

dR

2F

test

of�

R2

Cha

nge

inR

2T

otal

R2

Adj

uste

dR

2F

test

of�

R2

Cha

nge

inR

2T

otal

R2

Adj

uste

dR

2F

test

of�

R2

Ana

lysi

s1

Step

1:B

asic

cogn

itive

skill

sm

easu

resa

.467

.459

F(1

0,59

7)�

1,08

3.2,

p�

.000

1.1

32.1

18F

(10,

597)

�96

4.8,

p�

.000

1.3

48.3

38F

(10,

597)

�11

4.1,

p�

.000

1St

ep2:

Add

num

erac

ybto

cogn

itive

skill

s.0

39.5

06.4

97F

(1,

596)

�47

.0,

p�

.000

1.0

45.1

77.1

63F

(1,

596)

�32

.6,

p�

.000

1.0

72.4

20.4

10F

(1,

596)

�74

.0,

p�

.000

1A

naly

sis

2St

ep1:

Num

erac

yby

itsel

f.2

53.2

52F

(1,

606)

�20

5.5,

p�

.000

1.1

04.1

03F

(1,

606)

�70

.6,

p�

.000

1.2

80.2

79F

(1,

606)

�23

5.5,

p�

.000

1St

ep2:

Add

basi

cco

gniti

vesk

ills

.253

.506

.497

F(1

0,59

6)�

30.5

,p

�.0

001

0.07

3.1

77.1

63F

(10,

596)

�5.

29,

p�

.000

1.1

40.4

20.4

10F

(10,

596)

�14

.4,

p�

.000

1Fu

llm

odel

.506

.497

F(1

1,59

6)�

1,08

2.8,

p�

.000

1.1

77.1

63F

(11,

596)

�93

9.1,

p�

.000

1.4

20.4

10F

(11,

596)

�12

3.6,

p�

.000

1

aM

easu

res

ofba

sic

cogn

itive

skill

sin

clud

eV

ocab

ular

y,Sy

nony

ms,

Ant

onym

s,R

aven

’sA

dvan

ced

Prog

ress

ive

Mat

rice

s,L

ocat

ions

,Bac

kwar

dD

igit

Span

,Pai

red

Ass

ocia

tes,

Let

ter

Com

pari

son,

and

Dig

itSy

mbo

lSu

bstit

utio

n.b

Num

erac

ysc

ores

are

base

don

the

thre

e-ite

mm

easu

reby

Schw

artz

etal

.(1

997)

.


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

the effects of decision training on later outcomes (e.g., Downs etal., 2004).

Despite the above limitations, the results indicate progress to-ward a reliable and valid test of older adults’ DMC. This tool iscritical for future studies aimed at determining how to facilitateoptimal functioning in a rapidly aging society.

Theoretical Implications

In this and previous papers, we conceive of DMC as a multidi-mensional concept, including the skills of comprehension, consis-tency, identifying information relevance, and tempering impulsiv-ity. Each of these abilities is expected to tap functionally differentareas of decision processes. Although we were unable to developa reliable index of consistency, reliability of the other indices wasadequate and moderate correlations among the indices suggestedrelated yet distinct components of DMC. These results show theimportance of distinguishing among decision skills. Other compo-nents of DMC need to be investigated to further our understandingof complex constructs such as “rational” information integrationand how and why DMC changes with age.

A view of DMC as a complex, “compiled” form of cognition(Appelbaum & Grisso, 1988; Park, 1992; Salthouse, 1990; Willis& Schaie, 1986) is supported by our finding that basic cognitiveabilities were the strongest predictors of DMC. Age-related

changes in DMC are thus likely to have characteristics similar toage-related changes in basic abilities. The structural modelingresults suggest that only a small number of explanatory mecha-nisms may be needed to account for relationships between exog-enous factors and DMC performance. Finding that numeracy ac-counts for additional DMC variance above and beyond goodcognitive abilities suggests that models of sound decision makingshould include an ability to understand and work with numbers.

Policy Implications

A robust method for measuring individual differences in DMCcan enable decision support to be tailored toward those who needit in a timely manner. The present research has advanced a way ofmeasuring exactly when, where, and how older adults need helpand whether the needs of old-older adults differ from those ofyoung-older adults. Decision support may come in the form ofdecision agents or optimal information presentation formats andtraining tools. Longer life spans and the rapid aging of the world’spopulation (United Nations Population Division, 2002) demandways of creating decision environments that rely less on qualitiestypical of youth (e.g., speed) and rely more on qualities typical ofage (e.g., crystallized intelligence, wisdom, emotional regulation).

In sum, we have developed and evaluated new, standardizedindices of comprehension, dimension weighting, and cognitive

Figure 5. Final model indicating significant associations among model variables. Covariances join two latentfactors on the same side of the model with a double-pointed arrow, whereas error terms have a one-way arrowbetween observed and error variables. Variances have a two-headed arrow, with both heads pointing at the samelatent factor. Significant standardized coefficients are on a single-headed arrow between latent factors (structuralmodel) or between latent and observed factors (measurement model).


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

reflection, applicable across the adult life span. The present resultssupport the overall construct validity of DMC as well as thepredictive validity of tasks drawn from behavioral decision re-search. Our systematically developed battery of items, which weretested on a large, diverse sample that included old-older adults,suggests that the DMC indices have promise for measuring indi-vidual differences in decision skills.

References

Adams, C., Smith, M. C., Nyquist, L., & Perlmutter, M. (1997). Adultage-group differences in recall for the literal and interpretive meaningsof narrative text. Journals of Gerontology, Series B: PsychologicalSciences and Social Sciences, 52, 187–195.

Allaire, J. C., & Marsiske, M. (1999). Everyday cognition: Age andintellectual ability correlates. Psychology and Aging, 14, 627–644.

Appelbaum, P. S., & Grisso, T. (1988). Assessing patients’ capacities toconsent to treatment. New England Journal of Medicine, 319, 1635–1638.

Appelbaum, P. S., Grisso, T., Frank, E., O’Donnell, S., & Kupfer, D.(1999). Competence of depressed patients for consent to research. Amer-ican Journal of Psychiatry, 156, 1380–1384.

Artistico, D., Cervone, D., & Pezzuti, L. (2003). Perceived self-efficacyand everyday problem solving among young and older adults. Psychol-ogy and Aging, 18, 68–79.

Baltes, P. B. (1993). The aging mind: Potential and limits. Gerontologist,33, 580–594.

Baltes, P. B., & Baltes, M. M. (1990). Psychological perspectives onsuccessful aging: The model of selective optimization with compensa-tion. In M. M. Baltes & P. B. Baltes (Eds.), Successful aging: Perspec-tives from the behavioral sciences (pp. 1–34). Cambridge, England:Cambridge University Press.

Baltes, P. B., Cornelius, S. W., Spiro, A., Nesselroade, J. R., & Willis, S. L.(1980). Integration versus differentiation of fluid/crystallized intelli-gence in old age. Developmental Psychology, 16, 625–635.

Baron, R. M., & Kenny, D. A. (1986). The moderator–mediator variabledistinction in social psychological research: Conceptual, strategic, andstatistical considerations. Journal of Personality and Social Psychology,51, 1173–1182.

Betz, N. E. (1996). Test construction. In F. T. Leong & J. T. Austin (Eds.),The psychology research handbook: A guide for graduate students andresearch assistants (pp. 239–250). Thousand Oaks, CA: Sage.

Brown, S. C., & Park, D. C. (2003). Theoretical models of cognitive agingand implications for translational research in medicine. Gerontologist,43, 57–67.

Bruine de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individualdifferences in adult decision-making competence. Journal of Personalityand Social Psychology, 92, 938–956.

Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal ofPersonality and Social Psychology, 42, 116–131.

Carroll, J. B. (1974). Psychometric tests as cognitive tasks: A new “struc-ture of intellect.” Educational Testing Service, 74–116.

Carstensen, L. L. (1995). Evidence for a life-span theory of socioemotionalselectivity. Current Directions in Psychological Science, 4, 151–156.

Carstensen, L. L. (1998). A life-span approach to social motivation. InJ. Heckhausen & C. W. Dweck (Eds.), Motivation and self-regulationacross the life span (pp. 341–364). New York, NY: Cambridge Univer-sity Press.

Cattell, R. B. (1971). Abilities: Their structure, growth and action. NewYork, NY: Houghton Mifflin.

Clark, L. A., & Watson, D. (1995). Constructing validity: Basic issues inobjective scale development. Psychological Assessment, 7, 309–319.

Damasio, A. R. (1994). Descartes’ error: Emotion, reason, and the humanbrain. New York, NY: Avon.

Denes-Raj, V., & Epstein, S. (1994). Conflict between intuitive and ratio-nal processing: When people behave against their better judgment.Journal of Personality and Social Psychology, 66, 819–829.

Denney, N. W. (1989). Everyday problem solving: Methodological issues,research findings, and a model. In L. W. Poon, D. C. Rubin, & B. A.Wilson (Eds.), Everyday cognition in adulthood and late life (pp. 330–351). New York, NY: Cambridge University Press.

Donelle, L., Hoffman-Goetz, L., & Arocha, J. F. (2007). Assessing healthnumeracy among community-dwelling older adults. Journal of HealthCommunication, 12, 651–665.

Downs, J. S., Murray, P. J., Bruine de Bruin, W., White, J. P., Palmgren,C., & Fischhoff, B. (2004). An interactive video program to reduceadolescent females’ STD risk: A randomized controlled trial. SocialScience and Medicine, 59, 1561–1572.

Edwards, W. (1954). The theory of decision making. Psychological Bul-letin, 51, 380–417.

Ekstrom, R. B., French, J. W., Harman, H., & Derman, D. (1976). Manualfor kit of factor-referenced cognitive tests (Rev. ed.). Princeton, NJ:Educational Testing Service.

Epstein, S. (1994). Integration of the cognitive and the psychodynamicunconscious. American Psychologist, 49, 709–724.

Epstein, S. (2003). Cognitive–experiential self-theory of personality. In T.Millon & M. J. Lerner (Eds.), Handbook of psychology: Personality andsocial psychology (Vol. 5, pp. 159–184). Hoboken, NJ: Wiley.

Epstein, S., Pacini, R., Denes-Raj, V., & Heier, H. (1996). Individualdifferences in intuitive–experiential and analytical–rational thinkingstyles. Journal of Personality and Social Psychology, 71, 390–405.

Finucane, M. L., Alhakami, A., Slovic, P., & Johnson, S. M. (2000). Theaffect heuristic in judgments of risks and benefits. Journal of BehavioralDecision Making, 13, 1–17.

Finucane, M. L., Mertz, C. K., Slovic, P., & Schmidt, E. S. (2005). Taskcomplexity and older adults’ decision-making competence. Psychologyand Aging, 20, 71–84.

Finucane, M. L., Slovic, P., Hibbard, J., Peters, E., Mertz, C. K., &MacGregor, D. G. (2002). Aging and decision-making competence: Ananalysis of comprehension and consistency skills in older versusyounger adults considering health-plan options. Journal of BehavioralDecision Making, 15, 141–164.

Fitten, L. J., Lusky, R., & Hamann, C. (1990). Assessing treatmentdecision-making capacity in elderly nursing home residents. Journal ofthe American Geriatrics Society, 38, 1097–1104.

Frederick, S. (2005). Cognitive reflection and decision making. Journal ofEconomic Perspectives, 19(4), 25–42.

Grisso, T., Appelbaum, P., & Hill-Fotouhi, C. (1997). The MacCAT-T: Aclinical tool to assess patients’ capacities to make treatment decisions.Psychiatric Services, 48, 1415–1419.

Hess, T. M. (1990). Aging schematic influences on memory. In T. M. Hess(Ed.), Aging and cognition: Knowledge organization and utilization (pp.93–160). Amsterdam, the Netherlands: North-Holland.

Hibbard, J. H., Peters, E., Dixon, A., & Tusler, M. (2007). Consumercompetencies and the use of comparative quality information. MedicalCare Research and Review, 64, 379–394.

Hofer, S. M., Berg, S., & Era, P. (2003). Evaluating the interdependence ofaging-related changes in visual and auditory acuity, balance, and cog-nitive functioning. Psychology and Aging, 18, 285–305.

Horn, J. L., & Hofer, S. M. (1992). Major abilities and development in theadult period. In R. J. Sternberg & C. A. Berg (Eds.), Intellectualdevelopment (pp. 44–99). Cambridge, England: Cambridge UniversityPress.

Janofsky, J., McCarthy, R., & Folstein, M. (1992). The Hopkins Compe-tency Assessment Test: A brief method for evaluating patients’ capacityto give informed consent. Hospital and Community Psychiatry, 43,132–136.

Johnson, M. (1990). Age differences in decision making: A process meth-


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

odology for examining strategic information processing. Journals ofGerontology, Series B: Psychological Sciences and Social Sciences, 45,75–78.

Kahneman, D. (2003). A perspective on judgment and choice. AmericanPsychologist, 58, 697–720.

Kim, S. Y. H., Caine, E. D., Currier, G. W., Leibovici, A., & Ryan, J. M.(2001). Assessing the competence of persons with Alzheimer’s diseasein providing informed consent for participation in research. AmericanJournal of Psychiatry, 158, 712–717.

Kim, S. Y., Karlawish, J., & Caine, E. (2002). Current state of research ondecision-making competence of cognitively impaired elderly persons.American Journal of Geriatric Psychiatry, 10, 151–165.

Kitamura, F., Tomoda, A., Tsukada, K., Tanaka, M., Kawakami, I.,Mishima, S., & Kitamura, T. (1998). Method for assessment of compe-tency to consent in the mentally ill: Rationale, development, and com-parison with the medically ill. International Journal of Law and Psy-chiatry, 21, 223–244.

Kitamura, T., & Kitamura, F. (2000). Reliability of clinical judgment ofpatients’ competency to give informed consent: A case vignette study.Psychiatry and Clinical Neurosciences, 54, 245–247.

Kroenke, K., Spitzer, R. L., & Williams, J. B. W. (2003). The PatientHealth Questionnaire–2: Validity of a two-item depression screener.Medical Care, 41, 1284–1292.

Kutner, M., Greenberg, E., & Baer, J. (2005). A first look at the literacy ofAmerica’s adults in the 21st century (NCES 2006-470). Washington,DC: U.S. Department of Education, National Center for EducationStudies.

LeBoeuf, T. A., & Shafir, E. (2003). Deep thoughts and shallow frames:On the susceptibility to framing effects. Journal of Behavioral DecisionMaking, 16, 77–92.

Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001). General performance ona numeracy scale among highly educated samples. Medical DecisionMaking, 21, 37–44.

Löckenhoff, C. E., & Carstensen, L. L. (2007). Aging, emotion, andhealth-related decision strategies: Motivational manipulations can re-duce age differences. Psychology and Aging, 22, 134–146.

MacPherson, S. E., Phillips, L. H., & Sala, S. D. (2002). Age, executivefunction, and social decision making: A dorsolateral prefrontal theory ofcognitive aging. Psychology and Aging, 17, 598–609.

Marson, D. C. (1994). Determining the competency of Alzheimer patientsto treatment and research. Alzheimer Disease & Associated Disorders, 8,5–18.

Marson, D. C., McInturff, B., Hawkins, L., Bartolucci, A., & Harrell, L. E.(1997). Consistency of physician judgments of capacity to consent inmild Alzheimer’s disease. Journal of the American Geriatric Society, 45,453–457.

Meyer, B., Marsiske, M., & Willis, S. L. (1993). Text processing variablespredict the readability of everyday documents read by older adults.Reading Research Quarterly, 28, 234–249.

Meyer, B. J. F., Russo, C., & Talbot, A. P. (1995). Discourse comprehen-sion and problem solving: Decisions about the treatment of breast cancerby women across the life span. Psychology and Aging, 10, 84–103.

Mueller, R. O., & Hancock, G. R. (2008). Best practices in structuralequation modeling. In J. Osborne (Ed.), Best practices in quantitativemethods (pp. 488–508). Thousand Oaks, CA: Sage.

Palmer, B. W., Nayak, G. V., Dunn, L. B., Appelbaum, P. S., & Jeste, D. V.(2002). Treatment-related decision-making capacity in middle-aged andolder patients with psychosis: A preliminary study using the MacCAT-Tand HCAT. American Journal of Geriatric Psychiatry, 10, 207–211.

Park, D. C. (1992). Applied cognitive aging research. In F. Craik & T. A.Salthouse (Eds.), The handbook of aging and cognition (pp. 449–493).Hillsdale, NJ: Erlbaum.

Park, D. C., Willis, S. L., Morrow, D., Diehl, M., & Gaines, C. L. (1994).

Cognitive function and medication usage in older adults. Journal ofApplied Gerontology, 13, 39–57.

Parker, A. M., Bruine de Bruin, W., & Fischhoff, B. (2007). Maximizersversus satisficers: Decision-making styles, competence, and outcomes.Judgment and Decision Making, 2, 342–350.

Parker, A. M., & Fischhoff, B. (2005). Decision-making competence:External validation through an individual-differences approach. Journalof Behavioral Decision Making, 18, 1–27.

Peters, E., Dieckmann, N., Vastfjall, D., Mertz, C. K., & Slovic, P. (2009).Bringing meaning to numbers: The impact of evaluative categories ondecisions. Journal of Experimental Psychology: Applied, 15, 213–227.

Peters, E., Vastfjall, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert,S. (2006). Numeracy and decision making. Psychological Science, 1,407–413.

Raiffa, H. (1968). Decision analysis: Introductory lectures on choicesunder uncertainty. Reading, MA: Addison-Wesley.

Raven, J., Raven, J. C., & Court, J. H. (1998). Manual for Raven’sProgressive Matrices and Vocabulary Scales: Section 4. The AdvancedProgressive Matrices. San Antonio, TX: Harcourt Assessment.

Reyna, V. F., & Brainerd, C. J. (1995). Fuzzy-trace theory: An interimsynthesis. Learning and Individual Differences, 7, 1–75.

Reyna, V. F., Lloyd, F. J., & Brainerd, C. J. (2003). Memory, development,and rationality: An integrative theory of judgment and decision-making.In S. Schneider & J. Shanteau (Eds.), Emerging perspectives on decisionresearch (pp. 201–245). New York, NY: Cambridge University Press.

Roccaforte, W. H., Burke, W. J., Bayer, B. L., & Wengel, S. P. (1992).Validation of a telephone version of the Mini-Mental State Examination.Journal of the American Geriatrics Society, 40, 697–702.

Rönnlund, M., Karlsson, E., Laggnas, E., Larsson, L., & Lindström, T.(2005). Risky decision making across three arenas of choice: Areyounger and older adults differently susceptible to framing effects?Journal of General Psychology, 132, 81–92.

Rubin, D. B. (1987). Multiple imputations for nonresponse in surveys. NewYork, NY: Wiley.

Russo, J. E., Krieser, G., & Miyashita, S. (1975). An effective display ofunit price information. Journal of Marketing, 39, 11–19.

Ryan, J. J., Lopez, S. J., & Sumerall, S. W. (2001). Understanding testconstruction. In W. Dorfman & M. Hersen (Eds.), Understanding psy-chological assessment (pp. 1–15). New York, NY: Kluwer Academic/Plenum.

Salthouse, T. A. (1990). Cognitive competence and expertise in the aging.In J. E. Birren & K. W. Schaie (Eds.), Handbook of the psychology ofaging (3rd ed., pp. 310–319). New York, NY: Academic Press.

Salthouse, T. A. (1991). Mediation of adult age differences in cognition byreductions in working memory and speed of processing. PsychologicalScience, 2, 179–183.

Salthouse, T. A. (1993). Speed and knowledge as determinants of adult agedifferences in verbal tasks. Journals of Gerontology, Series B: Psycho-logical Sciences and Social Sciences, 48, 29–36.

Salthouse, T. A. (1996). The processing-speed theory of adult age differ-ences in cognition. Psychological Review, 103, 403–428.

Salthouse, T. A., & Babcock, R. L. (1991). Decomposing adult agedifferences in working memory. Developmental Psychology, 27, 763–776.

Salthouse, T. A., & Ferrer-Caja, E. (2003). What needs to be explained toaccount for age-related effects on multiple cognitive variables? Psychol-ogy and Aging, 18, 91–110.

Schaie, K. W., & Willis, S. L. (1999). Theories of everyday competenceand aging. In V. L. Bengtson & K. W. Schaie (Eds.), Handbook oftheories of aging (pp. 174–195). New York, NY: Springer.

Schmand, B., Gouwenberg, B., Smit, J. H., & Jonker, C. (1999). Assess-ment of mental competency in community-dwelling elderly. AlzheimerDisease & Associated Disorders, 13, 80–87.

Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

role of numeracy in understanding the benefit of screening mammogra-phy. Annals of Internal Medicine, 127, 966–972.

Shanteau, J. (1992). How much information does an expert use? Is itrelevant? Acta Psychologica, 81, 75–86.

Smith, A. D. (1996). Memory. In J. E. Birren (Ed.), The encyclopedia ofgerontology (pp. 107–119). New York, NY: Academic Press.

Smith, S. M., & Levin, I. P. (1996). Need for cognition and choice framingeffects. Journal of Behavioral Decision Making, 9, 283–290.

Stanley, B., Guido, J., Stanley, M., & Shortell, D. (1984). The elderlypatient and informed consent: Empirical findings. Journal of the Amer-ican Medical Association, 252, 1302–1306.

Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning:Implications for the rationality debate? Behavioral and Brain Sciences,23, 645–726.

Tanius, B. E., Wood, S., Hanoch, Y., & Rice, T. (2009). Aging and choice:Applications to Medicare Part D. Judgment and Decision Making, 4,92–101.

Thurstone, L. L. (1928). Attitudes can be measured. American Journal ofSociology, 33, 529–554.

United Nations Population Division. (2002). World population ageing:

1950–2050. Retrieved from http://www.un.org/esa/population/publications/worldageing19502050/index.htm

Vellinga, A., Smit, J., van Leeuwen, E., van Tilburg, W., & Jonker, C.(2004). Instruments to assess decision-making capacity: An overview.International Psychogeriatrics, 16, 397–419.

Weber, E. U., Goldstein, W. M., & Barlas, S. (1995). And let us not forgetmemory: The role of memory processes and techniques in the study ofjudgment and choice. In J. R. Busemeyer, R. Hastie, & D. L. Medin(Eds.), Decision making from the perspective of cognitive psychology(pp. 33–81). San Diego, CA: Academic Press.

Wechsler, D. (1987). Wechsler Adult Intelligence Scale—Third Edition.New York, NY: Psychological Corporation.

Willis, S. L., & Schaie, K. W. (1986). Practical intelligence in lateradulthood. In R. Sternberg & R. Wagner (Eds.), Practical intelligence(pp. 236–270). New York, NY: Cambridge University Press.

Received March 13, 2009Revision received January 19, 2010

Accepted January 29, 2010 �


This

doc

umen

t is c

opyr

ight

ed b

y th

e A

mer

ican

Psy

chol

ogic

al A

ssoc

iatio

n or

one

of i

ts a

llied

pub

lishe

rs.

This

arti

cle

is in

tend

ed so

lely

for t

he p

erso

nal u

se o

f the

indi

vidu

al u

ser a

nd is

not

to b

e di

ssem

inat

ed b

road

ly.

Developing a Tool for Measuring the Decision-Making Competence.pdf

Documents

Transcript of Developing a Tool for Measuring the Decision-Making Competence.pdf