Teorii Analize Memory Bias

8/10/2019 Teorii Analize Memory Bias

1/13

Psychological Bulletin1990, Vol . 107, No. 3.401-413

Copyright 1990 by the Amer ican Psychological Association, Inc.0033-2909/90/S00.75

Response Bias: Characteristics of Detection Theory, Threshold Theory,and Nonparametric Indexes

Neil A. MacmillanBrooklyn College of the City University of New York

C. Douglas CreelmanUniversity of Toronto

Toronto, Ontario, Canada

Models of discrimination based o n statistical decision theory distinguish sensitivity (the ability of anobserver to reflect a stimulus-response correspondence denned by the experimenter) from responsebias (the tendency to favor 1 response over others). Measures o f response bias have received lessattention than those o f sensitivity. Bias measures are classified here according to 1 characteristics.First, th e di stributions assumed or implied to underlie the observer's decision m ay be normal, logis-tic, or rectangular. Second, th e bias index ma y measure criterion location, cr iterion location relativeto sensitivity, o r likelihood ratio. Both parametric an d nonparametric indexes are classified in thismanner. The various bias statistics are compared on pragmatic and theoretical grounds, and it isconcluded that criterion location measures have many advantages in empirical work.

In a simple bu t important type of discrimination experi-ment, a stimulus from one of two classes (S \ or S2) is presentedon each trial, and an observer makes one of two correspondingresponses ( no or yes ). Examples of possible stimulusclasses are noise and signal-plus-noise in a detection task; newand old words in a recognition memory experiment; and nor-m al x-rays and those displaying abnormalities in medical diag-nosis.

Th e data from such an experiment can be summarized bytw o proportions. The hit rate H is the proportion of yes re-sponses on S 2 trials, and the false-alarm rate F is the proportion

of yes responses on 51

, trials. It is often useful, however, tosum marize the data by using tw o other statistics, a measure o fsensitivity and a measure of response bias. Sensitivity is the o b-server's ability to respond in accordance with the stimulus pre-sented, yes to S 2 and no to 5 ,. Response bias is the observ-er's tendency to use one of the two responses.

In this article, we compare alternative measures of responsebias. To begin, we propose some desirable attributes of a biasmeasure . We then present specific measures that derive fromone of three theoretical approaches: detection theory, thresholdtheory, and nonparametric analysis. In all cases, w e considermodels that assume that trials are independent, making noeffort to deal with sequential effects. We describe the relationsamong the bias measures and their relations to indexes of sensi-tivity. In a final section, we consider several possible rationalesfor deciding among bias measures and apply them to the candi-dates nominated earlier.

We are grateful to Howard Kaplan for many useful discussions ofthese issues. We also th ank Doris Aaronson, reviewers A. E. Dusoir andJ. G. Snodgrass, and an anonymous reviewer for helpful comments onprevious drafts. Th e work w as supported by a PSC-CUNY ResearchAward t o Neil A. Macmillan.

Correspondence concerning this article should be addressed to NeilA. Macmillan, Department of Psychology, Brooklyn College of the CityUniversity of New York, Brooklyn, N ew York 11210.

Dusoir (1975) discussed some of these measures, distinguish-ing those based on detection theory from nonparametric in-dexes. W e extend his theoretical work by placing the variousindexes in a more systematic psychophysical framework, show-ing that each measure of bias can be interpreted in terms of anassumed underlying decision space.

Our atti tude toward the problem of selecting an appropriatemeasure of response bias closely resembles the stance of Swets(1986b) in evaluating sensitivity indexes. Like Swets, we arguethat all measures have psychophysical consequences an d makedistributional assumptions; none is nonparametric. Swets's rec-

ommendations about sensitivity were ultimately empirical (seeSwets, 1986a). Because of some important differences betweensensitivity and bias, our conclusions depend more on practicalan d theoretical considerations.

General Characteristics of Sensitivity and Bias Measures

Treatments of bias and of sensitivity are parallel in someways, divergent in others. Although our interest is in bias, it isuseful to begin with some brief comments about sensitivity.

Sensitivity

Consider a modest requirement for sensitivity indexes: If twohit/false-alarm pairs have the same value of F, then the pair withthe higher hi t rate has higher sensitivity; if two pairs have t hesame value of//, then the pair with the lower false-alarm ratehas higher sensitivity. When this condition holds, the measureof se nsitivity is monotonically increasing with //and monotoni-cally decreasing with F. All proposed sensitivity measures thatwe are aware of have this property. A more stringent conditionrequires that H and F be treated symmetrically: Sensitivity isthe difference between H an d F, each transformed by a mono-tonic function u, the total possibly transformed by anothermonotonic function t;:

sensitivity = v [(//) u(F)]. (1)

40 1


2/13

402 NEIL A. MACMILLAN AND C. DOUGLAS CREELMAN

The interesting characteristics of a sensitivity measure canbe summarized by its receiver operating characteristic (ROC)curve, the function relating H to F when sensitivity is constantbut bias varies. Sensitivity indexes that monotonicaily increasein H and decrease in F have monotonicaily increasing ROCs.Those that take the form of Equation 1, as most do, are sym-metric around the minor diagonal of ROC space (that is, theuni t square).

Response Bias

Parallel to our requirement for sensitivity statistics is a pro-posal about bias measures: If two hit/false-alarm pairs have thesame value of F, then the pair with the higher hit rate displaysthe greater bias toward saying yes ; if two pairs have the samevalue of//, then the pair with the higher false-alarm rate showsgreater bias toward yes . When this constraint is satisfied, themeasure of response bias is monotonic with H and F in thesame direction (both increasing or both decreasing). This

monotonicity condition implies that when sensitivity declines,an observer who is maintaining the same bias should showboth a decreased hit rate and an increased false-alarm rate. Al-though this property seems as modest as the corresponding onefor sensitivity, many indexes satisfy it for only some values of//and F. T he false-alarm rate itself, for example, fails to satisfythe monotonicity condition, which requires that response ratesto both S, and S 2 contribute to an assessment of bias.

A more restrictive condition, similar to Equation 1, is

bias = v(u(H) + u(F)], (2 )

where u and v are monotonic functions. We consider some in-dexes that satisfy Equation 2, some that satisfy the monotonic-ity condition but not Equation 2, and some that violate eventhat requirement in some parts of their range.

The isobias curve of a bias statistic, analogous to the ROC ofa sensitivity measure, is the locus of (H, F) points for which biasis constant but sensitivity varies. The monotonicity conditionimplies monotonicaily decreasing isobias curves.

All significant information about a bias measure is capturedby its isobias curve. If a measure m\ is related to another mea-sure mi by any one-to-one transformation, both indexes havethe same isobias curves, for hit/false-alarm pairs that have aconstant value of m l also have a constant (in general, different)value of m 2 . We call measures having the same isobias curveequivalent; several instances of equivalent indexes are discussedlater. For comparing point estimates of bias across conditions,choices among equivalent measures are arbitrary; for otherpurposes, secondary considerations (such as the availability ofstatistical analysis) may render one particular transformation ofan index more useful than others.

Detection Theory Measures of Bias

Using the machinery of statistical decision theory, Equations1 and 2 can be translated into statements about the observer'sdecision space. According to detection theory, both 5, and S 2lead to underlying probability densities that are continuous.Two classes of detection-theoretic models have been especially

important: those in which the distributions are Gaussian

C r i t e r i o n

Given S

Figure 1. Decision space for the yes-no experiment, showing likelihooddistributions for stimulus classes S, and S 2 according to detection the-ory; the observer responds yes (classifies the stimulus as S2) for eventsabove the criterion.

(Green & Swets, 1966) and those in which they are logistic (asin Choice Theory; Luce, 1959, 1963a). Historically, the termSignal Detection Theory was used by the developers of Gaussianmodels; we sometimes use the abbreviation SDT for suchmodels.

If Si an d S2 give rise to densities that form a shift family, thatis, differ only in mean, then the transformation u in Equations1 and 2 is the inverse of the shared distribution function. Figure1 shows the decision space for a simple discrimination experi-ment, according to such a model. The observer is assumed torespond yes for observations above a fixed criterion and nofor those below that criterion. The function u converts hit andfalse-alarm proportions into distances from the criterion to themeans of the S2 and S\ distributions, and the difference betweenthese transformed proportions equals the difference betweenthe means of S2 and S,. (The function v in Equation 1 is theidentity function in this case.)

Three possible measures of response bias can be describedwithin the scheme of Figure 1: (a) the location of the criterion,(b) the location of the criterion divided by sensitivity, and (c) thelikelihood ratio, the height of the S 2 distribution at the criteriondivided by the height of the 5, distribution. The three indexesyield very different isobias curves whose exact shape dependson the assumed form of the underlying distributions.

Underlying Gaussian Distributions

Consider first the best-known SDT model, in which the un-derlying normal densities have equal variance. The meandifference d' is found from

d' = z(H) - z(F), (3 )

so that z, the inverse of the normal distribution function, is thefunction u of Equation 1. In Figure 2, the origin is placed atthe midpoint between the two densities, so that the distributionmeans are at d'/2.

Criterion location is measured relative to the origin, thecrossover point of the two distributions. Although this is tosome degree an arbitrary choice, two other possible referencepoints, the means of 51, and S 2 , can be ruled out. If measured

relative to the mean of 5 t , criterion location depends only on


3/13

RESPONSE BIAS 403

Figure 2. Decision space under Gaussian assumptions. (The decisionaxis is measured in units of the common standard deviation of the twodistributions [z scores]. Th e point of zero bias is arbitrarily set to zero.)

F; if measured relative to the m ean of 5 2 , only on H . Such in -dexes fail to satisfy the m onotonicity condition.

For the Gaussian SDT model, Figure 2 shows that the crite-rion location relative to the origin is the average of the trans-formed hit and false-alarm rates:'

c = -Q.5(z(H) (4)

Ingham (1970) was apparently the first to propose c as a biasmeasure. Clearly c satisfies Equation 2 and therefore the mono-tonicity condition. Isobias curv es imp lied by c are shown in Fig-ure 3a .

To normalize c by sensitivity, divide by d' :

c' = eld' = -0.5[z(H) + z(F)]/[z(H) - z(F)]. (5)

Isobias cu rves for c', shown in Figure 3b, fan out from the pointH = F= .5 . They satisfy the mon otonicity condition in the up-pe r left quadrant of ROC space but not elsewhere.

The likelihood ratio fiG (G for Gaussian) is the ratio of twonorm al ordinates:

fc = exp[-0.5z 2(//)]/exp[-0.5z 2(F)]. (6 )

The relation between likelihood ratio and c turns out to be ex-tremely simple, for

log()3G) = -0.5z2 (//) + 0.5z2(F)

= -0.5[z(//) + z(F)][z(H) - z(F)]

= c e f . (7 )

Log likelihood ratio equals the criterion location mu ltiplied bysensitivity. Notice that likelihood ratio can be computed by us-ing the z transformation only; values of normal ordinates arenot required.

Isobias curves for /3c fan out from the perfect-performancepoint, as shown in Figure 3c . Like those for c' , they satisfy themonotonicity condition only in the upper left quadrant.

Underlying Logistic Distributions

In Choice Theory, the underlying densities are logistic withvariance ir2/3 . The mean difference 21og(a) is found from2

log() = 0.5{log[///(l - H)] - log[F/(l - F)]}. (8)

Th e transformation from p to log[p/( 1 - p)] is the inverse of thelogistic distribution function. In Figu re 4, the origin is placed atthe midpoint between the two densities, and the distributionmeans are at log(a) (see McN icol, 1972; Noreen, 1977). Thecriterion location log(b) is defined by

log(fc) = -0.5{log[///(l - H)] + log[F/(l - F)]}, (9)so this bias me asure satisfies Equ ation 2.

To normalize log(A) by sensitivity,

b' = \og(b)/2\og(a). 1 0 )

The likelihood ratio f tL can be expressed in terms of the crite-rion and sensitivity parameters,

H)

or in terms of the hit and false-alarm rates (see Appen dix):

log(/?L) = - log[F(l - F)]

1 2 )Equation 1 1 displays relations amon g sensitivity, criterion loca-tion, and likelihood ratio that are similar to those found inEquation 7. For a criterion that is a fixed distance above thezero-bias point, likelihood ratio increases with sen sitivity; for anegative criterion, the relation i s inverse.

Because the logistic and Gaussian distribution functions arequite similar in shape, the isobias curves implied by the threemeasures are very m uc h like those of Figure 3.

Threshold Theory Measures

Threshold theories (Luce 1963a; Krantz, 1969) assume thatthe decision space is characterized by a small number of dis-crete states, rather than the con tinu ous dimensions of SDT andChoice Theory. Different threshold models a re d istinguished inthe ways these states are evoked by stimu lus classes and in theways they lead to responses. For each of the two models pre-sented here, we develop a state diagram that spells out theseconnections and determ ines the model's ROC.

A single ROC is, however, consistent with more than one setof und erlying distributions. Each threshold ROC can be inter-preted as arising from a decision space in which the S[ and S2densities are rectangular (Green & Swets, 1966, also adoptedthis conv entio n). These representations a re not mathem aticallyequivalent to the corresponding state diagrams, bu t becauseboth generate the same ROC, both are legitimate representa-tions of the model. The rectangular version has the importantadvantage of enabling comparison with the detection theorymodels already discussed.

We consider only two threshold theories. Both are character-ized by high thresholds; that is, both assume that some in ter-na l state(s) arise o nly from one stimulu s, whereas others may beactivated by either. Although low-threshold theory (Luce,

1 The minus sign is necessary to ensure that positive values of c corre-spond to locations above the equal-bias point.

2 We use a to denote the inverse of Luce's statistic T J , following theusage of McNicol (1972).


4/13

404 NEIL A. MACMILLAN AND C. DOUGLAS CREE LMAN

b )

0 .2 .4 .6 .8FALSE-ALARM RATE

Figure 3. Isobias curves, assuming Gaussian distributions, for (a) criterion location,(b) criterion location relative to sensitivity, and (c) likelihood ratio.

1963a, 1963b) predicts an ROC that more nearly agrees withexisting discrimination data than does high-threshold theory, ithas an important practical disadvantage: Two parameters areneeded to specify sensitivity and two to specify bias. Perhapsbecause of this complexity, the theory is little used.

Single H igh-Threshold Theory

The first model we consider is also the oldest (e.g., Blackwell,1953). Numerous authors (Egan, 1958; Green & Swets, 1966;Luce, 1963a) have pointed out that the theory's predicted ROCis contrary to data, and it is widely recognized that the singlehigh-threshold sensitivity index is inappropriate. The theory's

bias statistic, however, is the false-alarm rate F. The connection

of F to threshold theory is less generally appreciated, and thismeasure continues to be used by psychologists to assess bias.

State diagram and sensitivity measure. Single high-thresholdtheory proposes that sensitivity be measured by the adjusted( true ) hit rate:

This relation can be derived from a state diagram (Figure 5a)containing only two internal states, A and D 2 . The true hit rateq is the probability that stimulus S 2 leads to state D 2 . (Noticethat S t never leads to Z)2.) Observers respond yes wheneverin state D 2 and with probability u when in state D t; it is theselatter contaminating guesses that make the correction recom-

mended by Equation 13 necessary.


5/13

RESPONSE BIAS 405

The dependence of H an d F on the adjusted hit rate and theguessing rate can be calculated directly from the state diagram:

/>( yes |S2) = < 7 + M l - < ? )

/> ( yes |S , )=M. (14)

Eliminating the guessing parameter u leads to Equation 13.The ROC implied by the model is obtained by solving Equa-

tion 13 for H in terms of F. As shown in Figure 5b, it is a straightline from (q , 0) to (1, I) .3 It is this prediction of the theory thathas led to its rejection.

Underlying distributions. Now let us infer the underlying dis-tribut ions (see Figure 6). To allow for the point (q , 0), there mustbe a region on the decision axis where events only occur due toS2 otherwise the false-alarm rate w ould not be zero. In therest of the decision space, corresponding to the ROC segment(q, 0) to (1 ,1) , the 5, and S2 distribu tions could have any shape.They m ust be proportion al to e ach other, however, because theratio of their heights the likelihood ratio is constant whenthe R OC has constant slope. For simplicity, the u nd erlyin g dis-tribut ions are represented in Figure 6 as rectangles. The boun d-ary between these tw o areas is the threshold, the decision-axisvalue above which only S2 even ts occur. The sen sitivity parame -te r q is the p roportion of the S2 distribution that does not over-lap th e S, distribution.

Bias measure. A changing value of the parameter u (the pro-portion of A trials on which the observer responds yes ) ismodeled in this decision space by a shift in the criterion bu tnot by a change in likelihood ratio. The crite rion is located at aproportion 1 u of the distance spanned by the S, distribution,for H is the false-alarm rate. Accordingly, the criterion location,relative to the Si d istribution, is equivalen t to F . Isobias curve sfor the parameter u (or for F ) are vertical lines in RO C space,so u does not depend on H. A s a bias statistic, F is doubly flawed:It fails to satisfy the m onotonicity condition and it implicates amodel that has been rejected on othe r grounds.

Double High-Threshold Theory

Double high-threshold theory has a long history.4 Wood-worth (1938) presented a method for correcting recognitionmemory data for guessing that is equivalent to it. Egan (1958;

log D) - log n) 0 +log o)

l o g i t F | logil(H) H

Figure 4. Decision space under logistic assumptions. (Distances on the

decision axis are measured in logit un i t s of log \p/( 1 p)].)

(a)

STIMULUSSPACE

INTERNALSTATE RESPONSE

.2 .4 .6 .8

FALSE-ALARM R AT E

1.0

Figure 5. Single high-threshold theory: (a) state diagram, with twointernal states; and (b) receiver operating characteristic curve.

summarized in Green & Swets, 1966, pp. 337-341) was the firstto plot its ROC; he also pointed out th at single and d ouble high-threshold theories each call for a different correction for guess-ing and are special cases of the sam e gene ral model. The doublehigh-threshold theory has been ex plicitly discussed by M acmil-

lan and Kaplan (1985), Swets (1986b), Snodgrass and Corwin(1988), and Morgan (1976). It often arises implicitly via its sen-sitivity parameter, for this model has proportion correct as itsconstant performance measure.

State diagram. The state diagram of the un derlying model isshown in Figure 7a. There are three discrete states: D\ arisesonly when Si occurs, D2 can be triggered only by S 2 ,

ard aninterm ediate state, A, can occur for either stimulus. The modelassumes two high thresholds, each in sur mo un table by one ofthe stimuli.

3 We denote points in ROC space as (hit, false-alarm) pairs, the re-verse of the usual graphical convention, in which the order (x, y) is used.

4 We than k our anonymous reviewer for pointing this out to us.


6/13


1

ctn _X

x = DECISION AXIS

Figure 6. Rectangular underlying distributions consistent with single high-threshold theory.

Sensitivity measure p(c). As for single high-threshold theory,the sensitivity parameter in this model is a true hit rate q. Forsimplicity, this parameter is assumed to equal both the propor-tion of S 2 presentations leading to the D2 state and the propor-tion of Si presentations leading to the D\ state. The hit andfalse-alarm rates are

H = q + (1 q)v and

F=(l-q)v. (15)

It is easy to see that this model predicts an ROC of the formH = q + F; some examples are shown in Figure 7b.

For equal presentation probabilities, the proportion correctis the average of the hit and correct rejection rates, and

q = H - F = 2[0.5(// + l-F)]-l =2p(c)- 1. (16)Proportion correct is monotonically related to q and is thereforean equivalent sensitivity parameter of the model. If p(c) equals0.8, for example, the proportion q of trials falling in the sureZ > i and DI states is 0.6. Other trials lead to the uncertain stateDI and elicit yes and no responses according the observer'sresponse bias v.

A decision space consistent with the double high-thresholdROC is shown in Figure 8. The sensitivity parameter q equalsthe distance between the means of two rectangular distributionsand also equals the proportion of each distribution that doesnot overlap the other. The S\ and S 2 distributions each cover aregion corresponding to two states: S, presentations can lead toeither >, or Z >7 > S 2 presentations to either D 2 or D 7 . Unlike thedistributions of single high-threshold theory, those of doublehigh-threshold theory are symmetric. There are three values oflikelihood ratio: zero, one, and infinity.

Bias measures. Double-high threshold theory has two alter-native bias measures (both discussed by Dusoir, 1975, 1983),each of which parallels one from detection theory. We first cal-culate the criterion location relative to the equal-bias point. In

Figure 8, the center of the region of overlap is set to zero, and

the criterion k is measured with respect to this origin. ThenH = p(c) k and F = 1 p(c) k; solving these equationsyields

k=0.5[l-(F+H)]. (17)

The criterion is thus a simple linear transformation of 0.5(F +H), the yes rate when presentation probabilities are equal. Likethe c of SDT and log(b) of Choice Theory, the yes rate reflectsthe location of the criterion relative to the halfway point be-

tween the Si and S 2 distributions. The relation between k andp(c) is suggested by the similarity between Equation 17 and thecorresponding expression for sensitivity in the equal presenta-tion probability case:

The false-alarm and hit rates are added in Equation 17 and sub-tracted in Equation 18, and the same transformation is appliedto the result. We encountered a similar parallel for detection-theory models (compare, for example, Equations 3 and 4).

Like c and \og(b), the yes rate measures the same distancealong the decision axis whether the sensitivity measure is largeor small. If we linearly transform k into a new variable k' that

varies from 0 to 1, no matter what p(c) is, we obtain

that is, something that depends only on ( 1 - H)/F, the errorratio. The parameter k' is, in fact, equal to 1 t; and is thereforeequivalent as a bias measure to v, which was nominated for thisrole by Snodgrass and Corwin (1988). Atkinson (1963) pro-posed a different transformation of the same statistic.

Isobias curves for the yes rate and the error ratio are shownin Figure 9. As might be expected from their interpretation interms of criterion location, curves for the yes rate resembleSDT curves for criterion location, and curves for the error ratioresemble SDT curves for criterion location relative to sensi-

tivity.


7/13

RESPONSE BIAS 7

a

STIMULUS

SP E

INTERNALST TE RESPONSE

b )

Figure 7. Double high-threshold theory: (a) state diagram, with threeinternal states; and (b) receiver operating characteristic curve.

Measures Based on Areas in ROC Space

Sensitivity Measure A'

Th e better an observer perform s in a discrimination task, t hemore closely the ROC approaches the upper and left limits ofRO C space. An appealing measure of sensitivity, therefore, isthe area u nde r the ROC; like proportion correct w ith equal pre-sentation probabilities, this area increases from 0.5 for chanceperformance to 1.0 fo r perfect responding. This area eq uals theproportion correct by an unbiased observer in a two-intervalforced-choice ex periment (Green & Swets, 1966), giving it somecredibility as a natu ral, non parame tric statistic.

If only one point in ROC space is obtained in an experiment,there are m an y possible ROCs on which it could lie, an d someassump tions must be made to estimate the area un der the ROC.The measure A proposed by Pollack an d Norman (1964) is a

kind of average between min imu m an d maxim um performance

and can be calculated (Aaronson & Watts, 1987; Grier, 197 1;Snodgrass & Corwin, 1988) by

A = 0.5 + (H- F)(l + H- F)/[4H(l - F] i f H ^ F ,

A' = 0.5 + (F -//)( +F-H)/[4F(\ - H )] ifH


8/13


.2 4 6 8FA L S E A L A R M R AT E

1.0

Figure 9. Isobias curves for (a) k (yes rate) and (b) k' (error ratio),the bias measures of double high-threshold theory.

bias curves) above the major diagonal, for one is a monotonic

function of the other (see Appendix).The Hodos (1970) or Grier (1971) statistic is often paired

with A, just as /3G and d' are paired by users of Gaussian SDT,or b and a by users of Choice Theory. There is, however, nomodel that unifies A ' and B , as there is for the others. In fact,the relation of B to A is entirely superficial, but its connectionwith another sensitivity index is firm: Both B and B'H are mono-tonic functions of, and are therefore equivalent to, the likeli-hood ratio statistic ft in the logistic detection-theory model (seeAppendix). Their isobias curve, accordingly, is similar to thatshown in Figure 3c. These nonparametric indexes are consis-tent with a parametric likelihood ratio measure. That the para-metric-nonparametric distinction is vacuous has been argued

previously5

(Macmillan & Kaplan, 1985; Swets, 1986b; and

with regard to B 'H, by Dusoir, 1975). The equivalence of twomeasures from opposite sides of this supposed divide serves tomake the point concrete.

Comparing Response Bias Measures

We turn now to the task of comparing the many competingresponse-bias indexes. Faced with the analogous problem forsensitivity, Swets (1986b) was able to propose a single compel-ling touchstonethe form of the empirical ROC curvebywhich the quality of a sensitivity statistic might be ascertained.It is natural to ask whether empirical isobias curves might sim-ilarly provide an answer to the present problem, especially asthe shapes implied by the indexes we have considered are sovaried.

Dusoir (1983) essayed this approach. Using data from a seriesof tone-in-noise detection experiments with stimuli of varyingdetectability, he plotted isobias curves based on rating re-sponses. No measure was consistently supported; the three in-dexes rejected least often (for 9 of 21 data sets) were likelihoodratio, criterion location, and relative criterion location, all as-suming the Gaussian model.

Dusoir's (1983) ambiguous results probably reflect not justbad luck but an important asymmetry between sensitivity andbias. To demonstrate the in variance of a sensitivity statistic, itis necessary to hold relevant stimulus characteristics constantwhile varying, say, instructions. In most applications, it is quiteclear which stimulus characteristics are relevant. To show theinvariance of a bias index, on the other hand, we must holdmotivation constant while varying sensitivity. In general, ourunderstanding of even simple aspects of motivation lags far be-hind ou r understanding of, for example, psychoacoustics. Ex-periments like Dusoir's may be too optimistic: It is hard to besure that changes in stimulus factors do not also affect motiva-tion.

When isobias curves are collected empirically, they are morelikely to reflect truly constant bias if roving rather than fixeddiscrimination designs are used, that is, if stimulus pairs differ-ing in sensitivity are intermixed, so that the observer is encour-aged as much as possible to assign a constant meaning to aninstruction or rating. Unfortunately, this precaution does notguarantee success. Wood (1976) conducted a roving voice-onsettime experiment, but found that the Choice Theory parameter(here called b) varied systematically nonetheless. Wood pro-posed that his observers adjusted their bias from trial to trial onthe basis of their estimate of the discriminability of the pairpresented. If such strategies occur, they render empirical testingof isobias predictions very difficult.

The present obstacles to testing isobias predictions may even-tually be overcome by improved technique or theoretical ad-vance. As an example of the former, Snodgrass and Corwin(1988) collected isobias data for word recognition by varyingthe imageability of words; Snodgrass (personal communication,March 1989) pointed out that this method of manipulating dis-

3 The conclusion applies only to single (H, F) pairs. In rating experi-ments, area under the ROC is truly nonparametric (Green & Swets,

1966). Estimates of th is area from single ROC points, however, are not.


9/13

RESPONSE BIAS 409

Table 1Classification of Bias Measures

Bias parameter

Form of Criterionunderlying Criterion relative to Likelihood Sensitivity

distribution location sensitivity ratio parameter

GaussianLogistic

c(4)b(9)

c'(5)# (10 )

180(6) d'(3)

Rectangular I A : (17) 1[ yes rate J

3 ' (21)

( N A ) 'errorratio

Note. Numbers in parentheses refer to equations in which m easures aredefined. When two or more measures are bracketed, they are equivalent.NA = not applicable.

criminability is relatively opaque to the subject, who is thereforeunlikely to adopt strategies like that apparently used by Wood's(1976) observers. A useful theoretical advance would relate re-sponse bias to measurable aspects of the exper imental situation.Progress on this route has been made, for operant situationsusing animals, by McCarthy and Davison (1981).

For now, however, the data provide little guidance in evaluat-in g competing bias measures, and we use a different strategy.First, we note that all proposed statistics can be categorized ac-cording to two independent attributes of the underlying deci-sion space. The psychophysical parameter estimated may be cri-terion location, criterion location relative to sensitivity, or likeli-hood ratio; and the form of the underlying distributions may beGaussian, logistic, or rectangular. Combining these two parti-tions generates the taxonomy shown in Table 1.

Theoretical isobias curves for the measures in Table 1 arereplotted in Figure 10, this time including the below-chance re-gion of ROC space. The figure makes it clear that the decisioncharacteristics of a bias measure are much more importantthan the distributional assumptions. Curves for criterion loca-tion (or the yes rate) are nonintersecting, curves for criterionrelative to sensitivity (or the error ratio) fan out from a singlepoint, and curves for likelihood ratio fan out from ROC spacecorners. Our comparison of bias measures therefore focuses ondecision aspects but considers distributional assumptions

where necessary. We discuss five characteristics of our candi-date measures: (a) the form of the isobias curve, especially nearand below chance; (b) the relation between a bias index's rangeof values and sensitivity; (c) the distortions introduced by aver-aging data across observers and the likely need to use this strat-egy; (d) available strategies for statistical analysis; and (e) thetheoretical status of the measure.

Isobias Functions

We have argued that a minimum requirement for an isobiascurve is that H be a monotonically decreasing function of F.All measures satisfy this condition in the upper-left quadrant of

ROC space, where H ;> 0.5 and F


10/13


L I N E A R A X E S N O R M A L A X E S

2

0 -

-2

- 2 0 2

CONS TA NTCRITERION

2 -

0 -

-2-

- 2 0 2

CONSTANTR E L AT I V ECRITERION

. 5 -

2 -

0 -

-2-1

- 2 0 2

C O N S TA N TLIKELIHOODRAT IO

. 5 -

2

0 -

-2

.5 - 2 0 2

C O N S TA N T1'ES-RATE

2 -

0 -

-2-1

C O N S TA N TERROR-RATIO

0 5 - 2 0 2


11/13

RESPONSE BIAS 411

dition applies to criterion location with rectangular distribu-tions (for which th e curves are parallel on linear coordinates)and criterion location with logistic or Gaussian distributions(for which they are parallel on transformed coordinates). Forthe error ratio, isobias curves converge only at very h igh sensi-tivity. The condition is not met by relative criterion location orlikelihood ratio (w ith logistic or Gaussian distribu tions), whosecurves converge for both very high and very low sen sitivities.The issue of range independence is clearly ve ry close to the pre-viously discussed issue of behavior near chan ce, and our evalua-tions by these tw o criteria are, predictably, similar.

Averaging Across Observers

In some applications, frequency data must be averaged acrossobservers before summary statistics are computed. Such aver-aging can be more or less benign, in the sense that the statisticbased on pooled data may or may not be close to the averageof the same statistic based on in dividu al data. M acmillan and

Kaplan (1985) considered the effects of pooling on sensitivitystatistics; w hich of our bias indexes are best in this sense?

In the simplest situation of this type (and the only one weconsider here), tw o observers have the same bias bu t differentsensitivity. Collapsed bias equ als average bias when ever isobiascurves are straight lines (of any slope) on linear coordinates.Both m easures based on rectangular distributions, the yes rateand the error ratio, have this property. Next best are criterionlocation and relative criterion location, with logistic or Gauss-ian distributions. Isobias curves for these index es are linear onz coordinates, and the error involved in averaging is moderateas long as sensitivities are not too different. (This conclusion isanalogous to that of Macmillan and Kaplan, 1985, about sensi-

tivity.)Likelihood ratio is most susceptible to distortion due to aver-

aging, because the curves for this statistic are most deeplycurved. The discontinuities o f these curve s in going from aboveto below chance are e specially disruptive.

Statistical Analysis

When bias is estimated from experimental data, it is impor-tant to k now the variability of the estimate so as to constructconfidence intervals and test hypotheses. Bias indexes whosestatistical properties are simple or well-understood are to bepreferred over those whose properties are n ot.

Criterion location, assuming logistic, Gaussian, or rectangu-lar distributions, is statistically the most tractable conceptionof bias. For Gaussian distributions, Banks (1970) pointed outthat, because var(c) = var(J')/4 (Equations 3 and 4), the vari-ance of c can be estimated by using the approximation ofGourevitch and Galanter (1967). Assuming logistic distribu-tions, \og(b) can be estimated, and hypotheses about it tested,by u sing log-linear modeling (Macm illan & Kaplan, 1986), a

standard technique. Under the rectangular-distribution model,the yes rate has a binomial distribution (M organ, 1976).

We are not aware of any statistical studies of likelihood ratioor relative criterion. Ce rtainly it would be valuable to find, bycalculation or sim ulation, the distributions of these indexes.

Theoretical StatusIn this final category, we allow theoretical properties of bias

measures to intrude on our pragm atic e valuation . Specifically,we consider the historical importance of likelihood ratio andthe virtues of bias m easures derived from otherwise successfulmodels of discrimination.

In SDT, likelihood ratio plays a cen tral role. Green an d Swets(1966, chap. 1) showed that comparing observations with afixed likelihood ratio was the optimal decision procedure, undervarious definitions o f optimal. In fact, the specific optimal levelcan be calculated from such experimen tal variables as presenta-tion probabilities an d payoffs. Also, optimal models can bespecified in terms of likelihood ratio for discrimination designstoo complex for a simple criterion construct (e.g., Macm illan,Kaplan, & Creelman, 1977).

Likelihood ratio was able to play such an imp ortant role his-torically, in spite of its imperfections, because of a significantlimitation of early work in SDT: In most experiments, th e alter-native stim uli were fixed, so that sensitivity did not vary. Whe nthere is only a single RO C curve to consider, any measure ofbias will serve. This history has led to the dutiful use of /3G incases for which some measure of criterion location would bemore instructive.

Second, how important is the assumed form of the underlyingdistribution in choosing a bias measu re? In much of the forego-ing discussion, the best measures have come with equal creden-tials from continu ous (Gaussian or logistic) and discrete (or rec-tangular) distributions. Considering on ly the bias problem, nei-ther type of distribution al assumption is stronger th an the other,and neither can be viewed as the default choice. But an argu-ment can be made for preferring c and b, fo r example, over th eyes rate: Other things being equal, one should use sensitivityand bias statistics that arise from the same rather than differentunderlying models. On the basis of sensitivity predictions(ROCs), normal and logistic models are superior to discrete orrectangular ones.

Considerations of theoretical con sistency also have relevancefor the ma ny researchers who report A' as a measure of sensitiv-ity an d B as a measu re of bias. Although both statistics can bemotivated by examining the geometry of the unit square, wehave seen that they have no legitimate relationship. A moreprincipled choice is to pair B with logistic sensitivity.

Overall Comparison of Measures

In Table 2, we summ arize our evaluation of bias statistics.For each test, a measure is rated + if it performs well with

Figure 10. Isobias curves for c, c', /3 0, k, and k', on both linear and z coordinates; all curves include theregion below the chance diagonal. (Logistic indexes, including B , a re omitted because o f their similarity

to Gaussian ones.)


12/13


Table 2Comparison of Measures

Property

Bias measure

c, l og Z>) k, yes rate c b k', error ratio 0 0 , ft,, B , B 'H

Isobias is monotonicIsobias is we ll-behaved

at chanceIsobias is well-behaved

below chanceRange is independent

of sensitivityAveraging does not

distortStatistical analysis is

well-knownAssociated with

desirable index ofsensitivity

Useful in theory

+

+

0

Note. + = measure performs well with regard to property; - = measure performs poorly; 0 = marginalsuccess. If statistic is inverted for below-chance performance, entry is +.

regard to that criterion, - if it performs poorly, and 0 formarginal success. As the bases of comparison are neither ofequal weight nor wholly independent, we do not attempt to re-duce our assessments to single numbers.

Still, one specific comparison seems justified: Criterion loca-tion, especially as calculated from normal or logistic models,deserves more attention as a bias index. Contrariwise, the im-portance of likelihood ratio in theory does not argue for its au-tomatic use in data analysis.

SummaryWe have considered eight distinct measures of response bias,

in addition to some that are equivalent to one or another ofthese. All can be classified as reflecting criterion location, crite-rion location relative to sensitivity, or likelihood ratio and asassuming Gaussian, logistic, or rectangular underlying distribu-tions.

Measures were judged superior if their isobias curves weremonotonic, if they measured bias sensibly at and below chance,if their range was independent of sensitivity, if they were notdistorted by averaging across observers, if they lent themselvesto statistical analysis, and if they were of theoretical impor-tance. By these standards, indexes of criterion location (c, b,and the yes rate) are the best response bias statistics. Detection-theory measures (those assuming logistic or normal distribu-tions) have some advantages over those derived from thresholdtheory, but nonparametric indexes are actually equivalent toa logistic detection-theory statistic. An important problem forfuture research is to discover experimental manipulations forwhich specific bias measures are invariant.

ReferencesAaronson, D., & Watts, B. (1987). Extensions of Crier's computational

formulas for A' an d B to below-chance performance. Psychological

Bulletin, 102, 439-442.

Atkinson, R. C. (1963). A variable sensitivity theory of signal detection.Psychological Review, 70,91-106.

Banks, W. P. (1970). Signal detection theory and human memory. Psy-chological Bulletin, 74,61-99.

Blackwell, H. R. (1953). Psychoph ysical thresholds: Experimental stud-ies of methods of measurement (Engineering Research Bulletin 36).An n Arbor: University of Michigan.

Dusoir, A. E. (1975). Treatments of bias in detection and recognitionmodels: A review. Perception & Psychophysics, 17, 167-178.

Dusoir, T. (1983). Isobias curves in some detection tasks. Perception &Psychophysics, 33, 403-312.

Egan, J. P. (1958). Recognition memory and the operating characteristic(Tech. Note AFCRC-TN-58-51). Bloomington: Indiana University,Hearing and Communication Laboratory.

Gourevitch, V., & Galanter, E. (1967). A significance test for one-parameter isosensitivity functions. Psychometrika, 32, 25-33.

Green, D. M., & Swets, J. A. (1966). Signal detection theory and psycho-physics. N ew York: Wiley.

Grier, J. B. (1971). Nonparametric indexes for sensitivity and bias:Computing formulas. Psychological Bulletin, 75, 424-429.

Hodos, W. (1970). Nonparametric index of response bias for use in detec-tion and recognition experiments. Psychological Bu lletin, 74, 351-354.

Ingham, J. G. (1970). Individual differences in signal detection. ActaPsychologica, 34, 39-50.

Krantz, D. H. (1969). Threshold theories of signal detection. Psycholog-ical Review, 76, 308-324.

Luce, R. D. (1959). Individual choice behavior. New York: Wiley.Luce, R. D. (1963a). Detection and recognition. In R. D. Luce, R. R.

Bush, & E. Galanter (Eds.), Handbook of mathematical psychology(Vol. 1, pp. 103-189). New York: Wiley.

Luce, R. D. (1963b). A threshold theory for simple detection experi-ments. Psychological Review, 70, 61-79.

Macmillan, N. A., & Kaplan, H. L. (1985). Detection theory analysisof group data: Estimating sensitivity from average hit and false alarmrates. P sychological Bulletin. 98, 185-199.

Macmillan, N. A., & Kaplan, H. L. (1986). Log-linear models provideanswers to signal-detection questions (abstract). Bulletin of the Psy-

chonomic Society, 2 4, 342.


13/13

RESPONSE BIAS 413

Macmillan, N. A., Kaplan, H. L., & Creelman, C. D. (1977). The psycho-physics of categorical perception. Psychological Review, 84 ,452-471.

McCarthy, D., & Davison, M. (1981). Towards a behavioral theory ofas in signal detection. Perception & Psychophysics, 29 , 371-382.

McNicol, D. (1972). A primer o f signal detection theory. London: Allen& Unwin.

Morgan, B. J. T. (1976 ). The u niform distribution in signal detectiontheory. British Journal of Mathematical and Statistical Psychology,29,81-88.

Noreen, D. L. (1977). Relations among some models of choice. Paperpresented at Mathematical Psychology Meeting, University of Cali-fornia, Los Angeles.

Pollack, L, & Norm an, D. A. (1964). A nonparametric analysis of rec-ognition experiments. Psychonomic Science, I, 125-126.

Snodgrass, J. G., & C orwin, J. (1988). Pragmatics of me asuring recogni-tion memory: Applications to dementia and amnesia. Journal o f Ex-perimental Psychology: General, 117, 34-50.

Swets, J. A. (1986a). Form of em pirical ROCs in discrimination anddiagnostic tasks. Psychological Bulletin, 99, 181-198.

Swets, J. A. (1986b). Indices of discrimination or diagnostic accuracy:

Their ROCs and implied models. Psychological B ulletin, 99, 100-117.Waldmann, M. R., & Gottert, R. (1989). Response bias in below-chanceperformance: Computation of the parametric measure /3 . Psychologi-cal Bu lletin, 106, 338-340.

Wood, C. C. (1976). Discriminability, response bias an d phoneme cate-gories in discrimination of voice onset time . Journal o f th e AcousticalSociety o f America, 60, 1381-1389.

Woodworth, R. S. (1938). Experimental psychology. New York: Holt.

Appendix

Relations Among Logistic-Distribution Statistics

Likelihood Ratio, Criterion Location, and Sensitivity

We wish to show that, according to Choice Theory, the likeli-hood ratio is given by Equation 12 . Hit and false-alarm ratesare as follows (Luce, 1963a, p. 123):

H = /( + b) ,

F= l / ( l +ab).

Substituting these expressions fo r H an d F into Equation 12 ,we obtain

H( 1 - H)/[F( 1 - F)] = ((1 + ab)/(a + b)] 2. (A 1)

We now show that the right side of Equation A 1 (and there-fore the left side) equals the likelihood ratio. The logistic densityfunction can be written

+e x- ) 2 . (A2)

The Choice Theory yes-no model is consistent with two logisticdistributions having means of log(a) and a criterion locatedat log(). Substituting /u = log(a) and x = \og(b) into EquationA2,

, -log(a)]

= [(1 + ab)/(a + b)}2.

Thus, combining Equations A 1 and A 3,

(A3)

which is Equation 12 .

Likelihood Ratio, B , a nd B'H

W e wish to show that /3L, B , an d B'H are equivalent, that is ,have the same isobias curve. To do this, it is sufficient to showthat they are monotonic functions of each other.

From Equations 2 1 an d 12 ,

= |8 L -1 if (A4)Equation A4 represents B'H as a monotonic increasing functionof j3L over its entire range, so these two measures are equivalent.

The relation between B and ft L is also monotonic. Combin-ing Equations 22 and 12 (for above-chance performance),

Because B'H an d B ar e each monotonic functions of /3L , theyare themselves monotonically related. The actual relation is

B = B'H/(2 + if H> F

= B 'H/( 2 - B'H) if H

Teorii Analize Memory Bias

Documents

Transcript of Teorii Analize Memory Bias