Misresponse to Reversed and Negated Items in Surveys: A Review
description
Transcript of Misresponse to Reversed and Negated Items in Surveys: A Review
Misresponse to Reversed and Negated Items in Surveys: A Review
Bert WeijtersHans Baumgartner
Misresponse to Reversed and Negated Items
Motivation Should reversed items be included in multi-item
summative scales?□ The use of reversed items in surveys often leads to
problems (low reliabilities, poor fit, distorted factor structure);□ Reversed items control for acquiescence, serve as cognitive
“speed bumps” and may encourage more complete coverage of the construct’s domain of content;
If reversed items are to be used, does it matter whether the reversal is achieved through negation or through other means?
What’s the link between reversal and negation, what types of MR result, what psychological mechanisms are involved, and how can MR be avoided?
Misresponse to Reversed and Negated Items
Item reversal vs. item negation Authors often fail to draw a clear distinction between
reversals and negations and use ambiguous terms such as ‘negatively worded items’, which makes it unclear whether they refer to reversed or negated items, or both;
Examples from the Material Values scale (Richins and Dawson 1992):
□ It sometimes bothers me quite a bit that I can't afford to buy all the things I’d like.
□ I have all the things I really need to enjoy life.□ I wouldn't be any happier if I owned nicer things.
Misresponse to Reversed and Negated Items
Empirical data we analyzed items from volumes 1 through 36 of
JCR (1974 till the end of 2009) and volumes 1 through 46 of JMR (1964 to 2009);
we included all Likert-type scales for which the items making up the scale were reproduced in the article and factor loadings or item-total correlations were reported;
total of 66 articles in which information about 1330 items measuring 314 factors was provided;
of the 1330 distinct items in the data set, 608 came from JCR and 722 from JMR;
Misresponse to Reversed and Negated Items
Item negation Items can be stated either as an assertion (affirmation) or
as a denial (disaffirmation) of something (Horn 1989); Negation is a grammatical issue; Classification of negations in terms of two dimensions:
□ what part of speech is negated (how a word is used in a sentence: as a verb, noun/pronoun, adjective, adverb or preposition/conjunction);
□ how the negation is achieved (by means of particle negation, the addition of no, the use of negative affixes, negative adjectives and adverbs, negative pronouns, or negative prepositions);
Misresponse to Reversed and Negated ItemsNegated by means of
Part of speech
Not, n’t No negative affixes negative adjectives
and adverbs
negative pronouns
negative prepositions
Total
Verb [This salesperson does not make false claims.]
n.a. dislike, disagree, etc.
[I dislike food shopping very
much.]
reluctant, hesitant, never, rarely, seldom, hardly (ever), less, little, etc.
[I seldom daydream.]
n.a. without, instead of, rather than,
etc.[This supplier
sometimes promises to do things without actually doing
them later.]
135 (44.7%) 5 (1.7%) 26 (8.6%) 5 (1.7%) 171 (56.6%)
Noun/pronoun
not everyone, not (only), etc.
[I and my family will consume only certain brands, not
others.]
no object, no reason, no
purpose, etc.[Clipping,
organizing, and using coupons is
no fun.]
discomfort, disagreement,
etc.[There is
considerable disagreement as
to the future directions that this hospital should take.]
little, few, a lack of, none of the,
not much, neither of, etc.
[Many times I feel that I have little influence over things that happen to me.]
no-one, nobody, none, nothing,
etc.[Energy is really not my problem because there is simply nothing I can do about it.]
except for, without, with the
exception of, instead of, rather
than, etc.[American
people should always buy
American-made products instead
of imports.]
4 (1.3%) 17 (5.6%) 5 (1.7%) 10 (3.3%) 4 (1.3%) 14 (4.6%) 54 (17.9%)
Adjective n.a. n.a. uninterested, dishonest, etc.
[Most charitable organizations are
dishonest.]
rarely, less, etc.[I would be less loyal to this rep
firm, if my salesperson
moved to a new firm.]
n.a. n.a.
55 (18.2%) 4 (1.3%) 59 (19.5%)
Misresponse to Reversed and Negated ItemsNegated by means of
Part of speech
Not, n’t No negative affixes negative adjectives
and adverbs
negative pronouns
negative prepositions
Total
Adverb not much, etc.[After I meet
someone for the first time, I can
usually remember what they look like, but not much about them.
no longer, etc.[Hard work is no longer essential
for the well-being of society.]
[I often dress unconventionally even when it's likely to offend
others.]
rarely, less, etc.[I feel I have to do things hastily and maybe less
carefully in order to get everything
done.]
n.a. n.a.
2 (.7%) 1 (.3%) 3 (1.0%) 1 (.3%) 7 (2.3%)
Preposition/ Conjunction
not for, not (just) in, not (only) as, not until, not if (incl. unless),
not because, etc.[I enjoyed this
shopping trip for its own sake, not just for the items
I may have purchased.]
n.a. n.a. n.a. for nothing, etc.[I don't believe
in giving anything away for nothing.]
n.a.
10 (3.4%) 1 (.3%) 11 (3.7%)
Total 151 (50.0%) 18 (6.0%) 68 (22.5%) 41 (13.5%) 5 (1.7%) 19 (6.3%) 302
Misresponse to Reversed and Negated Items
Item reversal an item is reversed if its meaning is opposite to a
relevant standard of comparison (semantic issue); three senses of reversal:
□ reversal relative to the polarity of the construct being measured;
□ reversed relative to other items measuring the same construct:
reversal relative to the first item reversal relative to the majority of the items
□ reversal relative to a respondent’s true position on the issue under consideration (Swain et al. 2008);
Misresponse to Reversed and Negated Items
Item reversal (cont’d) in our data set of 1330 items, between 83 and 86
percent of items were nonreversed (depending on the definition of reversal);
the proportion of factors (or subfactors in the case of multi-factor constructs) that do not contain reversed items was 70 percent;
only 8 percent of factors (out of 314 factors) were composed of an equal number of reversed and nonreversed items (i.e., the scale was balanced);
Misresponse to Reversed and Negated Items
Integrating item negation and item reversal
Item reversal
Non-reversed Reversed
Item negation
Non-negated
Regular (RG) items
Talkative, enjoying talking to people
Polar opposite (PO) items
Quiet, preferring to do things by oneself
Negated
Negated polar opposite (nPO) items
Not quiet, preferring not to be by oneself
Negated regular(nRG) items
Not talkative, not getting much pleasure chatting with people
Misresponse to Reversed and Negated Items
Cross-classification of negation and reversal
Reversal relative to
Polarity of construct Polarity of first item Dominant keying direction Total
Non-reversed Reversed Non-
reversed Reversed Non-reversed Balanced Reversed
No negation(s) 71.1% 8.7% 71.0% 8.7% 70.3% 5.2% 4.2% 79.7%
Negation(s) 11.7% 8.7% 15.4% 4.9% 13.7% 2.9% 3.7% 20.3%
Total 82.8% 17.4% 86.4% 13.6% 84.0% 8.1% 7.9% 100.0%
Misresponse to Reversed and Negated Items
Misresponse to negated and reversed items
Item Consistent responding
Misresponse to negated items
(NMR)
Misresponse to reversed items
(RMR)
Misresponse to polar opposites
(POMR)
Talkative (RG) A A A A
Not talkative (nRG) D A A D
Quiet (PO) D D A A
Not quiet (nPO) A D A D
MR → within-participant inconsistency in response to multiple items intended to measure the same construct;
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Comprehension
Careless responding (Schmitt and Stults 1985):□ respondents fail to pay careful attention to individual
items and respond based on their overall position on an issue [RMR];
□ more likely when a reversed item is preceded by a block of nonreversed items;
□ Remedies: alert Rs to the presence of reversed items, avoid
tedious surveys, and be careful with involuntary Rs; increase Rs’ ability and avoid distractions; use balanced scales, alternate the keying direction, and
disperse the items;
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Comprehension (cont’d)
Reversal ambiguity: Rs may not view antonyms as polar opposites [POMR]; contradictories vs. contraries:
Antonyms can be contradictories or contraries, depending on whether they are bounded or unbounded (Paradis and Willners 2006);
Negations of the same core concept are clearly contradictories, but things are more complicated when the core concept differs;
simultaneous disagreement is more likely when items are worded extremely (McPherson and Mohr 2005);
“Buddhism’s ontology and epistemology appear to make East Asians relatively comfortable with apparent contradictions” (Wong et al. 2003, p. 86) [RMR];
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Comprehension (cont’d)
□ Remedies: use more sophisticated procedures to identify
appropriate antonyms (formulate linguistic contrasts in two stages; see Dickson and Albaum 1977);
may be particularly useful in cross-cultural research; bounded antonyms have to be pretested and
unbounded antonyms have to be used with care; extreme statements should be avoided;
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Retrieval
Item-wording effects:□ Confirmation bias (Davies 2003; Kunda et al. 1993);□ Directly applicable to antonymic reversals;□ For negation reversals, confirmation bias can lead to
MR if a non-negated polar opposite schema is readily available (Mayo et al. 2004);
□ Remedies: Use polar opposite reversals to get richer belief samples,
even though they may increase apparent MR; Negation reversals have few retrieval benefits;
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Retrieval
Positioning effects:□ Dispersed PO items reduce carryover effects and can
increase coverage, but the task is more taxing for Rs and internal consistency may suffer;
□ Item similarity may determine whether Rs engage in additional retrieval when items are grouped together;
□ Remedies: The use of dispersed antonyms should encourage the
generation of distinct belief samples; Avoid very similar (negated) statements when items are
grouped;
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Judgment
Item verification difficulty (Carpenter and Just 1975; Swain et al. 2008):□ MR is a function of the complexity of verifying the
truth or falsity of an item relative to one’s true beliefs, which depends on whether the item is stated as an affirmation or negation [NMR];
□ Remedies: Negations are problematic because they increase the
likelihood of making mistakes (remember there are many types of negations);
Negated polar opposites are most error-prone; Mix of regular and PO reversals should be best;
Misresponse to Reversed and Negated Items
Item verification difficulty
Truth valueTrue False
Negation
Affirmation
MR
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Response
Acquiescence: Rs initially accept a statement and subsequently re-consider it based on extant evidence; the first stage is automatic, the second requires effort (Knowles and Condon 1999) [RMR];
Remedies: □ Although response styles are largely individual
difference variables, situational factors may be under the control of the researcher (e.g., reduce the cognitive load for Rs);
□ Problems with online surveys;
Misresponse to Reversed and Negated Items
Theoretical explanations of MR:Response (cont’d)
Asymmetric scale interpretation: the midpoint of the rating scale may not be the boundary between agreement and disagreement for Rs (esp. if the response categories are not labeled; cf. Gannon and Ostrom 1996) [RMR];
Remedies:□ Use fully labeled 5- or 7-point response scales;
Misresponse to Reversed and Negated Items
Consequences of MR Effect on scale reliability: [based on 4 studies]
RG . .68nRG .60PO .57nPO .49
Effect on correlations between items measuring the same construct:
□ when content saturation is high, social desirability is low, and endorsement frequencies are moderate, the correlation between the four types of items can be quite high, although the findings showed evidence of POMR (Jackson and Lay 1968);
□ Ahlawat (1985) showed that the correlations between nPO and other items were particularly low;
Misresponse to Reversed and Negated Items
Consequences of MR (cont’d) Effect on factor structure: [based on three studies]
□ The factor structure can emerge clearly when the items are high in content saturation and free of social desirability, but RMR and POMR are apparent;
□ In Schriesheim and Eisenbach (1995), RG items performed best and nPO items did very poorly;
Effect on correlations with other scales:□ Holden et al. (9185) showed that item negation was
more detrimental to criterion-related validity than item reversal;
Misresponse to Reversed and Negated Items
Example item with 4 negationsTop management in my company has let it be known in no uncertain terms that unethical behaviors will not be tolerated.