Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe..........
-
Upload
ralf-gaines -
Category
Documents
-
view
218 -
download
0
Transcript of Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe..........
![Page 1: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/1.jpg)
Perspectival Diversity and Consensus Analysis
John Gatewood . . . . . . Lehigh University
John Lowe . . . . . . . Cultural Analysis Group
AAA Meetings, Philadelphia, Dec 5, 2009
![Page 2: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/2.jpg)
Preview
• INTRODUCTORY REMARKS– Problem of “culture-sharing” (and non-sharing)– Basic patterns of inter-informant agreement … adding
“perspectival diversity” to the list
• OUR CURRENT STUDY– Assessing effects of different distributional patterns on
consensus analysis’s key indicators– Some initial findings
• CONCLUSIONS– Methodological “lessons” for researchers– Future directions
![Page 3: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/3.jpg)
INTRODUCTORY REMARKS
![Page 4: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/4.jpg)
Problem of “Culture-Sharing”
• By definition, culture is socially transmitted knowledge; hence, it must be “shared” … but sharing is always a matter of degree
• Hence, two related issues for any given cultural domain:1. How much knowledge is shared?
(the AVERAGE “cultural competence”)
2. How is the knowledge socially distributed?(the DISTRIBUTIONAL PATTERN)
• KEY INSIGHT = assess degree of culture-sharing by examining patterning of inter-informant agreement
![Page 5: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/5.jpg)
Basic Patterns of Agreement
• Boster (1980, 1985) … four basic patterns of agreement (paraphrasing & expanding):
1. UNIFORM agreement – traditional view of culture
2. RANDOM agreement – free variation → no culture
3. EXPERTISE gradient – experts tend to agree with one another whereas non-experts deviate randomly
4. SUBCULTURAL variation – more than one ‘school of thought’a. Competing answer sets – different groups, different truths
b. Complementary knowledge – different groups systematically know different things
• Romney, Weller & Batchelder (1986) … cultural consensus theory & consensus analysis
![Page 6: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/6.jpg)
Perspectival Diversity … a 5th Pattern
• Pilot study of credit union employees (Gatewood & Lowe 2006)
No consensus in sample *+
No identifiable subcultural groups
=> Fish-scale overlappings of partial knowledge… “perspectival diversity”
i.e., social interaction and knowledge among the employees was rather departmentalized … their understandings of ‘credit unions’ reflected what they needed to know to perform their own jobs, not necessarily what might be relevant to other people
_______________________ * Pilot study’s conclusion about “no consensus” turned out to be an artifact of our failure to
counter-balance items in the questionnaire form (see Gatewood & Lowe 2008) … but that’s another story
![Page 7: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/7.jpg)
To generalize, perspectival diversity occurs when…
a) All individuals have limited knowledge with respect to a given domain and to approximately the same degree
b) Each individual’s range of knowledge only partially overlaps with the ranges known by others
And, consistent with this definition, different geometries of perspectival diversity are possible … e.g., circular pattern, linear pattern, taxonomic-hierarchical,
overlapping polygons on a surface, etc.
![Page 8: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/8.jpg)
OUR CURRENT STUDY
![Page 9: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/9.jpg)
RESEARCH QUESTION:• Ceteris paribus, do the different distributional patterns
affect the key indicators of consensus analysis?– As the average knowledge in a sample varies, do different
distributional patterns “show” consensus more readily than other patterns?
– Do some distributional patterns “mask” cultural consensus when other patterns “reveal” it?
• If NO … nothing to worry about [ yippee! ]• If YES distributional patterning has an independent
effect that needs to be taken into account when interpreting results of consensus analyses
![Page 10: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/10.jpg)
ITEM FORMAT:• Counter-balanced Likert-style questions, i.e., 6-point
“strongly agree” to “strongly disagree” response scale … because these are so common in survey research
ANALYSES:• Such data can be analyzed two ways:
– INFORMAL MODEL of consensus analysis … i.e., input to factor analysis is a Resp x Resp correlation matrix (data treated as interval-scale)
– FORMAL MODEL of consensus analysis … i.e., input to factor analysis is a chance-corrected agreement matrix (data treated as nominal-scale, e.g., dichotomized responses)
![Page 11: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/11.jpg)
Research Design … with average knowledge and distributional pattern as manipulated variables
Distributional Patterns
Key Indicators
[ variety of other
measures ]
Ratio of eigenvalues
Mean 1st factor
loading
Number of negative loadings
Uniform-to-Random ? ? ? ?
Expertise ? ? ? ?
Subcultures ? ? ? ?
Perspectival ? ? ? ?
![Page 12: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/12.jpg)
“Theoretical” Predictions
Distributional Pattern Prediction
Uniform-to-Random None … serves as benchmark for other patterns
Expertise Gradient INCREASE consensus indicators
Subcultures DECREASE consensus indicators
Perspectival ???
![Page 13: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/13.jpg)
Implementation
• How to “experimentally manipulate” key parameters for different distributional models while holding others constant ??
… computer simulation to the rescue !
See: Excel “data-generating” file & Excel “findings” file
![Page 14: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/14.jpg)
SOME INITIAL FINDINGS
![Page 15: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/15.jpg)
![Page 16: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/16.jpg)
![Page 17: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/17.jpg)
![Page 18: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/18.jpg)
![Page 19: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/19.jpg)
![Page 20: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/20.jpg)
![Page 21: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/21.jpg)
![Page 22: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/22.jpg)
![Page 23: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/23.jpg)
![Page 24: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/24.jpg)
![Page 25: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/25.jpg)
![Page 26: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/26.jpg)
Key Findings
1. Distributional pattern has an independent effect with respect to consensus indicators– w/r/to RATIO OF EIGENVALUES
( compared to the Uniform-to-Random model )
• Expertise patterns INCREASE the ratio • Subcultural patterns DECREASE the ratio • Perspectival patterns DECREASE the ratio
– w/r/to MEAN 1st FACTOR LOADING• Distributional patterns have little effect on this indicator,
AND consensus analysis estimates actual competence very well … with one exception:
• Expertise (triangular) pattern INFLATES mean competence as well as the ratio of eigenvalues … (because it violates the “homogeneity of items” assumption?)
![Page 27: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/27.jpg)
2. Expertise (rectangular) pattern– The range of expertise about the same average competence
also makes a difference:greater range larger ratio of eigenvalues
3. Subcultural patterns– As expected, systematic differences in sub-group knowledge
undermine consensus:• “By question” sub-groups may still show consensus overall,
with the groups showing up on the 2nd factor• Different “answer keys” just destroy consensus
4. “Formal consensus model” (on dichotomized data) and “informal consensus model” yield very similar results
![Page 28: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/28.jpg)
CONCLUSIONS
![Page 29: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/29.jpg)
“Lessons” for Researchers
1. Since the ratio of eigenvalues is particularly sensitive to the distributional pattern of knowledge, REPORT MORE than just the ratio
– Minimally, include:• Ratio of 1st to 2nd eigenvalues• Mean 1st factor loading (and st.dev. of those loadings)• Number of negative loadings
– And, comparable “guidelines” should be established for evaluating these additional measures: e.g., 0.500 for mean loading; fewer than ~5% negative loadings in sample
– These output statistics are necessary for more meaningful interpretations of one’s data
![Page 30: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/30.jpg)
2. IF your data show a hefty mean 1st factor loading but a low ratio of eigenvalues… DO NOT leap to the conclusion that either (a) subcultures exist or (b) there is free variation in the domain
– You may be dealing with a case of PERSPECTIVAL DIVERSITY … which would warrant further investigation, such as examining the inter-person correlation matrix and the response-profiles of individuals one at a time to see if you can detect a subtle social patterning to who-knows-what
![Page 31: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/31.jpg)
3. Try to formulate questions that are “EQUALLY DIFFICULT” ( and ask lots of questions )
– Violations of Assumption 3 will inflate both the obtained ratio of eigenvalues & the mean 1st factor loading
• e.g., Expertise (triangular) pattern INFLATES both indicators
– So … ex post facto…if you notice that some questions were “much easier” than others, then either: (a) use higher threshold criteria before claiming the data conform to the cultural consensus model, and/or (b) remove the very easy questions and re-analyze
![Page 32: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/32.jpg)
Future Directions
• Developing additional ‘geometries’ of perspectival overlapping
• Analyzing relations between a variety of measures describing the initial Resp x Resp correlation matrix and the key indicators from consensus analysis
• Exploring different instantiations of “guessing” (binomial, truncated-normal, beta distributions)
• Exploring other possible measures from the factor analysis as predictors of culture-sharing, e.g., 1st eigenvalue divided by sample size
![Page 33: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/33.jpg)
Thank you
… and we would be happy to continuetalking with interested folks
after the session
![Page 34: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/34.jpg)
DEIGRAT
14121086420
EIG
EN
RA
T
14
12
10
8
6
4
2
0
DMFLOAD
.8.7.6.5.4.3.2
MF
LO
AD
.8
.7
.6
.5
.4
.3
.2
Scalar data analyzed via Informal Method (vertical axis)VS
Dichotomized data analyzed via Formal Method (horizontal axis)
Ratios of eigenvalues( r = .933 )
Mean 1st factor loadings( r = .969 )
![Page 35: Perspectival Diversity and Consensus Analysis John Gatewood...... Lehigh University John Lowe....... Cultural Analysis Group AAA Meetings, Philadelphia,](https://reader036.fdocuments.in/reader036/viewer/2022062322/5697bfbe1a28abf838ca26e4/html5/thumbnails/35.jpg)
B6
strongly agree
agree
slightly agree
slightly disagree
disagree
strongly disagree
Co
un
t200
100
0
Ck=2,p=.5
6.005.004.003.002.001.00
Co
un
t
20
10
0
POTENTIAL PROBLEM…
Frequency distributions of itemsfrom “real” surveys (top panels)are more graded than our“simulated” data (lower right)
something we’re trying to resolve,but not there yet …
“Real” Survey Item
Simulated Item
B2
agree
slightly agree
slightly disagree
disagree
strongly disagree
Co
un
t
200
100
0
“Real” Survey Item