Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement
description
Transcript of Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement
![Page 1: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/1.jpg)
Moving Away From Aggregate Statistics: Observation Oriented Modeling and
Assessement
Lisa D. Cota, MSStatistical Analyst
Office of University Assessment & TestingOklahoma State University
![Page 2: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/2.jpg)
Why not NHST?• Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
Retrieved from http://psycnet.apa.org/journals/amp/49/12/997/• Haller, H., & Krauss, S. (2002). Misinterpretations of significance: A problem students
share with their teachers. Methods of Psychological Research, 7(1), 1–20. Retrieved from http://www2.uni-jena.de/svw/metheval/lehre/0405-ws/evaluationuebung/haller.pdf
• Hoekstra, R., Morey, R. D., Rouder, J. N., & Wagenmakers, E.J. (2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review. doi:10.3758/s13423-013-0572-3
• Lamiell, J. T. (2013). Statisticism in personality psychologists’ use of trait constructs: What is it? How was it contracted? Is there a cure? New Ideas in Psychology, 31(1), 65–71. doi:10.1016/j.newideapsych.2011.02.009
• Lykken, D. What’s wrong with psychology anyway? (1991). In Thinking Clearly About Psychology. University of Minnesota Press.
• Toomela, A. (2010). Quantitative methods in psychology: Inevitable and useless. Frontiers in Psychology, 1(July), 29. doi:10.3389/fpsyg.2010.00029
![Page 3: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/3.jpg)
Why not NHST?
• Assessment should be understandable– NHST: Not easily understood…in fact easily
misunderstood• Oakes, 1986; Gigerenzer, 2004• Hoekstra, Morey, Rouder, & Wagonmaker, 2014
• One problem (of many) with NHST:– Variance Explained ≠ Accuracy
![Page 4: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/4.jpg)
y = 0.5x + 3, R2 = 0.663
![Page 5: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/5.jpg)
y = 0.5x + 3, R2 = 0.663
![Page 6: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/6.jpg)
y = 0.5x + 3, R2 = 0.663
![Page 7: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/7.jpg)
y = 0.5x + 3, R2 = 0.663
![Page 8: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/8.jpg)
Assumptions
• Assumptions for independent samples t-test under NHST:– Null hypothesis is true– Observations are independent between and
within groups– Population variances are exactly equal– Random sampling / random assignment (???)* Continuous DV* DV has a normal population distribution
![Page 9: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/9.jpg)
Observation Oriented Modeling
• No meaningless aggregates• Person-centered • No assumption-laden p-values
![Page 10: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/10.jpg)
Observation Oriented Modeling: Ordinal Pattern Analysis
![Page 11: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/11.jpg)
Results
![Page 12: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/12.jpg)
![Page 13: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/13.jpg)
![Page 14: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/14.jpg)
![Page 15: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/15.jpg)
![Page 16: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/16.jpg)
![Page 17: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/17.jpg)
![Page 18: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/18.jpg)
Randomization Test
![Page 19: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/19.jpg)
Results, with chance-value
![Page 20: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/20.jpg)
VALUE Rubric Data
![Page 21: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/21.jpg)
Results
![Page 22: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/22.jpg)
![Page 23: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/23.jpg)
![Page 24: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/24.jpg)
Significant ≠ Meaningful
![Page 25: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/25.jpg)
Significant ≠ Meaningful
![Page 26: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/26.jpg)
![Page 27: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/27.jpg)
Significant ≠ Meaningful
![Page 28: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/28.jpg)
Transparency, Honesty, and Progress
![Page 29: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/29.jpg)
Raw Numbers
![Page 30: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/30.jpg)
Proportions
![Page 31: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/31.jpg)
Compare w. SPSS….
![Page 32: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/32.jpg)
SPSS Legacy Dialog, k Independent Samples
![Page 33: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/33.jpg)
Proportions
![Page 34: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/34.jpg)
![Page 35: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/35.jpg)
![Page 36: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/36.jpg)
Advantages of OOM
• Assumption-free• Transparent• Honest
![Page 37: Moving Away From Aggregate Statistics: Observation Oriented Modeling and Assessement](https://reader036.fdocuments.in/reader036/viewer/2022062520/568160c3550346895dcfeec1/html5/thumbnails/37.jpg)
OOM website
• “We’ve never actually looked at the data!” …before now!
• www.idiogrid.com/OOM