Significance and Meaningfulness

Significance and MeaningfulnessEffect Size & Statistical Power1

1. Effect SizeHow meaningful is the significant difference?1

KNR 445StatisticsEffect sizesSlide 3Significance vs. meaningfulnessAs sample size increases, likelihood of significant difference increases

The fact that this sample size is buried down here in the denominator of the test statistic means that as n , p 0. So if your sample is big enough, it will generate significant results12

KNR 445StatisticsEffect sizesSlide 4Significance vs. meaningfulnessAs sample size increases, likelihood of significant difference increasesSo statistical difference does not always mean important differenceWhat to do about this?Calculate a measure of the difference that is standardized to be expressed in terms of the variability in the 2 samples, but independent of sample size = EFFECT SIZE1

Effect SizeEFFECT SIZE - FORMULAKNR 445StatisticsEffect sizesSlide 5

12

KNR 445StatisticsEffect sizesSlide 6Effect SizeEFFECT SIZE from SPSSUsing appendix B data set 2, and submitting DV salary to test of difference across gender, gives the following output (squashed here to fit):

T-Test

1

KNR 445StatisticsEffect sizesSlide 7Effect SizeEFFECT SIZE from SPSS

T-Test

SDs to poolMean difference to use1

KNR 445StatisticsEffect sizesSlide 8Effect SizeEFFECT SIZE from SPSS

So

12so

KNR 445StatisticsEffect sizesSlide 9Effect SizeEFFECT SIZE from SPSS

Substituting12

KNR 445StatisticsEffect sizesSlide 10Effect SizeEFFECT SIZE from SPSS

Calculating

1

KNR 445StatisticsEffect sizesSlide 11Effect SizeFrom Cohen, 1988:d = .20 is smalld = .50 is moderated = .80 is largeSo our effect size of .25 is small, and concurs on this occasion with the insignificant resultThe finding is both insignificant and small (a pathetic, measly, piddling little difference of no consequence whatsoever trivial and beneath us)

12

2. Statistical PowerMaximizing the likelihood of significance1234

KNR 445StatisticsEffect sizesSlide 13Statistical PowerThe likelihood of getting a significant relationship when you should (i.e. when there is a relationship in reality)Recall from truth table, power = 1 - 1Truth TableReality (unknown)Null TrueNull FalseDecisionAccept NullType II error ()Reject NullType I error ()Power = 1 - (1- type II error)

KNR 445StatisticsEffect sizesSlide 14Factors Affecting Statistical PowerThe big ones:Effect size (bit obvious)Select samples such that difference between them is maximizedCombines the effects of sample SD (need to decrease) and mean difference (need to increase)Sample sizeMost commonly discussed: as n increases, SEM decreases, and test statistic then increases12

KNR 445StatisticsEffect sizesSlide 15Factors Affecting Statistical PowerThe others:Level of significanceSmaller , less powerLarger , more power1-tailed vs. 2-tailed testsWith good a priori info (i.e. research literature), selecting 1-tailed test increases powerDependent samples Correlation between samples reduces standard error, and thus increases test statistic123

KNR 445StatisticsEffect sizesSlide 16Calculating sample size a prioriSpecify effect sizeSet desired level of powerEnter values for effect size and power in appropriate table, and generate desired sample size:Applet for calculating sample size based on above:http://www.stat.uiowa.edu/~rlenth/Power/Applets for seeing power acting (and interacting) with sample size, effect size, etchttp://statman.stat.sc.edu/~west/applets/power.html

http://acad.cgu.edu/wise/power/powerapplet1.html

http://www.stat.sc.edu/%7Eogden/javahtml/power/power.html1