1

On statistical significance testingCoffee talk 19/02/2015

Nora Baka

2

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

My

Mx

Sequence 1 Sequence 2 Sequence 3

Is there a significant difference between method X and Y?

Time fram

esMeasurements are not independent!

Clustered data

Ignore

Take cluster meansSpecial tests

3

The Wilcoxon Signed Rank Test for Paired Comparisons of Clustered Data

B. Rosner, R. J. Glynn and M. T. LeeBiometrics 2006

4

Recap: Wilcoxon Signed Rank Test

• Paired data, Notion of bigger-lesser, (Data symmetric)• Independent Random samples• H0: median difference between the pairs is zero

1. Exclude all sample size 2. Rank from small to large 3. Test statistic

, where 5. If small Compare to distribution of all possible W

(random assignment of )6. If large Gaussian approximation

5

Clustered Wilcoxon Signed Rank Test, balanced

• Paired data, Notion of bigger-lesser• Clustered samples (m clusters), g correlated

samples in each cluster• H0: median difference between the pairs is zero

1. , where i is cluster, and j is sample in cluster2. Exclude all sample size 3. Rank for pooled data from small to large 4. Test statistic

6

Clustered Wilcoxon Signed Rank Test, balanced (2)

• Test statistic

• Randomization unit is the cluster distribution of

• If m is small calculate all possible

• If m is large Gaussian approximation

,where with probability ½.

7

Clustered Wilcoxon Signed Rank Test, unbalanced

• Different number of samples per cluster

• Modified statistic

• Randomization unit is the cluster distribution of

From here on the same as before.

𝑇 𝑐𝑠𝑜𝑏𝑠=∑

𝑖=1

𝑚

𝑤𝑖𝑆𝑖=∑𝑖=1

𝑚

𝑤𝑖∑𝑗=1

𝑔𝑖

𝑅𝑖𝑗𝑉 𝑖𝑗 𝑤𝑖=1

𝑉𝑎𝑟 (𝑆𝑖𝑗)1

1+(𝑔¿¿ 𝑖−1)𝜌 ¿,where

Intra-class correlation

8

Compare with no clustering taken into account

9

Results

10

What more?

• See below paper for overview of clustered data analysis[1] S. Galbraith, J. A. Daniel, and B. Vissel, “A study of clustered data and

approaches to its analysis.,” J. Neurosci., vol. 30, no. 32, pp. 10601–8, Aug.

2010.