Small Sample Comparison of Modified Chi-Square Test Statistics for Survey Data

Small Sample Comparison of Modified Chi-Square Test Statisticsfor Survey Data

Anil Rai, Arun K. Srivastava and Harish C. Gupta

Indian Agricultural Statistics Research Institute New DelhiIndia

Summary

The problem of categorial data analysis in survey sampling arises because of non-independence ofsample elements of the sample obtained through imposed sampling design. In this article the perfor-mance of modified c2 statistic for testing independence of attributes have been evaluated for smallsample sizes with the help of log-linear models with respect to its achieved level of significance forfixed nominal level at 5%, through simulation technique. It hase been observed that the perfiormanceof these test statistics depends on average and coefficient of variation of eigen values of design effectmatrix. The first order corrected statistic is able to capture the effect of sampling design to a greatextent but the performance of second order corrected statistic is much better. Further, these modified c2

test statistics were applied to a real survey data and their performance were evaluated with respect totheir achievied level of significance.

Key words: Categorical data; Simulation; Small sample; Complex survey.

1. Introduction

Let there be I cells in a multidimensional contingency table, which can be num-bered lexicographically as i ¼ 1; 2; . . . ; I. Let finite population proportions bedenoted by mi along with their estimates mmi ði ¼ 1; . . . ; IÞ. A log-linear model maybe written as

log m ¼ ~mmðqÞ J þ Xq : ð1:1ÞHere, log m is ðI � 1Þ-vector of log probabilities, X is a known I � r matrix of fullrank r � ðI � 1Þ, such that X0J ¼ 0, q is an ðr � 1Þ-vector of parameters, 0 is anull vector and ~mmðqÞ is the normalizing factor which ensures that

Pmi ¼ 1. It is

possible to partition X and q of the equation (1.1) as X ¼ ðX1; X2Þ andq ¼ ðq1; q2Þ0 respectively. Here, X1 is I � r1 dimensional matrix of rank r1 and X2

is I � r2 matrix of rank r2, where r1 þ r2 ¼ r. In case of null hypothesis of testingof independence of attributes, q2 ¼ 0, the reduced form of the model (1.1) can beobtained with r1 parameters. Further, let miðqqÞ and mið

^qqqqÞ denotes “pseudo MLE”

based on a sample of size n0, under model (1.1) and its reduced form respectively.

Biometrical Journal 43 (2001) 4, 483–495

# WILEY-VCH Verlag Berlin GmbH, 13086 Berlin, 2001 0323-3847/01/0407-0483 $ 17.50þ.50/0

The sample is selected from a finite population with the help of a complexsampling design. The general Pearson c2 statistics in this case is given by

X2P ¼ n0

PIi¼1

½miðqqÞ � mið^qqqq1Þ2=mið

^qqqq1Þ ð1:2Þ

The problem of testing of independence of attributes from complex samples hasbeen examined extensively by Chapman (1966), Bhapkar and Koch (1968),Nathan (1969, 1972, 1973, 1975) etc. Their approach was essentially to devel-op test statistics based on null hypothesis such that, expected value of thesestatistics becomes zero as desired. However, the test statistics proposed by thistechnique behaves poorly with respect to its achieved level of sifgnificance.Koch and Freeman (1975) proposed Wald statistics. The statistics, which wereproposed by Fellegi (1980), were also not found to be satisfactory in terms ofits achieved level of significance. Rao and Scott (1981, 1984) proposed themethod of correcting first and second central moments of ordinary c2 test statis-tic. It has been shown that ordinary c2 statistic is asymptotically distributed asweighted c2 distribution. The weights are eigen values of the generalized designeffect matrix. Hence, the first order correction is based on average design effect,whereas, second order correction is analogous to Satterthwaite (1946) approx-imation. The performance of these competing techniques for the case of good-ness-of-fit statistic under two stage cluster sampling have been evaluated throughMonte Carlo studies by Brier (1980), Kumar and Rao (1984), Fay (1989), Tho-

mas and Rao (1984, 1985, 1987) etc. An attempt was made by Thomas et al.(1989) to study the performance of above test statistics in case of testing indepen-dence of attributes, which is most often used test in categorical data. The approachof these Monte Carlo studies are confined to model based generation of populationwith clustered structure. As pointed out by the authors that this study in case oftesting of independence of attributes are preliminary in nature and properties ofthese statistics are remains to be seen under different situations in case of surveydata.

In this article, an attempt has been made to evaluate the performance of usualc2 test statistic along with the first and second order corrected c2 statistics forsurvey data. These statistics have been chosen due to their established superiorityover their competitors. The performance of these statistics have been studiedunder the null hypothesis of independence of attributes in case of small samplesizes, when sample is selected through stratified sampling designs with differentkinds of sample size allocations in various strata. In addition, the populationgenerated for this simulation study is based on real data. Further, the log-linearmodels are used for the analysis, even for two-dimensional contingency tables, asthis provides a general procedure of analysis for multi-dimensional contingencytables. To demonstrate its practical utilities in real agricultural survey, the perfor-mance of these statistics has been evaluated on a real data taken from a fisheriessurvey.

484 A. Rai, A. K. Srivastava, H. C. Gupta: Modified Chi-Square Test

2. Modified c2 Test statistics

Rao and Scott (1984) showed that X2P is distributed asymptotically as

Pr2i¼1

diWi,

where, Wi is c21 and di is the ith eigen value of the design effect matrix DD as given

below.

DD ¼ ½ ~XX02Wð^qqqq1Þ ~XX2�1 ½ ~XX2

PP~XX2

where,

~XX2 ¼ ½I � X1fX01Wð^qqqq1Þ X1g�1 X1Wð^qqqq1Þ X2

and

Wð~qqÞ ¼ n�10 fD½mð~qqÞ � mð~qqÞ mð~qqÞ0g :

Here, ~qq equals ^qqqq1 or qq1 as appropriate.

PPis estimated variance-covariance matrix

of cell proportions based on survey design. The first order corrections to the teststatistics X2

P is given by

X2F ¼ X2

P=dd ð2:1Þwhere,

r2dd ¼ Tr ½fX0Wð^qqqq1Þ Xg�1 fX0 PP Xg � Tr ½fX01Wð^qqqq1Þ X1g�1 fX0

1

PPX1g :

Here, Tr denotes the trace of matrix. In addition, the second order correctionsusing Satherthwaite approximation is given by

X2S ¼ X2

F=½1þ aa2 ð2:2Þwhere, this statistics follows c2

n, a c2 random variable withn ¼ ðI � r � 1Þ=ð1þ aa2Þ degrees of freedom. Heren aa is the coefficient of varia-tion of the eigenvalues ddi, of the estimated design effect matrix DD i.e.

aa2 ¼PI�r�1

i¼1dd2i =ðI � r � 1Þ dd2

� �� 1

where,

ðI � r � 1Þ dd ¼ Tr ðDDÞand

PI�r�1

i¼1d2i ¼ Tr ðDD2Þ :

More details regarding c2 tests in case of data from complex survey designs aregiven in a book by Skinner et al. (1989).

Biometrical Journal 43 (2001) 4 485

3. Simulation

A multivariate (10 variables) normal population of size 5000 has been simulatedwith the help of given parameters i.e. mean vector m and variance covariance matrixV ¼ ðnijÞ. These parameters m and V , which are needed to simulate the above popu-lation, are based on certain characteristics of villages as obtained from Census

(1981) of Muktsar block of Faridkot district from Punjab (India). The variables con-sidered are, (1) total area of the village, X1, (2) total irrigated area of the village, X2,(3) total unirrigated area of the village, X3, (4) Other area of the village, X4, (5) No.of households in the village, X5, (6) population of the village, X6, (7) No. of Sche-dule cast in the village X7, (8) No. of literate in the village X8, (9) No. of main work-ers, X9 and (10) No. of cultivators in the village, X10. The above population has beenstratified in to five equal strata based on X6. Within each stratum three categories ofeach X3, X4, X7, X8 and four categories of each X9 and X10 are made in such a waythat each cell of the two dimensional contingency tables formed with the help of thepairs of these variables contains almost same number of units. This was done mainlyto avoid the sparse nature of the contingency tables and to ensure the independenceof categorical variables in population.

Samples of different sizes namely 50, 75, and 100 are selected with the help of thesampling designs given in Table 1. The sampling units are selected independentlyfrom each stratum. In case of sample surveys, the sampling design used for selectionof samples can be classified with respect to characteristics of ultimate sample unit intothree broad categories (i) Self weigthing, (ii) Varying size and (iii) Balanced samplingdesigns. In this study, all these categories of sampling designs are considered with thehelp of stratification of the population in a particular way. In most of the simulationstudies in the literature, the effect of sampling design on the parameter of interest hasbeen studied following this approach, as the selecting sample with the help of allpossible sampling designs in case of survey sampling is not possible. The samplingdesign varies with respect to allocations in different strata as given in Table 1.


Table 1

Allocation of sample size in to different strata for various sampling designs

Design n ¼ 50 n ¼ 75 n ¼ 100Stratum

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

D1 10 10 10 10 10 15 15 15 15 15 20 20 20 20 20D2 6 8 10 12 14 9 12 15 18 21 12 16 20 24 28D3 2 8 10 12 18 3 12 15 18 27 4 16 20 24 36D4 2 3 5 15 25 3 4 8 22 38 4 6 10 30 50D5 1 1 2 8 38 2 2 3 12 56 2 2 4 16 76D6 15 7 5 8 15 22 11 8 12 22 30 14 10 16 30D7 23 2 1 2 22 34 3 2 3 33 46 4 2 4 44D8 24 1 1 1 23 35 2 2 2 34 48 2 2 2 46

Clearly, D1 is self-weighting design i.e. stratified sampling with propotional alloca-tion, D2, D3, D4, and D5 are increasing allocations parallel to varying probabilitysampling, whereas D6, D7, and D8 are U-shaped allocation equivalent to balancedtype of sampling design. The total numbers of 200 samples are selected for eachdesign, and for each sample size.

Let the column vector of dimension I ¼ R� C of observed cell counts, i.e.Z; r ¼ 1; 2; . . . ;R; c ¼ 1; 2; . . . ; C is created by listing these values in lexico-graphic order as

Z ¼ ðZ 0þ1; Z

0þ2; . . . ; Z

0þcÞ

where, Zþc is the cth column of the table. Let VV be I � I estimated covariancematrix of Z. Again, let PP be the I-dimensional column vector of estimated cellproportions under the null hypothesis of independence of attributes. Then, theestimated variance-covariance matrix of PP is given byPP

¼ Z�1ðþþÞ½I � ðPPð*Þ J 0Þ VV ½I � ðPPð*Þ J 0Þ0

where I, is a unit matrix. The different statistics are calulated for each selected sam-ple from the simulated population with the help of the formulate given in section 2.

3:1 Estimation of Level of Significance

The level of significance with respect to fixed nominal level 5% for various statis-tics have been calculated with the help of algorithm given by Shiel and Muirch-

eartaigh (1977). This algorithm is mainly given for the distribution of non-nega-tive quadratic forms in normal variables. Let n-dimensional vector Z has amultivariate normal distribution with expected value m and variance covariancematrix

P. This algorithm computes the distribution function of the quadratic form

such as ðzþ aÞ0 Cðzþ aÞ for a fixed vector a and symmetric positive definite orpositive semi-definite matrix C. By making the linear transformation

Z � m ¼ L0RX ; aþ m ¼ L0Rb

It has been shown that

Pr fðZ þ aÞ0 CðZ þ aÞ � tg ¼ Pr fðx þ bÞ0 Aðx þ bÞ � tgwhere, the components of the vector X are independent standard normal variablesand L is the upper triangular matrix defined by

P¼ L0L, A is a matrix whose

columns are eigen vectors of LCL0, and A ¼ dig ðaiÞ is a diagonal matrix whoseelements ai are the eigen values of LCL0. Again, it has been proved that

Pr fðZ þ aÞ0 CðZ þ aÞ � tg ¼P1k¼1

CkFðn0 þ 2K; t=bÞ ; if t > 0

¼ 0 ; if t < 0


where, Fðn0; yÞ denotes distribution function of c2n, and its corresponding density

function is denoted by f ðn0; yÞ, b is a distribution free constant and n0 is rank ofC. In order to fix the nominal level of significance (SL) for different modified c2

test statistics following equation is used.

SL ðX2Þ ¼ PrðX2 � X2nðaÞ

where, X2nðaÞ is the upper a –– percentage point of a c2 random variable with n-

degrees of freedom. The following statistics are calculated for each selected sam-ple.

(i) Average design effect or eigen values of design effect matrix dd.(ii) Average sum of squares of eigen values of design effect matrix,

PIi¼1

dd2i .

(iii) Co-efficient of variation of eigen values of design effect matrix, aa2.(iv) Degrees of freedom for Satherthwaite c2 test statistics, u.(v) Pearson’s c2 statistic, X2

P.(vi) First order corrected Pearson c2 statistics, X2

F .(vii) Second order corrected Pearson c2 statistics, X2

S .(viii) Actual level of significance of X2

P at 5% nominal level of significance,PðX2

PÞ.(ix) Actual level of significance of X2

F at 5% nominal level of significance,PðX2

FÞ.(x) Actual level of significance of X2

S at 5% nominal level of significance,PðX2

SÞ.Final tables of this simulation study are prepared after taking average of these

statistics over all sample of given size, selected by each of the designs under con-siderations.

4. Results and Discussion

The number (%) of samples for which different statistics i.e. X2P, X

2F , and X2

S arefound to be significant at 5% level of significance for each sampling design anddifferent sample sizes in case of two dimensional contingency table for X3 and X4

simulated variables are presented in Table 2.The total number of sample selected are m ¼ 200 for different sample sizes

corresponding to each sampling designs under consideration. Since, the inherentassumptions of c2 tests are, independent and identical nature of sample observa-tions, the c2 statistics are likely to be non-significant under the null hypothesis ofindependence of attributes. The higher number of samples having significant val-ues X2

P, X2F , and X2

S depicts the extent of effect of sampling design towards respec-tive c2 statistics. This table clearly shows that in case of selfweighting design i.e.D1, none of the statistics are found to be significant. As described earlier, D2, D3,D4 and D5 are non self weighting designs with increasing allocations and D6, D7,


D8 are having increasing U-shaped allocation of sample sizes to the five strata.Thus, skewness of sample allocations increases steadily from D2 to D5 and alsofrom D6 to D8 with smaller amount of skewness in D2 and D6. From the tablefollowing observations are evident.

i) Pearson’s c2 test provides misleading results for non-self weighting designsand extent of drawing misleading inference is more frequent as one movesaway from self-weighting designs.

ii) The first order corrections i.e. X2F provides definite improvement in eliminat-

ing the effect of sampling designs.iii) The X2

S , which is second order correction based on Satherthwaite approxi-mation further improves the situation and in the present analysis, the extentof misinterpretation was within 5% limits.

These observations are in agreement with the earlier findings of Thomas et al.(1989), about the performance of testing independence of attributes. Thus, theseresults strengthen the earlier findings for more versatile situations as the method ofanalysis and complex survey design in this study is different from earlier one.

The Table 3 shows the average of estimated statistics over 200 selected sam-ples each of size 50 through different sampling designs for two-dimensional con-tingency tables formed with help of X3 and X4. Following inferences can be drawnfrom this table:

i) The average of the calculated values of usual c2 statistic i.e. X2P is high as

compared to first and second order corrected statistics for non-self weight-ing designs. This gap increases as we move away from self-weighting de-signs.

ii) In case of self-weighting designs i.e. D1, the nominal level and actual levelof significance almost coincides for all the statistics under consideration.


Table 2

Number of samples for which X2P, X

2F and X2

S are significant(X3 � X4 table, 5% level of significance, m ¼ 100)

Design n ¼ 50 n ¼ 75 n ¼ 100

X2P X2

F X2S X2

P X2F X2

S X2P X2

F X2S

1 2 3 4 5 6 7 8 9 10

D1 –– –– –– –– –– –– –– –– ––D2 6 6 4 5 5 2 4 4 3D3 21 6 1 14 3 2 10 2 2D4 30 7 1 23 3 1 31 1 ––D5 52 11 2 46 7 2 63 4 1D6 8 5 2 10 4 2 4 2 2D7 72 15 5 73 13 5 75 10 3D8 79 12 4 76 10 4 77 10 5

But as the design effect increases, the actual size of the critical region of X2P

inflated at much higher rate and in case of some designs i.e. D4, D5, D7and D8, the estimated size of actual critical region for fixed nominal size at5% becomes one. Since, average of eigen values and coresponding coeffi-cient of variations of eigen values are much more higher in these cases, sothere is strong possibility of failure of the approximation used for calcula-tion of size of the critical region for small sample size.

iii) The first order corrected statistic i.e. X2F is able to reduce the effect of sam-

pling design on X2P to some extent, but this also fails in case of higher coeffi-

cient variation of eigen values of design effect matrix. Similar, findings werereported by Thomas and Rao (1987) in their simulation study for testinggoodness-of-fit under Direchilect distribution in case of small sample sizes.

iv) In case of second order corrected statistic, the size of the actual criticalregion is almost near the fixed nominal size at 5% except in case of highlyU-shaped design, where it is slightly more, this is again because of highervalues of coefficient of variation in these designs.

Table 4 and Table 5 presents average of estimated statistics from each sampleselected with the help of different sampling designs for two-dimensional contin-gency tables of X3 � X4 for samples sizes 75 and 100 respectively. The trendsfollowed by these tables are similar to Table 3. It can be seen that coefficient ofvariations of eigen value decreases by increasing the sample size even though de-sign effect is increasing. Due to this fact there is slight improvement in the actualsize of the critical region in case of some of the designs for different c2 statisticsunder consideration as samples size increases. The results presented in Table 3,Table 4 and Table 5 clearly support the results presented in Table 2. The similarpattern was found for other six pairs of categorical variables for two-dimensionalcontingency tables. So, the results of those pairs are not presented in this article.


Table 3

Average of estimated statistics from each selected sample of size 50 under each samplingdesign for two-dimensional contingency table X3 � X4.(Total number of samples for each design ¼ 200)

Design �dd�dd aa2 l=IJP

dd2i u X2

P X2F X2

S PðX2PÞ PðX2

FÞ PðX2SÞ

1 2 3 4 5 6 7 8 9 10 11

D1 1.00 0.04 1.24 3 3.84 3.91 3.71 0.06 0.05 0.05D2 1.09 0.04 1.24 3 3.46 3.22 3.07 0.16 0.10 0.05D3 1.34 0.20 2.22 3 6.84 4.86 4.04 0.76 0.40 0.06D4 1.82 0.24 4.18 3 8.81 4.96 3.87 1.00 0.81 0.07D5 1.98 0.60 6.72 2 10.78 5.18 3.14 1.00 1.00 0.09D6 1.13 0.06 1.34 3 3.57 3.20 3.01 0.24 0.14 0.06D7 2.17 0.59 7.70 3 19.11 9.36 5.28 1.00 1.00 0.11D8 2.18 0.80 9.03 2 15.18 6.89 3.92 1.00 1.00 0.11

The findings of this study indicates that, due to the effect of sampling designs,the actual size of the critical region inflated for X2

P statistic under the null hypothesisof testing independence of attributes in case of small sample sizes, selectedthrough various non-self weighting designs. As a consequence of this, the infer-ences drawn with the help usual c2 statistic are highly misleading. This is with theagreement of the result obtained by Fellegi (1980), Thomas and Rao (1987),

Thomas et al. (1989). Also, the first order corrected statistics X2F improves the

situation to a greater extent but as the complexity of the design increases it fails tocapture and eliminate the effects of sampling designs. The second order correctedstatistics is able to control the size of the critical region around the fixed nominallevel of significance. But as we move away from self weighting designs to non-self weighting designs, the coefficient of variation of eigen values increases and


Table 4

Average of estimated statistics from each selected sample of size 75 using each samplingdesign for two-dimensional contingency table X3 � X4.(Total number of samples for each design ¼ 200)


dd2i u X2

P X2F X2

S PðX2PÞ PðX2

FÞ PðX2SÞ

1 2 3 4 5 6 7 8 9 10 11

D1 1.00 0.02 1.02 3 3.65 3.66 3.60 0.05 0.05 0.05D2 1.06 0.04 1.14 3 3.73 3.61 3.42 0.12 0.09 0.05D3 1.42 0.19 2.45 3 7.52 5.46 4.64 0.93 0.48 0.07D4 1.97 0.15 4.50 3 5.56 3.01 2.51 1.00 0.78 0.06D5 3.17 0.34 14.12 2 13.73 4.43 3.30 1.00 1.00 0.08D6 1.14 0.05 1.38 3 6.78 5.91 5.61 0.24 0.13 0.05D7 2.75 0.32 10.11 3 12.18 4.69 3.45 1.00 1.00 0.09D8 2.90 0.50 12.68 2 15.72 5.76 3.76 1.00 1.00 0.09

Table 5

Average of estimated statistics from each selected sample of size 100 under each samplingdesign for two-dimensional contingency table X3 � X4.(Total number of samples for each design ¼ 200)


dd2i u X2

P X2F X2

S PðX2PÞ PðX2

FÞ PðX2SÞ

1 2 3 4 5 6 7 8 9 10 11

D1 1.00 0.02 1.01 3 3.97 3.96 3.90 0.05 0.05 0.05D2 1.08 0.02 1.20 3 3.49 3.22 3.16 0.06 0.08 0.05D3 1.50 0.01 2.64 3 10.62 7.27 6.26 0.96 0.49 0.06D4 2.08 0.11 4.82 3 13.59 4.16 3.71 1.00 0.89 0.06D5 3.88 0.33 20.33 2 19.93 4.22 3.93 1.00 1.00 0.07D6 1.16 0.04 1.41 3 5.57 4.78 4.59 0.26 0.14 0.05D7 3.20 0.27 13.07 3 15.22 5.27 3.98 1.00 1.00 0.07D8 3.96 0.54 24.70 2 25.65 6.79 5.02 1.00 1.00 0.08

the actual size of the critical region becomes almost double than fixed nominallevel of significance at 5% in case of small sample size. This reveals the fact thatsampling design and related sample size plays a vital role in the applications ofthese modified statistics. Hence, there is a need to take care of sampling designwhile determining the sample size, whenever these tests are to be applied in caseof complex survey data. It is expected that, as the sample size becomes consider-ably large X2

S will be able to control the level of significance.

5. Application to Real Data

The data for this empirical investigation is taken from a sample survey for estima-tion of inland fisheries resources and catch conducted in Cuttack, Bolangir andSambhalpur districts (administrative units) of Orissa State (India). The designadopted in this survey was stratified multistage random sampling with districts asstrata, blocks (subadministrative unit) within each district as Primary SamplingUnits (PSU’s) and ponds and tanks within each block as Secondary SamplingUnits (SSU’s). The PSU’s were selected by probability proportional to size withreplacement (PPSWR) sampling, using number of water units in each block assizes. Therefore, within each block total numbers of PSU’s were 14. The SSU’s ineach of the PSU’s were completely enumerated for the estimation of area. Thetotal numbers of SSU’s were 6154. The SSU’s were observed and classified ac-cording to following attributes namely, sesonality (S), source of water supply (W),soil type (T), depth during the monsoon (D), extent of silting (H) and utilisationfor fishing or other purposes (U), so as to form five two dimensional contingencytables. Seasonality is classified as perennial and seasonal. The source of watersupply as rained, river, canal, sewage, coastal water and others; Depth during themonsoon as 0–2 meters and above 2 meters; Soil type as sandy, laomy andothers; Extent of silting as partially silted and badly silted; Utilisation as fishcultivation, not fish cultivation, irrigation, multiple purpose and others. Now two-dimensional contingency tables, which provide meaningful association among thevarious attributes, were formed. Only five relevant two-dimensional contingencytables from the above attributes have been considered for the purpose of proposedinvestigation.

Table 6 shows that the calculated values of ordinary c2 statistics are very highand are closely followed by the Likelihood Ratio test statistic (G2) for all twodi-mensional contingency tables considered for this study. The basic reason, whichcan be attributed for these high values, is non-independence of sample observa-tions because of imposed sampling designs. Other two statistics presented in Table 6are modified test statistics and therefore, gives reasonably low values as comparedto the above statistics.

It was found that null hypothesis of independence of attribute is rejected at 5%level of significance for all contigency tables when ordinary c2 and Likelihood


ratio test statistics are applied. In case of modified statistics situation, the nullhypothesis of no association between the variables of contingency tables under theconsideration is not rejected at the same level of significance except for contin-gency table of seasonality vs depth of water units at the time of the monsoon. Thereason for this exceptional behaviour will be discussed later. This clearly reflectsthe extent of misleading inferences that can be drawn in applying c2 or Likelihoodratio test statistics in case of survey data. Further Table 6 shows that according tothe Satterthwaite c2 statistics, the null hypothesis of no association is rejected incase of contingency table of seasonality of ponds and depth of ponds during mon-soons. This clearly means that there is strong association between these two vari-ables.

Table 7 shows the actual size of critical region of various test statistics consid-ered in Table 6 i.e. their actual rejection rates, when null hypothesis is true. Theinflation produced in the size of critical region of ordinary c2 test statistics is ashigh as 97.8%, when evaluated at 5% of nominal level. The actual size of thecritical region for modified c2 test statistic namely X2

F due to Rao (1981) wasfound to be conservative at the nominal level of 5%. The exceptional behavior ofthe contingency table in respect of seasonality vs depth during the monsoon maybe attributed to less degree of freedom associated with c2 variable due to whichapproximation used may not be very accurate. Comparing the second order cor-rected c2 statistic i.e. Satterthwaite approximation with other modified c2 proce-dure, it was found that Satterthwaite c2 statistic i.e. X2

S has actual size of critical


Table 7

Percentage level of significance of various test statistics

Classifying variableX � Y

k X2P G2 X2

F X2S

S� D 1 1 77.34 77.85 18.80 38.38S�W 4 2 97.81 98.58 4.73 4.59W � D 4 3 77.25 79.16 2.22 4,76W � H 4 3 26.86 26.09 0.80 4.76U � T 12 9 67.64 66.89 4.14 5.91

Table 6

Calculated values of various test statistics

Classifying variableX � Y

k X2P G2 X2

F X2S

S� D 1 1 765.99 789.99 24.71 137.65S�W 4 2 789.98 100.54 3.72 2.50W � D 4 3 64.65 67.64 2.47 4.72W � H 4 3 15.49 16.61 0.74 4.00U � T 12 9 264.74 261.74 6.46 10.94

region very close to nominal critical region, except for the seasonality vs depthduring the monsoon contingency table.

Hence, from the result obtained in this empirical investigation, it can be con-cluded that Satterthwaite correction to the ordinary c2 test statistic performs better,wheres the first order modification is conservative with respect to the nominalcritical region. The emprical investigation clearly resembles with the result of thissimulation study. Hence it can be concluded that Satterthwaite approximated c2

statistics should be applied for getting the reliable inference for small sample sizewith sufficient degree of freedom of the contingency table, whenever data is col-lected with the help of imposed complex sampling designs.

References

Bhapkar, V. P. and Koch, C. G., 1968: Hypothesis of No Interaction in Multidimensional ContingencyTables. Technometrics 10, 107–123.

Brier, S. E., 1980: Analysis of Contingency Tables Under Cluster Sampling. Biometrika 67, 591–596.Chapman, D. W., 1966: An Approximation Test of Independence based on Replication of Complex

Survey Design. Unpublished Master’s thesis, Cornell University.District Handbook Census, 1981: R. G. Office, R. K. Puram, New Delhi.Fay, R. E., 1989: Additional Evaluation of Chi-Square Methods for Complex Samples. Proc. Am.

Statist. Assoc. Sur. Meth. Sec, 680–685.Fellegi, I. P., 1980: Approximate Tests of Indepencence and Goodness of Fit based on Stratified Multi-

stage Samples. J. Am. Statist. Assoc. 75, 261–268.Koch, C. G. and Freeman, J. L., 1975: Strategies in Multivariate Analysis of Data from Complex

Surveys. Int. Statist. Rev. 43, 59–78.Kumar, S. and Rao, J. N. K., 1984: Logistic Regression Analysis of Labour Force Survey Data. Sur.

Meth. 10, 62–81.Nathan, G., 1969: Test of Independence in Contingency Tables from Stratified Samples. In: N. L.

Johnson and H. Smith (eds.): New Developments in Survey Sampling. Wiley, New York, 578–600.Nathan, G., 1972: On the Asymptotic Power of Tests for Independence in Contingency Tables from

Stratified Samples. J. Am. Statist. Assoc. 67, 917–920.Nathan, G., 1973: Approximate Tests of Independence in Contingency Tables from Stratified Samples.

N. C. H. S., Vital and Health Statistics. Series-2, No. 53 Washington D.C.Nathan, G., 1975: Test of Independence in Contingency Tables from Satisfied Proportional Samples.

Sankhya 37, (C) Part I, 77–87.Rao, J. N. K. and Scott, A. J., 1981: The Analysis of Categorical Data from Complex Sample Sur-

veys. Chi-Square Tests for Goodness-of-fit and Independence of Two-way Tables. J. Am. Statist.Assoc. 76, 221–230.

Rao, J. N. K. and Scott, A. J., 1984: On Chi-Square Tests for Multiway Contingency Tables withProportions Estimated from Survey Data. Ann. Statist. 12, 46–60.

Satterthwaite, F. F., 1946: An Approximate Distribution of Estimates of Variance Components. Bio-metrics 2, 110–114.

Sheil, J. and O. Muircheartaigh, I., 1977: Algorithm, AS 106. The Distribution of Non-negativeQuadratic Forms in Normal Variables. Appl. Statist. 25, 92–99.

Skinner, C. J., Holt, D., and Smith, T. M. F., 1989: Analysis of Complex Surveys. John Wiley &Sons, New York.

Thomas, D. R. and Rao, J. N. K., 1984: A Monte Carlo Study of Exect Levels of Chi-squared Good-ness-of-fit Statistics Under Cluster Sampling. Technical report 66, Carlton Univ. Ottawa.


Thomas, D. R. and Rao, J. N. K., 1985: On the Power of Some Goodness-of-fit Tests Under ClusterSampling. Technical Report. 66, Carlton Univ. Ottawa.

Thomas, D. R. and Rao, J. N. K., 1987: Small Sample Comparisons of Level and Power for SimpleGoodness-of-fit Statistics Under Cluster Sampling. J. Am. Statist. Assoc. 82, 630–636.

Thomas, D. R., Singh, A. C., and Roberts, G., 1989: Size and Power of Independence Tests forR� C Tables from Complex Surveys. Proc. Am. Statist., 763–768

Dr. Anil Rai Received, January 1997Indian Agricultural Statistics Reseearch Institute Revised, September 1998Library Avenue Revised, September 2000New Delhi –– 110012 Accepted, November 2000India


Small Sample Comparison of Modified Chi-Square Test Statistics for Survey Data

Documents

Transcript of Small Sample Comparison of Modified Chi-Square Test Statistics for Survey Data