DIFFERENCES IN READING STRATEGIES AND

DIFFERENTIAL ITEM FUNCTIONING ON PCAP

2007 READING ASSESSMENT

by

Tanya Scerbina

A thesis submitted in conformity with the requirements

for the degree of Master of Arts

Department of Human Development and Applied

Psychology

Ontario Institute for Studies in Education

University of Toronto

© Copyright by Tanya Scerbina 2012

ii

DIFFERENCES IN READING STRATEGIES AND

DIFFERENTIAL ITEM FUNCTIONING ON PCAP

2007 READING ASSESSMENT

Master of Arts 2012

Tanya Scerbina

Department of Human Development and Applied

Psychology

University of Toronto

Abstract

Pan-Canadian Assessment Program (PCAP) 2007 reading ability item data and contextual data

on reading strategies were analyzed to investigate the relationship between self-reported reading

strategies and item difficulty. Students who reported using higher- or lower-order strategies

were identified through a factor analysis. The purpose of this study was to investigate whether

students with the same underlying reading ability but who reported using different reading

strategies found the items differentially difficult. Differential item functioning (DIF) analyses

identified the items on which students who tended to use higher-order reading strategies

excelled, which were selected response items, but students who preferred using lower-order

strategies found these items more difficult. The opposite pattern was found for constructed

response items. The results of the study suggest that DIF analyses can be used to investigate

which reading strategies are related to item difficulty when controlling for students‟ level of

ability.

iii

ACKNOWLEDGMENTS

My deepest gratitude goes to my supervisor, Dr. Ruth Childs, for her expertise,

guidance and support throughout my Master‟s program, as well as for her insight and feedback

in the process of completing this thesis. I would also like to thank Monique Herbert, the second

member of my supervisory committee, for being an invaluable mentor to me in my program of

study.

The completion of this thesis would not have been possible without the statistical

expertise of Olesya Falenchuk and professional input of Pierre Brochu; I thank both for their

patience and ongoing assistance. I would also like to express my gratitude to the employees of

the Canadian Ministers of Education, Canada, specifically Kathryn O‟Grady and Pierre Brochu

for providing me with indispensable resources and services.

Finally, I would like to thank all of the members of Datahost and my classmates;

especially Amanda, Christie, Jayme and Marija for giving excellent advice. Special thank-you

goes to my family and friends for their immense support and encouragement.

iv

TABLE OF CONTENTS

Abstract ..................................................................................................................................... ii

Acknowledgments ................................................................................................................... iii

Table of Contents ..................................................................................................................... iv

List of Tables ........................................................................................................................... vi

List of Figures ........................................................................................................................ viii

1 Introduction............................................................................................................................ 1

1.1 Differential Item Functioning .......................................................................................... 1

1.2 Reading Process and Strategies ....................................................................................... 1

1.3 Objectives ........................................................................................................................ 6

2 Method ................................................................................................................................... 8

2.1 Data .................................................................................................................................. 8

2.2 Grouping Variable ........................................................................................................... 9

2.3 Analyses ......................................................................................................................... 15

3 Results and Discussion ........................................................................................................ 17

3.1 Score Distributions ........................................................................................................ 17

3.2 Classical Item Analysis.................................................................................................. 19

3.3 Reading Strategies and Test Scores ............................................................................... 32

3.4 DIF and DSF Analyses .................................................................................................. 33

3.4.1 Dichotomous Items ............................................................................................... 33

3.4.2 Polytomous Items and DSF .................................................................................. 36

3.4.3 DIF with Scaled Matching Score, Dichotomous Items ........................................ 37

3.4.4 DIF with Scaled Matching Score, Polytomous Items and DSF ........................... 40

v

4 Implications and Conclusion ............................................................................................... 46

4.1 Limitations and Future Directions ................................................................................. 48

References................................................................................................................................ 50

Appendix A. Factor Analyses for Booklet 1 and Booklet 2 .................................................... 54

Appendix B. Means, Standard Deviations and Frequencies of Students‟ Questionnaire

Responses for Grouping Variable Sample, Booklet 1 and Booklet 2...................................... 56

Appendix C. The Relationship between Total and Scaled Scores for Booklet 1 and Booklet 2

................................................................................................................................................. 58

Appendix D. Missing Item Data for Booklet 1 and Booklet 2 ................................................ 60

Appendix E. Grouping Variable Sample: Item Statistics for Booklet 1 and Booklet 2 .......... 63

Appendix F. Anchor Items Eliminated: Item Discrimination for Booklet 1 and Booklet 2 .... 69

Appendix G. Item Analysis be Section for Booklet 1 and Booklet 2 ...................................... 72

Appendix H. Chi-Square Analyses for Reading Strategies by Item, Booklet 1 and Booklet 2

................................................................................................................................................. 75

Appendix I. DIF with Total Matching Score for Booklet 1 and Booklet 2 ............................. 78

vi

LIST OF TABLES

Table 1 Student Questionnaire: Assessment of Reading Strategies .......................................... 9

Table 2 Means, Standard Deviations and Frequencies of Students’ Questionnaire Responses,

Booklet 1 .................................................................................................................................. 10

Table 3 Means, Standard Deviations and Frequencies of Students’ Questionnaire Responses,

Booklet 2 .................................................................................................................................. 11

Table 4 3-Factor Model: Reading Strategies .......................................................................... 13

Table 5 2-Factor Model ........................................................................................................... 14

Table 6 Item Statistics, Booklet 1 ............................................................................................ 20

Table 7 Item Statistics, Booklet 2 ............................................................................................ 21

Table 8 Distractor Analysis, Booklet 1 .................................................................................... 24

Table 9 Distractor Analysis, Booklet 2 .................................................................................... 25

Table 10 Item Discrimination by Subscores, Booklet 1 ........................................................... 27

Table 11 Item Discrimination by Subscores, Booklet 2 ....................................