Post on 14-Jun-2015
Highlights from PIRLS and TIMSS 2011
Jack Buckley
National Center for Education Statistics
Washington, DC
December 11, 2012
OverviewWhat are PIRLS and TIMSS and which education
systems participate in them?
How are U.S. students performing compared to their international peers in: 4th-grade reading? 4th- and 8th-grade mathematics? 4th- and 8th-grade science?
How are students performing in participating U.S. states?
2
What are…?
3
• Progress in International Reading Literacy Study (PIRLS)
• 4th-grade reading assessment
* In 1999, no grade 4 assessment.
TIMSSPIRLS
2001 2006 2011
1995 1999* 2003 2007 2011
• Trends in International Mathematics and Science Study (TIMSS)
• 4th- and 8th-grade mathematics and science assessment
• Developed by the TIMSS & PIRLS International Study Center at Boston College, under contract to the International Association for the Evaluation of Educational Achievement (IEA)
PIRLS participating education systems (in 4th grade)
4
26 education systems
201132 education systems53 education systems
5
TIMSS participating education systems (in either/both grades)
44 education systems37 education systems47 education systems57 education systems74 education systems
2011
Participating states
6
• 9 states participated as separate entities to obtain scores in PIRLS or TIMSS or both
• Each participated as part of the nation and on its own• NCES funded participation at grade 8 of all states except Florida as
part of a study to statistically link NAEP (National Assessment of Educational Progress) and TIMSS (Florida received other Education Department funding)
PIRLS TIMSS grade 4 TIMSS grade 8
Florida FloridaNorth Carolina
FloridaNorth CarolinaAlabamaCaliforniaColoradoConnecticutIndianaMassachusettsMinnesota
U.S. national sample size
4th gradeSchools: 370
Students: 12,726
4th grade Schools: 369
Students: 12,569
8th gradeSchools: 501
Students: 10,477
7
TIMSSPIRLS
U.S. state sample sizes(public schools only)
4th grade(FL)
Schools: 77
Students: 2,598
4th grade (FL, NC)
Schools: 46 – 77
Students:
1,792 – 2,661
8th grade(AL, CA, CO, CT, FL, IN, MA, MN, NC)
Schools: 53 – 82
Students:
1,712 – 2,614
8
TIMSSPIRLS
READING
PIRLS
Definition The ability to understand and use those written language forms required by society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers in school and everyday life, and for enjoyment.
Content Literary and informational texts
Cognitive dimensions Focus on and retrieve Make straightforward inferencesInterpret and integrateExamine and evaluate
10
PIRLS 2011 framework
What is on the PIRLS assessment?
11
Percentage of 4th-graders answering correctly:U.S.: 90%Int’l Avg.: 89%
What is on the PIRLS assessment?
12
Percentage of 4th-graders answering correctly:U.S.: 42%Int’l Avg.: 29%
13
U.S. average score (556) higher than the PIRLS scale average (500)
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
14
U.S. average score (556) lower than in 5 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
15
U.S. average (556) not measurably different than in 7 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
16
U.S. average score (556) higher than in 40 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
17
Average reading scores of 4th-grade students increased from 2006 to 2011 in 13
education systems, including the U.S.
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average reading score in 2011.
Change in average score
Hong Kong-CHN
Singapore
UNITED STATES
Denmark
Chinese Taipei-CHN
England-GBR
Slovenia
Poland
Norway
Georgia
Trinidad and Tobago
Iran, Islamic Rep. of
Indonesia
-20 -10 0 10 20 30 40
7*
9*
16*
8*
18*
12*
9*
6*
9*
17*
35*
36*
24*
Alberta-CAN
Sweden
Italy
Germany
Hungary
Bulgaria
Austria
Lithuania
-30 -20 -10 0 10 20 30
-12*
-8*
-10*
-7*
-12*
-15*
-9*
-9*
Change in average score
18
Average reading scores of 4th-grade students decreased from 2006 to 2011 in 8
education systems
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average reading score in 2011.
PIRLS international reading benchmarks
19
Grade 4Advanced (625) Students can interpret figurative language, distinguish and
interpret complex information from different parts of text, and integrate ideas across text to provide interpretations about characters’ feelings and behaviors.
High (550) Students can recognize some textual features, make inferences on the basis of abstract or embedded information, and integrate information to recognize main ideas and provide explanations.
Intermediate (475) Students can identify central events, make straightforward inferences from the text, and begin to make connections across parts of the text.
Low (400) Students can retrieve explicitly stated details from literary and informational texts.
Percentages of U.S. 4th-grade students reaching PIRLS reading benchmarks were higher than international medians in 2011
Low Intermediate High Advanced0
102030405060708090
100 98
86
56
17
95
80
44
8
United StatesInternational median
Benchmark
Percent
20
NOTE: All U.S. percentages are significantly higher than the corresponding PIRLS international median at the .05 level of statistical significance.
NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U.S. students reaching the Advanced benchmark are not included in figure.
21
Two systems had higher percentages of 4th-grade students reaching Advanced than the U.S.
Higher than U.S. (p < .05)
Not measurably different than U.S. (p < .05)
Singapore
Florida-USA
Russian Federation
Northern Ireland-GBR
Finland
England-GBR
Hong Kong-CHN
UNITED STATES
Ireland
Ontario-CAN
0 10 20 30 40 50 60
24
22
19
19
18
18
18
17
16
15
Percentage reaching Advanced benchmark
MATHEMATICS
TIMSS 2011 mathematics framework
23
TIMSS mathematicsContent dimensions
Grade 4 Number Geometric Shapes and Measures Data Display
Grade 8NumberAlgebraGeometryData and Chance
Cognitive dimensions
KnowingApplyingReasoning
24
GRADE 4
Percentage of 4th-graders answering correctly:U.S.: 80%Int’l Avg.: 54%
What is on the TIMSS mathematics assessment?
What is on the TIMSS mathematics assessment?
25
GRADE 4
Percentage of 4th-graders answering correctly:U.S.: 76%Int’l Avg.: 65%
26
GRADE 8
What is on the TIMSS mathematics assessment?
GRADE 8
Percentage of 8th-graders answering correctly:U.S.: 87%Int’l Avg.: 70%
27
GRADE 8
What is on the TIMSS mathematics assessment?
Percentage of 8th-graders answering correctly:U.S.: 66%Int’l Avg.: 45%
28
U.S. average score (541) higher than the TIMSS scale average (500)
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
29
U.S. average score (541) lower than in 8 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
30
U.S. average (541) not measurably different than in 6 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
31
U.S. average score (541) higher than in 42 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
32
U.S. average score (509) higher than the TIMSS scale average (500)
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
33
U.S. average score (509) lower than in 11 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
34
U.S. average (509) not measurably different than in 12 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
35
U.S. average score (509) higher than in 32 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
36
Average mathematics scores of 4th-grade students increased from 2007 to 2011 in 12
education systems, including the U.S.
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average mathematics score in 2011.
Chinese Taipei-CHN
Japan
UNITED STATES
Denmark
Quebec-CAN
Slovenia
Czech Republic
Norway
Dubai-UAE
Georgia
Iran, Islamic Rep. of
Tunisia
-20 -10 0 10 20 30 40
15*
17*
12*
14*
14*
11*
24*
22*
24*
12*
28*
32*
Change in average score
Average mathematics scores of 8th-grade students increased from 2007 to 2011 in 10
education systems
37
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average mathematics score in 2011.
Korea, Rep. of
Singapore
Chinese Taipei-CHN
Russian Federation
UNITED STATES
Italy
Ukraine
Dubai-UAE
Georgia
Bahrain
Palestinian Nat'l Auth
-20 -10 0 10 20 30 40
16*
18*
11*
27*
1
19*
17*
17*
22*
11*
37*
Change in average score
38
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average mathematics score in 2011.
Average mathematics scores of 8th-grade students decreased from 2007 to 2011 in 6
education systems
Hungary
Sweden
Malaysia
Thailand
Jordan
Syrian Arab Republic
-40 -30 -20 -10 0 10 20
-12*
-7*
-34*
-14*
-21*
-15*
Change in average score
TIMSS international mathematics benchmarks
Grade 4 Grade 8
Advanced (625) Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning.
Students can reason with information, draw conclusions, make generalizations, and solve linear equations.
High (550) Students can apply their knowledge and understanding to solve problems.
Students can apply their understanding and knowledge in a variety of relatively complex situations.
Intermediate (475)
Students can apply basic mathematical knowledge in straightforward situations.
Students can apply basic mathematical knowledge in straightforward situations.
Low (400) Students have some basic mathematical knowledge.
Students have some knowledge of whole numbers and decimals, operations, and basic graphs.
39
Percentages of U.S. 4th-graders reaching TIMSS mathematics benchmarks were
higher than international medians in 2011
Low Intermediate High Advanced0
10
20
30
40
50
60
70
80
90
100 96
81
47
13
90
69
28
4
United StatesInternational median
Benchmark
Percent at or above
40
NOTE: All U.S. percentages are significantly higher than the corresponding TIMSS international median at the .05 level of statistical significance.
Higher than U.S. (p < .05)
Not measurably different than U.S. (p < .05)
NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U.S. students reaching the Advanced benchmark are not included in figure.
41
Seven systems had higher percentages of 4th-grade students reaching Advanced than the U.S.
Finland
UNITED STATES
Russian Federation
Florida-USA
North Carolina-USA
England-GBR
Northern Ireland-GBR
Japan
Chinese Taipei-CHN
Hong Kong-CHN
Korea, Rep. of
Singapore
0 10 20 30 40 50 60
12
13
13
14
16
18
24
30
34
37
39
43
Percentage reaching Advanced benchmark
Low Intermediate High Advanced0
10
20
30
40
50
60
70
80
90
10092
68
30
7
75
46
17
3
United StatesInternational median
Benchmark
Percent at or above
42
NOTE: All U.S. percentages are significantly higher than the corresponding TIMSS international median at the .05 level of statistical significance.
Percentages of U.S. 8th-graders reaching TIMSS mathematics benchmarks were
higher than international medians in 2011
NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U.S. students reaching the Advanced benchmark are not included in figure.
43
Eleven systems had higher percentages of 8th-graders reaching Advanced than the U.S.
Higher than U.S. (p < .05)
Not measurably different than U.S. (p < .05)
California-USADubai-UAE
New ZealandLithuaniaRomania
Quebec-CAN UNITED STATES
Indiana-USATurkey
HungaryColorado-USA
Florida-USAEngland-GBR
AustraliaConnecticut-USA
IsraelMinnesota-USA
North Carolina-USARussian Federation
Massachusetts-USAJapan
Hong Kong-CHNKorea, Rep. of
SingaporeChinese Taipei-CHN
0 10 20 30 40 50 60
555556
777
8888
910
1213
1414
1927
344748
49
Percentage reaching Advanced benchmark
SCIENCE
TIMSS 2011 science framework
45
TIMSS science
Content dimensions Grade 4Earth scienceLife sciencePhysical science
Grade 8BiologyChemistryEarth sciencePhysics
Cognitive dimensions
KnowingApplyingReasoning
What is on the TIMSS science assessment?
46
GRADE 4 Percentage of 4th-
graders answering correctly:U.S.: 96%Int’l Avg.: 83%
What is on the TIMSS science assessment?
47
GRADE 4
Percentage of 4th-graders answering correctly:U.S.: 90%Int’l Avg.: 73%
What is on the TIMSS science assessment?
48
GRADE 8
Percentage of 8th-graders answering correctly:U.S.: 28%Int’l Avg.: 27%
What is on the TIMSS science assessment?
49
GRADE 8
Percentage of 8th-graders answering correctly:U.S.: 76%Int’l Avg.: 61%
U.S. average score (544) higher than the TIMSS scale average (500)
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
50
51
U.S. average score (544) lower than in 6 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
52
U.S. average (544) not measurably different than in 3 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
53
U.S. average score (544) higher than in 47 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
54
U.S. average score (525) higher than the TIMSS scale average (500)
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
55
U.S. average score (525) lower than in 12 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
56
U.S. average (525) not measurably different than in 10 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
57
U.S. average score (525) higher than in 33 education systems
Higher than U.S.
Not measurably different than U.S.
Lower than U.S.
58
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average science score in 2011.
Average science scores of 4th-grade students increased from 2007 to 2011 in 9
education systemsJapan
UNITED STATES
Czech Republic
Sweden
Netherlands
Denmark
Norway
Georgia
Iran, Islamic Rep. of
Tunisia
-20 -10 0 10 20 30 40
11*
5
21*
9*
8*
11*
17*
37*
17*
27*
Change in average score
59
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average science score in 2011.
Average science scores of 4th-grade students decreased from 2007 to 2011 in 5
education systems
Hong Kong-CHN
England-GBR
Italy
Australia
New Zealand
-30 -20 -10 0 10 20 30
-19*
-13*
-11*
-12*
-7*
Change in average score
60
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average science score in 2011.
Average science scores of 8th-grade students increased from 2007 to 2011 in 9
education systems
Singapore
Korea, Rep. of
Minnesota-USA
Russian Federation
UNITED STATES
Quebec-CAN
Ukraine
Norway
Iran, Islamic Rep. of
Palestinian Nat'l Auth
-20 -10 0 10 20 30 40
23*
7*
15*
13*
5
13*
16*
8*
15*
16*
Change in average score
61
*p < .05. Change in average scores is significant.NOTE: Education systems ordered according to average science score in 2011.
Average science scores of 8th-grade students decreased from 2007 to 2011 in 7
education systems
Hungary
Bahrain
Thailand
Jordan
Malaysia
Syrian Arab Republic
Indonesia
-50 -40 -30 -20 -10 0 10
-17*
-15*
-20*
-33*
-44*
-26*
-21*
Change in average score
TIMSS international science benchmarks
62
Grade 4 Grade 8Advanced (625) Students can apply knowledge
and understanding of scientific processes and relationships and show some knowledge of the process of scientific inquiry.
Students communicate an understanding of complex and abstract concepts in biology, chemistry, physics, and earth science.
High (550) Students apply their knowledge and understanding of the sciences to explain phenomena in everyday and abstract contexts.
Students demonstrate understanding of concepts related to science cycles, systems, and principles.
Intermediate (475)
Students have basic knowledge and understanding of practical situations in the sciences.
Students recognize and apply their understanding of basic scientific knowledge in various contexts.
Low (400) Students have some elementary knowledge of life science and physical science.
Students can recognize some basic facts from the life and physical sciences.
Low Intermediate High Advanced0
10
20
30
40
50
60
70
80
90
100 96
81
49
15
92
72
32
5
United StatesInternational median
Benchmark
Percent at or above
63
NOTE: All U.S. percentages are significantly higher than the corresponding TIMSS international median at the .05 level of statistical significance.
Percentages of U.S. 4th-graders reaching TIMSS science benchmarks were higher
than international medians in 2011
NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U.S. students reaching the Advanced benchmark are not included in figure.
64
Higher than U.S. (p < .05)
Not measurably different than U.S. (p < .05)
Three systems had higher percentages of 4th-grade students reaching Advanced than the U.S.
Singapore
Korea, Rep. of
Finland
Russian Federation
Chinese Taipei-CHN
UNITED STATES
Japan
Florida-USA
Hungary
North Carolina-USA
0 10 20 30 40 50 60
33
29
20
16
15
15
14
14
13
12
Percentage reaching Advanced benchmark
Low Intermediate High Advanced0
10
20
30
40
50
60
70
80
90
10093
73
40
10
79
52
21
4
United StatesInternational median
Benchmark
Percent at or above
65
Percentages of U.S. 8th-graders reaching TIMSS science benchmarks were higher
than international medians in 2011
NOTE: All U.S. percentages are significantly higher than the corresponding TIMSS international median at the .05 level of statistical significance.
NOTE: Education systems with lower percentages of students reaching the Advanced benchmark than the percentage of U.S. students reaching the Advanced benchmark are not included in figure. 66
Higher than U.S. (p < .05)
Not measurably different than U.S. (p < .05)
Twelve systems had higher percentages of 8th-graders reaching Advanced than the U.S.
Singapore
Massachusetts-USA
Chinese Taipei-CHN
Korea, Rep. of
Japan
Minnesota-USA
Colorado-USA
Connecticut-USA
Russian Federation
England-GBR
Slovenia
Florida-USA
Finland
North Carolina-USA
Alberta-CAN
Israel
Australia
Indiana-USA
UNITED STATES
Hong Kong-GBR
New Zealand
Hungary
Turkey
0 10 20 30 40 50 60
40
24
24
20
18
16
14
14
14
14
13
13
13
12
12
11
11
10
10
9
9
9
8
Percentage reaching Advanced benchmark
Summary of change in average U.S. scores over time
67
PIRLS Reading2006-2011
TIMSSMathematics2007-2011
TIMSS Science2007-2011
TIMSSMathematics2007-2011
TIMSSScience2007-2011
Grade 4
Grade 8
Change over time in U.S. average scores was statistically significant.
Change over time in U.S. average scores was not measurably different.
For more information
PIRLS at NCES:
http://nces.ed.gov/surveys/pirls/
International Data Explorer:
http://nces.ed.gov/surveys/pirls/idepirls/
Contact:
Sheila Thompson
NCES
Sheila.Thompson@ed.gov
202-502-7425
68
TIMSSPIRLSTIMSS at NCES:
http://nces.ed.gov/timss/
International Data Explorer:
http://nces.ed.gov/timss/idetimss/
Contact:
Stephen Provasnik
NCES
Stephen.Provasnik@ed.gov
202-502-7480