Comparing
-
Upload
yenny-chandra -
Category
Documents
-
view
20 -
download
1
Transcript of Comparing
Indonesian Performance Compared to other Countries 1
INDONESIAN PERFORMANCE COMPARED TO OTHER COUNTRIES
Introduction
The results of the Third International Mathematics and Science Studies (TIMSS)
undertaken by the IEA indicate that among the participating countries, Singapore
outperforms all other countries in both Mathematics and Science, followed by
Korea, Japan, Hong Kong, and Belgium (Fl) in Mathematics and Czech
Republic, Japan, Korea and Bulgaria in Science. At the other end of the scale,
South African Junior Secondary School students are the lowest achievers in both
Mathematics and Science (Lokan et al., 1996).
The purpose of this article is to report the results of the analysis of the
Indonesian Population 2 data in order to get some information on the
performance in the Indonesian Junior Secondary School students in Mathematics
and Science compared to other countries participating in the TIMSS. The data
used for this comparison is the Grade 8 student responses to the mathematics and
science achievement tests. For this purpose, it is necessary to put the Indonesian
data on the same scale as that of the TIMSS in order to get comparable scores.
The procedure of placing two or more tests onto the same scale is known as
equating (Jaeger, 1981; Petersen et al., 1989; Cook and Eignor, 1991).
It has been stated previously that Indonesia was not included in the main
reports of the TIMSS because of problems that were encountered in the data
collection. Eng (1996) has reported that the exclusion of Indonesia and four other
countries (Argentina, Italy, Mexico, and the Philippines) in the report were
because Indonesia and the other four countries did not “complete the steps
necessary for their data to appear in the report (p. 4). However, in 1996,
Indonesia readministered the Mathematics and Science tests employing the same
instruments which had been used earlier so that data are available for analysis.
The TIMSS reports and the reports of its predecessors have been
discussed quite extensively by many researchers. Some researchers extend their
Ramon Mohandas
Indonesian Performance Compared to other Countries 2
support to such studies while some others question the significance of the studies.
Stedman (1997) has raised questions about how large the achievement
differences were among the countries participating in TIMSS. He argued that
when the scores of students in different countries were ranked, the position of a
country in the rank assigned might differ greatly in the rank compared with the
rank assigned to a particular country even though in fact their scores might be
relatively close together. For this reason, he suggested that the use of ranking
might give misleading impression on the performance of a particular country.
Nevertheless, Baker (1997, p. 16) claimed that the ranking of participating
countries in an international study such as the one conducted by TIMSS was
tempting and might be “useful in a limited way”. However, he cautioned that this
ranking “should not be the main focus, particularly among researchers”.
Method of the Equating
In order to place the Indonesian data on the same scale as that of the TIMSS
reports, and hence to compare the achievement of the Indonesian Junior
Secondary School students with other students at the same level in the
participating countries, the Indonesian data have to be equated with the data of
some other participating countries. Fortunately, the data of one of the
participating countries, namely Australia, were accessible for this purpose.
Several studies have shown that concurrent equating for horizontal
equating is better than the anchor item equating (Mohandas, 1996). For this
reason, concurrent equating in this study is employed to locate Indonesian and
Australian data on the same scale. In the concurrent equating, the data for the
student responses from the different forms to be equated are combined together
to form a large data file, provided that there are common items available in each
form. In this study, there were eight forms of both the mathematics and science
tests administered to the Junior Secondary School students at Grade 8 in
Indonesia and the same tests with the same number of forms were also
administered to Year 8 students in Australia.
Ramon Mohandas
Indonesian Performance Compared to other Countries 3
The report of the analysis in this chapter is divided into three parts. The
first part involves comparing the mathematics achievement of the Grade 8
students in Indonesia with other countries participating in the TIMSS. The
second part involves comparing the science achievement of the Grade 8 students
in Indonesia with other countries participating in TIMSS, and the third part
provides the conclusion.
The concurrent equating of the Indonesian data and the Australian data
was undertaken by first combining the two data files to become a single large
data file. In the Indonesian data file, there were 3929 Grade 8 students, while in
the Australian data file, there were 7926 Year 8 students. The mathematics and
science tests each consisted of eight different forms, with some items being
common across the different forms (the details of the test structure was presented
in Chapter 4). Since the Indonesian data contained only student’s responses from
the multiple-choice items, the concurrent equating procedure reported in this
study analyses only the student’s responses to the multiple-choice questions in
the Indonesian data as well as in the Australian data.
It has been described earlier that the mathematics and science tests each
consisted of eight different forms, with several common items across the
different forms. The structures of the mathematics and science tests were also
presented in that chapter. The mathematics and science tests used in Indonesia
were basically the same as those used in Australia, although there was the
necessary difference in the language used in the tests. Consequently, the correct
response codes given to each item were the same for the Indonesian and the
Australian data. However, for some items in the Indonesia data in this study, the
number of alternatives for certain multiple-choice items was changed from their
original five options to four options. For this reason, these items had to be treated
as different items in the Indonesian data file and the Australian data file. There
were 15 items which were treated as different items because of this problem,
consisting of 12 mathematics items (a02, b10, e01, e05, g06, h08, j14, m02, p17,
q01, q09, and r09) and three science items (l01, n01, and q14). While these items
Ramon Mohandas
Indonesian Performance Compared to other Countries 4
were excluded from calibration and equating, they were not excluded in the
scoring.
As has been described earlier, there were also items in Indonesia which
had to be deleted from the analysis because of translation and test construction
problems. There were 12 items with this these problems, consisting of eight
mathematics items (a07, d10, f10, j11, j15, k08, l14, and p15) and four science
items (a12, f06, j02, and p04). In addition to these 12 items, two items had to be
deleted from further analyses because the preliminary analysis of the Indonesian
data showed that these two items had very poor item statistics. These items were
mathematics items, namely item f08 with a point biserial value of 0.07 and item
o04 with a point biserial value of -0.01. Consequently, there were 14 items
deleted for the analyses of the Indonesian data. In the combined data file,
consequently, the Indonesian student’s responses for these items were treated as
blank so that they could be ignored in the analysis.
Comparing Mathematics Achievement
In this section, the comparison of the Indonesian student achievement in
mathematics with that in other countries participating in the TIMSS is reported in
three parts. The first part discusses the results of the concurrent equating
procedure, the second part discusses the procedures employed to locate the
results of the equating onto the same scales as the ones presented in the TIMSS
reports issued by IEA, and the third part presents the comparison of the
Indonesian student achievement in mathematics with other countries
participating in TIMSS.
The results of the concurrent equating in the first part are required to
obtain the case estimates of the Australian data by State using the Rasch scale.
The second part, presenting the data of the Australian Year 8 student
achievement in mathematics by states as reported by Lokan et al. (1996), is
required to put the scales of the resulting equating procedure onto the same scale
as the scales used in the TIMSS report. By obtaining the Australian data by State
Ramon Mohandas
Indonesian Performance Compared to other Countries 5
in the form of the Rasch scale from the equating procedure and the scale used in
the TIMSS report, the correlation coefficient between these two scales could be
obtained, together with their corresponding standard deviation values.
Subsequently, the regression equation could be used to calculate the equivalent
value of the Indonesian performance similar to that used in the TIMSS report,
which records the performances of other countries that participated in the study.
Results of the Concurrent Equating Procedure for Mathematics
The concurrent equating of the Indonesian and the Australian data was
undertaken using the QUEST program (Adams and Khoo, 1993). The results of
the analysis indicate that five mathematics items had to be deleted from the
analysis because their infit mean square values were greater than the acceptable
values, which ranged from 0.77 to 1.30 as described by Adams and Khoo (1993).
The results of the final run of the mathematics item estimates are presented in
Appendix 12.2.
The QUEST output that is required to obtain the scores of each individual
student in Indonesia as well as in Australia is the output from the show case
command line. However, the output of this command line does not provide the
scores for students who answered all questions correctly (perfect scores) or who
answered all questions incorrectly (zero scores). Consequently, these perfect and
zero scores have to be calculated manually before further analysis takes place.
For this purpose, individual students in the Indonesian and Australian data had to
be identified and grouped according to the test booklet they worked on so that
each group would consist of students who attempted exactly the same questions.
The calculation of the perfect scores for each group of students who took the
same test booklets is carried out by taking into consideration the first three
highest scores for the groups of students who made one, two, and three errors
respectively. Similarly, the calculation of the zero scores is carried out by taking
into consideration the last three lowest scores for the corresponding group of
students who obtained one, two, and three items correct respectively. For
Ramon Mohandas
Indonesian Performance Compared to other Countries 6
example, if the first three highest scores in the particular group of students are
3.42, 2.68, and 2.22, the perfect score is:
3.42 + (3.42 - 2.68) + {(3.42 - 2.68) - (2.68 - 2.22)} = 4.44.
If the last three lowest scores in the particular group of students are -2.56, -2.21,
and -1.92, the zero score is:
-2.56 + (-2.56 - -2.21) + {(-2.56 - -2.21) - (-2.21 - -1.92)} = -2.97.
The scores for all individual students in the Indonesian as well as in the
Australian data, after calculating the perfect and the zero scores, are then
transferred into the SPSS program to calculate the mean score for Indonesia and
the mean scores for each State in Australia after taking into account the weights
assigned to the Indonesian and the Australian data. The procedure used in the
calculation of the weights for the Indonesian and the Australian data is the same
as the procedure discussed in Chapter 11.
Table 12.1 presents the mean scores of the Australian states in Rasch
scores (logit) based on the results obtained from the SPSS calculations. The
figures in the last column are the mean scores of the Australian States using the
scale presented in the TIMSS reports, as given by Lokan et al. (1996). The mean
scores reported here are taken for Year 8 students in the States rather than by
using upper grade and lower grade categories employed in the report. In other
words, for Australian Capital Territory, New South Wales, Victoria, and
Tasmania, the mean scores taken from Table A.6: Mathematics Achievement by
State, Upper Grade in Lokan et al. (1996), while for Western Australia, South
Australia, Queensland, and Northern Territory, the mean scores are taken from
Table A.7: Mathematics Achievement by State, Lower Grade. This decision was
made because of the lack of equivalence in the upper and lower grades in the
school systems in the States concerned. Furthermore, the Australian data
analysed
Table 12.1: Mathematics Mean Scores for Australian States in Logit and TIMSS
Ramon Mohandas
Indonesian Performance Compared to other Countries 7
State Logit TIMSSACT 0.800 548NSW 0.775 522VIC 0.503 511QLD 0.602 512SA 0.624 516WA 0.861 532TAS 0.445 501NT 0.352 478Mean 0.620 515Standard Deviation 0.182 21Correlation 0.910
in this study consisted of Year 8 students, so that by making this decision, the
year level of students in the data would match the year level of students in the
table. In this report, the scale units of the scores derived from this equating
procedures are referred to as logits, while the scores obtained from the TIMSS
report are referred to as TIMSS units. The Logit scale is a scale with meaningful
units, while the units of the scale employed in the TIMSS report are not clearly
specified in a way that would facilitate equating, in either the Australian national
or the international reports.
From these two sets of data (Logit and TIMSS), the correlation coefficient
between the two sets can be calculated, together with their Means and Standard
Deviations to permit the use of the regression equation.
Based on these figures, the next section discuss the use of the regression
equation to calculate the estimates for the Indonesian data using the TIMSS
scale.
Locating the Indonesian Mathematics Scores to the TIMSS Scale
In this section, a regression equation is used to predict the Indonesian scores on
the TIMSS scale so that they can be compared to the results presented in the
reports on mathematics achievement in the TIMSS. Guilford and Fruchter (1981,
Ramon Mohandas
Indonesian Performance Compared to other Countries 8
p. 346) have argued that the main use of a regression equation is “to predict the
most likely measurement in one variable from the known measurement in
another”. In this procedure, the accuracy of the prediction is determined by the
correlation coefficient between the two known measures. As Guilford and
Fruchter (1981, p. 346) put it, “the higher the correlation, the greater the accuracy
of prediction and the smaller the errors of prediction”.
Hinkle et al. (1988, p. 128) suggest that the procedure of prediction
involves two steps, namely (1) determining the regression line as a mathematical
equation, and (2) using the mathematical equation to predict scores. The
mathematical equation to determine the regression line is: $Y = bX + a Equation 12.1
where $Y = the predicted score
b = the slope for the line
a = the intercept.
Hinkle et al. (1988) further show that the slope for the line (b) is also
called the regression coefficient, and is calculated using the following formula:
b = (r)SySx
Equation 12.2
where
r = the correlation between variables X and Y
Sy = standard deviation of the Y scores
Sx = standard deviation of the X scores.
The intercept of the regression line (a), is also called the regression constant, and
is calculated using the following formula:
a = Y - b X Equation 12.3
where
Y = the mean score of Y
Ramon Mohandas
Indonesian Performance Compared to other Countries 9
b = the slope of the regression coefficient
X = the mean score of X.
By using the three formulae presented above, it is possible to calculate the
predicted value of the new scores for each state in Australia, and at the same
time, using the same formulae, to calculate the predicted value of the new score
for Indonesia. This can be done because as a result of the concurrent equating,
the scores of the Indonesian students are already in the same logit scale as the
scores of the Australian students. Hence, the new predicted scores of the
Indonesian students on the TIMSS scale are also on the same scale as the
Australian scores, which also means they are located on the same scale as the
other countries participating in the TIMSS.
Table 12.2 presents the calculation of the predicted scores for the
Australian data based on the information provided in Table 12.1 and using the
three formulae described above. Since there are two groups of scores given in
Table 12.1 (Logit and TIMSS), the correlation coefficient (r) of the two groups
and the standard deviation of each group (Sx and Sy) can be calculated.
Consequently, the slope (b) and the intercept (a) of the regression line can also be
calculated.
The results in Table 12.2 show that the correlation coefficient (r) between
the Logit scores (X) and the TIMSS scores (Y) is 0.91, which indicates a strong
correlation. The mean of the resulting predicted scores is the same as the mean of
the TIMSS score, which is 515. Based on these results, it can be argued that the
calculation of the Indonesian score can be undertaken with relatively high
accuracy.
The correlation between the two sets of scores for the Australian States is
not perfect (ie equal to unity as might be expected) for the following reasons.
1. Some items were excluded from consideration in the calculation of the logit
scale, although this should not lead to scale differences.
Ramon Mohandas
Indonesian Performance Compared to other Countries 10
Table 12.2: Calculation of Predicted Scores for Australia in Mathematics Achievement
STATES Logit
(X) TIMSS
(Y) PREDICTED
SCORES ACT 0.800 548 $Y bX a= + = 534 NSW 0.775 522 $Y bX a= + = 531 VIC 0.503 511 $Y bX a= + = 503 QLD 0.602 512 $Y bX a= + = 513 SA 0.624 516 $Y bX a= + = 515 WA 0.861 532 $Y bX a= + = 540 TAS 0.445 501 $Y bX a= + = 497 NT 0.352 478 $Y bX a= + = 487
Mean 0.620 515 515
r 0.910 Sy 20.764 Sx 0.182 b 103.964 a 450.516 Indonesia -0.510 $Y bX a= + = 397
2. The weights assigned in the calculation of the logit scale scores did not
employ the procedures used in the TIMSS which were considered to have
involved errors.
3. The numbers of cases on the two data files obtained for the Australian data did
not correspond, nor did they correspond with the numbers of cases reported
for the Australian TIMSS (Lokan et al., 1997).
Nevertheless, these shortcomings in the Australian analyses are not likely to have
led to gross errors in the estimation of the Indonesian scores, and are probably no
greater than the unknown bias introduced into the Australian data by the
relatively low level of response. Likewise while items were excluded from the
calibration, equating, and scoring of the Indonesian data, the Rasch scaling
procedure makes allowance for these exclusions to produce equivalence to the
Ramon Mohandas
Indonesian Performance Compared to other Countries 11
scores obtained for the Australian States through the concurrent equating
procedure employed.
From the results of calculation of the mean score for Indonesia undertaken
in the same way as the calculation of the Australian data, the mean score for
Indonesia in mathematics is -0.51. By applying the mathematical equation used
for the Australian data to calculate the predicted score, the predicted score for
Indonesia is shown at the foot of Table 12.2, which is 397. This figure is taken as
the score of the Indonesian students at Grade 8 in mathematics, which is on the
same scale as the scores presented in the TIMSS reports.
The next section discusses the mathematics achievement of the Indonesian
students at Grade 8 by comparing the scores obtained from the above
calculations to the scores of other countries participating in TIMSS.
Comparing Indonesian Student Achievement in Mathematics with Other Countries Participating in TIMSS
The comparison of the Indonesian students’ achievement in mathematics with
other countries participating in TIMSS is undertaken by using the predicted score
as shown in Table 12.2, which is 397.
Table 12.3 shows the position of Indonesia among other countries
participating in the TIMSS. It is seen in the table that Indonesia is ranked number
39 out of 42 countries in the table. The three countries below Indonesia are
Kuwait, Colombia, and South Africa with scores of 392, 385, and 354
respectively. The four top positions are all held by Asian countries, namely
Singapore, Korea, Japan, and Hong Kong with the scores of 643, 607, 605, and
588 respectively. Australia, a country used as a base to put Indonesia onto the
same scale as the one reported in TIMSS, is ranked number 16 with a score of
539.
Another Asian country, Thailand, is ranked number 20 with a score of
522. However, Beaton et al. (1996b) report that the difference in the mathematics
achievement of students in Australia and Thailand is not significant, in spite of
Ramon Mohandas
Indonesian Performance Compared to other Countries 12
Table 12.3: Comparison of Mathematics Achievement at Year 8 among Countries Participating in the TIMSSa
No. Country Mean Standard Error1 Singapore 643 4.92 Korea 607 2.43 Japan 605 1.94 Hong Kong 588 6.55 Belgium (Fl) 565 5.76 Czech Republic 564 4.97 Slovak Republic 547 3.38 Switzerland 545 2.89 Netherlands 541 6.710 Slovenia 541 3.111 Bulgaria 540 6.312 Austria 539 3.013 France 538 2.914 Hungary 537 3.215 Russian Federation 535 5.316 Australia 530 4.017 Ireland 527 5.118 Canada 527 2.419 Belgium (Fr) 526 3.420 Thailand 522 5.721 Israel 522 6.222 Sweden 519 3.023 Germany 509 4.524 New Zealand 508 4.525 England 506 2.626 Norway 503 2.227 Denmark 502 2.828 United States 500 4.629 Scotland 498 5.530 Latvia (LSS) 493 3.131 Spain 487 2.032 Iceland 487 4.533 Greece 484 3.134 Romania 482 4.035 Lithuania 477 3.536 Cyprus 474 1.937 Portugal 454 2.538 Iran 428 2.239 Indonesia 397 b40 Kuwait 392 2.541 Colombia 385 3.442 South Africa 354 4.4 a Adapted from: Beaton, A.E., et al., 1996b. Mathematics Achievement in the Middle School Years, p. 22 b
No estimate could be made for the standard error of the score for Indonesia.
Ramon Mohandas
Indonesian Performance Compared to other Countries 13
the gap in the ranking position (16 and 20). The mean achievement of the top
performer, Singapore, is significantly higher than the mean achievement of any
other countries listed in the table. Furthermore, the mean achievement of the
country at the bottom, South Africa, is significantly lower than the mean
achievement of any other countries listed in the table.
Lokan et al. (1996) report that the scale used in the TIMSS reports is
based on item response theory (IRT) and the scale has a mean of 500 and a
standard deviation of 100. It should be noted here that while the scale mean is
presumably the average difficulty level of the mathematics test items, the
reference sample to obtain the standard deviation is not reported, nor is a reason
given for not using the natural unit of the Rasch scale, the logit. Several sub-
studies in the TIMSS project indicate that 30 scale points of mathematics
achievement reported in the TIMSS is equivalent to a year’s growth in
mathematics achievement at the middle secondary school stage (Stedman, 1997;
Beaton et al., 1996b). If this is the case, then, based on the figures in Table 12.3,
Indonesia is lagging four years behind Thailand and Australia, and eight years
behind Singapore, which is clearly several years of mathematics learning ahead
of Australia.
This results raise concerns about the performance of the Indonesian
students in mathematics at the Junior Secondary School level.
Comparing Science Achievement
In this section, the comparison of Indonesian student achievement in science with
other countries participating in the TIMSS, as in the section for mathematics, is
reported in three parts. The first part discusses the results of the concurrent
equating procedure, the second part discusses the procedures employed to place
the results of the equating onto the same scales as the ones presented in the
TIMSS reports issued by IEA, and the third part presents the comparison of the
Ramon Mohandas
Indonesian Performance Compared to other Countries 14
Indonesian student achievement in science with other countries participating in
TIMSS.
The results of the concurrent equating in the first part are required to
obtain the case estimates of the Australian data in science by State using the
Rasch scale. The second part, presenting the data of the Australian Year 8 student
achievement in science by states as reported by Lokan et al. (1996), is required to
locate the scales of the resulting equating procedure onto the same scale as the
scales used in the TIMSS report. By obtaining the Australian data by State in the
form of the Rasch scale from the equating procedure and the scale used in the
TIMSS report, the correlation coefficient between these two scales could be
obtained, together with their corresponding standard deviation values.
Subsequently, the regression equation could be used to calculate the equivalent
value of the Indonesian performance on the scale that is similar to that used in the
TIMSS report.
Results of the Concurrent Equating Procedure for Science
The concurrent equating of the Indonesian and the Australian data for science
was undertaken using the QUEST program (Adams and Khoo, 1993). The results
of the analysis indicate that for the science items, there were no items that had to
be deleted from the analysis because the infit mean square values for all items
were within the acceptable range, from 0.77 to 1.30, as described by Adams and
Khoo (1993). The results of the final run of science item estimates are presented
in Appendix 12.3.
The QUEST output that is required to obtain the science scores of each
individual student in Indonesia as well as in Australia is the output from the show
case command line. However, the output of this command line does not provide
the scores for students who answered all questions correctly (perfect scores) or
for those who answered all questions incorrectly (zero scores). Consequently,
these perfect and zero scores have to be calculated manually before further
analysis takes place. For this purpose, individual students in the Indonesian and
Ramon Mohandas
Indonesian Performance Compared to other Countries 15
Australian data had to be identified and grouped according to the test booklet
they took so that each group would consist of students who attempted exactly the
same questions. The procedures employed in the calculation of the perfect scores
and the zero scores are the same as those undertaken for mathematics.
The science scores for all individual students in the Indonesian as well as
in the Australian data, after calculating the perfect and the zero scores, are then
transferred to the SPSS program to calculate the mean score for Indonesia and
the mean scores for each State in Australia by taking into account the weights
given for the Indonesian and the Australian data as presented in mathematics.
Table 12.4 gives the science mean scores of the Australian States in Rasch
score (logit) units based on the results from the SPSS calculation. The figures on
the last column are the mean scores of the Australian States using the scale
presented in the TIMSS report, as given by Lokan et al. (1996). The mean scores
reported here are taken for Year 8 students in the States rather than by using the
upper grade and lower grade categories employed in the TIMSS reports. In other
words, for Australian Capital Territory, New South Wales, Victoria, and
Tasmania, the mean scores are taken from Table A.8: Science Achievement by
State, Upper Grade in Lokan et al. (1996), while for Western Australia, South
Australia, Queensland, and Northern Territory, the mean scores are taken from
Table A.9: Science Achievement by State, Lower Grade. This decision is made
because of the differences in the school systems in the States concerned.
Furthermore, the Australian data analysed in this study consisted of Year 8
students, so that by making this decision, the year level of students in the data
would match the year level of students in the table. In this report, the scores
derived from the equating procedures are referred to as Logit, while the scores
obtained from the TIMSS report are referred to as TIMSS.
From these two sets of data (Logit and TIMSS), the correlation coefficient
between the two types of figures is calculated, together with their standard
deviation.
The next section discusses the use of the regression equation to calculate
the estimates for the Indonesian data using the TIMSS scale.
Ramon Mohandas
Indonesian Performance Compared to other Countries 16
Table 12.4: Science Mean Scores for Australian States in Logit and TIMSS
State Logit TIMSSACT 0.793 564NSW 0.662 538VIC 0.448 521QLD 0.503 519SA 0.611 531WA 0.724 545TAS 0.528 525NT 0.510 504Mean 0.597 531Standard Deviation 0.121 18Correlation 0.910
Locating Indonesian Science Scores to the TIMSS Scale
In this section, the regression equation is used to predict the Indonesian scores in
the TIMSS scale so that they can be compared to the results in the reports on
science achievement in the TIMSS. The mathematical equations used to
determine the regression line are the same as the ones presented for mathematics,
namely Equations 12.1, 12.2, and 12.3.
By using the three formulae given earlier, it is possible to calculate the
predicted value of the new science scores for each State in Australia, and at the
same time, using the same formulae, to calculate the predicted value of the new
science score for Indonesia. This can be done because as a result of the
concurrent equating, the science scores of the Indonesian students are already on
the same scale as the science scores for the Australian students. Hence, the new
predicted science scores for the Indonesian students on the TIMSS scale are also
on the same scale as the Australian scores, as well as that for other countries
participating in the TIMSS.
Ramon Mohandas
Indonesian Performance Compared to other Countries 17
Table 12.5 presents the calculation of the predicted science scores for the
Australian data based on the information provided in Table 12.4 and using the
three formulae given earlier. Since there are two groups of scores given in Table
12.4 (Logit and TIMSS), the correlation coefficient (r) of the two groups and the
standard deviation of each group (Sx and Sy) can be calculated. Consequently,
the slope (b) and the intercept (a) of the regression line can also be calculated.
The results in Table 12.5 show that the correlation coefficient (r) between
the Logit scores (X) and the TIMSS scores (Y) is 0.91, which indicates a strong
correlation. The mean of the resulting predicted scores is the same as the mean of
the TIMSS score. From these results, it can be argued that the calculation of the
estimated Indonesian mean score can be undertaken with a relatively high degree
of accuracy.
From the results of calculation of the mean score for Indonesia undertaken
in the same way as the calculation of the Australian data, the mean score for
Indonesia in science is -0.18. By applying the regression equation used with the
Australian data to calculate the predicted score, the predicted score for Indonesia
is shown at the foot of Table 12.5, which is 424. This figure is taken as the mean
score for the Indonesian student at Grade 8 in science, which is on the same scale
as presented in the TIMSS reports.
The next section discusses the mean science achievement of the
Indonesian students at Grade 8 by comparing the score obtained from the above
calculations to the scores of other countries participating in TIMSS.
Comparing Indonesian Student Achievement in Science with Other Countries Participating in TIMSS
The comparison of the mean Indonesian student achievement in science with
other countries participating in TIMSS is undertaken by using the predicted score
as shown in Table 12.5, which is 424.
Table 12.6 shows the position of Indonesia among other countries
participating in the TIMSS. It is seen in the table that Indonesia is ranked number
Ramon Mohandas
Indonesian Performance Compared to other Countries 18
Table 12.5: Calculation of Predicted Scores for Australia in
Science Achievement STATES Logit TIMSS PREDICTED ACT 0.793 564 $Y bX a= + = 558 NSW 0.662 538 $Y bX a= + = 540 VIC 0.448 521 $Y bX a= + = 510 QLD 0.503 519 $Y bX a= + = 518 SA 0.611 531 $Y bX a= + = 533 WA 0.724 545 $Y bX a= + = 548 TAS 0.528 525 $Y bX a= + = 521 NT 0.510 504 $Y bX a= + = 519
Mean 0.597 531 531
r 0.910 Sy 18.295 Sx 0.121 b 137.615 a 448.667 INA -0.180 $Y bX a= + = 424
40 out of 42 countries. The two countries below Indonesia are Colombia and
South Africa with scores of 411 and 326 respectively. The four top positions are
held by Singapore, Czech Republic, Japan, and Korea with the scores of 607,
574, 571, and 565 respectively. Hong Kong, which is ranked number 4 in
mathematics, is performing relatively poorly in science compared to its level of
achievement in mathematics, and is ranked number 24 in science with a score of
522. On the other hand, the Czech Republic, which is ranked number 6 in
mathematics, is performing very well in science, and is ranked number 2.
Australia, a country used as the base to locate Indonesia onto the same scale as
that reported in TIMSS, is ranked number 12 with a score of 545. Another Asian
country, Thailand, is ranked number 22 with a score of 525. However, Beaton et
Ramon Mohandas
Indonesian Performance Compared to other Countries 19
Table 12.6: Comparison of Science Achievement at Year 8 among Countries Participating in the TIMSSa
No. Country Mean Standard Error1 Singapore 607 5.52 Czech Republic 574 4.33 Japan 571 1.64 Korea 565 1.95 Bulgaria 565 5.36 Netherlands 560 5.07 Slovenia 560 2.58 Austria 558 3.79 Hungary 554 2.810 England 552 3.311 Belgium (Fl) 550 4.212 Australia 545 3.913 Slovak Republic 544 3.214 Russian Federation 538 4.015 Ireland 538 4.516 Sweden 535 3.017 United States 534 4.718 Canada 531 2.619 Germany 531 4.820 Norway 527 1.921 New Zealand 525 4.422 Thailand 525 3.723 Israel 524 5.724 Hong Kong 522 4.725 Switzerland 522 2.526 Spain 517 1.727 Scotland 517 5.128 France 498 2.529 Greece 497 2.230 Iceland 494 4.031 Romania 486 4.732 Latvia (LSS) 485 2.733 Portugal 480 2.334 Denmark 478 3.135 Lithuania 476 3.436 Belgium (Fr) 471 2.837 Iran 470 2.438 Cyprus 463 1.939 Kuwait 430 3.740 Indonesia 424 b41 Colombia 411 4.142 South Africa 326 6.6 a Adopted from: Beaton, A.E., et al., 1996a. Science Achievement in the Middle School Years, p. 22 b No estimate could be made for the standard error of the score for Indonesia.
Ramon Mohandas
Indonesian Performance Compared to other Countries 20
al. (1996a) report that the difference in science achievement of students in
Australia and Thailand is significant, with the achievement in science for
students from Thailand being significantly lower than the achievement of the
Australian students. The mean achievement of the top performer, Singapore, is
significantly higher than the mean achievement of any other country listed in the
table. On the contrary, the mean achievement of the country at the bottom, South
Africa, is significantly lower than the mean achievement of any other country
listed in the table.
Lokan et al. (1996) report that the scale used in the TIMSS reports is
based on the item response theory (IRT) and the scale has a mean of 500 and a
standard deviation of 100. However, the scale mean and standard deviation relate
to an undefined population, and these statistics were presumably obtained by
some ill-defined pooling of a sample of cases drawn from the countries that
participated in the TIMSS testing program. Several analyses of the TIMSS data
report that 37 scale points of science achievement reported in the TIMSS would
be equivalent to a year’s growth in science achievement (Stedman, 1997; Beaton
et al., 1996a). If this is the case, then as in mathematics and based on the figures
in the table, Indonesia is lagging in science by almost three years behind
Thailand, and more than three years behind Australia, and almost five years
behind Singapore.
This result also raises concerns about the performance of the Indonesian
students in science at the Junior Secondary School level.
Conclusion
The results of the analysis indicate that in comparison with the other countries
participated in the TIMSS, Indonesia is performing very poorly both in
mathematics and science. In mathematics, Indonesia is ranked number 39 out 42
countries, while in science, Indonesia is ranked number 40 out of 42 countries. In
contrast, Singapore is the top performer for both mathematics and science.
Ramon Mohandas
Indonesian Performance Compared to other Countries
Ramon Mohandas
21
The apparently low level of performance of the Indonesian Junior
Secondary School students in mathematics and science found in this study raises
the question of why this could happen. What are the reasons for the low level of
performance?
The results of the analyses presented earlier indicate that at the student as
well as at the school levels, factors that consistently have significant influences
on achievement in mathematics and science are extra lessons and absenteeism.
This might suggest that students who spend more time engaging in mathematics
and science are more likely to achieve at a higher level in mathematics and
science than the students who spend less time on mathematics and science. If this
is the case, then should it mean that students might be encouraged to take extra
lessons after schools hours? The idea of encouraging students to take extra
lessons might not be wise because it would give extra burden to the parents
financially. One possible solution might be to decrease the seemingly
overcrowded curriculum and to put extra hours into the teaching of mathematics
and science.
Keeves (1995) reported that in the previous studies conducted by the IEA,
the achievement level of students from the developing countries is considerably
lower than the achievement level of students from the developed countries.
Several studies conducted in Indonesia also indicate that the achievement level of
the Indonesian students in mathematics and science is very low. The results of
comparison in this study confirm these findings.
Nevertheless, the reasons for the lower levels of achievement of the
Indonesian Junior Secondary School students in mathematics and science needs
to be investigated further in order to find ways of improving the quality of
mathematics and science teaching in Indonesia.