The role of TIMSS, PIRLS & PISA in Providing a True ... · skills and knowledge of 15-year-old...

The role of TIMSS, PIRLS & PISA in Providing a True Picture of Maltese

Students’ Attainment

Dr. Vincent Marmarà Ph.D.(Stir.), M.Sc.(Sheff.) Statistics, B.Sc.(Hons.)(Melit.), FRSS

for the Malta Union of Teachers

November 2018

Contents1) Main objectives

2) Methodology

3) PISA, TIMSS & PIRLS

4) One-to-one interviews

5) The quantitative study:

Methodology & Demographics

Results – Familiarity with PISA,TIMSS and PIRLS

Results – Performance of the three assessments

Results – Involvement in the three assessments

Results – a. Your workplace and the three assessments – level of importance

b. Your workplace and the three assessments – level of preparation (academic staff)

c. Your workplace and the three assessments – level of preparation (students)

Results – Opinions about the three assessments

Results – The level of students’ knowledge in relation to these assessments

Results – The three assessments and the Maltese syllabus

Results – The three assessments and the language impact

Results – Design and implementation of the assessments

Conclusion of the quantitative study

2

Main Objectives3

Main ObjectivesBased on the information provided by the Malta Union of Teachers (MUT), thescope of work covered the following:

To review and understand PISA, TIMSS and PIRLS by analysing the documentation related to these assessments and the perceptions of

these tests amongst educators and key personnel within the education sector

4

Methodology5

MethodologyThe Methodology for this project included:

1. Setting the necessary meetings with the Malta Union of Teachers (MUT) to understand theircontribution in the whole process;

2. Setting the necessary meetings with the Maltese educational authorities to understand theircontribution in the whole process. These meetings helped to understand the assessmentsbeing carried out in further detail as well as the curriculum and the process of data collection;

3. Setting the necessary meetings to understand the educational system and the perceptions ofkey stakeholders in relation to these tests. The latter meetings were held with the followingmembers:

a. Representatives from the Education department and officials from the Department forCurriculum, Lifelong Learning and Employability;

b. Officials from the Faculty of Education, University of Malta;

c. Independent Schools representatives;

d. Church Schools representatives;

4. Obtained the required data and documentation in relation to the scope of this project to carryout the analysis;

6

Methodology5. Furthermore, a quantitative survey study was carried out amongst Learning Support Educators

(LSEs), Kindergarten Educators (KGEs) Teachers, Heads of Departments (HoDs), Assistant Heads(AHs), Heads of Schools (HoSs) and Education Officers (EOs) to:

“Study and analyse the role of international studies (TIMSS, PIRLS and PISA) to provide a true picture of Maltese Students’ Attainment.”

Throughout this study we analysed questions such as:

a. “What do you consider your level of knowledge to be about these assessments?”

b. “Do you think that your workplace is giving enough importance to these assessments?”

c. “To what extent is your workplace preparing the students for these assessments?”

Disclaimer: This data collection was carried out during the month of October when someeducators were being given training related to the TIMSS assessment. Hence, some educators mayhave been more knowledgeable about this particular assessment.

7

PISA, TIMSS and PIRLS8

PISAWhat is PISA?

The Programme for International Student Assessment (PISA) organised by the

Organisation for Economic Co-operation and Development (OECD) is a triennial

international survey which aims to evaluate education systems worldwide by testing the

skills and knowledge of 15-year-old students. Moreover, the PISA programme investigates

and compares the performance of schools and education systems worldwide.

PISA focuses primarily on the assessment of student performance in reading,

mathematics and science as they are foundational to a student’s continuing education.

However, it also takes into account student attitudes, motivations, and collaborative

problem-solving.

Reference: oecd.org

9

Key findings for PISA 2015• Based on the current literature, no specific issues or flaws with respect to the actual

research methodology exists (data collection and analysis);

• PISA 2015: 38 countries had a mean Science score higher than Malta (465), while 32countries had a lower mean score. Malta’s mean science score was found to be lowerthan the international average (493);

• When comparing European countries, Malta scored higher in Science than in Greece

(455), Bulgaria (446), Romania (435), Cyprus (433), Moldova (428), Albania (427),

Turkey (425), Montenegro (411), Macedonia (384) and Kosovo (378);

• Malta scored significantly lower than Estonia (534), Finland (531), Slovenia (513),

Netherlands (509), Germany (509), United Kingdom (509), Switzerland (506), Ireland

(503), Denmark (502), Belgium (502), Poland (501), Portugal (501), Norway (498),

Austria (495), France (495), Sweden (493), Czech Republic (493), Spain (493), Latvia

(490), Luxembourg (483), Hungary (477), Croatia (475), Italy (481), Lithuania (475)

and Iceland (473);

10

Key findings for PISA 2015• The mean Science score of Maltese students in the PISA 2015 cycle (465) was 4 points

higher than the PISA 2009 cycle (461). Compared to PISA 2009, the mean Sciencescores in PISA 2015 increased by 2 points in State and Church schools and by 20points in Independent schools;

• Mathematics was a minor domain in the PISA 2015 study. 33 countries had a meanMathematics score higher than Malta (479), while 34 countries had a lower meanscore. Malta’s mean mathematics score was lower than the international average(490);

• Reading was a minor domain in the PISA 2015 study. 41 countries had a mean readingscore higher than Malta, while 28 countries had a lower mean score. The readingaverage score for Maltese students (447) was lower than the international average(493);

• Independent and Church schools are scoring higher than the State schools. Studentattainment in Science differs significantly between school types. Male and femalestudents attending church (509) and independent schools (540) are scoringsignificantly higher in Science, compared to the international average (493).Conversely, male and female students attending state schools (427) are scoringsignificantly lower. 11

TIMSSWhat is TIMSS?

The Trends in International Mathematics and Science Study (TIMSS) is a comparative

international study of mathematics and science achievement organised by the

International Association for the Evaluation of Educational Achievement (IEA). IEA is an

independent international cooperative of national research institutions and government

agencies that has been conducting studies of cross-national achievement since 1959.

TIMSS has the goal of helping countries make informed decisions about how to improve

teaching and learning in mathematics and science.

Reference: timssandpirls.bc.edu

12

Key findings for TIMSS 2015

• TIMSS 2015: Malta’s mean Science score (481) was lower than the internationalaverage (500) and was ranked 22nd place out of the 39 participating countries;

• Science attainment of Maltese students was comparable to students from United

Arab Emirates (477). It was found to be higher than 16 other countries including

Malaysia (471), Bahrain (466), Qatar (457), Iran (456), Thailand (456), Oman (455),

Chile (454), Georgia (443), Jordan (426), Kuwait (411), Lebanon (398), Saudi Arabia

(396), Morocco (393), Botswana (392), Egypt (371) and South Africa (358);

• The 21 countries that scored higher than Malta in Science included Singapore (597),

Japan (571), Chinese Taipei (569), Republic of Korea (556), Slovenia (551), Hong Kong

(546) Russian Federation (544), England (537), Kazakhstan (533), Ireland (530), United

States (530), Hungary (527), Canada (526), Sweden (522), Lithuania (519), New

Zealand (513), Australia (512), Norway (509), Israel (507), Italy (499) and Turkey (493);

13

Key findings for TIMSS 2015• On average, Independent school students (550.3 females, 556.4 males) scored higher

in Science than Church school students (519.3 females, 517.3 males) who in turn

scored higher than State school students (452.9 females, 434.3 males).;

• Malta’s mean Mathematics score (494) was 6 scale points lower than the

international average (500) and was ranked 20th out of the 39 participating countries;

• On average, Independent school students (541.3 females, 553.0 males) scored higher

in Mathematics than Church school students (518.3 females, 524.3 males) who in

turn scored higher than State school students (473.0 females, 458.1 males);

• In the TIMSS 2015 cycle, Maltese students scored higher than the 2007 cycle in all

content and cognitive domains. This improvement was achieved by both female and

male students.

14

PIRLSWhat is PIRLS?

The Progress in International Reading Literacy Study (PIRLS) is a comparative

international study of the reading attainment of ten-year-olds (Year 5 students). PIRLS is

organised by the International Association for the Evaluation of Educational Achievement

(IEA). Furthermore, it investigates reading literacy and the factors involved in acquiring

this skill.

Reference: timssandpirls.bc.edu

15

Key findings for PIRLS 2016

• PIRLS 2016: Malta’s mean reading score (452) was significantly lower thanthe international average (500) and was ranked 40th of 50 participatingcountries;

• Reading attainment of Maltese students was comparable to students fromUnited Arab Emirates (450) but was higher than nine countries includingBahrain (446), Qatar (442), Saudi Arabia (430), Iran (428), Oman (418),Kuwait (393), Morocco (358), Egypt (330) and South Africa (320);

• In all countries, female students scored higher in reading literacy than males.In Malta, female (463) students scored 21 scale points higher than males(442);

16

Key findings for PIRLS 2016• On average, Church school students (470.6) scored higher in reading than State

school students (447.5) who in turn scored higher than Independent school students(411.8) (*). Mean reading scores of female students exceed those of males by 26.2scale points in State schools, 15.4 scale points in Church schools and 5.2 scale pointsin Independent schools;

(*) There is societal bilingualism in Malta and this is reflected in the different schoolsectors in Malta. The first language of most students in State schools is Maltese. Thelanguage situation in Church schools is more varied. The first language of most studentsin Independent schools is English. This is made clear by the present PIRLS results whereMaltese was the test language. There was a significant decrease in the performance forstudents from Church schools, but especially for those from Independent schools. It isclear that the language of the test was a huge factor which determined the generalunderperformance of Maltese students on the PIRLS assessment. Source:curriculum.gov.mt (PIRLS 2016, Malta Report)

17

One-to-one interviews

18


19

Several one-to-one interviews were carried out with key players within our educationalsystem. The following are the salient points:

1. There are socio-economic differences between state schools, church schools andindependent schools;

2. Our system is very much exam oriented. If students fail the exams, they will most likelybe lost in the system. Hence, our students are not trained to focus on other tests whichare not their exams;

3. The items found in these tests (PISA, PIRLS, TIMSS) are not in sync with the knowledgeprovided to students;

4. Family background (incl. economic status) is still a dependent factor on the type ofschool (state, church and independent) and this affects test scores;

5. In most foreign countries, the above tests are given more importance, hence greaterawareness;

6. These tests are mainly based on reasoning and thinking skills. This is an issue for Maltaas our system is still exam oriented;


20

7. Our educational system is not based on evidence and research;

8. In order for our ratings to improve we need to put less emphasis on coaching and moreemphasis on reasoning and thinking skills. Decisions need to be taken based on evidence;

9. The vision of the different categories of schools towards these tests is different.

10. Perceptions about PISA, PRILS and TIMSS: Teaching in the Independent schools incorporatesthe same mind set of these assessments.

11. Science teaching: For the independent schools science is a core subject involving around 4lessons per week. This is completely different when compared to other schools (a lowernumber of science lessons);

12. More focus needs to be put on primary schooling if we want to improve Malta’s ranking.

Teachers are not confident enough when teaching science at a primary level;

13. More focus needs to be placed on the science subjects, especially at primary level.

The quantitative study

21

Methodology22

• The survey was carried out online. Educators (Learning Support Educators

(LSEs), Kindergarten Educators (KGEs) Teachers, Head of Departments (HoDs),

Assistant Heads (AHs), Heads of Schools (HoSs) and Education Officers (EOs))

were approached several times through e-mail by inviting them to participate

in this research study;

• A sample size of 475 individuals was collected amongst the educators;

• Level of confidence: 95%;

• Confidence interval: +/- 4.3%;

• The data was collected between the 12th October and the 26th October 2018.

Methodology and Demographics

23

Results – Familiarity with PISA,PIRLS

and TIMSS24

Results – Familiarity with PISA,PIRLS and TIMSS

25

• Most of the respondents heard about these assessments. PISA is the most familiar

assessment amongst the educators (86.6%) while PIRLS is the least familiar (73.6%)

86.6%

13.4%

Have you heard about PISA?

73.6%

26.4%

Have you heard about PIRLS?

84.1%

15.9%

Have you heard about TIMSS?

Yes

No


26

• Result 3.1 by school type

85.5% 87.9% 85.7%

14.5% 12.1% 14.3%

0%

20%

40%

60%

80%

100%

State Church Independent


78.0%

64.8%53.6%

22.0%

35.2%46.4%

0%

20%

40%

60%

80%

100%



85.3% 83.7%

69.0%

14.7% 16.3%

31.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%



Yes

No


27

• Result 3.1 by occupation

50.0%

55.0%

10.1%

0.0%

0.0%

0.0%

3.1%

50.0%

45.0%

89.9%

100.0%

100.0%

100.0%

96.9%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Learning Support Educator (LSE)

Kindergarten Educator (KGE)

Teacher

Education Officer (EO)

Assistant Head (AH)

Head of School (HoS)

Head of Department (HoD)


Yes


28


61.4%

57.9%

28.8%

0.0%

2.3%

0.0%

18.2%

38.6%

42.1%

71.2%

100.0%

97.7%

100.0%

81.8%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%



Teacher


Assistant Head (AH)




Yes

No


29


56.5%

55.6%

12.6%

0.0%

0.0%

0.0%

9.1%

43.5%

44.4%

87.4%

100.0%

100.0%

100.0%

90.9%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%



Teacher


Assistant Head (AH)




Yes

No


30

• When asked about the level of knowledge regarding these assessments, 26.9% have

some knowledge about PISA, while 33.3% are not knowledgeable at all about PIRLS.

Only 27.6% are knowledgeable or very knowledgeable about PISA and only 22.1% are

knowledge or very knowledgeable about PIRLS.

7.2%

20.4%

26.9%

25.2%

20.4%

0% 5% 10% 15% 20% 25% 30%

Very knowledgeable

Knowledgeable

Some knowledge

Slightly knowledgeable

Not knowledgeable at all

Level of knowledge (PISA)

5.0%

17.1%

25.4%

19.3%

33.3%

0% 5% 10% 15% 20% 25% 30% 35%

Very knowledgeable

Knowledgeable

Some knowledge



Level of knowledge (PIRLS)


31

• 25.2% of the respondents are knowledgeable about TIMSS; however 67% are either

‘Not knowledgeable at all’ or ‘Slightly knowledgeable’ or have ‘Some knowledge’.

• The mean and median of the level of knowledge of these assessments is also

illustrated in the table below.

7.8%

25.2%

24.8%

20.7%

21.5%

0% 5% 10% 15% 20% 25% 30%

Very knowledgeable

Knowledgeable

Some knowledge



Level of knowledge (TIMSS)

Mean/Median

PISA Some knowledge

PIRLS Slightly knowledgeable

TIMSS Some knowledge


32

• The mean and median of the level of knowledge of these assessments by school

type and occupation are illustrated in the tables below.


Mean Median Mean Median Mean Median

PISA 3 3 3 3 3 3

PIRLS 2 3 2 2 2 1

TIMSS 3 3 3 3 3 3

PISA HoD HoS AH EO Teacher KGE LSE

Mean 3 3 3 4 3 2 2

Median 3 3 4 4 3 1 2

PIRLS HoD HoS AH EO Teacher KGE LSE

Mean 3 4 3 4 2 2 2

Median 3 4 3 4 2 1 1

TIMSS HoD HoS AH EO Teacher KGE LSE

Mean 3 4 4 4 3 2 2

Median 3 4 4 4 3 1 1

1:Not knowledgeable at all

2:Slightly knowledgeable

3:Some knowledge

4:Knowledgeable

5:Very knowledgeable

Results – Performance of the three

assessments33


34

• More than half of the interviewed educators think that the Maltese students

performed below average in all three assessments when compared to other

countries.

56.0%

41.4%

2.6%

Performance of Maltese students (PISA)

57.1%

40.6%

2.4%

Performance of Maltese students (PIRLS)

59.3%

37.7%

3.0%

Performance of Maltese students (TIMSS)

Below average

Average

Above average


35

• Result 3.2 by school type.

• Respondents working in independent schools were more optimistic and think that the

Maltese students’ performance was of an average performance.

58.1%54.4%

48.1%

39.7% 41.2%

51.9%

2.2% 4.4%0.0%

0%

10%

20%

30%

40%

50%

60%

70%



59.9%53.8%

44.4%37.6%

43.4%

55.6%

2.5% 2.8% 0.0%0%

10%

20%

30%

40%

50%

60%

70%



61.8%56.6%

48.1%

35.3%38.9%

51.9%

2.8% 4.4%0.0%

0%

10%

20%

30%

40%

50%

60%

70%



Below average

Average

Above average


36


24.3%

20.0%

56.9%

92.9%

64.1%

69.2%

66.7%

70.3%

80.0%

39.9%

7.1%

33.3%

30.8%

33.3%

5.4%

0.0%

3.2%

0.0%

2.6%

0.0%

0.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%



Teacher


Assistant Head (AH)




Above average

Average

Below average


37


32.4%

15.4%

56.8%

85.7%

67.5%

76.9%

64.5%

64.9%

76.9%

40.0%

14.3%

32.5%

23.1%

35.5%

2.7%

7.7%

3.2%

0.0%

0.0%

0.0%

0.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%



Teacher


Assistant Head (AH)




Above average

Average

Below average


38


32.5%

15.4%

60.9%

92.9%

65.0%

80.8%

60.6%

57.5%

84.6%

35.5%

7.1%

35.0%

19.2%

39.4%

10.0%

0.0%

3.5%

0.0%

0.0%

0.0%

0.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%



Teacher


Assistant Head (AH)




Above average

Average

Below average

Results – Involvement in the three

assessments39

Results – Involvement in the three assessments

40

• Very few of the respondents were involved in these three assessments. 16.8% of the

educators were involved in PISA while only 8.8% were involved in PIRLS.

16.8%

8.8%

15.8%

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

PISA PIRLS TIMSS

Involvment in these assessments

16.6%

18.5%

20.7%

9.1% 8.9%

6.9%

15.3%

17.7% 17.2%

0%

5%

10%

15%

20%

25%


Involvment in these assessments (by school type)

PISA

PIRLS

TIMSS

Results – Your workplace and the

three assessments – level of

importance41

Results – Your workplace and the three assessments – level of

importance

42

• Only 4.8% (PISA), 2.7% (PIRLS) and 3.8% (TIMSS) of the educators said that a high level of

importance is given by their workplace to these three assessments respectively. The

majority said that their workplace gives a ‘Neutral’ level of importance (38.3%, 43.5% and

37.6% respectively.)

4.8%

20.7%

38.3%

19.8%

16.3%

0% 10% 20% 30% 40% 50%

Very high level of importance

Good level of importance

Neutral

Low importance

No importance at all

Level of importance (PISA)

2.7%

17.5%

43.5%

16.6%

19.7%

0% 10% 20% 30% 40% 50%



Neutral

Low importance


Level of importance (PIRLS)


importance

43

• The mean and median of the Likert scale is at ‘Neutral’ level of importance for all the

three assessments. When analysing the Likert scale by school type, all the mean and

medians for all the assessments are ‘Neutral’; however educators from the independent

schools rated the level of importance by their workplace at a higher level (E.g. PISA:

Independent (Mean=3.1), Church (Mean=2.9) and State (Mean=2.7)).

3.8%

25.2%

37.6%

17.1%

16.2%

0% 10% 20% 30% 40%



Neutral

Low importance


Level of importance (TIMSS)

Mean/Median

PISA Neutral

PIRLS Neutral

TIMSS Neutral



preparation (academic staff)44


preparation (academic staff)

45

• Around a third (33.6%, 36.8% and 32.9%) of the respondents said that their workplace level

of preparation for the academic staff for such assessments is ‘Neutral’. On the other hand,

a considerable amount (31.0%, 30.0% and 27.3%) of the respondents said that there is ‘no

preparation at all’ for these assessments.

0.9%

14.2%

33.6%

20.4%

31.0%

0% 5% 10% 15% 20% 25% 30% 35% 40%

Very high level of preparation

Good level of preparation

Neutral

Low preparation

No preparation at all

Level of preperation for academic staff (PISA)

1.3%

12.1%

36.8%

19.7%

30.0%

0% 5% 10% 15% 20% 25% 30% 35% 40%



Neutral

Low preparation


Level of preperation for academic staff (PIRLS)



46

• When calculating the mean for the latter question, this resulted in a ‘low preparation’

rating for all assessments while the median for PISA resulted in a ‘Low preparation’ rating

and for PIRLS and TIMSS the result was ‘Neutral’.

Level of preparation for academic staff

Mean Median

PISA Low preparation Low preparation

PIRLS Low preparation Neutral

TIMSS Low preparation Neutral

3.0%

16.5%

32.9%

20.3%

27.3%

0% 5% 10% 15% 20% 25% 30% 35%



Neutral

Low preparation


Level of preperation for academic staff (TIMSS)



Level of preparation for academic staff



PISALow

preparation

Low

preparation

Low

preparation

Low

preparationNeutral Neutral

PIRLSLow

preparationNeutral

Low

preparation

Low


TIMSS Neutral NeutralLow

preparation

Low


47

• The mean and median of the Likert scale by school type is illustrated below. It is evident

that the educators at the independent schools claimed that their workplace prepare

more their academic staff for these assessments.



preparation (students)48


preparation (students)

49

• 38.0%, 36.5% and 34.3% of the respondents rated their workplace level of preparation for

the students for such assessments as ‘Neutral’. On the other hand, a considerable amount

of the respondents (30.8%, 31.5% and 30.0%) said that there is ‘no preparation at all’ for

these assessments.

0.9%

10.0%

38.0%

20.4%

30.8%

0% 5% 10% 15% 20% 25% 30% 35% 40%



Neutral

Low preparation


Level of preperation for students (PISA)

1.4%

10.0%

36.5%

20.5%

31.5%

0% 5% 10% 15% 20% 25% 30% 35% 40%



Neutral

Low preparation


Level of preperation for academic staff (PIRLS)



50

• When calculating the mean for the latter question, this resulted in a ‘Low preparation’ for

all assessments while the median for PISA and PIRLS resulted in a ‘Low preparation’ rating

and a ‘Neutral’ rating for TIMSS.

Level of preparation for students

Mean Median

PISA Low preparation Low preparation

PIRLS Low preparation Low preparation

TIMSS Low preparation Neutral

2.2%

13.5%

34.3%

20.0%

30.0%

0% 5% 10% 15% 20% 25% 30% 35% 40%



Neutral

Low preparation


Level of preperation for academic staff (TIMSS)



Level of preparation for students



PISALow

preparation

Low

preparation

Low

preparation

Low


PIRLSLow

preparation

Low

preparation

Low

preparation

Low


TIMSSLow

preparationNeutral

Low

preparation

Low


51

• The mean and median of the Likert scale by school type is illustrated below. Similar to

the prior findings, it is evident that the educators at the independent schools claimed

that their workplace prepares their students more for these assessments.

Results – Opinions about the three

assessments52


53

• The majority of the respondents have a “neutral” opinion or said “it is just a test” for all

the three assessments when asked ‘What is your opinion about these assessments?’

15.3%

35.3% 35.3%

14.0%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Positive Neutral It is just a test Negative

Opinion about PISA

12.9%

37.5%36.2%

13.4%

0%

5%

10%

15%

20%

25%

30%

35%

40%


Opinion about PIRLS


54

16.0%

34.9% 34.9%

14.3%

0%

5%

10%

15%

20%

25%

30%

35%

40%


Opinion about TIMSS


55


13.7%

17.0%

28.6%

37.3%

34.0%

28.6%

37.3%

32.1%

28.6%

11.8%

17.0%

14.3%

0%

5%

10%

15%

20%

25%

30%

35%

40%


Opinion about PISA (by school type)

12.5%

15.7%

7.1%

39.4%

31.4%

50.0%

37.5% 37.3%

21.4%

10.6%

15.7%

21.4%

0%

10%

20%

30%

40%

50%

60%


Opinion about PIRLS (by school type)

Positive

Neutral

It is just a test

Negative


56

• Result 3.5 by school type

14.5%

20.8%

14.3%

37.0%

28.3%

42.9%

35.8% 35.8%

21.4%

12.7%

15.1%

21.4%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%


Opinion about TIMSS (by school type)

Positive

Neutral

It is just a test

Negative

Results – The level of students’ knowledge in

relation to these assessments57

Results – Do you think these assessments capture what

students know about the topics and provide a true picture?

58

• Only a fraction of the respondents (6.0%, 5.7% and 5.9%) said that these assessments provide

a true picture of the students’ knowledge. Many of the respondents (33.6%, 30.4% and

35.9%) said that only a small part of the content is relevant. There was a significant amount

of respondents (25.0%, 33.9% and 25.7%) who were unsure about the latter question.

6.0%

12.9%

33.6%

22.4%25.0%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Yes Most of thecontent

Small partsof the

content

No I don't know

Does PISA capture what students know about the subject and provide a true

picture?

5.7%

10.0%

30.4%

20.0%

33.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%


Small partsof the

content

No I don't know

Does PIRLS capture what students know about the subject and provide a true

picture?



59

5.9%

11.4%

35.9%

21.1%

25.7%

0%

5%

10%

15%

20%

25%

30%

35%

40%


Small partsof the

content

No I don't know

Does TIMSS capture what students know about the subject and provide a true

picture?



60


• Educators at the Independent schools selected the option ‘Most of the content’ more often

for the ‘PISA’ assessment.

5.6%7.8% 7.1%

11.9%13.7%

21.4%

31.9%

41.2%

28.6%

23.1%

15.7%

28.6%27.5%

21.6%

14.3%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


Does PISA capture what students know about the subject and provide a true picture? (by

school type)

6.3% 6.1%

0.0%

12.5%

4.1%

0.0%

31.3%32.7%

21.4%

18.8%

16.3%

35.7%

31.3%

40.8%42.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


Does PIRLS capture what students know about the subject and provide a true picture? (by school type)

Yes

Most of the content

Small parts of thecontent

No

I don't know



61


6.7% 5.8%

0.0%

12.1%

9.6%

7.1%

36.4%

40.4%

21.4%20.6%19.2%

28.6%

24.2% 25.0%

42.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


Does TIMSS capture what students know about the subject and provide a true picture? (by school type)

Yes

Most of the content

Small parts of thecontent

No

I don't know

Results – The three assessments and the

Maltese syllabus62


63

• 44.9% (PISA), 48.2% (PIRLS) and 38.9% (TIMSS) of the respondents do not know if the

content of these assessments is in line with the Maltese syllabi. Very few of the

respondents said that it is in line with the Maltese syllabi (2.2%, 1.3% and 0.9%).

2.2%

14.1%

20.3%18.5%

44.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%


Small partsof the

content

No I don't know

The content is in line with the syllabi? (PISA)

1.3%

13.4%

21.0%

16.1%

48.2%

0%

10%

20%

30%

40%

50%

60%


Small partsof the

content

No I don't know

The content is in line with the syllabi? (PIRLS)


64

0.9%

17.9%

24.8%

17.5%

38.9%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%


Small partsof the

content

No I don't know

The content is in line with the syllabi? (TIMSS)

• Throughout this survey, it is evident that the number of responses ‘I don’t know’ is high.

This shows the lack of knowledge that exists amongst the Maltese educators regarding

these three assessments.


65


1.9% 2.0%

7.1%13.2%

19.6%

7.1%

17.0%

27.5%

35.7%

21.4%

9.8%

14.3%

46.5%

41.2%

35.7%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%


The content is in line with the syllabi? (PISA) by school type

1.9%0.0% 0.0%

13.9% 14.3%

7.1%

21.5% 22.4%

14.3%16.5%

12.2%

21.4%

46.2%

51.0%

57.1%

0%

10%

20%

30%

40%

50%

60%


The content is in line with the syllabi? (PIRLS) by school type

Yes

Most of thecontentSmall parts ofthe contentNo

I don't know


66


1.2%0.0% 0.0%

17.0%

25.0%

7.1%

24.2%

28.8%

21.4%18.8%

13.5% 14.3%

38.8%

32.7%

57.1%

0%

10%

20%

30%

40%

50%

60%


The content is in line with the syllabi? (TIMSS) by school type

Yes

Most of the content

Small parts of the content

No

I don't know

Results – The three assessments and the

language impact67

Results – The three assessments and the language impact

68

• A considerable amount of the respondents (61.4%) said that the language of the

assessments has an impact on the Maltese results. Only 5.7% think that the language is not

affecting the result.

• Those working in an independent school (78.6%) are the most in favour that the language is

affecting the results.

9.3%

23.6%

5.7%

61.4%

0% 10% 20% 30% 40% 50% 60% 70%

I don't know

Maybe

No

Yes

Does the language of the assessment impact on the results?

60.2% 59.3%

78.6%

5.3%7.4% 7.1%

25.1% 24.1%

7.1%9.4% 9.3% 7.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%


Does the language of the assessment impact on the results? (by school type)

Yes

No

Maybe

I don'tknow

Results – Design and implementation of the

assessments and improvement of the Maltese results69

Results – Design and implementation of the assessments and

improvement of the Maltese results

70

• Below are the opinions of the respondents when they were asked ‘In what ways can these

assessments be better designed and implemented?’

• 23.5% of the respondents think that such assessments need to be more in line with the

Maltese syllabi.

4.5%

5.3%

6.8%

6.8%

6.8%

14.4%

15.2%

16.7%

23.5%

0% 5% 10% 15% 20% 25%

Better timing to conduct these assessmnents

Incentives and motivation for the students

More student friendly

More training to the educators

The assessments are very long

Others

Do not know

Choice of the assessment's language

To be more related to our syllabi

Better ways for design and implementation of such assessments

Results – Design and implementation of the assessments and

improvement of the Maltese results

71

• There were different suggestions on how the Maltese results can be improved. 15.9% of the

respondents think that the Maltese students have to be better prepared for such

assessments. Less routine learning for students was another valid suggestion on how to

improve the results.

2.2%

2.9%

7.2%

8.0%

8.0%

10.9%

11.6%

15.9%

33.3%

0% 5% 10% 15% 20% 25% 30% 35%

Try to be more inline with these assessment

More importance to science subjects

Choice of the assessment's language

More training to the educators

To be more related to our syllabi

Less rote learning

Do not know

Better preparation for students for these assessments

Others

How to improve Maltese results

Conclusion of the quantitative

study72

Conclusion

❖ The majority of the respondents heard about PISA, PIRLS and TIMSS.

❖ Very few of the respondents are ‘very knowledgeable’ about these assessments

(7.2% (PISA), 5.0% (PIRLS) and 7.8% (TIMSS)).

❖ Most of the interviewees think that the performance of the Maltese students is

below average in these assessments.

❖ 16.8% of the educators were involved in PISA, 15.8% in TIMSS while only 8.8% were

involved in PIRLS.

❖ A ‘Neutral’ level of importance was given for the three assessments by the

educators.

❖ A considerable amount of the respondents (30.8% (PISA), 31.5% (PIRLS) and 30.0%

(TIMSS)) said that there is ‘no preparation at all’ for these assessments.

73

Conclusion

❖ Many of the respondents have a ‘neutral’ opinion or said ‘it is just a test’ for all

the three assessments.

❖ Only a fraction of the respondents (6.0% (PISA), 5.7% (PIRLS) and 5.9% (TIMSS)) said

that these assessments provide a true picture of the students’ knowledge.

❖ Very few of the respondents said that that these assessments are in line with the

Maltese syllabi (2.2% (PISA), 1.3% (PIRLS) and 0.9% (TIMSS)).

❖ Only 5.7% of the respondents think that the language is not affecting the result.

❖ The independent schools are more positive about the fact that their workplace is

giving ‘enough importance to these assessments’, that their workplace is

‘preparing the academic staff for these assessments’ and that their workplace is

‘preparing the students for these assessments’.

74

The [email protected]

The role of TIMSS, PIRLS & PISA in Providing a True ... · skills and knowledge of 15-year-old...

Documents

Transcript of The role of TIMSS, PIRLS & PISA in Providing a True ... · skills and knowledge of 15-year-old...