TIMSS - edu.gov.qa · International Assessment Section 2018 - 2019 TIMSS Guidelines for Rules &...

International Assessment Section 2018 - 2019

TIMSS

Guidelines for Rules &

Procedures to be followed

during the Implementation of

International Tests in Qatar

Trends in International

Mathematics & Science Study

TIMSS

This Manual was prepared by International Assessment

Section experts.

Issued in 2018-2019

Guidelines to the Processes and

Controls for the application of

international assessments in

Qatar



TIMSS

2 International Assessment Section 2018 - 2019

Contents

Introduction……………………………………………………………………3

General objectives of the assessment processes and controls…………………4

Roles and Responsibilities……………………………………………………..5

Assessment Materials…………………………………………………………..6

Assessment Forms…………………………………………………….………..7

Students Excluded from the Assessment…………………..…………………..8

General Procedures……………………………………………………………..9

Main Assessment Session……………………………………………………..10

Make-Up Session……………………………………………………………...11

Students Participation …………..…………………………………………….12

Confidentiality of Materials………………….………………………………..13

Glossary……………………………………………………………………….14

Appendices………………………………..………………………………….15


Introduction

Evaluation has great importance in teaching and learning, as it is a tool that

will determine the degree of achievement of the educational goals of

students through their educational achievement, Thus measuring the

quality of the educational systems and correct their trend. Recently, the

focus has been on the techniques used to design assessments and their role

in the evaluation process.

Given the world's accelerated scientific and technological sciences, and

interest in computing education (e-learning) and the use of information

technology, Qatar has been keen to participate in the international

Assessment- Trends in International Mathematics and Science Study

(TIMSS)1 - in it’s electronic version, which is Supervised by the

International Association for Evaluation of Educational Achievement

(IEA)2.

The IEA has recently moved to conduct future assessments in an

interactive way across computers or tablets. In addition to the elements

normally found in the paper and pen assessments. Providing an attractive

assessment experience to motivate students through color, animation and

dynamism.

This manual contributes in supporting the efforts of International

Assessment Section3 teams, In the context of activating procedures

related to the international assessment field implementation, which is held

to a high degree of professionalism, starting from the preparation stage, to

the end of field implementation stage.


General objectives of the assessment Processes and controls

This manual includes procedures, policies and detailed information

required so that the School Coordinator4 and Test Administrators5 can

refer to them to achieve the following objectives: to be followed during

the Implementation of International Tests in Qatar

1- Compliance with the requirements of the international organizations for test implementation.

2. Standardize the testing environment for all participants.

3. Responsibility for following all the restrictions.

4. Improving quality of tests' results.


Roles & Responsibilities

To activate the general objectives of the assessment, the roles and

responsibilities of the team involved in the assessment implementation

should be distributed according to the following:

1- IEA role:

• Provide all materials required for the assessment implementation, and identify the tasks to be accomplished according to a specific timeline.

• Verify the information and data provided by the country.

• Analyzing students' results and issuing international reports.

2. Role of the International Assessment Section:

• Preparing plans and programs that will help the implementation of the assessment with highest quality, according to the specifications requested by the organization.

• Downloading all test materials from the IEA's website, to translate and adapt all materials according to the standards specified.

• Conducting the electronic scoring of students' responses, and start data entry using a software provided by the organization.

• Reconcile and clean the data before submission.

• Issuing schools' reports and national report after issuing of international reports by the organization.

3. Role of Paricipated Schools:

• Preparing and familirise students and teaching staff.

• Provide us with all assessment requirements, such as students' & teachers' information, test rooms specifications, no. of computers available, and other requirements that comply with the organization's rules and restrictions.

• Conducting the assessment according to the controls and processes, with supervision of the International Assessment Section.


Assessment Materials

Participating countries contribute in the preparation of the assessment framework,

and formulation of its content with the IEA experts. Each country will be

responsible for translating and adapting the following contents of the assessment

materials6:

Assessment software:

An electronic assessment that contains equivalent math and science questions

(70% of the multiple-choice type, and 30% questions with short answers based

on conclusion), they will be assigned to students randomly through a software.

Student Questionnaire:

A paper-based questionnaire, that includes information about the family and

academic background of the students, their attitudes, ambitions, and

classroom practices for mathematics and science teachers from their point of

view.

School questionnaire:

An electronic questionnaire directed to school principals, that includes

information about the school environment, teaching staff, students, curriculum,

study programs, resources, employee development programs and school’s

relationships with the community.

Teacher Questionnaire:

An electronic questionnaire directed to mathematics and science teachers. It

is related about their scientific and academic backgrounds, teaching practices

and teaching methods for the students participating in the test.

Early Learning Survey:

A paper questionnaire, directed to the student's guardian. It is related to

information about pre-primary education stages and the integrative

relationship between home and school.


Assessment Forms

The International Assessment Section should conduct several procedures in

cooperation with schools to issue assessment forms7, to facilitate the field

implementation as follows:

1. Student Listing Form:

2. Student Teacher Linkage Form:

3. Student Tracking Form:

4. Teacher Tracking Form:

5. Session Report Form: 6. Calculating Response Rate Form:

The form is sent to schools to be filled with

information about all the target classes such as

students, and teacher’s names, class name, and

educational branch. Appendix 1

After completing the Student Listing Form by

schools and selecting the participating classes by

the International Assessment Section, this form is

extracted and sent to schools to be filled with

names of students and teachers of mathematics

and science in these classes, date of birth, gender

of students and exclusions. Appendix 2

Include names of students in the participating

classes (who were listed in the Student Teacher

Linkage Form). And test instruments assigned to

them. It is used by the Test Administrator during the

assessment session, to ensure that students got the

correct assessment instrument assigned to them

(assessment/questionnaire). And to indicate the status

of participation of students in these classes. Appendix3

Includes a list of students' teachers who have

been selected in the Student Tracking Form, and

the subject taught by these teachers. It is used by

the school coordinator to indicate if the questionnaire

is completed by the participating teachers. Appendix4

This Form is used to document assessment timing8.

Appendix 5

And recording notes on any problems or conditions

that occurred during the assessment session.

Appendix6

The form determines whether the school needs a

make-up session, in the case of many students

were absent from the main session in each

participating class, and response rate should not

be less than 90%. Appendix 7


Students Excluded from the Assessment

The test is conducted under unified environmental conditions based on the principle of

equal opportunity for all students. To achieve the principle of equity among students,

students with special needs are excluded, as their school need to provide them special

preparations to participate, or provide separate assessment rooms and additional time,

or by adapting the assessment to suit their level of competence or complete it with the

help of one of the staff member at the school.

Therefore, the school coordinator, should note the existence of such cases when he is

filling out the Student Teacher Linkage Form.

Special needs students are classified as follows:

Students with functional disabilities

These are students who have physical disabilities in such a way

that they cannot perform in the TIMSS testing situation. and need

special accommodation .

Students with intellectual disabilities:

This includes students who are emotionally or mentally unable to

follow even the general instructions of the test. Students should not be excluded solely because of poor academic performance or normal

disciplinary problems.

Non-native language speakers:

These are students who are unable to read or speak the language(s) of the assessment and would be unable to overcome the language barrier, A student who has received less than

six months of instruction in the assessment language should be

excluded.


General Procedures

The success of the assessment implementation is due to several factors, the most

important one is the cooperation between the International Assessment Section and the

other participating parties, this can be achieved through the following standardized

implementation procedures and polices listed in the School Coordinator's Manual9 and the

Test Administrator Manual10 in details. The general procedures are:

1. Checking the readiness of school computers to ensure their ability to run the test

software.

2. Test Administrators nomination who can administer the assessment, providing

that they are not teaching test subjects (mathematics and science).

3. Nominating an IT technician to manage any technical problems may occur

during assessment implementation.

4. Attending the seminar about the procedure of conducting the assessment

according to the organization’s standards.

5. Scheduling test dates, considering holidays or any other activities in the school.

6. Receiving, inspecting and securing assessment materials.

7. Test Administrator is not allowed to answer any questions about the content of

the assessment, or provide any specific information, or instructions about any

item.

8. Students will not be provided with calculators or other tools (ruler, protractor,

etc.), these tools automatically appear on the screen with questions - in the test

software, if needed.

9. Electronic devices (such as mobile phones, laptops, or Video cameras) are not

allowed during assessment sessions.

10. Recording notes about any problems encountered while administering the

assessment in the provided form.

11. Returning all test materials to the International Assessment Section.


Main Assessment Session

1. Prepare rooms (according to selected classes in each school) following to the organization’s

specifications.

2. Make sure that each student is seated in the correct place according to the student tracking form,

before log in the student into the assessment software.

3. Internet connection is necessary to upload students' answers after the ending the sessions.

Before assessment session:

1. Administer the main session11 based on the rules and procedures listed in the Test Administrator

manual.

2. Administer the student questionnaire session directly after the test session.

3. Track timing of the sessions accurately using the timer, and record the time for the sessions

(preparation, beginning of part1, break,…etc) in the Test Administration Form;

4. Complete the forms.

5. Follow up the completion of the school questionnaire and the teacher questionnaire.

6. Count the number of absent students, to determine the date of the makeup session12 (if needed).

During assessment session:

1. Completion of all the session administration forms by the Test Administrator, and then deliver

them to the School Coordinator.

2. It is necessary to state whether the school was visited by the Tnternational13 or National14 Quality

Control Monitor in the session administration form.

3. Calculate the Student Response Rate Form, and arrange for makeup session (if needed).

4. Return assessment materials to the International Assessment Section.

After assessment session:


Make-Up Assessment Session

In case of students absence in more than one section:

one makeup session can be held for all school students who were absent during the main session,

in one test room.

Only students of the originally sampled class and who are listed in the Student Tracking Form should be tested. Replacing students is not

allowed.

Note that there are specific rates determined by the International Organization for students absence within each school, which will

affect the analysis of the country resutls. If this percentage exceeds the rate, Qatar will be excluded from the international reports.


Students Participation

Students are logged in and begin answering with the rest of the students.

If the student arrives late, before starting to read the instructions:

1

The student should not be allowed to enter the test room.If the student arrived late and the instructions were read and the first session of the assessment began:

2

Can participate in the second session of the assessment, by logging on to student data and registering the access password on the second part of the assessment.

If the student arrives during the break after the first session and before the second session has begun:

3

He can not enter the quesstionaire session, and he must attend the make-up assessment session (if any).

In the case of absence of the student from the assessment main session:

4

This is done in the presence of the School Coordinator. The Test Administrator keeps the student's computer after the log-out process and records the reason for the student leaving the session in the session Administration Form. At the end of the session, the Test Administrator will reopen the student's computer and submitt his answers with the answers of the rest of the students.

Only in emergency cases (for example, if sick) is it permitted to leave the testing room:

5

He can continue to answer questions in the time remaining from the assessment session.

If a student leaves the session and returns:

6


Confidentiality of Materials

The school is responsible for the safety, and confidentiality of assessment materials in the

school, starting from the moment it is received from the International Assessment

Section until delivery. In addition, the following required:

1. Test parties should sign the

confidentiality agreement sent by the

International Assessment Section.

-2 Keep test materials safely and under

observation throughout the testing

period.

3-Maintaining the confidentiality of all the information, such as passwords, and it

shouldn't be disclosed to any unauthorized

person.

4. photocopying and copying the test

software is prohibited, in addition to taking

pictures of the computer screen, or discussing any of the assessment items or

questions.


Glossary

Term Definition

1



{TIMSS}

It is an International assessment that explores skills of students in the

fourth and eighth grades in science and mathematics. It aims at focusing

on educational policies and systems, studying (evaluating) the

effectiveness of the applied curricula, teaching methods and its practical

application.

2

International Association for

Evaluation of Educational

Achievement (IEA):

It is the organization responsible for the assessment, where it supervises

the assessment implementation centrally in cooperation with

participating countries. Based in Amsterdam / The Netherlands.

3 International Assessment

Section:

It is one of the sections of the Department of Students Assessment in

the Evaluation Institute at the Ministry of Education and Higher

Education, which was established in recognition of the importance of

the international assessments, and to achieve the high quality of its

implementation in accordance with international standards.

4 School Coordinator:

One of the school staff who is nominated by the school principal, and he

is the link between the International Assessment Section and the school.

He cooperates in providing the section with all the requirements &

information for the assessment.

5 Test Administrator

One of the school staff who is nominated by the school principal. He

will administer the assessment session, and he should not be a

mathematics or science teacher.

6 Assessment Materials: The electronic test and the questionnaires.

7 Assessment Forms: Guidance information to be completed in sequential steps, to simplify

the implementation of field procedures.

8 Assessment Timing: A table showing the time for each assessment session and breaks.

9 School Coordinator Manual:

Defines the main procedures that the School Coordinator must follow,

in logical sequence for each stage, to ensure the highest quality and

accuracy of assessment implementation.

10 Test Administrator Manual:

Determines the main procedures that the Test Administrator must

follow, in logical sequence for each stage, to ensure the highest quality

and accuracy of assessment implementation.


Glossary

Term Definition

11 Main Session:

Is the session determined for the students present to complete the

assessment and the questionnaire, and takes 3 hours with short breaks in

between.

12 Make- up Session: Is the session determined for the students who were absent from the

main session.

13 International Quality Control

Monitor:

An employee who is nominated by the (IEA), who is responsible of

visiting schools to supervise testing procedures within test rooms

according to standard controls.

14 National Quality Control

Monitor (Observer):

An employee who is nominated by the International Assessment

Section (IAS), who visits schools to supervise the testing procedures

within test rooms according to standard controls.


❑ You can visit the following websites for more information:

1. http://www.edu.gov.qa/Ar/SECInstitutes/EvaluationInstitute/StudentAssessmentOffice/InternationalStudies/TIMSS/Pages/default.aspx

2. http://timssandpirls.bc.edu/ 3. http://www.iea.nl/

http://www.edu.gov.qa/Ar/SECInstitutes/EvaluationInstitute/StudentAssessmentOffice/InternationalStudies/TIMSS/Pages/default.aspx

http://www.edu.gov.qa/Ar/SECInstitutes/EvaluationInstitute/StudentAssessmentOffice/InternationalStudies/TIMSS/Pages/default.aspx

http://timssandpirls.bc.edu/

http://www.iea.nl/


Appendices


Appendix (1)

Class Listing Form

eTIMSS - Class Listing Form - [Grade 4 or 8]

TIMSS Participant Country Qatar

School Name The Gulf Privet

School ID 7001

School Coordinator Name Noor Khan

Phone Number (553*****)

Email @Education.gov.qa Or @hotmail.com

Class Name Grade Class Group

Number of

Students

Class Exclusion

Status

Name of Mathematics

Teacher

Name of Science

Teacher

A 4 30 Mr Ali said Mr Jason Abi

B 4 28 Ms Josh Rafi Mr Alen Zamir

<_list_end_> Class Group (column 3): Class groups occur when students are assigned to specific classes based on their ability/prior achievement. Class Exclusion Status (column 5): As a rule, all classes are to be included. Examples of class-level exclusions include classes where all students belong to at least one of the following three exclusion status categories: 1 = Students with functional disabilities; 2 = Students with intellectual disabilities; 3 = Non-native language speakers. If all students in the excluded class do not belong to the same exclusion category, please identify the category corresponding to the majority of students. All class-level exclusions must be approved by the national center. Name of Mathematics Teacher (column 6): Name of the teacher teaching mathematics content to the class. Name of Science Teacher (column 7): Name of the teacher teaching science content to the class.

mailto:[email protected]

mailto:****@Education.gov.qa

mailto:****@hotmail.com


Appendix (2)

Student-Teacher Linkage Form

eTIMSS - Student-Teacher Linkage Form - [Grade 4 or 8]

School Name:

TIMSS Participant Country:

The Gulf Privet

Qatar

[a] [b] [c] [d]

School ID Class ID Class Name Grade

7001 700101 A 4

Student Name and Number

Seq

uen

ce N

um

ber

Date

of

Bir

th

Gen

der

Exclu

sio

n S

tatu

s

Teacher Information

No

Math

em

ati

cs T

each

er

No

Scie

nce T

each

er

Mr A

li s

aid

Math

em

ati

cs

1

Mr J

aso

n A

bi

Scie

nce

6

Teach

er’s

Na

me:

Cla

ss (

Co

urse) N

am

e:

Su

bje

ct

Co

de:

Teach

er’s

Na

me:

Cla

ss (

Co

urse) N

am

e:

Su

bje

ct

Co

de:

Teach

er’s

Na

me:

Cla

ss (

Co

urse) N

am

e:

Su

bje

ct

Co

de:

Teach

er’s

Na

me:

Cla

ss (

Co

urse) N

am

e:

Su

bje

ct

Co

de:

D

D MM YYYY

Bareerah Asad-(ID Number) 1 15 09 2007 2 × ×

Teeba Abdur-Rahman-(ID Number) 2 31 07 2008 1 × ×

Sienna Pendleton-(ID Number) 3 20 09 2007 1 × ×

Mehmood Ali -(ID Number) 4 13 08 2007 2 × ×

Alexander Chalmers-(ID Number) 5 13 12 2007 2 × ×

Gender (column 4): 1 = Female; 2 = Male Exclusion Status (column 5): 1 = Students with functional disabilities; 2 = Students with intellectual disabilities; 3 = Non-native language speakers Subject Codes (column 6): 1 = Mathematics; 6 = Science; 7 = Mathematics and Science


Appendix (3)

Student Tracking Form

eTIMSS - Student Tracking Form - [Grade 4 or 8]

School Name:


The Gulf Privet

Qatar

[a] [b] [c] [d] [e]

School ID Class ID Class Name Grade Language of Test

7001 700101 A 4 English

Student Name

or Number

Student

ID Password

Date

of

Bir

th

Gen

der

Exclu

sio

n S

tatu

s

Participation Status

Retu

en

Sta

tus o

f th

e H

om

e

Qu

esti

on

nia

re

Main

Session

Make-Up

Session

Ach

ievem

en

t S

essio

n

Sessio

n N

um

ber

Qu

esti

on

nair

e S

essio

n

Ach

ievem

en

t S

essio

n

Qu

esti

on

nair

e S

essio

n

DD MM YYYY

Bareerah Asad-(ID Number) 70010101 **** 15 09 2007 2 C 1 C P

Teeba AbdurRahman-(ID Number) 70010102 **** 31 07 2008 1 2 A 1 A N

Sienna Pendleton-(ID Number) 70010103 **** 20 09 2007 1 C 1 C N

Mehmood Ali -(ID Number) 70010104 **** 13 08 2007 2 1 A 1 A N

Alexander Chalmers-(ID Number) 70010105 **** 13 12 2007 2 NA 1 NA N

<A> 70010106

<A> 70010107

<A> 70010108

Gender (column 5): 1 = Female; 2 = Male Exclusion Status (column 6): 1 = Students with functional disabilities; 2 = Students with intellectual disabilities; 3 = Non-native language speakers Participation Status (column 7): C = Participated; A = Absent; ; NA = Left school permanently. Session Number (Column 7): Session number is applicable if eTIMSS is administered in more than one session due to the number of computers available Return Status of the Home Questionnaire (column 8): N = Not returned or not applicable (e.g., denied parental permission, student left school, or student did not participate); P = Returned paper;


Appendix (4)

Teacher Tracking Form

eTIMSS - Teacher Tracking Form [Grade 4 or 8]

School Name:


The Gulf Privet

Qatar

[a] [b]

School ID Grade

7001 4

Teacher Name Teacher ID Link

Number Selected Class ID

Class

(Course)

Name

Number of

Eligible

Students

Subject Code

Questionnaire

Return

Status

Mr Ali said 700101 01 70010101 1 30 1

Mr Jason Abi 700102 02 70010102 1 30 6

Ms Josh Rafi 700103 03 70010103 2 28 1

Mr Alen Zamir 700104 01 70010104 2 28 6

Subject Code (column 7): 1 = Mathematics; 6 = Science; 7 = Mathematics and Science Questionnaire Return Status (column 8): N = Not returned; O = Returned online


Appendix (5)

Test Administration Form

eTIMSS - [Grade 4 or 8]

Test Administration Form

Class ID:

[1] School Name:

[2] Class Name:

[3] School Coordinator Name:

[4] Test Administrator Name:

[5] Test Administrator’s Position:

<_ TIMSS national center staff

Teacher from school but not teacher of the

selected class

Other, please

describe

[6] Type of testing session: Main Session Session

Number Make-up

[7] Date of testing: DD: Day of

testing MM: Month of testing

Year of testing: 2018 <_flh_p

_2018_

> 2019

Timing of Test and Questionnaire Sessions

Start time End time Task

(8a) (8b)

Administrative tasks (assigning computers to students,

preparation of students, instructions, etc.)

(9a) (9b)

Testing, first part

(10a) (10b)

Preparation of students for second part

(11a) (11b)

Testing, second part

(12a) (12b)

Responding to the eTIMSS questionnaire

(13a) (13b)

Session for the Student Questionnaire


14 Were there any special circumstances or unusual events during the testing session (e.g., loud

noises, students leaving or disturbing the testing session, or attempting to cheat, fire or smoke

alarms, etc.)?

No Yes, please explain

15

Did students complain about any particular problems with the test (e.g., test too difficult,

confusing, struggling with any particular item etc.)?


16

Were there any problems with the testing materials (e.g., errors or omissions in the Student

Tracking Forms, insufficient supply of materials)?


17

Were there any technical problems that prevented data collection from any of the students?


18 Did any technical problems, such as speed of the test software or malfunctioning

keyboards/mice, occur during the testing that seemed to frustrate students or prevent them

from giving their "best" responses?


19

Did a Quality Control Monitor observe the testing session?



Appendix (6)

Assessment Timings

First: Grade 4

The activity The time

setting up the testing room and logging in each student to his/her computer 30 minutes

assign students to their devices, read instructions from the Test

Administrator Manual, and lead students through the test directions 30 minutes

answering Part 1 of the eTIMSS assessment 36 minutes

A short break 10 minutes


answering a short eTIMSS Questionnaire 5 minutes

break 15 minutes

Answering the paper student questionnaire 30 minutes

Distribution of the paper-based early learning questionnaire (completed at

home by the parents) 5 minutes

Second: Grade 8

The activity The time

setting up the testing room and logging in each student to his/her computer 30 minutes

assign students to their devices, read instructions from the Test

Administrator Manual, and lead students through the test directions 30 minutes


A short break 10 minutes


Answer a survey about using the calculator 5 minutes

break 15 minutes

Answering the paper student questionnaire 30 minutes


Appendix (7)

Student Response Rate Form

eTIMSS

Student Response Rate Form [Grade 4 or 8]

Country:

School ID: School Name:

Main Session (1) (2) (3) (4) (5)

Class ID Number of Students

Present

Number of Students Not Present Number of Students

Present + Number of

Students Not Present

Response Rate

(Count students who

participated in at least

one part of the testing

session)

(Count students who were absent. Do

Not include students who have left

school permanently)

If Student Response Rate is less than 90% then a make-up session is required.

Make-up Session (1) (2) (3) (4) (5)

Class ID Number of Students

Present

Number of Students

Present + Number of

Students Not Present

Response Rate

(Count students who

participated in at

least one part of the

testing session)

(Count students who were absent

. Do Not include students who

have left school permanently)

The response rate is the rate calculated as follows:

Response Rate =

students present

students present +students not present x 100 = %

TIMSS - edu.gov.qa · International Assessment Section 2018 - 2019 TIMSS Guidelines for Rules &...

Documents

Transcript of TIMSS - edu.gov.qa · International Assessment Section 2018 - 2019 TIMSS Guidelines for Rules &...