Timss Model Kbat

7/21/2019 Timss Model Kbat

1/61

TIMSS International ResultsTIMSS Released Items

TIMSS Technical Report

http://timssandpirls.bc.edu/timss2011/international-results-mathematics.html

Personal Communication with IEA

S.Kanageswari Suppiah Shanmugam, PhD.


2/61

TIMSS

Trends in International Mathematics and Science

Study International assessment on Maths/Science for

grade 4/Grade 8

Conducted by International Association forEvaluation of Educational Achievement (IEA)

Every four years since 1995 (TIMSS 2011- 5th cycle)

Development and Administration Maths/Sc Test Booklets

Student/Teacher/School/Curriculum Questionnaires

2


3/61

Maths Questionnaires

Student Questionnaire(S)

Home Experience School Experience

Teacher Questionnaire (T) Education

Professional Development

Teaching Experience

School Questionnaire(P)Availability of Resources

Types of Programmes

Learning Environment

Curriculum Questionnaire(C) Organisation of Maths/Sc Curriculum

Content of Maths/Sc Curriculum

3


4/61

Student Testing Time

Student Achievement Booklet Part 1 45 minutes

Break

Student Achievement Booklet Part 2 45 minutes

Break

Student Questionnaire 30 minutes

4


5/61

Student Achievement Booklet28 item blocks 14 Maths

14 Sc Assigned to 14 Booklets

Each booklet has 4 blocks 2 maths 2 Sc

Each block has 12- 18 items and in two booklets(Linking) MCQ Structured Response Questions

- Provide explanation- Support an answer with reasons- Numerical evidence- Draw diagram- Display data

One block- score points of 18 (on average)Maths/Sc blocks alternately begin (balance position

effect) 5


6/61

TIMSS Mathematics AssessmentFramework

6


7/61

TIMSS 2011 Mathematics Framework

is organized around two dimensions

a content dimension

specifying the domains or subject matter to be assessedwithin mathematics (number, algebra, geometry, and

data and chance)Has several topic areas

Each topic area is presented as a list of objectives

a cognitive dimension

specifying the domains or thinking processes to beassessed (knowing, applying, and reasoning)

The cognitive domains describe the sets of behaviorsexpected of students as they engage with the

mathematics content.


8/61

TIMSS Maths Content Domain

Number (30%) Algebra

(30%)Whole NumFractions/DecimalsIntegersRatio/Proportion/Percent

PatternsAlgebraic ExpressionsEquation/Formula and Function

Geometry(20%)

Data & Chance(20%)

G ShapesG MeasurementLocation & Movement

Data organisation &representationData InterpretationChance

8


9/61

Using mathematics Depends on mathematical knowledge Familiarity with mathematics concept Facts - factual knowledge that provides the basic

language of mathematics, and the essential

mathematical facts and properties that form thefoundation for mathematical thought. Procedures- entails recall of sets of actions and

how to carry them out & computationalprocedures and tools

Knowledge of concepts - make connectionsbetween elements of knowledge, judge the validityof mathematical

statements and methods, and create mathematicalrepresentations

Cognitive Domain

Knowing(35%)


10/61

application of mathematical tools in arange of contexts.

The facts, concepts, and procedures oftenare very familiar to the student, with theproblems being routine ones.

apply mathematical knowledge of facts,

skills, and procedures or understanding ofmathematical concepts to createrepresentations

Problem solving is central but the problem

settings are more routine in theimplemented curriculum.

have been standard in classroom exercises

textbook problems

Applying

(40%)

Cognitive Domain

set in real-life situations

purelymathematicalquestionsemphasismore familiarand routine

tasks


11/61

for logical, systematic thinking.

includes intuitive and inductive reasoning

based on patterns and regularities that can beused to arrive at solutions to non-routine

problems. Non-routine problems - problems that are

very likely to be unfamiliar to students.

They make cognitive demands over andabove those needed for solution of routine

problems, even when the knowledge andskills required for their solution have beenlearned.

Cognitive DomainReasoning(25%)

noveltyof the contextthe complexityof thesituation,any solution to theproblem must involveseveral steps, drawing on knowledgeand understandingfrom different areas ofmathematics

involve transfer ofknowledge and skills tonew situationsInteractions amongreasoning skills


12/61

Cognitive Domain

Knowing

(35%)

Applying(40%)

Reasoning(25%)

12


13/61

13

TOPIC AREA FRACTIONS AND DECIMALS

CONTENT DOMAIN NUMBER

COGNITIVE DOMAIN KNOWING


14/61



COGNITIVE DOMAIN KNOWING

14


15/61



COGNITIVE DOMAIN APPLYING

15


16/61

16





17/61

TOPIC AREA RATIO, PROPORTION AND PERCENT



17


18/61

18

TOPIC AREA GEOMETRIC SHAPES

CONTENT DOMAIN GEOMETRY

COGNITIVE DOMAIN REASONING


19/61

19

TOPIC AREA INTEGER




20/61

20

TOPIC AREA GEOMETRIC SHAPES

CONTENT DOMAIN GEOMETRY



21/61

TOPIC AREA DATA

CONTENT DOMAIN DATA AND CHANCE


21


22/61

INTERNATIONAL BENCHMARKADVANCED (625)

-reasonwith information, draw conclusions, make generalizations, and solve linearequations.-solve a variety of fraction, proportion, and percent problems andjustify

conclusions.-express generalizations algebraically and model situations.-solve a variety of problems involving equations, formulas, and functions.-reason with geometric figures to solve problems and with data from several sourcesor unfamiliar representations to solve multi-step problems

Content Domain: Geometry

Cognitive Domain: Reasoning

Description: Solves a wordproblem involving filling athree-dimensional shape with

rectangular solids

22


23/61

INTERNATIONAL BENCHMARKHIGH (550)

-applyunderstanding and knowledge in complex situation-use information from several sources to solve problems involving differenttypes of numbers and operations.

-relate fractions, decimals, and percents to each other.-show basic procedural knowledge related to algebraic expressions.-use properties of lines, angles, triangles, rectangles, and rectangular prisms tosolve problems.-analyse data in a variety of graphs.

Peter, James, and Andrew each had 20 tries at throwing ballsinto a basket. Complete the missing boxes below.

Name Number of Percentage ofSuccessful Shots Successful

Shots

Peter 10 out of 20 50 %

James 15 out of 20

Andrew out of 20 80%

Content Domain:Number

Cognitive Domain:Knowing

Description: Given

the part and thewhole, can expressthe part as apercentage, andgiven the whole andthe %, can find the

part 23


24/61

-apply basic mathematical knowledge in a variety of situations.-solve problems involving decimals, fractions, prop,%

-understand simple algebraic relationships.-relate a two-dimensional drawing to a three-dimensional object.-read, interpret, and construct graphs and tables.-recognise basic notions of likelihood.

INTERNATIONAL BENCHMARKINTERMEDIATE (475)

What does xy + 1 mean?o a Add 1 to y, then multiply by x.

o b Multiply x and y by 1.

o c Add x to y, then add 1.

od Multiply x by y, then add 1.

Content Domain: Algebra


Description: Knows the

meaning of a simplealgebraic expressioninvolving multiplicationand addition

24


25/61

Content Domain:Number


Description: Adds atwo-place and athree-place decimal

Have some knowledge of whole numbers and decimals,

operations, and basic graphs

INTERNATIONAL BENCHMARKLOW (440)

42.65 + 5.748 =

Answer: ________

25


26/61

Malaysian Students Performance

26


27/61

Administration in Malaysia45 countries

Four countries in SEA(Indonesia, Malaysia, Singapore, Thailand)

180 Schools

5733 Form Two students

Test Booklets Bahasa Malaysia and English

Student Questionnaire Bahasa Malaysia

Maths Teacher Questionnaire English

School Questionnaire English

27


28/61

Malaysia Performance Trend

Only in Grade 8 Participated since 1999 1999- 16th (39 countries) - BM 2003- 10th (48 countries) - BM 2007- 20th (46 countries) - BM/ENG 2011- 26th (45 countries) - BM/ENG

Recorded as one of the countries that declined greatly (40points or more)

519

508

474

440

400

420

440

460

480

500

520

540

1999 2003 2007 2011

28


29/61

494

455

474

459

474

380

400

420

440

460

480

500

Number Algebra Geometry Data & Chance Overall

451

430 432 429

440

2011

Performance Trend- Content Domain

Num Geometry Data & Chance / Algebra

Num OverallGeometry / Data & Chance / Algebra Overall 29


30/61

Performance Trend - Cognitive Domain

Low performance in Reasoning

Not much difference between Knowing/Applying

473

477

466

474

415420

425

430

435

440445

450

455

460

465

470

475

480

Knowing Applying Reasoning Overall

2007

444

439

426

440

2011

30


31/61

Performance Trend -Content Domain by gender

Girls Overall Boys

460

440438

435

410

415

420

425

430

435

440

445

450

455

460

465

470

Number Algebra Geometry Data&Chance

Girls

441

419 425

422

Boys

451

430432

428

Overall

31

P f T d C i i D i


32/61

Performance Trend- Cognitive Domainby gender

Girls Overall Boys

456

445

432

415420

425

430

435

440

445

450

455

460

Knowing Applying Reasoning

Girls

431 432

420

Bo s

444

439

Overall

32


33/61

ACHIEVEMENT FOR MALAYSIA AT INTERNATIONAL BENCHMARK

At every benchmark, there is a decline

93

70

36

10

0

10

20

3040

50

60

70

80

90

100

Low Intermediate High Advanced

1999

93

66

30

6

200382

50

18

2007

65

36

13

2

2011

33


34/61

388

455

507

562

630

0

100

200

300

400

500

600

700

5th 25th 50th 75th 95th

2003319

416

475

530

608

2007

290

373

440

507

589

0

100

200

300

400

500

600

700

5th 25th 50th 75th 95th

2011

Percentile points for Malaysia

34


35/61

290

319

373

440

507

560

589

0

100

200

300

400

500

600

700

800

5th 10th 25th 50th 75th 90th 95th

Malaysia

453

494

559

620

672

713734

Singapore

PERCENTILE POINTS for MALAYSIA & SINGAPORE

in TIMSS 2011

35


36/61

TIMSS Mathematics Items

36


37/61

Question Types and Scoring Procedures

2 question formats- MCQ & constructed-response

Each MCQ is worth 1 score point.

Constructed-response questions worth 1 or 2 scorepoints, depending on the nature of the task and the

skills required to complete it. The choice of item format depends on the mathematics

or being assessed, and the format that best enablesstudents to demonstrate their proficiency.

M lti l Ch i Q ti (MCQ)


38/61

Multiple-Choice Questions (MCQ) MCQ provide students with 4 response options.

Select one correct response

can be used to assess any of the behaviors in the cognitivedomains.

allow valid, reliable, and economical measurement of a widerange of content in a relatively short testing time.

do not allow for students explanations or supporting statements less suitable for assessing students ability to make more

complex interpretations or evaluations.

linguistic features need to be appropriate.

questions are written clearly and concisely. response options also are written to minimise the reading

load of the question.

The options that are incorrect are written to be

plausible, but not deceptive.


39/61

Constructed-Response Questions (CRQ)

students are required to construct/give a writtenresponse, rather than select a response from a set of options

allow students to provide explanations, support an answerwith reasons or numerical evidence, draw diagrams, ordisplay data,

well-suited for assessing aspects of knowledge and skillsthat require students to explain phenomena or interpretdata based on their background knowledge and experience

real-world setting - the setting is familiar to students.

need to prepare a Scoring Rubric

i b i


40/61

Scoring Rubric describes the essential features of appropriate and complete

responses.

focus on evidence of the type of behavior the question assesses.

describe evidence of completely correct, partially correct andcompletely incorrect responses.

student responses at each level of understanding provideimportant guidance to those who will be rating the studentsresponses.

In scoring, the focus is solely on students achievement withrespect to the topic being assessed, not on their ability to write

well. students need to communicate clearly

ypes o tems- ra e


41/61

ypes o tems- ra e

T f It G d 8


42/61

Types of Items- Grade 8


43/61

Scoring Rubric

T f It G d 8


44/61


T f It G d 8


45/61


T pes of Items Grade 8


46/61



47/61

TIMSS Item Writing Process

47

Item Writing Process and Guidelines


48/61

Item Writing Process and Guidelines

consider the timing, grade appropriateness, difficultylevel, potential sources of bias (cultural, gender, orgeographical, ).

Make sure that item validity is not affected by factors thatunnecessarily increase the difficulty of the item, such asunfamiliar or difficultvocabulary, grammar, directions, contexts, or stimulusmaterials

be sensitive to the possibility of unintentionally placingparticular groups of students at an unfair disadvantage

diagrams and graphs are drawn accurately (to scale unlessotherwise noted), and are correctly and fully labeled.

What should the student know? What should the student be able to do?

MCQ


49/61

MCQ

Stem is the initial part of the item in which the task is defined. Options refer to the entire set of labeled response choices presented

under the stem. Key is the correct response option. Distracters are the incorrect response options. ask a direct question with only one correct answer, and provide plausible

distracters the question must be able to stand alone, and be answerable without

the response options do not include extraneous information that can confuse students avoid questions for which a wrong method yields the correct answer

(e.g., a question about a circle with a radius of 2, since computing either

area or circumference get 4) avoid writing items where students can work backwards from the

response options to find the correct answer (e.g., solving for x in anequation).(CRQ is more appropriate)

CRQ


50/61

CRQWrite a full credit

answer in terms of the

language, knowledge, and skills that studentcould be expected topossess & determine

whether to allocate 1 or2 score points.

Develop a specificscoring guide


51/61

Take Home information

Each of the items needs to contribute to the overallmathematics test

Some relatively easy items and some challenging items

Avoid items that almost all students or almost no

students are able to answer correctly


52/61

Checklist- MCQ


53/61

Checklist-CRQ


54/61

Each item

1The Content Domain, topic area, and objective theitem measures

2. The Cognitive Domain

3. The item number (1, 2, 3, etc.)

4. The key (multiple-choice items only) or

5. The scoring guide

1


55/61

1

2


56/61

2


57/61


58/61


59/61


60/61


61/61

Timss Model Kbat

Documents

Transcript of Timss Model Kbat