Kit-Tai HauThe Chinese University of Hong Kong

1

Assessments of National Educational Progress, School Improvement and

Students’ Learning – Dynamics and Practices in Asia

Outstanding Asian Performance Converging evidence:– International studies: Asian outstanding over widely

diverse representative population– e.g., 2nd IEA: 13-yr China, Taiwan among the top – TIMSS: G.4, 8 Hong Kong, Japan, South Korea,

Singapore outperformed others– PISA: Singapore, Hong Kong, S Korea top 3– Recent PISA (2012): Shanghai (China), 1st in all 3

subjects (Reading, Maths, Science), Hong Kong 2nd, 2nd, 3rd

2

– Hong Kong, Taiwan, China, Japan, South Korea (except Singapore) have Ed Monitoring systems, some (Hong Kong, S Korea, part of Japan, some parts in China) producing individual school reports to the public (league table)

– Will use Hong Kong as an example to illustrate the dynamics /issues involved

3

Presentation OutlineI. Educational Monitoring in SE Asia/PacificII. General FrameworkIII. Hong Kong Attainment Test (HKAT)IV. Problems of HKATV. Consultancy Report: Can one test serve all?VI. Divided into: Student + System AssessmentsVII. Student AssessmentVIII. System AssessmentIX. Positive Uses of Monitoring Assessment ResultsX. Current Problems/Issues

4

I. Ed Assessment in SE Asia/PacificCountry Level Subjects First YrAustralia G 3, 5, 7, 9 Literacy, Numeracy 2008

G 6, 10 ICT 2005G 6, 10 Science (Sc) 2003G 6, 10 Civil Knowledge 2004

Bangladesh G 3, 5 Language (Lang), Math 2006G 1, 5 Lang, Eng, Math, Sc, Soc Sc 2004

G 8 Lang, Eng, Math 2008Bhutan G 2, 4 Lang, Eng, Math 2007

G 4, 6, 8 Eng, Math, Sc 2008G 6 Eng, Math 2003G 10 Lang, Eng, Math, History 2006

5

Country Level Subjects First YrCambodia G 3, 6, 9 Lang, Math 2005China G 4, 8 Chinese, Eng, Math, Sc, Psycho

health, physical Ed/ Health2007

Cook Isld G 4, 6 Lang, Eng, Math 1999G 4, 5, 6 Eng, Lang 1994G 3, 7 Math 1994

Fuji G 4, 6, 8 Literacy, Numeracy 2007India G 3 Lang, Math 2004

G 5 Lang, Math, Enviro Sc 2002G 8 Lang, Math, Sc, Soc Sc 2003Age 5-16 Literacy, Numeracy 2005

Japan G 6, 9 Jap, Math, Sc 2007Kiribati G 4, 6 Lang, Eng, Numeracy 2004Lao G 1, 2, 5 Lang, Math, Sc 1996Maldives G 4, 7 Lang, Eng, Math, EnvSc, SocSc 2008

G 9 Phys, Chem, Bio, Hist, Geog 20136

Country Level Subjects First YrMyanmar G 3, 5 Lang, Math 2007Nepal G 3, 5, 6,

8, 10 Nepali, Math, Soc Studies, Eng, Sc, Health

1999

N Zealand G 4, 8 Sc, Visual Arts, ICT, Lang, Tech, Music, Math, Soc Studies, Health

1995

Pakistan G 3, 4, 8 Lang, Maths, Sc, Soc Studies 2003Philippines G 3, 6, 12 Eng, Filipino, Math, Sc, Soc Sc,

Critical thinking2004

S Korea G 6, 9, 11 Korean, Eng Math, Sc, Soc Studies 2000Samoa G 4, 6 Samoan, Eng, Numeracy 1996Singapore G 6, 10,

12Lang, Math, Sc, Soc Sc, Applied subjects

1960

Salomon I G 4, 6 Literacy, Numeracy 2005Sri Lanka G 4, 8, 10 Singalese, Tamil, Eng, Math, Sc,

Tech2003

7

Country Level Subjects First YrThailand G 2, 3, 5,

6, 9, 11, 12

Thai, Foreign Lang, Math, Sc, Soc Sc, Religion, Health, PE, Art, Career, Occupational Studies

2005

Tonga G 4, 6 Tongan, Eng, Numeracy 2011Tuvalu G 4, 6 Literacy, Numeracy 2000Vanuatu G 4, 6 Literacy, Numeracy 2007Vietnam G 5, 6, 9,

11Vietnamese, Math, Eng, Physics, Bio

2001

8

II. General Framework: Monitor + Feedback

9

Level Purpose/ Help……

Comparison targets

Student Student learn

Own past, classmate, national norms

Class Teacher teach

previous yr, other classes, national norm

School School Accountable

previous yr, other schools, national norm

State/City

Plan, accountable

previous yr, other states, national norm

Country Monitor progress

previous yr, other countries (e.g., PISA,TIMSS)

III. Hong Kong Attainment test (HKAT)• 1976-2003• One test at each grade, G.1 - 9• Subjects: Chinese, English, Maths• Operation: – test/questions left with school– changed every few years – marked by own teachers– school submit 1/30 (e.g., birth day 13th) for Ed Bureau

to mark (build HK norm, monitor HK)– later change to 1/3 to monitor school

• Main purpose: to monitor whole Hong Kong

10

• Little information provided:– for students: question paper too short, no

diagnostic value– for schools: only a relative ranking (cf. HK norm)– for whole Hong Kong: only a summary total (2015

M=87.6, no other information)• Once a year, insufficient to help daily learning• Teachers already have other means (school tests) to

understand own students’ strength/weakness• Teachers’ heavy load in marking• Paper too easy for top schools

11

IV. Problems of HKAT

V. Consultancy Report (review 2000): Can one test serve both purposes (Monitor/ Feedback)?

12

• Seek consultant’s advice

• Visit ETS, etc.• Public

consultation

Prof W. J. van der Linden

Can one test serve all purposes?Student School Country

Once a year Not enough Too much, has to rotate subjects as in PISA

paper long short

Good, but expensiveCoverage not enough

Each student answers part of the questions

Difficulty:key competencemid difficulty

Not enough discrimination, low qualityBest psychometric properties, but students, parents feel too difficult

High Stake Narrowing of curriculum, teachers cheat results not valid

13

Ideal Monitoring System• To Student: test daily, auto-marking, low stake, can retake, in

line with self-regulation• To school/city/country: should attempt to reduce the

unavoidably high stake, can change focus each year (academic subject on rotation, as in PISA)

• Schools have to supervise test administration, to reduce incentive in cheating, should not allow school to publicize results

• If cheating too serious/pressure on students too large, do not report school results, only provide higher level summaries

• But too costly if the whole scheme is just to monitor the country and does not produce school reports

14

VI. Divided into Student + System Assessments

Decision of the HK government in 2000:•Will monitor Chinese, English, Maths basic competence•To help learning + monitor school/provide support

15

(A)Secured test (G.3, G.6, G.9)System Assessment (slightly high stake)

(B)Web based adaptive test(G.1-G.9)(Student Assessment)

Hong Kong Attainment test

VII. Student Assessment• Web based• Purpose: help teachers in assessing their students• Supplement and adapt to students’ need,

difficulty appropriate to students’ ability• Can fully serve the daily need throughout the

school term

16

17

18

19

20

Problems of Student Assessment

21

• In-charge by the Examination Authorities (HK Examinations and Assessment Authorities): high quality but too few items

• Not enough promotion of usage, self-regulation, self-monitoring function

• Student mark recording system too complicated (does not facilitate widespread use)

• Does not link to learning or remedial packages, thus cannot be a fully standalone learning system

VIII. System Assessment• Mandatory tests for all students in all government

funded schools• To understand the achievement of basic competence• Only report % students (e.g., 85%) meeting basic

competence in each school, cannot publicize to avoid building league table and creating additional pressure to students/schools

• On Chinese/English/Maths, 30-90 min G.3, 65-1 – 2 hour G.9

• Only at G.3, 6, 9 (no G.1, cannot calculate value added for primary schools)

22

• One external teacher help invigilation• Centrally marked• Include even oral exam for Chinese, English (only randomly select 10%-20% of

students)

• Example report at system level (% meeting Basic Competence

23

G.3 G.6 G.9Chinese 85% 77% 76%English 79% 71% 69%Maths 87% 84% 80%

IX. Positive uses of Monitoring Assessment Results

• Analyses on item/domain performance on student + system assessment can generate useful feedback on school performance (on their strength/weakness)

• Exam pressure, over-drilling already serious in Asia Singapore no monitoring tests, other countries careful not add pressure through school reports, league tables

• In PISA, Hong Kong 1st in equity - Social Economic Status (SES) has least effect on students’ achievement

24

• Help for new migrants, low SES students through– Extra funds when schools have (i) new migrants,

(ii) low SES, (iii) SEN (special ed need) students – but must allow migrants to join System Assessment (for accountability)

– Ed Bureau (government) special team to provide school based professional support: jointly prepare teaching/ assessment materials

– University-school partner teams (supported by government funds) help needy schools

25

X. Current Problems and IssuesSystem Assessment •Only % pass/fail basic competence, no fine grade •After a few years, as these figures are quite stable, no one is interested, no one care (good, no pressure on students), but little diagnostic information as well•Items focus basic competence too easy low discrimination low quality in monitoring whole Hong Kong more difficult items added recently•To enrich the reports, now providing item performance information for schools in System Assessment but schools drill on identical item types (possible solution generate item analyses from Student Assessment)

26

• Not rotating academic subjects across years yet• questionnaire now added on students’ attitude,

family background, etc.• Low monitoring in test administration schools may

cheat, not totally cheating-proof• Item types too narrow, not creative enough, not

having a positive effect on teaching• Population declines primary schools great

pressure to drill to get good results asked government not to conduct test/or release results now stop releasing results to primary schools

• Secondary schools (G.9-12) have a high stake university entrance exam anyway results continued to be released to them

27

• Student Assessment– Not widely used by students– Insufficient coverage of fully curriculum– Item types not attractive– No linkage to remedial teaching packages (to enable self-

learning)– Insufficient convenience for students’ self-usage

• Conclusion– A good system is a compromise of educational,

psychometrical (measurement), and political considerations

28