ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   1  

The Assessment, Curriculum and Technology Research Centre is a partnership between the University of Melbourne and the University of the Philippines supported by Australian Aid.

LearnARMM

Esther Care Melissa Digo Hazelle Preclaro Eufracio Abaya ACTRC Research Forum 18th July 2015

ACTRC has been established as a joint research centre between the Assessment Research Centre at the University of

Melbourne and the University of the Philippines. The Centre is funded by the Australian Government.

LearnARMM Project Team

Project Team Leader Dr Esther Care

Test Development, Fieldwork, and Data

Analysis Staff

Case study Academic Lead

Research Officer

Overview

Goals - To explain the information we have gathered - To describe our approach to assessment so that you see how research data can be used at system, school, and teacher level

•  Section 1: Research overview and student assessment results

•  Section 2: Language issues in test development

•  Section 3: Contextual factors at case study level

Definitions

• Curriculum – the outline of subjects in terms of content and skills to be taught, together with information about how these subjects will contribute to the higher level goals of education

• Assessment – the gathering of information that shows how students are achieving the objectives of the education system

Curriculum and assessment

•  The curriculum is built according to an understanding of a typical order or sequence in how students learn

• Assessment tools, such as tests, need to reflect the curriculum:

- Some tools will be built to assess content knowledge - Some tools will be built to assess skills - Some tools will be built to assess both

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   2  

Assessment approach

•  The curriculum is just like stairs • We need to know what step the student is on in terms of

acquiring the skills and the content that is described in the curriculum

•  Then we can make sure the student is properly balanced on that step, so that we can help them up to the next step

• A good assessment tool will not just tell you þ or ý • A good assessment tool will tell you where the student is

on their way up the stairs

What does the LearnARMM tool do?

þ •  It tells us what the current skills

of the students are •  It reflects the skills in the

curriculum •  It helps teachers to know what

the student is ready to learn •  It is designed to sample skills

on each step along the way in the Kinder and Grade 1 curriculum

•  It can provide information about how a student progresses from one test event to the next

ý •  It does not give us a score •  It is not designed for pass

or fail

What are we testing? Mathema'cs  •  Numbers  &  number  sense  •  Measurement  •  Geometry  •  PaBerns  •  Data  

Literacy  •  Oral  language  •  Phonological  skills  •  Book  and  print  knowledge  •  Alphabet  knowledge  •  Word  recogniNon  •  Fluency  •  Spelling  •  HandwriNng  •  Composing  •  Grammar  awareness  •  Vocabulary  and  concept  

development  •  Reading  comprehension  •  Listening  comprehension  

Numeracy: Level descriptions

Solve one-step word problems using addition and subtraction. Identify missing elements from a number pattern. Read data from a table. Identify units of time (days in

a week, 15-min intervals, months in a year) and how many elements are in a group. Perform numeric operations (subtraction) using grouping and strategies such as

decomposition.

Perform numeric operations (addition) using grouping and strategies such as decomposition. Recognise fractions as part of a whole. Identify magnitude of a place

value. Perform skip counting by 2s.

Recognise equality of sets and numbers. Perform operations on two-digit numbers without grouping. Perform relative ordering using a mental number line. Recognise

numeric patterns. Tell the date/read a calendar.

Write and read numbers. Recognise zero. Identify parts of the day (non-formal time). Tell analog time in half hours. Recognise numerical order. Perform addition and

subtraction without regrouping. Classify objects based on geometric properties. Identify geometric properties. Use numbers to perform addition. Identify basic geometric shapes

by name. Compare sets using estimation and/or counting strategies.

Group and classify objects according to property. Write and record numbers. Recognise money in familiar denominations.

Count objects and compare quantity. Perform simple operations on numbers less than ten using counting strategies.

Literacy: Level descriptions Identify text and story structure (beginnings and endings). Recall and connect information in the story one has read. Read with comprehension paragraph-level text that is age appropriate. Write short paragraph about a given topic. Observe paragraph-level structural rules such as capitalisation, indentation, and proper punctuation.

Apply phonics knowledge consistently to read and spell complex word structures. Recognise and read with attention to sentence-level structure. Read simple sentences. Write text (sentence) describing everyday events.

Know the names and sounds of letters and provide words that begin with less frequently-occurring letters. Apply phonics knowledge to read and spell words with simple word structure. Record ideas using symbols and/or words.

Recognise that words are made of smaller units (sounds and syllables). Know the names and sounds of letters and provide words that begin with frequently-occurring letters. Use phonics knowledge to distinguish between words that sound similar. Demonstrate understanding of proper book orientation and concepts about print. Identify words that rhyme. Listen and respond to questions about key events in a text that is read aloud. Record ideas on a given topic using drawings and single words. Distinguish between letters, numbers and other symbols. Identify similar and different sounds in words. Understand that words are read from left to right. Listen to a story and make inferences about the plausibility/reality of the described events.

Respond to questions relating to personal experience. Describe familiar situations and provide additional detail when prompted. Identify familiar animals, things, and objects. Understand that meaning is assigned to words.

Types of reports

①  Student ②  Students within class by

skill level ③  Students within class from

Time 1 to Time 2

① 

② 

③ 

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   3  

    Province   DepEd   LCs  

    GRADE   Total   GRADE   Total  

    Kinder   Grade  1   Kinder   Grade  1  

2014   Lanao  del  Sur   480   600   1080   445   586   1031      

Maguindanao   450   662   1112   447   595   1042  

2015   Lanao  del  Sur   405   525   930   406   557   963      

Maguindanao   355   475   830   394   423   817  

Students across Time 1 and Time 2 testing

725 students or (17% of 2014 students) were not involved in the Time 2 (2015) assessment

LIT_T1   LIT_T2   LIT_T1   LIT_T2  DepED   LCs  

Male   50.2688   57.5520   48.9295   55.2140  

Female   51.4598   58.9188   49.8111   56.4599  

40  

42  

44  

46  

48  

50  

52  

54  

56  

58  

60  

Scores  

Male  

Female  

Progress  in  Kinder  Literacy  by  gender  

Progress  in  Kinder  Maths  by  gender  

MATH_T1   MATH_T2   MATH_T1   MATH_T2  DepED   LCs  

Male   51.0706   58.9324   49.3279   56.5860  

Female   50.5594   58.2973   48.3768   56.0091  

40  

42  

44  

46  

48  

50  

52  

54  

56  

58  

60  

Scores  

Male  

Female  

Factors Home and family

Child characteristics

Teacher

School and infrastructure

Environment

Gender

Language

Student learning

outcomes

e.care@unimelb.edu.au m.digo@actrc.org

www.actrc.org

The Assessment, Curriculum and Technology Research Centre is a partnership between the University of Melbourne and the University of the Philippines supported by the Australian Government.

TranslaNon  Processes  in  the  Development  of  Achievement  Tests  Across  Three  Languages:  Meranao,  Maguindanaoan  and  Sama  Maria  Hazelle  Preclaro  University  of  the  Philippines  

Esther  Care  University  of  Melbourne  

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   4  

The translation process

• The method adopted for translation was of multiple forward translations, rather than a forward, then backward approach

English Core Doc

Translation by Linguist 1

Translation

by Linguist 2

Harmonised

by educators 1 & 2

(3 STEPS)

Proofreading

Content Experts

Content Experts

F T I E N S A T L

The translation process

English Core Doc

Translation by Linguist 1

Translation

by Linguist 2

Harmonised

by educators

1 & 2 (3 STEPS)

Proofreading

Content Experts

Content Experts

F T I E N S A T L

Key actors

Psychometricians  

Criteria  -­‐  Expert  in  educa/onal  and  psychological  assessment  

Role  • Ensures  the  objec/ve  assessment  of  the  learning  achievement  in  Mathema/cs  and  Literacy  

Test developers and content-area experts

Criteria  • Proficient  in  source  language    Role  • Developed  items  to  measure  the  intended  construct;  worked  to  parallel  items  across  languages  

Factors that influence translation

1.  Content/curriculum  –  based  factors  2.  Developmental  and  cultural  factors  3.  LinguisNc  factors  -  Orthographic  -  SemanNc  -  SyntacNc  

Mathematics Test

Ques'ons:  Ø Which  terms  are  being  used  in  the  schools/centers?    

     (English  or  the  local  terms)  Ø Given  MTB-­‐MLE  and  the  revised  K-­‐12  curriculum  are  students  expected  to  use  local  terms?  

Ø Is  this  informa/on  found  in  the  teachers’  guides/IRR  of  MTB-­‐MLE?  

 

Literacy Test

Ques'ons:  Ø Which  competencies  must  be  present  at  the  end  of  the  year/  must  the  test  cover?  

Ø In  what  ways  do  the  instruc/onal  sequence  of  skills  differ  across  languages?  

 

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   5  

2. Development and cultural factors

Ques'ons:  •  In  what  ways  will  the  tool  need  to  differ  in  order  for  it  to  be  ‘localized/contextualized?’  

•  Which  events  are  within  the  child’s  range  of  experience?  

•  What  prac/ces  are  acceptable  given  their  way  of  life?  

2. Development and cultural factors

•  Contextualiza'on              (e.g.  Environment  and  way  of  life)      Word  problems  and  stories  involved  daily  experiences    ü  Use  of  local  names            (e.g.  Miss  =  Aling  =  Babo/Peter  vs.  Juan)  ü  Gathering  toys,  fruits    (e.g.  bikes,  bananas)  

ü  Buying  something  from  the  store    (e.g.  Kind  of  store)  

ü  Preparing  a  meal    (e.g.  gender  roles,  kind  of  viand)  

 

Literacy Test

Ques'ons:  Ø Which  domains  have  the  children  been  exposed  to?  

Ø In  what  ways  were  the  local  languages  similar    or  different  with  regards  to  their  orthography?  

Ø Which  words  must  be  avoided  to  reduce  confusion  in  test  administra/on?  

3. Linguistic factors

Orthography  - Transparent  - Consistent  correspondence  Alphabet  Knowledge:  - 5  vowels  - 23  consonants  Phonological  Knowledge:  - Word  level  - Syllable-­‐level  - Phoneme-­‐level  

SimilariNes:  ü  Transparent  

orthography  ü  WriBen  in  

Romanized  script  

3. Linguistic factors

§ Phonological  Knowledge  § Word/syllable/phoneme  levels  

§ Word  Structure  Challenges:    ü  Finding  equivalence  ü  Sequencing  

decodable  words  from  easiest  to  more  difficult  

English Local Language

•   Two  syllable  words  are  usually  composed  of  a  consonant  and  a  vowel  •   Simple  words  ojen  have  the  CVC  paBern    

•   Two  syllable  words  may  consist  of  two  vowels  (e.g.  vv  =  oo,  aa,  ee)  •   Simple  words  ojen  have  the  VCV  or  the  CVCV  paBerns      

3. Linguistic factors

• Rhyme  • Difference  in  accentuaNon  /stress  (e.g.  baso    =    laso  =    pa’so/paso’)  

 •  Avoidance of Homographs - words that are spelled in

the same way but can be pronounced in 2 or more ways

(e.g. pa’so VS. paso’) burn flower pot

 

Challenges:      ü  Rules  on  Stress    ü  Presence  of  

Homographs  

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   6  

Conclusion

Given  that  the  influences  on  test  construcNon  include:  •  Content/Curriculum-­‐Based  Factors  •  Developmental  and  Cultural  Factors  •  LinguisNc  Factors  •  TranslaNon  

-  Forward  translaNon  -  HarmonisaNon  -  Involvement  of  key  parNcipants  

 

Recommendations

Considering  the  major  orthographic  differences  between  English  and  the  local  languages:  

•  Development  of  a  Filipino  Core  Version  (due  to  similarity  in  orthography)  

•  TranslaNons    are  made  from  the  Filipino  Core  Version  rather  than  from  the  English  Core  Version  

The Assessment, Curriculum and Technology Research Centre is a partnership between the University of Melbourne and the University of the Philippines supported by the Australian Government.

LearnARMM: Case Studies Dr. Eufracio C. Abaya

Backdrop

• Objective of the Study - To identify factors that might explain differential student learning

outcomes.

• Research Methods -  In-depth field interviews and observations, guided by the case

study protocols - Photo-documentation

• Participants - Kindergarten students - Teachers - School Administrators - Parents

RESEARCH SITE

Framework

BLC DepEd Elementary

School

Family

Learner •  Child’s status and dispositions

•  Formal learning achievements

•  Family structure and organization

•  Resources •  Home

Literacy environment

•  Facilities and resources

•  Organization of teaching and learning

•  Classroom environment

•  Facilities and resources

•  Organization of teaching and learning

•  Classroom environment

Community / Neighborhood (Social, Economic, Political, Linguistic, Ethnic,

and Spiritual Orientations) and

Physical Environment

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   7  

Conditions of the Classroom 1 • Colorful decorations • Sufficient number of books and teaching devices • No tables and chairs for learners • Poor ventilation • No electricity •  Inadequate water supply •  Foul smell from the nearby canal • Noise due to footsteps from the second floor

Conditions of the Actors in the Classroom 1

•  Females are more attentive than males. • Noisy students are difficult to control. • Older students tend to bully younger classmates. • Many students do not eat breakfast before going to

school. • Many students incur absences due to illness. •  Teacher imposes discipline by threatening students that

they will have to pay one peso for every observed misdemeanor.

Conditions of the Classroom 2

• Colorful decoration • Poor ventilation • Poor lighting •  Inadequate water supply • Cramped space • Writing exercises made difficult because there are no

tables and chairs for learners • Sari-sari stores near the classroom

Conditions of the Actors in the Classroom 2

• Noisy students are difficult to control. • Some male students use abusive language. • Many students go to class without breakfast. • Most students have short attention span. • Some sneak out to buy snacks from the nearby sari-sari

store.

Classroom Condition 3

• Colorful decorations • Adequate shelves • Good ventilation • Poor lighting • Noise heard from outside •  Foul smell from the comfort room • No potable water supply

Condition of the Actors in the Classroom 3 •  Students are noisy and talkative. •  Students switch seats often. •  Males are more into math. •  Females are more into reading. •  Students are observed to be sickly. •  Many students do not eat breakfast before going to school. •  Administrator disciplines absentees and bullies by requiring

them to clean the classroom, comfort room, and playground •  Administrator punishes students who fight each other either by

locking them inside a room or by letting them clean the toilet

ACTRC  Research  Forum   18th  July  2015  

(c)  ACTRC  2015   8  

Classroom Condition 4

• Colorful decorations • Good ventilation • No foul smell • No power supply • Noise coming an ongoing constrution of the school gym •  Inadequate water supply

Condition of the Actors in the Classroom 4

•  Females are more attentive than males. • Some males run around the classroom while some

females play with dolls. • Some do not participate in activities. • Many students do not eat breakfast before going to

school. •  Teacher punishes noisy students by pinching their ears.

Conclusions §  Differential Learning Achievements

§  not amenable to unitary explanations §  Constellation of Conditions and Paradoxes

§  Home Context §  Family background

§ Socioeconomic status § Level of education § Familial relations § Child’s health status

§  Child’s dispositions § Preference between literacy and numeracy § Classroom demeanor § Response to bullying § Performance of household chores and other tasks

§  Home literacy environment § Learning materials § Assistance in doing homework

§  School Context §  Proximity of school and home §  School facilities §  Classroom condition §  Organization of teaching and learning

§ Classroom management and behavior modification

ecabaya.upd@gmail.com

www.actrc.org

@ACTRC_edu

www.facebook.com/ACTRC.org