Applying New Voice Recognition Technology to Formative Assessment Margaret Heritage UCLA Graduate...
-
Upload
trevor-robinson -
Category
Documents
-
view
213 -
download
0
Transcript of Applying New Voice Recognition Technology to Formative Assessment Margaret Heritage UCLA Graduate...
Applying New Voice Recognition Technology to Formative
AssessmentMargaret Heritage
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)
Markus IseliHenry Samueli School of Engineering, UCLA
CRESST Conference,Los Angeles, CA
September 8th, 2005,
Overview
• Project Aims and Components
• Features of the Program
• Automatic Speech Recognition Technology
• Interface Demo
• Assessment Framework
• Looking to the Future
Project Components• Develop speech recognition technology for
children
• Apply technology to create an on-demand, easy-to-use system of assessment in reading for students in grades K-2
• Develop system capacity to present auditory, text, graphical stimuli, and to score, analyze and adapt to responses
• Develop query-based data mining to monitor students’ progress
• Develop easy-to-understand displays of data analysis
Specific Aims
Develop assessment system that :
Is helpful for teachers (i.e. has instructional utility and saves time)
Reduces variability (e.g., consistent instructions, consistent delays, consistent scoring)
Automatically scores and analyzes children’s performance on reading assessment tasks
Distinguishing Features
• Strong interdisciplinary interactions among electrical engineering, computer science, education, psychology and linguistics
• Collaboration with expert teachers
• Focus on bilingual (Mexican-Spanish accented English) students
• Validation of the system
Lens of Project
• Information that teachers can use next day in their instruction?
• Sensitivity to English Language Learners(ELLs)
• Sensitivity to Language Knowledge
Instructional Utility
Effective classroom is assessment-centered (NRC, 2000)
• ongoing assessment of students’ learning that provides the
day-to-day fuel for instruction
Formative assessment
• ‘ used to adapt the teaching work to meet the learning needs’
(Black, Harrison, Lee, Marshall, & Wiliam, 2003, p.2).
Reading error or pronunciation difference?
In Spanish, compared to English
Phonetics p t k closer to Eng b d g than to p t k
s z n t d: tongue on teeth, not behind them
Sounds missing: th, oy, etc.
Phonology s+ptkbdg only across syllables
Distinctions like ‘bit-beat’ not made
Literacy Words spelled ‘y’ pronounced ‘j’, (by some)
Words spelled ‘i’ pronounced ‘ee’, etc.
Academic Language
Among the factors contributing to non-comprehension of text:
• inadequate knowledge of the words used,
• lack of familiarity with the syntactic structures
(Lyon, 1998)
Automatic Speech Recognition (ASR)
• Teach the computer to understand human speech
OR• Teach the human being how to talk
to be understood by a computer
Three main challenges:• Speaker: gender, pronunciation,
health, dialect, language
• Environment: noise, other speech
• System: devices, program
Child vs. Adult Speech
• Children don’t talk just to be understood by the computer, they just talk!
• Children cannot yet control their articulators as well as adults
• Children have different anatomical features (shorter vocal tract), and these features change fast
• Children have very high pitch frequency
Designing an ASR System
Questions
• Who is going to use the ASR system?
• In what environment will it be used?
Implementation
• Collect “a lot” of appropriate data
• Train the system
• Test the system and make changes
Our Assessment SystemThe system will:
• Assess each student in the same manner (no dependence on teacher)
• Produce visual stimuli (letters, words, phrases) and record the child’s vocal response
• Measure response times very accurately
• Analyze the recorded audio and other data to generate reports which are useful to teachers (ASR)
• Be easily accessible (internet)
• Handle multiple users at the same time
System Architecture
Interface
Client Side Server Side
More detailed…
Flash Interface: Live Demo
http://kittychan.icsl.ucla.edu/tball
Flash interface design by Larry Casey
System Design BenefitsAccessible
Software: common web browser Hardware: standard microphone
Flexible Command structure is open-ended Allows for any audio-visual testing set-up
Stable Constant audio stream: everything is
captured Stimulus/response data is recorded in real
time
Scalable Content, display, navigation are
independent
Assessment Framework
• Recall the lens
• Guiding questions:
Are the assessments embedded in an instructional framework?
What is the instructional value of the information?
How much assessment is too much?
Assessment Framework
• Guiding questions privilege a hierarchical rather than a uniform approach to assessment.
• All students take benchmark assessments as a check on progress
• Some students take 'drill down' assessments related to specific skills on an as-needed based for diagnosis
• Teachers have guidance on what to assess
Assessment Framework
Skills Assessed:
• Phonemic Awareness
• Word recognition
• Oral reading (accuracy and rate)
• Comprehension
• Syntax
Assessment Framework
Links to:
• English Language Development Standards
• English Language Arts Standards
Narrative Oral
Reading
NarrativeReading
Comp
Narrative Oral
Reading
NarrativeReading
Comp
Basic Monitoring Spine of Reading Assessment
Framework
Narrative Oral
Reading
NarrativeReading
Comp
The Dam (decodable word list)
K/1 High Frequency Word List
Rapid Naming
LetterSound
Comp.
BPST
Narrative Oral
Reading
NarrativeReading
Comp
Narrative Listening Comp.
After student demonstrates
mastery of listening
comprehension and word reading
tasks, begin assessing in
connected text skills.
After student demonstrates
mastery of letter sound and naming tasks, begin assessing
regular and irregular word
reading.
1. 2. 3. 4.
Basic Reading Assessment Framework - Kindergarten
Narrative Oral
Reading
NarrativeReading
Comp
Begin the framework with
screening assessments in
listening comprehension, letter sound,
and naming tasks .
Repeat assessing
connected text skills
throughout year as needed.
If problem arises, recheck word reading development
I.I.
I.I.
Reading Assessment Framework with Interventions - Kindergarten
Phonemic Awareness
I.I. I.I.
I.I.
The Dam (decodable word list)
I.I. I.I.
Irregular Word list
Narrative Listening Comp.
The Dam (decodable word list)
K/1 High Frequency Word List
Rapid Naming
LetterSound Comp.
BPST
If student demonstrates skill
level below mastery, provide instructional
intervention and reassess.
Narrative Oral
Reading
NarrativeReading
Comp
Continue assessing
connected text skills
throughout year as needed.
Vocab and Topic
Knowledge
I.I. I.I.
Written Lang.Comp.
(Syntax)
Narrative Oral
Reading
NarrativeReading
Comp
Oral Lang. Comp.
I.I.
I.I.Vocab and
Topic Knowledge
Oral Language
Comp.
San Diego High
Frequency Word List
BPST
Narrative Oral
Reading
NarrativeReading
Comp
Repeat assessing connected text
skills throughout year as needed. If
problem with progress arises, recheck word
reading development.
After student demonstrates mastery
of listening comprehension, letter
sound, and word reading tasks, begin
assessing in connected text skills.
Begin the framework with screening assessments in
listening comprehension,
letter sound, and word reading tasks .
1. 2. 3.
1st and 2nd Grades
Narrative Oral
Reading
NarrativeReading
Comp
San Diego High
Frequency Word List
BPST
Narrative Listening Comp.
Reading Assessment Framework with Interventions- First and Second Grades
The Dam (decodable word list)
Irregular Word list
I.I. I.I.
Vocab and Topic
Knowledge
Written Lang.Comp.
(Syntax)
I.I. I.I.
Vocab and Topic
KnowledgeOral Lang.
Comp.(Syntax)
I.I. I.I.
I.I.
The Dam (decodable word list)
K/1 High Frequency Word List
I.I.
Phonemic Awareness
I.I.
Rapid Naming
I.I.
LetterSound Comp.
I.I.Continue assessing connected text skills throughout year as
needed.
Narrative Oral
Reading
NarrativeReading
Comp
ExampleKindergarten/1st grade BPST
Blending Words with:
Short map rip met rub mop
Final-e fine rope rake cute kite
Long soap leak pain feed ray
r-control fur sort sir tar serve
OVD coin moon round lawn foot
2 syllable silent ladder napkin locate cactus
Reporting
Looking to the Future
• Validation of assessment system
• Development of ASR to include discourse level performances
• Leverage other CRESST technology( QSP)
• Applications to other domains ( e.g., math and science)
• Applications to other grade levels