David Pearson and TBALL

Technology Based Assessment of Language and LiteracyPatti Price - Intro - Decoding - Comprehension - Opportunities/Discussion PPRICE Speech & Language Technologywww.pprice.com

45 minutes.if a child sees the word "car" and reads it as "cow", or another reads "sit" as "seat", how do we know if these are pronunciation issues or reading issues? Since no one classroom teacher can be expected to be familiar with as many dialect/ideolect systems as might occur in an urban classroom today, making progress in automated assessments in this area can improve the consistency and fairness of reading assessment.This talk will survey some recent work in using speech and language technology in the area of language and literacy. It will also introduce some issues in this area that are ripe for further research!1PrimatesGorillasHumans

HumansStone toolsFireShelterSpearsLanguage: ~ 50,000 years agoClothingBurialHuntingArt

Farming, 10,000 years agoBronze, 5500 years agoWritten language: ~5 6,000 years ago

Intro Decoding Comprehension DiscussionBut first lets set the stage for language as the first information technology and how important that was2

Pre-language: Information access by direct experienceOral language: Information access also from someone in same time and place with same languageWritten language: Information access not limited by time and placeMultimedia: The power of direct experience, but not limited by time, place, language, learning style, sensory abilities Intro Decoding Comprehension DiscussionTechnology to give best of direct experience AND writing 3Written and Spoken Language are DifferentThese differences are important in learning to read.How difficult it is to spell (many letter combos to one sound)How difficult it is to read (many sounds for the same letter)How closely your dialect matches written conventionsGot to say this for you, you got guts. Guts and no brains. But guts alone dont mean nothin.

I gotta say this for you-- you have guts. Guts and no brains. But guts alone mean nothing.

Ive got to say this for you: You have guts. You have guts and no brains, but guts alone mean nothing.

Intro Decoding Comprehension Discussion4Differences Between Speech and WritingVocabularyDistinct wordsSentence lengthSentence structureGrammarProsodyWrittenSpokenthings stuff20,000+ 2,000+10 30 words ?Complex shallowConservative LiberalA . , ! ? ; : () Rich Intro Decoding Comprehension Discussion5Written and spoken language as dialectsStuff http://www.engl.polyu.edu.hk/corpuslinguist/stuff.htm ~130 words(1) Ive been talking Ive been multiplying matrices already, but certainly time for me to discuss the rules for matrix multiplication.(2) And the interesting part is the many ways you can do it, and they all give the same answer.(3) So its and theyre all important.(4) So matrix multiplication, and then, uh, come inverses.(5) So were uh, we mentioned the inverse of a matrix, but theres thats a big deal.(6) Lots to do about inverses and how to find them.(7) Okay, so Ill begin with how to multiply two matrices.(8) First way, okay, so suppose I have a matrix A multiplying a matrix B and giving me a result well, I could call it C.(9) A times B. Okay.(10) Uh, so, l- let me just review the rule for w- for this entry.

Transcript from a linear algebra lecture (From Glass, Hazen, Lee and Wang, Analysis and Processing of Lecture Audio Data, 2004)

Example~45 wordsWeve been multiplying matrices. Now lets discuss the rules for matrix multiplication. Note that there are many ways to do it and get the same answer. One way to multiply two matrices, A and B, to get a new matrix C is the following rule:

Intro Decoding Comprehension DiscussionAnalysis and Processing of Lecture Audio Data: Preliminary InvestigationsJames Glass, Timothy J. Hazen, Lee Hetherington, and Chao Wang, MIT Computer Science and Artificial Intelligence LaboratoryPlanning issues, vocabulary issues, restarts and pause fillers6Overview of TBALLTechnology Based Assessment of Language & Literacy Child-friendly testing system Measure & analyze child reading & pronunciation 5-8 year olds Native speakers of English & Spanish

Multidisciplinary collaboration: Engineering Education Computer science Linguistics Neuroscience

UCLA, USC, UCB

Intro Decoding Comprehension DiscussionTBALL overview (mention DAM) (5) - Dialect issues and assessment (expand 10)) - Assessment of comprehension (expand 10) - The BARLA paradigm (expand 10) - Opportunities/Discussion (10)

78TBall Goals and Components

Reliable and objective automatic reading assessmentsComprehensive instructional framework for Diagnosis InterventionThree main components:

Childrens interface Teacher interface Assessment module Decoding (DAM) Comprehension (BARLA)

Intro Decoding Comprehension Discussion89Decoding

Comprehension

Need to be balancedIn alphabet-based languages, hints about pronunciationIf you know the language you are reading, the hints may be enough If your first language is another language, and/or if you know the LTS rules in another language, there may be problems.

MOSTLY depends on successful decoding Much more to learn than LTS rules

Intro Decoding Comprehension DiscussionEnglish gives hints about how old English was pronounced, and about morphologyOf the kids I tutor, they typically have trouble with mostly one or the other Those still learning English have trouble with comprehension for many reasons but may be great decoders and may have gotten into the habit of not assigning meaningSpecific reading problems can occur in very bright people, who use the minimal cues they get from decoding to focus on meaning.Assessment of reading - in very early grades, alphabet, letter names, letter sounds, decoding skills, sight words - after about grade 3, comprehension

9DecodingNecessary (but not sufficient) for reading comprehensionNecessary for reliable and efficient processing of textStrongly related to speed of initial reading acquisitionA predictor of difficulties in comprehension developmentAccounts for much variance in reading ability at all ages

Intro Decoding Comprehension Discussion 10English Letter to Sound RulesExample: pronunciations of oPronunciations of the letter o observed in the 100 most frequent words of English

9 OW: go, home, most, no, only, over, so (know, own)6 UW: into, to, who, do, (too, you)6 AH: from, of, other, another, (some, come)6 AO: for, or, on, often, (your, more)5 OU: (about, hour; down, how, now) 3 UH: (good, goodbye would)3 -: (people, work, world)2 AA: got, not1 W AH: one

Intro Decoding Comprehension Discussion11Written Language, ComplexityLanguageLetterSoundEnglishc/s/, /k/hot, car, walk, ah/aa/Spanisha/aa/Frencha/aa/allo, pot, eau /ow/SpellingSoundache + past = ached/ey k/ + /t/aim + past = aimed/ey m/ + /d/paint + past = painted/p ey n t/ + /ax d/ Intro Decoding Comprehension Discussion12English 1 to many and many to 1, Spanish, mostly 1 to 1French, mostly decodable, but hard to spell from earDecoding AssessmentAre the words read correctly? Decodability: closer to one letter one sound is betterIn sample of 100 most frequent words in grade 1 materialm, j, b, d pretty straight forward (always /m jh b d/)Digraphs (2 letters -> 1 sound): -ng, -ck, (wh-)2 letters -> 2 sounds (or): th -> /dh, th/ (thy, thigh) 1 letter -> 2 sounds (and): x -> /k s/ (fox, box)G: get, thing, laugh, largeE: because, get, been, they, great, oftenO: go, do, from, for, how, good, people, got, one

Intro Decoding Comprehension Discussion 13Decoding AssessmentSome words are more decodable than othersHigh frequency words tend to be less decodableHigh frequency words tend to be sight words (not decoded but recognized instantly) (this is important)Decoding skills are also importantIf a word is well-known by a child, it may be treated as a sight word and NOT decoded (tyrannosaurus)Therefore, unless we know which words are sight words for a child, its hard to assess decoding skillsNonsense words must be decodedDecoding Assessment Measure

So DAM!

Intro Decoding Comprehension DiscussionStress decoding skills for everyoneBut letter combinations differ in frequencyDecoding Assessment Measure14Decoding Issues: Dialect, Idiolect and AssessmentIf the child has trouble with the letter 'r' (common in English), and says 'cow' when shown 'car' is it a misreading? If the child is from Boston and reads 'Carl' as 'Call' is it a misreading? What if the teacher has never heard anyone from Boston? if the child sees the word 'ask' and says 'aks'? if the child is a native speaker of another language, e.g., Spanish, and says 'seat' when presented the word 'sit', is it a misreading? if the child is already learning to read, e.g., Spanish, and sees the word 'sore' and pronounces it 'sore-ay', is it a misreading? SOLUTION: Map the child's system, whatever it is Augment the dictionary to diagnostic labels.

Intro Decoding Comprehension DiscussionComplicating factors: - if the child has trouble with pronouncing the letter 'r' or 'w' (common in English), and says 'cow' when shown 'car' is it a misreading or a mispronunciation? - if the child is from Boston and reads 'Carl' as 'Call' is it a misreading? What if the teacher has never heard anyone from Boston? - if the child sees the word 'ask' and says 'aks'? - if the child is a native speaker of another language, e.g., Spanish, and says 'seat' when presented the word 'sit', is it a misreading or a mispronunciation? - if the child is already learning to read in another language, e.g., Spanish, and sees the word 'sore' and pronounces it 'sore-ay', is it a misreading? SOLUTION: map the child's system, whatever it is and map the dictionary to diagnostic labels.

- if the child loves dynosaurs, tyranosaurus may be a sight word (assessing reading skill does not generalize very well to decoding new words) - for very frequent words, it's hard to tell - one person's arcane word is another's nonsense word; one person's well known word could be a nonsense word for someone else. SOLUTION: assess on nonsense (or very rare words) to stress decoding, generalizable skills - diagnose skills from simple . b always is pronounced b in English to more difficult . see lts.xls in work/dictionaries15Dialect Modeling in ASRDialect is a systemThe system is not well modeled in ASR

These are being merged

When the point is the Contrast systemHead sections adapted from Gunnar Fants Acoustic Theory of Speech Production Intro Decoding Comprehension Discussion1617We are lumping all the whites together. When in fact, here, the black on the left = the white on the rightWhat is important is the relative contrast between light and dark.Now, on to something even more slippery meaning

Decoding Issues: Dialect, Idiolect and Assessment

Grape g r ey pGrape g w ey p Grape g r aa p ey Bird b er dBird b uh d Bird b iy r d Rocks w aa k s Horse h ow s Three th w iyRun w ah n

Intro Decoding Comprehension Discussion18Decoding Resultsnug: /n uw g/ - Sp. LTS, phonology, or short/long?

jore, tay, bap: /hh ao r ey, t ay, b aa p/ - Sp. LTSyan, vag, hine: /zh aa n, b aa g, ay n/ - Sp. Phon.rin, bap, mot: /r iy n, b ey p, m ow t/ - short/long

jore: /zh ow r ey/ - mixed Spanish phonology, LTSjick: /hh ih s k/ - Spanish LTS, misapplied English LTS

Intro Decoding Comprehension Discussion19Comprehension AssessmentGRADE 2: Kim has a dog. The dog's name is "Dot." Dot is a very white dog, and he has a black dot on his leg. Sam, Ben, and Kim like to play with Dot. They play ball with Dot all day long. Ben lost the ball in the mud, and Dot went into the mud to get the ball. Now, Dot is a brown dog from the mud!

What does Dot look like? A DalmatianBrownI dont rememberLike a dogA white dogA white dog with a black dot on his legWhite and a dot on his legWell hes white and he has a brown dot on his legLike a rat

Please describe Dots physical appearance? Intro Decoding Comprehension DiscussionLike a dog, white are popular.. White with black dot on legThe point is: people dont necessarily agree on correct answerThose who say exactly what was in the text may be parroting without understanding, others may be extrapolating Last: as intro to next in the guise of being more precise, we could make the question not understandable, and then we dont know if the child had trouble with the text or with the question. Also, choice of whether to ask question, or to write it.

20

Motivation/DesiderataEquivalent measures for reading or listening (to compare)Separate understanding of probe from understanding of text (probe as picture adds interest and is not text or speech)Diagnose reading error patterns (by selection of alternate items)Easy to automate (compared to most comprehension tests)Items ascend in difficulty (several errors in a row stops the test)Many items (to allow retesting without repeating items)Equal number of plausible/implausible items (to mitigate guessing)PIAT-R (Peabody Individual Achievement Test Revised) is best match, but Not automated Reading onlyThey did not respond to our request

So BARLA! Intro Decoding Comprehension DiscussionPeabody Individual Achievement Test -- Revised21 Intro Decoding Comprehension Discussion

VocabularySyntaxMorphologyAccent adjustmentProsodyFind wordsPrint adjustmentPunctuationDecoding (LTS)

In your first language, this might be easierIn a new language, this might be easierOral language= listening and speaking (and some sight); written language = reading, writingComprehension (purple glow) is easier to assess than production (speaking and writing)So, thats the goal, assess language skills for reading and listening, and separate out understanding of item vs. question for language learners and native speakers22BARLA Example (pot and box reversed)(dog in box)(box on boy)(correct)

The pot is on a box.Hear OR Read:The pot is on a box. Intro Decoding Comprehension Discussion23BARLA ExampleHere is a fire.tire (rhymes)fire (correct)fur (vowel differs)fairy(shares, f, i, r)

Hear or read:Here is a fire. Intro Decoding Comprehension Discussion24Decoding OpportunityWe sampled the space with 25 itemsHigh Frequency Word: tell, napLow Frequency Word: fell, zapNonsense Word: zell, fapHowever, desirable would beFrequent retesting (perhaps in a game environment)Diagnostic of which decoding skills need more workItems need to be generated and coded for diagnosis

Intro Decoding Comprehension Discussion25An Opportunity!

Intro Decoding Comprehension DiscussionContentGamePedagogyUser Model

Letter sounds, English:a /ae/b /b/c /k/, /s/z /z/By font, audio files Levels: Lower case simple to complex Upper case Digraphs Move up if: 100% correct Move down if: 3 wrong in a row

Last playedSounds masteredDiagnosisPreferencesEngine

bpd

bpd26An Opportunity!

Intro Decoding Comprehension DiscussionContentGamePedagogyUser Model

Vocabulary, English:bat Levels: Lower case simple to complex Upper case Digraphs Move up if: 100% correct Move down if: 3 wrong in a row

Last playedSounds masteredDiagnosisPreferencesEngine

bpd

btaSimplify discussion of text simplification read some articlesMake this into a comprehension example27A Comprehension Opportunity!Data collected for sentences analogous to PIAT-R But not analyzedParadigm ready to hatch more uses For more information about the project: http://nautilus.icsl.ucla.edu/tball/

For more information about the data:iseli@ucla.edu

Intro Decoding Comprehension Discussion28An Opportunity!

Intro Decoding Comprehension DiscussionNatural language processing and reading augmentation29Simplify VocabularyBy "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. ViewBy "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify bothBy "adding to people's brains" we mean adding to the way we deal with difficult things, to add to understanding according to our needs, and to figure out solutions to problems. Intro Decoding Comprehension Discussion30Simplify SyntaxBy "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. ViewBy "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify bothBy "augmenting human intellect" we mean increasing the capability of a person to approach a complex problem situation. We also mean how a person gains comprehension to suit his particular needs and derives solutions to problems. Intro Decoding Comprehension Discussion31Simplify Vocab & SyntaxBy "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. 1. Select Text2. Select View3. ViewBy "augmenting human intellect" we mean increasing the capability of a man to approach a complex problem situation, to gain comprehension to suit his particular needs, and to derive solutions to problems. Increased capability in this respect is taken to mean a mixture of the following: more-rapid comprehension, better comprehension, the possibility of gaining a useful degree of comprehension in a situation that previously was too complex, speedier solutions, better solutions, and the possibility of finding solutions to problems that before seemed insoluble. And by "complex situations" we include the professional problems of diplomats, executives, social scientists, life scientists, physical scientists, attorneys, designers--whether the problem situation exists for twenty minutes or twenty years. We do not speak of isolated clever tricks that help in particular situations. Simplify vocabulary Simplify syntax Simplify bothBy "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. Intro Decoding Comprehension Discussion32Simplify Vocabulary & Syntax: ResultBy "adding to people's brains" we mean adding to how we deal with difficult things. We also mean adding to understanding that is suited to our needs, and to how we can figure out solutions to problems. By adding to what we can do, we mean a mix of several things. Faster and better understanding are examples. We also mean making it possible to get some understanding when before it was too difficult. And, we mean faster solutions, better solutions, and being able to find solutions that didn't seem possible before. By "difficult situations" we mean the problems people have in different kinds of jobs. The jobs can be in the government, running businesses, being scientists, lawyers, or designers. The problems can be there for twenty minutes or twenty years. We do not mean clever tricks that help in only some situations. We mean a way of life. In this new way of life many things are useful. Useful things include hunches, cut-and-try, things we can't touch, and our "feel for a things. These things are more useful with great ideas, ways to talk about the ideas and ways to write them, intelligent ways of doing things, and powerful technologies to help us.The number of people and what they create are growing fairly quickly. But our problems are getting harder even more quickly. Also, the importance of finding solutions becomes greater because things are changing more quickly than they used to change. Things that before happened only in one place now are spreading to the whole world. This also makes it important to find solutions. If we could make our brains better in this way, any intelligent person would run as fast as possible to get it. To make them want to run, we need to show a way to do it. We also need to show what we might get out of it. Intro Decoding Comprehension Discussion33Summary of Views and Reading Ease

Original

Words

239

Sentences

7

Word/sentence

34.1

Characters/ word

5.6

Reading Ease

10.7%

Flesch-Kincaid Grade Level

12.0

Vocabu-lary

Syntax

Both

275

249

315

7

15

22

39.2

16.6

14.3

4.7

5.6

4.4

42.3%

28.2%

71.7%

12.0

12.0

6.8

Intro Decoding Comprehension Discussion34Flesch reading ease: 206.835 1.015 (tot words/tot sents) 84.6 (tot sylls/tot words)Reader's Digest magazine has a readability index of about 65, Time magazine scores about 52, an average 6th grade student's (an 11-year-old) written assignment has a readability test of 6070 (and a reading grade level of 67), and the Harvard Law Review has a general readability score in the low 30s.Flesch-Kincaid = .39 (tot words/tot sents) + 11.8 (tot sylls/tot words) + 15.59

Questions?

Intro Decoding Comprehension Discussion35Chart10000001

humansstone toolsfirespearsjewelryleapHumans to the leap

Sheet1Timeline: Human Evolution, 25 August 2006 NewScientist.com news service , John Pickrell100000http://www.newscientist.com/channel/being-human/human-evolution/in165timenumnberwhat(85,000,000.00)-850-8.51primateswhatprimatesgorillashumans(2,000,000.00)-20-0.21humansMYA-55-7-5.8(200,000.00)-2-0.021modern0.0001now1000000gorillashumansstone toolsfireshelterspearsjewelry-55-45-35-25-15-50-7-5.8-2.5-1.6-0.5-0.4-0.11primatesgorillashumans1000000-5-4-3-2-10humansshelterspearsjewelryleapcave artfarmingbronze-0.5-0.4-0.11-0.05-0.033-0.01-0.0055clothingburialhunting

Sheet1111111

primatesgorillashumansstone toolsfirespearsEvolution, primates to humans, millions of years ago

Sheet2000001

humansstone toolsfirespearsjewelryleapHumans to the leap

Sheet3000010.50.2

stone toolsfirespearsjewelryleapfarmingbronzeMYAStone tools to Bronze