Kathy McCoyKathy McCoy

Artificial IntelligenceArtificial Intelligence

Natural Language ProcessingNatural Language Processing

Applications for People with Applications for People with DisabilitiesDisabilities

Primary Research AreasPrimary Research Areas

Natural Language GenerationNatural Language Generation – problem – problem of choice. of choice. Deep Generation --- structure and content of Deep Generation --- structure and content of

coherent textcoherent text Surface Generation – particularly using TAG Surface Generation – particularly using TAG

(multi-lingual generation and machine translation)(multi-lingual generation and machine translation)

Discourse ProcessingDiscourse Processing Second Language AcquisitionSecond Language Acquisition Applications for people with disabilitiesApplications for people with disabilities

affecting their ability to communicateaffecting their ability to communicate

ProjectsProjects Augmentative CommunicationAugmentative Communication – –

Center for Applied Science and Center for Applied Science and Engineering in Rehabilitation (ASEL) – Engineering in Rehabilitation (ASEL) – Word Prediction and Contextual Word Prediction and Contextual Information (Keith Trnka, (Jay McCaw), Information (Keith Trnka, (Jay McCaw), Chris Pennington, Debbie Yarrington)Chris Pennington, Debbie Yarrington)

ICICLEICICLE – CALL system for teaching – CALL system for teaching English as a second language to ASL English as a second language to ASL natives (Rashida Davis, Charlie natives (Rashida Davis, Charlie Greenbacker)Greenbacker)

Text SkimmingText Skimming – for someone who is blind – for someone who is blind to skim a document to find an answer to a to skim a document to find an answer to a question (Debbie Yarrington).question (Debbie Yarrington).

Generating Textual Summaries of GraphsGenerating Textual Summaries of Graphs – (Sandee Carberry, Seniz Demir)– (Sandee Carberry, Seniz Demir)

Developing Developing Intelligent Intelligent

Communication Communication Aids for People Aids for People with Disabilitieswith DisabilitiesKathleen F. McCoyKathleen F. McCoy

Computer and Information Sciences & Center for Applied Science and Engineering in Rehabilitation

University of Delaware

Augmentative Augmentative CommunicationCommunication

Intervention that gives non-speaking person Intervention that gives non-speaking person an alternative means to communicatean alternative means to communicate

User PopulationUser Population May have severe motor impairmentsMay have severe motor impairments

Unable to speakUnable to speak Unable to writeUnable to write Cannot use sign languageCannot use sign language

May have cognitive impairments and/or May have cognitive impairments and/or developmental disabilitiesdevelopmental disabilities

May be too young to have developed May be too young to have developed literacy skillsliteracy skills

Row-Column ScanningRow-Column Scanning

Row-Column Scanning IIRow-Column Scanning II

Language Language Representation: WordsRepresentation: Words

Still Need to Spell!Still Need to Spell!

Predicting Fringe Predicting Fringe VocabularyVocabulary

Word Prediction of Spelled Words Word Prediction of Spelled Words (infrequent context-specific words)(infrequent context-specific words)

MethodsMethods Statistical NLP MethodsStatistical NLP Methods Learning from the context of the Learning from the context of the

individualindividual Other Contextual CluesOther Contextual Clues

Geographic Location, Time of Day, Geographic Location, Time of Day, Conversational Partner, Topic of Conversational Partner, Topic of Conversation, Style of the DocumentConversation, Style of the Document

Prediction ExamplePrediction Example

Trigram Model: P(w|h)=P(w|wTrigram Model: P(w|h)=P(w|w--

22 w w-1-1))

Can we do better??Can we do better??

Intuitively all possible words do not Intuitively all possible words do not occur with equal likelyhood during a occur with equal likelyhood during a conversation.conversation.

The topic of the conversation affects The topic of the conversation affects the words that will occur. the words that will occur. E.g., when talking about baseball: ball, E.g., when talking about baseball: ball,

bases, pitcher, bat, triple….bases, pitcher, bat, triple…. How often do these same words occur How often do these same words occur

in your algorithms class?in your algorithms class?

Topic ModelingTopic Modeling

Goal: Automatically identify the topic Goal: Automatically identify the topic of the conversation and increase the of the conversation and increase the probability of related words and probability of related words and decrease probability of unrelated decrease probability of unrelated words.words.

QuestionsQuestions Topic RepresentationTopic Representation Topic IdentificationTopic Identification Topic ApplicationTopic Application Topic Language Model UseTopic Language Model Use

Topic Modeling Topic Modeling ApproachApproach

Topic IdentificationTopic Identification

Topic IdentificationTopic Identification

Topic ApplicationTopic Application

How do we use those similarity How do we use those similarity scores?scores?

Essentially weight the contribution Essentially weight the contribution of each topic by the amount of of each topic by the amount of similarity that topic has with the similarity that topic has with the current conversation.current conversation.

Results Using TopicsResults Using Topics

Current WorkCurrent Work

What happens with significantly larger What happens with significantly larger corpora?corpora?

What other kinds of tuning to the user What other kinds of tuning to the user can we do:can we do: RecencyRecency StyleStyle

Does keystroke savings translate into Does keystroke savings translate into communication rate enhancement?communication rate enhancement?

Text SkimmingText Skimming

Debra Yarrington, Kathleen Debra Yarrington, Kathleen McCoyMcCoy

Problem:Problem: Blind and dyslexic individuals cannot skim textBlind and dyslexic individuals cannot skim text

Example: “What’s the syntax for calling a function Example: “What’s the syntax for calling a function with template parameters?” (skimming through with template parameters?” (skimming through code)code)

““Why was Ayers Rock renamed?”Why was Ayers Rock renamed?” ““What type of tree produces leaves with three What type of tree produces leaves with three

distinct shapes?”distinct shapes?” ““Where can I find more information about Portugal?”Where can I find more information about Portugal?”

People who cannot read text rely on People who cannot read text rely on screen readers (Jaws, Window-Eyes)screen readers (Jaws, Window-Eyes) braille outputbraille output

more difficult to come by more difficult to come by extremely bulky to carry aroundextremely bulky to carry around

Example of Jaws Output at 400 Example of Jaws Output at 400 wpmwpm

LinkLink““What psychological and philosophical significance should we attach to What psychological and philosophical significance should we attach to

recent efforts at computer simulations of human cognitive capacities? In answering recent efforts at computer simulations of human cognitive capacities? In answering this question, I find it useful to distinguish what I will call "strong" AI from "weak" or this question, I find it useful to distinguish what I will call "strong" AI from "weak" or "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the computer in the study of the mind is that it gives us a very powerful tool. For example, computer in the study of the mind is that it gives us a very powerful tool. For example, it enables us to formulate and test hypotheses in a more rigorous and precise fashion. it enables us to formulate and test hypotheses in a more rigorous and precise fashion. But according to strong AI, the computer is not merely a tool in the study of the mind; But according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other computers given the right programs can be literally said to understand and have other cognitive states. In strong AI, because the programmed computer has cognitive states, cognitive states. In strong AI, because the programmed computer has cognitive states, the programs are not mere tools that enable us to test psychological explanations; the programs are not mere tools that enable us to test psychological explanations; rather, the programs are themselves the explanations.rather, the programs are themselves the explanations.

I have no objection to the claims of weak AI, at least as far as this article is I have no objection to the claims of weak AI, at least as far as this article is concerned. My discussion here will be directed at the claims I have defined as those of concerned. My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states and that the programs thereby explain human cognition. When I has cognitive states and that the programs thereby explain human cognition. When I hereafter refer to AI, I have in mind the strong version, as expressed by these two hereafter refer to AI, I have in mind the strong version, as expressed by these two claims.claims.

I will consider the work of Roger Schank and his colleagues at Yale (Schank I will consider the work of Roger Schank and his colleagues at Yale (Schank & Abelson 1977), because I am more familiar with it than I am with any other similar & Abelson 1977), because I am more familiar with it than I am with any other similar claims, and because it provides a very clear example of the sort of work I wish to claims, and because it provides a very clear example of the sort of work I wish to examine. But nothing that follows depends upon the details of Schank's programs. The examine. But nothing that follows depends upon the details of Schank's programs. The same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human mental phenomena.”mental phenomena.”

Proposed Solution:Proposed Solution:

A system that takes a question and a A system that takes a question and a document or a few documents, and document or a few documents, and returns a small set of text links where returns a small set of text links where potential answers to the question might potential answers to the question might be foundbe found

In order to accomplish this, we will In order to accomplish this, we will potentially use:potentially use: Techniques used in existing Question Techniques used in existing Question

Answering systemsAnswering systems Data collected from skimming text with an Data collected from skimming text with an

eye tracking deviceeye tracking device

ExampleExample

Gaze PlotGaze Plot

linklink

Hot SpotsHot Spots

WhatWhatArtArtMiddleMiddleinfusedinfusedpurpose purpose with with also servedalso servedpeople believedpeople believedwriting doeswriting doeswho readwho readSculpture. The missionSculpture. The missionas well as decorateas well as decorateBiblical talesBiblical taleslessons tolessons towerewerechurch sculpture; church sculpture; animalsanimalslifelife““Green man” peeringGreen man” peeringcarefullycarefullywroughtwroughtforthforthRomanesque eraRomanesque eraclassicalclassicalconventionsconventionsof figuresof figures

RomanesqueRomanesqueAt the beginningAt the beginningera the style ofera the style ofarchitecturearchitecturethat was in voguethat was in vogueKnown as Romanesque Known as Romanesque because it copied the patternbecause it copied the patternproportionproportionof the architectureof the architecturethe Roman Empirethe Roman Empirechief characteristics of the chief characteristics of the Romanesque style wereRomanesque style werevaults, round arches,vaults, round arches,and few windowsand few windowsThe easiest point to lookThe easiest point to lookfor is the rounded arch, seen for is the rounded arch, seen in door openingsin door openingswindowswindowsIn generalIn generalchurches were heavychurches were heavyCarrying about them an airCarrying about them an airsolemnity andsolemnity andThese earlyThese earlytapestries ortapestries orlook closelylook closely

werewereFrance called it “gothic”France called it “gothic”was a reference was a reference Ransacked RomeRansacked RometwilighttwilightarchitecturalarchitecturalRomanesqueRomanesquevaultsvaultsincorporatedincorporatedof windowof windowThe easiest point of The easiest point of archarchdoors. Alsodoors. Alsolater Gothiclater Gothicveryveryespecially theespecially thethethechurcheschurchesoutdo eachoutdo eachofofFor theFor theconstruction, througtconstruction, througtThe architectThe architectsame placesame place

Text SkimmingText Skimming

Debra Yarrington, Kathleen Debra Yarrington, Kathleen McCoyMcCoy

Problem:Problem: Blind and dyslexic individuals cannot skim textBlind and dyslexic individuals cannot skim text

Example: “What’s the syntax for calling a function Example: “What’s the syntax for calling a function with template parameters?” (skimming through with template parameters?” (skimming through code)code)

““Why was Ayers Rock renamed?”Why was Ayers Rock renamed?” ““What type of tree produces leaves with three What type of tree produces leaves with three

distinct shapes?”distinct shapes?” ““Where can I find more information about Portugal?”Where can I find more information about Portugal?”

People who cannot read text rely on People who cannot read text rely on screen readers (Jaws, Window-Eyes)screen readers (Jaws, Window-Eyes) braille outputbraille output

more difficult to come by more difficult to come by extremely bulky to carry aroundextremely bulky to carry around

Example of Jaws Output at 400 Example of Jaws Output at 400 wpmwpm

LinkLink““What psychological and philosophical significance should we attach to What psychological and philosophical significance should we attach to

recent efforts at computer simulations of human cognitive capacities? In answering recent efforts at computer simulations of human cognitive capacities? In answering this question, I find it useful to distinguish what I will call "strong" AI from "weak" or this question, I find it useful to distinguish what I will call "strong" AI from "weak" or "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the "cautious" AI (Artificial Intelligence). According to weak AI, the principal value of the computer in the study of the mind is that it gives us a very powerful tool. For example, computer in the study of the mind is that it gives us a very powerful tool. For example, it enables us to formulate and test hypotheses in a more rigorous and precise fashion. it enables us to formulate and test hypotheses in a more rigorous and precise fashion. But according to strong AI, the computer is not merely a tool in the study of the mind; But according to strong AI, the computer is not merely a tool in the study of the mind; rather, the appropriately programmed computer really is a mind, in the sense that rather, the appropriately programmed computer really is a mind, in the sense that computers given the right programs can be literally said to understand and have other computers given the right programs can be literally said to understand and have other cognitive states. In strong AI, because the programmed computer has cognitive states, cognitive states. In strong AI, because the programmed computer has cognitive states, the programs are not mere tools that enable us to test psychological explanations; the programs are not mere tools that enable us to test psychological explanations; rather, the programs are themselves the explanations.rather, the programs are themselves the explanations.

I have no objection to the claims of weak AI, at least as far as this article is I have no objection to the claims of weak AI, at least as far as this article is concerned. My discussion here will be directed at the claims I have defined as those of concerned. My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states and that the programs thereby explain human cognition. When I has cognitive states and that the programs thereby explain human cognition. When I hereafter refer to AI, I have in mind the strong version, as expressed by these two hereafter refer to AI, I have in mind the strong version, as expressed by these two claims.claims.

I will consider the work of Roger Schank and his colleagues at Yale (Schank I will consider the work of Roger Schank and his colleagues at Yale (Schank & Abelson 1977), because I am more familiar with it than I am with any other similar & Abelson 1977), because I am more familiar with it than I am with any other similar claims, and because it provides a very clear example of the sort of work I wish to claims, and because it provides a very clear example of the sort of work I wish to examine. But nothing that follows depends upon the details of Schank's programs. The examine. But nothing that follows depends upon the details of Schank's programs. The same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's same arguments would apply to Winograd's SHRDLU (Winograd 1973), Weizenbaum's ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human ELIZA (Weizenbaum 1965), and indeed any Turing machine simulation of human mental phenomena.”mental phenomena.”

Proposed Solution:Proposed Solution:

A system that takes a question and a A system that takes a question and a document or a few documents, and document or a few documents, and returns a small set of text links where returns a small set of text links where potential answers to the question might potential answers to the question might be foundbe found

In order to accomplish this, we will In order to accomplish this, we will potentially use:potentially use: Techniques used in existing Question Techniques used in existing Question

Answering systemsAnswering systems Data collected from skimming text with an Data collected from skimming text with an

eye tracking deviceeye tracking device

WhatWhatArtArtMiddleMiddleinfusedinfusedpurpose purpose with with also servedalso servedpeople believedpeople believedwriting doeswriting doeswho readwho readSculpture. The missionSculpture. The missionas well as decorateas well as decorateBiblical talesBiblical taleslessons tolessons towerewerechurch sculpture; church sculpture; animalsanimalslifelife““Green man” peeringGreen man” peeringcarefullycarefullywroughtwroughtforthforthRomanesque eraRomanesque eraclassicalclassicalconventionsconventionsof figuresof figures

RomanesqueRomanesqueAt the beginningAt the beginningera the style ofera the style ofarchitecturearchitecturethat was in voguethat was in vogueKnown as Romanesque Known as Romanesque because it copied the patternbecause it copied the patternproportionproportionof the architectureof the architecturethe Roman Empirethe Roman Empirechief characteristics of the chief characteristics of the Romanesque style wereRomanesque style werevaults, round arches,vaults, round arches,and few windowsand few windowsThe easiest point to lookThe easiest point to lookfor is the rounded arch, seen for is the rounded arch, seen in door openingsin door openingswindowswindowsIn generalIn generalchurches were heavychurches were heavyCarrying about them an airCarrying about them an airsolemnity andsolemnity andThese earlyThese earlytapestries ortapestries orlook closelylook closely

werewereFrance called it “gothic”France called it “gothic”was a reference was a reference Ransacked RomeRansacked RometwilighttwilightarchitecturalarchitecturalRomanesqueRomanesquevaultsvaultsincorporatedincorporatedof windowof windowThe easiest point of The easiest point of archarchdoors. Alsodoors. Alsolater Gothiclater Gothicveryveryespecially theespecially thethethechurcheschurchesoutdo eachoutdo eachofofFor theFor theconstruction, througtconstruction, througtThe architectThe architectsame placesame place

Current DirectionsCurrent Directions

Have collected eye-tracking data from close Have collected eye-tracking data from close to 100 people (on several documents each)to 100 people (on several documents each)

Analysis quite interesting – enough data to Analysis quite interesting – enough data to find patterns in where the skimmers are find patterns in where the skimmers are looking.looking.

Analyzing data with “text tiling methods” to Analyzing data with “text tiling methods” to pick out places in the text where “same pick out places in the text where “same thing” being discussed.thing” being discussed.

Incorporate question extraction techniquesIncorporate question extraction techniques How to present this to the user?How to present this to the user?

ModelingModeling the the Acquisition of English Acquisition of English in the ICICLE Systemin the ICICLE System

Kathleen F. McCoyDepartment of Computer and Information

SciencesUniversity of Delaware

PeoplePeople

Current PeopleCurrent People Rashida DavisRashida Davis Charlie GreenbackerCharlie Greenbacker

OthersOthers Chris Pennington, Dan Blanchard, Mike Bloodgood, Chris Pennington, Dan Blanchard, Mike Bloodgood,

Greg Silber, Meghan Boyle, Mohamed Mostagir, Greg Silber, Meghan Boyle, Mohamed Mostagir, Stephanie Baker, Heejong Yi, David DermanStephanie Baker, Heejong Yi, David Derman

Graduates: Matthew Huenerfauth, Jill Janofsky, Graduates: Matthew Huenerfauth, Jill Janofsky, Lisa Masterman Michaud, Litza Stark, David Lisa Masterman Michaud, Litza Stark, David SchneiderSchneider

The ICICLE ProjectThe ICICLE Project

IInteractive nteractive CComputer omputer IIdentification and dentification and

CCorrection of orrection of LLanguage anguage EErrorsrrors Interactive writing tutor for native Interactive writing tutor for native

signers of American Sign Language signers of American Sign Language (ASL)(ASL)

Purpose: analyze student-written Purpose: analyze student-written English texts and provide individualized English texts and provide individualized feedback and instruction on grammarfeedback and instruction on grammar

The ICICLE ProjectThe ICICLE Project

system provides student with system provides student with tutorial instruction on the errorstutorial instruction on the errors

student has opportunity to make student has opportunity to make corrections and request re-corrections and request re-analysisanalysis

TheICICLESystem

student provides piece of textstudent provides piece of text system analyzes text for system analyzes text for

grammatical errorsgrammatical errors

Cycle of user input, system Cycle of user input, system responseresponse

Current ImplementationCurrent Implementation

the student enters text

here

the system shows which

sentences have errors

explanations shown here

Writing From Deaf Writing From Deaf StudentsStudents

Literacy is a serious issue for the Deaf Literacy is a serious issue for the Deaf population.population.

Lots of variation in level of acquisition.Lots of variation in level of acquisition. Marked Differences from writing of Marked Differences from writing of

hearing peers.hearing peers. Dropped be: Dropped be: She really pretty.She really pretty. Missing Possessives: Missing Possessives: She age is 13.She age is 13. Subject/verb agreement, plural markers, Subject/verb agreement, plural markers,

determiners: determiners: She really like go with friend to She really like go with friend to mall.mall.

Work on ICICLEWork on ICICLE

Previous work focused on developing Previous work focused on developing grammar and mal-rules and modeling grammar and mal-rules and modeling the user’s level of acquisition (so the user’s level of acquisition (so different analyses can be found different analyses can be found depending on it)depending on it)

Current WorkCurrent Work Tutorial ResponsesTutorial Responses Probabilistic Parsing – need help!Probabilistic Parsing – need help! NEED SYSTEM HELP!!!!!NEED SYSTEM HELP!!!!!

What Mal-Rules do We What Mal-Rules do We Use?Use?

BeginnerBeginner:: Over-application of auxiliary Over-application of auxiliary IS, missing simple present IS, missing simple present morphology:morphology: She She teachesteaches piano on Tuesdays. piano on Tuesdays.

IntermediateIntermediate:: Botched progressive Botched progressive tense:tense: She is teachShe is teachinging piano on Tuesdays. piano on Tuesdays.

AdvancedAdvanced:: Botched passive voice: Botched passive voice: She is She is taughttaught piano on Tuesdays. piano on Tuesdays.

“She is teach piano on Tuesdays.”

Current DirectionsCurrent Directions

Have collected eye-tracking data from close Have collected eye-tracking data from close to 100 people (on several documents each)to 100 people (on several documents each)

Analysis quite interesting – enough data to Analysis quite interesting – enough data to find patterns in where the skimmers are find patterns in where the skimmers are looking.looking.

Analyzing data with “text tiling methods” to Analyzing data with “text tiling methods” to pick out places in the text where “same pick out places in the text where “same thing” being discussed.thing” being discussed.

Incorporate question extraction techniquesIncorporate question extraction techniques How to present this to the user?How to present this to the user?