eHealthResearchPresentation

8/6/2019 eHealthResearchPresentation

1/22

11

Speech Matrix SystemsSpeech Matrix Systems

Automated Speech Recognition Automated Speech Recognitionforfor

SelfSelf--reports in Health Researchreports in Health Research

Al ex Levin, Al ex Levin,Spacegate, Inc.Spacegate, Inc.

al [email protected] [email protected]

Esther Levin Esther Levin Spacegate, IncSpacegate, Inc

City Coll ege of New YorkCity Coll ege of New [email protected]@spacegate.com


2/22

22ConfidentialConfidential Spacegate, Inc.Spacegate, Inc.

Acknowledgement Acknowledgement

Speech Matrix Research & Development is supported by NIH, NCI SBIRSpeech Matrix Research & Development is supported by NIH, NCI SBIRPhase I programPhase I program


3/22


4/22


Electronic Data CaptureElectronic Data Capture

Goals & ChallengesGoals & ChallengesReal data in realReal data in real- -timetime

Data validation Data validation Regulatory complianceRegulatory complianceIncrease clinical trial efficiencyIncrease clinical trial efficiency

Reduce timeReduce time- -toto--market market Cost Cost Patient/Subject burden Patient/Subject burden


5/22


Automated Speech Automated SpeechTechnologyTechnology Science & Art Science & Art

Voice to Data Voice to Data - - voice data entry via automatedvoice data entry via automatednatural dialoguenatural dialogueAutomated Dialogue Automated Dialogue Voice Forms Voice FormsVoice Interface Design Voice Interface Design RealReal--time Monitoring & Reportingtime Monitoring & Reporting

Privacy & SecurityPrivacy & SecurityRegulatory ComplianceRegulatory Compliance


6/22


ASR System for Data Collection ASR System for Data Collection


7/22


Why Speech?Why Speech?Speech is a natural modality of interactionsSpeech is a natural modality of interactionsPhone is user friendly and ubiquitous and no special training for itsPhone is user friendly and ubiquitous and no special training for itsuse is required.use is required.

Dynamic Dialogue FlowDynamic Dialogue Flow

personalization of both content and style based on the profile andpersonalization of both content and style based on the profile andhistory.history.

Real time feedback and monitoringReal time feedback and monitoringrealreal--time reports of captured datatime reports of captured dataAutomated Compliance monitoring Automated Compliance monitoring

Flexible and extensive schedulingFlexible and extensive schedulingInbound/outbound call sessionsInbound/outbound call sessionsthe calls can be initiated by a system following a prescribed protocolthe calls can be initiated by a system following a prescribed protocol

Overall ASR based system offers an extensive and practical tool toOverall ASR based system offers an extensive and practical tool tofacilitate efficient and convenient realfacilitate efficient and convenient real- -time, twotime, two- -waywaycommunications.communications.


8/22


Speech MatrixSpeech Matrix -- VDC VDC SystemsSystems

Extensive data collection system for health, clinical, lifeExtensive data collection system for health, clinical, life

science and behavioral research.science and behavioral research.Another bra n ch of EDC Another bra n ch of EDCRealReal--time data capture i n a participa n t n ativetime data capture i n a participa n t n ativeenv iron me n t env iron me n t


9/22


Applicatio n s Applicatio n s

Pain Mon itorin g Diary (Pain Mon itorin g Diary (PMD PMD )) prototype applicatio n prototype applicatio n -- based o n based o n questio n s in sta n dard pai n questio n ers (BPI)questio n s in sta n dard pai n questio n ers (BPI)Drug use Diary ( VDMDDrug use Diary ( VDMD )) : Prototype applicatio n : Prototype applicatio n -- based o n based o n questio n s in sta n dard Drug Use Diary (DR. Li n da Sobell)questio n s in sta n dard Drug Use Diary (DR. Li n da Sobell)

Impleme n ts Dy n amic Questio nn airesImpleme n ts Dy n amic Questio nn airesIn teracti vity & Commu n icatio n sIn teracti vity & Commu n icatio n sReporti n g a n d Ma n ageme n t Reporti n g a n d Ma n ageme n t


10/22


Dialog Desig nDialog Desig nTask characteristics:Task characteristics:

Need to guara n tee data validity, accuracy a n d in tegrity, taki n g in to accou n t Need to guara n tee data validity, accuracy a n d in tegrity, taki n g in to accou n t speech recog n ition errorsspeech recog n ition errors

impro ve the o verall accuracy usi n g dialog actio n s such as reimpro ve the o verall accuracy usi n g dialog actio n s such as re- -prompts,prompts,con firmatio n s, error ha n dlin g, a n d, if n ecessary, recordi n g a n d flaggi n g thecon firmatio n s, error ha n dlin g, a n d, if n ecessary, recordi n g a n d flaggi n g theun recog n ized uttera n ces for later tra n scriptio nun recog n ized uttera n ces for later tra n scriptio n

The system should accommodate both n ovice a n d experie n ced callersThe system should accommodate both n ovice a n d experie n ced callers

en ough i n formatio n an d help to guara n tee questio n un dersta n din g a n den ough i n formatio n an d help to guara n tee questio n un dersta n din g a n dsuccessful sessio n completio n for n ovice.successful sessio n completio n for n ovice.short a n d effecti ve call flow for experie n ced callershort a n d effecti ve call flow for experie n ced caller

Subjects ide n tify themsel ves i n the begi nn in g of the each sessio n .Subjects ide n tify themsel ves i n the begi nn in g of the each sessio n .Opportu n ity to use the k n owledge accumulated across sessio n s forOpportu n ity to use the k n owledge accumulated across sessio n s forperso n alizatio n .perso n alizatio n .

Subjects may recei ve some trai n in g o n the use of the spoke n dialog systemSubjects may recei ve some trai n in g o n the use of the spoke n dialog systemduri n g the e n rollme n t sessio n .duri n g the e n rollme n t sessio n .

Dialog Design Issues:Dialog Design Issues:con trollin g the captured data accuracy.con trollin g the captured data accuracy.adapti ve le vel of user support adapti ve le vel of user support


11/22


Controlling the Accuracy of Data Capture:Controlling the Accuracy of Data Capture: ASR 101 ASR 101

Speech Recog n ition Grammar Desig nSpeech Recog n ition Grammar Desig nExample: yes/ n o grammar {yes, n o}Example: yes/ n o grammar {yes, n o}

Caller uttera n ce is matched compared to theCaller uttera n ce is matched compared to the

possibilities described by the grammarpossibilities described by the grammarThe output of ASR is the best matchi n g sen te n ce ,The output of ASR is the best matchi n g sen te n ce ,an d a scorean d a scoreIf the score is too low => rejectio nIf the score is too low => rejectio nOut Out--of of--vocabulary uttera n ces ca nn ot be recog n izedvocabulary uttera n ces ca nn ot be recog n izedcorrectlycorrectly

Desig n tradeoff: mi n imize out Desig n tradeoff: mi n imize out--of of-- vocabularyvocabularyan d mi n imize grammar sizean d mi n imize grammar size


12/22


Controlling the Accuracy of Data Capture.Controlling the Accuracy of Data Capture.Improved rejection mechanisms to deal with outImproved rejection mechanisms to deal with out- -ofof--vocabulary utterancesvocabulary utterances ..

System prompt:System prompt: Wa s th a t your l eft shou ld er?Wa s th a t your l eft shou ld er?User:User: no, l eft e l bowno, l eft e l bowSystem prompt:System prompt: I did n t get th a t.I did n t get th a t. Wa s th a t your l eft shou ld er? Pl ea se s a yWa s th a t your l eft shou ld er? Pl ea se s a y yes or no . yes or no .

Relia bl e conf i rma t i on recogn i t i on.Relia bl e conf i rma t i on recogn i t i on.

Using confirmations as the way to control the larger grammar s accuracy.Using confirmations as the way to control the larger grammar s accuracy.

Using recording to capture the outUsing recording to capture the out- -ofof--grammar answers and problematicgrammar answers and problematicuser inputs.user inputs.

System prompt: System prompt: Wa s th a t your l eft shou ld er? Wa s th a t your l eft shou ld er? User: User: No No System prompt: System prompt: Sorry a bout th a t. Let s try i t th i s w a y. Pl ea se choose c a refu ll y a bod y p a rt from theSorry a bout th a t. Let s try i t th i s w a y. Pl ea se choose c a refu ll y a bod y p a rt from thefoll owi ng li st th a t best d escr i bes the l oca t i on of your p ai n, a n d just s a y i t. If none of the l oca t i onsfoll owi ng li st th a t best d escr i bes the l oca t i on of your p ai n, a n d just s a y i t. If none of the l oca t i onsma tch, p l ea se s a y none of those . Here i s the li st: a bd omen

, a nkl es ma tch, p l ea se s a y none of those . Here i s the li st: a bd omen

, a nkl es User (b a rges i n) User (b a rges i n) none of those none of those System prompt:System prompt: Ok. Let me j u st recor d you r a nswer. Pl ea se d escr i be the l oca t i on of yo u r p ai n i n yo u r Ok. Let me j u st recor d you r a nswer. Pl ea se d escr i be the l oca t i on of yo u r p ai n i n yo u rown wor d s. own wor d s. User:User: < >< >System prompt ( af ter recor di ng i s fi n i she d ): System prompt ( af ter recor di ng i s fi n i she d ): Tha nks, I got th a t. Let s move on. Tha nks, I got th a t. Let s move on.

recor d ed u tter a nce i s c a pt u red a n d flagg ed a s tra nscr i pt i on i s nee d ed f or la ter process i n g .recor d ed u tter a nce i s c a pt u red a n d flagg ed a s tra nscr i pt i on i s nee d ed f or la ter process i n g .

sa me mech a n i sm o f fall sa me mech a n i sm o f fall --back to re c ordi n g i nste ad of re c og n i t i on i s u se d af ter sever al repe a te d back to re c ordi n g i nste ad of re c og n i t i on i s u se d af ter sever al repe a te d re c og n i t i on failu res.re c og n i t i on failu res.


13/22


Adaptive Level of User Support Adaptive Level of User Support- -I.I.

P rompt Design.P rompt Design. Wh ere do es it hurt? Wh er e do es it hurt? .. For exampl e, y our h ead For exampl e, y our h ead stomach or back? stomach or back? .. Rem em be r , if you do n t k n ow how Rem em be r , if you do n t k n ow how to a ns w er this qu estio n, j ust say I need h elp .to a n sw er this qu estio n, j ust say I need h elp .

Context sensitive helpContext sensitive help ..help i n formatio n describes a n d clarifies the curre n t questio n,help i n formatio n describes a n d clarifies the curre n t questio n,Provides examples of possible a n swersProvides examples of possible a n swersExample: help for theExample: help for the where does it hurt where does it hurt questio n : questio n : O k ay .O k ay .Here is the help i n formatio n . At this poi n t I n eed to fi n d out theHere is the help i n formatio n . At this poi n t I n eed to fi n d out thepart of y our bod y that hurts the most. Please choose carefull y apart of y our bod y that hurts the most. Please choose carefull y abod y part from the followi n g list that best describes the locatio n bod y part from the followi n g list that best describes the locatio n of y our pai n, an d just sa y it. If n on e of them matches , pleaseof y our pai n, an d just sa y it. If n on e of them matches , pleasesa y n on e of those . Here is the list: abdome n , an k lessa y n on e of those . Here is the list: abdome n , an k les , bac k , ...( list co n tin ues) , toes . , bac k , ...( list co n tin ues) , toes .W hich o n e is it ?" W hich o n e is it ?"


14/22


Adaptive Level of User Support Adaptive Level of User Support- -IIII

Detecting speech recognition failuresDetecting speech recognition failures ..The reThe re--prompts are desig n ed as a n escalati n g list , pro vidin g in creasi n glyprompts are desig n ed as a n escalati n g list , pro vidin g in creasi n glymore i n formatio n an d progressi vely co n strai n in g the caller.more i n formatio n an d progressi vely co n strai n in g the caller.

W here does it h u rt ? For example , you r head , stomach orW here does it h u rt ? For example , you r head , stomach orbac k? . Remember , if you do n t k n ow how to a n sw er thisbac k? . Remember , if you do n t k n ow how to a n sw er thisqu estio n, ju st say I n eed help .qu estio n, ju st say I n eed help . , , I didn t get that. P lease tell me the part of yo u r body that h u rts theI did n t get that. P lease tell me the part of yo u r body that h u rts themost , Remember , you cou ld al w ays say I n eed help most , Remember , you cou ld al w ays say I n eed help , ,

Detecting MisunderstandingsDetecting Misunderstandings ..the user says n o to a co n firmatio n questio n as i n :the user says n o to a co n firmatio n questio n as i n :

System prompt:System prompt: W as that your l eft shou ld er ? W as that your l eft shou ld er ?

User:User: No.No.System prompt:System prompt: Sorry about that. Let s try it this w ay. Pl ease chooseSorry about that. Let s try it this w ay. Pl ease choosecarefu ll y a bo d y part from the fo ll ow in g l ist that best d escribes thecarefu ll y a bo d y part from the fo ll ow in g l ist that best d escribes thel ocatio n of your pai n , a n d just say it. If n on e of them matches,l ocatio n of your pai n , a n d just say it. If n on e of them matches,pl ease say n on e of those . Here is the l ist: ab d ome n , pl ease say n on e of those . Here is the l ist: ab d ome n , (l ist co n tin ues). W hich o n e is it ? (l ist co n tin ues). W hich o n e is it ?


15/22


Adapti ve Adapti ve Level of Level of

User Support User Support- -IIIIIIDialog Personalization .

Taking advantage of alarger inter-sessioncontext by designing

two types datacollection sessions:normal and follow up .Both content and styleare different.Reminding prompts:

Is the pain still in your left shoulder?.


16/22


SystemSystem :: Your Your pinpin pleaseplease. .CallerCaller: : ********SystemSystem: : Ok,Ok, ***** ,***** , letslets begi nbegi n .. Are Are youyou inin

pai npai n n ow?n ow?CallerCaller: : yesyesSystemSystem: : WhereWhere doesdoes it it hurt?hurt? ForFor

example ,example , youryour head ,head , stomstom......CallerCaller: : mymy left left shouldershoulderSystemSystem: : WasWas that that youryour left left shoulder?shoulder?CallerCaller: : yesyesSystemSystem: : PleasePlease raterate youryour pai npain onon aa scalescale of of

zerozero toto te n,te n, withwith zerozero bei n gbei n g n on o pai npain andand

te nte n bei n gbei n g thethe worst worst possiblepossible pai npai nCallerCaller: : it it isis se vense venSystemSystem: : asas that that 77?? PleasePlease saysay yesyes

oror n on o..CallerCaller: : yeahyeahSystemSystem: : HowHow doesdoes thethe pai npai n feel?feel?CallerCaller :: II do n' t don' t kn owkn owSystemSystem :: Okay ,Okay , HereHere isis thethe helphelp i n formatio nin formatio n ..

At At thisthis poi n t poin t II n eedn eed toto fin dfin d out out thethe typetypeof of youryour pai npai n .. PleasePlease choosechoose carefullycarefully aawordword fromfrom thethe followi n gfollowin g list list that that best best describesdescribes youryour pai npai n an dan d just just saysay it it.. If If n othi n gn othi n g matches ,matches , pleaseplease saysay n on e n on e of of those those .. HereHere isis thethe list list:: achi n g,achi n g, < pause>< pause>bur n in g ,bur n in g , < pause>< pause> dulldull

NORMALNORMAL SESSIONSESSIONCallerCaller :: it it feelsfeels sharpsharpSystemSystem :: WasWas that that sharp ? sharp ?

Caller:Caller: yeahyeahSystemSystem: : IsIs thethe pai npain con sta n t con sta n t oror doesdoes it it comecomean dan d go?go?

CallerCaller: : comescomes a n dan d goesgoesSystemSystem: : DoDo youyou ha veha ve a n yan y otherother symptoms?symptoms?CallerCaller: : yeahyeahSystemSystem: : PleasePlease telltell meme youryour most most disturbi n gdisturbi n g

symptomsymptom. . ForFor......CallerCaller: : II ha veha ve fe verfeverSystemSystem: : II heardheard fever fever .. IsIs that that right?right?CallerCaller: : yesyesSystemSystem: : WouldWould youyou likelike toto lea velea ve aa messagemessage

forfor thethe staff?staff?CallerCaller: : n on oSystemSystem: : Tha n ksTha n ks forfor youryour cooperatio ncooperatio n .. I 'v eI'v e

got got allall thethe i n formatio nin formatio n that that II n eedn eed.. I 'llI'll callcallyouyou agai nagai n inin 44 hourshours.. GoodbyeGoodbye..


17/22


SystemSystem: : IsIs thethe pai npai n stillstill inin youryour left left

shoulder?shoulder?CallerCaller: : yesyesSystemSystem: : DoesDoes thethe pai npai n stillstill feelsfeels sharp?sharp?CallerCaller: : n on oSystemSystem: : HowHow doesdoes thethe pai npai n feel?feel?CallerCaller :: It sIt s dulldullSystemSystem :: WasWas that that dull ? dull ?

CallerCaller: : yeahyeahSystemSystem: : IsIs thethe pai npain con sta n t con sta n t oror doesdoes it it comecomean dan d go?go?

CallerCaller: : It sIt s con sta n t con sta n t SystemSystem: : DoDo youyou ha veha ve a n yan y otherother symptoms?symptoms?CallerCaller: : n on oSystemSystem: : WouldWould youyou likelike toto lea velea ve aa messagemessage

forfor thethe staff?staff?CallerCaller: : n on oSystemSystem: : Tha n ksTha n ks forfor youryour cooperatio ncooperatio n .. I 'v eI'v e

got got allall thethe i n formatio nin formatio n that that II n eedn eed.. I 'llI'll callcallyouyou agai nagai n tomorrowtomorrow. . GoodbyeGoodbye..

FOLLOWFOLLOW--UPUP SESSIONSESSION

SystemSystem :: Your Your pinpin pleaseplease. .CallerCaller: : ********SystemSystem: : Hi,Hi, Esther ,Esther , ThisThis isis youryour followfollow--upup callcall

Are Are youyou stillstill ha vin gha vin g pai n ?pain ?CallerCaller: : yehyehSystemSystem: : DidDid youyou taketake youryour pai npain medicatio nmedicatio n today?today?CallerCaller: : yesyesSystemSystem: : What What timetime diddid youyou taketake thethe medicatio n ?medicatio n ?CallerCaller :: TwoTwo hourshours agoago

SystemSystem: : WasWas that that arou n darou n d 99::4545 am?am?CallerCaller: : yehyehSystemSystem: : DidDid thethe medici n emedici n e dimi n ishdimin ish youryour pai n ?pai n ?......

PleasePleaseCallerCaller: : yehyehSystemSystem: : PleasePlease raterate youryour pai npai n onon aa scalescale of of zerozero toto

te n,te n, withwith zerozero bei n gbein g n on o pai npai n an dan d te nte n bein gbein g thetheworst worst possiblepossible pai npain ..

CallerCaller: : fourfourSystemSystem: : WasWas that that 44??CallerCaller: : yeahyeah


18/22


Normal sessio n report Normal sessio n report

CapturedCaptured Value Value ConfirmedConfirmed (yes/no)(yes/no) ConfidenceConfidence ScoreScore

PinPin ******** n on o 6666

Are Are youyou inin pain ?pain ? yesyes n on o 8080

PainPain Locatio nLocatio n left left shouldershoulder yesyes 8686PainPain In te n sityIn te n sity 77 yesyes 8888

PainPain T ypeType sharpsharp yesyes 8888

PainPain con sta n t?con sta n t? pai npai n comescomes a n dan d goesgoes n on o 4747

SymptomsSymptoms fe verfever yesyes 8686

MessageMessage n on en on e n on o 7878


19/22


FollowFollow--up sessio n report up sessio n report

CapturedCaptured Value Value ConfirmedConfirmed ConfidenceConfidence ScoreScore

PinPin ********** n on o 7474

InIn pain ?pain ? yesyes n on o 8585

Medicatio nMedicatio n take n ?take n ? yesyes n on o 7676Medicatio nMedicatio n timetime 99::4545 amam yesyes 6969

Medicatio nMedicatio n helped?helped? yesyes n on o 7575

PainPain Ratin gRatin g 44 yesyes 8787

PainPain Locatio nLocatio n left left shouldershoulder yesyes 8787

Pain

Pain T

ypeT

ype dulldull yesyes 8686PainPain con sta n t?con sta n t? co n sta n t con sta n t n on o 5454

SymptomsSymptoms n on en on e yesyes 8282

MessageMessage n on en on e n on o 8484


20/22


21/22


ResultsResultsSessio nSessio n duratio nduratio n (sec)(sec) 10 5105 ..66((4646..7878))

NumberNumber of of dialogdialog u n itsun its perper sessio nsessio n 77..8 585 ((22..66))

Duratio nDuratio n of of dialogdialog u n it un it (sec)(sec) 1313..4 64 6 ((44 ..5454 ))

DialogDialog tur n stur n s perper dialogdialog u n it un it 11..8888 ((00..4 64 6))

Perce n tagePerce n tage of of tasktask orie n tedorie n ted tur n stur n s 8080%% ((1616))

Perce n tagePerce n tage of of bargedbarged--inin promptsprompts6666%% ((1313))

TimeTime duratio nduratio n of of aa dialogdialog tur ntur n (sec)(sec) 77..1919 ((11..1010))

TimeTime duratio nduratio n of of aa dialogdialog tur ntur n whe nwhe n bargebarge--inin waswasdisableddisabled

1010..6363((11..55 ))


22/22


SummarySummaryASR & Spoke n Dialog Methodology for data ASR & Spoke n Dialog Methodology for datacapture ca n pro vide:capture ca n pro vide:

Additio n al real Additio n al real--time data collectio n tooltime data collectio n toolFlexible protocol desig nFlexible protocol desig nImpro ves data validatio n an d complia n ceImpro ves data validatio n an d complia n ceCen tralized collectio n an d mo n itorin gCen tralized collectio n an d mo n itorin gTelepho n e as ubiquitous de viceTelepho n e as ubiquitous de vice

System desig n n eeds to take i n to accou n t theSystem desig n n eeds to take i n to accou n t thespecificities of the task a n d the limitatio n s of thespecificities of the task a n d the limitatio n s of thetech n ologytech n ology

Flexible le vel of user support Flexible le vel of user support Con trolled accuracy of the captured dataCon trolled accuracy of the captured data

eHealthResearchPresentation

Documents

Transcript of eHealthResearchPresentation