Multimodal user interfaces: Implementation Chris Vandervelpen chris.vandervelpen@uhasselt.be.

download Multimodal user interfaces: Implementation Chris Vandervelpen chris.vandervelpen@uhasselt.be.

of 16

  • date post

    16-Jan-2016
  • Category

    Documents

  • view

    216
  • download

    0

Embed Size (px)

Transcript of Multimodal user interfaces: Implementation Chris Vandervelpen chris.vandervelpen@uhasselt.be.

  • Multimodal user interfaces: ImplementationChris Vandervelpenchris.vandervelpen@uhasselt.be

  • OverviewIntroductionVoiceXmlX+VFrom models to X + VDemo: ACCESS NetfrontConclusionsQuestions

  • IntroductionFocus on speech/direct manipulation on mobile deviceHow can we deploy a multi modal UIBuild our own framework using speech synthesizer/recognizers that interpret the designed models (reinventing the wheel)

    Build software that generates standardized markup from the models (use existing technologies) start point

  • VoiceXmlMarkup language for speech only interfacesTelephone interfacesUsing grammars for speech recognitionJava Speech Grammar Format (JSGF)Nuance Grammar Specification Language (NGSL)Speech outputSynthesisPrerecorded audio http://www.voicexml.org

  • VoiceXml = brussels | antwerp | amsterdam; ]]> What departure city do you like?? For example, brussels, antwerp or amsterdam Your departure city is

  • VoiceXmlMixed-initiative formsSingle user input for several fieldsSupports more natural languageFor example I want to fly from brussels to amsterdamFilling in departure_city and destination_city fields

  • X + VX + VXHtml: visual channelVoiceXml snippets: speech channelSynchronization between modalities using Xml EventsMultimodal browsers supporting X+VACCESS Netfront multimodal browser (PocketPC)Operahttp://www.voicexml.org/specs/multimodal/x+v/12/

  • X + V

  • X + V = brussels | antwerp | amsterdam; ]]> What departure city do you like?? For example, brussels, antwerp or amsterdam .

  • X + VAlso usable with XFormsVoiceXml snippets and XForms influence same XForms instance model synchronization

  • Models to X + V

  • Models to X + VAnnotate UI description for speech [Shao2003: Transcoding HTML to VoiceXML Using Annotations]Extend this approach to UIML and X + VIdentify particular information structuresText areasMenu/List structuresTop-level visual regionDefine their representation in XHTML and VoiceXmlGenerate the synchronization XML eventing code

  • Model to X + VDefine a generic UIML widget vocabulary mapping for both GUI and speech [Plomp2002]TextEntry (VoiceXml) (XHtml)System.Windows.Forms.TextBoxCollection (VoiceXml) (XHtml)System.Windows.Forms.Panel

  • DemoAccess Netfront multimodal browserPocketPCOrdering pizzaOrdering Chinese

  • ConclusionsX + Vbuilt-in modality synchronizationalternative to own multimodal implementationdeclarativetransformation from UIML possible

  • Questions?