Conversational Internet - Creating a natural language interface for web pages
-
Upload
dale-lane -
Category
Technology
-
view
4.241 -
download
3
description
Transcript of Conversational Internet - Creating a natural language interface for web pages
![Page 1: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/1.jpg)
© 2009 IBM Corporation
Conversational Internet:A natural language interface for webpages
Dale Lane – IBM United Kingdom
14 May 2013
![Page 2: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/2.jpg)
© 2012 IBM Corporation2
Challenge
![Page 3: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/3.jpg)
© 2012 IBM Corporation3
Challenge
![Page 4: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/4.jpg)
© 2012 IBM Corporation4
Motivation
![Page 5: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/5.jpg)
© 2012 IBM Corporation5
Understanding the page : Identifying type
![Page 6: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/6.jpg)
© 2012 IBM Corporation6
Understanding the page : Identifying navigation options
![Page 7: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/7.jpg)
© 2012 IBM Corporation7
Understanding the page : Identifying calls-to-action
![Page 8: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/8.jpg)
© 2012 IBM Corporation8
Understanding the user : Retrieving information
![Page 9: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/9.jpg)
© 2012 IBM Corporation9
Understanding the user : Mouse actions
![Page 10: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/10.jpg)
© 2012 IBM Corporation10
Understanding the user : Mouse actions
![Page 11: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/11.jpg)
© 2012 IBM Corporation11
Understanding the user : Keyboard actions
![Page 12: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/12.jpg)
© 2012 IBM Corporation12
Understanding the user : Keyboard actions
![Page 13: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/13.jpg)
© 2012 IBM Corporation13
Understanding the user : Keyboard actions
![Page 14: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/14.jpg)
© 2012 IBM Corporation14
Conversational Internet
Presenting an early-stage prototype being developed to explore the potential for question answering as an alternative approach to screen-readers for retrieving information from web pages
Architecture and approach inspired by active area of research and development in question answering on a knowledge derived from a corpus of documents(Ferucci, Lally, Chu-Carroll, et al) http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6177717
![Page 15: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/15.jpg)
© 2012 IBM Corporation15
Implementation
browser
extension
UIMA
LanguageWare
Java
client server
![Page 16: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/16.jpg)
© 2012 IBM Corporation16
Implementation
new request new request
response with
conv. idresponse
follow-up request
request with
conv. id
“what can I do?”
“the options are...”
“I want to do...”
![Page 17: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/17.jpg)
© 2012 IBM Corporation17
Implementation : Client
Firefox extension
Submits current state of the page to server for analysis
Training mode
![Page 18: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/18.jpg)
© 2012 IBM Corporation18
Implementation : Server
Analyzing the page
Processing user queries
![Page 19: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/19.jpg)
© 2012 IBM Corporation19
Implementation : Server : Understanding the page
![Page 20: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/20.jpg)
© 2012 IBM Corporation20
Implementation : Server : Understanding the page
What type of site is this?
Machine learning classifiers
Whitelists of known domains
![Page 21: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/21.jpg)
© 2012 IBM Corporation21
Implementation : Server : Understanding the page
What can we infer from markup used?
Semantic tags
ARIA
CSS class names
![Page 22: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/22.jpg)
© 2012 IBM Corporation22
Implementation : Server : Understanding the page
What does the structure of the page layout suggest?
Machine learning models for common page elements
![Page 23: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/23.jpg)
© 2012 IBM Corporation23
Implementation : Server : Understanding the page
What does the text of the page tell us?
Natural Language Processing using LanguageWare to recognize common forms of call-to-action
![Page 24: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/24.jpg)
© 2012 IBM Corporation24
Implementation : Server : Responding to queries
![Page 25: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/25.jpg)
© 2012 IBM Corporation25
Implementation : Server : Responding to queries
Interpreting the query
NLP rules created with LanguageWare to map to closest known command type
WordNet to attempt matches using synonyms of unknown terms
![Page 26: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/26.jpg)
© 2012 IBM Corporation26
Implementation : Server : Responding to queries
Extracting the requested information
Information requested from page extracted from serialized CAS created by first pipeline, by retrieving sections with relevant annotations
![Page 27: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/27.jpg)
© 2012 IBM Corporation27
Implementation : Server : Responding to queries
Preparing a response
Speech generated using Nuance NDev and streamed to client
![Page 28: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/28.jpg)
© 2012 IBM Corporation28
Future work / Limitations
Use cases
Usability testing
RIA / AJAX sites
![Page 29: Conversational Internet - Creating a natural language interface for web pages](https://reader034.fdocuments.in/reader034/viewer/2022051209/547c34a75906b59f798b46b7/html5/thumbnails/29.jpg)
© 2012 IBM Corporation29
Paper submitted to W4A – http://goo.gl/3X2iv
Overview presentation– http://youtu.be/uS6oquJdgbw
Demonstration of the prototype – http://youtu.be/tSGyPCcO-bY
Dale Lane
@dalelane