02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and...
-
Upload
monique-trueblood -
Category
Documents
-
view
218 -
download
0
Transcript of 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and...
![Page 1: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/1.jpg)
02/04/09 Danica Damljanović 1
Natural Language Interfaces to conceptual models: usability and performance
Danica Damljanović
![Page 2: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/2.jpg)
University of Sheffield NLP
Outline
• NLIs to KBs and their usability
• QuestIO – Question-based Interface to Ontologies Demo and evaluation
• Towards better usability using FREYA
• Conclusion
202/04/09 Danica Damljanović
![Page 3: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/3.jpg)
University of Sheffield NLP
Natural Language Interfaces
• Allow users to interact with a system using written or spoken language to perform tasks which require knowledge of a formal query language NLIs to structured data:
• NLIs to DBs: TEAM, PRECISE• NLIs to KBs
NLIs to semi-structured data• Open-domain question answering systems
Other:• E.g. NLC: replacement for a programming language• Dialog and tutoring systems
![Page 4: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/4.jpg)
University of Sheffield NLP
Danica Damljanović
NLIs to knowledge bases
(Kaufmann and Bernstein, 2007)
• Natural Language Interfaces preferred to keywords, menu-guided, and graphical interfaces
(Linckels, 2007): • keywords preferred
to NL interfaces
![Page 5: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/5.jpg)
University of Sheffield NLP
8Danica Damljanović
NLIs to KBs
![Page 6: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/6.jpg)
University of Sheffield NLP
9Danica Damljanović
Usability of NLIs
• Who uses NLIs? Application developers: customisation End users: search
• Usability Effectiveness Efficiency User satisfaction
![Page 7: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/7.jpg)
University of Sheffield NLP
10Danica Damljanović
Usable NLIs to KBs: challenges
• Robustness
• Portability
• What to show?
• Understanding information need
• Habitability
![Page 8: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/8.jpg)
University of Sheffield NLP
12Danica Damljanović
Habitability
• Can the User and the System speakthe same language?
![Page 9: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/9.jpg)
University of Sheffield NLP
14Danica Damljanović
Question-based Interface to Ontologies
![Page 10: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/10.jpg)
University of Sheffield NLP
16Danica Damljanović
NL --> SPARQL query
Filtering concepts
Ranking concepts
Query Creator
Query Execution
![Page 11: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/11.jpg)
University of Sheffield NLP
An Example
1.15
1.19
compare
![Page 12: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/12.jpg)
University of Sheffield NLP
Demo
• http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/questio/questio.html
• http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/prototype-tutorial/prototype-tutorial.html
20Danica Damljanović
![Page 13: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/13.jpg)
University of Sheffield NLP
User-centric task-based evaluation
• Training: using video tutorials
• 12 participants, 4 tasks: 3 defined, e.g. “Find runtime parameters of
Cebuano gazetteer.” 1 free task: “Think of any task that you would
like to perform using this prototype.”
• Measured: Efficiency, Effectiveness and User satisfaction
26Danica Damljanović
![Page 14: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/14.jpg)
University of Sheffield NLP
Efficiency: average time per task
![Page 15: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/15.jpg)
University of Sheffield NLP
Effectiveness: how successfully the tasks were finished?
![Page 16: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/16.jpg)
University of Sheffield NLP Post –task survey: question 1
![Page 17: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/17.jpg)
University of Sheffield NLP
Post –task survey: question 2
![Page 18: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/18.jpg)
University of Sheffield NLP
Post –task survey: question 3
![Page 19: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/19.jpg)
University of Sheffield NLP
Post –task survey: question 4
![Page 20: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/20.jpg)
University of Sheffield NLP
Evaluation: conclusion• Browsing ontology mostly helpful – but
only for users who are familiar with ontologies
• Refinement pane: primitive, still favourable for defined tasks
• Tasks not finished: in cases when the user information need was
not precisely expressed (undefined tasks) When the answer was not in the ontology
34Danica Damljanović
![Page 21: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/21.jpg)
University of Sheffield NLP “find out which are the runtime parameters of Cebuano Gazetteer”
• “cebuano gazetter parameters”• “ What are the runtime parameters of cebuano
gazetteer?“• “what are the parameters of cebuano gazetteer?”• “Cebuano gazetteer runtime parameters“• “Runtime parameters of cebuano gazetteer“• “Cebuano runtime parameters“• “Cebuano gazeteer“>>“Cebuano gazetteer“
Find parameters by browsing the ontology35Danica Damljanović
![Page 22: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/22.jpg)
University of Sheffield NLP
“runtime parameters of Cebuano Gazetteer”
• “Cebuano runtime parameters“GATE plugin Resource Parameter
Processing Resource
Contains
Resource hasInitTimeParameter
hasRuntimeParameter
![Page 23: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/23.jpg)
University of Sheffield NLP
Undefined tasks
• “Developer of Tokeniser”• ”Projects about GATE ”, ”GATE web site”• ” Tokenizer ” vs. ” Tokeniser ”• ” Gazetteer ” vs. ”Gazeteer ”• ”horacio saggion publications” vs. ”horacio
saggion articles”• ”Author of morphological analyser ” >>
”Developer of morphological analyser ” >> ”Developer and morphological analyser ”
3704/06/09 Danica Damljanović
![Page 24: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/24.jpg)
University of Sheffield NLP
Next steps: FREyA
38Danica Damljanović
FREyA (Feedback, Refinement, Extended vocabulary, Agregation)
![Page 25: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/25.jpg)
University of Sheffield NLP
Example
• the user's query: Cities in Europe
• FREyA: Cities >>locatedIn >>Countries >> locatedIn >>
Europe Cities >>locatedIn >> Countries >> partOf >> Europe
• Attempto: which cities are located in countries that are
part of Europe?
3904/06/09 Danica Damljanović
![Page 26: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/26.jpg)
University of Sheffield NLP
Challenges
• Return answers in real time: which rivers flow through Germany >>
which rivers flow through (cities in) Germany
• Refinement model - an ontology? with concepts such as query (hasAnswer)
answer, refinement, feedback, subject of query (main subject), domain, context.
find out what the user wants and instantiate the refinement ontology.
40Danica Damljanović
![Page 27: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/27.jpg)
University of Sheffield NLP
41Danica Damljanović
Expected contribution
• Usability improvement of NLIs: Eliminate training Improve the performance by implementing
user-system interaction using FREYA:• help the user to easily familiarize himself
with the system capabilities• express his need more precisely, in a way
which is understandable by the system
![Page 28: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/28.jpg)
University of Sheffield NLP
42Danica Damljanović
Thank you
• Questions?
![Page 29: 02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović danica@dcs.shef.ac.uk.](https://reader030.fdocuments.in/reader030/viewer/2022032516/56649c7e5503460f94933b60/html5/thumbnails/29.jpg)
University of Sheffield NLP
References• (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M.
Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001.
• (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004.
• (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM.
• (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006.
• (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007.
• (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.