AQUAINT Testbed

John Aberdeen, John Burger,Conrad Chang, Scott Mardis

The MITRE Corporation

© 2002, The MITRE Corporation

Testbed Activities• Testbed

– Provide access to Q&A systems on classified data.– Solicit user feedback and conduct user studies.

• Testweb– Provide public access to a broad variety of Q&A capabilities.– Evaluate systems and architectures for inclusion in testbed.

• User Studies– Obtain feedback from users on utility and usability

• Architecture support

Testbed

• Classified network (ICTESTNET)– access to users, data, scenarios will be restricted

• Evaluate systems prior to installation– Testweb is a good first step – MITRE installations are more than rehearsal

Testbed Progress

• MITRE installations– QUIRK (CYCorp/IBM) – Javelin (CMU)– LCC

• Assessing– Portability to the IC environment– Maintainability– Features– Resources– Configuration

Deployment Issues

• Restrictive version requirements• Fixed pathname/hostname issues• Configure scripts

• Remote debugging (installation, configuration) is inefficient

• Differences in machine configuration, environment• Unknown or unrealized dependencies• Schedule conflicts

• Firewalls and other security issues

Developing a Testbed Vision

• USNR Analyst Task• MITRE gains first-hand knowledge about:

– analyst’s working environment• tools• methods of analysis

– nature of analyst’s tasking– long-term analysis goals

• MITRE obtains expert feedback about:– suitability of QA tools for analysis tasks– representativeness of CNS scenarios– methodology for further tool assessment– methodology for future user studies

Plan for USNR Visit• Discussions of analyst's history with analysis tasks• Hands-on presentation of QA and Search tools• MITRE and USNR analyst jointly work through 1 to 2 CNS

scenarios using QA tools, search tools, telephone, etc.• Analyst formally works through 1 to 3 more CNS

scenarios on their own– MITRE monitors the activity through logging and survey

instruments– The analyst provides feedback so adjustments can be provided as

necessary

• The analyst and MITRE will jointly analyze the results• MITRE and the analyst develop a draft test plan

Testweb

Q/Asystem

CNS

TREC2002

Javelin, LCC,Qanda, TextMap

Q/APortal/Demo

Q/A repository

Google API

Othercollections

IRservice

Clarityservice

UsersQ/APortal/Demo

•Loose federation of systems

•Simple services over TCP/HTTP/SOAP

Q&A Demo

CNS Data

http://aquaint.mitre.org/cns

UMASS Clarity

• Many thanks to Steve Cronen-Townsend & Bruce Croft• Clarity measures …• Clarity service

– SOAP interface– Access via Google Java client – Data sets: AQUAINT CNS, TREC-11 (soon)

• User interface– For demonstration, simple output of clarity score– To come, low clarity warning

CNS Search w/ UMASS Clarity

http://aquaint.mitre.org/clarity

IR Service

• Utilizing Google SOAP API• Tied SOAP to Lucene search• Data sets: AQUAINT CNS, TREC-11• Access via Google Java client

Q/A Architecture

Meeting: April 22, 2003 @ IBM in Hawthorne, NY

Eric Nyberg (CMU), David Ferrucci & Chris Welty (IBM), Scott Mardis & John Burger (MITRE)

Topics:– Javelin– PIQUANT– Catalyst– UIMA– AQUAINT Testbed

Some Directions

• Architecture is important– Enable collaboration– May be task dependent

• Common Components/Subsystems– Question analysis– Query formulators– IR– Taggers – Knowledge sources

MITRE’s Mission on Architecture

• Encourage collaborations– Promote regularization of interfaces

• Facilitate broader use of best components

• Advocate for users (IC analysts) and user tasks

• Coordinate with Ex. Comm.

MITRE’s Roles on Architecture

Services

CommunitySupport

APIs, Standards

InfrastructureResources

Data Resources

MITRE’s Roles on Architecture

• User logs, including Q&A, simple search, documents viewed, etc.

• Extended WN & XWN APIs

• Query formulation API• Corpora URIs

Services,Libraries

APIs,Standards

InfrastructureResources

Data Resources

APIs, Standards:CommunitySupport

MITRE’s Roles on Architecture

• IR• Clarity• Unit conversions ?• Time normalization ?

CommunitySupport

InfrastructureResources

Data Resources

Services, Libraries:

APIs,Standards

Services,Libraries

MITRE’s Roles on Architecture

• Modified CNS corpus• Ontology library ?

CommunitySupport

InfrastructureResources

Data Resources:

APIs,Standards

Services,Libraries

DataResources

MITRE’s Roles on Architecture

• Middleware tools ?

Infrastructure Resources:

APIs,Standards

Services,Libraries

DataResources

CommunitySupport

InfrastructureResources

MITRE’s Roles on Architecture

• Tool website (index)• Wiki for architecture ?• Mailing lists

InfrastructureResources

Community Support:

APIs,Standards

Services,Libraries

CommunitySupport

DataResources

Component Offerings

• LCC– Ontology tools (development & browsing)– Extended WordNet (XWN)

• ISI– Question analysis (QTarget)– Question paraphraser– Query formulator– Web-based IR– Answer identifier (extractor)

Component Offerings (cont’d)

• IBM– NE tagger (Resporator – 80 types)

• CyCorp– Answer sanity checking

– Taxonomy (well-developed for CNS/WMD)– Unit conversion– Cyc concept paraphrases

• UMASS– Clarity

Component Needs ?

• IR subsystems• User context & history representation• More Ontologies & Tools• Answer combination• Entity normalization (time, location, organization)• Structured data services (DBs)• Other media

Many Faces of AQUAINT• Leverage all of AQUAINT’s successes• Expose a greater range of interfaces to analysts

– They may find something they can use now– Make the Q/A system more transparent

• Exploiting the middle ground– Queries that are more complex than boolean search, but more

structured than free natural language questions