Using an Ontology to Support Evaluation of Soldier-Worn Sensor Systems for ASSIST

Randolph Washington

DCS Corporation 1330 Braddock Pl

Alexandria, VA USA rwashing@dcscorp.com

Christopher Manteuffel DCS Corporation 1330 Braddock Pl

Alexandria, VA USA cmanteuffel@dcscorp.com

Christopher White DCS Corporation 1330 Braddock Pl

Alexandria, VA USA cwhite@dcscorp.com

Abstract--ASSIST (Advanced Soldier Sensor Information Systems Technology) is a DARPA-funded effort whose goal is to exploit soldier-worn sensors to augment the soldier’s recall and reporting capability to enhance situation understanding. An ontology is a data model that represents a domain and is used to reason about the concepts in that domain and the relations between them. It is a mechanism that enables a community to share a common conceptual model of a domain to facilitate communication of knowledge in that domain. This paper provides an overview of the development and use of an ASSIST ontology to support the evaluation of ASSIST technologies. Keywords: DARPA, ASSIST, Ontology, evaluation methodology

I. INTRODUCTION

The Advanced Soldier Sensor Information Systems and Technology (ASSIST) program is a Defense Advanced Research Projects Agency (DARPA) advanced technology research and development program. The objective of the ASSIST program is to exploit soldier-worn sensors to augment a soldier’s recall and reporting capability to enhance situational understanding in military operations in urban terrain (MOUT) environments. The program is split into two tasks:

• Task 1, named Baseline System Development, stresses active information capture and voice annotations exploitation. The resulting products from Task 1 will be prototype wearable capture units and the supporting operational software for processing, logging and retrieval.

• Task 2, named Advanced Technology Research, stresses passive collection and automated activity/object recognition. The results from this task will be the algorithms, software, and tools that will undergo system integration in later phases of the program.

The National Institute of Standards and Technology (NIST) Intelligent Systems Division (ISD), along with NIST’s subcontractors (Aptima and DCS Corporation), are funded to serve as the Independent Evaluation Team (IET) for Task 2. As the IET for Task 2, NIST is responsible for:

• Understanding the Task 2 contractor technologies

• Determining an approach for testing their technologies

• Identifying a MOUT site to evaluate the technologies

• Devising and executing the tests • Analyzing the data and documenting the

outcome The following three metrics are the focus for the IET’s Task 2 evaluation:

1) The accuracy of object/event/activity identification and labeling

2) The system’s ability to improve its classification performance through learning

3) The utility of the system in enhancing operational effectiveness

To evaluate the ASSIST systems of the three Task 2 research teams, the IET developed a two-part test methodology to produce these metrics. Metrics 1 and 2 were evaluated through “elemental tests”, and metric 3 was evaluated through “vignette tests”. Elemental tests were designed to measure the progressive development of the ASSIST system’s technical capabilities; and vignette tests were designed to predict the impact these technologies will have on warfighter’s performance in a variety of missions and job functions. In specifying the detailed procedures for each elemental and vignette test, the IET attempted to define evaluation strategies that would provide a reasonable level of difficulty for system and soldier performance at both the 6-month

172

(November 2005) and 12-month (May 2006) evaluations. This paper describes how an ontology is being used to support the evaluation of ASSIST technologies. Section II of this paper defines an ontology and identifies the objectives of the ASSIST ontology. Section III provides an overview of DCS’ approach to developing the ASSIST Ontology. Section IV describes the use of the ASSIST Ontology during the 6-month and 12-month technology evaluations. Section V concludes the paper and identifies recommended future work in this area.

II. ASSIST ONTOLOGY OBJECTIVES

In computer science, an ontology is a data model that represents a domain and is used to reason about the concepts in that domain and the relations between them. It is a mechanism that enables a community to share a common conceptual model of a domain to facilitate communication of knowledge in that domain. A good ontology focuses on defining the meaning of concepts through its relationship with other concepts rather than merely enumerating concepts. An ontology together with a set of individual instances of the concepts constitutes a knowledgebase. Since a knowledgebase is strictly defined by the ontology, a knowledgebase can be easily understood by consumers of the knowledge. Therefore, ontologies are particularly useful in the exchange of knowledge between computer systems. The development of languages to build ontologies and populate a knowledgbase has recently been a very active area for standards organizations such as World Wide Web Consortium [1]. NIST has been active in the development and application of ontologies and had previously collaborated with DCS on the development of an ontology for defining the behavior of intelligent ground combat vehicle systems for the U.S. Army Tank Automotive Research Development, and Engineering Center. As the ASSIST program was focused on the automated acquisition and distribution of knowledge within the dismounted infantry domain, NIST believed that the development of an ontology for the ASSIST program would be beneficial and contracted with DCS to support the effort. The objectives of DCS’ ASSIST Ontology effort were to:

1. Elicit and document the specific knowledge requirements in the ASSIST Ontology

2. Develop the ASSIST Ontology using formal knowledge representation techniques and standardized tools and representations

3. Apply the ASSIST ontology for evaluating the results of the developmental technology.

III. ASSIST ONTOLOGY DEVELOPMENT

DCS’ approach to developing the ASSIST Ontology was to A) conduct an analysis of the information requirements in the ASSIST domain to identify the concepts and relationships to be included in the ontology; B) survey available ontology languages and tools and select the ones most appropriate for the ASSIST Ontology; and C) design an ontology that efficiently captures the key ASSIST Domain concepts for the purpose of performance evaluation of the ASSIST systems and implement the Ontology in the selected language. Each of these steps is described in the following paragraphs.

A. Information Requirements Analysis

In order to evaluate the information requirements for the ASSIST Ontology, DCS participated in discussion on potential applications of the ASSIST technologies NIST conducted with soldiers recently returned from Iraq. The focus of these discussions was identifying the kinds of information dismounted infantry soldiers were expected to recall at the end of a patrol through potentially hostile urban environments and identifying how this information was conveyed to other soldiers to support the development of an intelligence assessment of the area and to provide beneficial information for subsequent patrols. It was assumed that reports made by soldiers supported by ASSIST technology would be aggregated with reports from other soldiers on the same patrol as well as other patrols over periods of time. A key part of the analysis was identifying information that, when combined with other reports, would support the identifications of patterns of hostile or friendly activities. In the course of these discussions there was a general assumption of the kinds of technologies to be applied to capture the information but no presumption of the format in which it was to be conveyed. The soldiers discussed how information is currently collected by soldiers (eyes, ears, cameras, GPS) and conveyed to squad/platoon leaders who in turn convey the information to battalion intelligence officers. The soldiers provided lists of concepts (i.e., objects, actions (both of the soldiers and external objects), and relations) they felt would be significant to be conveyed to the unit’s intelligence officer. At the same time the soldiers where working with NIST to define evaluation vignettes for the ASSIST technologies. These vignettes included most of the

173

significant objects and actions identified by the soldiers. DCS analyzed the soldier’s lists and the evaluation vignettes and generated a list of concepts that needed to be contained in the ASSIST ontology to enable a full description of the environment and activities of the vignettes from the ASSIST-wearing soldier’s perspective.

B. Ontology Language Format and Tools Selection

DCS evaluated several current ontology development languages including opencyc [2], KIF [3], UML [4], and Web Ontology Language (OWL) [5] for use in development of the ASSIST Ontology. Key criteria in the evaluation were descriptive ability, ease of human use, ease of computer use, reliance on open standards, and availability of associated open-source environments focused on knowledge capture and retrieval. OWL, an XML-based markup language for publishing and sharing data using ontologies on the Internet, was selected as the best language for the ASSIST Ontology. OWL is a vocabulary extension of the Resource Description Framework (RDF) and is derived from the DAML+OIL (DARPA Agent Markup Language, Ontology Interchange Language) Web Ontology Language developed by DARPA. The OWL specification is maintained by the World Wide Web Consortium (W3C). OWL provides somewhat intuitive mechanisms for defining classes, properties, property restrictions, and individuals that are more than adequate to support the ASSIST domain. In addition, being based on XML it is simple to translate OWL documents to other formats using eXtensible Stylesheet Language Transformations (XSLT). A number of open-source OWL editors including (Protégé-OWL, SWOOP, and OILEd) were evaluated for use in development of the ASSIST Ontology. Protégé [6] with the OWL-plug-in was selected as the most suitable development environment due to the maturity of its ontology-editing GUI, the availability of a wide variety of plug-ins for data inferencing and visualization, and the ease of integrating user applications through Protégé’s plug-in framework. Protégé and the OWL plug-ins are developed by Stanford University. DCS took advantage of this plug-in architecture by developing and integrating a custom query tab that supported advanced queries of ASSIST knowledgebases.

C. Ontology Development

Given the long list of important concepts and relationships identified during the information requirements analysis, there are many ways to organize these terms in an ontology. As the ASSIST technologies are primarily intended to aid soldiers observe things on the battlefield, it was decided to organize the ASSIST ontology around the concept of an observation. An Observation is a description of one or more related actions or objects, or lack thereof, which is made by the assisted soldier at a specific time. An observation may be supported by one or more multimedia files. An observation may also be related to an event which may be a part of another event.

Fig. 1 illustrates the top level of the ASSIST ontology. ASSIST concepts are shown as black boxes and ASSIST relationships are shown as blue lines between concepts. Asterisks by a blue arrow indicate a possible one-to-many relationship. A negative relationship between an observation and an action or object (i.e., hasNotObservedObject or hasNonAction) is necessary to report the observed lack of something (i.e., there were no children in the village). The ontology provides many sub-concepts for Action and Object that allow the Observation to be very specific and allows other objects and data properties to be associated with the observed object. The primary driver for the creation of sub-concepts was the vignette descriptions. During the creation of the knowledgebases representing individual after-action reports by the soldiers generated during the evaluations, new concepts where added to the ASSIST ontology if suitable concepts did not exist to express all of the observations. In this fashion the ontology was iteratively improved as more and more of the after-action reports were described with it.

Fig. 2 shows a breakdown of the resulting major subclasses of SpatialThing and Fig. 3 shows the breakdown of the major resulting subclasses of Action. Note that SpatialThing has spatial relations to itself allowing a knowledgebase to identify the relative locations of objects. In addition, multiple observations of the same object could be correlated by associating a name with the object. Also note that object relationships can be chained to form complex descriptions such as “observed a human being named Joe, who wore a white Dishdasha and carried an AK-47, in front of the building with two floors that was located west of the village marketplace”.

174

Fig. 1. Ontology Class Diagram

Fig. 2. Spatial Things Class Diagram

Fig. 3. Action Class Diagram

175

The ASSIST ontology was generated in the OWL language using Protégé. The resulting XML file can be found on the world-wide-web at www.isontology.org/ISOntology/ASSIST/Assistv10.owl. Knowledgebases reflecting observations made during evaluation vignettes were also created using Protégé.

IV. EVALUATION SUPPORT

The ASSIST ontology was used during the ASSIST technologies evaluation to quantify the amount of tactical information conveyed by soldiers during the vignette tests. This allowed the information content of after-action reports and intelligence officer debriefs generated without the aid of ASSIST technologies to be compared to the information content of the same reports and debriefs generated with the aid of each research teams ASSIST systems. During the 6-month evaluation, two vignettes based on presence patrols were executed by a squad of two fire teams. After each vignette, each fire team generated an after-action report without access to the information from the ASSIST technologies. Each fire team then generated a list of additional information that they would have liked to have known but could not directly recall. This list of additional information requirements was provided to each ASSIST research team. One after another, each ASSIST research team was then given the opportunity to show the fire teams the information requested using their ASSIST system. DCS observers listened to each fire team generate the unassisted after-action report and recorded each of the observations stated by members of the fire team. The DCS observers then listened to each ASSIST research team as they addressed the fire team’s information requests. Observations identified by the research team’s ASSIST system that corresponded to the fire team’s information requests were recorded by the DCS observers. After execution of the vignettes, DCS generated ASSIST ontology knowledgebases documenting the unassisted after-action reports from each fire team for each vignette. DCS also generated separate ASSIST ontology knowledgebases documenting the additional observations provided by each research team’s ASSIST system for each fire team for each vignette. Each of the 12 resulting knowledgebases was analyzed to quantify the number of observations, actions, observed objects, properties of each observed object, and multimedia files. The data showed that the soldiers were capable of recalling a lot of the militarily important observations from their exercises

and that the user interfaces of the vendors ASSIST systems were capable of detecting many of the militarily significant objects in the vignettes, but were not capable of describing the relationships between the objects, or describing the objects in as much detail as the soldiers were. It was also noted that as the ASSIST technology observations were recalled based on soldier inputs, these observations tended to include multimedia files (images and audio clips) of objects and actions already observed by the fire teams for the purpose of confirming the soldier’s observations. During the twelve-month evaluation, two vignettes based on presence patrols were again executed by a squad of two fire teams. After each vignette the squad generated an after-action report without access to the information from the ASSIST technologies. As the soldiers were provided two digital cameras which were not considered as part of the ASSIST technology during this evaluation, the unassisted after-action reports also included selected images from the digital cameras. After the unassisted after-action report was generated for each vignette it was given to a soldier acting as the unit intelligence officer. After the intelligence officer reviewed the report he interacted with one soldier from the squad to clarify information contained in the after-action report. The intelligence officer then generated a list of additional information that he would have liked to have known but could not get directly from the after-action report or through the clarifications of the soldier. This list of additional information requirements was provided to each ASSIST research team. One after another, each ASSIST research team was then given the opportunity to show the intelligence officer the information requested using their ASSIST system. DCS observers listened to the squad generate the unassisted after-action report for each vignette and recorded each of the observations stated by members of the squad. The DCS observers then listened to each ASSIST research team as they addressed the intelligence officer’s information requests. Observations identified by the research teams that corresponded to the intelligence officer’s information requests were recorded by the DCS observers. After execution of the vignettes, DCS generated ASSIST ontology knowledgebases documenting the unassisted squad after-action reports for each vignette. DCS also generated separate ASSIST Ontology knowledgebases documenting the additional observations provided by each ASSIST research team for the intelligence officer for each vignette. Each of the eight resulting knowledgebases

176

was analyzed to quantify the number of observations, action, observed objects, properties of each observed object, and multimedia files. The data again showed that the soldiers were capable of recalling a lot of the militarily important details from their exercises and, given hand held digital cameras, had captured still images of many of the significant observed objects. The ASSIST systems provided by each research team were able to provide varying amounts of supplemental information in response to the intelligence officer requests for additional information. While each of these ASSIST systems had collected large quantities of data, the actual amount of information conveyed to the intelligence officer was limited by the significant amount of information contained in the squad’s after-action report and the limited ability to search for information regarding specific objects and relationships between those objects with the ASSIST systems. DCS also generated ground-truth ASSIST ontology knowledgebases to support the evaluations. These knowledgebases documented the test environment during a vignette rather than merely what the soldiers observed. This allowed the independent evaluation team to determine how much progress had been made from the 6-month to the 12 month evaluations in terms of environmental complexity and the ASSIST systems ability to characterize that complexity. The number of objects of various categories placed in the MOUT site for the 12-month evaluation (May 2006) and 6-month evaluation (November 2005) are shown in Table 1. For comparison purposes Table 1 also lists the number of each category of objects expected to be found, on the average, for a site of the same area as the test site in the Iraq Governorates of Babylon and Baghdad based on a UN Development Program report (“Iraq Living Conditions Survey 2004”). As can be seen in the table, the total number of objects used in Vignette 1 in the May 2006 evaluation was approximately four times the number of objects used in the November 2005 evaluation. Also, the total number of objects used in Vignette 1 in the May 2006 evaluation was approximately 18 times the number of objects expected, on average, in an equivalent area in the rural Babylon Governorate and one-half the number of objects expected, on average, in an equivalent area in the very urban Baghdad Governorate.

May

2006November

2005 Babylon BaghdadPeople 50 14 3.45 108.41Vehicles 23 8 0.11 5.49Radio 3 0 0.29 12.69Television 1 0 0.41 17.23Video Player 1 0 0.12 7.20Bicycle 1 0 0.04 1.33Firearms 2 0 0.07 4.92Computers 1 0 0.01 1.70Total 81 22 4.49 158.97

Table 1 - Environmental Complexity

V. CONCLUSIONS

The ASSIST Ontology proved to be capable of supporting the manual generation of knowledgebases that represented the information contained in soldier after-action reports and the additional relevant information provided for each vignette by each of the research team’s ASSIST systems. It also proved capable of manual generation of knowledgebases characterizing the test site to support analysis of environmental complexity during each evaluation and comparison to statistics from Iraq. In both cases it proved to be a relatively simple matter to extend the ASSIST ontology as needed by adding new object and action concepts when the appropriate concepts were not found in the original ontology. While the generation of the unassisted after-action report knowledgebases was relatively straight forward because the soldiers articulated each observation, the generation of the knowledgebases representing the additional information provided by the ASSIST technologies was not as simple because it was not always easy to tell whether the information presented by the research teams was what the soldiers or intelligence officer had asked for or was not relevant. This made these knowledgebases and therefore comparison of the information contribution of each research team’s ASSIST systems more subjective than DCS would have preferred. When originally conceiving the ASSIST ontology, DCS had envisioned that the ASSIST systems would eventually be capable of automatically generating ASSIST ontology knowledgebases from the sensor data captured during a mission. This would allow each system’s knowledgebases to be compared

177

directly to ground truth knowledgebases and allow direct comparison of the accuracy and completeness of the knowledgebases. However, the processing capabilities of each system were limited to recognizing a relatively small set of object classes in still or video imagery and recognizing a relatively small set words or sounds in audio data. This allowed the research teams to tag still or video images with metadata specifying the classes of objects contained in each image (i.e., a person) but did not allow them to uniquely identify those objects (i.e., the person named John Q. Smith from Silver Springs, MD). This inability to differentiate multiple observations of the same instance of an object class from multiple observations of different instances of the same object class prevented them from extracting specific instances of object classes from the sensor data and reporting their location and relationships to other objects over time. Although each still or video image could have been treated as an observation and stored in an ASSIST knowledgebase, this would have led to a huge amount of uncorrelated observations. As an example, a video stream taken while walking past the John Q. Smith on two successive days would have generated hundreds of observations of an object of class person each day rather than two observations of an object of class person both named John Q. Smith”). Beyond its use for ASSIST technology evaluation, the ASSIST ontology has great potential to facilitate efficient storage and analysis of data from ASSIST technologies. Once the ASISST technologies have matured to the point where they can correlate specific objects over space and time, observations of these individual objects can be efficiently stored in ASISST knowledgebases. The ASSIST users can then annotate the observations of people with additional relationships such as names, family ties, business affiliations, and political affiliations to describe the social networks [7] of the communities observed either verbally as the imagery is taken or via a computer program after the completion of the mission. The ASSIST users can also annotate observations of manmade objects with additional

relationships such as “owned by” and “resides at” to define the relationships between the objects and the social network. These relationships can then form the basis for intuitive, relational queries into the ASSIST knowledgebase. Relational queries, such as “show me a picture of everybody in the al-Lami family associated with the United Iraqi Alliance who was observed in Mosul on Monday and Tikrit on Tuesday”, offer tremendous advantages to using ontologies for data storage and analysis. Commercially available reasoners exist which are capable of handling queries of almost infinite complexity. Storing information in ontological knowledgebases allows such reasoners to determine objects, relationships and patterns that would otherwise be difficult for humans to identify. Therefore it is recommended that automated object identification and storage within ASSIST knowledgebases as well as efficient approaches for operator annotation be investigated in future phases of the ASSIST program.

ACKNOWLEDGEMENT

This work was supported by the National Institute of Standards and Technology (NIST) Intelligent Systems Division (POC. Craig Schlenoff)

REFERENCES [1] Nigel Shadbolt, Tim Berners-Lee and Wendy Hall, The Semantic Web Revisited. IEEE Intelligent Systems 21(3) pp. 96-101, May/June 2006 [2] “The Cyc Foundation” http://www.opencyc.org/ [3] Geneserth, M.R. and R.E. Fikes (Eds), Knowledge Interchange Format, version 3.0. Computer Science Department, Stanford University, Technical Report Logic-92-1, June 1992. [4] S. Ambler, The Object Primer. New York, NY: Cambridge University Press, 2001. [5] “OWL Web Ontology Language Overview” http://www.w3.org/TR/owl-features/ [6] N. F. Noy, R. W. Fergerson, M. A. Musen. The knowledge model of Protege-2000: Combining interoperability and flexibility. 2th International Conference on Knowledge Engineering and Knowledge Management (EKAW'2000), Juan-les-Pins, France, 2000. Also see http:// http://protege.stanford.edu/ [7] “Social Networks” http://en.wikipedia.org/wiki/Social_network

178