DEVS Agents to Support Conformance Testing of Emerging

Defense Information Standards

Bernard P. Zeigler,Arizona Center for Integrative Modeling and Simulation

Tucson Arizonawww.acims.arizona.edu

andJoint Interoperability Test Command

(JITC)Fort Huachuca, Arizona

Outline• “Agent-supported simulation deals with the use of agents as a support

facility to enable computer assistance in problem solving or enhancing cognitive capabilities.” (ADS’06 definition)

• The agent-supported simulation metaphor applies to testing the conformance of multi-agent systems to complex defense information standards

• Structure control agents induce structural change in themselves or others to effectuate different behaviors under different circumstances

• Structure control implemented in Dynamic Structure DEVS enables automation of standards conformance testing

• DEVS formalism is capable of capturing the information-processing complexities, including the dynamics, underlying the MIL-STD-6016C standard

• DEVS modeling and simulation methodology is attaining core-technology for automated testing of military tactical data link standards

Background: Simulation-based development implies the need for simulation-based testing

-- distributed simulation allows testing a system that is first formulated as an abstract model -- both the System under Test (SUT) and the test device are coupled by an common interface

Network

Test Device

send/receive messages

System Under Test (SUT)

send/receivemessages

Raises the question: How to develop the Test Device in an authoritative manner?

Specified as abstract

model, e.g. UML

Connecting middle ware, e.g. HLA

Problem: Conflicting Requirements

To deal with the increasing complexity and advanced decision capabilities of C4ISR systems

=>testing methodology has to become more rigorous, in-depth and thorough

To keep up with the rapid change and short development life cycles expected from the system builders

=> tests have to be ready to conduct in time scales compatible with the agile development strategies of new systems.

Solution: employ DEVS-based M&S

• to increase capabilities for simulation-based testing and •as a basis to increase the automation of testing processes.

Testing of interface standards is a focus area for automated simulation-based testing.

Link-16 is required in all Joint and multi-national operations.

The Joint Interoperability Test Command (JITC) is developing an automated test generation methodology as its core technology for testing conformance of systems to Link-16 This methodology is fundamentally enabled by the DEVS formalized modeling and simulation approach

AWACS

TheaterWarning

ABL

DSP/SBIRS

F-15

JLENS

THAAD

PATRIOT

MEADS

ATACMSAVENGER

TEL

AEGIS (CEP)

SIS(MSCS)SIS(MSCS) Link-16specification

DEVS Representation of Link-16

Constraints(Exception)Rules

Stop

Modify C2Record for TN

1 2 3

RuleProcessing

Stop, Do Nothing,Alerts, Or jump to other

Transaction

TrackDisplay

Operatordecisions

Validity checking

TransmitMsg

Other ConsequentProcessing

Jumps (stimuli) to other

Transactions of specification

Transaction Level - example P.1.2 = Drop Track Transmit

Preparation Processing

Timeouts

PeriodicMsg

Input to

systemDEVS

Output from

system

t1

t2 t

3t4

Level

3 CoupledSystem

2 I/O System

1 I/O Function

0 I/O Frame

System Theory Provides Levels of Structure/Behavior

The JITC employed such testing for the initial major milestone evaluation of the Integrated Architecture Behavior Model (IABM) developed by the Joint Single Integrated Air Picture (SIAP) System Engineering Organization (JSSEO) in 2005.

The test exercise produced significant results that uncovered flaws in the model design and added acknowledged value to the model development.

Recent Successful Application

Types of Distributed Simulation Testing

Testing Description Example

standards conformance test whether system conforms to standardsupports interoperability

Tactical Data LinkStandards, Link-16, VMF, USMTF,

interoperability test whether systems can interoperate (at the syntactic, higher, levels)

Joint Translator Forwarder (JXF) – air to/from land exchange of tactical data

mission/capabilities test whether system of systems have capabilities required for mission

Joint Close Air Support (JCAS)

Test Driver controls the scenario

Multiplatform Distributed Simulation – controlled testing

Platform(System,

Component)

Platform(System,

Component)

Platform(System,

Component)

Test Driver

Distributed Observers look for opportunities to test

Multiplatform Distributed Simulation - uncontrolled testing

Platform(System,

Component)

Platform(System,

Component)

Platform(System,

Component)

Observer Observer Observer

Test Coordinator

Test Driver for Controlled Testing

Middleware

Coupled Test ModelCoupled Test Model

Jx1,data1Jx2,data2Jx3,data3

Jx4,data4 Jx1,data1Jx2,data2Jx3,data3

Jx4,data4 Jx1,data1Jx2,data2Jx3,data3

Jx4,data4

Component Test Model

1

Component Test Model

2

Component Test Model

3

SUT

Test Model Generation for Controlled Testing

Mirroring (flipping) the transactions of a SUT model (system model behavior selected as a test case) allows automated creation of a test model

holdSend(Jx1,data1,t1) holdSend (Jx2,data2,t2)

holdSend (Jx3,data3,t3) waitReceive(Jx4,data4)

receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)

receiveAndProcess(Jx3,data3) transmit(Jx4,data4)

Jx1,data1Jx2,data2Jx3,data3

Jx4,data4

t1 t2 t3 t4time

Test ModelTest Model

SUT Model

Test Manager for Opportunistic Testing• Replace Test Models by Test Detectors• Deploy Test Detectors in parallel, fed by the Observer• Test Detector activates a test when its conditions are met• Test results are sent to a Collector for further processing

Jx1,data1Jx2,data2Jx3,data3Jx4,data4

Test Detector 1

Test Detector 2

Test Detector 3

SUO ObserverOtherFederates

ResultsCollector

Test Manager

The Test Detector watches for the arrival of the given subsequence of messages to the SUO and then watches for the corresponding system output

• Define a new primitive, processDetect, that replaces holdSend• Test Detector

– Tries to match the initial subsequence of messages received by the SUO– When the initial subsequence is successfully matched, it enables waitReceive (or

waitNotReceive) to complete the test

processDetect(Jx1,data1,t1) processDetect(Jx2,data2,t2)

processDetect(Jx3,data3,t3) waitReceive(Jx4,data4)

receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2)

receiveAndProcess(Jx3,data3) transmit(Jx4,data4)

Jx1,data1Jx2,data2Jx3,data3

Jx4,data4

t1 t2 t3 t4time

Test Test DetectorDetector

SUO

Test Detector Generation for Opportunistic Testing

Observer & System Under Observation (SUO)

System(e.g. DEVS)

inports outports

ObserverForSystem

inports outports

Observeroutports

Tap into inputs and outputs of SUO

Gather input/output dataand forward for testing

Example: Joint Close Air Support (JCAS) Scenario

Natural Language Specification

JTAC works with ODA!JTAC is supported by a Predator!JTAC requests ImmediateCAS to AWACS !AWACS passes requestImmediateCAS to CAOC! CAOC assigns USMCAircraft to JTAC!CAOC sends readyOrder to USMCAircraft !USMCAircraft sends sitBriefRequest to AWACS !AWACS sends sitBrief to USMCAircraft !USMCAircraft sends requestForTAC to JTAC !JTAC sends TACCommand to USMCAircraft !USMCAircraft sends deconflictRequest to UAV!USMCAircraft gets targetLocation from UAV!!

Observer of AWACS with JCAS

Data gathered by

Observer

addObserver(USMCAircraft, JCASNUM1);

Observer is connected to SUO andmonitors itsI/O traffic

Test Detector Prototype:Sequence Matcher

processDetect(J2.2,data1,t1)

processDetect(J3.2,data2,t2)

waitReceive(J7.0,data3,t3)

Sequential triggering, same as test models

Example of Effect of State: AWACS

Rules

R1: if phase = “passive” & receive= "ImmediateCASIn“then output = "CASResourcesSpec" & state = "doSurveillance“

R2: if state = "doSurveillance“ & receive= "sitBriefRequestIn“then output = "sitBriefOut“ & phase = “passive”

matchsequence 1:initial state = passiveprocessDetect(ImmediateCASIn,””,1)waitReceive(CASResourcesSpec,””)

matchsequence 2:initial state = doSurveillanceprocessDetect(sitBriefRequestIn,””,1)waitReceive(sitBriefOut,””)

state = doSurveillance

need to know the state to enable this sequence

i1 i2

o2o1

state = passive

Solution: make activation of matchsequence2 conditional on matchsequence1

matchsequence2 can only start when matchsequence1 has successfully been performed

Observation Test Of AWACS

Observer ofAWACS

AWACSTestManager

Problem with Fixed Set of Test Detectors

• after a test detector has been started up, a message may arrive that requires it to be re-initialized

• Parallel search and processing required by fixed presence of multiple test detectors under the test manager may limit the processing and/or number of monitor points

• does not allow for changing from one test focus to another in real-time, e.g. going from format testing to correlation testing once format the first has been satisfied

Solution

• on-demand inclusion of test detector instances• remove detector when known to be “finished”• employ DEVS variable structure capabilities• requires intelligence to decide inclusion and removal

Dynamic Test Suite: Features

• Test Detectors are inserted into Test Suite by Test Control

• Test Control selects Detectors based on incoming message

• Test Control passes on just received message and starts up Test Detector

• Each Detector stage starts up next stage and removes itself from Test Suite as soon as the result of its test is known– If the outcome is a pass (test is successful) then next stage is

started up

Dynamic Inclusion/Removal of Test Detectors

message arrives

add induced test detectors into test set

test detector subcomponent removes its enclosing test detector when test case result is known (either pass or fail)

Test Manager Active Test Suite

Test Control

removeAncestorBrotherOf(“TestControl");

addModel(‘test detector”);addCoupling2(" Test Manager ",“Jmessage",“test detector", “Jmessage");

Adding Ports and Coupling by Relation Methods can be issued by any atomic model:

• add my ports and coupling to ancestor relation of another model – E.g., find my ancestor that is a brother of the given model, and add my ports and associated internal and external coupling to this ancestor

A

B

C D

E

A

addMyPortsToAncestorBrotherOf(B)

Issued by E, adds E’s ports to C, which is ancestor brother of B, and does associated coupling

B

A

BC

D E

C D

E

Adding Ports and Coupling (cont’d) Methods can be issued by any atomic model: • add output ports and coupling from source to destination – adds outports of source as input ports of destination and couples them• add input ports and coupling from source to destination – adds inports of source as outputs of destination and couples them

A

addOutportsNcoupling(C,B))

Issued by E, allows E to communicate with B

B

C D

E

A

B

C D

E

addInportsNcoupling(C,B))

in1 out1

In1 out1

Removing Models by Name and by Relation Methods can be issued by any atomic model:

•remove model by name – accepts the unique name of any atomic or coupled model within hierarchy and removes it and its immediate coupling

• remove model by relation to another model – e.g., remove ancestor that is a brother of the given model, can be used to avoid providing atomic models with names of other models

A

B

C D

E

A

removeModel2(C)

removeAncestorBrotherOf(B)

Issued by D, removes its parent which in this case is its ancestor that is a brother of B

B

A

BC

D E

AWACS Opportunistic Testing in JCAS

Test Control

CAS Model with AWACSobservation

Initially empty Test Suite

AWACS Opportunistic Testing in JCAS (cont’d)

Test Control adds appropriate Test Detector and connects it in to interface,

Test Control observes CAS request message to AWACS

AWACS Opportunistic Testing in JCAS (cont’d)

Test Control passes on start signal and request message

First stage detector verifies request message receipt and prepares to start up second stage

AWACS Opportunistic Testing in JCAS (cont’d)

second stage waits for expected response from AWACS to request

First stage detector removes self from test suite

AWACS Opportunistic Testing in JCAS (cont’d)

Second stage observes correct AWACS response and removes itself and starts up second part

AWACS Opportunistic Testing in JCAS (cont’d)

At some later time, second part of Test Detector observes situation brief request message to AWACS First stage removes itself and starts up second stage

AWACS Opportunistic Testing in JCAS (cont’d)

Second stage observes situation brief output from AWACS thus passing test, It removes itself and enclosing Test Detector

Summary• DEVS formalism is capable of capturing the information-processing

complexities, including the dynamics, underlying the MIL-STD-6016C standard

• DEVS modeling and simulation methodology is attaining core-technology for automated testing of military tactical data link standards

• Structure control implemented in Dynamic Structure DEVS enables automation of standards conformance testing

• Structure control agents induce structural change in themselves or others to effectuate different behaviors under different circumstances

• The agent-supported simulation metaphor applies to testing the conformance of multi-agent systems to complex defense information standards