Agent Based Testing Report

COMSATS INTITUTE OF INFORMATION TECHNOLOGY ISLAMABAD

Hafsah SP14-RSE-012 Qaisar Afzal SP14-RSE-013 Ahsan Azam SP14-RSE-014 Tahir Mehmood SP14-RSE-016 Junaid Kiani SP14-RSE-017

Testing In Agent Base Systems



1. What is Agent?

“One that acts or has the power or authority to act… or represent another”

1.1. What is Software Agent?

According to Z. Houhamdi:

A software agent is a computer program that works toward goals in a dynamic context on behalf of another entity (human or computational), perhaps for a long period of time, with discontinuous direct supervision or control, and exhibits a significant flexibility and even creativity degree in how it tries to transform goals into action tasks [3].Software agents have (among others) the following properties:

1. Reactivity: agents are able to sense contextual changes and react appropriately;

2. Pro-activity: agents are autonomous, so they are able to select which actions to take in order to reach their goals in given situations;

3. Social ability: that is, agents are interacting entities, which cooperate, share knowledge, or compete for goal achievement

OR

A software agent is defined as a software program that can perform specific tasks for a user and possessing a degree of intelligence that permits it to perform parts of its tasks autonomously and to interact with its environment in a useful manner.

COMSATS institute of Information Technology

Environment


1.2. Agent Types:

The following are the types of Agents:

Intelligent Agents Collaborative Agent Interface Agent Mobile Agent Information/Internet Agent Reactive Software Agent Heterogeneous Agent System

1.2.1. Intelligent Agent

Intelligent agents continuously perform three functions: perception of dynamic conditions in the environment; action to affect conditions in the environment; and reasoning to interpret perceptions, solve problems, draw inferences, and determine actions.”

Sensor Input Agent Action Output

1.2.1.1. Characteristics of Intelligent Agent

Autonomy: The ability to take goal directed autonomous decisions without intervening with human or other agents.Goal Oriented: Does not simply act in response of environment.Communicative: communicate with other agents including human.Learning: changes its behavior based on previous experience.Flexible: Actions are not scripts



Pro-activity: The ability to initiate actions to accomplish the goals rather than responding to the unpredictable environment.Reactivity: The ability to respond in selective and timely fashion by perceiving the environment.Social ability: The ability to work in collaboration with other agents and human in order to achieve the goal.Mobility: The ability to move from one platform/environment to another in self-directed way.

1.2.2. Collaborative Agents:

Collaborative agents emphasize autonomy and cooperation (with other agents) in order to perform tasks for their owners. They may learn, but this aspect is not typically a major emphasis of their operation. In order to have a coordinated set up of collaborative agents, they may have to negotiate in order to reach mutually acceptable agreements on some matters.

In brief the key general characteristics of these agents include autonomy, social ability, responsiveness and pro-activeness. Hence, they are (or should/would be) able to act rationally and autonomously in open and time-constrained multi-agent environments.

1.2.3. Interface Agents:

Interface agents emphasize autonomy and learning in order to perform tasks for their owners. A key proponent of this class of agents, points out that the key metaphor underlying interface agents is that of a personal assistant who is collaborating with the user in the same work environment.

Note the subtle emphasis and distinction between collaborating with the user and collaborating with other agents as is the case with collaborative agents. Collaborating with a user may not require an explicit agent communication language as one required when collaborating with other agents.

Figure: How Interface Agents Work



Figure depicts the functioning of interface agents. Essentially, interface agents support and provide assistance, typically to a user learning to use a particular application such as a spreadsheet or an operating system. The agent observes and monitors the actions taken by the user in the interface), learns new short-cuts, and suggests better ways of doing the task. Thus, the interface agent acts as an autonomous personal assistant which cooperates with the user in accomplishing some task in the application. As for learning, interface agents learn typically to better assist its user in four ways all shown in Figure:

By observing and imitating the user (i.e. learning from the user); Through receiving positive and negative feedback from the user (learning from

the user); By receiving explicit instructions from the user (learning from the user); By asking other agents for advice (i.e. learning from peers)

Their cooperation with other agents, if any, is limited typically to asking for advice, and not in getting into protracted negotiation deals with them as is the case with collaborative agents. The learning modes are typically by rote (memory-based learning) or parametric, though other techniques such as evolutionary learning are also being introduced.

1.2.4. Mobile Agents

Mobile agents are computational software processes capable of roaming wide area networks (WANs) such as the WWW, interacting with foreign hosts, gathering information on behalf of its owner and coming back home having performed the duties set by its user. These duties may range from a flight reservation to managing a telecommunications network.

Mobile agents are agents because they are autonomous and they cooperate, differently to collaborative agents. For example, they may cooperate or communicate by one agent making the location of some of its internal objects and methods known to other agents. By doing this, an agent exchanges data or information with other agents without necessarily giving all its information away.

1.2.5. Information/Internet Agents:Information agents perform the role of managing, manipulating or collating information from many distributed sources.



1.2.6. Reactive Software Agents:

Reactive agents represent a special category of agents which do not possess internal, symbolic models of their environments; instead they act/respond in a stimulus-response manner to the present state of the environment in which they are embedded.

There are three key ideas which underpin reactive agents.

Firstly, emergent functionality which we have already mentioned, i.e. the dynamics of the interaction leads to the emergent complexity. Hence, there is no a priori specification (or plan) of the behavior of the set-up of reactive agents.

1.3.Agent Based Systems

Agent based systems are the Ones in which the key abstraction used is that of an agent. It might be conceptualized in terms of agents or analogy with OO Systems. A number of software tools exist that allow a user to implement software systems as agents, and as societies of cooperating agents.

1.3.1. Agents Based System Example

a) Autonomous AirplanesCurrently large airplanes have most of the technologies for autonomous flight like

control of individual airplanes Coordination of activities with other airplanes relies on central authorities. requires dealing with non-autonomous airplanes older or smaller ones

Proposals have been put forward to allow more autonomy

UAVs for military and security applications are in use. Most of them are to a large degree remotely controlled.

b) Apple’s Siri:“Siri is the intelligent personal assistant that helps you get things done just by asking. It allows you to use your voice to send messages, schedule meetings, place phone calls, and more. But Siri isn’t like traditional voice recognition software that requires you to remember keywords and speak specific commands. Siri understands your natural speech, and it asks you questions if it needs more information to complete a task.”

2. Multi-Agent System:



A multi-agent system (MAS) is a computational context in which individual software agents interact with each other, in a collaborative (using message passing) or competitive manner, and sometimes autonomously trying to attain their individual goals, accessing resources and services of the context, and occasionally producing results for the entities that initiated those software agents [4].Consequently, they are difficult to debug and test.

2.1.Testing In Multi Agent Systems

2.1.1. Test Levels

Over the last years, the view of testing has evolved, and testing is no longer seen as a step which starts only after the implementation phase is finished. Software testing is now seen as a whole process that filters in the development and maintenance activities. Thus, each development phase and maintenance phase should have a corresponding test level.

Work in testing software agents and MAS can be classified into different testing levels: unit, agent, integration, system, and acceptance. Here we use general terminologies rather than using specific ones used in the community like group, society. Group and society, as called elsewhere, are equivalent to integration and system, respectively. The testing objectives, subjects to test, and activities of each level are described as follows:

Unit testing: tests all units that make up an agent, including blocks of code, implementation of agent units like goals, plans, knowledge base, reasoning engine, rules specification, and so on; make sure that they work as designed.

Agent testing: tests the integration of the different modules inside an agent; test agents' capabilities to fulfill their goals and to sense and effect the environment.

Integration or Group testing: tests the interaction of agents, communication protocol and semantics, interaction of agents with the environment, integration of agents with shared resources, regulations enforcement; Observe emergent properties, collective behaviors; make sure that a group of agents and environmental resources work correctly together.

System or Society testing: tests the MAS as a system running at the target operating environment; test the expected emergent and macroscopic properties of the system as a whole; test the quality properties that the intended system must reach, such as adaptation, openness, fault tolerance, performance.

Acceptance testing: tests the MAS in the customer's execution environment and verifies that it meets stakeholder goals, with the participation of stakeholders.



2.1.2. Multi Agent Systems are Difficult to Test

Software agents and MAS testing is a challenging task because these systems are distributed, autonomous, and deliberative. They operate in an open world, which requires context awareness. In particular, the very particular character of software agents makes it difficult to apply existing software testing techniques to them. There are issues concerning communication and semantic interoperability, as well as coordination with peers. All these features are known to be hard not only to design and to program [1], but also to test. There are several reasons for the increase of the difficulty degree of testing MAS:

Increased complexity, since there are several distributed processes that run autonomously and concurrently;

Amount of data, since systems can be made up by thousands of agents, each owning its own data;

Irreproducibility effect, since we can’t ensure that two executions of the systems will lead to the same state, even if the same input is used. As a consequence, looking for a particular error can be difficult if it is impossible to reproduce it each time [2].

They are also non-deterministic, since it is not possible to determine a priori all interactions of an agent during its execution.

Agents communicate primarily through message passing instead of method invocation, so existing object-oriented testing approaches are not directly applicable.

Agents are autonomous and cooperate with other agents, so they may run correctly by themselves but incorrectly in a community or vice versa.

.

3.Continuous Testing of Multi Agent System:

Multi-Agent systems are systems composed of multiple software agents that interact with one another in order to achieve their intended goals, and the goals of the systems as a whole. A MAS is usually a distributed system, its agents can be located in different hosts, and they communicate mainly through message passing. Each host provides a specific environment for the agents located at that host. Due to these peculiar properties, testing MAS is a challenging task.



4. Testing Techniques4.1. Role Oriented Approach

Role is an important attribute of an agent that has a huge scope and support for the analysis, design and implementation of Multi-Agent System (MAS). A role can be defined as the capability enabler that exposes to the agent that plays it a set of actions. Roles are created to do something and it has the responsibility of achieving specific system goals and sub goals. Roles provide a well-defined interface between agents and cooperative processes.

Role Oriented Testing

Testing the responsibilities corresponding to their respective role ensures that the agent plays its role as per the intended goal. As long as the goals are getting satisfied, it is understood that the individual agent is performing well that suits the system.

Σ An = Σ Gi ( Σ Rj ( Σ Rek ) ) where i = j = k = x = 1

Following are the steps involved in role-oriented testing.

STEP 1: Select the Agent to be tested. STEP 2: Identify Goals (Gi), Roles (Rj) and their corresponding Responsibilities (Rek) STEP 3: Design Role Model Diagram (Ax (Gi Rj Rek) STEP 4: Analyze role model (Ax(Gi Rj Rek) STEP 5: Define the interacting agents and situations. STEP 6: Identify environmental factors pre-conditioning input trigger Rek STEP 7: Identify fulfillment criteria that satisfies Responsibility STEP 8: Run test cases

Role Oriented Unit Testing

Role Oriented Unit Testing focuses primarily on the smallest building block of the Multi-Agent System. The basic idea behind the unit testing in MAS is to verify whether the individual agent (unit) in isolation performs its responsibilities under various conditions. Every individual agent has its own goal to be achieved and plans to do to fulfill the goal. In addition to goal and plan, role is one important mental state of the agent, which is defined as a set of capabilities and expected behavior. A role can be represented as <Goal, Responsibilities, Protocol, and Permissions>

Goal, for which the agent playing this role is responsible

Responsibilities, Which indicates the functionalities of agents playing such roles

Protocol, which indicates how an agent playing such role can interact with agents playing other role



Permissions, which are a set of rights associated with the role.

<Agent><Goal>

<Role1><Responsibilities 1><Protocol>

. <Responsibilities n><Protocols>

</Role1><Role n><Responsibilities 1><Protocol>

. <Responsibilities n><Protocol></Role n>

</Goal></Agent>

Role Oriented Integration Testing

Integration testing in conventional software development is that to progressively integrate the tested units (module, program, procedure, function).After ensuring that the individual agent in isolation is working as per the requirement; the next immediate step is to integrate the agents involved in the MAS so as to test the interaction, and communication among agents.

Role Oriented System Testing

System testing verifies that all elements (hardware, people, and databases) are integrated properly so as to ensure whether the overall product met its requirement and achieved the expected performance. System testing also deals with non-functional requirements of the software such as recovery testing, security testing, stress testing and performance testing. System testing in agent oriented approach will test the complete functionality and test the system as a whole.

Role Schema

Role schema provides a well-defined interface between agents and cooperative processes. This allows an agent to read and follow, normative rules established by the cooperation process even if not previously known by the agent. Their major motivation to introduce such roles is to increase the agent system’s adaptability to structural changes. Role schema involves role name, agent name, goal to be achieved, description of the role, protocol and related activities, permissions and responsibilities.



Role Schema

Advantages of Role Based Approach

Role based approach helps the multi agent system developers in testing the individual unit of the agent based system.

Disadvantages of Role Based Approach

Testing whether the agent performs its role properly is a challenging task.

4.2. Unit Testing in Multi-Agents Systems using Mock Agents

In a multi-agent system it may not always be possible to scan/gather the complete agent requirements, due to the fact that agent changes its functionalities based on the environmental need so as to achieve the goal. Agent is an important building block of a multi-agent system and it is very essential to verify whether the agent in isolation matches with the specification under normal and abnormal condition. Thus in an agent-oriented programming, the smallest building block/unit is an agent itself. Unit testing with respect to agent based system is to individually test all the agents involved in the system. Testing an agent is based on its own mental attributes such as goal, plan, role and action. Test cases are to be generated so as to ensure the following,

1. To test whether the goal of an agent is achieved.



2. To test whether the plan get triggered by the event that is supposed to handle.

3. To test whether the agent takes appropriate role as expected by the environment

4. To test whether the agent performs the activity as per the functionality.

The primary aim of unit testing is to uncover errors within a given unit

A Mock Agent is a regular agent that communicates with just one agent: the AUT. It has just one plan to test the AUT. The Mock Agent’s plan is equivalent to a test script, since it defines the messages that should be sent to the AUT and the messages that should be received from it. By testing an agent in isolation using Mock Agents the programmer is forced to consider the agent’s interactions with its collaborators (or competitors), possibly before those collaborators (or competitors) exist.Agent unit test approach that is composed of five participants:

1- Test Suite: which consists in a set of Test Cases and a set of operations performed to prepare the test environment before a Test Case starts.

2- Test Case: defines a scenario – a set of conditions – to which an Agent under Test is exposed, and verifies whether this agent obeys its specification under such conditions.

3- Agent under Test (AUT): is the agent whose behavior is verified by a Test Case.4- Mock Agent: consists in a fake implementation of a real agent that interacts with the

AUT. Its purpose is to simulate a real agent strictly for testing the AUT.5- Agent Monitor: is responsible for monitoring agents’ life cycle in order to notify the Test-

Case about agents’ states.



Each agent unit test follows the common structure depicted in Figure. In step 1, the Test Suite creates the agent’s platform and whatever element needed to set up the test environment.After that, a Test Case is started. The Test Case creates Mock Agents to every role that interacts with the Agent under Test - in the scenario defined by the Test Case (step 2).

Next, it creates the Agent under Test (step 3) and asks the Agent Monitor to be notified when the interaction between the AUT and the Mock Agents finishes (step 4).

At this point, the AUT and the Mock Agent start to interact. The Mock Agent sends a message to the AUT, and it replies (steps 5 and 6) or vice-versa. They can repeat steps 5 and 6 as many times as necessary to perform the test codified in the Mock Agent’s plan (for instance, the Mock

Agent can reply three messages before finalizing its test activity). During all this interaction process, the Agent Monitor keeps track of changes in agents’ life cycle. In order to do that it uses three lists as illustrated in Figure : (i) Created Agents List: maintains IDs of the agents that have been created, but are not running yet – an ID is any information that uniquely identifies an agent.(ii) Running Agents List: maintains IDs of the agents in the running state. (iii) Work Done Agents List: maintains IDs of the Mock Agents that have completed their plan (equivalent to a test script).When a Mock Agent concludes its plan, the Agent Monitor includes the Mock Agent’s ID in the Work done list. And then notifies the Test Case that the interaction between the Mock Agent and the AUT have concluded (step 7).

Lastly, the Test Case asks the Mock Agent whether or not AUT acted as expected (step 8).

This agent unit testing approach has two main concerns: (i) the design of a Test Case based on the use of Mock Agents; (ii) and the Test Case execution which relies on the Agent Monitor to notify when the test script (codified in Mock Agent’s plan) was concluded. As each Mock Agent exercises just one role of the AUT, rather than the wide interface that comprises all the features provided it, we call this approach “Role Driven Unit Testing”.

Pros:1- In MASs testing approach Unit Testing aims to remove faults along the application

development.2- Unit testing is independent of any development life cycle used.3- Unit testing in MAS verifies whether each agent in isolation respects its specifications

under normal and abnormal conditions.4- It is a role based approach, we can find out the individual behavior of

agents.



Cons:1- It doesn’t create complete test cases, this technique needs to be

combined with other strategies to make a thorough set of test cases.2- This approach only tests individual agent, group testing of agents or

system of agents as a whole is not considered

5. Conclusion:

Most of the existing research work on testing software agent and MAS focuses mainly on agent and integration level. Basic issues of testing software agents like message passing, distributed/asynchronous have been considered; testing frameworks have been proposed to facilitate testing process. And yet, there is still much room for further investigations, for instance [5]:

A complete and comprehensive testing process for software agents and MAS.

Testing MAS at system and acceptance level: how do the developers and the end-users build confidence in autonomous agents?

Test inputs definition and generation to deal with open and dynamic nature of software agents and MAS.

Test oracles, how to judge an autonomous behavior? How to evaluate agents that have their own goals from human tester's subjective perspectives?

Testing emergent properties at macroscopic system level: how to judge if an emergent property is correct? How to check the mutual relationship between macroscopic and agent behaviors?

Deriving metrics to assess the qualities of the MAS under test, such as safety, efficiency, and openness.

Reducing/removing side effects in test execution and monitoring because introducing new entities in the system, e.g., mock agents tester agents, and monitoring agent as in many approaches, can influence the behavior of the agents under test and the performance of the system as a whole.

References:



1. F. Bergenti, M. Gleizes, and F. Zambonelli, “Methodologies and Software Engineering for Agent Systems”, The Agent-Oriented Software Engineering Handbook, Springer, Vol. 11, 2004.

2. I. Sommerville, “Software Engineering”, 9th edition, Addison Wesley, 2011.

3. Z. Houhamdi, “Test Suite Generation Process for Agent Testing”, Indian Journal of Computer Science and Engineering, Vol. 2, 2, 2011.

4. J. Botia, A. Lopez-Acosta, and G. Skarmeta, “ACLAnalyser: A tool for debugging multi-agent systems”, Proceeding of the 16th European Conference on Artificial Intelligence, pp. 967-968, IOS Press 2004.

5. Multi-Agent System Testing: A Survey

6. Sivakumar, N., and K. Vivekanandan. "Exploring the Need for Specialized Testing Technique for an Agent-Based Software." PROMETHEUS 2002.7 (2002).

7. Agent Oriented Software Testing – Role Oriented approach (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 3, No.12, 2012 by N.Sivakumar and K.Vivekanandan

8. Unit Testing in Multi-agent Systems using Mock AgentsBy Roberta Coelho, Uirá Kulesza, Arndt von Staa, Carlos Lucena.

9. Computer Science Department, Pontifical Catholic University of Rio de Janeiro – PUC-Rio, Brazil10. {roberta, uira, arndt, lucena}@inf.puc-rio.br

11. Agent Oriented Software Testing – Role Oriented approach (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 3, No.12, 2012 by N.Sivakumar and K.Vivekanandan

12. http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs01-introduction-2x2.pdf Last Accessed at 27-11-2014

13. http://www.apple.com/iphone/features/siri.html Last Accessed at 25-11-2014

Resources for Further Reading


http://www.apple.com/iphone/features/siri.html

http://www.inf.ed.ac.uk/teaching/courses/abs/slides/abs01-introduction-2x2.pdf


Research Papers:

1. Kissoum, Y., Sahnoun, Z., & Barkaoui, K. (2009, April). An approach for testing mobile agents using the nets within nets paradigm. In Research Challenges in Information Science, 2009. RCIS 2009. Third International Conference on (pp. 207-216). IEEE.

2. Padgham, L., Zhang, Z., Thangarajah, J., & Miller, T. (2013). Model-based test oracle generation for automated unit testing of agent systems. Software Engineering, IEEE Transactions on, 39(9), 1230-1244.

3. Kalboussi, S., Bechikh, S., Kessentini, M., & Said, L. B. (2013, November). On the Influence of the Number of Objectives in Evolutionary Autonomous Software Agent Testing. In Tools with Artificial Intelligence (ICTAI), 2013 IEEE 25th International Conference on (pp. 229-234). IEEE.

4. Malz, C., & Jazdi, N. (2010, May). Agent-based test management for software system test. In Automation Quality and Testing Robotics (AQTR), 2010 IEEE International Conference on (Vol. 2, pp. 1-6). IEEE.

5. Moser, T., Merdan, M., & Biffl, S. (2010, February). A pattern-based coordination and test framework for multi-agent simulation of production automation systems. In Complex, Intelligent and Software Intensive Systems (CISIS), 2010 International Conference on (pp. 526-533). IEEE.

6. Gurcan, O., Dikenelli, O., & Bernon, C. (2011, September). Towards a generic testing framework for agent-based simulation models. In Computer Science and Information Systems (FedCSIS), 2011 Federated Conference on (pp. 635-642). IEEE.

7. Miller, T., Padgham, L., & Thangarajah, J. (2011). Test coverage criteria for agent interaction testing. In Agent-Oriented Software Engineering XI (pp. 91-105). Springer Berlin Heidelberg.

8. Jing, G., & Yuqing, L. (2009, December). Automatic Test Task Allocation in Agent-Based Distributed Automated Testing Framework. In Computational Intelligence and Software Engineering, 2009. CiSE 2009. International Conference on (pp. 1-5). IEEE.



9. Paydar, S., & Kahani, M. (2011). An agent-based framework for automated testing of web-based systems. Journal of Software Engineering and Applications, 4(02), 86.

Web Links:

1. http://www.agentbuilder.com/Documentation/whyAgents.html2. http://www.sce.carleton.ca/netmanage/docs/AgentsOverview/ao.html3. http://www.ai.mit.edu/projects/humanoid-robotics-group/cog/capabilities.html4. http://www.slideshare.net/ToniMorenoURV/introduction-to-agents-and-multiagent-

systems5. http://www.agentbuilder.com/Documentation/product.html6. www.askjeeves.com7. http://ti.arc.nasa.gov/tech/asr/planning-and-scheduling/remote-agent/8. http://www.agentbuilder.com/AgentTechnology/agentApplications.html9. http://www.agentbuilder.com/AgentTools/index.html10. http://en.wikipedia.org/wiki/Distributed_artificial_intelligence


Agent Based Testing Report

Documents

Transcript of Agent Based Testing Report