Simulation Validation Using Direct Execution of Wireless Ad-Hoc RoutingProtocols ∗

Jason Liu, Yougu Yuan, David M. Nicol,Robert S. Gray†, Calvin C. Newport†, David Kotz†,

and Luiz Felipe Perrone‡

Coordinated Science Laboratory, University of Illinois at Urbana-Champaign1308 West Main Street, Urbana, IL 61801{jasonliu,yuanyg,nicol}@crhc.uiuc.edu

† Department of Computer Science, Dartmouth College6211 Sudikoff Laboratory, Hanover, NH 03755

{robert.s.gray, calvin.c.newport}@dartmouth.edu, dfk@cs.dartmouth.edu

‡ Department of Computer Science, Bucknell University, Lewisburg, PA 17837perrone@bucknell.edu

Abstract

Computer simulation is the most common approach tostudying wireless ad-hoc routing algorithms. The results,however, are only as good as the models the simulationuses. One should not underestimate the importance of vali-dation, as inaccurate models can lead to wrong conclusions.In this paper, we use direct-execution simulation to vali-date radio models used by ad-hoc routing protocols, againstreal-world experiments. This paper documents a commontestbed that supports direct execution of a set of ad-hocrouting protocol implementations in a wireless network sim-ulator. The testbed reads traces generated from real experi-ments, and uses them to drive direct-execution implemen-tations of the routing protocols. Doing so we reproducethe same network conditions as in real experiments. Bycomparing routing behavior measured in real experimentswith behavior computed by the simulation, we are able tovalidate the models of radio behavior upon which protocolbehavior depends. We conclude that it is possible to havefairly accurate results using a simple radio model, but the

∗This work was supported in part by Dartmouth Center for MobileComputing, DARPA (contract N66001-96-C-8530), the Department ofJustice (contract 2000-CX-K001) and the Department of Defense (MURIAFOSR contract F49620-97-1-03821). Points of view in this document arethose of the authors and do not necessarily represent the official position ofDARPA or the United States government. The U. S. Government retainsa non-exclusive, royalty-free license to publish or reproduce the publishedform of this work, or allow others to do so, for U.S. Government purposes.

routing behavior is quite sensitive to one of this model’s pa-rameters. The implication is that one should i) use a morecomplex radio model that explicitly models point-to-pointpath loss, or ii) use measurements from an environment typ-ical of the one of interest, or iii) study behavior over a rangeof environments to identify sensitivities.

1. Introduction

Using simulation one must take the precaution that themodel may not reflect the reality. Validation of a wirelessnetwork simulation is particularly difficult because not onlymust the implementation of the simulated protocol be val-idated against its design specifications, but also the modelmust be able to capture lower-level characteristics of thewireless environment with a proper level of abstraction [3].The validation problem is amplified when the routing pro-tocol deployed in a real system is implemented and main-tained as a separate code base from the one used in simula-tion.

Direct-execution simulation alleviates the problem ofmaintaining separate code bases for the same routing proto-col by executing the same code designed for real systems di-rectly inside a wireless network simulator. We compile therouting protocol’s source code with the simulator’s sourcecode with only moderate changes as necessary. The proto-col’s logic is executed inside the simulator and is driven by

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

the simulator’s time advancing mechanism. In a discrete-event simulation, the routing protocol code is invoked asa result of the simulator processing events stored in theevent queue. Since each protocol instance communicateswith other simulated mobile stations by sending and re-ceiving packets through well-defined system calls, we sub-stitute these system calls with calls to the simulator. Thepackets are redirected to go through the simulated wirelessnetwork—all transparent to the protocol implementation.Using direct-execution simulation is desirable for prototyp-ing a protocol implementation, which, after initial simula-tion evaluation, can be deployed directly in a real network.

Our interest in direct-execution simulation is that it canhelp us validate the wireless network simulator using the re-sults from real experiments. We run the same routing proto-col and application traffic generator code both in simulationand in the real experiment. The difference is that in the realexperiment packets are transmitted via the wireless chan-nel and are subject to delays and losses due to signal fadingand collisions during the transmission. In simulation, thesepackets are translated into simulation events scheduled withdelays calculated by the radio channel model. Dependingon the modeling details, the simulation result may or maynot reflect what would happen in reality. Direct-executionsimulation provides us a valuable opportunity to investigatethe effect of details of a wireless network model on the fi-delity of the simulation study.

This paper documents our effort in supporting direct ex-ecution of a set of wireless ad-hoc routing protocol imple-mentations and using direct-execution simulation to vali-date the underlying wireless network models by compar-ing the results from real-world experiments. We portedfive routing protocol implementations for direct execution:APRL, AODV, GPSR, ODMRP, and STARA. Versions ofall five protocols were implemented as part of the ActCommproject, whose goal is to provide information access througha wireless network to soldiers in the field.1 One contributionof our research is to provide a common testbed for directexecution of these protocols in simulation. More impor-tantly, we instrumented the testbed to enable validation ofvarious wireless network models. We embedded the routingprotocol code with various logging functions. Each laptopcomputer running the routing protocols in the real exper-iment has a Global Positioning System (GPS) device andperiodically records its location information and average re-ceiving signal quality from other laptops. We later trans-formed these logs into traces of node mobility and radioconnectivity. We adapted the simulator to read the tracesand combined them with different stochastic radio propa-gation models to reproduce the test scenario inside simula-tion. We compared the results from running these routingprotocols in simulation with those collected from the real

1http://actcomm.thayer.dartmouth.edu/.

experiment. Such comparison helps us understand the ef-fect of different wireless network models on the behavior ofad-hoc routing algorithms.

We present two set of experiments in this paper. The firstone compares two implementations of the AODV routingprotocol, both running in simulation. One AODV imple-mentation was specifically designed for the simulator—theprotocol sends and receives packets, and schedules time-outs using the special functions provided by the simula-tor. The other AODV implementation was developed by theActComm project, designed for the real network, and wasdirectly executed in the simulator. The goal of this exper-iment is to validate both protocol implementations and toidentify the overhead introduced by direct-execution simu-lation in terms of memory usage and execution time. Thesecond set of experiments compares the results from a realfield test with those from simulation. In the real experiment,we ran the routing protocols on 40 laptop computers, eachequipped with a wireless device and a GPS unit. The lap-tops were carried by people walking randomly in a largefield. In simulation, we applied different radio propaga-tion models together with the traces derived from the realexperiment. We directly executed the routing protocol im-plementations in simulation and compared the behavior ofthese protocols in both environments. The goal of this studyis to highlight the importance of modeling decisions on thevalidity of a wireless simulation study.

The paper is organized as follows. Section 2 provides anoverview of the implementations of the ActComm routingprotocols and outlines the architecture of our wireless net-work simulator on which we directly execute these protocolimplementations. In Section 3 we briefly describe issues re-lated to direct-execution simulation. Section 4 presents theaugmented simulation testbed designed for validation pur-poses. We focus on the experiments and results in Section 5.Section 6 concludes the paper.

2. Background

2.1. The Routing Protocols

We ported five protocols for direct execution. Any-PathRouting without Loops (APRL) is a proactive distance-vector routing protocol [5]. Rather than using sequencenumbers, APRL uses ping messages before establish-ing new routes to guarantee loop-free operation. Ad-hoc On-Demand Vector (AODV) is an on-demand rout-ing algorithm—routes are created as needed at connectionestablishment and maintained thereafter to deal with linkbreakage [10]. Greedy Perimeter Stateless Routing (GPSR)uses GPS positions of the mobile stations to forward pack-ets greedily along a path toward the target’s physical loca-tion [6]. GPSR uses a perimeter-following algorithm to for-

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

ward packets around the boundaries of empty regions thatcontain no laptops (and hence cause greedy forwarding tofail). On-Demand Multicast Routing Protocol (ODMRP)maintains a mesh, instead of a tree, for alternate and redun-dant routes for each multicast group [7]. It does not dependon another unicast routing protocol and, in fact, can be usedfor unicast routing. System and Traffic Dependent AdaptiveRouting Algorithm (STARA) uses shortest-path routing [2].The distance measure is calculated by the mean transmis-sion delay instead of the hop count.

We implemented these protocols for the ActCommproject in C++ on Linux. All five implementations per-form their routing in user space using IP tunneling and UDPsockets. An IP tunnel is a virtual network device with twoendpoints: one as a regular network interface, and the otheras a Unix file. Packets sent to the network interface, viaa standard UDP socket for example, can be read from thefile by any (authorized) user process, while packets writtento the file are delivered by the kernel as if they had arrivedover the network interface. Each mobile station has a virtualIP address (e.g., 11.0.0.1) associated with the network inter-face of the tunnel, and a physical IP address (e.g., 10.0.0.1)associated with the network interface of the physical wire-less device. The application communicates using virtual IPaddresses. The kernel IP routing table in each mobile sta-tion is configured to forward packets with virtual destina-tion addresses to the IP tunnel device. At the source of atransmission, the packet sent from the application is for-warded through the IP tunnel to the routing protocol read-ing the device file. The routing protocol then converts thevirtual addresses to physical addresses and selects the nexthop to forward the packet to according to its routing table.Packets are forwarded to their neighbors using UDP sock-ets through the physical (wireless) network device. Oncethe packet reaches its destination, the physical addresses aretranslated back into virtual addresses and the routing proto-col writes the packets to the device file that represents theIP tunnel, which then deliver the packet to the applicationvia the virtual network interface.

2.2. The Wireless Network Simulator

We developed a high-performance simulator calledSWAN as an integrated, flexible, and configurable environ-ment for evaluating different wireless ad-hoc routing proto-cols, especially in large network scenarios. SWAN is builtbased on a parallel discrete-event simulator called DaSSF,which has proved successful in simulating large-scale wirednetworks.2 We ported and implemented several protocolmodels that are used frequently in a wireless ad-hoc net-work. The protocol models can be readily assembled into aprotocol stack within each simulated mobile station. Using

2http://www.cs.dartmouth.edu/research/DaSSF.

SWAN, one can dynamically configure each protocol andthe underlying wireless network using a specially designedconfiguration language.

In this paper, we study the effect of various radio sig-nal propagation models on the behavior of the routing algo-rithms in simulation. In particular, we examine three sim-ple but frequently used stochastic radio propagation mod-els: a Friis free-space model, a two-ray ground reflectionmodel, and a generic propagation model. The Friis free-space model assumes an ideal radio propagation condition:the signals travel in a vacuum space without obstacles. Thepower loss is proportional to the square of the distance be-tween the transmitter and the receiver. The two-ray groundreflection model adds a ground reflection path from thetransmitter to the receiver. The model is more accurate thanthe free-space model when the distance is large and whenthere is no significant difference in elevation between themobile stations. The generic propagation model describesthe radio signal attenuation as a combination of two effects:small-scale fading and large-scale fading. The small-scalefading captures the characteristic of rapid fluctuation in sig-nal power over a short period of time or a small changein the node’s position—a result primarily due to the exis-tence of multiple paths that the signals travel. The clas-sic models that predict the small-scale fading effect includeRayleigh and Ricean distributions. Large-scale fading ismostly caused by the environmental scattering of the sig-nals and can be further divided into two components: thedistance path loss is the average signal power loss as a func-tion of distance and is proportional to the distance raised toa specified exponent; the shadow fading effect describes thevariations in signal receiving power measured in decibelsand can be modeled as a log-normal distribution. Readerscan refer to a textbook on wireless communications (suchas Rappaport’s book [11]) for a detailed discussion on theradio propagation models.

3. Direct Execution

In simulation, multiple instances of a routing protocolmust run simultaneously, driven by the same event queue.Conceivably, each routing protocol can run as a separateprocess and interact with the simulation kernel throughinter-process communication mechanisms. We only needto substitute the system calls related to either communica-tions (i.e., sending or receiving packets) or time (e.g., query-ing for the current wall-clock time or potentially blockingthe user process causing noticeable delays) with calls to thesimulator. The replacement can be done at link time aftercompilation. The major attraction of this approach is thatno source code modification is necessary. The drawback,however, lies in its complexity and the potential overheadintroduced by the inter-process communication.

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

We chose an easier yet faster approach that allows multi-ple instances of the same routing protocol to execute in thesame address space. The method involved moderate modifi-cations to the source code. Similar approaches can be foundin the literature [1, 8, 9]. We ported all five ActComm rout-ing protocols together with related programs, such as the ap-plication traffic generator used in the real experiment. Thenumber of lines changed accounts for only 3.8% and mostchanges were related to creating and configuring the rout-ing protocols individually in each simulated mobile station,separated from the protocol’s control flow.

3.1. Encapsulations

We modified the protocol code slightly to allow multipleinstances of a routing protocol to run simultaneously insidethe simulator. Since all these instances are executed in thesame address space, we need to provide wrappers so thatthese instances can be identified and separated in the sameexecution environment.

We created a protocol session object to represent eachrouting protocol instance in the simulator. The protocol’sinteraction with the operating system, such as system callsfor sending and receiving packets, was replaced by methodinvocations of the protocol session. These methods redi-rect the calls to simulator. We also replaced global vari-ables, which are data objects specific to a routing protocolinstance, with member data of the protocol session. We re-placed the original main function in the routing protocolimplementations with a method of the protocol session thatconfigures and initializes the instance.

3.2. Communications

The routing protocol implementations use system callsfor communications, such as sendto for sending messagesthrough UDP sockets. As mentioned earlier, we replacedthese system routines with those supplied by the simula-tor. Rather than replacing them manually at all places of thesource code, we provided a base class that contains meth-ods with the same names as the system routines and with thesame parameters. In this way, all classes in the protocol im-plementations default to call the methods in the base class.The base class contains a reference to the protocol sessionthat represents the routing protocol instance. The methodsin the base class forward control through the reference tothe protocol session, which passes on the messages throughthe simulated protocol stack.

We added support in the simulator for UDP sockets. AUDP protocol session manages the UDP sockets on top ofthe IP layer and its primary function is to multiplex and de-multiplex UDP datagrams. We replaced system calls re-lated to UDP sockets, such as socket, bind, sendto,

recvfrom, and setsockopt, with methods that interactwith the UDP protocol session. We also implemented theIP tunnel device in the simulator. The device is treated as anetwork interface below the IP layer in the protocol stack.Packets sent by the application with virtual destination ad-dresses (via UDP sockets) are diverted to the tunnel deviceby the IP layer. The routing algorithm accesses the IP tunnelthrough a regular file descriptor. We replaced the file accessfunctions, specifically open, read, write, and close,to distinguish the file descriptor for the tunnel device fromother regular files. We did not replace operations to regularfiles since they are used by the directly executed code forlogging purposes.

3.3. Timings

The routing protocols executed inside the simulator mustbe driven by simulation time rather than real time, whichmeans that we must deal with all time-sensitive system callscarefully. We replaced gettimeofday, which returns thewall-clock time of the mobile station, with a call to the sim-ulator querying for the current simulation time. We alsoreplaced select, which causes the running process to beblocked until any one of the specified set of file descriptorsis ready for reading or writing, or the given timeout inter-val has been elapsed. The ActComm protocol implemen-tations all center on an event loop that contains one call tothe select function. When the control returns from thisfunction—upon timeouts or incoming messages—the algo-rithm invokes the corresponding event handlers to processthe event. We bypassed the the event loop and directly in-voked the event handlers whenever a timeout occurred or amessage arrived at the protocol session.

One also has to be aware of the ramifications from thelack of a CPU work model in the wireless simulator. Thesimulator uses function invocations for packets to travel upand down the protocol stack, without advancing the simula-tion time. This bears no side-effect for a carefully designedprotocol model, where the packet processing time is simu-lated with proper random delays, but may create problemsfor a directly executed protocol implementation that pays nospecial attention to the packet processing time. If in simula-tion we assume zero packet processing time, the behavior ofall instances of a routing protocol could be synchronized insimulation time. This time synchrony could then lead to anunnaturally high probability of packet loss due to collisionsat the radio channel. To deal with this problem, we intro-duced packet jitters at the interface between the simulatorand the directly executed code. Each time a message goesthrough a UDP socket, we added a random delay to modelthe time needed by the operating system for processing thepacket.

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

4. Support for Simulation Validation

In this section we discuss our support to validate a wire-less simulation by comparing results from the real exper-iment and the direct-execution simulation. Validation ofsimulations in general and wireless ad-hoc network simu-lations in particular has been a focal point surrounding theapplicability of simulation studies. Johnson first suggestedusing the logging information from running ad-hoc routingalgorithms during the real experiment to simulate identicalnode movement and communication scenarios [4]. Takaiet al. studied the effect of wireless physical layer and radiochannel modeling on the performance evaluation of ad-hocrouting algorithms [12, 13].

Our research uses real experiments as the base for com-parison. In particular, we ran the routing protocol imple-mentations together with other applications, such as thetraffic generator, directly in the simulator. We derived bothmobility and radio connectivity traces from the real exper-iment and combined them with a stochastic propagationmodel in an attempt to recreate the real network conditionsin simulation. We compared the results against those fromthe real experiment to assess the validity of the radio prop-agation model.

In all five ActComm routing protocol implementationswe embedded a sophisticated logging mechanism, as shownin Figure 1. When the routing protocol runs, it generatesan event log that includes all types of events related to therouting algorithm, such as sending or receiving a controlmessage. We used the event log both for analyzing the per-formance of the routing algorithm and for debugging. Wealso instrumented the traffic generator with logging func-tions to record each packet sent and received. We later usedthis application log to calculate application-level statistics,such as packet delivery rate and end-to-end delay. The traf-fic generator executed directly in the simulator also read thislog to recreate the exact traffic behavior.

In the real experiment, we ran a third program calledthe service module together with the routing protocol andthe application traffic generator. The program periodicallyqueried the attached GPS device at the mobile station to logits current position. The program also used iwspy to pe-riodically record link quality information. iwspy allowsthe user to set a list of network addresses. The wireless de-vice driver gathers the link quality information, in signal-to-noise ratio (SNR), whenever a packet is received from oneof those addresses, that is, from any other laptop. The ser-vice module collected the link quality information and aver-aged it over the last sampling interval. Also, it periodicallybroadcasted beacon messages that contain position informa-tion of all known mobile stations. The original ActCommapplications used them to keep every soldier in the field up-dated with the positions of other soldiers. We recorded the

ConnectivityTrace

MobilityTrace

Beacon LogPosition Log

Mobile Node #1

REAL

EXPERIMENT

SIMULATION

Service

Module

Mobile Node #1

Service

Module

Routing ApplicationLogEvent Log

Routing

Protocol Generator

Traffic

Routing

Protocol Generator

Traffic

SignalQuality Log

Figure 1. Logs are generated and comparedfor validating simulation results.

beacon messages and used them to refresh the link qualityinformation.

In simulation, the routing protocols are running directlyinside the simulator together with the application trafficgenerator and the service module. We chose to directlyexecute the service model since we need to reproduce thebeacon messages and their effect on the MAC/PHY statesof the wireless network.

We further processed the position log from the real ex-periment to produce a mobility trace, which shows howeach mobile station moved during the experiment. In addi-tion, we generated a node connectivity trace from the bea-con logs recorded by the mobile stations during the real ex-periment. The mobility trace states whether a mobile sta-tions can receive a packet from another mobile station overthe wireless channel at a given time. The beacon log con-tains the times at which the beacon messages from othermobile stations were received. Receiving a beacon success-fully indicates a link from the sender to the receiver, whilemissing several consecutive beacons indicates that the re-ceiver may be beyond the transmission range of the sender.

The signal quality log recorded a series of averagedsignal-to-noise ratios for packets received at each mobilestation from other stations in the network. We did not in-clude the signal quality log in this study. We are currentlyinvestigating the use this log to reconstruct the connectivityof the network, as it may provide a better alternative to thebeacon log.

We used the radio connectivity trace as a baseline to de-termine whether two mobile stations could directly commu-nicate with each other. The connectivity information, how-ever, does not capture the state of interference—collisionscould happen due to the presence of “hidden terminals.” For

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

example, if node B can hear both node A and node C situ-ated on either side, but node A cannot talk to C and viceversa because of the distance, it is possible that node B can-not faithfully receive a packet from A if node C is transmit-ting another packet to node B. Although the 802.11 MAClayer protocol, which arbitrates packet transmissions overthe radio medium, allocates the radio channel before eachtransmission, it cannot totally prevent collisions. In thiscase, the simulator must use an interference model to sim-ulate what would happen when two packets arrive at thereceiver—one of the packets can be accepted if its receivingpower is significantly higher than the other, or both packetscan be lost due to interference. Since the interference modelrelies on the receiving signal power to determine packet re-ceptions, we still need a radio propagation model to simu-late the signal power attenuation.

5. Performance and Validation Studies

We conducted two experiments for validation: one com-paring the direct-execution simulation of the ActCommAODV protocol implementation with an AODV protocolmodel implemented natively in the simulator, and the othercomparing a real experiment with the simulated wirelessnetwork.

5.1. AODV vs. AODV

Our first experiment compared the direct execution ofthe ActComm AODV protocol implementation with anAODV protocol model implemented natively in SWAN.We ran both protocol implementations in simulation un-der the same simulated network conditions, with the sameapplication traffic pattern, and the same radio propagationmodel. Our goal is to validate both protocol implementa-tions against each other and determine how much overheaddirect-execution simulation requires.

In the simulation experiment, we tested a network of 50,100, and 200 mobile stations, out of which we chose 20mobile stations as traffic sources. We deployed these mo-bile stations in a square area, sized so that each mobile sta-tion had seven neighbors on average (796, 1126, and 1592meters for each dimension, respectively). We used the ran-dom way-point node mobility model: each node moves to arandomly selected point in the area with a speed chosen uni-formly between 1 and 10 meters/s; when reaching the point,it pauses for 60 seconds before selecting another point tomove to. We chose the IEEE 802.11 protocol for the MACand PHY layer with standard parameters according to theIEEE specification (with 11 MB/s bandwidth), and we usedthe generic radio propagation model (with an exponent of2.5 and shadow fading log-normal standard deviation of 6dB) to compute radio signal power attenuation. We used

a simple application traffic generator: each source period-ically sends one packet (of 1 KB in size) to a randomlyselected peer with an exponentially distributed inter-arrivaltime.

The behaviors of the two implementations differedslightly owing to variations in treatment of the AODV spec-ifications. In addition, the ActComm AODV protocol ranin user space using IP tunneling and UDP sockets, whileSWAN AODV ran directly on top of IP. The messagesfrom the application traffic generator, when delivered tothe ActComm AODV protocol through the IP tunnel, werewrapped with UDP and IP headers. Both the data and con-trol messages used by ActComm AODV were also aug-mented with UDP headers through UDP sockets. Nonethe-less, we found that, with varying traffic load, the overallpacket ratio—which is the total number of packets receivedby the application layer divided by the total number of pack-ets sent—differed only slightly between these two imple-mentation (less than 3%). The similarity in the behavior ofthe two implementations ensures that using the two imple-mentations to assess the cost of direct execution is mean-ingful.

Figures 2 and 3 show the difference in total executiontime and peak memory usage between the two implementa-tions of the AODV protocol. Clearly, the ActComm AODV(direct-execution) implementation requires more computa-tional resources, but marginally so. The greatest increasein the execution time (about 18%) is at larger network sizeand heavier traffic load. The increased execution time ismostly caused by the overhead of copying and serializationof real packets. The memory overhead of ActComm AODV(over 100%) is more significant. We attribute it to the ad-ditional data structures used by the direct-execution proto-col session, the IP tunnel device, and the UDP socket layer,which are proportional to the number of simulated mobilestations. Moreover, in simulation, the directly executedrouting protocol and the application send and receive realpackets with real message headers and real payloads. Theoverhead grows with increasing traffic intensity as packetsstay longer in the wireless network due to more contentions.

In conclusion, direct-execution simulation requires morecomputational resources, especially in memory usage. Thebenefit of directly executing a routing protocol implementa-tion in simulation is the assurance that the protocol imple-mentation exhibits the same behavior as in a real network. Arouting protocol model implemented natively in the simula-tor, however, may benefit from computational optimizationssuch as eschewing actual message headers and payloads.Thus, a protocol model, once validated, can be used in situ-ations where the resource requirement is critical, such as ina simulation of a large-scale wireless network. On the otherhand, the extra costs of direct-execution are not so onerousthat it disqualifies the technique as a means of experimen-

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

0

2

4

6

8

10

12

14

16

0.1 1 10

Exe

cutio

n Ti

me

(in 1

,000

sec

onds

)

Traffic Intensity (in packets/second)

200 nodes, dirx ActComm AODV200 nodes, SWAN AODV100 nodes, dirx ActComm AODV100 nodes, SWAN AODV50 nodes, dirx ActComm AODV50 nodes, SWAN ADOV

Figure 2. The simulation time for the twoAODV implementations with varying trafficload (in log scale).

0

10

20

30

40

50

60

70

80

90

0.1 1 10

Pea

k M

emor

y U

sage

(in

MB

)

Traffic Intensity (in packets/second)

200 nodes, direx ActComm AODV200 nodes, SWAN AODV100 nodes, direx ActComm AODV100 nodes, SWAN AODV50 nodes, direx ActComm AODV50 nodes, SWAN ADOV

Figure 3. Peak memory usage by the twoAODV implementations with varying trafficload (in log scale).

tation. There are obvious advantages to maintaining a com-mon code base between a protocol’s actual implementation,and that used to study its behavior in a simulator.

5.2. Simulation vs. Reality

As the second step in our validation, we compared theresults from an outdoor routing experiment with our simu-lation results. In particular, we compared the results fromthe real experiment with the simulation results using differ-ent radio propagation models. The purpose of this study isto reveal the sensitivity of the performance of the routingprotocols to the underlying wireless models.

5.2.1 The Real Experiment

The outdoor routing experiment took place on a rectangularathletic field measuring approximately 729 by 1408 feet (or

222 by 429 meters). Each of the 40 laptop computers usedin this experiment had a Lucent (Orinoco) 802.11B wire-less card operating in peer-to-peer mode at 2 MB/s. Eachlaptop had a Garmin eTrex GPS unit attached via the serialport. These GPS units did not have differential GPS capa-bilities, but were accurate to within 10 meters during theexperiment.

For this particular outdoor experiment, we includedAPRL, AODV, ODMRP and STARA (GPSR was still un-der development). The laptops, whose clocks were set to thetime reported by the GPS unit, automatically ran each rout-ing algorithm for 15 minutes, with two minutes of networkquiescence between each algorithm to handle cleanup andsetup chores. After each routing algorithm had been run-ning for one minute, providing time to reach an initial sta-ble routing configuration, the laptops automatically started atraffic generator that generated “streams” of UDP packets.The number of packets in each stream was Gaussian dis-tributed with mean 5 and standard deviation

√2; the time

between streams was exponentially distributed with mean15 seconds; the time between packets inside a stream wasexponentially distributed with mean 3 seconds; every packetcontained approximately 1200 data bytes; and the targetlaptop for each stream was uniformly randomly selectedfrom among the other laptops. We chose these numeri-cal parameters to approximate the (moderate) traffic volumeobserved during an earlier demonstration of a military ap-plication. The routing algorithm parameters, such as thebeacon interval for APRL and the forwarding group life-time for ODMRP, were set to “standard” values taken fromthe literature and our own experience.

During the course of the experiment, the laptops werecontinuously moving. The athletic field was divided intofour equal-sized quadrants, one of which was approxi-mately eight feet lower in elevation than the rest of the field.The hills from the higher to lower elevation were steep andshort, and thus did obstruct the wireless signal, increasingthe frequency with which the routing algorithms needed tofind a multi-hop route. At the start of the experiment, the40 participants were divided into equal-sized groups of 10each, each of which was instructed to randomly disbursein one of the four quadrants. The participants then walkedcontinuously, always picking a quadrant different than theone in which they were currently located, picking a randomposition within that quadrant, walking to that position in astraight line, and then repeating. This approach was cho-sen since it was simple, but still provide continuous move-ment to which the routing algorithms could react, as well assimilar laptop distributions across each of the four routingalgorithms.

Each laptop recorded extensive logs as described in Sec-tion 4. At the end of experiment, we discovered that sevenlaptops failed to generate any data or routing traffic due to

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

misconfiguration or hardware problems. Thus, the experi-ment, in practice, reduced to a 33-laptop experiment and thelogs from these 33 laptops were used as the starting point forcomparing the real-world and simulated results.

5.2.2 The Simulation

We processed the logs from the real experiments to derivethe mobility and radio connectivity traces for each laptopfor the duration of running each routing algorithm. We ranthe simulation for each algorithm for the designated period.We directly ran the routing protocol and the service mod-ule in each simulated mobile station. We modified the ap-plication traffic generator to read the application log andgenerate the same packets as in the real experiment. Wefocused only on the 33 laptops that actually transmitted, re-ceived, and forwarded packets in the real experiments. Toreproduce the traffic pattern in simulation, the applicationtraffic generator on each of the 33 nodes still included the 7crashed nodes as their potential packet destinations.3

The mobile stations in simulation followed the mobil-ity trace generated from the real experiment. We exam-ined three radio propagation models: a free-space model,a two-ray ground reflection model, and a generic propaga-tion model. The simulator delivered each transmitted packetto all neighbor stations that could receive the packet withan average signal power beyond a minimum threshold. Weused the propagation models to determine the power loss foreach packet transmission and calculate the signal-to-noiseratio to quantify the state of interference at the receiver—whether a packet that arrived at a mobile station could bereceived successfully, or dropped due to significant powerloss or collisions. We combined the three models with theconnectivity trace derived from the beacon logs, leading tosix different radio propagation models in simulation: threeusing the connectivity traces and the other three not. In thefirst three cases, we used the connectivity trace to deter-mine whether a packet from a mobile station could reachanother mobile station, and then we used the radio propa-gation models to determine the receiving power for the in-terference calculation. Comparison of models with mea-sured connectivity with those without give us a means ofdetermining whether the model contains accurate predictivepower for connectivity.

5.2.3 The Results

We first examine the packet delivery ratio. Figure 4 showsthe packet delivery ratio from the real experiment and thesimulation runs with six radio propagation models (three of

3Therefore, the packet-delivery ratios, both from the real experimentand the simulation, should be lower than expected, since those packetswith unknown destinations could not be delivered.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

AODV APRL ODMRP STARA

Pac

ket D

eliv

ery

Rat

io

real experimentgeneric model with connectivity

free-space with connectivitytwo-ray with connectivity

generic model no connectivityfree-space no connectivity

two-ray no connectivity

Figure 4. Comparing the data delivery ra-tio from the real experiment with various ra-dio propagation models. “With connectivity”means the connectivity trace was used.

which used the connectivity trace derived from the real ex-periment to determine the reachability of the signals). Eachsimulation result is an average of five runs; the variance isinsignificant and therefore not shown. The generic propaga-tion model in the experiment used typical parameters to de-scribe the outdoor environment of the real experiment: weused 2.8 as the path-loss exponent and 6 dB as the standarddeviation for shadow fading.

We found that the simple generic propagation model of-fered an acceptable prediction of the performance of therouting algorithms, although different propagation modelspredicted vastly different protocol behaviors. The differ-ence is significant in some cases that could result in mis-leading conclusions, for example, when comparing the per-formance of AODV and ODMRP. The inaccuracy in themodel prediction introduced by the propagation model isnon-uniform and can undermine a performance comparisonstudy of different protocols.

For AODV, APRL, and STARA, the figure shows a largeexaggeration of the packet delivery ratio using the free-space model and the two-ray ground reflection model. Bothmodels overestimated the transmission range of radio sig-nals causing shorter routes and therefore better packet de-livery ratio. Even with the connectivity trace, the mod-els overestimated the signal quality, failing to capture thelossy characteristic of the radio propagation environment.The performance of ODMRP was underestimated in sim-ulation. ODMRP is a multicast routing algorithm that de-livers packets using multiple paths to their destinations. Ithas a higher demand on the network bandwidth. The over-estimated transmission range and signal quality in the free-space and two-ray models caused more contentions and cre-ated a negative effect on the simulated throughput.

The packet delivery ratio does not reflect the entire ex-

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

0 1 2 3 4 5

Frac

tion

of T

otal

Dat

a P

acke

ts in

Tra

nsit

Hop Count

real experimentgeneric model with connectivity

free-space with connectivitytwo-ray with connectivity

generic model no connectivityfree-space no connectivity

two-ray no connectivity

Figure 5. The hop-count histogram of AODVin real experiment and in simulation.

ecution environment of the routing algorithm. Figure 5shows a histogram of the number of hops that a data packettraversed in AODV, before it either reached its destinationor dropped along the path. For example, a hop count ofzero means that the packet was dropped at the source node;a hop count of one means the packet went one hop: ei-ther the destination was the source’s neighbor or the packetfailed to reach the next hop. The figure shows the frac-tion of the data packets that traveled in the given numberof hops. We see clearly the free-space and two-ray modelsresulted fewer hops by exaggerating the transmission range.We also see that the connectivity trace was helpful in pre-dicting the route lengths, which confirms that the problemwith the free-space and two-ray models using the connec-tivity trace was that they did not consider packet losses dueto the variations in receiving signal power.

The generic propagation model with typical parametersto represent the outdoor test environment offered a rela-tively good prediction of the performance of the routing al-gorithms. However, one must carefully choose the correctparameters to reflect the wireless environment. The expo-nent for the distance path loss and the standard deviation inlog-normal distribution for the shadow fading are heavilydependent on the environment under investigation. In thenext experiment, we ran a simulation with the same num-ber of mobile stations and with the same traffic load as inthe real experiment. Figure 6 shows AODV performancein packet delivery ratio with the same network setting butvarying the path-loss exponent and the shadow log-normalstandard deviation.

The AODV behavior was more sensitive to the path-lossexponent than to the shadow standard deviation. That is, thesignal propagation distance had a stronger effect on the al-gorithm’s performance. A shorter transmission range meanspackets must travel through more hops (via longer routes)before reaching its destination, and therefore has a higher

2

2.5

3

3.5

4

02

46

810

12

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Path Loss ExponentShadow Stdev

Pac

ket D

eliv

ery

Rat

io

Figure 6. Sensitivity of AODV performance toparameters of large-scale fading model.

probability to be dropped. A larger shadow standard de-viation caused the links to be more unstable, but the ef-fect varied. On the one hand, when the path-loss exponentwas small—the signals had a long transmission range, thesmall variation in the receiving signal strength did not havea significant effect on routing, causing only infrequent linkbreakage. On the other hand, when the exponent was large,most nodes were disconnected. A variation in the receiv-ing signal power helped establish some routes which wereimpossible if not for the signal power fluctuation. Betweenthe extremes, a larger variation in the link quality generallycaused more transmission failures, and therefore resultedslightly lower packet delivery ratio.

The critical implication of this sensitivity study is thatwe cannot just grab a set of large-scale fading parameters,use them, and expect meaningful results for any specificenvironment of interest. On the one hand, pre-simulationempirical work to estimate path-loss characteristics mightbe called for, if the point of the experiment is to quantifybehavior in a given environment. Alternatively, one may re-quire more complex radio models (such as ray-tracing) thatinclude complex explicit representations of the domain ofinterest. On the other hand, if the objective is to compareprotocols, knowledge that the generic propagation model isgood lets us compare protocols using a range of path-lossvalues. While this does not quantify behavior, it may allowus to make qualitative conclusions about the protocols overa range of environments.

To summarize, we used simple stochastic radio propaga-tion models and the traces generated from a carefully de-signed real experiment. Direct-execution simulation pro-vided a common baseline for comparing the behavior ofrouting protocols both in the real experiment and in sim-ulation. We found that it is critical to choose a proper wire-less model that reflects a real-world scenario for studyingthe performance of ad-hoc routing algorithms. In contrast

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE

to earlier studies [12], we found that using a simple stochas-tic radio propagation model with parameters typical to theoutdoor environment can produce acceptable results. Wemust recognize, however, the results are sensitive to theseparameters. It is for this reason we caution that the conclu-sions drawn from simulation studies using simple propaga-tion models should apply only to the environment they rep-resent. The free-space model and the two-ray model, whichexaggerate the radio transmission range and ignore the vari-ations in the receiving signal power, can largely misrepre-sent the network conditions.

6. Conclusions

This paper reports our effort to support direct-executionsimulation of a set of wireless ad-hoc routing protocols tofacilitate validation of wireless network models.

In an experiment, we compared two implementationsof the AODV protocol: one with direct execution and theother implemented natively in the simulator. We found thatdirect-execution simulation requires more computationalresources, especially in memory usage, thus making themodeled protocol more attractive in a resource-constrainedsituation, such as studying protocol behaviors in a large net-work environment. The CPU overhead of direct-execution,however, is moderate and in most case cannot keep direct-execution simulation from being a valuable means of ex-perimentation with the obvious advantage of maintainingconsistency between a protocol’s actual implementation andthat used in simulation.

We conducted a real experiment running the protocolson 40 laptop computers in an outdoor environment. Weembedded a sophisticated logging mechanism in the pro-tocol implementations. All activities related to the rout-ing algorithms and the applications were recorded in files.Post-processing these files results in traces that we used insimulation to reproduce the same network condition. Wefound that one can use a simple stochastic radio propaga-tion model to predict the behavior of the routing protocolswith fairly good accuracy, but the results are quite sensitiveto the model’s parameters. We argue that choosing a properwireless model that represents the wireless environment ofinterest is critical in performance evaluation of the routingalgorithms.

Our future work includes further analysis to validate dif-ferent wireless models under different real experimentalconditions. We are currently investigating using the linkquality information collected by the wireless device driverto improve the accuracy of the connectivity trace. Also, wewant to translate the terrain information of the real experi-ment into a radio propagation gain matrix for a more real-istic representation of the wireless environment, and studythe effect of such modeling details on the performance eval-uation of wireless ad-hoc routing protocols.

7. Acknowledgments

We thank Nikita Dubrovsky, Aaron Fiske, Chris Masone,and Michael DeRosa for implementing the routing algo-rithms and most of the outdoor experiment infrastructure.Piyush Gupta and Brad Karp helped with the source code ofSTARA and APRL. Dennis McGrath helped with the initialclass structure for the routing algorithms. We thank ChipElliott at BBN for his valuable insights and suggestions. Wealso thank Lisa Shay, Susan McGrath, and Eileen Entin fordesigning the application scenario, and the sixty Dartmouthstudents and staff members who participated the outdoorexperiments.

References

[1] X. A. Dimitropoulos and G. F. Riley. Creating realistic BGPmodels. MASCOTS’03, October 2003.

[2] P. Gupta and P. R. Kumar. A system and traffic dependentadaptive routing algorithm for ad hoc networks. 36th IEEEConference on Decision and Control, pages 2375–2380, De-cember 1997.

[3] J. Heidemann, N. Bulusu, J. Elson, C. Intanagonwiwat,K. Lan, Y. Xu, W. Ye, D. Estrin, and R. Govindan. Effects ofdetails in wireless network simulation. SCS Multiconferenceon Distributed Simulation, pages 3–11, January 2001.

[4] D. B. Johnson. Validation of wireless and mobile networkmodels and simulation. DARPA/NIST Network SimulationValidation Workshop, May 1999.

[5] B. Karp and H. T. Kung. Dynamic neighbor discovery andloopfree, multi-hop routing for wireless, mobile networks.Hardvard University, May 1998.

[6] B. Karp and H. T. Kung. Greedy perimeter stateless rout-ing for wireless networks. MobiCom’00, pages 243–254,August 2000.

[7] S. J. Lee, M. Gerla, and C. C. Chiang. On-demand multi-cast routing protocol in multihop wireless mobile networks.Mobile Networks and Applications, 7(6):441–453, Decem-ber 2002.

[8] J. Liu, L. F. Perrone, D. M. Nicol, M. Liljenstam, C. El-liott, and D. Pearson. Simulation modeling of large-scale ad-hoc sensor networks. European Simulation InteroperabilityWorkshop (Euro-SIW’01), June 2001.

[9] M. Neufeld, A. Jain, and D. Grunwald. Nsclick: bridgingnetwork simulation and deployment. MSWiM’02, pages 74–81, September 2002.

[10] C. E. Perkins and E. M. Royer. Ad hoc on-demand distancevector routing. 2nd IEEE Workshop on Mobile ComputingSystems and Applications, pages 90–100, February 1999.

[11] T. S. Rappaport. Wireless Communications, Principles andPractice. Prentice Hall, 1996.

[12] M. Takai, R. Bagrodia, K. Tang, and M. Gerla. Effi-cient wireless network simulations with detailed propaga-tion models. Wireless Networks, 7(3):297–305, May 2001.

[13] M. Takai, J. Martin, and R. Bagrodia. Effects of wirelessphysical layer modeling in mobile ad hoc networks. Mobi-Hoc’01, pages 87–94, October 2001.

Proceedings of the 18th Workshop on Parallel and Distributed Simulation (PADS’04)

1087-4097/04 $20.00 © 2004 IEEE