Towards Improving Packet Probing TechniquesMatthew J. Luckie, Anthony J. McGregor, Hans-Werner Braun.

Abstract-Packet probing is an important Internet mea-surement technique, supporting the investigation of packetdelay, path, and loss. Current packet probing techniques useInternet Protocols such as the Internet Control Message Pro-tocol (ICMP), the User Datagram Protocol (UDP), and theTransmission Control Protocol (TCP). These protocols werenot originally designed for measurement purposes. Currentpacket probing techniques have several limitations that canbe avoided. The IP Measurement Protocol (IPMP) is pre-sented as a protocol that addresses several of the limitationsdiscussed.

Keywords- Packet Probing Techniques, Packet Delay,Network Path.

I. INTRODUCTION

As the Internet grows in scale and complexity, the needfor measurement increases. The underlying need for mea-surement is to understand why the Internet behaves theway it does in complex conditions. One form of mea-surement is active measurement; this involves introducingpackets into the network and measuring the way the net-work handles those packets. This paper focuses on packetdelay measurements.

Measurement packets can be encapsulated in existingprotocols such as the Internet Control Message Protocol(ICMP) [l], the User Datagram Protocol (UDP) [2], andthe Transmission Control Protocol (TCP) [3]. Examplesof packet probing techniques that are encapsulated in theseexisting protocols are ping, traceroute, and the IPPerformance Metrics (IPPM) group’s One-way Delay Pro-tocol (OWDP) [4]. The protocols used to encapsulate thesemeasurement packets were not designed with measure-ment as a consideration. There may be serious limitationsto measurements encapsulated in these protocols.

The authors are with the National Laboratory for Applied NetworkResearch (NLANR), Measurement and Network Analysis Group, SanDiego Supercomputer Center, University of California, San Diego, LaJolla, USA and the University of Waikato, Hamilton, New Zealand.Email: mjl,tonym,hwb@nlanr.net

This work is funded, in part, by NSF Cooperative Agreement No.ANI-9807479. The U.S. government has certain rights to this material.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page. To copyotherwise, or republish, to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.IMW’OI, November l-2.2001, San Francisco, CA, USA.Copyright 2001 ACM I-581 13-435-5/01/0011...$5.00.

Current packet probing techniques are not suited to mea-suring packet delay at the router level. This makes the taskof identifying where delay occurs in the network more dif-ficult. In addition, current approaches to ensuring clocksynchronisation where delay measurements include time-stamps from more than a single clock have a high cost,normally requiring a dedicated external time receiver beinstalled on each host or router involved in a measurement.Despite the implicit requirement for information regardingthe synchronisation of a clock when multiple independentclocks are represented in a measurement, the authors arenot aware of any protocol that provides a mechanism toretrieve this information.

The IP Measurement Protocol (IPMP) [5] is introducedas an example of a protocol that addresses some of the lim-itations of using existing protocols to encapsulate packetprobes. IPMP considers both the packet delay and thepath a packet takes in a single packet exchange betweenthe measurement host and the echo host. In the opinionsof the authors, the protocol is tightly constrained, efficient,and easy to implement. It is hoped that these characteris-tics will make IPMP suitable for implementation by routermanufacturers. This would enable packet delay measure-ments to indicate places of delay due to congestion be-tween two hosts as a single packet passes through the net-work.

This paper is organised as follows. In section II, adiscussion of the current measurement techniques is pre-sented. Limitations of these techniques are presented insection III. Section IV presents the IPMP protocol that in-troduces a new technique that allows path and delay mea-surements to be combined in a single packet exchange.Section V presents some remaining design issues with theIPMP protocol in its present specification.

II . D I S C U S S I O N O F CU R R E N T T E C H N I Q U E S

A. Ping

The most widely used method to investigate network de-lay is for a measurement host to construct and transmit anICMP echo request packet to an echo host as outlined inRFC 792 [l]. Round trip time (RTT) is calculated as thedifference between the time the echo request is sent andthe time a matching response is received by the ping ap-plication.

A variation of this method is to construct an ICMP

145

timestamp request packet, also outlined in RFC 792 [l].This packet contains three timestamps - the originate time-stamp, the receive timestamp, and the transmit timestamp.If both hosts involved in the timestamp exchange have syn-chronised clocks, the forward path delay can be calculatedusing the originate and receive timestamps. The reversepath delay can be calculated using the transmit timestampcontained in the timestamp response packet and the timethe response packet arrived back at the transmitter.

B. Traceroute

The traceroute technique allows a measurementhost to deduce the forward path to an echo host by sys-tematically sending a sequence of IP [6] packets with anincreasing time-to-live (TTL) value that is initially set toone. The forward path is deduced by extracting the sourceIP address from the ICMP TTL expired messages that aresent as successive routers discard the IP packets.

A variation of traceroute is mtrace [7], for in-vestigating the multicast path from the transmitter to a re-ceiver. mtrace uses features that are built into multicastrouters, and thus mtrace does not use ICMP TTL ex-pired messages as its method for tracing a path. mtracemeasures the reverse path from multicast transmitter to re-ceiver, and requires only a single query packet to be sent tothe transmitter for the trace to be conducted for the entirereverse path. The trace is conducted in parallel, with eachrouter sending a trace response to the request in addition toforwarding the request to the next multicast router on thereverse path to the receiver.

C. One- Way Delay Protocol

The IP Performance Metrics (IPPM) group has pub-lished several Request for Comment (RFC) documents thatdefine frameworks for measuring the performance of IPnetworks [8], [9]. The IPPM group is well advanced inthe engineering of a One-way Delay Measurement Proto-col (OWDP) [4] that will build on a framework designedin RFC 2679 [9].

The OWDP specification provides a mechanism formeasuring packet delay with UDP packet probes. In ad-dition, the specification describes a mechanism for con-trolling a measurement session between two hosts with aTCP connection, for negotiating the UDP port numbers in-volved in the delay measurement, and for encrypting thedata carried in the measurement packets to protect againstmanipulation by a third party.

III. LIMITATIONS OFCURRENT TECHNIQUES

A. Separation of Path and Delay Measurements

There are often large variations in delay between suc-cessive packets following the same route, particularlywhen a load balancing arrangement is in place or when anetwork is under high load. The path that successive pack-ets take to the same destination may change during mea-surement, resulting in tools like traceroute possiblyreporting a path that does not exist. This makes the task ofcorrelating a t racerout e measurement to a ping mea-surement difficult. A major problem is that it is more prob-lematic to measure a network under heavy load, preciselywhen measurement is most valuable. A measurement tech-nique that combines path and delay measurement wouldallow a measurement to ascertain not only network delay,but where delay occurs.

B. Limited Ability to Measure to a Router

Routers often make bad measurement targets becausethey are optimised for the relatively simple task of for-warding packets. Routers may process tasks that are re-source intensive and therefore an opportunity for a denialof service attack at low priority or not at all. This hasimplications for path and delay measurements taken withtechniques such as traceroute.

Some network administrators express concern that if theamount of active measurement activity on their networkincreases, significant network resources may be consumedhandling this activity. In many cases the total traffic addedto a network is often a very small fraction of the net-work’s capacity. It is important that packet probes disturbthe network as little as possible. In general, this meansadding the minimum necessary number of packets to thenetwork. Path measurement using tracerout e requiresmany packet probes per path.

Some measurement techniques construct measurementtraffic that can be difficult to efficiently detect amongstother network traffic. This type of measurement traffic pre-cludes measuring to a router.

C . Limited Consideration of Protocol Encapsulation

Packet probes encapsulated in general purpose protocolssuch as ICMP, TCP, and UDP, may be subject to protocolfiltering. This may result in delay measurements that notonly consider network delay, but protocol filters that maynot be appropriate for traffic encapsulated in other proto-cols. The reason for protocol filtering and rate limiting isoften to prevent denial of service attacks.

The ICMP protocol is filtered at many routers, andmay be blocked entirely despite RFC 18 12 [lo] requiring

146

ICMP. Section 4.3.2.8 of RFC 1812 allows for the router torate limit ICMP replies to avoid the consumption of band-width and the use of router resources. The implication ofthis is that ICMP is not a reliable protocol for conductingdelay and loss measurements.

The encapsulation of packet probes in UDP may beproblematic due to the general-purpose nature of the proto-col. UDP does not contain the well developed congestionmanagement algorithms inherent in TCP and it is possi-ble that UDP packets will be rate limited during periods ofpeak UDP usage in order to reduce their impact on TCPflows.

TCP has limitations for delay and loss measurement.Each TCP packet that arrives at an echo host incurs over-head in the TCP stack, matching that packet with a datastructure representing that TCP connection. This processis comparatively CPU intensive, and thus delay measure-ments encapsulated in TCP will include a component ofdelay introduced by the TCP stack in addition to the net-work delay.

Measurement traffic is subject to protocol-based prior-ity queuing policies that may be deployed in the path be-tween a pair of hosts. The choice of a particular protocoltype for conducting measurements results in a measure-ment that not only considers the propagation delay of thepacket, but also the effects of queuing policy. The impli-cation of protocol-based priority queuing is that if a mea-surement shows a change in delay from previous measure-ments, the change in delay is not necessarily the result ofincreased network load.

D. Limited Clock Support

Packet probing techniques that measure delay to sec-tions of a network, such as that with one-way delay, re-quire synchronised clocks. Clock behaviour is complexand outside the scope of this paper. A good discussion ofthe impact of clock behaviour on delay measurements ispresented in [ 111. The fundamentals of clock theory arethat clocks are of limited precision and that they drift atdiffering rates over time.

One method to address the effects of inaccurate clocksin one-way delay measurements is to insist on echoinghosts using a precise external time source [9], [4], [12]such as those provided by the Global Positioning System(GPS) and the Code Division Multiple Access (CDMA)network. These precise external time sources result in aclock synchronised to real time with an offset of a fewhundred nanoseconds. These methods have limitations.GPS time receivers are expensive and there can be logis-tical difficulties in placing antennae. CDMA networks arenot widely available at present.

In the case of one-way delay measurement, an accuracylimitation in the range of around two milliseconds can beacceptable because this limitation represents only a smallproportion of delay in the actual network. Existing mea-surement protocols do not provide a mechanism to retrieveinformation about clocks located on echo systems, despiteecho systems often knowing how far their clock is fromreal time.

IV. THE IP MEASUREMENT PROTOCOL

The IPMP protocol is based on an echo request and re-ply packet exchange for measuring packet delay and as-sociated path metrics, and is similar to the technique thatping uses with the ICMP echo capabilities. Full detail ofthe protocol can be found in [5].

IPMP is carried directly inside of an IP packet in orderto make an echo packet obvious to the routers connectingthe hosts involved in the measurement. The echo replypacket has been designed so that an echo host can constructan echo reply packet with very few modifications to theecho request packet. The echo protocol packet format ispresented in Figure 1.

: ,: ,: ,: 3IVersion Queue Type Checksum

00000000 TYPO Returned TTL Return Type

Length Path Pomter

(optional) data

(optional) Path Records

Padding (ifrequired)

Fig. 1. The IPMP Echo Request Format

The key fields in the echo packet are as follows. TheVersion field identifies the version of the IPMP protocolbeing used. The Queue Type field identifies the proto-col that the packet should be queued as (e.g., TCP). TheType field identifies the IPMP packet as one of echo re-quest or echo reply. The Returned TTL field identifies theTTL value of the IP packet as it arrives at the destination.The Return Type field becomes the type field for the echoresponse. The symmetric nature of the second 32-bit wordof the packet is intended to make creation of the echo re-ply packet more efficient. The Length field identifies theamount of data that has been allocated in the IPMP packetto store IPMP Path Records. The Path Pointer field identi-fies the offset at which the next IPMP path record shouldbe inserted, assuming that there is available space.

A major difference between the echo exchange in IPMPand other echo protocols is the introduction of the IPMPpath record, presented in Figure 2. Each path record struc-ture requires twelve bytes of data to be allocated in theIPMP packet. The first four bytes of a path record repre-sent the IPv4 address of the network interface the IPMPpacket was received on. If a host inserts a path record tosignify the time the packet leaves the kernel, it uses the net-work interface that the kernel uses to transmit the packeton. The last eight bytes of a path record is a fixed-pointrepresentation of time following the conventions of RFC1305 [13]. The first four bytes is the integer part of thetimestamp; the second four bytes is the fraction part. Thetimestamp represents seconds since January 1900.

In addition to the echo packet exchange, the IPMP pro-tocol contains an information packet exchange. This facil-ity is used to retrieve information regarding precision andaccuracy limitations of a clock represented in a delay mea-surement. In addition, the drift of the clock can be inferredwith linear interpolation if multiple IPMP Real Time Ref-erence Point (RTRP) objects, shown in Figure 3, are in-cluded in the response. The RTRP format presents a pointthat can be used to map between a reported time and theactual real time of a clock.

0 3

Real Time

: 0 I 1 38 6 ,4 IFonvarding Address

Reported Timei

Timestamp Fig. 3. The IPMP Real Time Reference Point

Fig. 2. The IPMP Path Record Format

A discussion of how IPMP addresses the limitationsidentified in Section III is now presented.

The minimum packet size that must be supported by anIP network, 576 bytes, allows for 45 path records to bestored in an echo request packet. The size of the pathrecord would increase to 24 bytes in an IPv6 environ-ment due to the storage requirements of a 16 byte address.In an IPv6 environment, the Minimum Transmission Unit(MTU) of 1280 bytes [ 141 allows for 50 path records to beinserted in an echo packet. Research from CAIDA’s Skit-ter project [ 151 indicates that the average distance betweenthe F root server and a customer of that server is 13 hops,while less than 0.5% of paths are longer than 22 hops. Fora path connecting a pair of hosts that is longer than canbe measured with the MTU-sized packet, a measurementhost may restart the measurement from one of the last IPaddresses in the path record by sending an echo request toan address that will answer the request (if there is such anaddress), or by sending a larger packet if the underlyingnetwork supports the increased size of the packet.

A . Combines Path and Delay Aspects of Measurement

IPMP combines path and delay aspects of measurementby providing an echo packet with predetermined space al-located for routers to insert path records. This provides ameasurement host with an ability to identify paths that arecongested or have an otherwise lengthy propagation delay,for both the forward and reverse paths. The ability to mea-sure to a router is important as the reverse path from therouter is often different from the forward path, as is thecase with hot-potato style routing policy [ 161.

An ISP or other network that wants to be able to dis-cover and demonstrate the degree of delay introduced bytheir network can deploy IPMP path record enabled routersat the boundaries of their network. IPMP echo packetswould have timestamped path records inserted as they passthrough border routers in the forward and reverse path.

In addition to the ability to infer end-to-end delay bysubtracting the time that a measurement packet was sentfrom the time when the packet returned, the echo requestpacket can be used to deduce path length in hops for eachdirection, and one-way delay if the echo host inserts a pathrecord when it responds to the echo request. Providedthat adequate space has been allocated in the echo requestpacket, each router or host that handles the packet is ableto insert a path record into the echo packet that signifies theIP address of the router and the time at which the packetwas processed.

B . Designed for E@cient Handling by Routers

The protocol is designed so that it can be manipulatedin an efficient manner. Path records are placed in allocatedspace in the packet. Simple word manipulations allow therouter to transform an echo request packet into an echoreply packet, removing the need for the IPMP checksumto be recalculated over this portion of the packet.

This makes the protocol more suitable for a kernel im-plementation where timestamps can be recorded that avoidpotential user-space process switching that can result in aless accurate timestamp [8]. The encapsulation of a packetprobe with a separate protocol type allows for more flex-

148

ible filtering and may avoid measurement activities frombeing blocked due to administrative policies designed toblock other packets. Administrative filters may rate limitor block ICMP packets, UDP packets, and even TCP-SYNpackets in order to limit the impact of a denial of serviceattack.

The design decision to make IPMP easy to implementfor router manufacturers requires IPMP traffic to be obvi-ous so that routers may participate in measurements in anefficient manner. This is not to say that there are no ways toengineer IPMP traffic to be less obvious, and to allow theprotocol to be used where administrative blocks or filtersmight otherwise prevent doing so. One possible method isto encapsulate an IPMP packet with IPSec using Transportmode [17] and an Encapsulating Security Payload (ESP)header [ 181 to hide the IPMP protocol type. Doing somakes it impossible to collect path information in additionto the packet delay measurement and precludes a routerfrom being a measurement target. The protocol remainsuseful for measurement of one-way delay between two au-thenticated hosts, although decrypting an echo packet in-troduces overhead onto the measurement hosts.

C. Provides a Mechanism for Priority-based Queuing

The IPMP echo protocol format has a queue type field asshown in Figure 1. The purpose of this field is to allow arouter to queue a probe packet as if it were another protocolsuch as TCP, UDP, or ICMP. In effect, it creates a numberof performance profiles depending on the protocol in use.

In reality, some routers may queue packets based on fivevalues: the source and destination IP address, the sourceand destination port numbers, and the protocol type. Atpresent, an echo packet contains the protocol type field andthe source and destination IP addresses. The echo packetrequires the addition of two 16-bit words that representthe source and destination ports to allow routers to queueIPMP echo packets based on the five-tuple.

In addition, consideration may need to be given to theeffect of the Differentiated Services Field in the IPv4header as outlined in RFC 2474 [ 191. This field is a sub-stitute for the Type of Service (TOS) field in the IPv4, andthus measurements taken with IPMP may need to considerthe effects of this field in addition to the five-tuple.

A measurement needs to be made in the context of aspecific protocol, as outlined in RFC 2330 [8]. IPMP pro-vides the potential to support this by allowing the measure-ment to request that the measurement traffic be treated asif it were another protocol for the purpose of filtering thepacket.

D. Provides a Mechanism for Exchanging Informationabout Clocks

The IPMP information packet provides a mechanism fora measurement host to collect information from hosts androuters that include time information in delay measure-ments. The information packet can be used to establishthe accuracy of individual clocks that are represented in anecho packet.

In addition, this mechanism allows a measurement hostto conduct measurements with echo hosts that do not haveexternal precision time sources, but do have a mechanismto retrieve clock offset data with known accuracy limita-tions. The separation of timestamp generation and correc-tion to real time reduces the effort required by a router toimplement IPMP and allows more sophisticated analysisby the measurement system. An IPMP information packetallows measurement hosts to ascertain, in a simple manner,the stability of a clock over a period of time and to makeefficient adjustments to the reported time with a degree ofcertainty.

The need for a precise pair of clocks for conducting one-way delay measurements in the Internet has been noted[8]. Indeed, some papers state that the use of NTP has adetrimental effect on one-way delay measurements [ 121.That simply installing an NTP daemon on a measurementhost will lead to invalid one-way delay measurements isnot disputed. It is well established that an NTP daemonshould not synchronise to hosts over network paths that area significant component of a path that is being measuredfor asymmetrical delay properties [20].

Studies such as [ 111, [21] have presented algorithms fordetecting clock adjustments between two hosts involved inone-way delay measurement. An experimental approach isto monitor system calls to ntp-adjtime by the NTP daemonand to make this information available to IPMP informa-tion requests. The default action of the NTP daemon is toamortise the offset from real time at a constant rate if it isless than 128ms from what it understands to be real time.

The information passed to the ntp-adjtime system callcan give useful information regarding the clock’s offsetfrom real time, and an estimation of how accurate this off-set is. The estimation of the accuracy of the offset valueis the most interesting piece of information for measure-ment purposes, as a measurement host may compensatefor the change in the offset over a period of time with lin-ear interpolation if the accuracy limitation of the offset isacceptable. A measurement host may present the forwardand reverse path delay with an estimated accuracy limita-tion, balancing the cost and utility of delay measurements

WI.

149

V. REMAINING IS S U E S REFERENCES

A. Per-Hop TTL Information .I. Pastel, “Internet control message protocol,” RFC 792, IETF,1981.

When an IPMP echo packet passes through a router orhost, an IPMP path record may be inserted if there is suf-ficient pre-allocated space available. As shown in Figure2, the path record contains an eight byte representation oftime on that router. If the seconds portion of the time-stamp was restricted to be relative to midnight UTC, thetimestamp would require 17 bits to represent the secondsportion of the timestamp from the current allocation of32 at present. The first 8 bits of the remaining 15 bitsavailable could store the TTL of the packet as it enters arouter. This information would allow analysis of the echoresponse packet to consider the placement of a router be-tween two points on a network without relying on all pre-ceding routers inserting a path record.

111

PI[31141

151

161[71

Fl

[91

[lOI

illI

.I. Postel, “User datagram protocol,” RFC 768, IETF, 1980.J. Postel, “Transmission control protocol,” RFC 793, IETF, 198 1.S. Shalunov, B. Teitelbaum, and M. Zekauskas, ‘A one-way delaymeasurement protocol,” IPPM work in progress, IETF, 200 1.A. J. McGregor, “The IP measurement protocol,” http://moat.nlanr.net/AMP/AMP/IPMPl.

v21B. The Role of Router Policy and IPMP

It is difficult to engineer a measurement packet thatrouters can understand and process almost as efficientlyas any other IP packet without making the packet very ob-vious amongst other packets. This is an important facetof the IPMP echo packet, due to the desire for routers toinsert IPMP path records so that per-hop delay can be con-sidered.

v31

[I41

v51

U61

A serious prospect is the issue of an administrator block-ing IPMP measurement traffic entirely, rendering the pro-tocol less useful for measuring the path between two hosts.As discussed in IV-B it is still possible to consider one-way delay by using mechanisms provided by IPSec.

VI. CONCLUSION

The limitations of existing packet probing techniqueswere discussed. The IPMP protocol was presented asa framework for discussing the issues of current packetprobing techniques. The IPMP protocol addresses someof the limitations of existing protocols by providing a pro-tocol that combines path and delay measurements, a mech-anism to profile the handling of a particular protocol type,and a mechanism to retrieve echo host clock information.These mechanisms are provided in a protocol that is ef-ficient and designed for a plausible implementation byrouter manufacturers.

Cl71

[I81

[I91

rw

[211

1221

J. Postel, “Internet protocol,” RFC 791, IETF, 198 1.S. Casner, “The mtrace (8) manual page,” http://ftp.parc.xerox.corn/pub/net-researchipmulti/.V. Paxson, G. Almes, J. Mahdavi, and M. Mathis, “Frameworkfor IP performance metrics,” RFC 2330, IETF, 1998.G. Almes, S. Kalidindi, and M. Zekauskas, “A one-way delaymetric for IPPM,” RFC 2679, IETF, 1999.F. Baker, “Requirements for IP version 4 routers,” RFC 1812,IETF, 1995.V. Paxson, “On calibrating measurements of packet transit times,”in Proceedings of ACM SIGMETRICS, Madison, WI, June 1998,pp. 11-21.A. Pasztor and D. Veitch, “A precision infrastructure for activeprobing,” in Proceedings of the PAM2001 workshop on Passiveand Active Measurements, Amsterdam, Apr. 200 1.D.L. Mills, “Network time protocol (version 3): Specification,implementation and analysis,” RFC 1305, IETF, 1992.S. Deering and R. Hinden, “Internet protocol, version 6 (IPv6)specification,” RFC 2460, IETF, 1998.Cooperative Association for Internet Data Analyis (CAIDA),“Hop count distribution,” http:l/www.caida.org/tools/measurementlskitter/RSSAC/.V. Paxson, “End-to-end routing behavior in the Internet,”IEEE/ACM Transactions on Networking, vol. 5, no. 5, pp. 601-615, 1997.S. Kent and R. Atkinson, “Security architecture for the Internetprotocol,” RFC 2401, IETF, 1998.S. Kent and R. Atkinson, “IP Encapsulating Security Payload(ESP),” RFC 2406, IETF, 1998.K. Nichols, S. Blake, F. Baker, and D. Black, “Definition of theDifferentiated Services field (DS field) in the IPv4 and IPv6 head-ers,” RFC 2474, IETF, 1998.K.C. Claffy, G.C. Polyzos, and H-W. Braun, “Measurement con-siderations for assessing unidirectional latencies,” Znternetwork-ing: Research andExperience, vol. 4, no. 3, pp. 121-132, 1993.S.B. Moon, P. Skelly, and D. Towsley, “Estimation and removal ofclock skew from network delay measurements,” in Proceedingsof I999 IEEE INFOCOM, New York, NY, Mar. 1999.A.J. McGregor and H-W. Braun, “Balancing cost and utility inactive monitoring: The AMP example,” in Proceedings of INET2000, Tokyo, Japan, July 2000.

A C K N O W L E D G E M E N T S

The authors are thankful for the valuable advice and ef-fort expended by our shepherd, Vern Paxson, and to theeditorial assistance provided by Maureen C. Cm-ran.

150