Improving performance of transport protocols in multipath transferring schemes

Available online at www.sciencedirect.com

www.elsevier.com/locate/comcom

Computer Communications 30 (2007) 3270–3284

Improving performance of transport protocols in multipathtransferring schemes

Maysam Yabandeh *, Sajjad Zarifzadeh, Nasser Yazdani

Router Laboratories, ECE Department, Faculty of Engineering, University of Tehran, Iran

Available online 2 March 2007

Abstract

One major drawback of multipath transferring schemes inspired by usage of different paths with diverse delays is the emergence ofreordering among packets of the same flow. In this paper, we present two separate approaches to resolve this problem for UDP and TCPconnections. By properly scheduling the packets among multiple paths, our UDP-based approach tries to deliver data to the receiver in-order, while imposing a minimum possible delay and small buffer space on the receiver’s application. We theoretically prove the opti-mality of the proposed method and then present its analytical results. Unfortunately, in the case of TCP, the reordering intensifiesthe problem by bringing more timeouts and many unnecessary fast-retransmits which eventually degrades the throughput of TCP con-nections considerably. To address these issues, we first present the general conditions that should be held to avoid timeouts in multipathschemes. Then, we enhance our approach by preventing nonessential fast-retransmit/recovery events in TCP. Moreover, we introduce ananalytical model to estimate the probability of triggering 3rd duplicate ACK in our method. Finally, through simulation experiments weshow that the performance of our multipath methods is comparable with the optimal one-path transmissions (with aggregated band-width); especially, in terms of throughput and fast-retransmit ratio parameters.� 2007 Elsevier B.V. All rights reserved.

Keywords: Multipath transferring; Transport protocols; Reordering; Multimedia communication; Delay

1. Introduction

The idea of multipath routing was initially used in wirednetworks to achieve load balancing [1] and more aggregatebandwidth. The applications which operate based on mul-tipath routing were always vulnerable in front of the prob-lems caused by usage of different paths with diverse delays.By emergence of Bluetooth [2] and 802.11, multipathscheme has found new applications in wireless networksto handle its inherent limitations (e.g., limited bandwidthand high error rates) [3–6]. A new emerged problem withwireless networks is the potential interference betweenpaths. To overcome this deficiency, the selected pathsshould be completely disjoint and also separated geograph-ically as far as possible. Utilizing more diverse paths

0140-3664/$ - see front matter � 2007 Elsevier B.V. All rights reserved.

doi:10.1016/j.comcom.2007.02.017

* Corresponding author. Tel.: +98 21 61114352; fax: +98 21 88778690.E-mail addresses: [email protected] (M. Yabandeh), szarifza-

[email protected] (S. Zarifzadeh), [email protected] (N. Yazdani).

implies more discrepancy between their delays. On theother hand, recent study [7] shows that load balancingcould not be really achieved in multipath schemes, unlessnumerous paths are used to convey the packets. However,using more paths entails more variations of delays amongpaths.

Plainly, such a scheme causes the transmitted packets toexperience different delays in both wired and wireless net-works. As a first result, this yields the packets to be deliv-ered with a different order with respect to what the sourcehas sent. This reordering results in (i) more demanded buf-fer space; and (ii) extra delays for the receiver’s application.Specifically, reordering in TCP inspires (i) unwantedretransmission due to TCP’s fast-retransmit mechanism;and (ii) bandwidth degradation due to TCP’s fault recoveryand timeout mechanism. Thus far, most of the multipathschemes have tried to bypass the reordering problem bystreaming all of the packets belonging to the same flowthrough a single path [3–6]. In fact, they divide the

mailto:[email protected]




M. Yabandeh et al. / Computer Communications 30 (2007) 3270–3284 3271

available flows individually between paths. Nevertheless, asreported in [8], a per-packet granularity results in muchbetter performance than the explained per-flow streamingapproach.

In this paper, we propose two novel end-to-end stream-ing mechanisms to forward packets of a single flow throughmultiple paths; one for UDP and another for TCP connec-tions. In our network model, which is presented in detail inSection 4, the delay of paths are not supposed to be con-fined by domination of either bandwidth or propagationdelay. Our approach for UDP schedules transmission ofpackets over multiple paths in such a way that they arereceived at the destination in-order while imposing the min-imum overall delay on the receiver’s application. Its TCPcounterpart significantly reduces the number of unwantedfast-retransmits and timeout events by properly configur-ing splitting of data over multiple paths at IP layer. It isnoticeable that we concentrate here only on the end-to-end multipath scheme. This is different from thoseapproaches that split a flow through multiple paths atintermediate routers, like the one proposed in [14] and[16]. In other words, in our scheme the sender is entirelyresponsible for managing multiple paths and transferringits packets through them.

The remainder of the paper is organized as follows. InSection 2, we survey the previous related works. Section3 presents the motivation behind our work and Section 4illustrates the assumed network model. In Section 5, theproposed method for UDP connections is presented. Itscounterpart for TCP connections is introduced in Section6. Our analysis is verified through simulation in Section7. Finally, we conclude the paper in Section 8.

2. Related works

In [10], Mao et al. extended Realtime Transmission Pro-tocol (RTP) to support the use of multiple paths in Multi-path Realtime Transmission Protocol (MRTP), anapplication layer protocol that could use UDP as trans-port. MRTP specifies session establishment, maintenanceand scheduling mechanisms over multiple paths.

Thus far, several algorithms have been proposed toeliminate the effects of reordering initiated by intermediaryrouters [11,12]. In end-to-end multipath scheme, we discussthe reordering introduced by sender due to the usage ofmultiple different paths. In this context, to enjoy the virtuesof multipath scheme without affecting end-to-end TCP per-formance, modifications to TCP have also been proposed,like the one proposed in [13]. These proposals modify andaugment TCP’s congestion control mechanisms to copewith reordering arise from the usage of multipath routing.Since in such schemes the sender has more information toeffectively handle this reordering, most of work should bedone in sender side.

Recent studies have revealed that false fast-retransmitscan be reduced by adapting the value of dupthresh (thethreshold of duplicate acknowledgement at which the

TCP fast-retransmit algorithm is triggered) according tothe underlying network conditions [11,14]. However, theapproach in [14] imposes excessive computational and stor-age overheads to construct a reordering histogram,whereas the techniques proposed in [11] may not well adaptto changing network conditions promptly. Moreover, Maand Leung [15] have studied this approach under multipathnetwork scenarios while they does not investigate the per-formance of the proposed method in networks with diversedelays and bandwidths.

Phatak and Goff [9] proposed distributing data at thenetwork (IP) layer transparent to the higher layers usingIP-in-IP encapsulation. Under the questionable assump-tion that the end-to-end delays are dominated by transmis-sion delay, they presented conditions under which theirmechanism would work without triggering incorrectretransmission timeouts. However, they failed to addresskey issues such as (i) reordering; (ii) propagation delaydominated paths; and (iii) paths with dynamically changingbandwidth or delay. In this paper, our model is notrestricted by these assumptions and we provide a probabi-listic model to measure the performance of our methodunder dynamically changing delay conditions.

3. Motivation

Multipath routing can provide load balancing, higheraggregate bandwidth, and fault-tolerance in network. Loadbalancing can be achieved by spreading the traffic alongmultiple routes. This can alleviate congestion and load ofbottlenecks. As a result of exploiting multiple paths, higheraggregate bandwidth can also be obtained for networkconnections. From a fault tolerance perspective, multipathrouting can provide route resilience against router or linkfailure. However, the usage of multipath transferringschemes creates some new challenges for transport proto-cols, i.e., UDP and TCP. In UDP, such schemes usuallydemand more buffer space at the receivers, due to higherpotential of reordering. This makes the application of mul-tipath transferring hardly possible for receiver devices withlimited resources, like handheld PDAs. In TCP, one prob-lem is that the RTT (Round Trip Time) estimation is neveraccurate under multipath routing. Different delays over dif-ferent paths lead to unstable RTT estimation and incorrectRTO (Retransmission Timeout) setting. For example, if theRTT difference of the longest path and the shortest pathwas large, TCP could prematurely timeout packets on thelongest path due to the incorrect RTT estimation. More-over, it is likely that the packets going through differentpaths arrive at destination out-of-order. Out-of-orderpackets trigger duplicate ACKs, which in turn cause unnec-essary TCP congestion reaction, namely fast-retransmit/recovery.

The main motivation behind this work is indeed to han-dle the above problems by introducing a new approach fordata transmission over multiple paths. Generally, our ideais to schedule the packets at the sender among multiple

3272 M. Yabandeh et al. / Computer Communications 30 (2007) 3270–3284

paths in a way that they arrive at the receiver in-order. Toachieve this goal, we consider two design principles in ourwork: (i) we stream the packets over paths of larger delayssooner than those with smaller delays, and (ii) we utilizepaths according to their available bandwidths. Since pack-ets of slower paths will receive at the destination later thenthose of faster paths, the first principle allows schedulingthe packets to be delivered in-order. As we describedabove, each of UDP and TCP transport protocols has adifferent kind of problems when combined with a multi-path scheme; i.e. larger buffer size for UDP and moretransmission timeout and fast-retransmit for TCP. Hence,we describe the details of our approach separately for eachtransport protocol.

Fig. 1. Timeline of sequential transmission steps in the presented model.The transmission time of bulk j lasts tj seconds and the delay of path k isdk.

4. Network model

In the rest of paper, the model similar to the one used by[9] will be considered. Assume two network nodes A and B,where node A has some amount of data towards node B.Moreover, assume that there exist n distinct paths from A

to B. We use di notation to indicate the average one-waydelay of path i (i.e., propagation plus queuing and trans-mission delays). Without loss of generality, suppose thatall paths are sorted with respect to their delays, i.e., di < dj

("i, j, i < j).In our model, we separate the notions of interface band-

width and effective bandwidth. The interface bandwidth ofpath i (represented by Bint

i ) is accounted as the bandwidthof the interface connected to the sender on path i. On theother hand, the effective bandwidth (referred to by Bi) rep-resents the bandwidth that could be achieved through pathi considering the limitation of the intermediary networkbetween A and B. This implies that Bint

i P Bi. For simplic-ity, hereafter we use bandwidth term to refer to the effectivebandwidth.

We also assume that the approximate values of band-width and delay parameters of paths are known for the net-work (IP) layer. This information could be acquired by asimple extension to some link-state routing protocol suchas OSPF (like the ones proposed in [17] and [18]). Anothermethod to realize this assumption is to keep track of on-going/receiving data/ACK packets at the sender to esti-mate the bandwidth and delay values of paths. Note thatin UDP based connections, the control packets of RTCPcan play the role of ACK packets. However, this methodneeds some minor changes in network layer of the senderto capture the characteristics of data/ACK packets. Inboth TCP and UDP methods, the IP layer of the senderis responsible for splitting traffic between the availablepaths. This approach is of interests, because it omits theneed for changing transport layer protocols like TCP andUDP.

Let C be the total size of transmitted data (in bytes)between A and B, and Ci be the size of traffic which willbe transmitted on path i. In our model, we suppose that

the ratio of data sent through each path i (i.e., fi) is propor-tional to the effective bandwidth of that path, namely:

fi ¼Ci

C¼ Bi

B; ð1Þ

in which B is total effective bandwidth over all paths. Here,we bring some explanation to justify this assumption.Obviously, it would be wisely to fully utilize the effectivebandwidth of all paths during the total transmission timet. Because there is no retransmission in UDP connections,we have C ¼

P16i6nCi. Then, it is clear that:

Bi

B¼

CitPjCj

t

¼ CiPjCj¼ Ci

C: ð2Þ

Moreover, [9] provides this assumption with proof of opti-mality for large traffic size C in TCP connections.

As Fig. 1 illustrates, the transmission process of a con-nection in our model is divided into some fixed-size steps.Formally, we define a transmission step as a part of trans-mission process in which all of the paths are participatedexactly once (namely, every path carries a continuous setof data in each step). We use bulk j to indicate the amountof data which is transmitted over path j in each step. Gen-erally, in each step we stream data bulks of slower paths(i.e. the paths with larger delays) sooner than those of fas-ter ones. As we will show in the next section, operating inthis manner allows us to schedule packets among multiplepaths in a way that they arrive at the destination in-order.Consider Fig. 1 again. Suppose that there exist only twoavailable paths 1 and 2 while path 2 is the slower one. Stepi starts with transmitting through path 2. This transmissionlasts t2 seconds. After that, the packets are carried overpath 1 for t1 seconds. With completion of transmissionover path 1, the i + 1th step starts with transmissionthrough path 2 again. It is worth noting that in this modelall sending (receiving) times are measured based on the exit(entrance) of packets from (to) IP layer. This justifies theserial transmission scheme which is depicted in Fig. 1,despite the existence of multiple network interfaces at theend hosts.


Logically, the time needed to receive a packet from anarbitrary path is a function of packet size, interface band-width, and latency of that path. More specifically, the one-way delay of the ith packet with size pi sent from A to B

along path k at time t is:

dABk ðpi; tÞ ¼

pi

Bintk

þ lABk ðpi; tÞ; ð3Þ

in which lABk is the latency of kth path between A and B

(i.e., propagation plus queuing delays). Specifically, in thecase of TCP connections, the RTT of the ith packet canbe computed by:

rðpi; qi; tÞ ¼ dABk ðpi; tÞ þ di þ dBA

k0 qi; tþ dABk ðpi; tÞ þ di

� �; ð4Þ

where di is the delay between when ith packet was receivedand when the corresponding ACK was sent, and qi is thesize of the ACK packet.

For the sake of simplifying our analysis, the followingassumptions are made:

• All data packets have the same size p. This allows thepacket dependency of delay to be removed from (3)and (4).

• We consider average latency of each path (instead of itsinstantaneous latency) to compute delay of packets sentover it. This allows the time dependency of delay to beremoved from (3) and (4). Based on these two assump-tions, we use dk to denote the one-way delay of a packet(with size p) over path k during a given step. Neverthe-less, we provide an analytical probabilistic model in Sec-tion 6 to analyze our TCP-based method in networkswith dynamic delay conditions.

• There is no delay between the time that a data packet isreceived and the time that its corresponding ACK issent, namely di = 0 in (4).

• All ACKs are sent back over the same and single path tothe source (e.g., the path on which the TCP SYN packethas been initially sent). We also assume that ACK pack-ets are much smaller than data packets. Hence, thelatency of transmission for all ACKs can be assumedto be the same value Dack. Note that Dack = 0 for UDPconnections.

Based on these assumptions, for the purpose of comput-ing the RTT of a packet sent along path k, we can rewrite(4) as:

rk ¼ dk þ Dack: ð5Þ

Fig. 2. Schematic view of the proposed idea for UDP connections. Themethod schedules packets at the source to arrive in-order at thedestination.

5. The proposed method for UDP connections

Consider a scenario in which we are going to split pack-ets belonging to a single UDP connection between multiplepaths. Clearly, packets which are moved over slower pathswill receive later at the destination in comparison withthose carried over faster paths. In general, our idea is toschedule the packets among these paths such that the des-

tination receives them in-order. The idea is schematicallydepicted in Fig. 2. The first row in the figure is sequentialraw data which is ready for transmission in step i. The sec-ond row shows the reordered data which is actually trans-mitted. Finally, the last row depicts the received data at thedestination. As we described before, in each step the sendertransmits a continuous bulk of data over every path. In thefigure, each data bulk is tagged by the path number whichis going to convey it. The intuition behind our idea is thatin each step the data bulks of the slower paths are streamedsooner than those of the faster ones. Because of moredelays of path j in comparison with path i ("i, 0 < i < j),the traffic could be properly scheduled so that the data bulkcarried over path j is delivered at the destination after thedata bulk of path i. In fact, the main issue we are goingto address in this paper is ‘‘how to schedule the traffic overpath 1 to n in a way that the packets sent on path i ("i,i > 1) are received exactly after those carried over pathi � 1’’. Since the sender breaks the original order of data,the data should be completely available before the startof transmission. Consequently, this approach is not suit-able for applications like VoIP [19] and video conferencing[20] which consume data directly. However, it is useful forsome other applications like video/voice on-demand appli-cations which usually have strong senders (i.e., powerfulservers), while their receivers are from a broader range, likePDA and PC. Since the limited memory space at the hand-held devices is one of the major factors influencing designoptions, reducing buffer space at the receiver is much ofinterest.

According to (1), we have

Ci

Cj¼ Bi

Bj; 8i; j: ð6Þ

We assume that the data bulk which is transmitted in a gi-ven step through path i has the average size ci. The time re-quired to accomplish this transmission is denoted by ti.Intuitively, the time required to transmit a data bulk overa given path would be related to the size of that bulk.Hence, we can draw the following equation:

ci

cj¼ ti

tj: ð7Þ


To measure the performance of our method, we now definea practical performance metric.

Definition 1. Beginning pause time (BPT): the amount oftime from the start of transmission (by the sender) that thereceiver should pause to get sure that delivering data to theupper layer will not be interrupted.

Fig. 3. Axis of receiving times from paths 1 and 2. The zero point of theaxis represents the start of transmission by the sender.

In other word, BPT is the sum of buffer underrun durationsfor the case that the receiver starts its playing just as thesender begins its transmission. As it can be seen fromFig. 2, the BPT of our solution is calculated by:

BPT ¼ d1 þXn

i¼2

ti: ð8Þ

Just for simplicity, suppose that there are totally m com-plete steps throughout the connection. Consequently, thetotal bytes which will be transmitted over path i is equal to:

Ci ¼ m � ci ð9ÞAccording to (7) and (9), we have the following n � 1 inde-pendent equations:

Ci

Cj¼ ti

tj: ð10Þ

To establish the scheduling illustrated in Fig. 2, the delay ofpath i should equal the sum of the following parameters: (i)the delay of path 1; (ii) the elapsed time for sending trafficthrough path j for all j, j < i (i.e. after transmitting overpath i); and (iii) the time required to get sure that the datawhich will be sent after path i in the same step will arrive.These issues are summarized in the following equation:

di ¼Xi�1

j¼1

tj þ d1 þXi

j¼2

tj ¼ d1 þ 2 �Xi

j¼1

tj � ti � t1: ð11Þ

This gives us another set of n � 1 independent equations.Based on (10) and (11), finally we will have 2n � 2 equa-tions and n unknown variables. Clearly, to deterministi-cally solve an equation system with n variables, thereshould be exactly n independent equations. Hence, we have2n � 2 = n which leads us to the value of 2 for n. In otherwords, our scheme can be applied in situations where onlytwo paths are available. Therefore, we assume n = 2 for therest of this section. As a consequence, (10) and (11) arerewritten as

C1

C2

¼ t1

t2

; ð12Þ

d2 ¼ d1 þ t1 þ t2: ð13Þ

Suppose that Dd = d2 � d1. Based on above equations, theperfect values of t1 and t2 are straightforwardly computedby

t2 ¼C2

C� Dd; ð14Þ

t1 ¼C1

C� Dd; ð15Þ

Together with (8), (14) and (15), we can conclude the fol-lowing BPT for our solution

BPT ¼ d1 �C1

Cþ d2 �

C2

C: ð16Þ

From (16), it is evident that by decreasing the amount ofdata which will be conveyed through the slower path, theBPT value will be reduced accordingly. However, weshould be careful not to decrease C2 so much that the over-head of packet header becomes intolerable.

Below, we prove that our solution delivers an optimalvalue for BPT. Assume a general streaming algorithmwhere the packets are received arbitrarily (i.e. in-order orout-of-order). We consider the best case in which all thereceived packets are in-order. Suppose an arbitrary pointin the axis of time at the receiver. The zero point of the axisrepresents the start of transmission by the sender. Fig. 3illustrates this:

According to (1), the expected sending rate over path i

would be proportional to Ci/C. Let t be a given time wheret P di. Thus, during [di,t], the receiver is supplied throughpath i with ui units of time for playing, where ui is calcu-lated by:

ui ¼Z t

di

Ci

C� dt ¼ Ci

C� ðt� diÞ: ð17Þ

From definition 1, the BPT is equal to the sum of the idletimes at the receiver in which it has no data for deliveringto the upper layer, namely:

BPTnopt ¼ t�

Xn

i¼1

Ci

C� ðt� diÞ

" #: ð18Þ

When n = 2 (i.e. we are using only two paths), the optimalvalue of BPT would be

BPT2opt ¼ t� C1

C� ðt� d1Þ þ

C2

C� ðt� d2Þ

� �

¼ C1

C� d1 þ

C2

C� d2: ð19Þ

This value equals the value of BPT achieved by our solu-tion, as shown in (16). Plainly, the BPT obtained by allother algorithms would be greater than or equal to this va-lue. Consequently, the BPT which the receiver experiencesby our algorithm is optimal.

Using Dd instead of d2 � d1, we can rewrite (16) as:

BPT� d1 ¼ Dd � C2

C: ð20Þ


Remind that d1 is the inevitable delay of transmission be-tween node A and B, because it is the smallest possible de-lay between two nodes. Hence, the above equation impliesthat the additional delay (i.e. BPT � d1) will rise by the in-crease in the difference between paths’ delays and also bythe raise in the fraction of data which will be conveyedacross the slower path. The analytical results of the BPTwith respect to Dd and C2/C are depicted in Fig. 4.

As the figure shows, the overall delay will increase mul-tiplicatively with the raise in the fraction of data which isconveyed through the slower path.

6. The proposed method for TCP connections

As mentioned in Introduction section, in the case ofTCP, concurrent transferring of data over multiple pathscan sometimes worsens the performance than using a singlepath [9]. This may happen because of the followingreasons:

(i) The bandwidth/delay mismatch between the selectedpaths causes that the sender-side TCP times out forpackets sent on the slower paths and so retransmitsthem. In addition, after each timeout, TCP drasticallyscales down its congestion window to zero andinvokes the slow-start mechanism, thereby under-uti-lizing the paths and further degrading the perfor-mance of multipath scheme.

(ii) Most TCP implementations include the ‘‘fast-retrans-mit/recovery’’ mechanism [21]. Even if the TCP time-out values are set high to prevent the scenariodescribed above, fast-retransmit may still incuranother serious problem. Here, we bring a simpleexample to elucidate this issue. Suppose the ratio ofthe RTTs of two employed paths is 4 and furtherassume that the first packet is transmitted on theslower path and the next four packets are transmittedover the faster path. The receipt of packets 2, 3, and 4at the destination will cause the receiver-side TCP tosignal a fast-retransmit of the first packet, therebywasting the prior transmission on the slower path.Worse yet, the recovery phase following a fast-

Fig. 4. The additional delay imposed by the proposed method on thereceiver’s application (i.e., BPT-d1) with respect to the delay difference(i.e., Dd) and bandwidth ratio of the slower path (i.e., C2/C).

retransmit scales down the congestion window tothe half of its current size, resulting in a high degra-dation in the throughput of the connections.

In subsequent sections, we first derive the general cir-cumstances that could be tolerated to prevent timeoutsduring the lifetime of a TCP connection. Then, we discussthe conditions that should be held to avoid invocation offast-retransmit mechanism.

6.1. Preventing TCP retransmission timeouts

In popular TCP Reno, the retransmission timeout(RTO) for the ith packet is set as:

RTOi ¼ Ri þ K � V i; ð21Þwhere Ri is the current smoothed estimate of RTT, Vi is thecurrent smoothed estimate of the deviation in RTT, and K

is a constant factor (typically K = 4) which adjusts themeasured retransmission timeout with respect to its vari-ance [9]. Moreover, Ri and Vi are calculated by the follow-ing recursive equations:

Ri ¼ w � Ri�1 þ ð1� wÞ � RTT i; ð22ÞV i ¼ b � V i�1 þ ð1� bÞ � jRTT i � Rij; ð23Þ

where RTTi is the sampled RTT for the ith packet. Indeed,Eqs. (22) and (23) act as a low-pass filter on the sampledRTT, smoothing out the variations [22].

According to the above discussion, the exact value ofRTO will closely depend on the sequence of RTT samples.In multipath transferring schemes, this dependency is basedon the sequence of paths which are chosen to send packetson. As an approximation, the average RTT and the aver-age deviation in RTT (both weighted according to the por-tion of traffic which is conveyed through each path) will beused to compute RTO [9]. The average RTT is hence calcu-lated by

�r ¼Xn

i¼1

ri � fi; ð24Þ

in which ri is the smoothed estimate of RTT for packetssent along path i and fi is the portion of traffic sent overpath i (i.e. fi = Ci/C). Similarly, the average deviation inRTT will be

�v ¼Xn

i¼1

fi � jri � �rj: ð25Þ

From (21), no timeout will occur if:

rj < �r þ K � �v; 8j; 1 6 j 6 n: ð26Þ

According to (5) and (24), the average RTT over n paths iscalculated by

�r ¼Xn

i¼1

fi � ðdi þ DackÞ ¼ Dack þXn

i¼1

fi � di ¼ Dack þ �d; ð27Þ

in which �d is the weighted average of one-way delays, i.e.:


�d ¼Xn

i¼1

fi � di: ð28Þ

Also, based on (5), (25) and (27), the average deviation inRTT for n paths will be:

�v ¼Xn

i¼1

fi � jdi þ Dack � Dack � �dj ¼Xn

i¼1

fi � jdi � �dj: ð29Þ

Considering (27) and (29), the inequality of (26) can be re-stated as:

djþDack < �dþDackþK �Xn

i¼1

fi � jdi� �dj 8j;16 j6 n: ð30Þ

After simplification, (30) turns into the following relation:

dj � �dPni¼1fi � jdi � �dj

< K 8j; 1 6 j 6 n: ð31Þ

Below, we try to rewrite (31) with respect to the differencebetween delays of paths. Since

Pni¼1fi ¼ 1, we clearly have:

dj � �d ¼Xn

i¼1

fi

!� dj �

Xn

i¼1

fi � di ¼Xn

i¼1

fi � ðdj � diÞ

¼Xn

i¼1

fi � Ddj;i 8j; 1 6 j 6 n; ð32Þ

where Ddj,i stands for dj � di. Using (32), we can rephrase(31) as:Pn

i¼1fi � Ddj;iPni¼1fi �

Pny¼1fy � Ddi;y

�� < K 8j; 1 6 j 6 n: ð33Þ

Inducing from the above equation, the condition for pre-venting timeout depends solely on the difference of paths’delays, i.e. Ddj,i, and not on their absolute values.

00.10.20.30.40.50.60.70.80.9

1

1 2 3 4 5 6 7 8 9K

Max

f1

Fig. 5. Maximum tolerable value of f1 (C1/C) under different values of K

(the constant factor in TCP’s timeout formula).

6.1.1. Two path case

Let’s consider more simple and also common case ofmultipath scheme in which merely two paths are exploitedto carry the traffic between A and B, meaning that n = 2.Remind that according to our model, we have assumedr1 6 r2. Then, by setting j = 1 in (33), we will have

f2 � Dd1;2

f1 � f2 � Dd1;2j j þ f2 � f1 � Dd2;1j j < K: ð34Þ

By taking into account Dd1,2 = �Dd2,1 and Dd1,2 < 0, (34)will be shortened to:

� 1

2 � f1

< K; ð35Þ

which is always satisfied, because both K and f1 are positivenumbers.

For j = 2, (33) will be equivalent to

f1 � Dd2;1

f1 � f2 � Dd1;2j j þ f2 � f1 � Dd2;1j j < K: ð36Þ

Similar to the simplification we made in (34), we can reduce(36) to:

f1 6 1� 1

2 � K : ð37Þ

Hence, if the employed TCP implementation is configura-ble, we could appropriately set the value of K to satisfythe above inequality. Otherwise, the sender should care-fully adapt the value of f1 (i.e. the ratio of data sent overpath 1). As Eqs. (35) and (37) show, the minimum valueof K in two-path case (unlike the multipath case) is interest-ingly independent of the delay difference of paths and re-lates only to the bandwidth ratio of paths. Based on thiscondition, Fig. 5 depicts the maximum value of f1 thatcan be chosen under different values of K to incur no time-out. In common implementation of TCP Reno, the value ofK is typically set to 4 [9]. It forces us to set the value of f1 nogreater than 0.875. In other words, the sending rate overthe faster path must be less than seven times of the slowerpath. Although this result is the same as the one achievedby [9], but our model is not confined by the assumptionof bandwidth domination for RTT.

6.2. Conditions for preventing TCP fast-retransmit

In most implementations of TCP, the fast-retransmit/recovery mechanism will be triggered if the sender receivesthree consecutive duplicate acknowledgements (ACKs).These duplicate ACKs have a very high potential to occurwhen using multiple paths with different delays. This is dueto the fact that packets moved on the faster paths willreceive sooner at the destination. Hence, they report theabsence of packets which has been sent on the slower paths.The sender wrongly infers these duplicate ACKs as packetloss and falls into the fast-retransmit/recovery phase.

Consider the situation in which two bulks of data comeconcurrently from two paths at the receiver. The basic ideais to guarantee (with a good approximation) that afterreception of one packet from the faster path, the receiverwill get at least one packet through the slower path. Thisprevents the third duplicate ACK and as a consequence,the fast-retransmit/recovery at the sender will not be trig-gered. This idea leads us to the concept of concurrent

Fig. 7. All possible overlapping cases for the reception of bulk i and i + 1.


reception through multiple paths in which the packets fromdifferent paths would be interleaved among each other. Inthe case that multiple different interfaces are attached tothe receiver, to realize the concurrent reception, we shouldslightly modify the network layer such that the packetsfrom different interfaces are fairly scheduled and deliveredto the transport layer. In other words, when multiple pack-ets from different interfaces arrive simultaneously at thereceiver, only one packet from each interface is processedat each round.

This concept is schematically illustrated in Fig. 6. Con-sider a stream of data from node A to B sent over twopaths i and i + 1 where di < di+1. As the time proceeds,packets of both paths are received at the destination inter-leavingly. The number above each packet is the sequencenumber of the packet and the number below it stands forthe ACK sequence number which is triggered by the recep-tion of that packet. For the sake of simplicity, it is assumedthat each packet contains just one byte of data. As shownin the figure, no more than two duplicate ACK is triggeredper each sequence number. Despite this improvement,when delays of paths change unexpectedly, we mayencounter a specific sequence of receptions which couldtrigger the third duplicate ACK. However, as our experi-mental results confirm, this situation happens rarely.

Considering start and end of reception of bulk i andi + 1 in an arbitrary step, four general cases can beassumed for possible interleaving. Fig. 7 depicts thesecases. As explained earlier in this section, the dangerousduplicate ACKs are triggered only by the packets of thefaster path i. Three possible situations could take placewhen a packet from bulk i is received: (i) no packet frombulk i + 1 has been received yet; (ii) packets of bulk i + 1are being received concurrently with this packet; and (iii)all packets of bulk i + 1 are received before this packet.In situation ii, packets from bulk i and i + 1 are receivedinterleavingly and remind that due to concurrent receptionat the receiver, it can hardly incur fast-retransmit. Hence,among these situations only the first one is dangerous(i.e. receiving some packets of bulk i before the receptionof bulk i + 1) which happens only in the third and forthcases of Fig. 7. To avoid the occurrences of this situation,the length of bulk i + 1 should be long enough to makesure that its start of reception will be before the start ofreception of bulk i. Operating in this manner, the thirdand forth cases in Fig. 7 will be converted into the firstand second one, respectively.

Path i Path i+1

15 16

10

17 1811 12 13 14

11 11 12 12 13 13 14

19

19

Time

Fig. 6. Interleaving reception of packets through two paths. With theassumption that each packet contains just one byte of data, the numberabove each packet represents the sequence number of that packet and theone below it stands for the last successful received sequence number.

Fig. 8. Schematic view for computing the minimum tolerable transmissiontime for bulk i + 1.

Fig. 9. Relative order between sending times of packets belonging to twoconsecutive bulk k and k + 1.


Fig. 8 helps to understand the analytical calculations foracquiring the minimum transmission time of bulk i + 1 (i.e.ti+1). As the figure shows, we always have:

diþ1 ¼ tiþ1 þ di þ x; 1 6 i < n: ð38Þ

The value of x is the duration of the aforementioned dan-gerous situation. Our destiny is that the variable x has a va-lue less than or equal to zero. This fact leads us to thefollowing important equation which determines the mini-mum value of ti+1 to avoid fast-retransmit:

diþ1 � di � tiþ1 6 0) tiþ1 P Ddiþ1;i; 1 6 i < n: ð39Þ

Since all paths are sorted with respect to their one-way de-lays, we can easily conclude that (39) has a transitive prop-erty for all i, 0 < i < n. In other words, if there is nointerleaving between the ith and i + 1th paths and also be-tween i + 1th and i + 2th paths, there will be no interleav-ing between the ith and i + 2th ones too. Therefore, if (39)is held for all i, 0 < i < n, it avoids interleaving between anypair of paths i and j ("i, j, 1 6 i, j 6 n).

Note that by holding the condition of (39), the 3rd dupli-cate ACK is avoided with a high probability. This uncer-tainty is due to the fact that packets sent along the samepath may actually experience various delays which is differ-ent from the assumed average delay of the path. Thismakes the reception behavior at the destination more com-plex and so inspires the potential of 3rd duplicate ACK inthe method. Here, we present an analytical model to obtainthis probability. Consider two consecutive bulks y andy + 1 (from path y and y + 1, respectively) in any arbitrarystep of the transmission. Since 3rd duplicate ACK happensonly when two data bulks arrives at the receiver simulta-neously, we take into consideration the durations in whichthe reception time of two bulks y and y + 1 overlaps witheach other. According to our previous discussion, to obtainthe probability of no 3rd duplicate ACK, we only need tosum the probability of all arrival scenarios in which thereis at least one packet from the slower path among everytwo successive packets of the faster path in overlappingdurations. Before that, we should explain some issuesrelated to the receiving times of packets.

Based on [23], the delay experienced by each packet overa path y is determined by two parameters: (i) the minimumconstant delay of path y (represented by Dy

min), which canbe thought as the propagation delay of that path; and (ii)the additional variable delay of path y (denoted by Dy

var),which can be seen as the queuing and processing delay ofthe intermediate routers. Inspired by [23] and [24], weassume that the additional delay of path y follows an expo-nential distribution with the average of ky. Now, let Ds bethe constant interval between transmission of two consecu-tive packets at the sender (Fig. 9). The value of Ds is deter-mined by the actual throughput of the TCP connectionwhich remains constant despite switching between paths.Also, suppose that Sy

i represents the start time of transmis-sion for the ith packet sent through path y and Ry

i denotesthe reception time of this packet at the receiver. Then,

according to above discussion, the following relations areobtained for every path y, 1 6 y 6 n:

Ryþ1i ¼ Syþ1

i þ Dyþ1min þ Dyþ1

var ; 1 6 i 6 myþ1; ð40ÞSyþ1

i ¼ ði� 1Þ � Dsþ Syþ11 ; 1 6 i 6 myþ1; ð41Þ

where my+1 denotes the number of packets in bulk y + 1. Inaddition, regarding the fact that in each step all packets ofthe faster path i are streamed exactly after those of theslower path i + 1, we can conclude that:

Syi ¼ ði� 1Þ � Dsþ myþ1 � Dsþ Syþ1

1

¼ ðiþ myþ1 � 1Þ � Dsþ Syþ11 ; 1 6 i 6 my : ð42Þ

Since all packets enter and leave the IP layer serially (notsimultaneously), then the probability of equality in recep-tion times, i.e. PðRyþ1

j ¼ Ryi Þ, becomes practically zero.

Thus, the probability that jth packet from path y + 1 is re-ceived between ith and i + 1th packets of path y with theassumption that the i + 1th packet arrives after the ithone (denoted by P y

i;j) will be

P yi;j ¼ PðRyþ1

j > Ryi Þ � P ðRyþ1

j > Ryiþ1Þ: ð43Þ

On the other hand, according to (40), we have:

PðRyþ1j > Ry

i Þ ¼ P ðSyþ1j þ Dyþ1

min þ Dyþ1var > Sy

i þ Dymin þ Dy

varÞ¼ P ðDy

var � Dyþ1var < Dyþ1

min � Dymin þ Syþ1

j � Syi Þ:ð44Þ

Together with (41), (42) and (44), the following relation isdrawn:

PðRyþ1j > Ry

i Þ¼ PðDy


min � Dymin þ ðj� i� myþ1Þ � DsÞ:

ð45Þ

In order to simplify the calculations, we define thresholdvariable a as follows:

a ¼ Dyþ1min � Dy

min þ ðj� i� myþ1Þ � Ds: ð46ÞThen, (45) converts into

PðRyþ1j > Ry

i Þ ¼ P ðDyvar � Dyþ1

var < aÞ: ð47ÞFig. 10 depicts the area of Dk

var � Dkþ1var < a in the (Dk

var,Dkþ1var )

plane. Clearly, to calculate the probability stated in (47),we must sum the probability of Dy

var � Dyþ1var for all points

in this area. As figure shows, depending on the value ofa, we may encounter two different cases:

Case A) if a > 0 then we should take into account thestriped area in Case A of Fig. 10. This leads us to the fol-lowing formula:

kDvar

1

var

+kD

α=− +1

varvar

kk DD

α>− +1

varvar

kk DD

0,α

α−,0

Case A) α > 0

kDvar

1

var

+kD

α=− +1

varvar

kk DDα>− +1

varvar

kk DD

0,α α−,0

Case B) α < 0

Fig. 10. The area in the (Dkvar,D

kþ1var ) plane that satisfies Dy

var � Dyþ1var < a.

The Case A depicts this area when a > 0 and Case B shows the mentionedarea when a < 0.


P ðDyvar � Dyþ1

var < aÞ

¼Z 1

dyþ1var ¼0

Z dyþ1var þa

dyvar¼0

Dyvar � Dyþ1

var � ddyvar � ddyþ1

var

¼Z 1

dyþ1var ¼0

Z dyþ1var þa

dyvar¼0

1

ky � e�dy

varky � 1

kyþ1� e�

dyþ1var

kyþ1 � ddyvar � ddyþ1

var :

¼ 1� ky

kyþ1 þ ky � e� a

ky ð48Þ

Case B) if a < 0 then we should consider the striped areain Case B of Fig. 10. Thus, similar to the former case, wewill have

P ðDyvar � Dyþ1

var < aÞ

¼Z 1

dyþ1var ¼�a

Z dyþ1var þa

dyvar¼0

Dyvar � Dyþ1

var � ddyvar � ddyþ1

var

¼Z 1

dyþ1var ¼�a

Z dyþ1var þa

dyvar¼0

1

ky � e�dy

varky � 1

kyþ1� e�

dyþ1var

kyþ1 � ddyvar � ddyþ1

var :

¼ kyþ1

kyþ1 þ ky � ea

kyþ1 ð49Þ

Using (48) and (49), we are now able to compute the valueof P y

i;j for every i and j. Bear in mind that to compute theprobability of having no 3rd duplicate ACK, we mustsum the probability of all arrival scenarios of two bulksin which there is at least one packet from the slower pathy + 1 between two consecutive packets of the faster pathy during the simultaneous reception of two bulks. To sim-plify our calculations, we concentrate on the most probable

one, namely an arrival scenario in which the i + 1th packetof path y + 1 arrives at the destination between the ith andi + 1th packets of path y for all 1 < i < by, in which:

by ¼ minðmy ;myþ1Þ � 1: ð50Þ

The probability of this desired scenario (represented byP ok

y ) would be equal to

P oky ¼

Yby

i¼1

P yi;iþ1: ð51Þ

According to (43) and (45), the probability of P yi;iþ1 would

be:

P yi;iþ1 ¼ P ðRyþ1

iþ1 > Ryi Þ � P ðRyþ1

iþ1 > Ryiþ1Þ

¼ P ðDyvar � Dyþ1

var < Dyþ1min � Dy

min � myþ1 � Dsþ DsÞ;� P ðDy


min � Dymin � myþ1 � DsÞ

¼ P ðDyvar � Dyþ1

var < a1Þ � P ðDyvar � Dyþ1

var < a2Þ ð52Þ

where a1 and a2 are threshold variables in two probabilitiesof (52) which can be obtained by setting j = i + 1 and j = i,respectively in (46). Since in most cases a1 is positive and a2

is negative, we can use (48) and (49) to compute the firstand second probability of (52), respectively. To get sureabout the accurate sign of a1 and a2 (i.e. a1 > 0 anda2 < 0) in all of the scenarios, the following constraintshould be taken into consideration, when choosing the va-lue of my+1

Ddminyþ1;y

Ds6 myþ1 <

Ddminyþ1;y

Dsþ 1; ð53Þ

where Ddminyþ1;y ¼ Dyþ1

min � Dymin. In this fashion, we reach to the

following value for P yi;iþ1:

P yi;iþ1 ¼ 1� ky

kyþ1 þ ky � e�

Ddminyþ1;y

�m�DsþDs

ky � kyþ1

kyþ1 þ ky

� eD

dminyþ1;y

�m�Ds

kyþ1 : ð54Þ

It can be easily understood from (54) that, P yi;iþ1 is not

dependent to the value of i. Hence, based on (51) and(54), we can compute Pok by

P oky ¼ 1� 1

kyþ1þky � ky � e�D

dminyþ1;y

�my �DsþDs

ky þkyþ1 � eD

dminyþ1;y

�myþ1 �Ds

kyþ1

!" #by

:

ð55Þ

Note that P oky provides a lower bound on the probability of

having no 3rd duplicate ACK, because we haven’t consid-ered some other desired scenarios in computing the aboveprobability. In the following, we bring an example to showhow the probability of 3rd duplicate ACK can be computedaccording to this analytical model.

Suppose the situation where n = 2, d1 = 0.05 s,d2 = 0.225 s, Dk

min ¼ 0:95 � dk, kk = 0.05 Æ dk, B1 = 128 kbps,B2 = 60 kbps. Referring to our model, we straightforwardlyfind the following values: Ddmin

2;1 ¼ 0:166, k1 = 0.0025,

0

50000

100000

150000

200000

250000

300000

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

BPT

Simple Multipath (Simulation)Enhanced Multipath (Simulation)Any Multipath (Analytical)One-path (Simulation)

Fig. 11. BPT achieved by UDP methods for transferring a file of size100 KB.


k2 = 0.01125. To obtain the value of Ds, the packet sizeshould be divided by throughput of the connection (i.e188 kbps). Hence, Ds = 0.064. To satisfy the constraint of(53), m2 should be 3 and consequently b becomes 2. Thesevalues reach us to the value of 0.83 for P ok

1 . In other words,the probability of triggering 3rd duplicate ACK in any stepfor two consecutive bulks is less than 0.17. In the next sec-tion, we compare the results of this probabilistic model withthose achieved by simulations.

7. Performance evaluation

This section first presents the simulation model used toevaluate the performance of the proposed methods andthen provides the experimental results.

7.1. Simulation model

For the purpose of simulation, we implemented the TCPReno protocol in Java. To provide UDP-based connec-tions, we also implemented the RTP/RTCP protocol. Thecomplete code of our simulator can be found in [25]. Inour simulations, two threads act as a client and a server.A file will be transferred from the server to the clientthrough a TCP (or UDP) connection. The implementedcode simulates the common characteristics and limitationsof real networks like dynamically changing latency of linksand drop behavior when the load exceeds the link capacity.The dynamic delay follows an exponential distributionwhich has been described thoroughly in Section 6.2. More-over, it enables the IP layer to split the outgoing trafficamong multiple distinct paths.

For the sake of simplicity, we just consider two availablepaths between the client and the server (i.e. path 1 and 2),while path 1 is the faster one. The interface bandwidth oftwo paths is the same (i.e. 2 Mbps) and the size of all datapackets is fixed to 1.5 KB. Also, the bandwidths of path 1and 2 are limited to B1 and B2, respectively. The configura-ble parameters of our experiments are (i) the size of thetransferred file; (ii) the relative latency of the paths; and(iii) the relative bandwidth of the paths.

Overall, three different UDP-based methods are simu-lated in our experiments: (i) the usual one-path UDP withaggregated bandwidth of B1 + B2; (ii) the simple multipathUDP which simply divides the traffic between two pathsaccording to their bandwidth ratio; and (iii) our enhancedmultipath whose transmission parameters are set based on(14) and (15). Similarly, three methods are simulated forTCP: (i) the usual one-path TCP with aggregated band-width of B1 + B2; (ii) the simple multipath TCP; and (iii)the multipath TCP enhanced by the techniques we pro-posed in this paper. Clearly, the results of the one-path sce-nario with aggregated bandwidth give the optimal valueswhich could be obtained by a perfect multipath method.In the simple multipath scenario, the outgoing packetsare divided at the IP layer between two available pathsaccording to the ratio of their bandwidths (i.e., in each

point of the time, path 1 has carried B1/(B1 + B2) portionof the outgoing data). We measure the BPT parameter tocompare performance of UDP methods. Also, the effectivethroughput, number of fast-retransmit events, number oftimeouts and number of dropped packets are our perfor-mance metrics for evaluating TCP methods.

In all of the following experiments, we assume that theaverage latency and the bandwidth of path 1 are fixed to0.05 s and 128 kbps, respectively. Also, all diagrams areplotted based on the relative latency and bandwidth ofpath 2 with respect to those of path 1.

7.2. Simulation results for UDP

Here, we compare all three UDP methods with respectto the BPT parameter. Fig. 11 shows the experimentalresults of BPT obtained by different methods for transfer-ring a file of size 100 KB. To give the reader better insight,the figure also presents the analytical result of BPT in ourapproach, calculated based on (16). As we proved in Sec-tion 5, this result shows the minimum value of BPT thatcan be obtained by any multipath method. According tothe figure, the one-path method completely outperformstwo other multipath approaches, especially when d2/d1 ishigh. However, our enhanced multipath method signifi-cantly improves the BPT, compared with simple multipathapproach. As much as the delay ratio of two paths becomeshigher, this improvement becomes more apparent. Also,the simulation results of our approach are on average20% above the analytical ones, which is mainly due tothe existence of packet delay variance and drop behaviorin our experiments.

7.3. Simulation results for TCP

This section presents the results of various simulationswe have carried out to assess our TCP-based method. Asthe first simulation result, Fig. 12 shows the results ofthroughput obtained by different methods for transferringa file of size 100 KB. According to the figure, the through-put will improve significantly by applying our method on

0

50

100

150

200

250

300

350

1.5 2 2.5 3 3.5 4 4.5 5

Thro

ughp

ut

B2/B1=0.07B2/B1=0.86B2/B1=1.64

0

50

100

150

200

250

300

350

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1d2/d1

B2/B1=0.07B2/B1=0.86B2/B1=1.64

0

50

100

150

200

250

300

350

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

B2/B1=0.07B2/B1=0.86B2/B1=1.64

a b c

Fig. 12. Throughput achieved by TCP methods for transferring a file of size 100 KB (a) simple multipath method, (b) our enhanced multipath method and(c) one-path method.


the multipath transferring scheme. In general, with theraise in the bandwidth of the auxiliary path 2, the through-put of all methods increases too. However, by the raise inlatency of path 2, the throughput considerably decreasesin the simple multipath scenario. Although this reductionexists in our approach too, but it’s slope is considerablysmoothed. As a numerical example, in the case that thebandwidth of path 1 and 2 are 128 kbps and 110 kbps(i.e. B2/B1 = 0.86) and their delays are 0.05 ms and0.25ms (i.e. d2/d1 = 5) respectively, the achieved through-put for transferring a file of size 100 KB by our methodis 197.8 kbps which is very close to the one obtained bythe optimal one-path transmission (i.e. 189.8 kbps). How-ever, this value is 128.3 kbps in simple multipath approachwhich is very fewer than what achieved by the formermethods.

Notice that at the beginning of a TCP connection, ittakes a while for TCP to learn the RTT and also thethroughput of the path. During this time, all methods arein their transient state in which their behaviors becomesomehow complicated.

Surprisingly, in some situations in Fig. 12, our methodoutperforms the one-path method in terms of throughput.To analyze this observation, we should note that the resultsdepicted in Fig. 12 are obtained by transferring a file ofsmall size (i.e. 100 KB) while the methods are often in theirtransient state. It means that the TCP parameters (e.g.timeout) have not been stabilized yet. In this situation,the timeout value in our method is more influenced bythe delay of path 2 at the beginning steps of transmission,because transmission starts with bulk 2. In consequence,

0

50

100

150

200

250

300

350

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

Thro

ughp

ut

B2/B1=0.07B2/B1=0.86B2/B1=1.64

0

50

100

150

200

250

300

350

1.5 2 2.5 3

d2/

a b

Fig. 13. Throughput achieved by TCP methods for transferring a file of size 1and (c) one-path method.

the timeout value is greater than that of the one-pathmethod in transient states. On the other hand, accordingto Fig. 14, the number of dropped packets in the one-pathmethod is considerably more than our method. It meansthat the drop probability is higher in the one-path method.Keeping these two parameters in mind, let’s look at theTCP performance formula [26]:

k̂ ¼ 1ffiffiffiffiffiffi2bpl

3

qþ T 0

R minð1; 3ffiffiffiffiffiffi3bpl

8

qÞplð1þ 32p2

l Þpackets=RTT;

ð56Þ

where k is the throughput, pl is the packet loss rate of aTCP streaming flow, R is the round-trip time, T0 is theretransmission timeout, b = 2 if delayed ACK is imple-mented at the receiver, and otherwise b = 1. Based on thisrelation, the throughput of TCP generally improves bydecreasing each of the drop probability pl and timeoutduration. However, the exact throughput of TCP in differ-ent situations depends on the compromise between thesetwo parameters. During the initial transient state, our en-hanced multipath approach has a smaller drop probabilitywhile the one-path method has a less timeout duration. Asindicated before, these complex behaviors are observedonly in transient states. When the file size is large enough,these behaviors disappear completely.

Fig. 13 depicts the resulted throughput for transferring afile of size 1000 KB by different methods. Still, theenhanced multipath approach shows very good results, ascompared with other methods. As a numerical example,in the case that the bandwidth of path 1 and 2 are 128 kbps

3.5 4 4.5 5

d1

B2/B1=0.07B2/B1=0.86B2/B1=1.64

0

50

100

150

200

250

300

350

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

B2/B1=0.07B2/B1=0.86B2/B1=1.64

c

000 KB (a) simple multipath method, (b) our enhanced multipath method

0

20

40

60

80

100

120

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

Dro

p C

ount

B2/B1=0.07B2/B1=0.86B2/B1=1.64

0

20

40

60

80

100

120

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

B2/B1=0.07B2/B1=0.86B2/B1=1.64

0

20

40

60

80

100

120

1.5 2 2.5 3 3.5 4 4.5 5

d2/d1

B2/B1=0.07B2/B1=0.86B2/B1=1.64

a b c

Fig. 14. Number of drops in TCP methods for transferring a file of size 1000 KB (a) simple multipath method, (b) our enhanced multipath method and (c)one-path method.

0

0.2

0.4

0.6

0.8

1

Prob

. of 3

rd d

up.

AC

K

0.07

1.25

54.543.532.521.5

B2/B1

d2/d1

Fig. 16. Analytical results for the probability of triggering 3rd duplicateACK computed through (55).


and 110 kbps (i.e. B2/B1 = 0.86) and their delays are0.05 ms and 0.20 ms (i.e. d2/d1 = 4), respectively, theachieved throughput for transferring a file of size1000 KB by our method is 229.8 kbps which is very closeto the one obtained by the optimal one-path transmission(i.e. 230.7 kbps). However, this value is 155.9 kbps in sim-ple multipath approach which is very fewer than whatachieved by other two methods.

The number of dropped packets is illustrated in Fig. 14.This parameter is different from fast-retransmit in a sensethat here fewer number of drops does not necessarily implya better performance. It may have another implicit mean-ing that TCP has fought less to achieve the maximumbandwidth.

Fig. 15 depicts the results of dropped throughput in ourapproach with respect to the one-path transferring method.Moreover, the analytical probability of triggering 3rd dupli-cate ACK calculated by (55) is demonstrated in Fig. 16.Comparing these two diagrams with each other, we caneasily justify the behavior of our method in terms ofdropped throughput. Ignoring the local changes, bothdropped throughput and probability of 3rd duplicateACK increase by the raise in bandwidth and delay of path2. In other words, the main reason for the increase indropped throughput of our method (when D2 and B2

increase) is the occurrence of large number of 3rd duplicateACKs in such conditions.

Table 1 compares the number of fast-retransmit eventsin all methods. The rows and columns represent the ratio

010

20

30

40

50

60

Dro

pped

Thr

ough

put

0.07

1.25

54.543.532.521.5

B2/B1

d2/d1

Fig. 15. Dropped throughput in our approach with respect to the one-path method. These results are measured by transferring a file of size1000 KB.

of bandwidth and delay of path 2 to path 1, respectively.In this table, the file size is fixed to 1000 KB. As the tableshows, the rate of fast-retransmit events decreases consid-erably in our approach and approximately reaches to theresults of the one-path method. As an example, considerthe case where the bandwidth and the delay of path 2 areequal to 110 kbps (i.e. B2/B1 = 0.86) and 0.25 s (i.e. d2/d1 = 5), respectively. By our method, the number of fast-retransmits which is 9 units by the one-path methodincreases just by 5 units. This increase in simple multipathapproach reaches 30 units which unfortunately could notbe tolerated.

Table 1Number of fast retransmission events for transferring a file of size 900 KB;The numbers in each cell belong to the number of fast retransmissionsobtained respectively by simple multipath (SM) approach, enhancedmultipath (EM) approach, and one-path (OP) method. The rows andcolumns represent the ratio of bandwidth and delay of path 2 to path 1,respectively

d2/d1 B2/B1

0.07 0.86 1.64

SM EM OP SM EM OP SM EM OP

2 23 11 5 21 14 9 14 12 72.5 26 11 6 26 18 9 17 15 63 34 11 7 30 17 9 19 15 83.5 47 8 6 36 13 10 25 14 74 61 10 7 38 13 9 30 13 84.5 62 9 6 39 15 10 27 13 85 65 10 5 39 14 9 30 15 7

0

5

10

15

20

25

1.5 2 2.5 3 3.5 4 4.5 5

D2/D1Num

ber o

f Tim

eout

s

B2/B1=0.07B2/B1=0.86B2/B1=1.64

Fig. 17. Number of timeout events in our method for transferring a file ofsize 1000 KB.


Finally, the number of timeout events in our method isschematically drawn in Fig. 17. In this experiment, the filesize is 1000 KB. As we expected beforehand from (37), thenumber of timeout events is nearly independent of thedelay ratio of two paths. Moreover, the rate of timeoutevents increases hugely when bandwidths of the paths donot satisfy the condition of (37) (i.e. when B2/B1 is less then0.14).

8. Conclusion

The main problem of multipath transferring schemes isthe difference between the delays of selected paths whichcauses reordering between packets of the same flow. Multi-path methods in both wired and wireless networks prefermore diverse paths and this unfortunately intensifies theproblem of reordering. In this paper, two novel streamingapproaches have been introduced for handling the reorder-ing problem in end-to-end multipath schemes. In the caseof UDP connections, we proposed a streaming methodfor scheduling packets at the sender among multiple pathsin a way that they arrive at the receiver in-order. It wasproven that our proposal imposes the minimum possibledelay on the receiver’s application to start after the senderbegins its transmission. In addition, our method reducesthe need for large buffer spaces at the receiver. Simulationresults also confirm the efficiency of applying the proposedmethod in multipath transmission.

Unfortunately, in the case of TCP, the reordering bringsmore serious problems, i.e. producing more timeouts andunnecessary fast-retransmit events which degrade thethroughput of TCP connections considerably. We first pre-sented the analytical conditions that should be preserved toavoid timeouts. Interestingly, we proved that these condi-tions depend on the differences of one-way delays of pathsnot on their absolute values. Then, we introduced a novelapproach to avoid unnecessary fast-retransmits/recoveryevents. Based on analytical discussions, it has been sug-gested to continue transmission over the slower path forduration longer than the delay differences of selected paths.The performance of our method has been compared with

both simple multipath approach and the usual one-pathmethod (with the aggregated bandwidth) through simula-tion. The performance has been studied in terms ofthroughput, number of fast-retransmit events, and numberof dropped packets. Simulation results show that the per-formance of our approach is comparable with the optimalone-path transmission almost in all scenarios.

For applications which deliver the prerecorded multime-dia content via TCP, minimizing the delay sensed by thereceiver’s application can be a potential subject for futureresearches. Also, the effect of the proposed probabilitymodel for 3rd duplicate ACK on the receiver buffer size isa good candidate for future work.

References

[1] R. Rom, I. Cidon, Y. Shavitt, Analysis of multi-path routing, IEEE/ACM Transactions on Networking 7 (6) (1999) 885–896.

[2] J.C. Haartsen, The Bluetooth radio system, IEEE Personal Commu-nication 7 (1) (2000) 28–36.

[3] S.J. Lee and M. Gerla, Split multipath routing with maximallydisjoint paths in ad hoc networks, in: Proceedings of IEEE Interna-tional Conference on Communications (ICC), Helsinki, Finland,2001, pp. 3201–3205.

[4] W. Wei and A. Zakhor, Robust multipath source routing protocol(RMPSR) for video communication over wireless ad hoc networks,in: Proceedings of IEEE Int’l Conference on Multimedia and Expo(ICME 2004), Taipei, 2004.

[5] M.K. Marina and S.R. Das, On-demand multipath distance vectorrouting in ad hoc networks, in: Proceedings of InternationalConference for Network Protocols (ICNP), 2001.

[6] Z. Ye, S.V. Krishnamurthy, and S.K. Tripathi, A framework forreliable routing in mobile ad hoc networks, in: Proceedings of 22ndAnnual Joint Conference of the IEEE Computer and Communica-tions Societies (INFOCOM ’03), San Francisco, CA, USA, 2003.

[7] Y. Ganjali and A. Keshavarzian, Load balancing in ad hoc networks:single-path routing vs. multi-path routing, in Proceedings of 23ndAnnual Joint Conference of the IEEE Computer and Communica-tions Societies (INFOCOM ’04), Hong Kong, vol. 2, pp. 1120–1125,2004.

[8] R. Krishnan and J.A. Silvester, Choice of allocation granularity inmultipath source routing schemes, in Proceedings of 12nd AnnualJoint Conference of the IEEE Computer and CommunicationsSocieties (INFOCOM ’93), San Francisco, CA, vol. 1, pp. 322–329,March 1993.

[9] D.S. Phatak and T. Goff, A novel mechanism for data streamingacross multiple IP links for improving throughput and reliability inmobile environments, in: Proceedings of 21nd Annual Joint Confer-ence of the IEEE Computer and Communications Societies, (INFO-COM ’02), New York, NY, vol. 2, pp. 773–781, June 2002.

[10] S. Mao, D. Bushmitch, S. Narayanan, and S.S. Panwar, MRTP: amulti-flow realtime transport protocol for ad hoc networks, in:Proceedings of IEEE Vehicular Technology Conference, October2003.

[11] E. Blanton, M. Allman, On making TCP more robust to packetreordering, ACM Computer Communication Review 32 (1) (2002)20–30.

[12] S. Bohacek, J.P. Hespanha, J. Lee, C. Lim, and K. Obraczka, TCP-PR: TCP for persistent packet reordering, in Proceedings of IEEEICDCS, Rhode Island, May 2003.

[13] M. Gerla, S.S. Lee, and G. Pau, TCP westwood simulation studies inmultiple-path cases, in Proceedings of SPECTS, California, July 2002.

[14] M. Zhang, B. Karp, S. Floyd, and L. Peterson, RR-TCP: Areordering-robust TCP with DSACK, in Proceedings of IEEE ICNP,2003.


[15] C. Ma and K.C. Leung, Improving TCP performance robustness inmultipath networks, in Proceedings of IEEE LCN, 2004.

[16] S. Norden, M.M. Buddhikot, M. Waldvogel, and S. Suri, Routingbandwidth guaranteed paths with restoration in label switchednetworks, in Proceedings of IEEE International Conference onNetwork Protocols (ICNP 2001), Riverside, CA, USA, pp. 71–79,November 2001.

[17] G. Apostolopoulos, et al., QoS routing mechanisms and OSPFextensions, IETF RFC 2676, 1999.

[18] W.H. Liao, Y.C. Tseng, S.L. Wang, and J.P. Sheu, A multi-path QoSrouting protocol in a wireless mobile ad hoc network, IEEEInternational Conference On Networking, France, part II, pp. 158–167, July 2001.

[19] J. Rosenberg and H. Schulzrinne, A framework for telephony routingover IP, IETF RFC 2871, June 2000.

[20] W. Chimiak, A comment on packet video remote conferencing andthe transport/network layers, IETF RFC 1453, April 1993.

[21] V. Jacobson, Modified TCP congestion avoidance algorithm,end2end interest group mailing list, April 1990.

[22] A. Mankin and K. Ramakrishnan, ‘‘Gateway congestion controlsurvey, IETF RFC 1254, 1991.

[23] C. Demichelis and P. Chimento, IP packet delay variation metric forIP performance metrics (IPPM), IETF RFC 3393, 2002.

[24] A. Corlett, D. Pullin, and S. Sargood, Statistics of one-way internetpacket delay, IETF Internet-Draft, draft-corlett-statistics-of-packet-delays-00.txt, Aug. 2002.

[25] http://web.ut.ac.ir/routerlab/members/yabandeh/tcp-sim.zip.[26] J. Padhye, V. Firoiu, D. Towsley, J. Kurose, Modeling TCP Reno

performance: a simple model and its empirical validation, IEEE/ACM Transaction on Networking 8 (2) (2000) 133–145.

Maysam Yabandeh received the B.S. degree inComputer Science and Engineering from Uni-versity of Tehran, Tehran, Iran, in 2005. He iscurrently perusing his M.S degree in the sameuniversity and working as a research assistant inthe Router Laboratory, University of Tehran.His main research interests include distributedcomputing, computer networks, operating sys-tems, and theoretical computer science.

Sajjad Zarifzadeh received the B.S. and M.S.degrees in Computer Science and Engineering
from University of Tehran, Tehran, Iran, in 2002and 2005, respectively. He is currently working asa research assistant in the Router Laboratory,University of Tehran. His main research interestsinclude algorithmic game theory, network algo-rithms, topology control and routing in wirelessad hoc and sensor networks.
Nasser Yazdani got his B.S. degree in Computer
Engineering from Sharif University of Technol-ogy, Tehran, Iran. He worked in Iran Telecom-munication Research Center (ITRC) as aresearcher and developer for few years. To pursuehis education, he entered to Case WesternReserve Univ., Cleveland, Ohio, USA, later andgraduated as a Ph.D. in Computer Science andEngineering. Then, he worked in different com-panies and research institutes in USA. He joinedthe ECE Dept. of Univ. of Tehran, Tehran, Iran,
as an Assistant Professor in September 2000. His research interest includesNetworking, packet switching, access methods, Operating Systems and
Database Systems.
http://web.ut.ac.ir/routerlab/members/yabandeh/tcp-sim.zip

Improving performance of transport protocols in multipath transferring schemes

Documents

Transcript of Improving performance of transport protocols in multipath transferring schemes