Stanley Francis Tara Catamount Algorithmic Trading, LLC Spring 2012

Advancements in Transmission Media in High-Frequency Trading Networks

Profit motives have driven improvements in data communications networks throughout history. In the 1920s, after Western Union ceased reporting horse-race results via their telegraph network, bettors organized1 and soon developed more efficient private communications systems. Over one-hundred years earlier, in 1815, Baron Henri James de Rothschild, a member of the Rothschild banking dynasty, used carrier pigeons to learn of Napoleon's loss at Waterloo and made a fortune shorting French bonds.2

Today, on Wall Street, efforts to develop and use faster data communications networks are more aggressive than ever before. According to data from the NYSE, high-frequency trading (HFT) - which is trading activity that occurs over high speed communications networks from servers co-located at the exchanges matching engines - accounts for 70% of all stock execution volume.3 Even more impressive, as we’ll soon see, certain terrestrial long-haul data transmissions are within a few micro-seconds of the speed of light.

A Tale of Two Cities.

This report explores recent advancements in transmission media with respect to HFT trading activities and provides an overview of the current alternatives. Specifically, given the limited scope of this paper, focus is placed on the New York City (NYC)4 to Chicago (CHI) corridor. Undoubtedly this cross connect, which links New York-based stock exchanges with Chicago based options & futures exchanges (and as a result is the most heavily trafficked securities trading corridor in the United States) is of utmost importance to HFT traders and their Information Technology and Network Managers.

Furthermore, since increasing speed is the primary objective with this long-haul circuit, there are only two suitable transmission mediums worth considering; optical fiber (a guided medium) and wireless (an unguided medium).

How far is Chicago from New York, anyway?

In 2009, whispers throughout the HFT technical community persisted that someone had established a significantly faster optical fiber data link (cross connect) between New York and Chicago. At that time, the speediest and most costly optical fiber circuits between the NY/CHI corridor delivered 15.8ms (milliseconds) round trip and most circuits ranged 17.5 to 23ms. Faster transmissions, via guided media, given the fixed distance between these two cities were thought unlikely.

However, by early 2011 the rumors proved true. A new private carrier, Spread Networks, announced a 13.33ms circuit between the CME’s market access center at 350 E. Cermack in Chicago to NASDAQ’s market access center in Carteret, New Jersey (NYSE’s SFTI POP at 165 Halsey Street, Newark, NJ was added shortly thereafter). Chicago is located 715 miles from New York City. However, the actual distance of guided transmission media – including optical fiber - is usually 925+ miles (1850+ miles round trip).5 This is because the media does not follow a direct path between the two cities but instead follows traditional railroad routes as Samuel Morse’s copper telegraph wire once did.6 Figure 1: illustrates Spread’s actual path between these two points.7

1 The Chicago Historical Society. Retrieved August 31, 2012. http://www.chicagohs.org/history/index.html 2 Nial Ferguson, The House of Rothschild (New York: Penguin Books, 1998) pp. 15-38 3 Aite Group. Retrieved December 21, 2011. http://www.aitegroup.com/ 4 Remark: NYC includes the market access centers that physically reside across the Hudson River in New Jersey i.e., NYSE Access Center in Weehawken & Mahwah and NASDAQ Market Access Center in Carteret, NJ. 5 Remark: Excess length – for future repairs - is also present and typically adds 3-5% to the total circuit length. 6 Frank Derfler Jr. and Les Freed, How Networks Work: (Indianapolis: Que Publishing, 2005) pp. 6-30 7 Spread Networks, LLC, Retrieved March 12, 2011. http://www.spreadnetworks.com/

Stanley Francis Tara Page 1

Figure 1: Spread Networks Map

Therefore, Spread Networks achieved the bulk of this 15.6% latency reduction through the only means they theoretically could; they shortened the length of the circuit by over a hundred miles. They achieved this by trenching a path, via a private right-of-way, through the Allegheny Mountains at a cost of $300 million dollars.8 The remaining latency improvements were primarily attributable to minimizing the sizeable hit associated with switching. By creating a dedicated route switching was basically eliminated. Another improvement was superior dispersion management. The optical transport fiber used was a nonzero dispersion fiber (instead of standard single-mode fiber) which minimized the amount of chromatic dispersion in the link. Other transmission impairments were reduced due to more tightly spaced amplification facilities which decreased attenuation. Analysis of Quoted Signal Propagation Latencies

Latency: describes propagation delay from point A to point B. In an optical fiber system, it’s approximately: a)the length of the optical fiber divided b) the speed of light in the optical fiber plus c) data processing overhead e.g, network elements such as switching, data packing, amplification, signal regeneration, excess cable installation, etc. Optical Fiber: is a thin (2 to 125 µm) thin flexible medium capable of guiding an optical ray.9 For long haul transmissions is it the best guided transmission medium due to: immense bandwidth capacity, smaller size and lighter weight, lower attenuation, electromagnetic isolation and economic feasibility. Table 1: Fiber Propagation Analysis and Table 2: Fiber Propagation Analysis Cont. calculate and analyze propagation latencies with optical fiber under three cases: 1) Speed of light in vacuum. 2) Speed of light through glass. 3) Propagation delays which includes processing overhead. The result is 13.31 ms round trip.

Table 1: Fiber Propagation Analysis

Theoretical Latency based on speed of light in a vacuum (SOL) Speed of Light ( c ) 299792 km per second

Distance (d) 1327 km (825 miles) Latency one way 4.43 ms 1000 * 1327 / 299792 Latency round trip 8.85 ms

8 Matthew Phillips, Stock Trading is about to get 5.2 MS Faster (Business Week, March 29, 2012) 9 William Stallings, Data and Computer Communications 9th Edition (New York: Prentice Hall, 2000) p.112.

Stanley Francis Tara Page 2

Table 2: Fiber Propagation Analysis Cont.

Theoretical Latency based on speed light propagates through glass Speed of Light ( c ) 299792 km per second Approx. Refractive Index of glass (n) 1.45

propagation speed thru glass is <( c )

Speed of Light ( c ) 206753 km per second 299792/1.45 Naïve Distance (d) 1327 km (825 miles) Latency one way 6.42 ms 1000 * 1327 / 206753 Latency round trip 12.84 ms

Inherent media / impairment loss ~31 % 1 - (206753 / 299792) Estimated Latency of Spread's circuit including network overhead & excess Cable Excess Length 46.45 km ~ 3.5 percent of excess cable Excess Length latency 0.22 ms length used for future repairs Transmission Impairments 0.25 ms

Realized Latency round trip 13.31 ms 12.84+.22+.25

Calculation Notes: Incremental Distance per unit of Latency

Distance 102.19 km 63.5 miles Latency round trip 1.00 ms 1000 * 102.2 / 206753 * 2

Spread Networks Distance Advantage Distance 161 km 100 miles

Latency improvement round trip 1.56 ms 1000 * 161 / 206753 *2

Beam me up, Scotty.

In 1943 Western Union pioneered the first commercial microwave communications system.10 Shortly thereafter, in 1945, the system beamed telegrams between rely stations from New York to Philadelphia and used radio frequencies that had previously only been used by military radar systems.11 Figure 2: Western Union's Microwave Telegraphy illustrates their telegraphy history during the first half of the 20th century. Microwave technology has also been used extensively by the radio and TV broadcast and cable television industries, as well as in other telecommunications applications, since the early 1950s. Today it is also used by backbone carriers in cellular networks.

10 Western Union. August 29, 2012. http://corporate.westernunion.com/history.html 11 Ibid.

Stanley Francis Tara Page 3

Figure 2: Western Union's Microwave Telegraphy

Microwave transmission refers to the technology of transmitting information or energy by the use of radio waves whose wavelengths are conveniently measured in small numbers of centimeter; these are called microwaves. This part of the radio spectrum ranges across frequencies of roughly 1.0 gigahertz (GHz) to 30 GHz. These correspond to wavelengths from 30 centimeters down to 1.0 cm.12

Some important characteristics of terrestrial microwave include: •Frequency is directly related to bandwidth (higher frequency = more bandwidth) •Frequency is inversely related to PTP distance (lower the frequency = longer transmission distance) •Bandwidth is inversely related to attenuation (more bandwidth = greater attenuation) •Attenuation is directly related to rainfall, fog, snow and other atmospheric conditions. • Point- to-point (PTP) communications and direct line-of-sight (LOS) antenna positioning is required • For long haul HFT applications, such as the NYC / CHI corridor, the radio usually uses a spectrum below 20GHz Therefore, wireless transmission mediums - in the form of terrestrial microwave - have been with us for some time. They’ve always been favored for right-of-way bypass and geographic avoidance. As we have seen, they are still widely used in certain industries. But, for most users, terrestrial microwave was displaced by optical fiber networks

12 Wikipedia, September 1, 2012. http://en.wikipedia.org/wiki/Microwave_transmission

Stanley Francis Tara Page 4

long ago. Therefore, what is happening with this old technology that has the HFT trading world - including Spread Networks “ultra-low latency” 13.33ms clients - scrambling to engineer long haul terrestrial microwave links?13 What is the speed limit on Wall Street? Have Trading Technologists been using the wrong metric? Everyone has been focused on finding the fastest fiber. However, it seems, speed of light propagation with - carrier class effectiveness - is suddenly the metric that counts with HFT firms. Figure 3: Speed Limit on Wall Street, illustrates a light-hearted spoof of what is becoming a reality.

Figure 3: Speed Limit on Wall Street

Earlier this year, several entrepreneurial firms including McKay Brothers, Aviat Networks and BGC Partners, made press releases essentially claiming their 6GHz wireless network between NY/CHI with a 100 Mb/s duplex circuit offers round trip latency of less than 10.0ms.14 As we’ve seen, this is ~3.3ms faster than Spread Networks line which is the fastest optical fiber cross-connect available! Even more impressive, as Table 3 illustrates, properly constructed future designs will enable wireless networks within 1.0ms of the speed of light!!

But, there are tradeoffs. Optical Fiber, as we have seen, offers immense bandwidth especially at spectrums of 1 GB and higher but microwave does not. Unlike microwave, optical fiber neither suffers from attenuation related atmospheric conditions nor frequent packet loss. However, as Table 3 illustrates, terrestrial microwave does offer much lower latency transmissions.

The competition among wireless vendors is fierce and they all seem to have proprietary designs that give them an edge. However, all the vendors are either benefiting from inherent latency advantages or are architecting systems that basically share following characteristics:

13 Melissa Harris, Some high-speed traders convinced microwave dishes serve up bigger returns (Chicago Confidential, Aug. 19, 2012) 14 BCG Partners, August 17, 2012. http://www.bgcpartners.com/news-centre/press-releases/press-releases-02/164413026.html

Stanley Francis Tara Page 5

• Straighter Paths: as we’ve seen with our optical fiber analysis, the “lowest hanging fruit” with respect to decreasing latency is creating shorter routes. The lowest latency wireless players share a common trait; they decreased the terrestrial path length from ~ 825 miles (Spread Networks) to ~ 735 – 760 miles for microwave. If a picture says a thousand words then

Figure 4: Microwave LOS Image illustrates this point effectively. Improved modem technology that allows for reliable transmissions over vast areas of water - such as Lake Michigan - helped make these low latency, long haul LOS relays possible, as illustrated in

Figure 5 : Microwave Dish beaming east over Lake Michigan towards the POPs of Wall Street.

Figure 4: Microwave LOS Image

Figure 5 : Microwave Dish beaming east over Lake Michigan towards the POPs of Wall Street

Stanley Francis Tara Page 6

• Inherent Propagation Advantages due to Physics: Table 2 and Table 3 prove two interesting facts. The first is that latency is primarily a function of the speed of light in a vacuum (SOL). Microwave signals propagate through air at approximately the same rate as the speed of light in a vacuum (see Table 3 line 14 column 2). In fact, microwave signals are only 90,000 meters per second slower. Secondly, Microwave signals propagate through air approximately 40% faster than light propagates through glass (optical fiber). The results were 7.84ms vs. 12.84ms respectively.

Table 3: Microwave Propagation Analysis

Terrestrial Microwave Propagation Analysis

Remarks Theoretical Latency based on speed of light in a vacuum

Speed of Light ( c ) 299792 km per second Distance (d) 1175 km (730 miles) Latency one way 3.92 ms 1000 * 1175 / 299792 Latency round trip 7.84 ms

Theoretical Latency based on speed light propagates through transparent materials Speed of Light ( c ) 299792 km per second Refractive Index of air (n) 1.0003

propagation speed thru air is < c

Speed of Light ( c ) 299702 km per second 299792/1.0003 Naïve Distance (d) 1175 km (730 miles) Latency one way 3.92 ms 1000 * 1175 / 299702 Latency round trip 7.84 ms

Speed of Light, air vs. ( c ) -90 km per second Remarks: The assertion that microwave signals travel through air at approximately the same speed as the speed of light in vacuum ( c ) is correct. In fact, the signals are only 90km/sec slower. Whereas optical has a media / impairment loss of ~ 31%.

•Superior Digital Signal Processing: Perhaps the most differentiating aspect of the new ultra low latency vendors who are vying for their share of the HFT business are their digital signal processing schemes and radio designs. At least one vendor, NexxCom, believes it’s gained a superior advantage through a patented OSI Layer 1 (physical layer) architecture.15 Their proprietary Fast Ethernet / GigE systems OSI Level 1 architected radio significantly improves link margin (which is a function of error coding, constellation and modulation scheme); which directly correlates to link

15 Nexxcom Wireless, LLC, July 2, 2012. http://www.nexxcomwireless.com/

Stanley Francis Tara Page 7

availability at any given time. The greater the distance, the fewer links required to complete the network which allows for a straighter LOS path and lower latency transmissions.

Another aspect to their Layer 1 architecture relates to switching. At Layer 1, they do not place any framing or routing information in the data that is received from the Fast Ethernet Switch. They believe it is deadly, from a latency perspective, in a PTP wireless HFT trading network to inject switching and routing information into the data stream.16 Therefore, the virgin bits are sent as data frames – at nearly the speed light – across the network with minimal latency injected by the radios along the path.17

Summary

Clearly, the latest long haul terrestrial wireless networks offer significantly lower latencies in comparison to the fastest of the new proprietary optical fiber networks. However, the tradeoff is vastly less bandwidth, higher error rates and probable reliability issues - including downtime - when severe atmospheric conditions persist. Certainly, microwave is not a one-size-fits-all solution. Less latency sensitive participants “mortals” have no valid reason to even consider abandoning optical fiber. However, terrestrial microwave is a compelling alternative transmission medium for a particular niche market; the HFT arbitrageurs who must trade securities in NYC against their synthetic equivalent(s) in Chicago. Without question, the same crowd who preleased Spread Networks “ultra- low latency” (13.31ms) fiber offering will embrace microwave too. They’ll have no choice since they need to stay competitive to stay alive.18 But, they’ll certainly keep an optical fiber circuit for redundancy – especially on those rainy days.

16 Ibid. 17 Ibid. 18 Remark: Although quantifying the value of a millisecond (ms) is beyond the scope of this paper it is satisfying to understand what such tiny increments of time mean to the HFT community. Time priority is the rule of trading and even microseconds (µs) count at the securities exchanges matching engines. So with arbitrage - where assets are bought and sold simultaneously for a profit - no profit will occur if their orders are not filled since their order messages arrived too late. Yes, even microseconds (µs) mean millions of dollars to a certain breed of HFT firms.

Stanley Francis Tara Page 8

Works Cited

"Al Capone." Famous Cases & Criminals. Federal Bureau of Investigation. Web. 08 Sept. 2012. <http://www.fbi.gov/>.

"BGC Partners And Thesys Technologies Form Epsilon Networks To Create The Fastest High-Speed Microwave Financial

Information Network." BGC Voice and Electronic Brokerage. BCG Partners, 07 July 2012. Web. 01 Aug. 2012.

<http://www.bgcpartners.com/news/>.

Bowley, Graham. "The New Speed Of Money." The New York Times. The New York Times, 02 Jan. 2011. Web. 05 Jan. 2011.

<http://www.nytimes.com/2011/01/02/business/02speed.html?_r=2>.

"Celoxica." Celoxica. Celoxica. Web. Aug.-Sept. 2012. <http://www.celoxica.com/>.

"Cisco Systems, Inc." Cisco Systems, Inc. Cisco Systems, Inc. Web. Aug.-Sept. 2012. <http://www.cisco.com/>.

Derfler, Frank, Jr., and Les Freed. "Part 1 - 3." How Networks Work. 7th ed. Indianapolis: Que, 2005. 1+. Print.

Fitchard, Kevin. "Wall Street Gains an Edge by Trading over Microwaves and Broadband News and Analysis." GigaOM. Gigaom,

10 Feb. 2012. Web. 15 Feb. 2012. <http://gigaom.com/broadband/>.

"Futures & Options Trading." CME Group. CME Group. Web. Aug.-Sept. 2012. <http://www.cmegroup.com/>.

"Golden Networking, the Premier Networking Community for Business and Financial Services Executives." Golden Networking.

Golden Networking, LLC. Web. Aug.-Sept. 2012. <http://goldennetworking.net/>.

"High Frequency Traders." Traders Look to Wireless for Lowest Latency. Http://highfrequencytraders.com, 12 Oct. 2011. Web. 12

Oct. 2011. <http://highfrequencytraders.com/news/>.

"Home | NeXXCom Wireless." Home | NeXXCom Wireless. NeXXCom Wireless. Web. 04 Sept. 2012.

<http://www.nexxcomwireless.com/>.

"InformationWeek: The Business Value of Technology." Informationweek. Information Week. Web. 30 Aug. 2012.

<http://www.informationweek.com/>.

Stanley Francis Tara Page 9

Lee, Ching-Cheng. "X433 Fundamentals of Data Communications and Networking." Berkeley.edu. UC Berkeley Extension. Web. 12

Aug. 2012. <https://elearning.berkeley.edu/>.

Martin, Michael J. "Network Switches." Understanding the Network: A Practical Guide to Internetworking. Indianapolis, IN: New

Riders, 2000. 253+. Print.

McCarty, Laton. "Wireless Startup Says It Will Beat Spread�s Chicago-to-New York Speed." Wireless Startup Says It Will Beat

Spread S Chicago-to-New York Speed. Securities Technology Monitor, 6 Jan. 2012. Web. 08 Jan. 2012.

<http://www.securitiestechnologymonitor.com/>.

"McKay Brothers - Low Latency Microwave Communications." McKay Brothers - Low Latency Microwave Communications. McKay

Brothers, LLC. Web. 29 Aug. 2012. <http://www.mckay-brothers.com/>.

"Most Rated Stocks of the Last Week." NASDAQ.com. Nasdaq OMX Group. Web. Aug.-Sept. 2012. <http://www.nasdaq.com/>.

NYSE Technologies. NYSE Technologies, Inc. Web. Aug.-Sept. 2012. <https://nysetechnologies.nyx.com/>.

Quarrell, David. "FPGA, The Next Wave of HFT Technology? - HFT Review's Blog - HFT Review." FPGA, The Next Wave of HFT

Technology? - HFT Review's Blog - HFT Review. HFT Review, 30 June 2011. Web. June-July 2012.

<http://www.hftreview.com/>.

"Raging Bulls: How Wall Street Got Addicted to Light-Speed Trading." Wired.com. Conde Nast Digital, 01 Aug. 2012. Web. 02

Sept. 2012. <http://www.wired.com/>.

Rothsein, David. "Western Union'��s Microwave Telegraphy Network." Adafruit Industries Blog. Adafruit Industries. Web. 02

Sept. 2012. <http://www.adafruit.com/>.

Schneider, David. "The Microsecond Market." Spectrum.IEEE,org. IEEE Spectrum, 1 June 2012. Web. 07 July 2012.

<http://spectrum.ieee.org/>.

Stallings, William. "Chapter 4 Transmission Media." Data and Computer Communications. 9th ed. Upper Saddle River, NJ:

Prentice Hall, 2011. Print.

Stanley Francis Tara Page 10

"Stanford Computer Science." Stanford Computer Science. Stanford Computer Science. Web. Aug.-Sept. 2012. <http://www-

cs.stanford.edu/>.

Stasburg, Jenny. "On Your Mark, Get Set…." On Your Mark, Get Set…. The Wall Street Journal, 10 Oct. 2011. Web. 12 Oct. 2011.

<http://online.wsj.com>.

TABB Group. "TABB Says Long-Distance, Low-Latency Networking Increasingly Critical To Trading (TABB Group)." TABB Says Long-

Distance, Low-Latency Networking Increasingly Critical To Trading (TABB Group). TABB Group, 21 June 2010. Web. 21

Dec. 2010.

"Western Union - History." Western Union - History. Western Union. Web. 02 Sept. 2012.

<http://corporate.westernunion.com/history.html>.

Williams, Christopher. "The $300m Cable That Will save Traders Milliseconds." The Telegraph. The Telegraph United Kingdom, 11

Sept. 2011. Web. 07 Sept. 2012. <http://www.telegraph.co.uk>.

"Wireless Transmission | Aviat Networks." Wireless Transmission | Aviat Networks. Aviant Networks, LLC. Web. 21 July 2012.

<http://www.aviatnetworks.com/>.

"Www.hfttechnologies.com." Www.hfttechnologies.com. HFT Technologies, LLC. Web. 07 Sept. 2012.

<http://www.hfttechnologies.com/>.

Stanley Francis Tara Page 11