1

THESIS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

High Performance Fiber-Optic Interconnection Networksfor Real-Time Computing Systems

%\

0DJQXV�-RQVVRQ

Department ofComputer Engineering

Chalmers University of TechnologySE-412 96 Göteborg

Sweden

Centre forComputer Systems Architecture

Halmstad UniversityBox 823

SE-301 18 HalmstadSweden

Göteborg, November 1999

2

High Performance Fiber-Optic Interconnection Networksfor Real-Time Computing SystemsMagnus JonssonGöteborg 1999ISBN 91-7197-852-6

Doktorsavhandlingar vidChalmers Tekniska HögskolaNy serie 1548ISSN 0346-718X

School of Electrical and Computer EngineeringTechnical Report No. 379

Copyright © Magnus Jonsson, 1999.All rights reserved.

Department of Computer EngineeringSchool of Electrical and Computer EngineeringChalmers University of TechnologySE-412 96 GöteborgSweden

Printed by Chalmers Reproservice.Göteborg, Sweden, 1999.

3

3UHIDFH�DQG�$FNQRZOHGJHPHQW

My work concerning fiber-optic network architectures and protocols for suchnetworks was initiated already in my master thesis work at the Centre forComputer Systems Architecture (CCA), Halmstad University, finished inSeptember 1994 and supervised by Kenneth Nilsson. Recently after that, Iwas registered as a doctoral candidate at the Department of ComputerEngineering (CE), Chalmers University of Technology, and the worktowards a Ph.D. thesis could start. The whole work was performed atHalmstad University but with support in various ways from Chalmers.

The work reported in this thesis has been part of two projects: (i) theREMAP project, financed by NUTEK, the Swedish National Board forIndustrial and Technical Development, and (ii) the PARAD project, firstfinanced by the Swedish Ministry of Education in c-ooperation with EricssonMicrowave Systems AB (EMW), later by the KK Foundation in co-operationwith EMW. Both CCA and CE have taken part in the two projects.

First, I want to express my gratitude to my wonderful wife Eva for endlesspatience and support. Of course, I also want to mention my two children,Marcus and Erica, who were born during this Ph.D. work. Many thanks goto my supervisor, Professor Bertil Svensson, especially for encouraging mywork. A special thank you goes to my co-authors: Carl Bergenhem, KlasBörjesson, Magnus Legardt, Kenneth Nilsson, Jörgen Olsson, BertilSvensson, Mikael Taveniku, and Anders Åhlander. I also want to thank allother people at CCA (Lars Bengtsson, Stefan Lund, Per-Arne Wiberg, andmany more), CE (Håkan Forsberg, Jan Jonsson, Henrik Lönn, and others),and EMW (Richard Larsson, Joachim Strömbergson, Per Söderstam, andothers) that have fed this work with valuable support, comments, anddiscussions.

I also express my gratitude to those (not mentioned above) for whom I havesupervised master or bachelor thesis work in the scope of the projects:Henrik Arleving, Ali Jamshid Far, Bassam Mahmoud, and AndreasSundgren. OPTOBUS and ECL components for the bulding of a prototypehave been sponsored by Motorola and National Semiconductor, respectively.

Last, I want to encourage curious people who have in mind taking a Ph.D.It´s a lot of work but it gives experiences well worth it.

5

$EVWUDFW

Parallel and distributed computing systems become more and morepowerful and hence place increasingly higher demands on the networks thatinterconnect their processors or processing nodes. Many of the applicationsrunning on such systems, especially embedded systems applications, havereal-time requirements and, with increasing application demands, high-performance networks are the hearts of these systems. Fiber-optic networksare good candidates for use in such systems in the future.

This thesis contributes to the relatively unexplored area of fiber-opticnetworks for parallel and distributed real-time computer systems andsuggests and evaluates several fiber-optic networks and protocols. Twodifferent technologies are used in the networks, WDM (Wavelength DivisionMultiplexing) and fiber-ribbon point-to-point links. WDM offers multiplechannels, each with a capacity of several Gbit/s. A WDM star network inwhich protocols and services are efficiently integrated to support differentkinds of real-time demands, especially hard ones, has been developed. Thestar-of-stars topology can be chosen to offer better network scalability.

The WDM star architecture is attractive but its future success depends oncomponents becoming more commercially mature. Fiber-ribbon links,offering instead an aggregated bandwidth of several Gbit/s, have alreadyreached the market with a promising price/performance ratio. This hasmotivated the development and investigation of two new ring networksbased on fiber-ribbon links. The networks take advantage of spatialbandwidth reuse, which can greatly enhance performance in applicationswith a significant amount of nearest downstream neighbor communication.One of the ring networks is control channel based and not only has supportfor real-time services like the WDM star network but also low level supportfor, e.g., group communication.

The approach has been to develop network protocols with support fordynamic real-time services, out of time-deterministic static TDMA systems.The focus has been on functionality more than pure performance figures,mostly on real-time features but also on other types of functionality forparallel and distributed systems. Worst-case analyses, some simulations,and case studies are reported for the networks. The focus has been onembedded supercomputer applications, where each node itself can be aparallel computer, and it is shown that the networks are well suited for usein the radar signal processing systems studied. Other application examplesin which these kinds of networks are valuable are distributed multimediasystems, satellite imaging and other image processing applications.

7

&RQWHQWV

Preface and Acknowledgement......................................................................... 3Abstract ............................................................................................................. 5Contents............................................................................................................. 7List of Figures ................................................................................................. 11List of Tables ................................................................................................... 15

Part I: Introduction and Tutorial Surveys ..................................171. Introduction ............................................................................................ 19

1.1 Contributions.................................................................................... 191.2 Published Papers and Research Reports......................................... 211.3 Parallel Computers .......................................................................... 251.4 Real-Time Communication .............................................................. 251.5 Optical Interconnections in Parallel Computers ............................ 271.6 Disposition of the Thesis .................................................................. 29

2. High Performance Fiber-Optic Networks.............................................. 312.1 Introduction ...................................................................................... 312.2 Multiple Access Methods.................................................................. 31

2.2.1 Wavelength Division Multiple Access....................................... 312.2.2 Time Division Multiple Access .................................................. 332.2.3 Subcarrier Division Multiple Access ......................................... 332.2.4 Code Division Multiple Access................................................... 342.2.5 Discussion................................................................................... 34

2.3 Components for WDMA Networks .................................................. 352.3.1 Tunable Receivers ...................................................................... 362.3.2 Tunable Transmitters................................................................ 382.3.3 Other WDM Components........................................................... 40

2.4 Interconnection Architectures ......................................................... 412.4.1 Point-to-Point Linked Networks ............................................... 412.4.2 All-Optical Multi-Access Networks ........................................... 43

2.5 Classifications of WDMA Networks ................................................ 462.5.1 Broadcast-and-Select and Wavelength Routing ....................... 462.5.2 Single-Hop and Multihop Networks.......................................... 472.5.3 Networks with and without Control Channel .......................... 472.5.4 Receiver and Transmitter Tunability ....................................... 482.5.5 Random-Access Protocols........................................................... 492.5.6 Time-Wavelength Assignment Schemes................................... 49

2.6 Single-Hop WDMA Star Networks and Protocols........................... 502.6.1 FOX............................................................................................. 532.6.2 Lambdanet ................................................................................. 532.6.3 DT-WDMA.................................................................................. 532.6.4 A Distributed Adaptive Protocol ............................................... 54

8

2.6.5 Interleaved TDMA ..................................................................... 542.6.6 FatMAC ...................................................................................... 55

2.7 Conclusions....................................................................................... 553. Interconnections in Parallel Computers................................................ 57

3.1 Design and Performance Parameters.............................................. 573.2 Static Networks ................................................................................ 613.3 Dynamic Networks........................................................................... 633.4 Shared Medium Networks ............................................................... 653.5 Hybrid Networks .............................................................................. 663.6 Group Communication ..................................................................... 663.7 Routing.............................................................................................. 673.8 High Performance Networks for Coarse Grained Parallel

Computers......................................................................................... 68

Part II: Review of Radar Signal Processing Systems and SuitableInterconnection Networks.....................................................71

4. A Sample Radar Signal Processing System .......................................... 734.1 Computation Module Design ........................................................... 744.2 System Design Issues....................................................................... 744.3 Inter-Module Communication Demands ......................................... 75

5. Configurations and Requirements of Signal Processing Systems........ 795.1 Pure Pipeline Chain ......................................................................... 795.2 Same Program Multiple Data.......................................................... 805.3 Parallel Hardware For Multiple Modes .......................................... 815.4 Concurrent Modes on the Same Hardware ..................................... 825.5 One Data Cube per Processing Element ......................................... 835.6 Traffic Types..................................................................................... 845.7 System Sizes and Communication Distances ................................. 84

6. Evaluation of Optical Interconnection Systems.................................... 856.1 Fiber-Ribbon Ring Network............................................................. 856.2 WDM Star Network.......................................................................... 866.3 WDM Ring Network......................................................................... 896.4 Integrated Fiber and Waveguide Solutions .................................... 906.5 Optical Interconnections and Electronic Crossbars........................ 936.6 Systems with Smart Pixel Arrays ................................................... 946.7 Optical and Optoelectronic Switch-Fabrics..................................... 966.8 Planar Free Space Optics................................................................. 976.9 Free Space Optical Backplanes ....................................................... 976.10 Summary .......................................................................................... 99

Part III: Proposed Network Architectures and Protocols .........1037. FT-TR WDM Star Network.................................................................. 105

7.1 Protocol Overview and Related Networks..................................... 106

9

7.1.1 Circuit Switched Networks...................................................... 1067.1.2 Packet Switched Real-Time WDM Networks ......................... 1067.1.3 A New Protocol with Real-Time Support ................................ 1077.1.4 Protocol and Real-Time Features............................................ 109

7.2 Protocol Description ....................................................................... 1107.2.1 Transmitter and Receiver Cycles ............................................ 1127.2.2 Distributed Slot Allocation Algorithm .................................... 113

7.3 Real-Time Services......................................................................... 1167.3.1 Arriving Messages.................................................................... 1167.3.2 Control Slots............................................................................. 1177.3.3 Data Slots ................................................................................. 1187.3.4 Slot Reserving .......................................................................... 118

7.4 Implementation Aspects ................................................................ 1197.4.1 Clock Synchronization ............................................................. 1197.4.2 Clock Recovery ......................................................................... 1197.4.3 Computational Complexity...................................................... 1207.4.4 Electronic Stars........................................................................ 120

7.5 Deterministic Performance ............................................................ 1217.6 Case Study...................................................................................... 1237.7 Simulation Results ......................................................................... 1247.8 Summary ........................................................................................ 127

8. WDM Star-of-Stars Network................................................................ 1298.1 Network Architecture and Protocol ............................................... 1298.2 Deterministic Performance ............................................................ 1318.3 Clock Synchronization Aspects...................................................... 1358.4 Summary ........................................................................................ 139

9. Control Channel Based Fiber-Ribbon Pipeline Ring Network ........... 1419.1 Network Overview.......................................................................... 1429.2 Related Networks ........................................................................... 1439.3 The CC-FPR Protocol ..................................................................... 1449.4 User Services .................................................................................. 148

9.4.1 Real-Time Virtual Channels.................................................... 1499.4.2 Guarantee Seeking Messages.................................................. 1509.4.3 Best Effort Messages................................................................ 1519.4.4 Barrier Synchronization .......................................................... 1519.4.5 Global Reduction...................................................................... 1529.4.6 Low Level Support for Reliable Transmission........................ 153

9.5 Implementation Aspects ................................................................ 1539.6 Case Study...................................................................................... 1559.7 Performance Analysis .................................................................... 1579.8 Summary ........................................................................................ 163

10. Packet and Circuit Switched Fiber-Ribbon Pipeline Ring Network .. 16510.1 Circuit Switched Traffic ................................................................. 16510.2 Packet Switched Traffic ................................................................. 16710.3 Circuit Establishment.................................................................... 167

10

10.4 Case Study...................................................................................... 16810.5 Mode Changes ................................................................................ 16810.6 Summary ........................................................................................ 169

11. Conclusions and Future Work.............................................................. 17111.1 Conclusions..................................................................................... 17111.2 Ongoing and Future work.............................................................. 171

References...................................................................................................... 175Abbreviations ................................................................................................ 221

11

/LVW�RI�)LJXUHV

Figure 1: Three passive optical network architectures: (a) ring, (b) (dual)bus, and (c) star........................................................................................ 44

Figure 2: Folded bus. ...................................................................................... 45

Figure 3: Some static topologies: (a) linear array, (b) ring, (c) 2-dimensionalmesh, (d) 2-dimensional torus, (e) 3-dimensional binary hypercube, and(e) 4-dimensional binary hypercube. ....................................................... 61

Figure 4: Omega network for an eight-node system. One path through thenetwork is highlighted............................................................................. 64

Figure 5: Possible states of a 2 × 2 switch. .................................................... 64

Figure 6: A rearrangeble Benes network. ...................................................... 65

Figure 7: Fat-tree of switches where nodes are leaves in the tree. .............. 65

Figure 8: Computation module with eight 8 × 8 meshes of PEs. .................. 74

Figure 9: The MIMD system enables a functional decomposition andallocation of the program to the modules. Each function or block offunctions is then mapped onto the processors in the computationmodule (CM). ............................................................................................ 75

Figure 10: Description of one mode of operation in the 64-channel groundbased radar system. ................................................................................. 76

Figure 11: Data flow between the modules in the signal processing chain..77

Figure 12: A pure pipeline chain where one or more stages in the chain aremapped on each module........................................................................... 79

Figure 13: If the SPMD model is used, all PEs work together on one stage inthe signal processing chain at a time...................................................... 80

Figure 14 : Several parallel groups of PEs, each running a pipeline chainprogram. ................................................................................................... 81

Figure 15: Several parallel groups of PEs, each running an SPMD program................................................................................................................... 82

Figure 16: Several PEs in parallel, where each data cube is processed by asingle PE................................................................................................... 83

Figure 17: Example of spatial bandwidth reuse. Node M sends to Node 1 atthe same time as Node 1 sends to Node 2 and Node 2 sends a multicastpacket to Nodes 3, 4, and 5...................................................................... 85

Figure 18: Control channel based network built up with fiber-ribbon point-to-point links. ........................................................................................... 86

12

Figure 19: WDM star network........................................................................ 87

Figure 20: WDM star multi-hop network. ..................................................... 89

Figure 21: Multi-hop topology. ....................................................................... 90

Figure 22: A foil of fibers connects four computational nodes. In addition, aclock node distributes clock signals to the computational nodes........... 91

Figure 23: Array of passive optical stars connects a number of nodes viafiber-ribbon cables.................................................................................... 92

Figure 24: Fully connected topology where each node has an array of N − 1laser diodes and an array of N − 1 photo diodes, where N is the numberof nodes. .................................................................................................... 93

Figure 25: Row of smart-pixel arrays............................................................. 95

Figure 26: Example of a planar free space system. The direction of the beamis steered by the optical element on the way between two chips........... 97

Figure 27: Optical backplane configurations: (a) with planar free spaceoptics, (b) with smart pixel arrays, and (c) with a mirror. ..................... 98

Figure 28: Multiple passive optical stars topology. ..................................... 108

Figure 29: Passive optical star network where the nodes are equipped withfixed transmitters and tunable receivers.............................................. 109

Figure 30: The transmitter cycle, lower table, is filled by taking all ownedslots in the receiver cycles, upper table, in all other nodes. Note that amulticast is possible in Slots 1, 9, and 10. ............................................ 112

Figure 31: A receiver cycle is partitioned into data slots and control slots................................................................................................................. 113

Figure 32: Allocation scheme for the receiver cycles in a four-node system................................................................................................................. 114

Figure 33: Electronic star. ............................................................................ 121

Figure 34: Latency calculation. .................................................................... 122

Figure 35: Real-time performance plotted as fraction of messages that misstheir deadlines versus traffic intensity. ................................................ 125

Figure 36: Latency for guarantee seeking messages plotted versus trafficintensity.................................................................................................. 126

Figure 37: Mean latency for best-effort messages in a network with 10 %guarantee seeking and 90 % best-effort messages. .............................. 126

Figure 38: Mean latency for best-effort messages plotted versus bandwidthutilization. The traffic consisted of 10 % guarantee seeking and 90 %best-effort messages............................................................................... 127

13

Figure 39: Gateway node with higher channel bandwidth on the backboneside (right side)....................................................................................... 130

Figure 40: Each backbone slot is divided into 5 sub-slots with the same pairof gateway nodes as source and destination. ........................................ 131

Figure 41: Worst-case latency when full slot reservation by other nodes isassumed. The x-axis represents the number of ordinary nodes and theslot length is assumed to be γ = 1.0 µs. ................................................. 133

Figure 42: Worst-case latency when no slot reservation is assumed. The x-axis represents the number of ordinary nodes and the slot length isassumed to be γ = 1.0 µs......................................................................... 134

Figure 43: Node bandwidth, when full slot reservation by other nodes isassumed, in number of high priority slots per total number of slots in acycle. The x-axis represents the number of ordinary nodes. ................ 136

Figure 44: Node bandwidth, when no slot reservation is assumed, in numberof high priority slots per total number of slots in a cycle. The x-axisrepresents the number of ordinary nodes.. ........................................... 137

Figure 45: With the synchronization scheme used, incoming traffic to agateway node can be forwarded immediately except for the case ofinternal delay in the gateway node....................................................... 138

Figure 46: Worst-case latency comparison between usage and no usage ofthe proposed synchronization scheme. No slot reservation is assumed.The x-axis represents the number of ordinary nodes and the slot lengthis assumed to be γ = 1.0 µs..................................................................... 139

Figure 47: (a) Bi-directional fiber-ribbon link. (b) Unidirectional ringnetwork built up of M/2 bi-directional links. ........................................ 141

Figure 48: The role of being slot initiator is cyclically repeated. Each of theM nodes is the slot initiator in one slot per cycle. ................................ 144

Figure 49: The node succeeding the slot initiator initiates the control packettransmission........................................................................................... 145

Figure 50: In each slot, a node passes/transmits one control packet and onedata packet, where the control packet is used for the arbitration of thenext slot. ................................................................................................. 146

Figure 51: A control packet contains a start bit, a link reservation field, anda destination field. ................................................................................. 146

Figure 52: Conceptual view of the control channel part of the transceiver................................................................................................................. 147

Figure 53: Conceptual view of the data channel part of the transceiver.... 147

Figure 54: A control packet travels around a network with five nodes. Node1 is the slot initiator. ............................................................................. 148

14

Figure 55: An example: Node 1 sends a single destination packet to Node 3,while Node 4 sends a multicast packet to Node 5 and Node 1. ........... 149

Figure 56: The bandwidth utilization depends on the ratio of the totalpropagation delay around the ring to the cycle length. The boxes withbold text show the link through which each slot first propagates....... 150

Figure 57: Detailed description of the control packet contents................... 152

Figure 58: Data flow between the modules in the radar signal processingchain. ...................................................................................................... 156

Figure 59: One cycle of 45 slots where Slots 16 through 45 are reserved.Each number in the table indicates, for Slots 16 through 45, the ownerof the corresponding link. Neighboring segments in each slot havedifferent background shading. The slot initiators are indicated in Slots1 through 15. .......................................................................................... 157

Figure 60: Slot distribution according to Cases A and B. In the example, Nr

= 2, No = 1, and M = 3. ........................................................................... 158

Figure 61: Worst-case latency when the only slots a node gets are thoseordinary slots for which it is the slot initiator. Case A is assumed..... 159

Figure 62: Worst-case latency when the only slots a node gets are thoseordinary slots for which it is the slot initiator. Case B is assumed..... 160

Figure 63: Comparison of worst-case latency for Cases A and B................ 161

Figure 64: Maximum aggregated throughput. Case A is assumed............. 162

Figure 65: Maximum aggregated throughput. Case B is assumed............. 163

Figure 66: Example of an allocation scheme for the links in a five-nodesystem. The slot initiators are in bold type, and different segments havedifferent background shading................................................................ 166

Figure 67: Network demonstrator with two nodes. The cable bundle in theupper part of the figure contains two fiber-ribbons ending in eachnode´s OPTOBUS module. .................................................................... 172

15

/LVW�RI�7DEOHV

Table 1: Experimental single-hop WDMA star networks. M is the number ofnodes. ........................................................................................................ 50

Table 2: Non-control channel based protocols for single-hop broadcast-and-select WDMA networks. M is the number of nodes, and N is a positiveinteger greater than zero. ........................................................................ 50

Table 3: Control channel based protocols for single-hop broadcast-and-selectWDMA networks. M is the number of nodes, and N is a positive integergreater than zero...................................................................................... 51

Table 4: Other proposed single-hop WDM star networks. M is the number ofnodes, and N is a positive integer greater than zero.............................. 52

Table 5 : Classification of system sizes and communication distances. ....... 84

Table 6 : A feasible allocation scheme of the wavelength channels in a WDMstar network for the sample radar system.............................................. 88

Table 7: Performance summary of some of the networks discussed withrespect to pipeline and SPMD mapping.................................................. 99

Table 8 : Performance summary of some of the networks discussed withrespect to pipeline and SPMD mapping when several concurrent modesrun in parallel, each on dedicated hardware. ....................................... 100

Table 9: Suitability in different system sizes. Empty cells in a column meanthat the technology/network is not suitable for the correspondingsystem sizes............................................................................................ 101

Table 10: Notations for descriptions of the network architecture andprotocol. .................................................................................................. 111

Table 11: Additional notation when describing the star-of-stars networkarchitecture. ........................................................................................... 129

Table 12: Node bandwidth in number of high priority slots per total numberof slots in a cycle. ................................................................................... 135

17

3DUW�,��,QWURGXFWLRQ�DQG�7XWRULDO�6XUYH\V

19

��� ,QWURGXFWLRQ

Fiber-optic communication systems with multiple channels obtainedthrough the use of WDM (Wavelength Division Multiplexing) have reachedthe commercial stage in the area of telecommunications (see Chapter 2 for asurvey of fiber-optic communication technology). In the future, the WDMtechnique is expected also to be applicable in local multiple-access networks.At the same time, bandwidth demands in parallel and distributedcomputing systems are increasing because of new, evolving applications.Fiber-optic multiple-channel communication systems can give newperspectives in these systems.

Another technology that can be very useful for the development of newparallel and distributed computing systems is the use of fiber-ribbon links.Commercially available products have appeared very quickly, and a goodprice/performance ratio is expected.

An area of high-performance fiber-optic networks that is relativelyunexplored is networks and protocols for hard real-time systems. This fieldhas been the main target in the work reported in this thesis. The focus hasalso been on case studies and special support for signal processingapplications, especially radar signal processing.

The contributions of the present work are summarized in Section 1.1, whileSection 1.2 states which papers the thesis is based on. Introductions toparallel computers, real-time communication, and optical interconnectionsin parallel computers are given in Sections 1.3, 1.4, and 1.5, respectively.The disposition of the thesis is given in Section 1.6.

���� &RQWULEXWLRQV

One of the motivating factors for this work was the small amount ofresearch done on high-performance fiber-optic networks for hard real-timecomputer systems (we do not focus on telecommunications and do not treat,e.g., FDDI, Fibre Distributed Data Interface, networks and extensions ofFDDI networks for high-performance networks in the framework of thisthesis). There is a need for this kind of network in, e.g., embeddedsupercomputers. The radar signal processing system described in Chapter 4is a typical example of an application and computer system which we placein the category of embedded supercomputing.

The approach has been to develop network protocols with support fordynamic real-time services, out of traditional time-deterministic staticTDMA systems (see, e.g., [Kopetz and Grünsteidl 1994] [Kopetz et al. 1989]

20

[Nilsson et al. 1993] [Nilsson et al. 1993B] [Nilsson 1994] [Wiberg 1993] fortime-deterministic communication systems for hard real-time systems),while considering emerging promising, fiber-optic technologies. The focushas been on functionality more than pure performance figures, mostly onreal-time features but, later in the work, on other types of functionality forparallel and distributed systems as well. The focus of the analyses made hasbeen on predictable performance, including case studies with feasibilitytests.

The WDM star network proposed in Chapter 7, and Chapter 8 for the star-of-stars variant, was developed to fill the gap in high performance real-timenetworking and supports real-time traffic for dynamic hard real-timesystems where each node can have a sustained output dataflow of severalGbit/s. This is offered in combination with other desired, or even required,properties such as broadcast capabilities, scalability in terms of number ofnodes, and scalability in terms of aggregated system bandwidth.

Although the WDM star network represented a breakthrough, there wereseveral reasons to continue the work:

1. High demands on the optoelectronic components. While componentswith low tuning-latency have been demonstrated, it might not bereasonable to expect components combining low tuning latency,large tuning range (many channels possible), and fair price toappear in the near future.

2. The appearance of fiber-ribbon links on the market, which madepossible new network architectures.

3. The pipelined dataflow that is common in many signal processingapplications may not always need the flexibility of the distributedcrossbar that the WDM star network implements.

The network architecture of a ring, built up of fiber-ribbon point-to-pointlinks, was chosen as an alternative to the WDM star network. With thepossibility of spatial reuse of the bandwidth in the ring (referred to as apipeline ring in this thesis), the ring offers good suitability for applicationswith a pipelined dataflow. At the same, it is a flexible network if a goodmedium access method is offered. Two fiber-ribbon pipeline ring networkshave been developed in the scope of this thesis. In the first network, the ringis divided into two sub-networks, one for data traffic and one for controlinformation (see Chapter 9). The sub-network for control traffic is usedprimarily for medium access information (for the arbitration of the data sub-network) but also to implement low-level support for, e.g., groupcommunication. Compared with other networks, this network offers a greatof functionality. In the second network, the ring is divided into two sub-

21

networks, one for circuit-switched traffic and one for packet-switched traffic(see Chapter 10).

It is contradictionary to have the highest possible performance andflexibility at the same time that availability and the cost of networkcomponents are considered. Different network solutions must be offered fordifferent application demands. The choice of network may depend on, e.g.,availability of components, scalability requirements, and traffic patterns. Abroad range of high-performance networks, with special consideration to theapplication of radar signal processing is briefly described in Chapter 6,including the WDM star (and star-of-stars) network and the two fiber-ribbonring networks developed in the scope of this work (described in more detailin Chapters 7 through 10).

The main contributions of this thesis work can be summarized as:

1. Development, analysis, and case study (with chosen industrialapplication) of a WDM star network where protocols and servicesare integrated in an efficient way to support different kinds of real-time demands, especially hard real-time demands, for parallel anddistributed computing systems.

2. Development of two new ring network architectures for whichfeatures of fiber-ribbon links are used advantageously.

3. Development, analysis, and case study (with chosen industrialapplication) of protocols and services for a fiber-ribbon ringnetworkbased on a control channel. Services for different kinds ofreal-time demands, especially hard real-time demands, aresupported. Mechanisms for spatial reuse of bandwidth and low-level support for, e.g., barrier synchronization and other kinds ofgroup communication are also efficiently integrated into thenetwork.

���� 3XEOLVKHG�3DSHUV�DQG�5HVHDUFK�5HSRUWV

The papers on which the thesis is based can be divided into four groups: (L)papers concerning the proposed WDM star and star-of-stars networks, (LL)papers concerning the two proposed fiber-ribbon ring networks, (LLL) surveysof optical networks (mainly fiber-optic) and network components, includingevaluations of network architectures for signal processing systems, and (LY)papers reporting research on radar signal processing systems. The papers inthe first three groups have been written mainly by me, while my workrepresents a minor part of the papers in the fourth group. The papers arelisted by group below.

22

*URXS���

[A] M. Jonsson, K. Nilsson, and B. Svensson, “A fiber-optic interconnectionconcept for scaleable massively parallel computing,” 3URF�� 0DVVLYHO\3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� SanAntonio, TX, USA, Oct. 23-24, 1995, pp. 313-320.

Initial ideas for how to achieve a WDM star network with support forboth hard real-time traffic and non real-time traffic by dividing a cycleof time slots into different parts with different roles.

[B] M. Jonsson, K. Nilsson, and B. Svensson, “Fiber-optic interconnectionsin massively parallel systems,” 3URF�� 6L[WK� 6ZHGLVK� :RUNVKRS� RQ&RPSXWHU�6\VWHP�$UFKLWHFWXUH� �'6$ª���, Stockholm, Sweden, June 1-2,1995, pp. 51-52.

Shorter version of Paper A presented at a national conference.

[C] M. Jonsson, A. Åhlander, M. Taveniku, and B. Svensson, “Time-deterministic WDM star network for massively parallel computing inradar systems,” 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���� Maui, HI, USA, Oct. 27-29, 1996, pp. 85-93.

Presentation of the TD-TWDMA protocol together with a case study ofthe suitability of the network in a radar signal processing system.

[D] M. Jonsson and B. Svensson, “On inter-cluster communication in a time-deterministic WDM star network,” 3URF�� �QG� :RUNVKRS� RQ� 2SWLFV� DQG&RPSXWHU�6FLHQFH��:2&6�� Geneva, Switzerland, Apr. 1, 1997.

Design alternatives and analysis of star-of-stars networks where eachstar cluster follows the TD-TWDMA protocol. An interpretation, madeby Borella et al., of this article is found in [Borella et al. 1998B].

[E] M. Jonsson, K. Börjesson, and M. Legardt, “Dynamic time-deterministictraffic in a fiber-optic WDM star network,” 3URF�� �WK� (XURPLFUR:RUNVKRS�RQ�5HDO�7LPH�6\VWHPV� Toledo, Spain, June 11-13, 1997.

Implementation details on real-time services for the TD-TWDMAnetwork, together with simulation results.

*URXS���

[F] M. Jonsson, B. Svensson, M. Taveniku, and A. Åhlander, “Fiber-ribbonpipeline ring network for high-performance distributed computingsystems,” 3URF�� ,QWHUQDWLRQDO� 6\PSRVLXP� RQ� 3DUDOOHO� $UFKLWHFWXUHV�

23

$OJRULWKPV� DQG� 1HWZRUNV� �,�63$1ª���� Taipei, Taiwan, Dec. 18-20,1997, pp. 138-143.

Presentation of a novel network architecture where fiber-ribbon point-to-point links form a ring network. The network is divided into two sub-networks, one for circuit-switched traffic and one for packet-switchedtraffic. A case study of the suitability of the network in a radar signalprocessing system is also provided.

[G] M. Jonsson, “Control-channel based fiber-ribbon pipeline ring network,"3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV�0332,ª���, Las Vegas, NV, USA, June 15-17, 1998, pp. 158-165.

Presentation of a new network architecture similar to that described inPaper F. The network is called CC-FPR (Control Channel based Fiber-ribbon Pipeline Ring) and is divided into one data channel and onecontrol channel. The CC-FPR protocol uses the control channel to co-ordinate the arbitration of the data channel. A case study of thesuitability of the network in a radar signal processing system is alsoprovided.

[H] M. Jonsson, “Two fiber-ribbon ring networks for parallel and distributedcomputing systems,” 2SWLFDO�(QJLQHHULQJ, vol. 37, no. 12, pp. 3196-3204,Dec. 1998.

Journal version of Papers F and G.

[I] M. Jonsson, C. Bergenhem, and J. Olsson, “Fiber-ribbon ring networkwith services for parallel processing and distributed real-time systems,”3URF�� ,6&$� ��WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 3DUDOOHO� DQG� 'LVWULEXWHG&RPSXWLQJ�6\VWHPV��3'&6����, Fort Lauderdale, FL, USA, Aug. 18-20,1999, pp. 94-101.

Services in the CC-FPR network for, e.g., real-time traffic and groupcommunication are briefly described (work to be continued beyond thescope of this thesis). A worst-case analysis is also provided for thenetwork.

[J] M. Jonsson, C. Bergenhem, and J. Olsson, “Fiber-ribbon ring networkwith services for parallel processing and distributed real-time systems,”3URF�� 61$57��� 5HDO�7LPH� 6\VWHPV� &RQIHUHQFH, Linköping, Sweden,Aug. 24-25, 1999, pp. 46-54.

Slightly longer version of Paper I presented at a national conference.

24

*URXS���

[K] M. Jonsson, “High-performance fiber-optic communication networks fordistributed computing systems,” 5HVHDUFK�5HSRUW�&&$�−�������&HQWUH�IRU&RPSXWHU� 6\VWHPV� $UFKLWHFWXUH� �&&$��� +DOPVWDG� 8QLYHUVLW\�� 6ZHGHQ,Apr. 1997.

Survey of fiber-optic networks and components for such networks.

[L] M. Jonsson, “Optical interconnections in parallel radar signal processingsystems,” 5HVHDUFK� 5HSRUW� &&$� −� ������ &HQWUH� IRU� &RPSXWHU� 6\VWHPV$UFKLWHFWXUH��&&$���+DOPVWDG�8QLYHUVLW\��6ZHGHQ, Apr. 1999.

Survey of fiber-optic and free-space optical network architectures. Thenetworks are briefly evaluated according to their suitability in differentconfigurations of signal processing systems. A tutorial part oninterconnection networks for parallel computers is also included.

[M] M. Jonsson, “Fiber-optic interconnection networks for signal processingapplications," 4WK� ,QWHUQDWLRQDO�:RUNVKRS� RQ� (PEHGGHG� +3&� 6\VWHPVDQG� $SSOLFDWLRQV� �(+3&���� KHOG� LQ� FRQMXQFWLRQ� ZLWK� WKH� ��WK,QWHUQDWLRQDO� 3DUDOOHO� 3URFHVVLQJ� 6\PSRVLXP� � ��WK� 6\PSRVLXP� RQ3DUDOOHO�DQG�'LVWULEXWHG�3URFHVVLQJ��,336�63'3����, San Juan, PuertoRico, Apr. 16, 1999. Published in /HFWXUH�1RWHV�LQ�&RPSXWHU�6FLHQFH. vol.1586, Springer Verlag, pp. 1374-1385, 1999, ISBN 3-540-65831-9.

Conference paper in which parts of Paper L were extracted (before thereport was finished) so that the paper emphasized on fiber-opticnetworks for signal-processing applications.

*URXS���

[N] M. Taveniku, A. Åhlander, M. Jonsson, and B. Svensson, “A multipleSIMD mesh architecture for multi-channel radar processing,” 3URF�,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 6LJQDO� 3URFHVVLQJ� $SSOLFDWLRQV� 7HFKQRORJ\� �,&63$7����, Boston, MA, USA, Oct. 7-10, 1996, pp. 1421-1427.

A MSIMD (Multiple SIMD) system for radar signal processing isproposed. The TD-TWDMA network is one part of the system proposal.

[O] M. Taveniku, A. Åhlander, M. Jonsson, and B. Svensson, "The VEGAmoderately parallel MIMD, moderately parallel SIMD, architecture forhigh performance array signal processing," 3URF�� ��WK� ,QWHUQDWLRQDO3DUDOOHO� 3URFHVVLQJ� 6\PSRVLXP� � �WK� 6\PSRVLXP� RQ� 3DUDOOHO� DQG

25

'LVWULEXWHG� 3URFHVVLQJ� �,336�63'3ª���, Orlando, FL, USA, Mar. 30 -Apr. 3, 1998, pp. 226-232.

In a continuation of the work presented in Paper N, several designalternatives for an MSIMD system for radar signal processing arepresented and discussed. The TD-TWDMA network and the fiber-ribbonring network described in Paper F are design alternatives for theinterconnection network in the system proposal.

���� 3DUDOOHO�&RPSXWHUV

Two categories, according to Flynn’s classification [Flynn 1966] [Flynn1972], into which parallel computers may be classified are MIMD (MultipleInstruction streams Multiple Data streams) [Hord 1993] and SIMD (SingleInstruction stream Multiple Data streams) [Hord 1990]. MIMD computersare typically coarser grained than SIMD computers, which instead havemore tightly coupled processing elements that are all controlled by the sameinstruction flow. MIMD computers may be further divided into distributed-memory multicomputers and shared-memory multiprocessors. Shared-memory multiprocessors of today normally employ mechanisms to retaincache coherency among the processing elements [Stenström 1990][Tomaševiü and Milutinoviü 1994] [Tomaševiü and Milutinoviü 1994B].Parallelism can exist on a lower level as well (i.e., on a more fine grainedlevel), e.g., instruction level parallelism (ILP).

Most, but not all of the interconnection networks mentioned in this thesisare more suitable for coarse-grained systems. Referring to a node in thisthesis means a computer system with one interface to the network. A nodecan actually be a parallel computer itself. For example, it can be a SIMDcomputer, i.e., the system is an MIMD computer on the high level,connecting multiple autonomous SIMD computers [Taveniku et al. 1998].This configuration is related to the MSIMD (Multiple-SIMD) architecture ofPASM [Siegel et al. 1981]. Optical interconnection technology can actuallybe employed on several levels in a system, e.g., fiber-optics connecting nodeson a coarse level, where each node contains free-space optics to connectprocessing elements internally in the node.

����5HDO�7LPH�&RPPXQLFDWLRQ

In a real-time computer system, the correct functioning of the systemdepends on the time at which a result is produced as well as the correctnessof the result [Stankovic 1988]. In many real-time systems, timing must beguaranteed in order to avoid life-threatening situations. An example iscontrol systems for nuclear power plants. Other real-time systems includeflight control systems, radar systems, robotics, and industrial control

26

systems. In distributed real-time systems, the interconnection networkplays a very important role in fulfilling the system functioningrequirements. Often, guaranteeing real-time services is much moreimportant than performance such as throughput or average latency in thesesystems. Some classifications and concepts important for the area of real-time communications are briefly described below.

Whether the traffic is SHULRGLF�or DSHULRGLF� (or mixed) plays an importantrole in real-time communication systems. An example of periodic traffic ismessages triggered by the sampling of a sensor at a regular interval.Aperiodic traffic may occur as a result of non-predictable events such asuser input or a signal indicating that a dangerous situation has beenobserved by a sensor.

There is a design choice as to whether communication traffic (or processes)should be scheduled VWDWLFDOO\, i.e., off-line, or G\QDPLFDOO\. Communicationschemes can also be setup during run-time semi-statically, e.g., having aschedule which can be exchanged with a new one when the system changesits mode of operation. The static approach allows for easier analysis butrequires more knowledge of the system behavior at the design stage. Thedynamic approach, on the other hand, allows for the interconnectionnetwork to adaptively change schedule or on the basis of the current systembehavior. Interconnection systems with truly dynamic features often allowfor treatment of real-time requirements for separate packets or messages.This thesis focuses on networks with both semi-static and dynamic features.

Real-time systems are often classified into VRIW or KDUG� real-time systemsdepending on how critical the (timing) correctness of their behavior is. Interms of deadlines for processes or messages, we can introduce a third term,ILUP [Kopetz 1997]. If, e.g., a result of a calculation or an arriving messagealso has utility after its deadline, the deadline is classified as soft. It isotherwise classified as firm. If the missing of a deadline may end in acatastrophe, the deadline is classified as hard. Services for real-time trafficwith hard and soft deadlines are often classified into guarantee-seeking andbest-effort, respectively. This work has focused on networks with support forboth guarantee-seeking and best-effort traffic.

Communication networks can be divided into PXOWLSOH�DFFHVV�QHWZRUNV� (thefocus of this thesis) and VZLWFKHG�QHWZRUNV. Multiple-access networks havesome form of shared medium. The access to this medium is regulated by amedium access method. In the case of real-time multiple-access networks[Malcolm and Zhao 1995], it is often the medium access method that isspecially designed to support real-time traffic. Examples of multiple-accessnetworks with some form of support for real-time traffic are: Token Bus,Token Ring, CAN (Control Area Network), FDDI, and TTP (Time-TriggeredProtocol) [Kopetz 1997].

27

With regard to (packet-) switched real-time networks [Zhang 1995], they arehere assumed to be constructed of switches interlinked with point-to-pointinterconnections, i.e., so called mesh networks. Most reported switched real-time networks are wide area networks and are therefore not in focus in thisthesis (which is focused on short distance systems). Real-time messagestreams, e.g., multimedia streams, are normally supported in the form ofvirtual circuits or the like. The handling of these message streams in theswitches during the lifetime of a connection and at setup can be done in adeterministic or a statistical way. A deterministic service guarantees acertain performance even in the worst-case, while a statistical service doesnot have a hundred percent guarantee for this. Instead, a statistical servicemight have a specified probability of packet loss and deadline or jitterviolation.

����2SWLFDO�,QWHUFRQQHFWLRQV�LQ�3DUDOOHO�&RPSXWHUV

Optics have been used for a while in communication systems where opticalfibers each acts as a single point-to-point channel. However, the broad fieldof optical interconnections does not only include traditional single-channelfiber-optics. Novel technologies allow for features such as multiple high-speed channels in a single fiber and two-dimensional arrays of optical free-space channels. Some work on comparative technology studies andclassifications are reviewed below.

In arguments for optical interconnections in parallel computing systems andsimilar systems, some stated drawbacks or limitations of electricalinterconnects are [Tooley 1996] [Yatagai et al. 1996]:

• pin bottlenecks, both on the chip and the board level

• clock/signal skew

• bandwidth limitation

• high power consumption

• limited fanouts

Some advantages of optical interconnects are [Caulfield 1998] [Guilfoyle etal. 1998] [Irakliotis and Mitkas 1998] [Jahns 1994] [Jahns 1998] [Lund1997] [Ozaktas 1997B] [Stunkel 1997] [Yatagai et al. 1996]:

• large spatial and temporal bandwidth are offered

• optical beams do not affect one another

• light is immune to electromagnetic interference

28

• light beams do not suffer from signal frequency-dependentattenuation

• parallel global high speed interconnects are offered

• the possibility of non-planar interconnections is offered

• multiplexing can be done in multiple domains, e.g., the time andwavelength domains

• high-density interconnects where light beams can cross each other arepossible

• large electrical backplane contacts can be avoided, thereby reducingthe chassis size

• optical interconnects offer low energy per bit and high speed-powerproduct

• low skew is possible

Of course, there are also arguments for not using optics [Bohr 1998]. Forexample, it is argued that the performance of electrical interconnections willcontinue to scale [Horowitz et al. 1998], and it has been demonstrated that abit rate of 4 Gbit/s over 100 m of twisted-pair copper cable is possible [Dallyet al. 1998]. Comparative studies of optical and electrical interconnectionshave been reported, e.g., comparing energy consumption and system speed[Feldman et al. 1987] [Tooley 1996] [Yayla et al. 1998]. The best thing,however, is not to choose either electronics or optics but to use bothtechnologies in the respective areas to which they are best suited [Caulfield1998].

A classification of more advanced optical interconnection systems can bemade in a number of ways. Some possible groups of systems are listed below[Kurokawa et al. 1998]:

• parallel optical fiber links (e.g., fiber-ribbons and image fibers)

• free-space optical interconnects

• optical waveguide circuits

Ozaktas views alternative optical interconnection architectures for parallelcomputing as a tree in which the top-level alternatives are two- and three-dimensional systems [Ozaktas 1997B] [Ozaktas 1997C]. Two-dimensionalsystems are further divided into planar free space and waveguides, whilethree-dimensional systems are further divided into free space and fibers orwaveguides. Ozaktas then further divides three-dimensional systems andargues for such systems where devices arrayed on planes are globallyconnected with a regular connection pattern. The focus is obviously onbuilding dense parallel computing systems and not distributed systems such

29

as clusters of workstations. Many references to work related to free-spacesystems are found in [Ozaktas 1997] [Ozaktas 1997B]. One possibleclassification of free-space optical interconnection systems is [Yatagai et al.1996]:

• stacked optics

• planar optics

• stacked and planar optics

for which examples of systems are described and referred to in Chapter 6.More or less general discussions on the use of optical interconnections inparallel computers have also been published [Goodman et al. 1984][Rudolph 1998] [Schenfeld 1995] [Schenfeld 1996].

����'LVSRVLWLRQ�RI�WKH�7KHVLV

The dissertation is divided into three parts as described below:

• Part I: background material and tutorial surveys. The introduction isfollowed by two chapters on fiber-optic communication systems andinterconnection networks for parallel computers, respectively.

• Part II: optical interconnection networks for signal processing systems,especially for radar signal processing systems. A brief introduction isgiven to the radar signal processing systems developed in the projects ofwhich this work has been a part of. Presented in this part are bothselected network architectures of interest which were described in Part I,suggestions for how to configure these and other network architecturesespecially for signal processing applications, and the networks that aredescribed in more detail in Part III.

• Part III: the main contributions of the thesis work presented. Fourchapters present the WDM star network, the star-of-stars network, thecontrol channel based fiber-ribbon ring network, and the simplified fiber-ribbon ring network, respectively. The part then ends with a concludingchapter of the thesis.

Because of the disposition chosen, and the fact that Parts II and III containenough overlapping background information, one may begin reading inParts II or III, especially if the contents of earlier chapters are well knownto the reader.

31

���+LJK�3HUIRUPDQFH�)LEHU�2SWLF�1HWZRUNV

���� ,QWURGXFWLRQ

Optical fibers for communication systems offer a bandwidth of more than 30THz per fiber [Brackett 1990], giving them great potential for futurecomputer communication networks for data intensive applications. Thissurvey describes representative examples of high-performance fiber-opticnetworks as well as MAC (Medium Access Control) protocols [Rom and Sidi1990] for those networks. WDMA (Wavelength Division Multiple Access,described below) networks and passive optical networks (PONs) are givenspecial attention. Previous reviews of high-capacity fiber-optic networks fordata communication are found in [Acampora and Karol 1989] [Chatterjeeand Pawlowski 1999] [DeCusatis 1998] [Green 1991] [Modiano 1999][Ramaswami 1993] [Senior et al. 1998]. The review given here has a focuson optical LANs [Kazovsky et al. 1994] and similar networks that can beused for flexible communication in parallel and distributed computingsystems. Many proposed fiber-optic networks, although developed with aspecific application in mind, can be used in a wide range of applications;thus we do not necessarily sort those networks out. Nevertheless, e.g., puretelecommunication networks are not treated here.

A review of fiber-optic network technologies, components, and classificationsis presented in Sections 2.2 through 2.5. Examples of WDMA networks aredescribed in Section 2.6. Section 2.7 gives a conclusion and summary.

����0XOWLSOH�$FFHVV�0HWKRGV

Accessing and sharing the huge amount of optical bandwidth among themultiple nodes in a fiber-optic network is a challenging problem. Fourmultiple access methods are described below: WDMA, TDMA (Time DivisionMultiple Access), SCMA (SubCarrier Division Multiple Access), and CDMA(Code Division Multiple Access). The most popular one for achieving highaggregated bandwidth seems to be WDMA, especially in end-user systemsowing to their simple transceiver designs. Additional information onmultiple-access methods is found in [Gagliardi and Karp 1995].

������ :DYHOHQJWK�'LYLVLRQ�0XOWLSOH�$FFHVVWhen using WDMA, multiplexing is done in the spectral domain of the light[Hill 1989] [Kaminow 1989] [Green and Ramaswami 1990] (WDMA systemsuse the WDM technology, and the two terms are therefore often used inter-changeably here). In this way, several optical carriers, or channels, are

32

implemented in the network. For systems with denser wavelength spacingthan 1 nm, the WDMA technique is also referred to as optical FDM(Frequency Division Multiplexing) [Agrawal 1992]. One way to use thewavelength channels is to allow several nodes to transmit simultaneouslyon different channels. Data are then sent bit-serially between nodes tunedto the same channel. The channel to which each transmitter and receiverare tuned at any time is controlled by the MAC protocol. Another way is tosynchronously transmit on several channels in parallel, i.e., bit-parallelbyte-serial transmission [Loeb and Stilwell 1988] [Loeb and Stilwell 1990].However, compensation for bit-skew caused by group delay dispersion(different wavelength channels travel at different speeds in the fiber) maybe needed in these systems [Jeong and Goodman 1996].

The practical limit (e.g., because of “the impracticality of administering anetwork with a very large number of differing lasers”) in the number ofwavelengths in LAN networks and similar networks is expected to besomewhere between 16 and 32 [Brackett 1996]. However, a WDM systemwith 100 channels has been demonstrated [Toba et al. 1990], and systemswith thousand 1 Gbit/s channels are likely in the future [Wailes and Meyer1991]. Experimental WDM demonstrations include:

• Experiments with 16 channels at 2 Gbit/s [Lin et al. 1988]. TunableFabry-Perot filters (see Section 2.3.1) were used for channel selection.

• A WDM experiment with 16 channels, each at 622 Mbit/s [Toba et al.1989]. One of the channels were selected by a tunable filter consistingof Mach-Zehnder interferometers (see Section 2.3.1).

• A system with 100 channels, each carrying data at a bit rate of 622Mbit/s [Toba et al. 1990].

• A commercially available WDM link called MuxMaster [Janniello et al.1995]. A MuxMaster link has 20 wavelengths on a single fiber whichimplement 10 full-duplex protocol-independent connections.Experiments with a bit rate of 200 Mbit/s on each wavelength werereported but bit rates in the order of Gbit/s are possible. Maximumdistance is 50 km.

• A 160 Gbit/s system with 8 × 20 Gbit/s channels and a channel spacingof 4 nm [Sorel et al. 1996].

• 1 Tbit/s aggregated capacity with 50 channels, each modulated at arate of 20 Gbit/s [Chraplyvy et al. 1996].

33

������ 7LPH�'LYLVLRQ�0XOWLSOH�$FFHVVTDMA networks can be divided into two groups, those using SDFNHW�LQWHUOHDYLQJ and those using ELW�LQWHUOHDYLQJ. Networks using packet-interleaving, for example [Barry et al. 1996], divide the access to themedium into time slots. The length of each slot normally equals thetransmission duration of a packet. Simple MAC protocols can be used, buteach node must work at the speed of the aggregated bit rate in the network.

With bit-interleaving, each node sends only one bit at a time at regularintervals. If 1 nodes transmit simultaneously, the bit stream from one nodewill have a bit-to-bit interval of 1 bit slots. The width of the optical pulseis, however, normally several times shorter than the duration of a bit slot.In a bit-interleaved TDMA network, clock synchronization must bemaintained with a precision in the order of less than a bit period, which isdifficult when a bit rate of multiple Gbit/s is used. Experiments with a 250Gbit/s network were reported in [Prucnal et al. 1994]. The optical pulsewidth was 1 ps, the bit slot was 4 ps and the bit-to-bit interval from onenode was 10 ns. This resulted in the capacity to support a 100 Mbit/s bit-stream to each of 2 500 nodes. Bit-interleaved TDMA networks andtechnologies to implement such networks are reviewed in [Spirit et al. 1994][Seo et al. 1996].

������ 6XEFDUULHU�'LYLVLRQ�0XOWLSOH�$FFHVVThe data from each node in a SCMA network are used to modulate node-specific microwave subcarriers, i.e., multiplexing is done in the microwavefrequency domain [Darcie 1987]. The subcarrier then modulates an opticalcarrier. All subcarriers are detected at the receiver photo-diode, but only thedesired one is demodulated using conventional microwave techniques.SCMA can be contrasted with WDMA, where multiplexing is also done inthe spectral domain but on the lightwave carrier, about 1014 Hz, while thesubcarrier microwave frequency is at 108 to 1010 Hz [Mestdagh 1995].

A typical application suitable for SCMA is distribution of analog videochannels [Chang et al. 1995]. The video channels can then remain analogthrough the whole network, and similar FM techniques as used today can beemployed. However, SCMA has also been proposed for use in datacommunication networks [Darcie 1987].

In [Georges and Lau 1993], an SCMA network experiment using self-pulsating CD laser-diodes was reported. By changing the pulse frequency (inthe range of 2 to 4 GHz), a node can tune to another subcarrier channel. Abit rate of 150 Mbit/s was used in the experiment.

34

������ &RGH�'LYLVLRQ�0XOWLSOH�$FFHVVSeveral CDMA methods for fiber-optic communication have been proposed.The majority of work is being done on the method having a set of orthogonalcode sequences of FKLSV, where each chip has a value of either “1” or “0”[Salehi 1989] [Salehi and Brackett 1989] [Marhic 1993]. Each code sequencethen corresponds to the destination address of a specific node. Duringtransmission, each “1” bit is encoded with the destination node’s codesequence of chips, while a “0” bit is encoded with a sequence of only “0”chips. The drawback of this scheme is that each transceiver must work at aspeed 1 times higher than the bit rate, where 1 is the length of the code,i.e., the number of chips. In wireless communication systems, this method isreferred to as Direct Sequence Spread Spectrum (DS-SS) [Bantz andBauchot 1994].

Decoders to be used in the receivers can be implemented optically [Prucnalet al. 1986]. CDMA experiments using an optical decoder were reported in[Prucnal et al. 1986B]. The optical decoder consisted of 3 delay lines where3 is the number of chips that are set to 1 in the receiving node’s codesequence. By matching the delay lines to the corresponding positions of the“1” chips in the code sequence, an auto correlation function sensitive to thecode sequence is obtained. A similar method was proposed for use in thetransmitter. A bit rate of 3.125 Mbit/s was achieved on a 100 Mbaud fiber-optic link, using 32 chips per bit.

Instead of having the code sequence spread in time, spectral encoding can beused. In [Zaccarin and Kavehrad 1993], the light from a LED (LightEmitting Diode) is spatially split into a discrete number of spectralcomponents by a diffraction grating. The light is then passed through aspatial amplitude mask with the code sequence and combined in the fiber.In this way, each of the spectral components is encoded with one of thechips, and the LED need only to be modulated with the bit rate. A similartechnique is used in the receiver. In [Weiner et al. 1988], the code sequenceis also applied in the spectral domain but with a phase mask instead of anamplitude mask.

Another CDMA method called PFDM (Pulse Frequency DivisionMultiplexing) is described in [Frenkel 1992]. Short laser pulses with a nodespecific repetition rate are used to encode the data. The received signal canbe decoded with optical delay lines, with delays proportional to the pulsefrequency, and optical AND gates.

������ 'LVFXVVLRQThe different properties of the multiple-access methods make them suitablefor different applications/systems. In the kind of systems in focus in thisthesis, a limited number of users share the network cost (the opposite of

35

telecommunication backbone networks). CDMA and bit-interleaved TDMAare therefore considered too complex. When a bit rate of several Gbit/s isneeded from each node in the network simultaneously, packet-interleavedTDMA will also be too complex, because every node must work at theaggregated bit rate of the network. SCMA is also excluded because of thelimited network capacity [Mestdagh 1995]. What is left is WDMA, whichgives easy access to the channels when tunable components reachreasonable costs, and where each node need only work at the speed of itsown bit rate.

Several proposed network architectures use WDMA in combination with oneof the other multiple access methods, for example, TDMA [Jonsson et al.1995] and SCMA [Shankaranarayanan et al. 1991]. As an example, datapackets can be carried on wavelength channels while network controlinformation is carried on subcarrier channels [Chiang et al. 1996]. Thesekinds of hybrids can give new, valuable properties to networks, for example,flexible real-time properties when combining TDMA and WDMA [Jonsson etal. 1996].

���� &RPSRQHQWV�IRU�:'0$�1HWZRUNV

A fiber-optic communication system typically consists of a transmitter, areceiver, and some form of transmission medium based on optical fibers.Common optoelectronic components used in the transmitter are LEDs andlaser diodes [Ettenberg 1998], where laser diodes can be divided intomultimode and single-mode laser diodes. Single-mode laser diodes havenarrower spectral widths than multimode laser diodes by the incorporationof a grating filter inside or outside the cavity. The conventional multimodelaser diode is called the Fabry-Perot laser, while two common single-modelaser diodes are the DBR (Distributed Bragg Region) and the DFB(Distributed FeedBack) laser diodes [Carroll et al. 1993]. In the receiver, thecommonly used components are PIN (P-insulator-N) diodes and APDs(Avalanche Photo Diode). The three classic types of optical fibers aremultimode step-index fiber, multimode graded-index fiber, and single-modestep-index fiber, each with its own core/cladding design and dispersioncharacteristics. Further reading on fiber-optic communication systems andtheir components is found in [Keiser 1991], while an overview of componentsspecially needed in WDMA networks is presented here.

Components for WDMA networks is a large research field, and examples ofproposed components are given below. The component descriptions are givenfrom a systems view, explaining what system function they are to performand what level of performance can be expected. The aim is to provide thenecessary amount of knowledge before proceeding to the presentation ofarchitectures and protocols and leaving implementation details. In most

36

parallel and distributed computing systems, the longest distance betweentwo end-nodes is rather short. Topics related to long distancecommunication are therefore not described here, e.g.:

RSWLFDO�DPSOLILHUV: may be needed when the optical signal is split to manynodes but are otherwise used when the signal is passed and attenuatedthrough long distances of fiber

FRKHUHQW� GHWHFWLRQ: although it offers narrow channel spacing, it is toocomplicated for end-user systems

GLIIHUHQW� NLQGV� RI� ILEHUV: performance trimming of fibers (lowerattenuation and dispersion) is important for telecommunication but lessrelevant for short-distance communication

More information on WDMA components is found in [Borella et al. 1997][Brackett 1990] [Green 1993] [Sadot and Boimovich 1998] [Tong 1998][Zirngibl 1998].

������ 7XQDEOH�5HFHLYHUVA receiver in a WDMA network must tune in one of all the wavelengthsfrom the incoming fiber. Since a photo diode itself detects a broad band ofwavelengths, a wavelength filter must be used. Tunable optical filtersinclude interferometer filters, filters based on mode coupling, filters basedon resonant amplification, and grating based filters. Fast tuning and abroad tuning range is desired while the pass-bandwidth should be adaptedto the incoming channels, i.e., it should pass the whole energy of onechannel while preventing energy from other channels from being detected.

The Mach-Zehnder interferometer filter splits the incoming light into twobeams, delays one of the beams slightly more than the other and then letsthe two beams interact with each other [Hecht 1987]. Tuning is obtainedwhen the delay is changed. By cascading several Mach-Zehnderinterferometer filters, a greater wavelength selectivity is achieved. In[Wooten et al. 1996], a three-stage electro-optically tunable Mach-Zehnderinterferometer with a 50-ns tuning latency over a tuning range of eightchannels is demonstrated. Experiments with a 16-channel (four stages)interferometer was reported in [Oda et al. 1989], while a 128-channel (sevenstages) interferometer were used in the 100-channel transmissionexperiment reported in [Toba et al. 1990].

The Fabry-Perot filter is another interferometer in which two mirrors forma resonant cavity [Hecht 1987]. With a mirror reflectivity of less than 100percent, some of the resonated light will escape out. The Fabry-Perot filteris tuned by changing the distance between the mirrors, i.e., moving one of

37

the mirrors. The distance must be short to limit the number of resonatingwavelengths to one in the working range of the communication system.Because of the need of mechanically moving one of the lenses, the tuninglatency is large. Piezoelectric transducers are commonly used to change thedistance [Miller and Janniello 1990], while another method is to rotate(relative to the incoming and outgoing beams) a Fabry-Perot filter with afixed distance [Frenkel and Lin 1988]. Experiments with cascaded Fabry-Perot filters to enhance performance have also been reported [Kaminow etal. 1989]. The two-stage filter described in [Miller and Miller 1992],supports 1000 channels in a 40 nm range. A tunable Fabry-Perot filterwithout moving parts is described in [Patel et al. 1990]. The cavity wasfilled with liquid crystals, and wavelength tuning was obtained by applyingan electric field to change the refractive index of the liquid crystals.

Tunable filters based on mode coupling refers to filters with wavelengthdependent coupling between optical fields (modes) where either acousto-optic, electro-optic, or magneto-optic effects are used [Kobrinski and Cheung1989]. The optical signal is first split into two orthogonal polarization statesand then passed through a resonant structure [Green 1993]. For awavelength corresponding to the resonant frequency, the polarization statewill be changed 90 degrees. A second polarization splitter at the output willfilter this wavelength out from the rest of the wavelength channels. Thedifferent kinds of mode coupling filters generate the resonant structure indifferent ways. Electro-optic filters have low tuning latencies, while acousto-optic filters have a broad tuning range and support multi-wavelengthfiltering by applying multiple acoustic frequencies. Acousto-optic filters arereviewed in [Cheung 1990] [Gupta 1997].

If a single-mode laser diode is biased below threshold, it will start emittingwhen incoming light, at the wavelength selected by the grating in the laser,is present. If a tunable single-mode laser (see below) is used, a wavelengthtunable filter with amplification is achieved. A tuning latency as low as 1 nshas been demonstrated [Kobrinski et al. 1988], but the tuning range is verylimited. As an example, the tuning range demonstrated in [Numai et al.1989] was 0.95 nm, although an 18-channel wavelength selection wasexpected with this filter.

In a grating based filter, the incoming light is spatially split into itswavelength components. If a photo diode array is used to detect all of thewavelength components (one wavelength per photo diode), as proposed in[Kirkby 1990], a multi-channel receiver is achieved in which the electronicswitching time sets the tuning latency, i.e., very fast. A grating integratedmonolithically with a photo diode array detecting 42 wavelength channelsspaced by 4 nm was reported in [Cremer et al. 1992]. In [Parker and Mears1996], one digitally tunable wavelength is steered to the output fiber byusing an SLM (Spatial Light Modulator) together with a fixed grating.

38

Tuning to discrete wavelengths spaced by 1.3 nm over a tuning range of38.5 nm was demonstrated. The demultiplexing of 120 channels by the useof a concave grating was reported in [Sun et al. 1998].

������ 7XQDEOH�7UDQVPLWWHUVWith wavelength-tunable transmitters, a certain transmission wavelengthcan be chosen, either from a continuous range or from a number of discretewavelengths. From a system engineer’s point of view, an ideal tunable laserdiode has a tuning range of about 100 nm and a tuning latency in the orderof nanoseconds [Mestdagh 1995]. Many components aiming to meet one ofthe two wishes have been proposed, and some meet both rather well. Thelinewidth (spectral width; normally measured as the FWHM, Full WidthHalf Maximum) of the emitted light gives a hint of the possible number ofchannels, but other parameters such as receiver selectivity must beconsidered when designing a WDM communication system. Wavelength-tunable laser diodes are reviewed in [Lee and Zah 1989].

The simplest way to achieve a tunable transmitter is by temperaturetuning, utilizing the fact that the refractive index of the active layer istemperature dependent. Although a tuning range of 10.8 nm has beendemonstrated [Kameda et al. 1993], only limited tuning range and longtuning latency are achieved with this method.

In [Kobrinski et al. 1990], a demonstration of a tuning latency of less than15 ns over a tuning range of 2.2 nm and experiments with 16 wavelengthchannels are reported. A three-section DBR laser was used where two of thesections were used for wavelength tuning by changing the values of theinjected currents and, hence, the index of refraction. As a consequence ofchanging the index of refraction of the grating in a DBR, the lasingwavelength (optical frequency) is changed. In [Goodman et al. 1988], eightchannels were achieved from a double-section DFB with a tuning speed ofless than 5 ns. A four-wavelength DBR laser array is reported in [Delormeet al. 1996]. Each laser is a three-section DBR tunable over a range of 12nm. The total tuning range of the component is 28 nm.

By activating one laser in an array of laser diodes emitting at differentwavelengths, a discretely tunable laser is achieved. A 7 × 11 [Chang-Hasnain et al. 1991B] and a 7 × 20 [Chang-Hasnain et al. 1991] VCSEL(Vertical Cavity Surface Emitting Laser) array with each laser emitting aunique, non-redundant wavelength has been reported (see [Giboney et al.1998] for general information on VCSELs). The latter array had awavelength span of 43 nm, and WDM experiments were carried out inwhich four of the lasers were each coupled to a fiber. The fibers werecombined by a star coupler, and each laser was simultaneously modulatedat 155 Mbit/s. A 2 × 8 VCSEL array with a small overlap in wavelength

39

range between the two rows but in which each laser was capable of 5 Gbit/soperation was reported in [Maeda et al. 1991]. Other WDM VCSEL arraysinclude a 2 × 2 densely-packed array [Huffaker and Deppe 1996], and otherWDM array components include a DFB laser array with 20 lasers emittingat ten 2-nm spaced wavelengths (two diodes per wavelength) [Lee et al.1996].

One of the disadvantages of array components is the difficulty in couplingthe light from all laser diodes into one fiber. However, the integration of a21-wavelength DFB laser array, a star coupler, and two optical amplifierson one chip was reported in [Zah et al. 1988].

Another way to get multiple parallel wavelength channels is by spectralslicing of the output light from a number of broad-spectrum LEDs [Zirngiblet al. 1996]. By filtering out a different wavelength from each diode, awavelength tunable transmitter is obtained when selecting one of the diodesfor modulation. Experiments with a 16-channel 150 Mbit/s systememploying spectral slicing were reported in [Chapuran 1991].

Several external cavity laser diodes with external filters for wavelengthtuning have been reported. By antireflection coating one of the output facetson a laser diode and having an external mirror, the cavity length isextended. If a diffraction grating is used as the external mirror, tuning isobtained by moving the grating (fine-tuning by changing the cavity length)or rotating the grating (a different longitudinal mode is selected from thegrating) [Mellis et al. 1988]. Tuning ranges exceeding 240 nm have beendemonstrated with external-cavity grating-based laser diodes [Bagley et al.1990] [Tabuchi and Ishikawa 1990]. The drawback of these grating-basedlasers is the large tuning latency caused by the mechanical movement of thegrating. However, devices with shorter tuning latencies have beendemonstrated. One way to avoid the mechanical moving mechanism is toexchange the diffraction grating with an acousto-optic filter that selects thelasing wavelength and allows for tuning latencies in the range ofmicroseconds [Coquin and Cheung 1988]. Another fully electronicallytunable laser diode is the MAGIC (Multistripe Array Grating IntegratedCavity) laser diode [Soole et al. 1992]. The MAGIC laser chooses onewavelength for lasing by activating one of several waveguide stripes. A fixeddiffraction grating couples a specific wavelength into each stripe.

In [Larson and Harris 1996], a 15-nm tuning range and a 0.14 nm linewidthare demonstrated for a single VCSEL laser. A movable mirror was placed ontop of the laser with an air gap between. By moving the mirror, the cavitylength and, hence, the wavelength of the emitted light are changed. Asimilar tunable VCSEL but with a tuning range of 31.6 nm was recentlyreported [Sun et al. 1998]. An LED with the same tuning mechanism, a

40

tuning range of 39 nm, and a linewidth of 1.9 nm is presented in [Larsonand Harris 1995].

A diode-laser pumped tunable fiber laser is reported in [Chieng andMinasian 1994]. Light in the 1550 nm range goes through an Erbium-dopedfiber ring, where it is amplified. A tunable Mach-Zehnder interferometerselects one wavelength which is re-amplified. A tuning range of 39 nm isachieved. A fiber laser with an electro-optic filter is reported in [Chollet etal. 1996]. The laser supports 50 wavelength channels over a spectral rangeof 10 nm with a 50-ns tuning latency.

������ 2WKHU�:'0�&RPSRQHQWVEssential system components when building fiber-optic networks like thosedescribed in the next section include splitters, combiners, stars, wavelengthconverters [Nesset et al. 1998], WDM demultiplexers, and WDMmultiplexers. Several of the components mentioned are commerciallyavailable with limited features, e.g., a small number of input/output ports.

A 1�×�1 fiber-optic splitter splits the light from one input fiber to 1 outputfibers. Both symmetric and asymmetric splitters are available, where anasymmetric splitter splits an unequal amount of light to the different outputports. A combiner works in the opposite way combining input signals fromseveral fibers to one output fiber. An 1 × 1 fiber-optic star (often referred toas a passive optical star) can be viewed as an 1 × 1 combiner followed by an1�×�1 split. The conventional way to build a star is to use a number of 2 × 1combiners and 1 × 2 splitters. However, it is difficult to build large starsusing this technique, and other techniques have been proposed to overcomethe problem [Okamoto et al. 1992] [Yun and Kavehrad 1992]. A 144 × 144passive optical star is described in [Okamoto et al. 1992B] and [Kato et al.1993].

A WDM demultiplexer is a splitter that splits the incoming signal on onefiber to several output fibers, each with a different spectral component ofthe input signal [Juma 1996]. The demultiplexer either splits the inputsignal into groups of wavelength channels or into one separate wavelengthchannel per output fiber. The most common methods for demultiplexing theinput signal are said to be the use of an interference filter or a diffractiongrating [Straus and Kawasaki 1987]. A WDM multiplexer couples togetherdifferent wavelengths on different fibers into one outgoing fiber. Althoughthere exist WDM multiplexers with wavelength selective inputs, thesimplest way to get a WDM multiplexer is to use an ordinary� 1 × 1combiner.

41

The ADM (Add-Drop Multiplexer) is another component in which one orseveral wavelengths may be added/dropped to/from a bypassing fiber. Thecomponent can be used in, e.g., WDM ring networks (described in the nextsection). An ADM based on two fixed wavelength, multi-layer Fabry-Perotfilters is described in [Hamel et al. 1995]. A component in which thewavelength to be added/dropped can be tuned over a discrete number ofwavelengths is described in [Glance 1996].

An arrayed waveguide grating (AWG) [Okamoto and Inoue 1995] can beused, e.g., as a demultiplexer in a wavelength router [McGreer 1998] (seeSection 2.5.1) or as an ADM. In the case of a single-input AWG, the lightfrom the input waveguide is split into a number of waveguides withdifferent lengths. Where all these waveguides end, the light beamsconstructively interfere at the entrance of one of several output waveguides.The output waveguide into which the light is transferred depends on thewavelength of the light and the input waveguide, if several exist, into whichthe light entered.

���� ,QWHUFRQQHFWLRQ�$UFKLWHFWXUHV

The simplest fiber-optic transmission system is to have a point-to-point linkbetween two nodes. A network can be built by using several point-to-pointlinks. These so-called point-to-point linked networks have beencommercially available for a while and are described in Section 2.4.1.However, all-optical multi-access networks are expected to be popular in thefuture (described in Section 2.4.2).

������ 3RLQW�WR�3RLQW�/LQNHG�1HWZRUNVIn a fiber-optic point-to-point linked network, a number of optically-isolatedlinks connect the nodes in the network. If, for example, wavelength divisionmultiplexing should be used to increase the bandwidth, the technology mustthus be implemented separately in each point-to-point link. The main typesof point-to-point linked networks that fall into the scope of this report arering networks [Davies and Ghani 1983] and switched networks.

In the FDDI network [Ross 1989] [Jain 1993], the nodes are connected in aunidirectional primary ring where each node receives from one link andtransmits on another. A node acts like a repeater for all incoming messagesexcept those addressed to the node and those sent by the node and should beremoved from the ring after one round. In addition to the primary ring,some or all nodes are connected to a secondary ring which, for example, canbe used as a backup ring. A transmission rate of 125 Mbaud is used, whichgives a data rate of 100 Mbit/s because of the 4b/5b encoding that is used. Atoken protocol is used for medium access control. Guarantees can be given

42

for both latency and bandwidth in the FDDI network and its successor,FDDI-II [Ross 1989] [Jain 1993].

In the Wavelength Distributed Data Interface (WDDI) network described in[Ramamurthy et al. 1995], multiple FDDI networks are obtained on logicalrings in a WDM ring network. Only a subset of the total number of nodes inthe network is connected to each logical ring. Some of the nodes act asbridges to interconnect two or more logical rings.

In switched networks, one or several switches connect the nodes together.All traffic in the network passes one switch at least. One well knownswitched network is ATM (Asynchronous Transfer Mode). ATM wasdeveloped to carry a wide range of traffic such as video, voice, and data, bothat the local and the global levels. To ease the transition from current LANtechnologies, ATM LAN emulation has been developed to support emulationof a broadcast medium [Truong et al. 1995] [Finn and Mason 1996].Examples of different services provided by an ATM network are [Reilly1994] [Suzuki 1994]:

• Constant Bit Rate (CBR) with minimal and constant delay betweenthe end-nodes. Can be used for, e.g., voice over ISDN.

• Variable Bit Rate (VBR) with minimal and constant delay betweenthe end-nodes. Designed to handle video and audio where, e.g., thecompression ratio can vary.

• Unspecified Bit Rate (UBR), e.g., for data transmissions without real-time requirements.

As indicated above, real-time services are supported, but guarantees are notmade for separate cells (data units in ATM each consisting of 48-byte dataand a 5-byte address) without a setup phase, due to the fact that ATM is aconnection-oriented network.

Electronic ATM switches and integrated switching circuits are commerciallyavailable [Goldberg 1994], while optical switches have been proposed[Nishio et al. 1993] [Jajszczyk and Mouftah 1993] [Blumenthal et al. 1994].The way in which ATM networks can be used in parallel and distributedcomputing systems is discussed in [Hariri and Lu 1996]. Another networkfor parallel processing systems, using fiber-optic point-to-pointinterconnections, is Nectar, where one or several electronic crossbars switchthe traffic [Arnould et al. 1989].

A multi-stage switching system with WDMA interconnections between thestages and with wavelength conversion inside the switching stages wasreported in [Fujiwara et al. 1988]. Wavelength conversion was obtained by

43

converting the optical signal first into electronic form and then back tooptical again, but at another wavelength.

A system component that has reached the market recently [Bursky 1994][Fibre Systems 1998] is the fiber-ribbon link [Buckman et al. 1998][Engebretsen et al. 1996] [Hahn 1995] [Hahn 1995B] [Hahn et al. 1996][Hartman et al. 1990] [Jiang et al. 1998] [Karstensen et al. 1995][Karstensen et al. 1998] [Kuchta et al. 1998] [Nagarajan et al. 1998][Nishimura et al. 1997] [Nishimura et al. 1998] [Schwartz et al. 1996] [Sialaet al. 1994] [Wickman et al. 1999] [Wong et al. 1995]. Several links can beused to build high bandwidth point-to-point linked networks [Hahn et al.1995]. With ten parallel fibers, each carrying data at a bit rate of 400Mbit/s, an aggregated bandwidth of 4 Gbit/s is achieved [Schwartz et al.1996]. Bi-directional links, with some fibers in the fiber-ribbon cablesdedicated for each direction, are also possible [Jiang et al. 1995]. Furtherdiscussions on fiber-ribbon aspects are presented in Section 9.5, page 153,while more references to reports on fiber-ribbon links are found in [Tooley1996].

Modules that support multiple high-speed channels but are not specificallyoptimized for fiber-ribbons have been reported, e.g., receiver andtransmitter modules with five channels, each channel with a bit rate of 2.8Gbit/s [Nishikido et al. 1995].

������ $OO�2SWLFDO�0XOWL�$FFHVV�1HWZRUNVIn an all-optical network, the data stream remains in the optical form allthe way from the transmitter to the receiver. Three basic architectures forall-optical multi-access networks are the ring, the bus, and the star (seeFigure 1). These network architectures will be discussed below.

An all-optical multi-access ring network differs from a traditional ringnetwork in the sense that all other nodes can be reached in a single hopwithout any intermediate optoelectronic conversion. In contrast to therepeating function of a node in, for example, a FDDI network, messagessimply pass a node through passive optics in an all-optical multi-access ringnetwork. This is true for all messages except a node´s own messages thatshould be removed from the ring after one round. Just a fraction of theoptical power contained in the bypassing fiber is tapped to the receiver,which gives all nodes the opportunity to read the message, i.e., a multicast(one to many) or a broadcast (one to all). Outgoing messages are insertedinto the ring and, in a multi-channel system, mixed together with bypassingmessages on other channels.

In the WDMA ring network described in [Irshid and Kavehrad 1992], eachnode is assigned a node-unique wavelength on which to transmit.

44

Transmitted messages are then accessed by tunable receivers. Experimentswith a WDMA unidirectional ring network are reported in [Chawki et al.1995], while a WDMA bi-directional ring network is described in [Elrefaie1993]. WDMA ring networks with central electronic switching are describedin [Wagner and Chapuran 1992]. WDMA ring networks for long distancecommunication have already reached the market [Grenfeldt 1998].

(a)

(b)

(c)

)LJXUH����7KUHH�SDVVLYH�RSWLFDO�QHWZRUN�DUFKLWHFWXUHV���D��ULQJ��E���GXDO��EXV��DQG��F��VWDU�

45

A hierarchical ring network is described in [Louri and Gupta 1996] [Louriand Gupta 1997]. One ring is dedicated to each cluster for intra-clustercommunication, while a hierarchical all-optical (no optical-to-electricalconversion or vice versa in the gateway nodes) ring network is used forinter-cluster communication. A TDMA protocol is investigated for use in thenetwork.

In an optical bus, the light travels only in one direction, making it necessaryto have two buses (upper and lower), one for each direction (higher or lowernode index of destination nodes). This kind of bus architecture is called GXDOEXV. The disadvantage of the dual bus is that two transceivers are needed ineach node. This is avoided in the IROGHG� EXV, where the two buses areconnected with a wrap-around connection at one end of the buses (seeFigure 2) [Tseng and Chen 1982]. In the folded bus, transmitters areconnected to the upper bus while receivers are connected to the lower bus.

Several bus architectures and hybrids in which the bus is part of thearchitectures are discussed in [Nassehi et al. 1985]. A WDMA dual busnetwork is described in [Cheung 1992], and a WDMA folded bus network isdescribed in [Chlamtac and Ganz 1988]. WDMA dual bus networks in whichmessages do not always remain unchanged until the end of the bus, but inwhich the protocols support several transmissions on the same wavelengthchannel simultaneously (wavelength reuse), are presented in [Huang andSheu 1996] [Huang and Sheu 1997].

In a star network, the incoming light waves from all nodes are combinedand uniformly distributed back to the nodes. In other words, the opticalpower contained in the middle of the star is equally divided between allnodes. WDMA star networks are described in numerous papers and arespecially treated in Section 2.6. In addition to LANs and similar networks,WDMA star networks have also been proposed for internal use in packetswitches [Eng 1988] [Brackett 1991].

Hierarchical WDM star networks include the wavelength-flat (all nodesshare the same wavelength space) tree-of-stars network [Dowd et al. 1993],

)LJXUH����)ROGHG�EXV�

46

the tree-of-stars network (called LIGHTNING) that has wavelength routingelements between each level [Dowd et al. 1996], the star-of-stars networkthat has an electronic gateway node between each cluster and the backbonestar [Jonsson and Svensson 1997] and the multiple star network where eachnode is directly connected to both a local star and a remote star [Ganz andGao 1992B].

All of the three basic network architectures have different advantages. Thering has the least amount of fibers, a bus network’s medium access protocolcan utilize the linear ordering of the nodes [Nassehi et al. 1985] and theattenuation for an (ideal) star only grows logarithmically with the numberof nodes. However, star networks are the most popular, judging from thenumber of published papers.

���� &ODVVLILFDWLRQV�RI�:'0$�1HWZRUNV

The classifications below give the terminology and the backgroundknowledge assumed in the overview of proposed networks and protocols(medium access control protocols) in the next section. If not otherwisestated, multi-access, single-hop broadcast-and-select networks (explainedbelow) are assumed in the following sections. More information on WDMAnetworks is found in [Borella et al. 1998] [Brackett 1990] [Green 1993][Mestdagh 1995]. Predictions for future directions in WDMA networking arefound in [Brackett 1996] [Green 1996].

������ %URDGFDVW�DQG�6HOHFW�DQG�:DYHOHQJWK�5RXWLQJThe difference between broadcast-and-select networks and wavelengthrouting networks is whether or not there exists any wavelength dependentrouting in the network [Gerstel 1996]. The path followed from a transmitterto a receiver in a wavelength routing network is determined by the selectedwavelength. As long as two paths do not have any common fiber links, theycan use the same wavelength simultaneously (wavelength reuse). This isnot possible in a broadcast-and-select network, where all receivers alwayshave all transmitted channels available on the incoming fiber. However, abroadcast-and-select network does not need any wavelength selectivedevices out in the network. Instead, the receivers decide (on the basis of theprotocol used) when to receive and/or which channel to tune in.

Several wavelength routing networks have been proposed, for example, thethree-level hierarchical network described in [Alexander et al. 1993].Another hierarchical wavelength routing network is LIGHTNING [Dowd etal. 1995] [Dowd et al. 1996]. The topology in LIGHTNING is a tree-structure of passive optical stars where each level is separated by a ODPEGDSDUWLWLRQHU. The lambda partitioner selects the wavelengths that are to stay

47

in the levels beneath the partitioner and the wavelengths that will pass tothe levels above. In this way, the number of wavelengths in each fiber hasits minimum at the top level (only top level wavelengths exist) andmaximum at the bottom level (wavelengths for all levels exist). Allwavelengths that are not passed to a certain level can be reused in eachcluster on the level below.

The focus in this section is on broadcast-and-select networks and examplesof such networks are found in Section 2.6.

������ 6LQJOH�+RS�DQG�0XOWLKRS�1HWZRUNVIn a single-hop network, all nodes can reach any other node in a single hop.This means that the transmitted data are not passed through anyintermediate routing stages and remain in optical form all the way from thesource node to the destination node. A disadvantage of single-hop networksis that each transmitter and/or receiver must be equipped with tunablecomponents. Advantages are flexibility and the absence of extra latency atintermediate nodes.

In a multihop network, fixed-wavelength (or slowly tunable) transmittersand receivers are configured to shape a virtual topology. As an example, anode may be able to transmit on λ1 , λ2, and λ3 in a broadcast-and-selectnetwork, where the three neighbors in a virtual binary three-dimensionalcube have receivers tuned to λ1 , λ2, and λ3, respectively. A number of virtualtopologies have been investigated, e.g., perfect shuffle in Shufflenet[Acampora and Karol 1989]. An experimental multihop network wasreported in [Gidron 1992]. An advantage of multihop networks is that nocomponents with low tuning latency are needed. By having slowly tunablecomponents, the virtual topology can be reconfigured when the overalltraffic demands in the network are changed.

Although multihop networks can be reconfigurable, single-hop networks aremore flexible. This chapter focuses on single-hop networks. Single-hopnetworks are reviewed in [Mukherjee 1992], and multihop networks arereviewed in [Mukherjee 1992B].

������ 1HWZRUNV�ZLWK�DQG�ZLWKRXW�&RQWURO�&KDQQHOA number of protocols for WDMA networks assume a separate controlchannel. This control channel is normally used to reserve access on the datachannels. Hence, these protocols are sometimes called UHVHUYDWLRQ protocols.Protocols for networks with more than one control channel have also beenproposed, for example, 1-DT-WDMA [Humblet et al. 1992] [Humblet et al.1993]. Non-control channel based networks often assign a fixed homechannel to each node, either on the transmitter side or on the receiver side.

48

Protocols for these networks are then called SUH�DOORFDWLRQ protocols.Protocols for networks without control channel but where access to the datachannels is divided into a control phase and a data phase are found in[Sivalingam and Dowd 1995] [Jonsson et al. 1996]. These protocols arecalled K\EULG protocols.

Instead of having an additional wavelength channel for the control channel,SCMA can be used to achieve a number of control channels [Su andOlshansky 1993]. In the proposed network, the subcarrier carrying a node’scontrol data modulates the same laser diode as that on which the data aretransmitted. Two photo diodes are used in the receiver, one with a filter thatfilters out the wavelength with the data and one without a filter to receiveall subcarriers in the network. One of the subcarriers is then selected usingmicrowave techniques.

������ 5HFHLYHU�DQG�7UDQVPLWWHU�7XQDELOLW\The most flexible networks are those with both tunable transmitters andtunable receivers. However, protocol complexity and node cost decreasewhen fixed wavelength units are used in either the transmitter or thereceiver. Using the classification scheme given in [Mukherjee 1992], WDMnetworks can be divided into:

• FT-FR (Fixed Transmitters and Fixed Receivers)

• TT-FR (Tunable Transmitters and Fixed Receivers)

• FT-TR (Fixed Transmitters and Tunable Receivers)

• TT-TR (Tunable Transmitters and Tunable Receivers)

Some networks have more than one transmitter and/or receiver in eachnode. A network with L fixed transmitters, M tunable transmitters, P fixedreceivers, and Q tunable receivers can be described as:

FTLTTM-FRPTRQ

if no control channel is used or as:

CC-FTLTTM-FRPTRQ

if the network is control channel based. The number of nodes in a network isdenoted as 0. As an example, a TT-FR0 network has one tunabletransmitter and 0 fixed receivers in each node.

In a packet switched network, the tuning latency is critical for the networkperformance. This issue was addressed in the discussion on tunable

49

components in Section 2.3. Also, fast locking clock recovery circuits must beused instead of the slow PLL based circuits used in point-to-point links.Several methods to recover the clock signal on one or a few bits have beenproposed [Banu and Dunlop 1992] [Jonsson and Moen 1994] [Cerisola et al.1995].

������ 5DQGRP�$FFHVV�3URWRFROVIn a network using a random-access protocol, all nodes compete for atransmission channel in a random (uncontrolled) way [Halsall 1995].Random-access protocols can be contrasted with protocols with dynamic orstatic pre-assigned transmission patterns. The original single-channelrandom-access protocol is the Aloha protocol, where transmission is donewith no regard to other nodes. If two messages from different nodes overlapin time, both are corrupted. Several protocols are more or less pureimprovements of the Aloha protocol, e.g., Slotted Aloha, CSMA (Carrier-Sense Multiple-Access), and CSMA/CD (CSMA with Collision Detection). Inslotted Aloha, all nodes are synchronized and transmissions are allowed tobe started only at the beginning of a time slot. In this way, the probability ofcollision is reduced. In CSMA, the transmission medium is sensed beforetransmission starts. If a carrier is sensed, the transmission is postponeduntil the current message has reached its end. If two nodes sense themedium at the same time and find it free, they both begin to send and acollision will take place. In this case, the medium is busy for the wholeduration of the corrupted transmissions. This is avoided in the CSMA/CDprotocol, where a collision can be detected during transmission. If a collisionis detected by two nodes, both nodes stop and wait for a random time beforetrying again, beginning with the carrier-sense mechanism. Common to therandom-access protocols is that they do not perform well at high traffic-loads owing to the increased probability of collision but experience lowlatency at light loads.

In multiple-channel networks, a random-access protocol can be used both onthe data channels and on the control channel if present. Random accessprotocols for control channel based networks are known as X/Y protocols,where X denotes the protocol used on the control channel while Y denotesthe protocol on the data channels. An example of an X/Y protocol is theSlotted Aloha/Aloha protocol described in [Habbab et al. 1987].

������ 7LPH�:DYHOHQJWK�$VVLJQPHQW�6FKHPHVSeveral algorithms have been proposed to solve different kinds of timewavelength assignment problems [Rouskas and Ammar 1995]. One specificproblem is to schedule a traffic matrix (containing the communicationdemands between every possible pair of nodes) on a limited number ofwavelength channels, minimizing the number of time slots needed [Pieris

50

and Sasaki 1994]. Other aspects may also be considered, for example,minimizing the number of tuning periods, that is, the periods in which thetuning of the transmitters and the receivers are not changed [Ganz and Gao1992] [Ganz and Gao 1992B].

���� 6LQJOH�+RS�:'0$�6WDU�1HWZRUNV�DQG�3URWRFROV

A number of single-hop broadcast-and-select WDMA star networks andprotocols for these kinds of networks have been proposed. They aresuggested for a wide range of applications but in general can all be used inLANs and distributed computing systems. Several of these networks aresummarized in Table 1, Table 2, Table 3, and Table 4. Table 1 lists networkexperiments, while Table 2 and Table 3 list MAC protocols for non-controlchannel based and control channel based networks, respectively. Althoughthe protocols listed are for packet-switched traffic, protocols for circuit-switched traffic have been proposed [Dono et al. 1990]. Some papers

Name/Reference

Tunabilityclassification

Number ofwavelengths

Channelbandwidth Organization/other info.

FOX[Arthurs et al. 1988]

TT-FR 2 1 Gbit/s Bellcore; two stars, one foreach direction

HYPASS[Kobrinski et al. 1988B]

TT-FR &FT-TR

8 1.2 Gbit/s Bellcore; two stars, one foreach direction

[Kaminow et al. 1988] FT-TR 2 45 Mbit/s AT&T Bell Lab.Lambdanet[Kobrinski et al. 1987]

FT-FR0 18 1.5 Gbit/s Bellcore

Rainbow-I[Janniello et al. 1992]

FT-TR 6 300 Mbit/s IBM T. J. Watson; circuitswitched traffic

7DEOH����([SHULPHQWDO�VLQJOH�KRS�:'0$�VWDU�QHWZRUNV��0�LV�WKH�QXPEHU�RIQRGHV�

Name/Reference

Tunabilityclassification

Number ofchannels

Protocol type Packetlength

Slotted Aloha[Dowd 1991]

TT-FR 1 Slotted Aloha withACK-control slots

Fixed

I-TDMA[Sivalingam et al. 1992]

TT-FR 1 Static TDMA Fixed

I-TDMA*[Bogineni et al. 1993]

TT-FR 1 Static TDMA Fixed

FatMAC[Sivalingam and Dowd 1995]

TT-FR 1 TDMA with controlphase for reserving

Variable(fixed units)

TD-TWDMA[Jonsson et al. 1996]

FT-TR 0 TDMA with controlphase for reserving

Fixed

7DEOH����1RQ�FRQWURO�FKDQQHO�EDVHG�SURWRFROV�IRU�VLQJOH�KRS�EURDGFDVW�DQG�VHOHFW�:'0$�QHWZRUNV��0�LV�WKH�QXPEHU�RI�QRGHV��DQG�1�LV�D�SRVLWLYH�LQWHJHUJUHDWHU�WKDQ�]HUR�

51

describing single-hop WDM star networks that do not fit into the scope ofTable 1, Table 2, or Table 3, are listed in Table 4. Several of these papersare application suggestions and early contributions of conceptual thoughts.

Of the network experiments in Table 1, Lambdanet [Kobrinski et al. 1987][Goodman et al. 1990] was the first to reach “industrial strength” [Green1996]. Other noteworthy experimental demonstrations include the two-starFOX network for shared memory multiprocessors [Arthurs et al. 1988] andthe circuit switched Rainbow network [Dono et al. 1990]. The experimentaldemonstrations of Rainbow-I were reported in [Janniello et al. 1992].

Name/Reference

Tunabilityclassification

Num. ofchan.

Control ch.protocol

Data ch.protocol

Packetlength

Aloha/Aloha[Habbab et al. 1987]

CC-TT-TR 1���� Aloha Aloha Fixed

Slotted Aloha/Aloha[Habbab et al. 1987]

CC-TT-TR 1���� Slotted Aloha Aloha Fixed

Aloha/CSMA[Habbab et al. 1987]

CC-TT-TR 1���� Aloha CSMA Fixed

CSMA/Aloha[Habbab et al. 1987]

CC-TT-TR 1���� CSMA Aloha Fixed

CSMA/N-Server Switch[Habbab et al. 1987]

CC-TT-TR 1���� CSMA N-ServerSwitch

Fixed

DT-WDMA[Chen et al. 1990]

CC-FT2-FRTR 0�+ 1 Uniform staticTDMA

Distributedalgorithm

Fixed

Improved Slotted Aloha/Aloha [Mehravari 1991]

CC-TT-TR 1���� Slotted Aloha Aloha Fixed

Slotted Aloha/N-ServerSwitch [Mehravari 1991]

CC-TT-TR 1���� Slotted Aloha N-ServerSwitch

Fixed

Slotted Aloha[Sudhakar et al. 1991]

CC-TT-TR 1���� Slotted Aloha Aloha Fixed

Reservation Aloha[Sudhakar et al. 1991]

CC-TT-TR 1���� Slotted Aloha Aloha Fixed

DAS[Chipalkatti et al. 1992]

CC-FT2-FRTR 0 + 1 Uniform staticTDMA

Distributedalgorithm

Fixed

Hybrid TDM[Chipalkatti et al. 1992]

CC-FTTT1-FR1+1 0 + 1 Uniform staticTDMA

Two differentprotocols

Fixed

TDMA-C [Bogineni andDowd 1992]

CC-TT-FRTR 0 + 1 Uniform staticTDMA

Distributedalgorithm

Var.

1-DT-WDMA[Humblet et al. 1993]

CC-FTTT-FRTR 20 slotted slotted Fixed

[Yan et al. 1996] CC-FTTT1-FR1+1 0 + 1 Token Distr. alg. Var.MultiS-Net [Jia andMukherjee 1996]

CC-TT-TR 1 + 1 Slottedrandom access

Distributedalgorithm

Fixed

7DEOH����&RQWURO�FKDQQHO�EDVHG�SURWRFROV�IRU�VLQJOH�KRS�EURDGFDVW�DQG�VHOHFW:'0$�QHWZRUNV��0�LV�WKH�QXPEHU�RI�QRGHV��DQG�1�LV�D�SRVLWLYH�LQWHJHUJUHDWHU�WKDQ�]HUR�

52

Of the protocols for non-control channel based networks (Table 2), severalare TDMA variants. For example, I-TDMA [Sivalingam et al. 1992] and I-TDMA* [Bogineni et al. 1993] are multiple-channel variants of thetraditional static TDMA. Random access protocols have also been proposedfor non-control channel based networks. One example described in [Dowd1991] is a variant of the Slotted Aloha protocol.

Numerous protocols for control channel based networks have been proposed(Table 3). One of the first papers, [Habbab et al. 1987], describing protocolsfor WDMA networks presents five random access protocols utilizing acontrol channel. Several variants and improvements of these protocols havelater been proposed, for example, those described in [Mehravari 1991].Several of the protocols for control channel based networks use uniformstatic TDMA on the control channel and some form of distributed algorithmto schedule access to the data channels. Some of the protocols supportvariable sized packets, for example, the TDMA-C protocol [Bogineni andDowd 1992]. Protocols with other noteworthy features include: protocolsutilizing multiple control channels, for example, 1-DT-WDMA [Humblet etal. 1993], where 1 is the number of nodes and control channels; tell-and-goprotocols (data are sent immediately after the control information, withoutwaiting for the control information from the own and/or other nodes toreturn), for example, the Aloha/Aloha protocol [Habbab et al. 1987]; andprotocols not requiring dedicated hardware for the control channel, forexample, MultiS-Net [Jia and Mukherjee 1996]. Protocols for single-hopnetworks are reviewed in [Mukherjee 1992].

Some of the networks and protocols listed in Table 1 to Table 4 aredescribed in Sections 2.6.1 - 2.6.6 for purposes of example. Sections 2.6.1 -2.6.2 are network oriented, while Sections 2.6.3 - 2.6.6 are protocol oriented.The protocol descriptions are given at a level trying to explain the basicfunction and important features. If nothing else is stated, a passive opticalstar is assumed as the physical topology when describing the protocols.However, the passive optical star can often be exchanged with anothermulti-access network without any large modifications of the protocol. The

Name/Reference

Tunabilityclassification Other features

Photonic knockout switch[Eng 1988]

FT-FR1 $SSOLFDWLRQ: centralized packet-switches.A separate electrical control network is used.

Broadcast SYMFONET[Westmore 1991]

FT-FR0 $SSOLFDWLRQ: shared memory multiprocessors.Synchronization issues and timing are discussed.

MCA[Wailes and Meyer 1991]

TT1-TR1 $SSOLFDWLRQ: massively parallel computers.

[Brackett 1991] FT-TR $SSOLFDWLRQ� centralized packet-switches.

7DEOH����2WKHU�SURSRVHG�VLQJOH�KRS�:'0�VWDU�QHWZRUNV��0�LV�WKH�QXPEHU�RIQRGHV��DQG�1�LV�D�SRVLWLYH�LQWHJHU�JUHDWHU�WKDQ�]HUR�

53

focus on single-hop broadcast-and-select networks is motivated by thesimple architecture needed to support flexible traffic between end-nodes.More WDM networks and protocols have been reported, e.g., those describedin [Jia et al. 1995] [Levine and Akyildiz 1995].

������ )2;The FOX network reported in [Arthurs et al. 1988] is an interconnectionarchitecture for parallel computers with shared memory. Two stars areused, one for communication from the processing elements and to thememory modules and one for the opposite direction. Both the star networksare TT-FR networks with a unique wavelength for each receiver. If twoprocessing elements transmit to the same memory module at the same time,a collision will occur. With the motivation of a low cache miss rate, collisiondetection and retransmission are proposed to be a feasible solution to thisproblem. Experiments with two wavelengths and less than 20 ns tuninglatency were reported.

A related network is the HYPASS switching system [Kobrinski et al. 1988B][Goodman et al. 1988]. HYPASS also employs one star for each direction,but internally in a switch. One star is used for the data transport, while theother passes back control information from the output stages.

������ /DPEGDQHWThe Lambdanet WDMA star network presented in [Kobrinski et al. 1987] isclassified as FT-FR0, where 0 is the number of nodes. Although Lambdanetwas designed to have one specific wavelength per node as a transmitterhome channel, two extra wavelengths were used in the 16-node experiment,i.e., a total number of 18 wavelengths. Commercial DFB laser diodes wereselected to obtain a channel spacing of 2 nm, while a grating wavelengthdemultiplexer was used in the receiver to select one of the incomingwavelengths. Further experiments was reported in [Goodman et al. 1990],where the channel bit-rate was increased from 1.5 Gbit/s to 2 Gbit/s.

������ '7�:'0$The DT-WDMA (Dynamic Time-Wavelength Division Multi Access) protocolfor CC-FT2-FRTR networks, described in [Chen et al. 1990], divides theaccess to both control channel and data channels into slots of equal size.However, slots on the control channel are further divided into mini-slots,one to each transmitter. When node L wants to transmit to node M, it waitsfor its next mini-slot on the control channel. An address field in the controlpacket is set with the address of node M. Another field is set with a delayvalue related to the generation time of the message. The data message isthen sent in the data slot succeeding the control slot. After each slot when

54

all mini-slots are received, a deterministic distributed algorithm (separatelycomputed in each node with the same outcome) is run in each node. On thebasis of the distributed algorithm, node M will choose to tune in the nodewith the largest delay value of all the nodes that have the address of node Min the address field. All other messages destined to node M will get lost. In apipeline fashion, several messages can be sent before success or not of thefirst message is known.

The DT-WDMA protocol tries to minimize the maximum packet delay. Italso tries to cope with large propagation delays by using the pipelinemechanism. In addition to packet-switched traffic, circuit-switched traffic isalso supported by DT-WDMA.

������ $�'LVWULEXWHG�$GDSWLYH�3URWRFROIn [Yan et al. 1996] and [Yan et al. 1996B], a protocol, not named in thepapers, supporting soft real-time traffic is described. The QoS (Quality ofService) associated with a real-time packet in the network is the probabilityof missing the deadline. The distributed algorithm used tries to globallyminimize the number of packets not managing the QoS by adaptivelychanging the priority of the queued packets. The network architecture isCC-FTTT1-FR1 + 1, where the fixed transmitter and one of the fixedreceivers are dedicated to the control channel. The 1 tunable transmittersand 1 of the fixed receivers are dedicated to the 1 data channels.

Access to the control channel is based on a circulating token. The token isalways broadcasted to every other node so that they can take part of thestatus information about the sending node that is contained in the token. Ineach node, the latest version of this information from every other node isstored in a table. In this way, the current system state is known by allnodes, and an identical copy of the distributed algorithm can be run on eachnode. When a node has the token, it determines which data channels toacquire or release, in a way trying to globally meet the QoS demands. Thedetermination is based on the current system state and on informationabout the own queued messages, e.g., deadline and acceptable deadlinemissing fraction. The node to which the token is passed is also determinedaccording to the QoS demands of all messages in the network.

Even though the protocol has sophisticated methods for real-time messages,it is targeted only for soft real-time traffic. Guarantees can not be given thata message will meet its deadline.

������ ,QWHUOHDYHG�7'0$The I-TDMA (Interleaved TDMA) protocol described in [Sivalingam et al.1992] is an extension of the traditional static uniform TDMA protocol. The

55

protocol assumes a non-control channel based network with tunabletransmitters and fixed receivers. The access to each channel is divided into0 slots, where 0 is the number of nodes. The assignment is fully static andgives each node one slot per cycle and channel in which to transmit. Eachnode has access to a maximum of one channel at a time. If there are 0channels in the network, each node always has access to exactly onechannel. An extension of I-TDMA, called I-TDMA*, is described in [Bogineniet al. 1993]. The only difference is that I-TDMA* has C queues, where C isthe number of channels, for outgoing messages in each node instead of onequeue. Head-of-line problems are hence avoided. The head-of-line problemmeans that a node has a packet to send but can not reach it in the queuebecause there are other packets (whose destination it is not possible totransmit to for the moment) in front of it. As with traditional TDMA, theseprotocols reach high bandwidth utilization but have relatively largelatencies at low traffic intensities.

������ )DW0$&The FatMAC protocol is proposed for use in distributed shared memorymultiprocessors [Sivalingam 1994] [Sivalingam and Dowd 1995]. No controlchannel is used, and the network is classified as TT-FT. By choosing a laserdiode array as the tunable transmitter, broadcast is made possible throughsimultaneous activation of all laser diodes in the array. The access to thechannels is divided into cycles of variable length. The cycles have two parts,a control phase followed by a data phase. Each node has a slot withbroadcast capability in the control phase, where it transmits itstransmission demands for the cycle. The packet is scheduled in the dataphase among other demanded transmissions, following the same order asthe control slots. The length of the data phase depends on the number ofdemanded transmissions. Positive features of the network include: supportfor variable length packets, no need for control channel, and collisionlesstransmission.

A related protocol is TD-TWDMA (see Chapter 7), which also has a controlphase and a data phase instead of a separate control channel [Jonsson et al.1996] [Jonsson et al. 1997]. The TD-TWDMA protocol is developed fordistributed real-time systems, however, and has features for those systems.

���� &RQFOXVLRQV

This chapter has presented an introduction to high-performance fiber-opticnetworks. The emphasis has been on multiple-channel passive opticalnetworks, especially WDMA networks, because these networks have thehighest potential for meeting future bandwidth demands at a reasonablecost. Networks for numerous applications have been proposed in the

56

literature, but most of the networks and protocols referred to in this chaptercan be used in distributed computing systems too. WDMA star networks areforeseen to have a key role in future high-performance computercommunication networks, and examples of such networks and protocols forthem have been described. Many of the components demonstrated forWDMA networks, as reviewed in this paper, further indicate that high-performance WDMA networks for end-user systems will be available in thenear future.

57

��� ,QWHUFRQQHFWLRQV�LQ�3DUDOOHO�&RPSXWHUV

Interconnection networks are often divided into static (also called direct)and dynamic (also called indirect) networks. Static networks have a definedstatic topology, where the nodes are directly connected to nearest neighborsvia point-to-point links. This forms a static topology of the network, e.g., atwo-dimensional mesh. In dynamic networks, the traffic is routed through aswitched-based network. Therefore, we can say that the nodes areindirectly connected to each other.

Not all networks fall into one of the two categories given above. Thus twomore categories can be added: shared-medium networks and hybridnetworks [Duato et al. 1997]. After a presentation of different parameters ofinterconnection networks in Section 3.1, static, dynamic, shared-medium,and hybrid networks are described in Sections 3.2, 3.3, 3.4, and 3.5,respectively. Section 3.6 then presents different kinds of groupcommunication. A discussion of routing is given in Section 3.7, and thechapter ends with an overview of different high-performance networks forcoarse grained systems in Section 3.8.

����'HVLJQ�DQG�3HUIRUPDQFH�3DUDPHWHUV

The terminology used in the litterature on interconnection networks canvary a bit. This chapter states definitions of terms for which some are usedin the thesis, while others are included just to give a hint of whichparameters that can characterize interconnection networks. Various designand performance parameters and desirable features of interconnectionnetworks for parallel computers are found in textbooks covering the area ofparallel computing [Almasi and Gottlieb 1994] [Casavant et al. 1996][Decegama 1989] [Hockney and Jesshope 1988] [Hwang 1993] [Hwang andBriggs 1985] [Lawson et. al. 1992], compilations of papers [Scherson andYoussef 1994] [Varma and Raghavendra 1994], and tutorial texts oninterconnection networks for parallel computers [Bhuyan et al. 1989] [Duatoet al. 1997] [Reed and Grunwald 1987] [Siegel 1990]. Furthermore, moregeneral network discussions and concept definitions are found in computercommunication textbooks [Halsall 1995] [Peterson and Davie 1996][Stallings 1997] [Tanenbaum 1996]. We will explain some of these below, ofwhich several have an influence on the analysis.

Fault tolerance Redundant paths in the network can bring faulttolerance.

58

Latency and delayLatency and delay are terms that can be defined in anumber of ways. One common definition, normallycalled message delay, is the time from messagegeneration in the source node until the message isfully received at the destination node and availablefor use by the application running on it.

Transmission capacityTransmission capacity is measured in bit/s or Byte/sand denotes the maximum amount of data that canbe transferred per time unit over a link oraggregately over a whole interconnection network.Transmission capacity is often, but somewhatmisleadingly [Freeman 1998], called bandwidth.

Throughput The term throughput is used when describingefficiently used transmission capacity at differentassumptions such as certain traffic patterns andmessage generation rates. It is usually measured inbit/s or number of delivered messages per time unit.In the ideal parallel computer system, a PE(Processing Element) can have a sustainedthroughput as high as the peak throughput,independent of the traffic pattern from other PEs.

Bisection bandwidthThe bisection of a system is the section that dividesthe system into two halves with an equal numbers ofnodes. The bisection bandwidth is the aggregatedbandwidth over the links that cross the bisection. Inasymmetric systems, the number of links across thebisection depends on where the bisection is drawn.However, since the bisection bandwidth is a worst-case metric, the bisection leading to the smallestbisection bandwidth should be chosen [Hennessy andPatterson 1996].

Cost An interconnection network designer always wishesto bring the cost down as far as possible. Instead ofmeasuring cost in monetary terms, it can bemeasured in, e.g., number of links or switches.

Conflict free The ideal network should be conflict-free. If thereexists a physical connection between each pair ofnodes (or a full crossbar is used), conflicts in the

59

network are avoided. Conflicts caused by, e.g., alimited amount of buffers must still be considered,however. Since 12 connections (or cross points in acrossbar) are needed, a conflict-free network will betoo expensive in larger systems.

Uniformity Sometimes it is desirable to have uniform latencyand bandwidth, independent of which two nodes thatcommunicate with each other. However, parallelcomputers with a non-uniform network can scale to alarge number of PEs for applications with a highdegree of local communication [Agarwal 1991].

Circuit switching vs. packet switchingWhen using circuit switching, a "physical" channel isallocated before the communication between a pair ofPEs can start. The "physical" channel does not needto be a purely physical but can, e.g., be a cyclicallyavailable time slot on a time multiplexed channel.The advantage of circuit switching is the guaranteedbandwidth, while disadvantages are long setup timesand low bandwidth utilization when the channel isidle for a long time, since the bandwidth normallycannot be reused. When using packet switching, thedata to be transferred are split into packets thatcompete for bandwidth with packets from othernodes. Traffic situations with temporary bursts oflarge volumes of data from one or a few PEs cantherefore be handled better in a packet-switchednetwork than in the case that much of the bandwidthis allocated by other nodes using circuit switching.Also, sporadic traffic often experiences a shorterlatency than in the case that a circuit must be setupeach time. Handling real-time traffic is, however,harder in a packet-switched network.

Connectionless vs. connection-oriented serviceWhen using connection oriented service, the twocommunicating parties first agree upon establishinga logical "error-free" connection. After the phase ofdata transfer, the logical connection is disconnected.During data transfer, each correctly received packet(or similar) is acknowledged to the transmittingparty. If it is not acknowledged, or retransmission isexplicitly requested, a packet is retransmitted. Also,to guarantee that packets arrive in the correct

60

(transmitted) order, each packet is stamped with asequence number. Connectionless service does notinvolve connection establishment and is not an error-free service. Because the protocol entity in adestination PE is not aware of forthcoming packets,it cannot ask for retransmission of packets that arecorrupted and discarded before arriving to it.

Scalability Sometimes, scalability refers only to scalability inone dimension, e.g., size. In this way, VL]H�VFDODELOLW\can denote the ability to build arbitrarily largesystems with a chosen (network) architecture andincrease the number of nodes with no or smallmodifications of the architecture. Just sayingscalability often denotes that performance (e.g.,bandwidth) scales with the number of nodes. As anexample, when more PEs are added to a scalablesystem, performance in terms of, e.g., networkcapacity should, in order to to avoid bottlenecks,increase proportionally with the number of PEs.

Incremental expandabilityIt is often desirable to be able to expand a systemwith another small subsystem instead of, e.g., beingforced to double the number of nodes to maintain acertain topology. It can also be the case that aparallel computer system is optimized for a certainsize which can lead to, e.g., unused communicationlinks at smaller system sizes. The term PRGXODULW\ issometimes used instead of, or at least related to, theterm incremental expandability.

Control strategy The control strategy can be either centralized (global)or decentralized (local) and normally refers to theway in which switches are steered in the network. Ina network with centralized control, a single controllersteers the whole network. This might imply that allswitches in a network are set at the same time tooptimize for a certain global communication pattern.In a network with decentralized control, each switchdecides on its own how to route incoming messages.

61

���� 6WDWLF�1HWZRUNV

A certain static topology is chosen in static networks. Some commontopologies are linear array, ring, 2-dimensional mesh, 2-dimensional torus,and binary hypercube (Figure 3). The parameters below are used to describestatic networks.

Diameter For each possible pair of nodes in the network, thereexists a shortest path. The diameter is defined as the

)LJXUH����6RPH�VWDWLF�WRSRORJLHV���D��OLQHDU�DUUD\���E��ULQJ���F����GLPHQVLRQDOPHVK���G����GLPHQVLRQDO�WRUXV���H����GLPHQVLRQDO�ELQDU\�K\SHUFXEH��DQG��H���GLPHQVLRQDO�ELQDU\�K\SHUFXEH�

(c) (d)

(f)(e)

(b)(a)

62

number of hops over the longest of these shortestpaths.

Node degree The node degree, the number of links that connect anode to its nearest neighbors, can either be constantfor the whole network or differ between the nodes. Asan example, the boundary nodes in a two-dimensional mesh have a node degree of 3 and cornernodes a node degree of 2, while the rest of the nodeshave a node degree of 4. A constant node degree is anexample of a feature that might make expansions ofthe system easier.

If we let 1 denote the number of nodes or PEs in a system and assume bi-directional links, we can, with these parameters, characterize the differenttopologies in the figure. The linear array has a node degree of two, exceptfor the end nodes, and a diameter of 1 − 1. The linear array topology isemployed in, e.g., REMAP-β [Bengtsson et al. 1993].

The ring has a constant node degree of 2, while the diameter is 1 / 2. It ischeap to expand a ring network but, since the diameter increases with 1,only small systems are practical. An example of a computer using a ringnetwork is KSR-1 [Almasi and Gottlieb 1994]. For larger systems, however,a hierarchy of rings must be used. Multiple access protocols for LAN typering and multiple ring networks are found in [Bhuyan et al. 1989B].

The node degree of the two-dimensional mesh is discussed above, while thediameter is 2(√N − 1). If a mesh network is extended with wrap-aroundconnections, it is called a torus. A torus has lower diameter than the mesh,√N for the two-dimensional case, and a uniform node degree. Examples ofparallel computers with a two-dimensional mesh interconnection networkare the distributed shared memory multiprocessor from MIT, the MITAlewife [Agarwal et al. 1995], and the MPP SIMD computer (with theextension of reconfigurable function of the boundary nodes) [Batcher 1980][Batcher 1980B]. A three-dimensional mesh network is used in the J-machine at MIT [Dally et al. 1993].

The MasPar MP-1 [Blank 1990] [Nickolls 1992], MP-2 [MasPar 1992], andthe embedded version by Litton/MasPar [Smeyne and Nickolls 1995] has atwo-dimensional torus network where each node is connected to its eightnearest neighbors by the use of shared X connections. Moreover, the FujitsuAP3000 distributed memory multicomputer has a two-dimensional torusnetwork [Ishihata et al. 1997], while the CRAY T3D [Kessler andSchwarzmeier 1993] [Koeninger et al. 1994] and T3E both use the three-dimensional torus topology.

63

A binary hypercube has two nodes along the side in each dimension, i.e., atotal of 2N nodes, where N is the dimension. The diameter is k since themaximum distance to be travelled is one hop in each dimension. Theconstant node degree of a binary hypercube is also N. Examples of computerswith hypercube interconnection networks are the Cosmic Cube with a 6-dimensional hypercube [Seitz 1985] and the Connection Machine CM-2 witha 12-dimensional hypercube, where each node in the hypercube consists of16 PEs [Thinking Machines 1991]. Replacement of 1024 wires in the CM-2by two optical fibers has been demonstrated [Lane et al. 1989].

More references and a discussion on static networks are found in[Stojmenovic 1996].

����'\QDPLF�1HWZRUNV

The crossbar is the most flexible dynamic network and can be comparedwith a fully connected topology, i.e., point-to-point connections between allpossible combinations of two nodes. The drawback, however, is the increaseby 1� in cost/complexity of the switch, where 1 is the number of nodes.Systems with a single true crossbar are therefore limited to small systems.Starfire from Sun Microsystems is a symmetric multiprocessor system witha 16 x 16 crossbar network for data transactions [Charlesworth 1998] (forthe snoopy cache-coherence protocol, a bus is also used in the system). Othercomputers with a crossbar network include the VPP500 [Miura et al. 1993]and VPP700 [Uchida 1997] from Fujitsu.

In multistage shuffle-exchange networks, the cost function is reduced 1 log2

1, but where log2 1 stages must be traversed to reach the destination. Anexample of such a network is the Omega network (Figure 4) which providesexactly one path from every input to every output. The four different switchfunctions of the 2 × 2 switch that is used as building block are shown inFigure 5, where the two rightmost configurations are used for broadcast.Switches larger than 2 × 2 can also be used. Each stage of the switches in anOmega network is preceded by a perfect-shuffle pattern. In contrast to acrossbar network, which is a QRQEORFNLQJ network, an Omega network is aEORFNLQJ network. This means that there might not always exist a paththrough the network as a result of already existing paths that block theway. An Omega network is used in the NYU Ultracomputer which is aglobally shared memory multiprocessor [Gottlieb et al. 1982] [Gottlieb et al.1983]. The network connects 1 processing elements to 1 memory modules.The Cedar system is another globally shared memory multiprocessor wherea multistage shuffle-exchange network connects processors to memorymodules [Kuck et al. 1993].

64

5HDUUDQJHDEOH networks is a another category of multistage networks whereit is always possible to find a path through the network. However, if not allpaths are routed at the same time, it may be necessary to reroute alreadyexisting paths. An example of a rearrangeable network is the Benesnetwork shown in Figure 6. Other multistage networks include Banyannetworks [Goke and Lipovski 1973].

A bi-directional multistage network [Stunkel et al. 1994] [Stunkel et al.1995] [Sethu et al. 1998] is used in the IBM SP2 [Agerwala et al. 1995].Switchboards each having multiple 8 × 8 switches, coupled as 4 × 4 bi-directional switches, are used as building blocks.

A tree network can be seen as a static network topology. However, as longas there are nodes only at the leaves, the tree of switches can be seen as aself-standing system connecting the nodes indirectly with each other. Theroot nodes in a tree network easily become a bottleneck. This problem issolved in the scalable fat-tree network where links closer to the root havehigher bandwidth [Leiserson 1985]. An example of a (not fully scalable)switched fat-tree network built of 6 × 6 switches is shown in Figure 7. Eachlink in the figure is bi-directional. The CM-5 is an example of a parallel

Transmitter side Receiver side

)LJXUH� ��� 2PHJD� QHWZRUN� IRU� DQ� HLJKW�QRGH� V\VWHP�� 2QH� SDWK� WKURXJK� WKHQHWZRUN�LV�KLJKOLJKWHG�

)LJXUH����3RVVLEOH�VWDWHV�RI�D������VZLWFK�

65

computer with a fat-tree network [Hillis and Tucker 1993] [Leiserson et al.1992]. The fat-tree topology is also used in the Meiko CS-2 [Beecroft et al.1994].

���� 6KDUHG�0HGLXP�1HWZRUNV

A common way of implementing a shared-medium network is to use the bustopology, but it can also be, e.g., a ring where only one node is allowed tosend at a time. The great advantage of a shared-medium network is theeasy implementation of broadcast, which is useful in many situations. Thedisadvantage is that the bandwidth does not scale at all with the number ofnodes. The bus is commonly used in small systems, e.g., in Silicon GraphicsPower Challenge [Silicon 1994]. Multiple buses can be used to enhanceperformance relative to single-bus systems [Mudge et al. 1987].

)LJXUH����$�UHDUUDQJHEOH�%HQHV�QHWZRUN�

)LJXUH����)DW�WUHH�RI�VZLWFKHV�ZKHUH�QRGHV�DUH�OHDYHV�LQ�WKH�WUHH�

66

����+\EULG�1HWZRUNV

An example of a hybrid network is the hierarchical network in the StanfordUniversity DASH [Lenoski et al. 1992] [Lenoski et al. 1992B] [Lenoski et al.1993]. A two-dimensional mesh network connects bus-based clusters. Theaim of the configuration is to get a scalable cache-coherent shared memorymultiprocessor. The cache-coherence protocols used are snoopy-on-the-businside each cluster and a distributed directory-based protocol between theclusters. The Paradigm instead uses a hierarchical bus network in its cache-hierarchy implementation [Cheriton et al. 1991]. The CRAY APP in turngroups the processing elements in groups of up to 12 processing elementsconnected to a common bus. Up to seven such buses of processing elementscan be connected, via a crossbar, to a globally shared memory [Carlile 1993].

����*URXS�&RPPXQLFDWLRQ

Many parallel programs can take advantage of special support for groupcommunication, or collective communication, i.e., communication wheremany nodes (or processes) are collectively involved. The nodes involved in agroup communication operation are said to be members of a group. Somekinds of group communication are:

• 0XOWLFDVW� One-to-many communication where one node sends thesame message to all members of the group. The special case in whichall nodes in the system are members of the group is called EURDGFDVW.

• 6FDWWHU� One-to-many communication where one node sends differentmessages to different members of the group.

• 5HGXFWLRQ� �JOREDO� FRPELQLQJ�� Many-to-one communication wheredifferent messages from different members of the group are combinedinto one message for delivery to one destination node. Some commonoperators used when combining are SUM, OR, and AND.

• *DWKHU� Many-to-one communication where different messages fromdifferent members of the group are concatenated in a defined orderfor delivery to one destination node.

• 5HGXFH� DQG� VSUHDG� Variant of the reduction operator, where theresult is spread to all group members.

• %DUULHU� V\QFKURQL]DWLRQ� A synchronization point is defined in theprogram code, at which all members must arrive before any of themembers may continue beyond the synchronization point. This is aspecial case of "reduce and spread" where no data is involved.

• 6FDQ� For each member of the group, PL, where 1 ≤ L ≤ 0 and 0denotes the number of nodes, a reduction is made where the node is

67

chosen to be the destination node. Each such reduction is made froma sub-group of 1 ≤ 0 nodes, e.g., nodes PM, where L − 1 + 1 ≤ M ≤ L.

The possibility to define groups and call group communication routines issupported in, e.g., PVM (Parallel Virtual Machine) [Geist et al. 1994]. Theunderlying group communication mechanisms can be implemented inseveral different ways: (L) in software using the same network as forordinary traffic, (LL) by the use of a more or less general network dedicatedfor group communication as in the CM-5 [Leiserson et al. 1992], and (LLL) bydedicated hardware specialized for, e.g., barrier synchronization [O’Keefeand Dietz 1990] [O’Keefe and Dietz 1990B].

����5RXWLQJ

The routing decision, or path selection, in a packet-switched interconnectionnetwork can be made either inside the network by routers or by the endnodes, so called source routing. Source routing can be more easily used inparallel computers than in, e.g., internet communication [Comer 1995],because the topology does not change as often and is not normally socomplex and/or irregular. Source routing is used in, e.g., Myrinet [Boden etal. 1995] and the IBM SP2 communication system [Stunkel et al. 1995],where the header of a packet includes the desired switch-setting of eachrouter on the way from source to destination. Each router drops itscorresponding switch-setting field in the header when the packet isforwarded.

If each router must make routing decisions based on a single destinationaddress in the header of a packet, this can be done in two ways. First, VWDWLFrouting tables can be loaded into the routers and only rarely changed. Thesecond method is more sophisticated and involves DGDSWLYH routing.Whenever a packet arrives at a router, the best next-hop is adaptivelychosen based on, e.g., congestion statistics. Adaptive routing can betterutilize the resources in the network as compared with static routing andsource routing, even though some support for different paths to choosebetween can be incorporated into source routing systems.

When a packet arrives at a router, it can be stored and error checked beforean attempt is made to forward it. This method is called store-and-forwardand may be simple to implement but requires some buffer memory and addssignificant latency for each router that is passed on the path from source todestination. The alternative is to use cut-through switching, i.e., only theheader of the packet with the destination address must arrive before thepacket can be forwarded to an output port [Kermani and Kleinrock 1979].This means that it is not necessary to store the whole packet and it will onlyexperience a low latency because the router begins to forward the packet

68

before it is fully received. If there is no suitable free output port, the rest ofthe packet can be received and stored as in store-and-forward.

Wormhole routing is a kind of cut-through where it is not needed to storethe whole packet in a router if an output port is busy [Dally and Seitz 1987].Instead, a flow control signal is propagated back to the transmitter to orderit, and intermediate routers, to stop sending. The transmission of the packetis resumed when the busy port becomes free. Methods to avoid deadlocks[Pinkston and Warnakulasuriya 1997] in wormhole routing networks (twoor more "worms" block each other) and similar networks have been proposed[Ni and McKinley 1993], e.g., dividing physical channels into multiplevirtual channels, each with dedicated buffers in the routers [Dally and Seitz1987]. Virtual channels can also be used, among other things, to decreaseblocking in a network and to guarantee bandwidth to virtual circuits [Dally1992].

����+LJK�3HUIRUPDQFH�1HWZRUNV�IRU�&RDUVH�*UDLQHG3DUDOOHO�&RPSXWHUV

There are several different general high-performance networks that can beused to connect rather powerful and possibly heterogeneous computingnodes, which might be physically separated by several tens of meters ormore. Both standards and ongoing research projects exist. Often, thesekinds of networks are used in NOWs (Networks of Workstations) or COWs(Clusters of Workstations) [Anderson et al. 1995]. Another possibility is tohave a heterogeneous system of both workstations and supercomputers.

Myrinet is a switch based solution with support for arbitrary topologies[Boden et al. 1995]. In the current version (November 1998), full-duplex 1.28+ 1.28 Gbit/s links connect switches and nodes in the selected topology.Electrical signals on the cables carry data, control information, and flowcontrol for the reversed direction. Host interfaces for PCI and SBus areavailable, while switches with four, eight, 12, and 16 ports exist. Myrinet isbased on earlier work performed at Caltech [Seitz and Su 1993] and theUniversity of Southern California [Felderman et al. 1994]. A parallelcomputer architecture with a hierarchy of Myrinet switches is reported in[Boggess and Shirley 1997]. The lowest level in the hierarchy has a switchthat connects several processors on a single board. The same switch is aninterface to the next level, which connects several boards in a backplane.Other work related to Myrinet has been reported [Prylli and Tourancheau1998] and includes a multicast protocol where the Myrinet network-interfaces forward multicast packets along a spanning tree [Bhoedjang et al.1998]. More references to reports on communication systems in whichMyrinet is used can be found [Bhoedjang et al. 1998B].

69

An interconnection system similar to Myrinet, but especially developed forembedded systems, is RACEway from Mercury Computer Systems[Kuszmaul 1995] [Einstein 1996] [Isenstein 1994] [Mercury 1998]. ARACEway system is built up of six-port (bi-directional) crossbar switches toget an active backplane. Several different topologies can be chosen, but thetypical one is a fat-tree of switches, where each switch has four children andtwo parents. Circuit switching with source routing are used. Support forreal-time traffic is obtained by using priorities, where a higher prioritytransmission preempts a lower priority transmission. The link bandwidth is160 MByte/s.

Other high performance networks for rather coarse grained systems include:

• 1HFWDU� switched-based source-routing network [Arnould et al. 1989][Steenkiste 1996]

• )LEUH�&KDQQHO��standardized network supporting different link speedsand topologies, e.g., switch-based [Anderson and Cornelius 1992][Boisseau et al. 1994] [Emerson 1995] [Sachs and Varma 1996][Saunders 1996]

• +,33,� standardized network where switches can be used to switchpoint-to-point links, each with 800 Mbit/s or 1.6 Gbit/s, simplex orduplex [Saunders 1996] [Tolmie and Renwick 1993]

• 71HW� switch based wormhole routing network [Horst 1995]

• 6&,� standardized network supporting cache coherence in differenttopologies of the network, e.g., ring or switch based [Gustavson andLi 1996] [IEEE 1993]. Used in, e.g., a system from Sequent [Lovettand Clapp 1996]

• 6SLGHU� short-distance (few meters) switch based network with 2 × 1GByte/s full duplex links [Galles 1997], used in SGI’s Origincomputer systems [Laudon and Lenoski 1997]

• +$/V�0HUFXU\�,QWHUFRQQHFW�$UFKLWHFWXUH� network based on crossbarswith six 1.6 + 1.6 Gbyte/s full duplex ports [Weber et al. 1997]

• *LJD5LQJ� a 600 + 600 MByte/s dual ring network developed by CrayResearch for use as a supercomputer interconnect [Scott 1996].

Experiments with ATM networks in parallel and distributed computingsystems have also been reported [Eicken et al. 1995].

The category of interconnection networks for rather coarse-grained systems(typically multicomputers) are especially in focus in this thesis.

71

3DUW�,,��5HYLHZ�RI�5DGDU�6LJQDO�3URFHVVLQJ6\VWHPV�DQG�6XLWDEOH�,QWHUFRQQHFWLRQ

1HWZRUNV

73

���$�6DPSOH�5DGDU�6LJQDO�3URFHVVLQJ�6\VWHP

The signal processing system under consideration is primarily developed foruse in applications with a phased array antenna, i.e., an antenna withmultiple fixed antenna elements (and digital beam forming) instead of amoveable antenna. The system has a number of different requirementsdepending on the application, but the algorithms are usually well known.They comprise mainly linear operations such as matrix-by-vectormultiplication, matrix inversion, FIR-filtering, DFT etc. In general, thefunctions will work on relatively short vectors and small matrices, but at afast pace and with large sets of vectors and matrices.

One of the goals of our research is to find a good scalable architecture whichcan give sufficiently high computing speed for this application without toogreat a loss in generality, i.e., the use of efficient programmable computersis preferred. A solution to this is to use the two-dimensional array SIMDmachine as a building block. The two-dimensional array is well known inthe literature, and a number of machines have been built, e.g., MasPar[Blank 1990] [Nickolls 1992] [MasPar 1992], Connection Machine [ThinkingMachines 1991], and DAP [Hord 1990]. Numerous successful mappings ofalgorithms on these machines have been done. However, these machines arenot efficient when the calculations are too small for the machine size, i.e., itis difficult to fit a 32 by 32 matrix problem on a 65k processor machine.Thus, it can be noted that the mesh is a promising architecture, but that thesize of the mesh should be in the order of the size of the data structures inthe calculations, i.e., the size of the matrices in the data set. To cope withthe large number of matrices in the data, many computation modules, eachwith the two-dimensional array topology, can be interconnected to share theload. A very powerful interconnection network is needed for this, becauseeach computation module (hereafter also called node) can produce asustained data flow of several Gbit/s. Guaranteed bandwidth must also besupported in order not to disturb the dataflow.

This chapter briefly describes our proposal for a computer system, which isa MSIMD mesh system, intended to meet the imposed requirements interms of computing power, generality, size, and power consumption. Thecomputation module architecture is presented in Section 4.1, and systemdesign issues are discussed in Section 4.2. The communication demands aretreated in Section 4.3. More detailed descriptions of the radar signalprocessing systems developed are found in other publications by ourresearch group [Taveniku et al. 1996] [Taveniku et al. 1998] [Ahlander1996] [Taveniku 1997].

74

���� &RPSXWDWLRQ�0RGXOH�'HVLJQ

Several similar computation module designs have been developed andevaluated in the scope of the projects of which this thesis has been part. Oneis the computation module shown in a simplified form in Figure 8 [Tavenikuet al. 1996]. Each module consists of eight 8-by-8 meshes working in anSIMD fashion. Internally in the meshes, the processors (PEs) are connectedwith nearest neighbor connections (x-grid), together with row and columnbroadcast lines. Computations and inter-PE communication can beperformed simultaneously. In addition to the PE meshes, the module holdsI/O buffers, memory, and control units.

���� 6\VWHP�'HVLJQ�,VVXHV

The overall system design is a scalable, moderately parallel MIMD systemwith moderately parallel SIMD modules interconnected with a scalable,optical real-time interconnection network. The applications areimplemented on the system as follows. A multi-mode radar application canin general be described as a set of independent modes of operation.Furthermore, each mode is described as a series of transformations on thesampled data stream. In addition to this, there are control functions andregistration functions controlling and monitoring the system.

)LJXUH����&RPSXWDWLRQ�PRGXOH�ZLWK�HLJKW������PHVKHV�RI�3(V�

75

The system implementation process is illustrated in Figure 9. First, themodes of operation can be designed separately. The modes are thenfunctionally decomposed and laid out on the system (MIMD level). In thisstage, only the computational demands and the network bandwidthdemands need to be considered. When the functions are assigned toprocessing nodes, these can be further mapped onto the processors, andbuffer space can be allocated. This task is simplified by the fact that only alimited number of algorithms are used, therefore making it possible to useand develop a function library. Furthermore, owing to the properties of theapplication, data are inherently parallel and therefore easy to map ontothese small processor arrays. The chosen approach gives: (i) intuitive use ofthe MIMD system, (ii) understandable, high level functional decomposition,(iii) scalability on the MIMD level, and (iv) efficient mapping of algorithmsbecause of small modules.

���� ,QWHU�0RGXOH�&RPPXQLFDWLRQ�'HPDQGV

As an example, we show how signal processing in a ground basedsurveillance radar system is implemented using the approach described inSection 4.2 [Taveniku et al. 1996]. In the system, 64 lobes are created thatuse 64 receiver channels. The data flow in one mode of operation can bedescribed as shown in Figure 10, while the total computational demand is40 GFLOPS. The nodes in this system are SIMD computers of the array-of-meshes type (see Section 4.1) with a sustained performance of 4 GFLOPS.The functions are mapped onto the nodes as shown, together with thebandwidth demands, in Figure 11. The algorithms are then individuallymapped onto the specific processor array, in this case an 8-by-8-by-8 meshPE array as shown in Figure 8. The chain will only figure as a sample

f0

f4

f3f2f1

CM4

CM2

CM3CM1

CM0

Control

mem

buff

f0 = FFT 4 * 8

p

p p

ppp

p

pppp

p

pppp

)LJXUH� ��� 7KH� 0,0'� V\VWHP� HQDEOHV� D� IXQFWLRQDO� GHFRPSRVLWLRQ� DQGDOORFDWLRQ�RI�WKH�SURJUDP�WR�WKH�PRGXOHV��(DFK�IXQFWLRQ�RU�EORFN�RI�IXQFWLRQVLV�WKHQ�PDSSHG�RQWR�WKH�SURFHVVRUV�LQ�WKH�FRPSXWDWLRQ�PRGXOH��&0��

76

system with different communication requirements. Thus no details aboutthe chain are covered here.

The figure shows how the work is split in a coarse grained MIMD fashion,where each computational module is itself powerful (containing multipleprocessors in this case). A data cube that initially comes from the antennacontains data in three dimensions (channel, pulse, and distance). After theprocessing of one stage, the new data cube is forwarded to the next node inthe chain. As a pipelined system, a module can start processing new data assoon as it has sent the results of the former calculation to the next node. Theaggregated throughput demand is about 45 Gbit/s, including the data fromthe antenna (Node 1) feeding the chain. As seen in the figure, the chaincontains both multicast, one-to-many, and many-to-one communicationpatterns.

Corner-turning of the data cube is done when the PEs must process dataalong another dimension of the data cube. The memory modules are used forthis task. A memory module stores incoming messages from thecommunication system in such a way that the whole corner-turned datacube is funally in memory. The memory modules may not be needed if thecomputational modules have enough memory to store a whole data cube.Also in this case, the communication system does a main part of the corner-turning.

A number of different working modes are possible in a radar system. Thetask of one mode can, for example, be to scan the whole working range,while the task of another mode can be to track a certain object. Normally,the algorithm mapping and communication patterns are different for twodifferent modes. The signal processing chain discussed above represents onemode.

)LJXUH�����'HVFULSWLRQ�RI� RQH�PRGH� RI� RSHUDWLRQ� LQ� WKH����FKDQQHO� JURXQGEDVHG�UDGDU�V\VWHP�

77

Although control I/O and high speed data I/O are logically separated in themodule, a single network interface handles all intermodule communication.Since each module itself can have a sustained output data rate of severalGbit/s, a powerful interconnection network is needed. Another importantfeature of the network is the ability to guarantee that the time constraintsare met, i.e., the data flow must not be disturbed by, for example, statusinformation that the network must also transport. A network that canguarantee delivery of semi-static high bandwidth traffic at the same timethat it carries rapidly changing control and status traffic is thereforevaluable, or even required. We suggest two different ways of implementingthe communication network, both employing fiber-optic technology andsupporting guaranteed timely delivery. The first network is a WDM starnetwork which scales to large systems, and the second is a fiber-ribbon ringnetwork suitable for systems of a moderate size and/or systems with highdegrees of nearest neighbor communication. These two main networkarchitectures, together with different design alternatives, are described inChapters 7 through 10.

BeamForming

6.0Gb/s

6.0Gb/s

3.0 Gb/sMulticast

4 x 750Mb/s

3.0Gb/s

ExtractorCFAR

= Computational Module

= Buffer Memory Module

EnvelopeDetection

DopplerFilter Bank

BufferMemory

2 x 3.0 Gb/s Pulse Compression

2 x 3.0Gb/s

BufferMemory

6.0Gb/s

6.0Gb/s2

7

3 54

8

10 11 12 13 14

96

)LJXUH�����'DWD�IORZ�EHWZHHQ�WKH�PRGXOHV�LQ�WKH�VLJQDO�SURFHVVLQJ�FKDLQ�

79

���&RQILJXUDWLRQV�DQG�5HTXLUHPHQWV�RI�6LJQDO3URFHVVLQJ�6\VWHPV

Future radar signal processing systems will have high computationaldemands, which implies that parallel computer systems are needed. In turn,the performance of parallel computers is highly dependent on theperformance of their interconnection networks. Next chapter, reviewsoptical interconnection networks from a signal processing perspective. Thenetworks are evaluated according to their suitability for mapping signalprocessing chains in, e.g., radar systems. Different ways to map areexplained. These kinds of mappings are simplified in their nature tofacilitate an evaluation of the diverse range of networks. More detaileddiscussions of algorithm mapping in similar signal processing systems arefound in [Liu and Prasanna 1998].

Sections 5.1 through 5.7, describe different kinds of parallel algorithmmapping and performance criteria for use in the analysis.

���� 3XUH�3LSHOLQH�&KDLQ

The simplest case considered is a single, straightforward pipeline chain.Such a system is shown in Figure 12, where each shaded box represents acomputational module and the cubes represent the data flow. Eachcomputational module runs one or several pipeline stages, and the arcsrepresent only the dataflow between modules. The physical topology can,e.g., be a ring or a crossbar switch. The chain is a purified form of thesample chain in Figure 11 (Page 77), without, e.g., multicasting to simplifythe discussion of performance.

Computationalmodule

)LJXUH�����$�SXUH�SLSHOLQH�FKDLQ�ZKHUH�RQH�RU�PRUH�VWDJHV�LQWKH�FKDLQ�DUH�PDSSHG�RQ�HDFK�PRGXOH�

80

���� 6DPH�3URJUDP�0XOWLSOH�'DWD

A common way of mapping radar signal processing algorithms onto parallelcomputers is to let each processor carry out the same operation but ondifferent sets of data, i.e., SPMD (Same Program Multiple Data). All thePEs in Figure 13 then work together on one of the pipeline stages in Figure11 (Page 77) at a time. After the processing of one stage, the data cube isredistributed if necessary. Since this is not a pure pipelined system, it mightbe harder to overlap communication and computation.

Although not considered here, there might occur communication during theprocessing of one stage, e.g., nearest neighbor communication, if a statictopology like the two-dimensional mesh is used. The number of times it isnecessary to redistribute the data cube between the PEs may however bereduced as compared with the number of times the data cube must betransferred to the next module for processing in a pipeline chain. Instead, itmight be harder to overlap communication and computation, which impliesextra bandwidth requirements [Teitelbaum 1998]. When corner turning thedata cube, one half of the data cube must be transferred across the networkbisection [Teitelbaum 1998]. The bisection bandwidth is therefore anessential performance parameter in this strategy.

Computationalmodule

)LJXUH�����,I�WKH�630'�PRGHO�LV�XVHG��DOO�3(V�ZRUN�WRJHWKHU�RQRQH�VWDJH�LQ�WKH�VLJQDO�SURFHVVLQJ�FKDLQ�DW�D�WLPH�

81

Incoming data from the antenna are not shown in Figure 13 but areassumed to be fed into the PEs in such a way that it does not affect thecommunication pattern shown in the figure. For example, special I/Ochannels can be used instead of the communication system carrying thetraffic indicated by the arcs in the figure.

���� 3DUDOOHO�+DUGZDUH�)RU�0XOWLSOH�0RGHV

When there are several concurrent operation modes of the radar instead ofonly one, these can be mapped in a number of different ways. One way is tohave several groups of processing elements, each living its own life anddedicated for one mode. Assuming one of the two kinds of process mappingdescribed in Sections 5.1 and 5.2, we have two ways of mapping concurrentmodes totally in parallel.

Figure 14 shows a system of parallel pipeline chains, chain with its owncomputational modules. The incoming data from the antenna are multicastto the first node in each chain, i.e., all modes operate on all data cubes(instead of operating in an interleaved way, which is also possible).

Figure 15 shows a system with parallel groups of PEs. Each group executesan SPMD program that corresponds to the dedicated working mode. In thiscase also, some form of multicasting to spread the incoming data isassumed.

Computationalmodule

)LJXUH������6HYHUDO�SDUDOOHO�JURXSV�RI�3(V��HDFK�UXQQLQJ�D�SLSHOLQH�FKDLQSURJUDP�

82

���� &RQFXUUHQW�0RGHV�RQ�WKH�6DPH�+DUGZDUH

The system configurations in which several modes run concurrently on thesame set of nodes place a demand on the network to be reconfigurable. Twodifferent ways of mapping are considered here and described below.

The first way is to have concurrent modes where all nodes switch mode ateach new data batch. This means that the signal processing chain for eachmode is only fed with one of 1 data batches, where 1 is the number ofmodes. Each node switches mode of operation in the order of once every tenms. It should hence be possible to reconfigure the network in a few hundredmicroseconds since reconfiguration might be needed for each mode change.The reason is that the communication pattern can differ from mode to mode.As an example, a node concurrently running several pipeline chains like theone shown in Figure 12 (but not pure pipeline) might perform a multicast inone mode and one-to-many or one-to-one in the next mode. Similar

Computationalmodule

)LJXUH�����6HYHUDO�SDUDOOHO�JURXSV�RI�3(V��HDFK�UXQQLQJ�DQ�630'�SURJUDP�

83

communication reasoning can be used in the system shown in Figure 13,which then runs several concurrent SPMD programs.

The second way to have concurrent modes on the same hardware is to leteach node switch mode several times per data batch. In this way, the signalprocessing chain for each mode is fed with all data batches. Of course, in thegeneral case, this requires more processing power. Also, which is of greaterinterest here, reconfiguration must be done faster. Depending on thenumber of working modes and how the algorithms are mapped onto thecomputational modules, reconfiguring the network in a few tens ofmicroseconds or less might be desirable.

����2QH�'DWD�&XEH�SHU�3URFHVVLQJ�(OHPHQW

One can allow the incoming data to be distributed such that each data cubeis given to a different processor, as shown in Figure 16. All calculations inthe whole signal processing chain are then performed by the same processorfor each data cube, therefore requiring no communication before the resultsare gathered. Although this way of mapping minimizes communication itwill typically not be a good solution because the computational latency willbe too great. For example, consider a system in which the data aredistributed to 100 processors, one data cube at a time to each processor. If100 percent processor utilization is assumed and the CPI (CoherentProcessing Interval, i.e., the time between the start of two subsequent datacubes) is ten ms, then the computational latency will be one s, which is not

Computationalmodule

)LJXUH� ���� 6HYHUDO� 3(V� LQ� SDUDOOHO�� ZKHUH� HDFKGDWD�FXEH�LV�SURFHVVHG�E\�D�VLQJOH�3(�

84

acceptable. Hence, this kind of mapping will not be further investigated.Further reasons for why this is not a good choice of mapping are found in[Liu and Prasanna 1998].

���� 7UDIILF�7\SHV

For many of the cases described above, circuit switching might work well forthe data flow in the signal processing chain because the mode is notchanged very often, e.g., not for every single data cube. However, packetswitching is needed to carry short messages like control and statusmessages. There is no time to set up a circuit for this kind of messages, andto always have circuits connected is not viable for sporadic traffic.Nevertheless, packet switching can be supported by a totally differentsubsystem, e.g., an optical subsystem for circuit switching and an electricalsubsystem for packet switching.

���� 6\VWHP�6L]HV�DQG�&RPPXQLFDWLRQ�'LVWDQFHV

Different communication systems may fit different ranges of system sizes orcommunication distances. One interconnection network might, for example,not offer sufficient throughput over a long distance. Another might bephysically too large or too expensive compared with other solutions for smallcomputer systems. An evaluation is made to come to qualitative judgementsabout the communication systems’ suitability for the different system sizesand communication distances given in Table 5.

.LQG�RIFRPPXQLFDWLRQ

&RPPXQLFDWLRQGLVWDQFHV

Intra chip 0.1 - 2 cmIntra MCM 1 - 10 cmIntra board 5 - 50 cmInter board 0.1 - 1 mInter cabinet 0.5 - 10 mInter and intra room 10 - 100 mIntra and inter building 100 m - 10 km

7DEOH�����&ODVVLILFDWLRQ�RI�V\VWHP�VL]HV�DQGFRPPXQLFDWLRQ�GLVWDQFHV�

85

���(YDOXDWLRQ�RI�2SWLFDO�,QWHUFRQQHFWLRQ�6\VWHPV

A number of proposals for optical or optoelectronic communication systemssuitable in a radar signal processing system is given in Sections 6.1 through6.9, some of which are hybrids of several other systems. Although thereexist many other optical interconnection architectures that might becandidates, only some selected groups or concepts are selected here to give areasonably broad view of possible solutions. Somewhat more emphasis isplaced on pure fiber-optic solutions than, e.g., free space systems. Thesurvey ends with a summarizing evaluation in Section 6.10. Theclassification made in the chapter is to some extent influenced by workreferred to in Section 1.5. There are some previous surveys and tutorialtexts in the field of optical interconnects [Goldberg 1997] [Kurokawa andIkegami 1996], the latter focusing on parallel computers and ATM switches.

���� )LEHU�5LEERQ�5LQJ�1HWZRUN

Bit-parallel transfer can be utilized when fiber-ribbon cables/links are usedto connect the nodes in a point-to-point linked ring network. In such anetwork, one of the fibers in each ribbon is dedicated to carry the clocksignal. Therefore, no clock-recovery circuits are needed in the receivers.Other fibers can be utilized for, e.g., frame synchronization.

As seen in Figure 17, aggregated throughputs higher than 1 can be obtained

Node 3Node 0

Node 4

Node 2

Node 5

Node 1

)LJXUH�����([DPSOH�RI�VSDWLDO�EDQGZLGWK�UHXVH��1RGH�0�VHQGVWR�1RGH���DW�WKH�VDPH�WLPH�DV�1RGH���VHQGV�WR�1RGH���DQG�1RGH��VHQGV�D�PXOWLFDVW�SDFNHW�WR�1RGHV�������DQG���

86

in ring networks with support for spatial bandwidth reuse (sometimescalled pipeline rings) [Wong and Yum 1994]. This feature can be effectivelyused in signal processing applications with a pipelined dataflow, i.e., most ofthe communication is to the nearest down stream neighbor. Two fiber-ribbon pipeline ring networks have recently been reported [Jonsson 1998B].The first network has support for circuit switching on 8+1 fibers (data andclock) and packet-switching on an additional fiber [Jonsson et al. 1997B].The second network is more flexible but is a little more complex, and hassupport for packet switching on 8+1 fibers and uses a tenth fiber for controlpackets (see Figure 18) [Jonsson 1998]. The control packets carries MACinformation for the collision-less MAC protocol with support for slotreserving. Slot reserving can be used to get RTVCs (Real-Time VirtualChannels) [Arvind et al. 1991] for which guaranteed bandwidth and a worst-case latency are specified (compare with circuit switching).

Another fiber-ribbon ring network is the PONI network (formerly USCPOLO), which is proposed to be used in COWs and similar systems[Raghavan et al. 1999] [Sano and Levi 1998]. Integrated circuits have beendeveloped for the network, and tests have been performed [Sano et al. 1996][USC 1997].

����:'0�6WDU�1HWZRUN

A passive fiber-optic star distributes all incoming light on the input ports toall output ports. A network with the logical function of a bus is obtained

Node 3Node 0

Node 4

Node 2

Node 5

Node 1

= Packet-switched datachannel (8 fibers)

= Clock channel (1 fiber)

= Control channel (1 fiber)

)LJXUH�����&RQWURO�FKDQQHO�EDVHG�QHWZRUN�EXLOW�XS�ZLWK�ILEHU�ULEERQSRLQW�WR�SRLQW�OLQNV�

87

when connecting the transmitting and receiving side of each node to oneinput and output fiber of the star, respectively. By using WDM, multiplewavelength channels can carry data simultaneously in the network[Brackett 1990]. In other words, each channel has a specific color of light. Aflexible WDMA network requires tunable receivers and/or transmitters, i.e.,it should be possible to send/listen on an arbitrary channel [Mukherjee1992].

Figure 19 shows an example of a WDM star network configuration. Eachnode transmits on a wavelength unique to the node, while the receiver canlisten to an arbitrary wavelength. The configuration is used in the TD-TWDMA network [Jonsson et al. 1996], which has support for guaranteeingreal-time services, both in single-star networks [Jonsson et al. 1997] andstar-of-stars networks [Jonsson and Svensson 1997]. One can say that thiskind of network architecture implements a distributed crossbar. Theflexibility is hence high, and multicast and single-destination traffic can co-exist. The number of wavelengths is practically limited to 16-32 [Brackett1996], but, as stated above, hierarchical networks with wavelength reusecan be built.

Tunable components (e.g., filters) with tuning latencies in the order of ananosecond have been reported, but they often have a limited tuning range[Kobrinski et al. 1988]. At the expense of longer tuning latencies, however,components with a broader tuning range can be used [Cheung 1990]. Suchcomponents can be used to achieve a cheaper network in systems wheremuch of the communication patterns remain constant for a longer period,

λ 1 λ 2

, ...λ 1 λ 2 λ Ν , ...λ 1 λ 2 λ Ν , ...λ 1 λ 2 λ Ν

λ Ν

Node NNode 1 Node 2

)LJXUH�����:'0�VWDU�QHWZRUN�

88

e.g., during the processing of a data cube in a radar system with a pipelinedmapping of the processing stages. To support some more rapidly changingtraffic patterns, the nodes can be extended with transmitters and receiversfixed-tuned to a special broadcast channel. This configuration can becompared to having support for both circuit switching and packet switching.

An example based on the signal processing chain shown in Figure 11 isgiven below to show that only one input and one output channel (in additionto the broadcast channel) are needed during the processing of a data cube,i.e., the normal minimum time running one pipeline chain (working mode ofthe radar). Figure 11 shows 13 nodes. In addition, the antenna is seen asone node (feeds the first node in the chain with data), Node 1, and there isone master node responsible for supervising the whole system andinteracting with the user, Node 15. For simplicity, we assume an efficientchannel bandwidth of 6.0 Gbit/s. A feasible allocation scheme of eightchannels (in addition to the broadcast channel) is shown in Table 6.

Complete removal of the ability to tune in a WDM star network gives amulti-hop network [Mukherjee 1992B]. Each node in a multi-hop networktransmits and receives on one or a few dedicated wavelengths. If a nodedoes not have the capability of sending on one of the receiver wavelengths ofthe destination node, the traffic must pass one or several intermediatenodes. The wavelengths can be chosen to get, e.g., a perfect-shuffle network[Acampora and Karol 1989] or a network with a pattern similar to the

1RGH ,QSXW�FKDQQHO 2XWSXW�FKDQQHO1 − λ1

2 λ1 λ2

3 λ2 λ3

4 λ3 λ4

5 λ3 λ4

6 λ4 λ5

7 λ5 λ6

8 λ6 λ7

9 λ7 λ7

10 λ7 λ8

11 λ7 λ8

12 λ7 λ8

13 λ7 λ8

14 λ8 λ8

15 λ8 −

7DEOH�����$�IHDVLEOH�DOORFDWLRQ�VFKHPH�RI�WKHZDYHOHQJWK� FKDQQHOV� LQ� D� :'0� VWDUQHWZRUN�IRU�WKH�VDPSOH�UDGDU�V\VWHP�

89

dataflow in a pipelined system. One can also choose to have a network inwhich several topologies are embedded, e.g., a ring and a hypercube. Anexample of a multi-hop network is shown in Figure 20. This configuration ofwavelength assignments corresponds to the topology shown in Figure 21.Dynamic real-time scheduling can be done in a multi-hop network [Yu andBhattacharya 1997]. The method works like static scheduling, but here acentral node runs the scheduling algorithm and high priority messagesmight preempt low priority messages. The highest priority level is used formessages with hard deadlines, while the other levels are used for messageswith soft deadlines. Lower priority levels are used for less importantmessages.

Some WDM star networks have been specially developed to be used inshared memory multiprocessors, e.g., using a dedicated wavelength forcache coherence control operations [Dowd and Chu 1994] [Ha and Pinkston1995] [Ha and Pinkston 1997]. A smart pixel based network interface chiphas been developed for a free space variant of one of these networks[Pinkston and Kuznia 1997].

����:'0�5LQJ�1HWZRUN

A WDM ring network utilizes ADMs in all nodes to insert, listen, andremove wavelength channels to/from the ring. In the WDMA ring networkdescribed in [Irshid and Kavehrad 1992], each node is assigned a node-unique wavelength on which to transmit. The other nodes can then tune inan arbitrary channel on which to listen. This configuration is logically the

λ 1

,λ 2 λ 4 ,λ 1 λ 3

λ 4

Node 4Node 1

λ 2

,λ 1 λ 3

Node 2 Node 3

,λ 2 λ 4

λ 3

)LJXUH�����:'0�VWDU�PXOWL�KRS�QHWZRUN�

90

same as that for the WDM star network with fixed transmitters and tunablereceivers. The distributed crossbar again gives good performance for generalcommunication patterns, such as in a radar system with the SPMD model.

Spatial wavelength reuse can be achieved by removing the transmitted lightat the destination node (last destination node for multicast). At high degreesof nearest downstream neighbor communication, as in systems with apipelined mapping, throughputs significantly higher than 1 can be achievedfor a single wavelength. In this way, a smaller number of wavelengthchannels is needed.

As discussed for the WDM star network, components with long tuninglatencies (ADMs in the case of a ring) can be used for traffic patterns that donot change rapidly. An additional broadcast wavelength dedicated forpacket switching keeps the network flexible. A single 6 Gbit/s channel (inaddition to the broadcast channel) is sufficient for the signal processingchain shown in Figure 11, if the ADMs in Nodes 1, 2, 3, 6, 7, 8, and 9terminate the channel for wavelength reuse.

���� ,QWHJUDWHG�)LEHU�DQG�:DYHJXLGH�6ROXWLRQV

Fibers or other kinds of waveguides (hereafter commonly denoted aschannels) can be integrated to form a more or less compact system ofchannels. Fibers can be laminated to form a foil of channels, for use asintra-PCB (Printed Circuit Board) or back-plane interconnection systems[Eriksen et al. 1995] [Robertsson et al. 1995] [Shahid and Holland 1996].Fiber-ribbon connectors are applied to fiber end-points of the foil. An

λ 1

λ 2

λ 4

λ 3

Node 3

Node 2

Node 4

Node 1

Transmitter side Receiver side

Node 3

Node 2

Node 4

Node 1

λ 1

λ 1

λ 2

λ 2

λ 3

λ 3

λ 4

λ 4

)LJXUH�����0XOWL�KRS�WRSRORJ\�

91

example is shown in Figure 22, where four computational nodes areconnected in a ring topology. In addition, there is a clock node thatdistributes clock signals to the four computational nodes via equal-lengthfibers to keep the clock signals in phase. In other words, a fiber-optic clockdistribution network [Kiefer and Swanson 1995] and a data network areintegrated into one system. If one foil is placed on each PCB in a rack, theycan be passively connected to each other via fiber-ribbon cables. Usingpolymer waveguides instead of fibers brings advantages such as thepossibility of integrating splitters and combiners into the foil, and thepotential for more cost-effective mass production [Eriksen et al. 1995].

Integrated systems of channels can be setup and used in a number ofconfigurations, some of which are discussed below. One way is to embed aring with bit-parallel transmission and the possibility for spatial bandwidthreuse, as described in Section 6.1. Of course, this leads to the same goodperformance for pipelined data flows as does the fiber-ribbon ring network;the medium is simply changed into a more compact form. Besides purecommunication purposes, channels for, e.g., clock distribution (as seen in theexample) and flow control can be integrated into the same system.

Another way is to follow the proposed use of an array of passive optical starsto connect processor boards in a multiprocessor system via fiber-ribbon

Node 3

Nod

e 4 N

ode

2

Node 1

ClockOut In

In OutIn

Out

Out

In

Out

)LJXUH�����$�IRLO�RI�ILEHUV�FRQQHFWV�IRXU�FRPSXWDWLRQDO�QRGHV��,Q�DGGLWLRQ��DFORFN�QRGH�GLVWULEXWHV�FORFN�VLJQDOV�WR�WKH�FRPSXWDWLRQDO�QRGHV�

92

links, for which experiments with 6 x 700 Mbit/s fiber-ribbon links weredone (see Figure 23) [Parker 1991] [Parker et al. 1992]. As indicated above,such a configuration can be integrated by the use of polymer waveguides.The power budget can, however, be a limiting factor to the number of nodesand/or the distance. Advantages are simple hardware owing to bit-paralleltransmission (like other fiber-ribbon solutions) and the broadcast nature.Many-to-many communication patterns, as used when corner-turning inSPMD mode, map easily on the broadcast architecture as long as the stararray does not become a bottle-neck. In a similar system, the star array isexchanged by a chip (with optoelectronics) that has one incoming ribbonfrom each node and one output ribbon [Lukowicz et al. 1998]. The outputribbon is coupled to an array of 1 × 1 couplers so that each node has aribbon connected to its receiver. The chip couples the incoming traffictogether in a way that simulates a bus. At contention, the chip cantemporarily store packets.

Electronic crossbars can be distributed on the PCBs and/or placed on aspecial switch PCB in a back-plane system, and be connected by integratedparallel channels. A distributed crossbar can instead be realized by bit-serial transmissions over a fully connected topology, i.e., there is onechannel between each pair of nodes (see Figure 24) [Li et al. 1998C].Broadcast is done by driving all the laser diodes of a node with the same bit-stream. A simple solution is to always drive all the laser diodes in thetransmitting node and couple the photo diodes together as one incomingchannel. In this way we get an architecture with the support of a singlebroadcast at a time. Sending on only one fiber when performing single-destination communication will, however, give the opportunity of havingmultiple transmissions in the system at the same time. The number ofnodes, 1, in this configuration is limited because the number of fibers growsby 12. Clock recovery circuits must also be involved. The distressed power

Node 1 Node 2 Node 0

Fiber-ribbon

)LJXUH� ���� $UUD\� RI� SDVVLYH� RSWLFDO� VWDUV� FRQQHFWV� D� QXPEHU� RI� QRGHV� YLDILEHU�ULEERQ�FDEOHV�

93

budget actually offers an advantage over a system with passive splitters orstars (this also holds for, e.g., the point-to-point connected ring). Theflexibility of a crossbar makes the network good for radar systems with theSPMD mode.

Other similar systems include the integration of fibers into a PCB for thepurpose of clock distribution [Li et al. 1998]. Distribution to up to 128 nodeswas demonstrated. The fibers are laminated on one side of the PCB, whileintegrated circuits are placed on the reversed side. The end section of eachfiber is bent 90 degrees to lead the light through a so called via hole to thereversed side of the PCB.

����2SWLFDO�,QWHUFRQQHFWLRQV�DQG�(OHFWURQLF�&URVVEDUV

Communication systems such as in Myrinet [Boden et al. 1995], wherearbitrary switched topologies can be built by using electrical switches, cansupport a number of different traffic patterns possible in radar systems.Fiber-ribbons can be used to increase bandwidth, compared to electricalsystems, while still sending in bit-parallel mode. It is possible to have bitrates in the order of 1 Gbit/s over each fiber in the ribbon over tens ofmeters using standard fiber-ribbons. As noted in Section 6.4, foils of fibersor waveguides (e.g., arranged as ribbons) can be used to interconnect nodesand crossbars on the PCB and/or back-plane level.

An alternative to ribbons is bit-parallel transfer over a single fiber usingWDM. In such configurations, each bit in the word, plus the clock signal, isgiven a dedicated wavelength. Wavelengths (or fibers in a fiber-ribbon

Node 3

Nod

e 4

Nod

e 2

Node 1

)LJXUH�����)XOO\�FRQQHFWHG�WRSRORJ\�ZKHUH�HDFK�QRGH�KDVDQ� DUUD\� RI� 1� −� �� ODVHU� GLRGHV� DQG� DQ� DUUD\� RI� 1� −� �SKRWR�GLRGHV��ZKHUH�1�LV�WKH�QXPEHU�RI�QRGHV�

94

cable) can also be dedicated to other purposes such as frame synchronizationand flow control. Significantly higher bandwidth distance products can beachieved when using bit-parallel WDM over dispersion shifted fiber insteadof fiber-ribbons [Bergman et al. 1998] [Bergman et al. 1998B]. If, however,there is only communication over shorter distances (e.g., a few meters), thebandwidth distance product is not necessarily a limiting factor.Transmission experiments with an array of eight pie-shaped VCSELsarranged in a circular area with a diameter of 60 µm, to match the core of amultimode fiber, have been reported [Coldren et al. 1998]. Other work onthe integration of components for short distance (non telecom) WDM linkshas been reported, e.g., a 4 × 2.5 Gbit/s transceiver with integrated splitter,combiner, filters, etc. [Aronson et al. 1998].

The switch itself can also be modified to increase performance or packingdensity. A single-chip switch core where fiber-ribbons are coupled directly tooptoelectronic devices on the chip is possible [Szymanski et al. 1998].Attaching 32 incoming and 32 outgoing fiber-ribbons with 800 Mbit/s perfiber translates to an aggregated bandwidth of 204 Gbit/s through theswitch when eight fibers on each link are used for data.

A 16×16 crossbar switch chip, with integrated optoelectronic I/O wasimplemented for switching packets transferred using bit-parallel WDM[Krisnamoorthy et al. 1996]. Each node has two single-mode fibers coupledto the switch, one for input and one for output.

Another switch chip with integrated optoelectronic I/O is intended to beattached to each node in a static topology like a multidimensional torus[Pinkston et al. 1998]. A special feature of the chips is that potentialdeadlocks are handled by a global mechanism. The mechanism operates inorder to get mutual exclusion among packets to let one packet at a time usededicated hardware to recover from a potential deadlock.

Finally, it can be noted that both Myricom (Myrinet) and Mercury(RACEway) are looking at optical technology for their future products [Lund1997].

���� 6\VWHPV�ZLWK�6PDUW�3L[HO�$UUD\V

In smart pixel based systems, the interconnection network can normally notbe seen as a stand-alone subsystem. Instead, processors and optoelectronicdevices for communication are integrated on the same substrate [Neff et al.1996]. Typically, smart pixels are organized in a two-dimensional array(e.g., on a chip) where each smart pixel consists of a processor, a laser diodeand a photo diode. Other, but similar, configurations exist, for example,where incoming light is modulated by a modulator in the smart pixel.

95

Several smart pixel arrays can be arranged in a row where data istransferred stage by stage (see Figure 25). Between arrays, holographicinterconnects or other optics might be used to steer or switch the opticalchannels. The row arrangement is especially suitable in applications inwhich computations can be mapped in a pipeline fashion with one pipelinestage per array, e.g., image processing [McArdle et al. 1996] [McArdle et al.1997] [Ishikawa and McArdle 1998], sorting [Gourlay et al. 1998], andapplications using FFT [Betzos and Mitkas 1998]. The row of smart pixelarrays can be placed on a PCB with the bottom edge of each arrayelectrically connected to the PCB [Neff 1994].

A system where smart pixel arrays are connected in a ring has beenreported [Chen et al. 1998] [Chen et al. 1998B]. Each array operates inSIMD mode on two-dimensional data. A modified CSMA/CD protocol is usedfor arbitration in the ring where some of the pixels in each array are usedfor data and some for addressing and clocking.

In [Kurokawa et al. 1998], an estimation of maximum number of pixels andI/O throughput allowed on a chip is reported for a VCSEL based smart pixelarray. Power consumption and pixel homogeneity (lack of variation of theVCSEL threshold current) are considered, and compact 150 MHz processorelements each with 200 gates are assumed. With a threshold currentvariation of 10 %, a single chip with more than 1000 processor elements(smart pixels) is possible, each processor having a 300 Mbit/s I/O port. Theestimated maximum total I/O throughput for such a system is 600 Gbit/s.VCSELs are better than edge-emitting laser diodes for free spacecommunication mainly because of the possibility of two-dimensional VCSELarrays and because integration with electronic circuits is simpler[Kurokawa et al. 1998].

)LJXUH�����5RZ�RI�VPDUW�SL[HO�DUUD\V�

96

����2SWLFDO�DQG�2SWRHOHFWURQLF�6ZLWFK�)DEULFV

The architecture with optical interconnections and electronic crossbars isflexible and powerful. Optics and optoelectronics can however also be usedinternally in a switch fabric, i.e., more than just in the I/O interface. Abroad spectrum of solutions has been proposed, and some examples aregiven below.

SDM (Space Division Multiplexing) switches [Goh et al. 1998] [Guilfoyle etal. 1998] [Kato et al. 1998] [Lai et al. 1998] [Moosburger and Petermann1998] [Sawchuk et al. 1987] and WDM switches (consisting of, e.g.,wavelength converters and wavelength selective components) [Pedersen etal. 1998] [Flipse 1998] can be used both as stand-alone switches and asbuilding components in larger switch fabrics [Reif and Yoshida 1994]. As anexample, a Banyan multistage network built of 2 x 2 switch elements hasbeen described [Chamberlain et al. 1998]. Another multistage network usesboth WDM and SDM switches but in different stages [Kawai et al. 1995]. Amultistage network can also be implemented using chips with processingelements placed on a two-dimensional plane [Christensen and Haney 1997].The processors then communicate with each other by a mirror that bouncesback the beam to the plane but to another processor. Switching is made onthe chips while each pass between two switch stages corresponds to abounce on the mirror.

A multistage switch incorporating both electrical and optical switching, butin different stages, has also been reported [Duan and Wilmsen 1998]. Somework has focused on the communication between stages, e.g., perfect shufflewith lenses and prisms [Lohmann et al. 1986]. Switch times for SDMswitches in the order of 1 ns have been reported [Kato et al. 1998], whilesome SDM switches have switch times in the order of 1 ms [Tajima et al.1998]. A switch can be placed on a dedicated board in a cabinet and beconnected to processor boards via fibers or via an optical backplane [Maenoet al. 1997].

A system that implements a distributed crossbar, or a fully connectedsystem, connecting 1 nodes with only passive optics between thetransmitters and receivers has been demonstrated [Li et al. 1998B]. Alloptical channels turned on from a transmitter’s two-dimensional √1 × √1VCSEL array are inserted into a fiber image guide. The fiber image guidesfrom all transmitters end at a central free space system with lenses. Thelenses are arranged in such a way that the light from each VCSEL pixel in aVCSEL array is focused on a single spot together with the correspondingpixels in all other arrays. This gives 1 spots where each is focused into asingle fiber leading to a receiver. Hence, selecting a pixel in a VCSEL arrayto be turned on corresponds to addressing a destination node.

97

���� 3ODQDU�)UHH�6SDFH�2SWLFV

By placing electronic chips (including optoelectronic devices) and opticalelements on a substrate where light beams can travel, we get a planar freespace system (Figure 26) [Jahns 1994] [Jahns 1998] [Sinzinger 1998].Electronic chips are placed in a two-dimensional plane, while light beamstravel in a three-dimensional space. In this way, optical systems can beintegrated monolithically, which brings compact, stable and potentiallyinexpensive systems [Jahns 1998].

The interconnection pattern in a planar free space system can, for example,be chosen with respect to a pipelined dataflow between chips. Anotherpossibility is to have a more general topology, such as the two-dimensionalmesh, or to have special optical or optoelectronic devices dedicated toswitchfunctions. The latter configurations may be the best choice if the SPMDprogram model is used.

���� )UHH�6SDFH�2SWLFDO�%DFNSODQHV

Several different optical backplanes have been proposed, three of which arediscussed below. As shown in Figure 27a, using planar free space optics isone means of transporting optical signals between PCBs. Holographicgratings can be used to insert and extract the optical signals to/from thewaveguide, which may be a glass substrate [Zhao et al. 1995]. Severalbeams or bus lines can be used, i.e., each arrow in the figure representsseveral parallel beams [Zhao et al.1996].

)LJXUH� ����([DPSOH� RI� D� SODQDU� IUHH� VSDFH� V\VWHP��7KH� GLUHFWLRQ� RI� WKHEHDP�LV�VWHHUHG�E\�WKH�RSWLFDO�HOHPHQW�RQ�WKH�ZD\�EHWZHHQ�WZR�FKLSV�

98

In the system shown in Figure 27b, two-dimensional arrays of optical beams(typically 10 000) link neighboring PCBs together in a point-to-point fashion[Szymanski 1995] [Hinton and Szymanski 1995]. Smart pixel arrays thenact as intelligent routers that can, e.g., bypass data or perform dataextraction operations where some data pass to the local PCB and some dataare retransmitted to the next PCB [Supmonchai and Szymanski 1998]. Eachsmart pixel array can typically contain 1 000 smart pixels arranged in atwo-dimensional array, where each pixel has a receiver, a transmitter, and asimple processing unit. One way of configuring the system is to connect thesmart pixel arrays in a ring, where the ring can be reconfigured to embedother topologies [Szymanski and Hinton 1995] [Szymanski and Supmonchai1996].

The configuration shown in Figure 27c is similar to the optical backplanebased on planar free space interconnects. The difference is the replacementof the waveguide by a mirror [Hirabayashi et al. 1998]. An optical beamleaving a transmitter is simply bounced once on the mirror before it arrivesat the receiver. A regeneration of the optical signal (multi-hop) might beneeded on the way from the source to the final destination.

(a) (b)

(c)

)LJXUH�����2SWLFDO�EDFNSODQH�FRQILJXUDWLRQV���D��ZLWK�SODQDU�IUHH�VSDFHRSWLFV���E��ZLWK�VPDUW�SL[HO�DUUD\V��DQG��F��ZLWK�D�PLUURU�

99

Of the three types of optical backplanes discussed, the one with smart pixelarrays seems to be the most powerful. On the other hand, a simple passiveoptical backplane may have other advantages. Other optical backplaneshave been proposed, e.g., a bus where optical signals can pass throughtransparent photo detectors or be modulated by spatial light modulators[Hamanaka 1991].

����� 6XPPDU\

Some of the networks that incorporate fiber-optics are summarized in Table7 with remarks on their suitability/performance for the two basic cases ofmapping (pipeline and SPMD). The pipeline mapping fits on a larger varietyof networks because of the absence of all-to-all traffic patterns. Limitingfactors tothe use of SPMD mapping vary from network to network; thesemay be tuning speed when switching from many different sources anddestinations, shared resources that become bottlenecks and a topology thatfavors nearest-neighbor communication.

Performance estimations for the case of mapping several parallel workingmodes, each with either pipeline or SPMD mapping, are summarized inTable 8. The networks should essentially handle the same kind of trafficpatterns as in the single-mode case. However, the incoming data from the

1HWZRUN 3LSHOLQH 630' 1RWHVFiber-ribbon pipeline ring Good Moderate Good for SPMD too if enough

bandwidthWDM star, rapid tuning Good Good Flexible passive networkWDM star, slow tuning andbroadcast channel

Good Poor WDM star alternative thatmay be cheaper

Multi-hop WDM star Moderate Moderate Can be optimized forpipelined mapping

WDM ring with rapid tuning Good Good More channels might beneeded for SPMD

WDM ring, slow tuning andbroadcast channel

Good Poor WDM ring alternative thatmay be cheaper

Fiber ribbons and array ofstars

Moderate Moderate Power and bandwidth limited

Fully connected topologywith broadcast driving

Moderate Moderate Bandwidth limited. Costgrows with N2

Fully connected topologywith flexible driving

Good Good Cost grows with N2

Optical fibers and electroniccrossbars

Good Good Optolectronics also needed inswitch

7DEOH� ��� 3HUIRUPDQFH� VXPPDU\� RI� VRPH� RI� WKH� QHWZRUNV� GLVFXVVHG� ZLWKUHVSHFW�WR�SLSHOLQH�DQG�630'�PDSSLQJ�

100

antenna must, in the pipeline case, be multicasted to the first node in eachgroup of nodes dedicated to a working mode. In the case of SPMD, theincoming data are distributed by multiple multicast transmissions, eachcarrying a subset of the data cube.

If several concurrent working modes are time-interleaved, all nodes worktogether on the data cube, one mode at a time. In this way, thecommunication patterns change several times per CPI. The tuning latenciesin appropriate networks must therefore be reduced proportionally to thenumber of working modes. This is not a problem for most networks.

Having optics inside a switch gives the same flexibility as electroniccrossbars, but it might be possible to build larger switch fabrics with hightransmission capacities using optics. The suitability of the different freespace systems for the mapping cases discussed depends a great deal on themore detailed configurations of the systems. For example, planar free spacesystems can be arranged in arbitrary topologies. However, the free spacetechnology chosen has some influence on possible topologies etc dependingon, e.g., the linear order (or similar arrangements) of smart-pixel arrays andthe two-dimensional plane to which components are restricted to whenusing planar optics.

The suitability of different technologies/networks in different system sizes issummarized in Table 9. The lower bounds on system sizes arise from such

1HWZRUN 3LSHOLQH 630' 1RWHVFiber-ribbon pipeline ring Moderate Poor Extra bandwidth needed

to distribute input dataWDM star, rapid tuning Good Good Broadcast support is usedWDM star, slow tuning andbroadcast channel

Good Poor Broadcast support is used

Multi-hop WDM star Moderate Moderate Depends on which virtualtopology is chosen

WDM ring with rapid tuning Good Good Broadcast support is usedWDM ring, slow tuning andbroadcast channel

Good Poor Broadcast support is used

Fiber ribbons and array of stars Moderate Moderate Broadcast support is usedFully connected topology withbroadcast driving

Moderate Moderate Broadcast support is used

Fully connected topology withflexible driving

Good Good Broadcast support is used

Optical fibers and electroniccrossbars

Good Good Broadcast support is used

7DEOH� �� �� 3HUIRUPDQFH� VXPPDU\� RI� VRPH� RI� WKH� QHWZRUNV� GLVFXVVHG� ZLWKUHVSHFW�WR�SLSHOLQH�DQG�630'�PDSSLQJ�ZKHQ�VHYHUDO�FRQFXUUHQW�PRGHV�UXQLQ�SDUDOOHO��HDFK�RQ�GHGLFDWHG�KDUGZDUH�

101

factors as miniaturization problems (e.g., fiber-ribbon connectors) andexpensive components that are primarily developed today for long distancecommunication (e.g., WDM components). Some reasons for upper bounds arechannel-to-channel skew in fiber-ribbons (especially when there is adedicated clock channel), high signal losses (e.g., foils of polymerwaveguides), and alignment problems in free space systems.

The increasingly good price/performance ratio for fiber-ribbon linksindicates a great success potential for several of the fiber-ribbon networksdiscussed. On the other hand, the WDM technique offers flexible multi-channel networks that can be implemented using passive technology.Integrated fiber and waveguide solutions make the building of compactsystems possible, especially for networks such as those using fiber-ribbons.The same reasoning about compactness can be argued for free spacesystems. Optical backplanes may earn their success from the similaritieswith current rack-based systems.

Not all optical interconnection networks are quite as easy to categorize intothe groups above, e.g. those described in [Chiarulli et al. 1994] [Teza et al.1995] [Louri et al. 1996], the latter describing a network using both free-space and fiber-ribbon interconnects. In another system, plastic modulesperforming different functions are snapped together to form a free spaceoptical interconnection system [Neilson and Schenfeld 1998]. Examples ofmodule functions are relaying and beam splitting of a two-dimensionalarray of incoming beams.

After describing a radar signal processing system in Chapter 4 and differentrequirements and configurations in Chapter 5, we have now, in Chapter 6,evaluated a number of optical network configurations.

.LQG�RIFRPPXQLFDWLRQ

:'0VWDU�ULQJ

)LEHU�ULEERQ

)RLO�RIILEHUV

IUHH�VSDFH

Intra chip GoodIntra MCM GoodIntra board Good GoodInter board Expensive Good Good (Good)Inter cabinet Expensive GoodIntra and inter room Good GoodIntra and inter building Good Moderate

7DEOH����6XLWDELOLW\�LQ�GLIIHUHQW�V\VWHP�VL]HV��(PSW\�FHOOVLQ� D� FROXPQ� PHDQ� WKDW� WKH� WHFKQRORJ\�QHWZRUN� LV� QRWVXLWDEOH�IRU�WKH�FRUUHVSRQGLQJ�V\VWHP�VL]HV�

103

3DUW�,,,��3URSRVHG�1HWZRUN�$UFKLWHFWXUHVDQG�3URWRFROV

105

���)7�75�:'0�6WDU�1HWZRUN

High-performance interconnection networks can be foreseen to have acentral role in future distributed real-time systems. If a number ofcomputation modules, each very powerful or even parallel, are used toobtain a massively parallel or distributed system, a modular interconnectionnetwork able to carry a huge amount of data is needed. Other key featuresof the network are time deterministic latency and guarantees to meetdeadlines. Application examples are future radar signal processing systems,distributed multimedia systems, and image processing systems. A typicalsystem is the radar signal processing system described in [Jonsson et al.1996] [Taveniku et al. 1996], where each module consists of a SIMDcomputer and a network interface. In this way, an MSIMD computer systemis formed. Other applications in which the MSIMD architecture with a highperformance interconnection network may be required are described in[Davis et al. 1992] [Svensson and Wiberg 1993]. It should noted that otherMIMD systems can have similar network requirements as MSIMD systems.

The WDM technique [Hill 1989] [Green and Ramaswami 1990] offersextremely high aggregated bandwidth by the use of multiple wavelengthchannels in fiber-optic communication systems, and is expected to have akey role in future computer networks. Each channel has a specificwavelength, i.e., color, of the light. A promising network architecture forhigh performance systems, for which commercial components have alreadyappeared, is the WDM star network with multiple Gbit/s channels. This isbased on a passive optical star which implements a fiber-optic multi-accessnetwork [Mestdagh 1995] where all incoming packets, in the form of trainsof light pulses, to the star are distributed to all nodes in the network bysplitting the light.

The remainder of the chapter is organized as follows. First, an introductionto the network and protocol is presented in Section 7.1 together with relatednetworks. The protocol is then presented in Section 7.2, while theimplementation of real-time services is described in Section 7.3. Section 7.4discusses some aspects of implementation. The analysis of the network ismade in Sections 7.5, 7.6, and 7.7, which respectively present deterministicperformance, the case study, and simulation results. Section 7.8 gives aconclusion and summary.

106

���� 3URWRFRO�2YHUYLHZ�DQG�5HODWHG�1HWZRUNV

������ &LUFXLW�6ZLWFKHG�1HWZRUNVA number of protocols for WDM star networks have been proposed.However, the area of real-time protocols for these networks is relativelyunexplored, with a few exceptions. The Rainbow network, described in[Dono et al. 1990] [Janniello et al. 1992], and the 1�DT-WDMA protocol,described in [Humblet et al. 1993], support guaranteed bandwidth for circuitswitched and virtual circuit switched traffic, respectively. However, thebandwidth utilization is reduced when there is no traffic on an establishedconnection, because the bandwidth cannot be reused by other nodes. TheI-TDMA [Sivalingam et al. 1992] and I-TDMA* [Bogineni et al. 1993]protocols are other examples where guaranteed bandwidth cannot bedynamically reused. These two protocols are multichannel extensions tostatic uniform TDMA.

������ 3DFNHW�6ZLWFKHG�5HDO�7LPH�:'0�1HWZRUNVA real-time protocol for packet switched communication in WDM starnetworks is described in [Yan et al. 1996B]. The QoS associated with a real-time packet in the network is related to the probability of missing itsdeadline. The protocol tries to globally minimize the number of packets notmanaging to keep their QoS by adaptively changing the priority of queuedpackets. Although dynamic real-time properties are supported, the matter ofthe success or failure of a packet transmission depends on the global state ofthe network, and transmission success can not be guaranteed in advance.

A class of real-time protocols for CC-FTTT-FRTR (see Section 2.5.4, page 48) WDM star networks is proposed in [Maode et al. 1998]. The protocolssupport global "optimization" of the real-time traffic, i.e., the order oftransmission is calculated such that packets from all nodes are sorted at thesame time according to their real-time requirements, instead of just beinglocally sorted in each node. However, each node in the network requiresboth a tunable transmitter and an additional transmitter/receiver pair forthe control channel. Moreover, the protocols do not support the ability toguarantee timely delivery at the time of message generation.

Other work reported on real-time support in similar WDM networksinclude:

• A star network with support for allocation of isochronous channels(guaranteed bit-rate and bounded network delay with low jitter), butwithout support for aperiodic traffic, which is essential for, e.g.,control traffic in parallel and distributed systems [Wang et al. 1997].

107

• A star network with support for virtual circuits, both with andwithout (best effort) guaranteed bit-rate, but with the need of adedicated control channel and a central scheduling node and withoutsupport for aperiodic traffic [Kam et al. 1998] [Kam et al. 1998B].

• A WDM bus network with support for real-time streams but not forguarantees to separate messages [Cho et al. 1995].

• A network in which all nodes track already guaranteed but not sentmessages to be able to decide whether to guarantee new traffic.However, there is no support for RTVCs or the like, and a controlchannel with 0 mini slots per data slot, where 0 is the number ofnodes, is needed (can be difficult to implement owing to, e.g., clocksynchronization problems) [Chen and Georganas 1993].

• A network with support for real-time traffic by the use of priorities,but without support to guarantee delivery of messages before theirdeadlines [Selvakennedy and Ramani 1996].

Research on pre-allocation of real-time message streams has also beencarried out [Dong et al. 1998] [Tyan et al. 1996].

������ $�1HZ�3URWRFRO�ZLWK�5HDO�7LPH�6XSSRUWThis chapter gives a proposal for a medium access protocol for packetswitched real-time communication in WDM star networks. The protocol,which is time deterministic, is called TD-TWDMA and supports guaranteedreal-time services for both single destination, multicast, and broadcasttransmission. Slot reservation is also supported, while bandwidth is usedefficiently owing to a simple slot release method. The protocol uses bothtime and wavelength division multiplexing and is targeted for distributedreal-time systems, especially very high performance systems, e.g., byparallelism within the nodes. The deterministic properties of the protocolare theoretically analyzed, while computer simulations show theperformance in a network with general traffic.

We propose to expand the single-star network to a star-of-stars topology(Figure 28) for larger systems, where a backbone star connects severalWDM star networks (clusters). Each cluster then has an electronic gateway-node as an interface to the backbone star. By the use of electronic gatewaynodes, we retain the popular WDM star network architecture in eachcluster, for which cheap components can be expected to appear in the future.With electronic gateway nodes, we also achieve wavelength reuse in eachcluster and in the backbone.

108

Other hierarchical WDM star networks include the wavelength flat (allnodes share the same wavelength space) tree-of-stars network [Dowd et al.1993], the tree-of-stars network (called LIGHTNING) that has wavelengthrouting elements between each level [Dowd et al. 1996] and the multiplestar network where each node is directly connected to both a local star and aremote star [Ganz and Gao 1992B]. The star-of-stars network proposed inthis thesis can be seen as a two-level tree-of-stars network.

Fixed wavelength transmitters and tunable receivers are used in eachcluster or single star network (Figure 29). A fixed unit is always tuned toone and the same wavelength channel, while a tunable unit can be tuned toan arbitrary wavelength channel. Each transmitter has a specificwavelength (home channel), and the network architecture can be describedas FT1-TR1 using the classification scheme given in [Mukherjee 1992].FT1-TR1 stands for one Fixed Transmitter and one Tunable Receiver pernode. Other FT-TR networks are described in [Brackett 1991] [Dono et al.1990], and general information on WDM star networks is given in

)LJXUH�����0XOWLSOH�SDVVLYH�RSWLFDO�VWDUV�WRSRORJ\�

109

[Mukherjee 1992] [Brackett 1990] [Mestdagh 1995]. Components for WDMAnetworks are reviewed in [Green 1993]. The receivers are tunable over thewhole range of channels used in the cluster. This makes the cluster a single-hop network where any receiver can be reached by any transmitter in asingle hop [Mukherjee 1992] [Ramaswami 1993]. The main reason forchoosing FT1-TR1 is the naturally embedded broadcast function obtainedwhen all receivers are tuned to the same transmitter channel.

A 100-channel WDM system has been demonstrated [Toba et al. 1993].However, the practical limit in number of wavelengths in the kind ofnetworks reported in this chapter is expected to be somewhere between 16and 32 [Brackett 1996]. This translates into star-of-stars networks ofmaximum sizes between 256 and 1024 nodes, gateway nodes included.

The TD-TWDMA protocol is used separately in each cluster and in thebackbone. Therefore, in this chapter, the protocol is described as running ina single-star stand-alone network. Inter-cluster communication [Jonssonand Svensson 1997] is covered in the next chapter.

������ 3URWRFRO�DQG�5HDO�7LPH�)HDWXUHVIn dynamic distributed real-time systems, messages may be classified intotwo categories [Arvind et al. 1991]: best effort messages and guaranteeseeking messages. While best effort messages normally have soft deadlines,such that the system need only try its best to meet the deadlines, guarantee

λ 1 λ 2

, ...λ 1 λ 2 λ Ν , ...λ 1 λ 2 λ Ν , ...λ 1 λ 2 λ Ν

λ Ν

Node NNode 1 Node 2

)LJXUH�����3DVVLYH�RSWLFDO� VWDU�QHWZRUN�ZKHUH� WKH�QRGHV�DUH� HTXLSSHG�ZLWKIL[HG�WUDQVPLWWHUV�DQG�WXQDEOH�UHFHLYHUV�

110

seeking messages have harder timing constraints. If the communicationsystem cannot guarantee the timing constraints of a guarantee seekingmessage, the owner of the message should be aware of it immediately.

In a network using the TD-TWDMA protocol, real-time services to supportboth guarantee seeking and best effort messages, exist for both singledestination, multicast, and broadcast transmission. With TDMA, the accessto each channel is divided into cycles of time slots. Each node has a numberof guaranteed slots to support guarantee seeking messages. However, ifthere are no guaranteed messages in a node, the slots will be released forbest effort messages from other nodes (or the same node) according to apredetermined scheme. This is a simple method to obtain an efficientbandwidth utilization at the same time time deterministic bandwidth isachieved. A node can increase its time deterministic bandwidth, used forguarantee seeking messages, by slot reserving. Another way of using slotreserving is for the allocation of RTVCs [Arvind et al. 1991]. The slot releasemethod is also used for the reserved slots.

The main function of the TD-TWDMA protocol is to allocate time slots foreither guarantee seeking messages or best effort messages. The allocation isdone using a deterministic distributed slot allocation algorithm. Thealgorithm temporarily changes the predetermined scheme according to thecurrent slot demands from each node. These slot demands are transmittedin advance on the same channels as the data are transmitted on and theycontain information about which guaranteed slots should be kept and whichshould be released. These types of WDM networks without a separatecontrol channel are denoted as QRQ� FRQWURO� FKDQQHO based networks.Networks in which a separate control channel is used to reserve access tothe data channels are denoted as FRQWURO�FKDQQHO based networks. Other noncontrol channel based networks than the one presented in this thesis arefound in [Dono et al. 1990] [Ganz and Koren 1991] [Ganz and Gao 1992],while control channel based networks are found in [Bogineni and Dowd1992] [Habbab et al. 1987] [Chen et al. 1990] [Chipalkatti et al. 1992].FatMAC, presented in [Sivalingam and Dowd 1995], is another non controlchannel protocol for WDM star networks where the access to the channels isdivided into a control phase and a data phase. However, FatMAC has nosupport for real-time services.

���� 3URWRFRO�'HVFULSWLRQ

The network and protocol proposed will now be described in detail. Thenotation used when describing the network is found in Table 10. Thisnotation is used in the next chapter as well. Because each transmitter in thenetwork has a specific wavelength, the number of wavelengths, &, hereafterdenoted as channels, equals the number of nodes, & = 0. The transmitter

111

and receiver parts of the transceiver are independent and can workconcurrently.

There are 20 queues in each transmitter, 0 queues for best effort messagesand 0 for guarantee seeking messages. For each of the two types ofmessages, one queue is for broadcast and 0 − 1 queues are for singledestination messages (one for each of the other nodes). The broadcastqueues are used for both true broadcast messages and for multicastmessages (messages destined to more than one node but not to all). We

0: Number of nodes in the network

&: Number of channels (wavelengths) in the network

PL: Designation of node L

%: Effective bandwidth of each channel in the network

γ: Slot length, including inter-slot gap

µ: Computing time of the distributed slot allocation algorithm

6: Total number of slots in a cycle

3: Number of cycles deadline guarantees can be forecast

.: Number of best effort multicast/broadcast packets to be sent in next cycle

VL: Designation of the L:th slot in a cycle

YLM: Index of the default high priority owner (transmitter) of slot L in thereceiver cycle in node M (YLM is an element in matrix 9 with size�0�×�(0 − 1))

ZLM: Index of the default low priority owner (transmitter) of slot L in the receivercycle in node M�(ZLM is an element in matrix : with size�0�×�(0 − 1))

[LM: Element in the control slot matrix ; with size�0�×� (0 − 1), sent from onenode, where status is given of slot L in the receiver cycle in node M

\L: Tells which queue to take a sub-message from for transmission in slot L (\L

is an element in matrix < with size�0(0 − 1))

]L: Tells which channel to tune in for receiving in slot L (]L is an element inmatrix = with size�0(0 − 1))

JLM: Number of high priority slots in the L:th next cycle, where M is thecorresponding queue (JLM is an element in matrix * with size�(0 + 1)��3)

KL: Current number of sub-messages in each guarantee seeking queue (KL is anelement in matrix + with size�0 + 1)

τ: Packet latency: delay from the moment a packet arrives at the transmitterbuffer until the moment the transmission begins

7DEOH�����1RWDWLRQV�IRU�GHVFULSWLRQV�RI�WKH�QHWZRUN�DUFKLWHFWXUH�DQG�SURWRFRO�

112

define the size of an entry in the queues as that of a packet, i.e., a part of amessage corresponding to one slot.

Section 7.2.1 explains the function of the cycles, while the distributed slotallocation algorithm is presented in Section 7.2.2.

������ 7UDQVPLWWHU�DQG�5HFHLYHU�&\FOHVBecause every transmitter has its own home channel, the only possibleconflict is when two or more nodes wish to transmit to the same node at thesame instant. To prevent conflicts, slots in the network are thereforeallocated in the receiver cycles, where each slot will have a specific owner.This allocation is done by the distributed slot allocation algorithm.

One cycle is also running in each transmitter, but this is only to tell whenand to whom the node is allowed to transmit. The transmitter cycle reflectsthe slots that the node owns in the receiver cycle of every other node(exemplified in Figure 30). In the receiver cycles, shown in the upper tablein the figure, each slot in each receiver cycle can be assigned to only onetransmitter at the same time. The lower table shows how the slots of atransmitter cycle are built up by copying its entries in the correspondingslots in each receiver cycle.

Figure 31 shows how a UHFHLYHU cycle is partitioned into data slots andcontrol slots. Each node PL, 1 ≤ L ≤ 0, is assigned one of the 0 control slots,where it broadcasts control information to all other nodes PM, 1 ≤ M�≤ 0 and

5HFHLYHU 'DWD�VORWVF\FOHV � � � � � � � � � �� �� ��

1RGH�� 2 2 3 4 3 2 3 4 4 2 3 41RGH�� 1 3 3 3 4 4 3 4 1 1 3 41RGH�� 1 4 4 4 1 2 1 4 1 1 2 41RGH�� 1 2 1 1 2 2 3 2 1 2 3 3

⇓'DWD�VORWV

� � � � � � � � � �� �� ��

7UDQVPLWWHUF\FOH�LQQRGH��

234

- 4 4 3 - 3 - 234

23

- -

)LJXUH�����7KH�WUDQVPLWWHU�F\FOH��ORZHU�WDEOH��LV�ILOOHG�E\�WDNLQJDOO� RZQHG� VORWV� LQ� WKH� UHFHLYHU� F\FOHV�� XSSHU� WDEOH�� LQ� DOO� RWKHUQRGHV��1RWH�WKDW�D�PXOWLFDVW�LV�SRVVLEOH�LQ�6ORWV�������DQG����

113

M ≠ L. The control slots are therefore identically assigned in every receivercycle. When control slots have been gathered from all other nodes, allocationof the data slots in the QH[W cycle can be calculated using the distributed slotallocation algorithm described in Section 7.2.2.

To reduce the latency, the control slots are placed as late as possible in thecycle. However, there must be time for the distributed slot-allocationalgorithm to be calculated before beginning the next cycle.

������ 'LVWULEXWHG�6ORW�$OORFDWLRQ�$OJRULWKPThe TD-TWDMA protocol consists of three steps:

1. Each node transmits a control slot

2. Each node separately runs the distributed slot allocation algorithm

3. The nodes transmit and receive data slots

The ways in which the contents of the control slots is calculated andincoming messages and buffers are handled partly determine the real-timeservices and are described in Section 7.3. When describing the distributedslot allocation algorithm, it is assumed for simplicity that all broadcastedcontrol slots are received before the algorithm is started. As described inSection 7.3, however, this is not a requirement in a real implementation.

The slot allocation algorithm is based on a predetermined allocation schemethat can be partly overloaded. As an example, the allocation scheme for afour-node system is shown in Figure 32. We do not call this scheme“reservation”, because that term is used when describing the overloading ofthe scheme. Only data slots are shown, and even though the control slotsmay be in the middle of the cycle, we assume here they have index 13 to 16.The elements denoted with ‘-‘ can be exchanged with node indexes but, forsimplicity, this is not investigated here.

)LJXUH�����$�UHFHLYHU�F\FOH�LV�SDUWLWLRQHG�LQWR�GDWD�VORWVDQG�FRQWURO�VORWV�

114

The total number of slots in a cycle is set to 6 = 02, and the number of dataslots is 0(0�− 1). Each pair of rows represents one receiver cycle, whereeach number is the index of the transmitter that owns the correspondingslot. The high priority row is the default scheme, but if the high priority slotowner does not need the slot it is temporarily released for the current cycle,i.e., the cycle where the data should have been sent. This is done by arelease message (described below) contained in the owner’s control slot. Thelow priority owner will then get the slot. If neither the high priority nor thelow priority owner needs the slot, it will be unused. This is the cost ofhaving a VLPSOH algorithm. However, compared to a static TDMA system,HIILFLHQW bandwidth utilization is achieved with the slot release method.

In the predetermined allocation scheme, YLM in matrix 9 identifies the indexof the high priority owner of slot L in the receiver cycle in node M. In the sameway, ZLM in matrix : identifies the index of the low priority owner. The highpriority owners are determined by

YLM = ((L − 1) mod 0) + 1 (1)

for 1 ≤ L ≤ 0(0 − 1), 1 ≤ M ≤ 0 and L ≠ M, while the low priority owners aredetermined by

ZLM = (((L − 1) div 0 + M) mod 0) + 1 (2)

The high priority slots are coordinated to allow for broadcasts but, asdescribed in Section 7.3.4, this is not always the case.

5HFHLYHUV 'DWD�VORWV3ULRULW\ � � � � � � � � � �� �� ��

1: High - 2 3 4 - 2 3 4 - 2 3 4�� /RZ � � � � � � � � � � � �

2: High 1 - 3 4 1 - 3 4 1 - 3 4�� /RZ � � � � � � � � � � � �

3: High 1 2 - 4 1 2 - 4 1 2 - 4�� /RZ � � � � � � � � � � � �

4: High 1 2 3 - 1 2 3 - 1 2 3 -�� /RZ � � � � � � � � � � � �

)LJXUH�����$OORFDWLRQ�VFKHPH�IRU�WKH�UHFHLYHU�F\FOHV�LQ�D�IRXU�QRGH�V\VWHP�

115

The release message mentioned above is actually a specified value in thecontrol slot matrix, ;, that each node transmits in the control slot. Thematrix has the same size as matrix 9, and each element in the matrix givesinformation about the corresponding high priority slot. All elements [LM,1 ≤ L ≤ 6, 1 ≤ M ≤ 0, in the matrix are set to zero except for thosecorresponding to high priority slots belonging to the node, which should notbe released. Those elements are instead set to one. This means that a zeroin the position of a high priority slot belonging to the node will release theslot.

When all the ; matrices (one from each node) have been gathered by a node,two new matrices are composed, < and =. Each element in < is used todetermine what queue to take a packet from, for transmission in a slot,while an element in = determines what channel to tune in at the beginningof a slot. The two matrices have elements corresponding to both data slotsand control slots during one cycle but, for clearness, the control slotelements are not treated here. The way the elements in < are set and usedto determine the current queue is described in Section 7.3. When thedistributed slot allocation algorithm is run, an element states the kind ofslot and is defined as:

M

L

M

M

M

\L

node to slot priority Highbroadcastmulticast/ priority High

slot in ontransmissi No node to slot priority Low

110

1

+

−−

= (3)

for slot L where 1 ≤ L ≤ 0(0 − 1). Each element is set using the followingalgorithm for node PN:

1. Let \L = − 1 − M, where M refers to the element ZLM = N in matrix :, i.e.,the receiver at which the node has a default low priority slot

2. If� [LM = 1 in matrix ;� from node� PO, where M refers to the elementZLM = N in matrix :�and� O =�YLM, then set \L = 0, i.e., the slot was notreleased by the high priority owner

3. If ∑=

0

M

LM[1

> 1 for matrix ; from node PN, then set \L = 1, i.e., a high

priority multicast/broadcast slot belonging to the node

4. If ∑=

0

M

LM[1

= 1 for matrix ; from node PN, then set \L = 1 + O, where O

refers to the element [LO = 1 in matrix ; from node PN, i.e., a highpriority single destination slot belonging to the node

116

Each element ]L, 1 ≤ L ≤ 0(0 − 1), in matrix = is set using the followingalgorithm for node PN:

1. Let ]L = ZLN, i.e., the default low priority owner

2. If�[LN = 1 in matrix ;�from node�PM, where M = YLN, then set ]L = YLN, i.e.,the slot was not released by the high priority owner

Since each node can independently perform the computations of the slotallocation scheme, it is called a distributed algorithm. No extra latency isrequired to return the result of the algorithm, which had been the case if,e.g., a master had calculated it.

����5HDO�7LPH�6HUYLFHV

In the description of how guarantee seeking and best effort messages arepassed through the transmitter to obtain real-time services, the followingparts will be explained:

1. Treatment of arriving guarantee seeking messages

2. Treatment of arriving best effort messages

3. The moment of transmitting a control slot

4. The moment at which all control slots are received

5. Action at the beginning of a data slot

Points 1 and 2 will first be explained. In 7.3.2, the third and fourth pointsare discussed, and the last point is described in 7.3.3. Finally, slot reservingand RTVCs are introduced in 7.3.4.

������ $UULYLQJ�0HVVDJHVThe ability to give deadline guarantees for a message relies on knowingwhen there are guaranteed slots in the forthcoming cycles. Element JLM,1 ≤ L ≤ 3, 1 ≤ M ≤ 0���1, in matrix * holds the number of high priority slotsbelonging to the node in the L:th cycle next to the currently running one,where M = 1 for broadcast slots and 2 ≤ M ≤ 0���1 for single destination slotsto node PM-1. 3 is chosen at system design and tells us how far in the future,relative to the current time, deadlines can be guaranteed. Broadcast andmulticast do not have to be treated separately here since the protocol doesnot allow the default scheme to be changed (by reservation) to have highpriority multicast to less than all other nodes.

By scanning JLM, starting with L = 1 and with M set to the actual type ofarriving message, we can see whether there are enough high priority slots,

117

before deadline, to guarantee that a message will be sent in time. If not, themessage will be rejected immediately and the owner will have time tohandle the situation. If, instead, a guarantee can be given, each element in* corresponding to required slots is decremented by the number of requiredslots for that element, i.e., the sum of all decrements equals the number ofpackets in the message. In the case of a single destination message with toofew available slots, the broadcast elements are also scanned. Packets arealways put in the guarantee seeking queue that corresponds to the elementin *� that was decremented in the order of transmission to facilitatereassembling the message.

Best effort messages may be transmitted in order, according to, e.g., theEDF (Earliest Deadline First)�algorithm, but in the scope of this thesis theyare assumed not to have deadlines specified and to be transmitted in orderof arrival. An arriving best effort message is hence simply put in the correctbest-effort destination-queue. Multicast messages are put in the broadcastqueue.

������ &RQWURO�6ORWVWhen it is time to send a control slot, the following algorithm is executed innode PN, 1 ≤ N ≤ 0:

if J11 > 0

. = MIN(L� J11), where L is the number of packets in the best effortbroadcast queue

endif

count the packets in each guarantee seeking queue to set KL, 1 ≤ L ≤ 0 + 1,L ≠ N + 1

[LM = 0, 1 ≤ L ≤ 0, 1 ≤ M ≤ 0(0 − 1)

for M = 1 to 0, M ≠ N

scan [LM starting with L = 1 and set [LM = 1 in KM�1 positions (if possible),where the corresponding element in 9� is�YLM = N�and YLO ≠ N, 1 ≤ O ≤ 0,O ≠ M

endfor

do the same scan of [LM for broadcast high priority slots, first for the K1

guarantee seeking packets and then for the . best effort packets, setting[LM = 1 for each corresponding destination that is set in the address fieldof a multicast (all destinations for broadcast), i.e., releasing slots todestinations not specified

send control slot

118

JLM = JL�1�M, 1 ≤ L ≤ 3 − 1, 1 ≤ M ≤ 0 + 1

JLM, L = 3, 1 ≤ M ≤ 0 + 1, are set to the corresponding number of default(after reservation) high priority slots for each corresponding kind ofdestination

where . is the number of best effort multicast/broadcast packets that are tobe sent in the next cycle. The elements� in matrix + state the currentnumber of packets in each guarantee seeking queue, where� K1 is thebroadcast queue and KL, 2 ≤ L ≤ 0 + 1, L ≠ N + 1, is the single destinationqueue to node PL�1.

When all control slots have been received, the elements in < are calculatedas described in the Section 7.2.2. However, each element is modified to statethe queue from which a packet should be taken when transmitting in thecorresponding slot, and is, for Slot L, defined as:

M

L

M

M

M

\L

node to seeking Guaranteebroadcast seeking Guarantee

slot in ontransmissi Nobroadcast effort Best

node to effort Best

110

11

+

−−−

= (4)

where 1 ≤ L ≤ 0(0 − 1). . of the elements in matrix <, where \L = 1, shouldbe set to \L = −1, beginning to scan the elements with� L = 0(0 − 1), i.e., indescending order.

������ 'DWD�6ORWVAt the beginning of each data slot, \L is used to determine the queue fromwhich to take a packet. By the definition of the protocol, there will alwaysbe a packet in the addressed queue when it is a guarantee seeking queue orthe best effort broadcast queue. However, the best effort single destinationqueues may be empty. In that case, an empty message is sent, telling thereceiver only that there was nothing to send. A packet generatedimmediately before a best effort single destination queue is checked will besent in that slot. The minimum latency for best effort single destinationmessages is therefore zero.

������ 6ORW�5HVHUYLQJA node can reserve slots to increase its guaranteed bandwidth. A maximumof 0(0 − 2) slots per cycle, Slots 5 to 12 in the example in Figure 32, areallowed to be reserved. Slots VL, 1 ≤ L ≤ 0, are not allowed to be reserved andslots� VL, 02 − 0 + 1 ≤ L ≤ 02, are control slots. When reserving slots, the

119

corresponding high priority entry (element in matrix 9) in the receivercycle, or cycles if broadcast is used, are exchanged with the index of thereserving node. To reduce complexity in the transmitters, the correspondingslots in all receivers must be reserved (i.e., for broadcast) if the slots are tobe used for multicast messages.

The assignment of reserved slots can be changed during run-time either byhigher layers, by a development system or by storing slot assignmentschemes for several working modes in the nodes. If the reservation is doneby higher layers, a reservation request to the nodes in question is sent in acontrol slot (together with matrix ;).

Slot reservation can also be used for the allocation of RTVCs where anRTVC, with its reserved slots, is typically dedicated to a specific applicationtask. An RTVC then has a specified worst-case latency and a guaranteedbandwidth (see Section 7.5). The allocation of RTVCs can also be used as aservice-primitive when reserving slots to increase the bandwidth forguarantee seeking traffic. Each such RTVC is then dedicated to be used forguarantee seeking messages generated by the task owning the RTVC. Themechanism for guarantee seeking traffic described above must then bereplicated for each RTVC but is somewhat simpler because only slotsreserved for the RTVC in question must be considered.

���� ,PSOHPHQWDWLRQ�$VSHFWV

������ &ORFN�6\QFKURQL]DWLRQThe TDMA method used requires synchronized nodes. Hardersynchronization will reduce the necessary gap between the time slots. Amethod to achieve harder synchronization in TDMA star networks is toaccount for the node-to-star propagation delay so that all packetstransmitted in the same slot pass the star simultaneously [Jonsson et al.1995] [Bengtsson et al. 1994]. A discussion of timing and dispersion in WDMstar networks can be found in [Semaan and Humblet 1993]. In this thesis,the interconnection distances are assumed to be short, and the propagationdelay is hence neglected.

������ &ORFN�5HFRYHU\A method to reduce the effect of clock-recovery and tuning latencies in thereceivers will increase performance. By duplicating the optoelectronic andclock-recovery parts of the receiver, the time for clock-recovery andwavelength tuning can even be eliminated. This is done by locking oneclock-recovery circuit to the currently used channel while the other onerecovers bit synchronization for the channel that will be used in the next

120

slot. In this way, the tuning time of the receiver only needs to be shorterthan the duration of one slot, minus the clock-recovery time. For thisreason, we assume that the tuning latency in the receivers can be neglected.

������ &RPSXWDWLRQDO�&RPSOH[LW\For each slot, the outcome of the distributed slot allocation algorithmdepends only on the control slot information from the corresponding highpriority owner. Therefore, the algorithm can start as soon as the first controlslot is received. A gap of only one data slot, between the control slots andthe beginning of the next cycle, to finish the computation is hence assumed.The small gap positively affects the latency. Only table indexing is thenused in the data slots, in transmitters to choose between buffered messagesand in receivers to choose channel tuning.

The protocol and its real-time services may seem rather computationallyintensive because, for the computation of each slot allocation, operations aredone on every element in a matrix column or in a whole matrix. In a realimplementation, however, the searches and scans of matrix 9 and : can beexchanged for simpler operations. The reason is that these matrices do notchange for each cycle and, e.g., a matrix search must only be done when thedefault allocation scheme is changed. Similar reasoning can be usedconcerning matrix *, because only a few elements are changed in each cycle.When all ; matrices are received, they can be composed into one ; matrixto save memory space by doing an element wise OR. Since all elements in Xare binary, the composition can be accelerated using bit-parallel operations.Each element in ; corresponds to the element with the same indexes inmatrix 9 (and :) and, even if the elements in ; change values in each cycle,their relation to 9 (and :) does not change. Hence, similar reasoning as isused with 9 and : to reduce the computational complexity can be used hereas well. The counting of packets in the queues can be eliminated bycontinuously updating counters when packets are inserted or removed.

������ (OHFWURQLF�6WDUVElectronic stars are also consistent with the network architecture; thesegive a network which can be implemented with cheap components today orin the near future depending on size and bandwidth (see Figure 33)[Jonsson et al. 1995]. The switching circuit should always be able tosimultaneously assign a path to an arbitrary output channel from everysingle incoming channel (true crossbar). The switch should also be able tomap one input channel to many output channels, multicasting, or one inputto all outputs, broadcasting (these features are automatically implementedwith a true crossbar switch). As shown in the figure, the control unit of thestar is connected to the switch in the same way as the other channels are.By this connection, the control unit also receives control packets and can,

121

with the information in these packets, configure the star according to theTD-TWDMA protocol. The electronic star simulates the passive optical starin a way almost invisible to the nodes. Therefore, the nodes still run thedistributed slot allocation algorithm.

Although the electronic star brings additional cost in, e.g., optoelectronictransceivers in the star, the point-to-point links have a positive effect. Thedifference as compared to the passive optical star is the simplified clock-recovery because bit synchronization can be maintained by continuouslysending information, either data or synchronization patterns, over the seriallink (see, e.g., [TriQuint 1992]). This can be done due to the fact that eachfiber is actually a point-to-point connection and not a shared medium.

����'HWHUPLQLVWLF�3HUIRUPDQFH

The real-time services rely on time-deterministic communication. Thedeterministic bandwidth and the worst-case packet latency for thisguaranteed bandwidth is analyzed below. As stated above, the analysistreats intra-cluster communication.

When a specific node is the only transmitter in the network, the bandwidthit receives is %(0 − 1) ⁄ 0, where % is the channel bandwidth. This gives 87,94, and 97 percent of the channel bandwidth for 8, 16, and 32 nodes,respectively. This bandwidth is then divided into %(0 − 1) ⁄ 02, which can

)LJXUH�����(OHFWURQLF�VWDU�

122

be used for broadcasting, and %(0 − 1)2 ⁄ 02 for point-to-pointcommunication, i.e., %(0 − 1) ⁄ 02 to each of all the other nodes. Whencompeting with other nodes, the broadcast bandwidth is always guaranteed(deterministic) and can be used for either single destination, multicast, orbroadcast messages. This deterministic bandwidth is 10.9, 5.9, and 3.0percent of the channel bandwidth for 8, 16, and 32 nodes, respectively. Ifmaximum allowed reservation in another node’s receiver cycle is possible,the deterministic bandwidth is increased to�%(02 − 20 + 1) ⁄ 02 includingthe broadcast bandwidth. This gives 77, 88, and 94 percent of the channelbandwidth for 8, 16, and 32 nodes, respectively. When some of the reservedslots are reserved in all receiver cycles, the deterministic broadcastbandwidth is also increased. A node that leaves all slots allowed forreservation to other reserving nodes will have the minimum guaranteedbandwidth possible: %� ⁄ 02. This gives 1.6, 0.4, and 0.1 percent of thechannel bandwidth for 8, 16, and 32 nodes, respectively. However, all thisbandwidth has broadcast capability.

The worst-case latency for a single packet gives an essential performancemeasurement of networks for real-time systems and will therefore beanalyzed for the proposed network. It is here assumed that only highpriority slots can be used and the control packet must hence be transmittedfirst to claim the use of the high priority slot. The time it takes to finish theslot allocation algorithm, µ, sets the limit for how late in the cycle thecontrol slots can be placed. As shown in Figure 34, the worst-case latency fornode PL, 1 ≤ L ≤ 0, is

τPD[ = 6γ + 0γ + µ = (0 + 1)0γ + µ (5)

which consists of: (i) the worst-case delay before the own control slotappears, i.e., one cycle as shown in the figure, (ii) the delay from the controlslot to the first owned data slot. By fine grain interleaving of the slots,where each node is assured to have one of the first 0 slots, we can minimizethe second part of the latency so that node PL, 1 ≤ L ≤ 0, will never have aslot later than slot VL as its first slot in the cycle. As explained above, µ canbe assumed to be the duration of one slot. This gives, for a network with a

)LJXUH�����/DWHQF\�FDOFXODWLRQ�

123

slot duration of 1 µs, a worst-case latency of 73 µs, 0.27 ms, and 1.1 ms for8, 16, and 32 nodes, respectively. Note that the actual latency may besignificantly lower, especially when many low priority slots are availablesince those slots can be used without first waiting for control slottransmission.

The worst-case latency of an RTVC belonging to node PL is

τPD[ = 6γ + (0 + M − L)γ + µ = (02 + 0 + M − L)γ + µ (6)

if VM is the first slot in the cycle reserved for the RTVC. The bandwidth of anRTVC is 1%�⁄ 02, where 1 is the number of slots reserved for the RTVC.

���� &DVH�6WXG\

The signal processing system shown in Figure 11 on page 77 was chosen asa case study, where the bandwidth demands on the inter-modulecommunication are also given. Although the data flow has a very simplestructure, there can exist control messages in other directions than the dataflow. Also, the algorithms, and hence the data flow, can be changed whenswitching to a different working mode. A general network is thereforerequired. For simplicity, the maximum data flow from one module isassumed to be 6.0 Gbit/s, excluding, e.g., error checking codes. The latencybudget for the whole chain is 100 ms, in which 10 ms is included for inter-module communication. With the ten pipeline stages (some of the stagesinclude several nodes) shown in the figure, including the antenna, themaximum allowed latency per link is τPD[ = 1.0 ms.

To guarantee a node bandwidth of 6.0 Gbit/s, maximum slot reservation ismade for these pipeline stages (RTVCs can be used). This gives

6.0 Gbit/s =�%0(0�− 2) /�0� (7)

which, e.g., for 0 = 16 leads to % = 6.9 Gbit/s. The network size of 0 = 16implies that the calculation holds for the 14 nodes in the radar system,including the antenna, plus two other nodes (e.g., for control purposes).With the calculated value of % we will, in addition to the reservedbandwidth, have a high priority bandwidth of %/0� = 27 Mbit/s. Thisbandwidth is contained in the part of the cycle that is not allowed to bereserved and can be used for control and status messages. According to theabove calculations, we have guaranteed bandwidths for both the dataflowand for control messages in the radar system.

With assumed values for parameters, it is possible to get the maximumnumber of nodes in a single-cluster network from Equation 5. Assuming aslot length of γ = 1 µs, a gap of one data slot between the last control slot

124

and the next cycle, i.e., µ = γ, and a worst-case latency of τPD[ = 1.0 ms, 0max

= 30 is found to be the maximum number of nodes. The value of 0max can beincreased by decreasing γ, since the number of bits in one slot is rather high.At an effective bandwidth of 6.0 Gbit/s, there will be 6000 bits in each slot,which can be compared to ATM with its 424-bit cells. Clustering the systeminto a star-of-stars topology can also increase 0max [Jonsson et al. 1996].

���� 6LPXODWLRQ�5HVXOWV

In addition to the worst-case analysis (deterministic performance), theaverage performance of the network was analyzed through computersimulations for general traffic. Single-star networks with 8, 16, and 32nodes were simulated. Other assumptions were:

• A gap of one data slot between the last control slot and the next cycle,i.e., µ = γ. To give an example of a real system, the slot duration was setto γ = 1 µs. This corresponds to a 1 kbit packet if the data rate is 1 Gbit/s.

• All guarantee seeking messages have a deadline at 5 ms from themoment of generation.

• Uniform traffic was assumed, i.e., all nodes had equal probability ofmessage generation and uniformly distributed destination addresses.Messages were generated according to a Poisson process, and allmessages were of single destination type.

• Message lengths were exponentially distributed between one and tenslots (discrete number of slots), with a length of one slot as the highestprobability.

• For simplicity, the propagation delay was neglected and no slotreservation was done.

• Infinite queue lengths were assumed.

• Packet generation rate is the message generation rate through the meanmessage length.

• Latency is defined as the time elapsed from the moment of arrival of amessage until the last packet of the message leaves the transmitter.

First, the deadline missing percentage (percentage of the messages that arenot accepted for transmission) versus packet generation rate of guaranteeseeking messages is plotted in Figure 35. At moderate traffic intensities, nomessages miss their deadlines for the given assumptions. The deterministicbandwidth fraction is (0 − 1) ⁄ 02. This gives 10.9, 5.9, and 3.0 percent for 8,16, and 32 nodes, respectively. As shown in the figure, guarantee seekingmessages begin to be partly rejected around these values. The plot is

125

independent of packet generation rate of best effort messages, sinceguarantee seeking messages are always given priority.

In Figure 36, the mean latency for guarantee seeking messages is plottedagainst packet generation rate. Again, the performance is independent ofthe amount of best effort traffic. Because messages that can not beguaranteed to meet their deadlines are discarded, the guarantee seekinglatency is upper bounded. The latency is rather uniform at low trafficintensities, but starts to grow earlier for larger networks because of thelower fraction of deterministic bandwidth.

The plot in Figure 37 shows the latency for best effort messages versus totalpacket generation rate (guarantee seeking plus best effort). The trafficconsisted of 10 % guarantee seeking messages and 90 % best effortmessages. The fraction of the bandwidth available for data packets is(0 − 1)⁄0, which gives 87.5, 93.8, and 96.9 percent for 8, 16, and 32 nodesrespectively. However, when the guarantee seeking traffic becomessaturated, some of the total number of generated packets are discardedguarantee seeking messages. Below saturation, the best effort latency isalmost uniform.

0 0.02 0.04 0.06 0.08 0.1 0.12 0.140

10

20

30

40

50

60

70

80

90

100

Guarantee-seeking packet generation rate

Dea

dlin

e m

issi

ng p

erce

ntag

eo 8 nodes

+ 16 nodes

* 32 nodes

)LJXUH�����5HDO�WLPH�SHUIRUPDQFH�SORWWHG�DV�IUDFWLRQ�RI�PHVVDJHV�WKDW�PLVVWKHLU�GHDGOLQHV�YHUVXV�WUDIILF�LQWHQVLW\�

126

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.160

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Guarantee-seeking packet generation rate

Mea

n gu

aran

tee-

seek

ing

late

ncy

[ms]

o 8 nodes+ 16 nodes

* 32 nodes

)LJXUH� ���� /DWHQF\� IRU� JXDUDQWHH� VHHNLQJ� PHVVDJHV� SORWWHG� YHUVXV� WUDIILFLQWHQVLW\�

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Total packet generation rate

Mea

n be

st e

ffort

late

ncy

[ms]

o 8 nodes

+ 16 nodes

* 32 nodes

)LJXUH� ���� 0HDQ� ODWHQF\� IRU� EHVW�HIIRUW� PHVVDJHV� LQ� D� QHWZRUN� ZLWK� ��� �JXDUDQWHH�VHHNLQJ�DQG������EHVW�HIIRUW�PHVVDJHV�

127

The mean best effort latency is plotted again in Figure 38, this time versusthe bandwidth utilization. The same ratio between guarantee seekingmessages and best effort messages was used. The plot looks very similar tothe previous best effort latency plot in Figure 37. This is a consequence ofthe almost linear relationship between packet generation rate andbandwidth utilization below saturation.

���� 6XPPDU\

A medium access protocol for WDM star networks in distributed real-timecomputer systems has been proposed. To obtain scalability, the networkarchitecture can be extended to a star-of-stars configuration. At the sametime as dynamic real-time properties are supported, the protocol givesalmost uniform latency nearly up to the theoretical saturation point.Deadline guarantees are supported, where the underlying deterministicbandwidth can be changed dynamically through slot reserving. An efficientbandwidth utilization is achieved by means of a simple slot release method.

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Bandwidth utilization

Mea

n be

st e

ffort

late

ncy

[ms]

o 8 nodes

+ 16 nodes

* 32 nodes

)LJXUH� ����0HDQ� ODWHQF\� IRU� EHVW�HIIRUW�PHVVDJHV� SORWWHG� YHUVXV� EDQGZLGWKXWLOL]DWLRQ��7KH� WUDIILF� FRQVLVWHG� RI� ����� JXDUDQWHH� VHHNLQJ� DQG������EHVW�HIIRUW�PHVVDJHV�

129

���:'0�6WDU�RI�6WDUV�1HWZRUN

By using electronic gateway nodes we retain the popular WDM star networkarchitecture in each cluster, for which cheap components can be expected toappear in the future [Jonsson and Svensson 1997]. With electronic gatewaynodes, we also achieve wavelength reuse in each cluster.

The same MAC protocol, TD-TWDMA (see the previous chapter) [Jonsson etal. 1996] is used in every cluster and in the backbone. Real-time services areimplemented for inter-cluster communication in a similar way as for intra-cluster communication. However, the larger number of nodes sharing thebackbone slots must be considered when calculating the deterministicperformance. We also present a new method for clock synchronization toreduce the worst-case latency in this kind of multi-cluster network usingTDM. The same notation as was used in the previous chapter (see Table 10on page 111) is used when describing the multi-cluster extension of thenetwork, but with the additional notation found in Table 11.

The remainder of the chapter is organized as follows. The networkarchitecture and associated protocols are described in Section 8.1. Section8.2 then presents the deterministic performance of the network, which isfollowed by a discussion of clock synchronization in Section 8.3. The chapteris then summarized in Section 8.4.

����1HWZRUN�$UFKLWHFWXUH�DQG�3URWRFRO

Although the same MAC protocol is used separately in each cluster, theclusters can be coordinated to improve performance and to get time-deterministic communication for inter-cluster communication as well.

0M: Number of nodes in cluster M, including all transceiver modules on thecluster side in the gateway node

4M: Number of ordinary nodes in cluster M, i.e., number of physical end-nodes

&M: Number of channels (wavelengths) in cluster M

/: Number of clusters, which is the same as the number of gateway nodes

%: Effective bandwidth of each channel in the clusters

(: Effective bandwidth of each channel in the backbone

5: Ratio between the channel bandwidth in the backbone and the channelbandwidth in the clusters

7DEOH� ���� $GGLWLRQDO� QRWDWLRQ� ZKHQ� GHVFULELQJ� WKH� VWDU�RI�VWDUV� QHWZRUNDUFKLWHFWXUH�

130

A network consists of / clusters. Each cluster has 0L, 1 ≤ L ≤ /, nodes whereone of the nodes is a gateway node. A gateway node contains networkinterfaces to both the backbone star and its dedicated cluster and buffermemories for both upward and downward traffic. The size of the buffermemories is significantly larger than the possible traffic in one cycle. Statusinformation on the buffers (flow control information) is always sent togetherwith the other information in the control slots.

In some systems, it is desirable that the bandwidth per channel in thebackbone be higher than in the clusters. The increase in bandwidth may beimplemented either by a higher bit rate or by having several wavelengthsper channel. If 5 is the ratio of backbone channel bandwidth, (, to clusterchannel bandwidth, % (i.e. 5 = ( / %), then the gateway nodes are designedto each have 5 transceiver modules on the cluster side in order to achievethe same aggregated bandwidth as on the backbone side (Figure 39). Also, agateway node has 5 dedicated home channels on the cluster side, one foreach transmitter. Since the transceiver modules in a gateway node are seenas connections to separate nodes by the MAC protocol, the number of nodesis still defined as 0L = &L, 1 ≤ L ≤ /. In this way, we have 4L = 0L − 5ordinary end-nodes in each cluster.

The cycle length in the backbone is always the same as that in the clusters,both when measured in time and when measured in number of slots.

)LJXUH� ���� *DWHZD\� QRGH� ZLWK� KLJKHU� FKDQQHOEDQGZLGWK�RQ�WKH�EDFNERQH�VLGH��ULJKW�VLGH��

131

Therefore, the number of bits in a backbone slot is 5 times higher than in acluster slot. Each backbone slot is divided into 5 sub-slots, all with the samepair of gateway nodes as source and destination (Figure 40). However, thesub-slots can have different pairs of end-nodes. A sub-slot has the samenumber of bits as a cluster slot, which makes the design of the gatewaynodes easier. Also, the latency may decrease when using sub-slots if severalslots in the same source cluster and with the same destination cluster canbe packed together.

����'HWHUPLQLVWLF�3HUIRUPDQFH

Deterministic performance is important, e.g., for the ability to giveguarantees for guarantee seeking messages. The worst-case latency forinter-cluster communication between two end-nodes is analyzed for twocases: (i) full slot reservation by other nodes and (ii) no slot reservation.When reservation is considered, full reservation is assumed in all clustersand in the backbone by RWKHU�QRGHV than the analyzed transmitting node. Inboth cases, the network size, 0total, is assumed to be the total number ofnodes in the network as seen by the MAC protocol. Also, it is assumed thateach cluster has the same number of nodes as the number of clusters in thenetwork:

2

1

/00/

L

LWRWDO== ∑

=

(8)

The gap between the last control slot and the next cycle is assumed to beone data slot, i.e., µ = γ. We can then get the worst-case latency of a singlecluster from Equation 5:

τPD[ = (02 + 0 + 1)γ. (9)

)LJXUH�����(DFK�EDFNERQH�VORW�LV�GLYLGHG�LQWR�5�VXE�VORWV�ZLWKWKH�VDPH�SDLU�RI�JDWHZD\�QRGHV�DV�VRXUFH�DQG�GHVWLQDWLRQ�

132

The guaranteed minimal bandwidth per source node, proportional to thenumber of high priority slots excluding reserved slots, is also analyzed. It isassumed in the analysis that no manipulation is made of the allocationscheme in order to utilize slots not having any function. These slots arethose which are allocated for traffic between two transceiver modules in thesame gateway node.

When IXOO�VORW�UHVHUYDWLRQ by other nodes is assumed, the number of slots towhich the source node has access in the receiver cycles in the destinationnode and in the gateway nodes of the source and destination clustersreaches its minimum. The first transit is from the source end-node and tothe gateway node of the source cluster. Since the source node has directaccess to its own cluster, the worst-case latency, τ1, for the first transit canbe obtained by using Equation 9 for intra-cluster communication, with0 = /, i.e.:

τ� = γ (/� + / + 1). (10)

The second transit is between the gateway nodes of the source cluster andthe destination cluster, through the backbone. Here, slots from all 4 = / − 5ordinary nodes in the source cluster must, in the worst-case, be multiplexedover several high priority backbone slots. At full slot reservation, a node hasonly one high priority slot per cycle in which to transmit. Therefore, / − 5 −1 extra cycles, in addition to the normal intra-cluster latency (Equation 9),are needed. The source gateway node is responsible for carrying out thismultiplexing using the round robin scheduling strategy. Hence, the sourceend-node is guaranteed its part of the bandwidth. The worst-case latency forthe second transit is

τ� = γ (/2(/ − 5) + / + 1). (11)

In the destination cluster, for transfer from the gateway-node to the end-node, slots from all (/ − 5)(/ − 1) ordinary nodes outside the cluster must, inthe worst-case, be multiplexed. The latency decreases when 5 > 1, becausethe multiple transceiver modules, in the gateway node, work in parallel:

( )( )

++

+−−= 11

1 2 //5/5/

7UXQF�

γτ (12)

The total worst-case latency, with full slot reservation, is

( )( )

++

−++−−

=++=

3321 2 //5/

5/5/

���PD[

γ

ττττ(13)

133

and is plotted in Figure 41, in which the horizontal axis represents the totalnumber, /(/ − 5), of ordinary nodes in the network and each curverepresents a specific value of 5. To give an example of a real system, the slotlength is assumed to be γ = 1.0 µs in all latency plots. Figure 41 indicateshow latency decreases with 5.

When QR�VORW�UHVHUYDWLRQ in any receiver cycle is assumed, a node has / − 1high priority slots in which to transmit. The worst-case latency, τ1, for thefirst transit is the same as when full reservation is considered (Equation10). The latency of the rest of the transit will however decrease as comparedto the full reservation latency. The second transit latency is τ2 = τ1, becausethe high priority slots in the transmitter of the gateway node, on thebackbone side, will always be enough to multiplex one slot from each clusternode in one cycle:

/ − 1 ≥ / − 5 (14)

0 20 40 60 80 1000

1

2

3

4

5

6

7

8

9

10

Number of nodes

Late

ncy

[ms]

o R = 1

+ R = 2

* R = 4

)LJXUH� ���� :RUVW�FDVH� ODWHQF\� ZKHQ� IXOO� VORW� UHVHUYDWLRQ� E\� RWKHU� QRGHV� LVDVVXPHG�� 7KH� [�D[LV� UHSUHVHQWV� WKH� QXPEHU� RI� RUGLQDU\� QRGHV� DQG� WKH� VORWOHQJWK�LV�DVVXPHG�WR�EH�γ� �����µV�

134

In the third transit, to the destination end-node, the gateway node canmultiplex 5(/ − 1) slots per cycle time. The latency is

++

= 12 //5/

�γτ (15)

The total worst-case latency, with no slot reservation, is

++

+=++= 332 2 //5/

���PD[γττττ (16)

Figure 42 shows how the worst-case latency when no slot reservation is usedvaries with the total number of ordinary nodes. The latency is significantlylower as compared to the case of full-reservation. This effect is related to thehigher number of slots in the gateway nodes that can be used formultiplexing incoming messages.

0 20 40 60 80 1000

0.5

1

1.5

Number of nodes

Late

ncy

[ms]

o R = 1

+ R = 2

* R = 4

)LJXUH�����:RUVW�FDVH�ODWHQF\�ZKHQ�QR�VORW�UHVHUYDWLRQ�LV�DVVXPHG��7KH�[�D[LVUHSUHVHQWV�WKH�QXPEHU�RI�RUGLQDU\�QRGHV�DQG�WKH�VORW�OHQJWK�LV�DVVXPHG�WR�EHγ� �����µV�

135

We define the deterministic bandwidth per node as the minimum highpriority bandwidth when there are no reserved slots. In this analysis weexpress the bandwidth as a ratio to the full channel bandwidth, %.Measured as the number of high priority slots per total number of slots in acycle, this so called bandwidth is shown in Table 12 for the different cases.The last transit is the bottleneck; the node bandwidth for it, as a function ofthe number of ordinary nodes, is plotted in Figure 43 for full slotreservation. As seen in the figure, the deterministic node bandwidth is arather low part of the full bandwidth but all of the bandwidth can be usedfor broadcasting. Nodes requiring more deterministic bandwidth can useslot reservation. The deterministic node-bandwidth, when no slotreservation is assumed, is plotted in Figure 44 and is higher than theformer.

The real-time services offered by the MAC layer rely on the deterministiclatency and bandwidth. A guarantee can be stated if the known minimumnumber of high priority slots along the whole path through the network issufficient to transfer the message in time. In the calculation of thisjustification, the deterministic latency, the deterministic bandwidth, andthe deadline of the message are used. A slot of a guaranteed message istagged to indicate its priority over other slots. The gateway nodes thenalways transmit the tagged messages before buffered best effort messages.

���� &ORFN�6\QFKURQL]DWLRQ�$VSHFWV

The nodes are synchronized to account for the propagation delay betweentransmitting and receiving nodes, i.e., receivers are synchronized to

)XOO�UHVHUYDWLRQ 1R�UHVHUYDWLRQ

)URP�VRXUFH�HQG�QRGH�WR�JDWHZD\�QRGH�

2

1/ 2

1// −

7KURXJK�EDFNERQH�

( )5//5

−2

( )( )5///5

−−

2

1

)URP�JDWHZD\�QRGH�WR�GHVWLQDWLRQ�HQG�QRGH�

( )( )12 −− /5//5

( )5//5

−2

7DEOH� ���� 1RGH� EDQGZLGWK� LQ� QXPEHU� RI� KLJKSULRULW\�VORWV�SHU�WRWDO�QXPEHU�RI�VORWV�LQ�D�F\FOH�

136

transmitters, proportional to the propagation delays, and thus slots are firstexpected when they have traveled through the network [Jonsson et al. 1995][Bengtsson et al. 1994]. This type of synchronization is always used inside acluster.

The inter-cluster worst-case latency can be reduced by using thesynchronization scheme shown in Figure 45. For clarity, in the example inthe figure, all fibers are assumed to have the same length and propagationdelay and to have a propagation delay significantly lower than the cycletime of 6 slots. These restrictions do not directly apply to a realimplementation.

The midpoint of the backbone star is used as the reference point. Eachcluster is then synchronized to the backbone by its gateway node. In agateway node, the receiver on the cluster side is synchronized so tightly tothe transmitter on the backbone side that incoming messages are directlyforwarded when possible. In this way, the information extracted from anincoming control slot (about next hop, i.e., the gateway node of the

0 20 40 60 80 1000

0.001

0.002

0.003

0.004

0.005

0.006

0.007

0.008

0.009

0.01

Number of nodes

Ban

dwid

th

o R = 1

+ R = 2

* R = 4

)LJXUH� ���� 1RGH� EDQGZLGWK�� ZKHQ� IXOO� VORW� UHVHUYDWLRQ� E\� RWKHU� QRGHV� LVDVVXPHG��LQ�QXPEHU�RI�KLJK�SULRULW\�VORWV�SHU�WRWDO�QXPEHU�RI�VORWV�LQ�D�F\FOH�7KH�[�D[LV�UHSUHVHQWV�WKH�QXPEHU�RI�RUGLQDU\�QRGHV�

137

destination cluster) can be transmitted to the other gateway nodes, like apipeline mechanism, before the data slots have arrived. This will reduce theworst-case latency by γ (/� + / + 1), i.e., eliminate the latency for one hop asdescribed by Equation 9. On the other hand, the receiver on the backboneside is not synchronized to the transmitter on the cluster side. This isbecause transmitters in a cluster are synchronized to the receivers in thesame cluster. Therefore, the same latency is experienced as for the case inwhich the clusters and the backbone are not synchronized (described above).

The improved worst-case latency (Equation 13) when full slot reservation isassumed is

( )( )

++

−++−−= 221

1 2 //5/5/5/

PD[

γτ (17)

The corresponding improved latency when no slot reservation is assumed(Equation 16) is

0 20 40 60 80 1000

0.005

0.01

0.015

0.02

0.025

0.03

Number of nodes

Ban

dwid

th

o R = 1

+ R = 2

* R = 4

)LJXUH�����1RGH�EDQGZLGWK��ZKHQ�QR�VORW�UHVHUYDWLRQ�LV�DVVXPHG��LQ�QXPEHURI�KLJK�SULRULW\�VORWV�SHU�WRWDO�QXPEHU�RI�VORWV�LQ�D�F\FOH��7KH�[�D[LV�UHSUHVHQWVWKH�QXPEHU�RI�RUGLQDU\�QRGHV��

138

++

+= 221 2 //5/

PD[

γτ (18)

The relative latency improvement is best in the case of no slot reservationfor networks with larger values of 5 (Figure 46). The figure compares thelatencies with and without synchronization between the clusters and thebackbone. Note that the latter latency is the same as that described byEquation 16 above. Even better relative improvement is achieved when aslot can pass through the whole network without multiplexing over severalcycles in the gateway nodes, e.g., by the use of reserved slots or at lowtraffic. In this case, the worst-case latency decreases from 3γ (/� + / + 1) to

)LJXUH� ���� :LWK� WKH� V\QFKURQL]DWLRQ� VFKHPH� XVHG�LQFRPLQJ� WUDIILF� WR� D� JDWHZD\� QRGH� FDQ� EH� IRUZDUGHGLPPHGLDWHO\� H[FHSW� IRU� WKH� FDVH� RI� LQWHUQDO� GHOD\� LQ� WKHJDWHZD\�QRGH�

139

2γ (/� + / + 1), i.e., an improvement of 33 percent. The synchronizationscheme is general and can be used in other similar networks to improveworst-case performance.

���� 6XPPDU\

We have shown how to calculate the worst-case latency for inter-clustercommunication in a WDM star network using the TD-TWDMA protocol. Theanalysis shows how the latency decreases for larger networks when theratio between the backbone bandwidth and the cluster bandwidth increases.A calculation of the minimum deterministic bandwidth obtained in the casein which no reservation is used by the analyzed node was presented, and asynchronization scheme was proposed. At reserved traffic or low traffic, theimprovement to the worst-case latency when using this synchronizationscheme is 33 percent, as compared to the case in which the clusters are notsynchronized with one another.

0 20 40 60 80 1000

0.5

1

1.5

Number of nodes

Late

ncy

[ms]

+ No synch., R = 1

o Synch., R = 1

x No synch., R = 4

* Synch., R = 4

)LJXUH�����:RUVW�FDVH�ODWHQF\�FRPSDULVRQ�EHWZHHQ�XVDJH�DQG�QR�XVDJH�RI�WKHSURSRVHG�V\QFKURQL]DWLRQ�VFKHPH��1R�VORW�UHVHUYDWLRQ�LV�DVVXPHG��7KH�[�D[LVUHSUHVHQWV�WKH�QXPEHU�RI�RUGLQDU\�QRGHV�DQG�WKH�VORW�OHQJWK�LV�DVVXPHG�WR�EHγ� �����µV�

141

���&RQWURO�&KDQQHO�%DVHG�)LEHU�5LEERQ�3LSHOLQH5LQJ�1HWZRUN

This chapter and the next chapter presents two ring networks suitable fordifferent situations, both based on fiber-ribbon links. The proposed networksare pipeline ring networks based on optical fiber-ribbon point-to-point links.In a pipeline ring network, several packets can be traveling through thenetwork simultaneously, thus achieving an aggregated throughput higherthan the capacity of a single link. Motorola OPTOBUS bi-directional links[Schwartz et al. 1996] with ten fibers per direction are used (similar linkscan be used, too) but the links are arranged in a unidirectional ringarchitecture (see Figure 47) where only 2/0 bi-directional links areneeded to close a ring of 0 nodes. Fiber-ribbon links offering an aggregatedbandwidth of several Gbit/s have already reached the market [Bursky 1994].The increasingly good price/performance ratio for fiber-ribbon linksindicates a great potential for success of the proposed kind of networks.

The remainder of the chapter is organized as follows. An overview of theCC-FPR network is presented in Section 9.1, while a review of related work

)LJXUH������D��%L�GLUHFWLRQDO�ILEHU�ULEERQ�OLQN���E��8QLGLUHFWLRQDOULQJ�QHWZRUN�EXLOW�XS�RI�M���EL�GLUHFWLRQDO�OLQNV�

142

is given in Section 9.2. The CC-FPR protocol is presented in Section 9.3.Section 9.4 describes different user services. Section 9.5 discussesimplementation aspects, and a case study is presented in Section 9.6 toshow the efficiency of the networks. This is followed by a performanceanalysis and a summary in Sections 9.7 and 9.8, respectively.

����1HWZRUN�2YHUYLHZ

The first network (described in this chapter) is called CC-FPR [Jonsson1998] [Jonsson 1998B] and has special features for both parallel processingin general and for distributed real-time systems [Jonsson et al. 1999][Jonsson et al. 1999B]. The physical ring network is divided into two rings:a data ring and a control ring. In each fiber-ribbon link, eight fibers carrydata and one fiber is used to clock the data, byte by byte. Together, thesefibers form a data channel that carries data packets. The access is dividedinto slots as in an ordinary TDMA network. The tenth fiber is dedicated tobit-serial transmission of control packets that are used for the arbitration ofdata transmission in each slot. The clock signal on the dedicated clock fiber,which is used to clock data, also clocks each bit in the control packets.Separating clock and control fibers simplifies the transceiver hardwareimplementation, which is verified by the current prototype development.The control channel is also used for the implementation of low level supportfor barrier synchronization, global reduction, and reliable transmission.

Real-time services in the form of best effort messages, guarantee seekingmessages and RTVCs [Arvind et al. 1991] are supported for singledestination, multicast and broadcast transmission by the network.Application examples requiring a high performance network of this kind areradar signal processing systems [Jonsson et al. 1996] [Taveniku et al. 1998]and multimedia systems. An aggregated throughput of tens of Gbit/s isneeded in soon-to-come radar systems, and this is achieved with theproposed network as shown in a case study (see Section 9.6) [Jonsson 1998][Jonsson 1998B].

The node synchronization requirement is more relaxed than for a traditionalTDMA network, and the network is somewhat similar to a slotted ringnetwork (but without the need of a central controller). This is because theaccess to the network circulates among the nodes according to the physicalorder of the nodes in the ring. In addition, the ring can dynamically (foreach slot) be partitioned into segments to obtain a pipeline ring networkwhere several transmissions can take place simultaneously. Evensimultaneous multicast transmissions are possible when the multicastsegments do not overlap.

143

����5HODWHG�1HWZRUNV

Two kinds of spatial reuse in ring networks can be identified. The intentionof the first type is to overcome performance degradation owing to longpropagation delays around the ring as compared to the frame length.Performance is increased if another node (or nodes) can start transmissionbefore the current transmission is finished. One example is early tokenrelease in the 16 Mbit/s token ring network, where the token is releasedwhen the last bit of the frame has left the sending node. When early tokenrelease is QRW used, the sending node does not release the token until theframe has traveled around the ring and is completely removed from thering. Another example is slotted ring (e.g., the Cambridge network [Greavesand Zielinski 1993] [Hopper and Needham 1988]) where several time slots,in which each can host a frame, travel around the ring at the same time.

The first type of spatial reuse increases the utilization, but it cannot exceed1 as long as data frames are not removed for reuse of the bandwidth for newframes. This states the definition of the second type of spatial reuse. Forexample, suppose the frame contained in a slot in a slotted ring is removedby the destination node [Adams 1994]. The destination node, or anothernode downstream, can then reuse the slot. If the distribution of source anddestination nodes is totally uniform, the average utilization cantheoretically reach 2. This means that the average source to destinationdistance is half the ring, which gives an average of two transmitted framesper time unit. The term "pipeline ring network" is used in this thesis todenote a network with spatial reuse of the second type. It should be notedthat spatial reuse of the bandwidth can be implemented in dual uni-directional bus networks as well [Garrett and Li 1991] [Ray and Jiang 1995][Ray and Mukherjee 1995].

Other pipeline ring networks are described in [Chen et al. 1991] [Ofek 1994](MetaRing), [Imai et al. 1994] (ATMR), [Wong and Yum 1994], [Jafari et al.1980], and [Xu and Herzog 1988] and further references are given in [Wongand Yum 1994]. Advantages of the CC-FPR network over these othernetworks include the use of high bandwidth fiber-ribbon links and the closerelation between a dedicated control channel and a data channel withoutdisturbing the flow of data packets. In other words, control and data areoverlapped in time. With less header overhead in the data packets the slotlength can be shortened to reduce latency without too great a sacrifice ofbandwidth utilization. The separate clock and control fibers also simplifythe transceiver hardware implementation. Another fiber-ribbon ringnetwork is the USC PONI (formerly called POLO) network, proposed to beused in multimedia applications [Sano and Levi 1998] [Raghavan et al.1999]. However, spatial reuse of bandwidth is not a function of the USCPONI, i.e., it is not a pipeline ring network.

144

The network described by Jafari et al. also relies on a separate controlchannel but needs a central control node that brings additional cost inhardware and in latency when waiting for response from the central controlnode [Jafari et al. 1980]. The CC-FPR network is insensitive to propagationdelay in the sense that no feedback is needed, from other nodes or from acentral controller, between control packet and data packet transmissions.

A network with a similar slot reuse mechanism for reserved slot as for theCC-FPR network (see Section 9.4.1) has been reported [Marsan et al. 1997].However, the network does not support concurrent transmissions indifferent segments of a single ring channel.

Other high performance ring networks include the WDM passive ring[Irshid and Kavehrad 1992] and the hierarchical WDM ring [Louri andGupta 1997], which are more closely related to the WDM star network andstar-of-stars network described in Chapters 7 and 8, respectively.

���� 7KH�&&�)35�3URWRFRO

Throughout Section 9.3, slot reserving, as described in Section 9.4.1, isassumed not to be used. Before we explain the arbitration mechanism, wewill describe how data packets travel on the ring.

The access to the network is cyclic; each cycle consists of 0 time slots, where0 is the number of nodes. Each node is denoted PL, 1 ≤ L ≤ 0. Each slotalways has one node that is responsible for initiating the traffic around thering. This node is called the slot initiator (SI). Each node is slot initiator inone slot per cycle, as shown in Figure 48. At the end of the slot, the role ofslot initiator is asynchronously handed over to the next node downstream.This can be done implicitly simply by sensing the end of the slot, i.e., thelast bit.

7LPH

P1 P2 P3 P0

P1 P2 P3 P0

Cycle 1+1Cycle 1

Slot

)LJXUH� ���� 7KH� UROH� RI� EHLQJ� VORW� LQLWLDWRU� LVF\FOLFDOO\�UHSHDWHG��(DFK�RI�WKH�M�QRGHV�LV�WKH�VORWLQLWLDWRU�LQ�RQH�VORW�SHU�F\FOH�

145

The CC-FPR medium access protocol is based on the use of a control packetthat, for each slot, travels almost one round (over 0�− 1 links) on the controlchannel ring, as shown in Figure 49. The node that will be the slot initiatorin the next slot initiates the transmission of the control packet, as shown inthe figure. We denote this node SI+1. In the time domain, the control packetalways travels around the ring in the time slot preceding the one for whichit controls the arbitration (see Figure 50). Accordingly, the control packetalways passes each node one time slot before the data packet to which it isrelated.

The contents of the control packet are shown in Figure 51. The controlpacket consists of a start bit followed by an 0-bit long link reservation fieldand an 0 bit long destination field, where 0 is the number of nodes. Eachbit in the link-reservation field states whether the corresponding link isreserved for transmission in the next slot. In the same way, each bit in thedestination field states whether the corresponding node has a data packetdestined to it in the next slot. Additional information, such as flow control,is also included in the control packet; for clarity, this is not shown in thefigure.

Each node succeeding SI+1 checks the control packet as it passes todetermine: (L) whether it will receive a data packet in the next slot, which isindicated by the node’s bit in the destination field, and (LL) if a data packetwill pass the node in the next slot, which is indicated by the bit in the link-

Node 4Node 5

Node 3

Node 2Node 1

Node 6

Data & clock Control

)LJXUH� ���� 7KH� QRGH� VXFFHHGLQJ� WKH� VORW� LQLWLDWRULQLWLDWHV�WKH�FRQWURO�SDFNHW�WUDQVPLVVLRQ�

146

reservation field corresponding to the outgoing link of the node. If no datapacket is to pass the node, i.e., the rest of the ring back to the slot initiatoris free, then the node can transmit a data packet in the next time slot in thispart of the ring.

When a node has a packet ready for transmission, it prepares in advancenew link reservation and destination fields to reserve needed links andnotify destination node(s). In this way, the node can immediately change thecontrol packet when it passes, provided the bit in the link-reservation fieldcorresponding to the outgoing link of the node is set to zero (see Figure 52for a conceptual view of the control channel part of the transceiver). Sincethere is no data packet that will pass the node, succeeding nodes have nouse for the overwritten information in the control packet.

Because all of the nodes succeeding the slot initiator repeat the procedure ofchecking the control packet, multiple transmissions in different segments ofthe ring can occur in the same slot. The transceivers are designed to allowfor both reception and transmission at the same time (see Figure 53), whichincreases the possibility of spatial bandwidth reuse. An example of how thecontrol packets travel around a five-node network is shown in Figure 54.

Slot 1+1Slot 1

7LPH

Slot 1+2

Data-packetData-packet Data-packet

Control-packet

Control-packet

Control-packet

)LJXUH�����,Q�HDFK�VORW��D�QRGH�SDVVHV�WUDQVPLWV�RQH�FRQWURO�SDFNHWDQG� RQH� GDWD� SDFNHW�� ZKHUH� WKH� FRQWURO� SDFNHW� LV� XVHG� IRU� WKHDUELWUDWLRQ�RI�WKH�QH[W�VORW�

Destination fieldLink-reservation field

Node 1Start bit Node 2 Node 0Link 1 Link 2 Link 0

Bit 0+1Bit 0 Bit 0+2 Bit 20Bit 1 Bit 2 Bit 0

)LJXUH� ���� $� FRQWURO� SDFNHW� FRQWDLQV� D� VWDUW� ELW�� D� OLQNUHVHUYDWLRQ�ILHOG��DQG�D�GHVWLQDWLRQ�ILHOG�

147

The arbitration results in two concurrent data packet transmissions in thenext slot, one single-destination and one multicast packet, as shown inFigure 55. Node P� is SI+1 in the example; it therefore initiates the controlpacket transmission described in Figure 54. It reserves Link 1 and Link 2for transmission to node P� and informs this node, by setting thecorresponding bits in the link reservation field and the destination field,respectively, that it will have a data packet destined to it in the next slot.While node P� and node P� do not change the control packet, they check itto see whether there will be any data packets destined to them in the nextslot. Node P� reserves Link 4 and Link 5 for a multicast transmission tonode P� and node P�. Node P� then receives the control packet and removesit from the ring.

The reason why the control packet travels only among the first 0�− 1 linksafter SI+1 is that the clock signal is interrupted by the SI (see Figure 49).The node that initiated the transmission of the control packet, SI+1, doesnot return the packet. Consequently, it will not be informed of whether ornot there is a data packet destined to it in the next slot. However, the node

Control-packet

Register

Preparedcontrol-packet

From up-streamneighbor

To down-streamneighbor

)LJXUH� ���� &RQFHSWXDO� YLHZ� RI� WKH� FRQWURO� FKDQQHOSDUW�RI�WKH�WUDQVFHLYHU�

Datapacket

Receivebuffer

From up streamneighbor

To down streamneighbor

Transmitbuffer

)LJXUH�����&RQFHSWXDO�YLHZ�RI�WKH�GDWD�FKDQQHO�SDUW�RIWKH�WUDQVFHLYHU�

148

will receive either a packet destined to the node or an empty packet.

It is essential for desirable performance that the delay of the control packetin each node it bypasses be minimal, especially in large networks. Onemethod is to organize the bits in the link-reservation field in the controlpacket for each slot, so that they appear in the same order as that in whichthe control packet travels. In other words, the first bit corresponds to theoutgoing link from the slot initiator. Thus, when a node wishes to changethe contents of a control packet, it does not have to store the whole packetbefore checking and possibly overwriting it. Instead, it can retransmit thepacket bit by bit and exchange the remaining part of the packet (iftransmission is possible) after reading the bit in the link reservation fieldcorresponding to its outgoing link. The node’s bit in the destination field inthe incoming control packet must, however, be checked before it is thrownaway. Using this method, the delay in each node can be reduced to only oneor a few bits.

As indicated in Figure 56, the bandwidth utilization depends on the ratio ofthe total propagation delay around the ring to the cycle length. This is aneffect related to the asynchronous passing mechanism of the slot initiatorassignment. Further evaluation of this phenomena is provided in Section9.7.

����8VHU�6HUYLFHV

The user services described below are: real-time virtual channels (Section9.4.1), guarantee seeking messages (Section 9.4.2), best effort messages(Section 9.4.3), barrier synchronization (Section 9.4.4), global reduction(Section 9.4.5), and reliable transmission (Section 9.4.6).

1RGH2XWJRLQJ�FRQWURO�SDFNHW/LQN 'HVW� 7UDQVPLVVLRQ�DOORFDWHG

1 1 1 0 0 0 0 0 1 0 0 To Node 3

2 1 1 0 0 0 0 0 1 0 0 Could not allocate

3 1 1 0 0 0 0 0 1 0 0 Could allocate transmission toNodes 4, 5, and 1 but hadnothing to send

4 0 0 0 1 1 1 0 0 0 1 Multicast to Nodes 5 and 1

5 0 0 0 1 1 1 0 0 0 1 Could not allocate

)LJXUH�����$�FRQWURO�SDFNHW�WUDYHOV�DURXQG�D�QHWZRUN�ZLWK�ILYH�QRGHV�1RGH���LV�WKH�VORW�LQLWLDWRU�

149

������ 5HDO�7LPH�9LUWXDO�&KDQQHOVMany computer systems have real-time demands where the network mustoffer logical connections with guaranteed bandwidth and bounded latency.This can be done in the network by using slot reserving. We refer to suchconnections as RTVCs. Either the whole ring is reserved for a specific nodein a slot, or several segments of the ring are dedicated to some specificnodes.

When slot reservation is allowed, the cycle is prolonged to contain0(1RUG + 1UHV) slots, where 1RUG and 1UHV are the number of slots for ordinaryuse and for reservation (per node and cycle), respectively. The values of 1RUG

and 1UHV are chosen at system design and remain unchanged duringoperation of the network if the system function does not change radically.The 1RUG ordinary slots cannot be reserved. However, all nodes in thenetwork can try to reserve a segment of the ring in a reservation slot, notonly the SI of that slot. In each node, a separate packet queue is providedfor each of its RTVCs.

When a node wishes to reserve a slot for an RTVC, it searches for slotswhere the required links are free to allow allocation of a new segment. First,the node’s own slots (i.e., where the node itself is the slot initiator) aresearched. If enough slots (actually only a segment in each slot) cannot beallocated for the reservation, the search is continued in other slots. In this

Node 3

Node 4

Node 2

Node 5

Node 1

Link 1 Link 2

Link 4

Link 3Link 5

)LJXUH� ���� $Q� H[DPSOH�� 1RGH� �� VHQGV� D� VLQJOHGHVWLQDWLRQ�SDFNHW�WR�1RGH����ZKLOH�1RGH���VHQGVD�PXOWLFDVW�SDFNHW�WR�1RGH���DQG�1RGH���

150

case, the node broadcasts a packet containing a request to all other nodes toallocate the desired segment in their slots. The packet contains informationabout the links required and the number of slots needed. Each node thenchecks whether any of its own slots have the required free links. All nodessend a packet back to the requesting node to notify which slots, if any, thathave been allocated. When the requesting node has received the answers, itdecides whether it is satisfied with the number of allocated slots. If not, itsends a release packet. Otherwise, it can start using the reserved slotsimmediately. However, it should still send a release packet if more slotsthan needed were allocated. The same medium access method as forordinary slots is used for reservation slots, but with the restriction not topass links in reserved segments of the ring. A reserved segment of the ringtemporarily not in use, however, can be reused by nodes down stream of theowner.

������ *XDUDQWHH�6HHNLQJ�0HVVDJHVGuarantee seeking messages normally have hard timing constraints. If thecommunication system cannot guarantee the timing constraints of aguarantee seeking message, the owner of the message should be madeaware of it immediately. In the CC-FPR network, a guarantee is given onlyif enough deterministic bandwidth (slots) owned by the node is free beforethe deadline of the message. The available deterministic bandwidthcorresponds to ordinary slots where the node is the slot initiator and which

)LJXUH�����7KH�EDQGZLGWK�XWLOL]DWLRQ�GHSHQGV�RQ� WKH� UDWLR� RI� WKHWRWDO� SURSDJDWLRQ� GHOD\� DURXQG� WKH� ULQJ� WR� WKH� F\FOH� OHQJWK�� 7KHER[HV� �ZLWK�EROG� WH[W� VKRZ� WKH� OLQN� WKURXJK�ZKLFK� HDFK� VORW� ILUVWSURSDJDWHV�

151

have not already been encountered for other guarantee seeking messagesqueued in the node.

Each transmitter has 0 − 1 queues for guarantee seeking messages, one foreach possible destination (the node itself excluded). When a multicastpacket arrives for queuing, it is put in the queue corresponding to themulticast destination furthest away from the source node downstream. Inthis way, multicast packets are treated in the same way as single-destination packets, and multiple multicast packets can travel in thenetwork at the same time whenever possible.

������ %HVW�(IIRUW�0HVVDJHVBest effort messages can be sent in all ordinary slots where the node is SIbut in which the node does not have any guarantee seeking messages thatcan be sent. In competition with the other nodes according to the CC-FPRprotocol, other ordinary slots can also be used but, again, only as long as noguarantee seeking messages can be sent. The same method is used forreservation slots that are not (or are only partly) reserved for the moment orthat are reserved but not used in the current slot.

As for guarantee seeking messages, each transmitter has 0 − 1 queues. Themessages in each of these queues are sorted according to the EDF-algorithm. If the deadline expires, the sending process is notified. Othersimilar algorithms can also be used.

������ %DUULHU�6\QFKURQL]DWLRQBarrier Synchronization (BS) is an operation to control the flow of processesin a distributed processing system. A logical point in the control flow of analgorithm is defined, at which all processes in a processing group mustarrive before any of the processes in the group are allowed to proceedfurther. When, during execution, a BS point is encountered in theapplication program, the node broadcasts the encountered BS_id in thecontrol packet when the node is SI+1. In this way, all nodes are notified thatthe node has reached the BS point. Nodes not belonging to the BS group canignore the broadcast, but nodes belonging to the same group, i.e., have thesame id, will make a note in an internal table. The control packet contains afield in which BS_id can be sent (see Figure 57). The id field contains eightbits, which permits ids ranging from one to 255. When the field is zero, noBS command is sent.

When a node participating in the BS group has received the correct BS_idfrom all the participants, it knows that all the other nodes are at the sameexecuting point and may proceed. The worst case latency for a node thatreaches the BS point until it can broadcast this to the other nodes is one

152

cycle. This assumes one slot per node and that each node is SI+1 only onceper cycle. Clearly, the implications of sending BS information in the controlchannel are both bounded latency and better bandwidth utilization in thedata channel. The whole BS mechanism is handled by the communicationinterface, transparent to the calling user processes.

In the description above, static allocation of barrier synchronization BS_idsis assumed. The programmer (or the compiler) allocates the requiredparameters for BS and GR (Global Reduction) off-line before run-time. Withminor adjustments, dynamic allocation is also possible, although this is notinvestigated further in this paper.

������ *OREDO�5HGXFWLRQGlobal reduction is similar to barrier synchronization, where data arecollected from distributed processes when they signal their arrival at thesynchronization point. A global operation, e.g., sum or product, is performedon the collected data so that only a single value is returned. At the end ofthe GR, all participating nodes have access to the same data. As in the caseof BS, we assume that the programmer (or the compiler) statically allocatesthe necessary parameters, off-line before run-time.

The GR command requires the following parameters: operation, length, datatype, GR_id, and the ids of the participating nodes. The last two parametersare similar to the parameters in the BS command. The operation parametertells the nodes what operation (sum, product, max, min, etc.) should beperformed on the data received. The data type parameter indicates how thedata field in the control packet (see Figure 57) is to be interpreted, and thelength parameter gives the length of the data field. In the studied case, thedata field is 320 bits long and may facilitate the transfer of, e.g., up to fivedouble precision floating point numbers (IEEE-754). Other data types mayalso be distributed. Except for the additional fields and the global function,the nodes treat GR commands in the same way as BS commands. The samereasoning of performance advantages also holds for GR.

1

ACK/NACKfield

StartLink

reservationfield

Destinationfield

7\SH

Flowcontrol field

IDField

M1 M M

1EU�RI�ELWV

4*(M-1) 8

Typebit

Data field

320

)LJXUH�����'HWDLOHG�GHVFULSWLRQ�RI�WKH�FRQWURO�SDFNHW�FRQWHQWV�

153

The type bit states whether the control packet contains a BS or a GRcommand and hence whether data are contained in the data field (seeFigure 57). The data field is currently only used for data reduction.

������ /RZ�/HYHO�6XSSRUW�IRU�5HOLDEOH�7UDQVPLVVLRQThe proposed network has low level support for reliable transmission[Bergenhem and Olsson 1999]. Network control information,acknowledge/negative acknowledge and flow control, is sent in the controlchannel instead of in ordinary data packets. This results in less or nooverhead in the data channel, i.e., better bandwidth utilization. The field inthe control packet named ACK/NACK contains bits that correspond to the 0packets that may have been received by the current SI+1 during theprevious 0 slots. The ACK/NACK information is therefore always sentwhen a node is SI+1. If a packet was correctly received (correct checksum), a“1” is written in the position of the ACK/NACK field that corresponds to theslot that the packet was received. If a faulty packet or no packet wasreceived, a “0” is written. All nodes must keep track of their transmissionsand can therefore resolve the meaning of the bits in the ACK/NACK field. Inthis way, the nodes can be notified as to whether their packet was correctlyreceived or must be retransmitted. The latency for a node to send andreceive ACK/NACK is bounded, which is desirable.

The 4(0 − 1) bits in the flow control field relate to independent logicalconnections. Put simply, each node can have up to four logical connections,with low-level support for flow control, from each other node. The SI+1 setsthe bit corresponding to a logical connection to "1" if it is to be haltedtemporarily; otherwise, the bit is set to "0".

���� ,PSOHPHQWDWLRQ�$VSHFWV

In addition to the high bandwidth offered by a fiber-ribbon cable, it alsooffers a ten-fold increase in packing density as compared to electrical cables,resulting in less rigid cables [Karstensen et al. 1995]. Furthermore, it is notnecessary for the designer to be concerned about electromagnetic emissions.These properties make possible new components such as the single-chipoptoelectronic switch core reported in [Szymanski et al. 1998]. In addition tothe switch function, the chip eliminates 32 OPTOBUS 800 Mbit/s per fibertransceivers. This translates to an aggregated bandwidth of 204 Gbit/sthrough the switch when eight of the ten fibers on each link are used fordata. Such a switch can connect multiple ring clusters when building largenetworks. The high bit-rate of a fiber-ribbon link makes it possible to reducethe slot duration in the proposed networks, still keeping the same number ofbits in a packet as on a slower link based on an electrical cable. This reducesthe latency.

154

In scaling up the bandwidth of a fiber-ribbon link where a dedicated fibercarries the clock signal, the main problem is channel-to-channel skew. Theskew is mainly the result of differences in propagation delay betweendifferent fibers and variations of lasing delay time among different laserdiodes [Kurokawa et al. 1998]. The 400 Mbit/s OPTOBUS has a specifiedmaximum skew of 200 ps, excluding the fiber-ribbon cable for which 6 ps/mis assumed for standard ribbons [Schwartz et al. 1996]. Since the datastream passing a node in the ring network is, at least, passing a pipelineregister, the channel-to-channel skew is not accumulated over severalnodes. The limited distance between two adjacent nodes, owing to skew, isin the same order of magnitude as in today’s LAN networks (a few hundredmeters). In parallel and distributed computer systems, so called SystemArea Networks (SAN), the maximum required distance is normally lowerthan this limit. It may however be difficult to increase the bit rate to severalGbit/s per fiber without significantly reducing the distance. It can also beargued that it should be possible to construct networks with more physicallydistributed nodes, since this may be valuable in some applications. Thelatency of the network is not dependant on the distance, except that thepropagation delay is added and the throughput is reduced as mentionedabove. This motivation calls for the discussion of techniques to reduce theeffect of the skew below.

One technique is to actually reduce the skew, either by using low skewribbons or employing skew compensation. Fiber-ribbons with about 1 ps/mskew [Siala et al. 1994] and below [Kanjamala and Levi 1995] have beendeveloped, which essentially increases the possible bandwidth distanceproduct. All the fibers in the same ribbon were sequentially cut to reducethe variation of refractive index among the fibers. In the fiber-ribbon linkdescribed in [Wong et al. 1995], a dedicated fiber carries a clock signal usedto clock data on 31 fibers. The transmitter circuitry for each channel has aprogrammable clock skew adjustment to adjust the clock in 80-psincrements.

Another technique is to extract the clock signal from the bit flow on eachfiber instead of using a separate fiber to carry the clock signal. Thedisadvantage is increased hardware complexity when adding a clockrecovery circuit and a buffer circuit for each channel in the receiver. Ahybrid solution is to skip the separate clock channel and encode clockinformation on the data channels while still sending in bit-parallel mode, asreported in [Yoshikawa et al. 1997] [Yoshikawa et al. 1997B]. In this case, adeskew unit relying on FIFO registers (First In First Out) ensures thatparallel data words that are output from the receiver are identical to thosewhich were sent. A possible ± 15-ns deskew was reported. A similar systemis reported in [Fujimoto et al. 1998].

155

The techniques mentioned above introduce either increased hardwarecomplexity or a more sophisticated fiber-ribbon manufacturing process. Ifthe manufacturing process allows for adding more fibers in each ribbon, thismay be a cheaper alternative. For example, 12 channel links with 1 Gbit/sper channel [Karstensen et al. 1995] and 2 Gbit/s per channel [Karstensen1995] have been reported, and array modules supporting 12 × 2.4 Gbit/s for,e.g., fiber-ribbon links were described in [Peall 1995]. A fiber-ribbon linkwith 32 fibers, each with a bit rate of 500 Mbit/s, was described in [Wong etal. 1995], and researchers at NEC have developed a module in which 8 × 2lasers are coupled to two fiber-ribbons [Kasahara 1998]. Instead of fiber-ribbons, fiber imaging guides (FIGs) with thousands of pixels can be used[Li et al. 1995]. In the system described in [Li et al. 1998B], both a 14000-pixel FIG and a 3500-pixel FIG were coupled to an 8 × 8 VCSEL array indifferent setups.

����&DVH�6WXG\

A typical application with high throughput requirements and a pipelineddata flow between the computational modules is future radar signalprocessing systems. In Figure 58, a signal processing chain, similar to thosedescribed in [Jonsson et al. 1996] [Taveniku et al. 1996] is shown togetherwith its bandwidth demands. Each computational module in the figurecontains multiple processors. The chain is a good example containing bothmulticast, one-to-many, and many-to-one communication patterns. Theaggregated throughput demand is 30 Gbit/s. Only the throughputrequirements are treated here; all details on the chain are covered in thetwo papers referred to above. The data flow must not be disturbed by, forexample, status information that the network must also transport. Slotreserving is therefore a good choice for the data flow in the signal processingchain when using the CC-FPR network.

We assume links with ten fibers and 800 Mbit/s per fiber in the case study.In the CC-FPR network, this translates to a bandwidth of 6.4 Gbit/s for datatraffic on eight of the fibers. For simplicity, we assume an efficientbandwidth of 6.0 Gbit/s after, for example, check-sums have been excluded.Figure 58 shows 13 nodes. In addition, the antenna is seen as one node(feeds the first node in the chain with data) and one master node isresponsible for supervising the whole system and interacting with the user.We denote the antenna as node P�, the modules shown in the figure as nodePL, 2 ≤ L ≤ 14, and the master node as P��. The numbers of the modules arealso indicated in the figure. Hence, the number of ordinary slots is 15, butthe cycle is extended so that it also contains 30 slots for reservation.Accordingly, there are 45 slots in a cycle, where one slot per cyclecorresponds to a bandwidth of 133 Mbit/s at a total efficient bandwidth of6.0 Gbit/s.

156

A feasible allocation scheme of the slots is shown in Figure 59. For clarity,all of the reservation slots are placed after the ordinary slots. In a realimplementation, however, there are two ways of spreading the reservationslots, as explained below in Section 9.7. Care must be taken, however, indistributing the reservation slots, the reason being that, when there areintermediate nodes between the source and destination nodes, allocation isnot possible in those slots in which one of the intermediate nodes is the slotinitiator.

The maximum data flow from one module is 4.0 Gbit/s, which corresponds toreserving having a segment of the ring in all of the 30 slots (for reservation).Slots for both of the two 2 Gbit/s data flows to the pulse compression nodescan be allocated, since one of the two data flows is tapped before adding thedata flow produced from the same node. The incoming data flow to theCFAR (Constant False Alarm Ratio) nodes is multicasted to all of these

BeamForming

4.0Gb/s

4.0Gb/s

2.0 Gb/sMulticast

4 x 500Mb/s

2.0Gb/s

ExtractorCFAR

= Computational Module

= Buffer Memory Module

EnvelopeDetection

DopplerFilter Bank

BufferMemory

2 x 2.0 Gb/s Pulse Compression

2 x 2.0Gb/s

BufferMemory

4.0Gb/s

4.0Gb/s2

7

3 54

8

10 11 12 13 14

96

)LJXUH� ���� 'DWD� IORZ� EHWZHHQ� WKH� PRGXOHV� LQ� WKH� UDGDU� VLJQDO� SURFHVVLQJFKDLQ�

157

nodes. Although this multicast data flow must remain unchanged until thelast CFAR node, it can coexist with the data flow produced from the CFARnodes. The reason for this is that the multicast bandwidth is only 2 Gbit/s.The rest of the data flows are pure pipeline flows and fit easily on thenetwork as long as the calculations are mapped on the nodes according tothe pipeline order.

���� 3HUIRUPDQFH�$QDO\VLV

To be able to give guaranteed real-time services, worst-case performancemust be determined. An analysis of how worst-case latency anddeterministic throughput vary with the network design parameters is givenbelow. Each node is assumed to have 1RUG ordinary slots and 1UHV slots forwhich reservation is allowed, per cycle. Two ways of circulating the role ofbeing slot initiator are treated and compared to each other (see Figure 60):Case A, where each node is slot initiator in 1RUG + 1UHV slots in sequence, andCase B, where the slots are interleaved in a way that each cycle is inpractice divided into 1RUG + 1UHV sub-cycles, where each node is slot initiatoronce per sub-cycle. Throughout the section, the total propagation time

'DWD�VORWV/LQNV � � �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� ��

1 − 2 � 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 12 − 3 � 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 23 − 4 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 34 − 5 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 3 3 3 35 − 6 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 56 − 7 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 67 − 8 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 78 − 9 ..... 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 - - - ..... -

9 − 10 - - - - - - - - - - - - - - - 9 9 9 910 − 11 10 10 10 10 - - - - - - - - - - - 9 9 9 911 − 12 10 10 10 10 11 11 11 11 - - - - - - - 9 9 9 912 − 13 10 10 10 10 11 11 11 11 12 12 12 12 - - - 9 9 9 913 − 14 10 10 10 10 11 11 11 11 12 12 12 12 13 13 13 13 - - -14 − 15 - - - - - - - - - - - - - - - - - - -15 − 1 �� - - - - - - - - - - - - - - - - - - -

)LJXUH�����2QH�F\FOH�RI����VORWV�ZKHUH�6ORWV����WKURXJK����DUH�UHVHUYHG��(DFKQXPEHU� LQ� WKH� WDEOH� LQGLFDWHV�� IRU� 6ORWV� ��� WKURXJK� ���� WKH� RZQHU� RI� WKHFRUUHVSRQGLQJ� OLQN�� 1HLJKERULQJ� VHJPHQWV� LQ� HDFK� VORW� KDYH� GLIIHUHQWEDFNJURXQG�VKDGLQJ��7KH�VORW�LQLWLDWRUV�DUH�LQGLFDWHG�LQ�6ORWV���WKURXJK����

158

around the ring is denoted as 7SURS, the skew between a control packet andthe data packet it arbitrates for as 7VNHZ, and 0 is the number of nodes inthe network. The value of 7VNHZ is assumed to be equal to the duration of oneslot, 7VORW, which is set to 1 µs in this analysis.

The worst-case latency a node experiences for guarantee seeking traffic is inthe case in which all reservation-slots are reserved for other traffic and theonly ordinary slots it gains access to are those in which it is the slotinitiator. If a packet is generated just after the node had the chance to send,the node must wait until the next ordinary slot in which it is slot initiator.The latency for Case A will then be:

VNHZSURS

RUGUHVRUGVORWODW

77

111077

+

++−+= )1)((. (19)

The equation holds for best effort traffic as well if no other (best effort orguarantee seeking) traffic is queued in the node to be sent before. Thelatency is plotted in Figure 61 against the number of nodes for differentcombinations of 1RUG and 1UHV, and for different values of 7SURS. The sameworst-case latency, but here for Case B, is:

VNHZUHVVORWSURSODW710777 +++= )1)(( (20)

and is plotted in Figure 62. As seen in the figures, Case B can be moresensitive to large propagation delays. With the same assumptions as above,further comparison between Cases A and B is shown in Figure 63. Case B

Cycle 1+1Cycle 1

O = Ordinary slotR = Reservation slotP

L = Slot-initiator node

7LPH

P3

ORR

P1

ORR

P2

ORR

P3 P3P1P1 P2 P2 P3

ORR

P1

ORR

P2

ORR

P3 P3P1P1 P2 P2

7SURS

7SURS

Cycle 1

RR RR

P3P1 P2

7SURS

R R

P3P1 P2

7SURS

O

P3P1 P2

7SURS

OO

Case A

Case B

)LJXUH�����6ORW�GLVWULEXWLRQ�DFFRUGLQJ�WR�&DVHV�$�DQG�%��,Q�WKH�H[DPSOH��1U� ���1R� ����DQG�0� ���

159

can offer lower latency in some situations in which 1RUG > 1, especially forlarge values of 0.

Even if worst-case parameters for guaranteed services must be determined,the worst-case latency in the case of no other traffic in the network is anessential parameter to form an idea about the more general performance.However, it is still assumed that all reservation slots are reserved for othertraffic. For communication to the nearest neighbor down stream, it ispossible to send in any ordinary slots. If the packet must instead travel overseveral links, the slots for which any of the intermediate nodes is slotinitiator cannot be used because the slot initiator always terminates bothclock and data. The worst-case latency for Case A is:

VNHZSURS

RUGUHVRUGVORWODW

77

111477

+

++−+= )1)(((21)

where 4 is the number of links a packet must pass to come to thedestination node. The gap between each cycle (see Figure 56) is always

0 10 20 30 40 500

50

100

150

200

250

300

Number of nodes

o Nord = 1, Nres = 0, Tprop = 2.5 µs (500 m)+ Nord = 1, Nres = 0, Tprop = 25 µs (5 km)* Nord = 1, Nres = 3, Tprop = 2.5 µs (500 m)x Nord = 1, Nres = 3, Tprop = 25 µs (5 km)

Late

ncy

[µs]

)LJXUH� ���� :RUVW�FDVH� ODWHQF\� ZKHQ� WKH� RQO\� VORWV� D� QRGH� JHWV� DUH� WKRVHRUGLQDU\�VORWV�IRU�ZKLFK�LW�LV�WKH�VORW�LQLWLDWRU��&DVH�$�LV�DVVXPHG�

160

experienced by a node when it is handing over the role of being slot initiatorto the next node. Because the worst-case latency appears when a node justmisses a slot where it is the slot initiator (the last one if there are several ina sequence), the whole 7SURS is always a part of the latency, even if 4 < 0.Because of the similarity between Equations 19 and 21, a plot of Equation21 would look the same as the plot of Equation 19 in Figure 61, except that4 is represented on the x-axis. The same worst-case latency, but here forCase B, is:

)1(

)(

UHVSURS

VNHZUHVVORWODW

17

740177

+

+++=(22)

and varies with respect to both 4 and 0 but not with respect to 1RUG.

For the ability to give a bounded latency coupled to an RTVC, considerationmust be given to the placement of the slots reserved for the RTVC in thecycle. The 1 slots reserved for an RTVC are denoted as VL, 1 ≤ L ≤ 1. Theworst-case latency is:

0 10 20 30 40 500

50

100

150

200

250

300

Number of nodes

o Nord = 1, Nres = 0, Tprop = 2.5 µs (500 m)+ Nord = 1, Nres = 0, Tprop = 25 µs (5 km)* Nord = 1, Nres = 3, Tprop = 2.5 µs (500 m)x Nord = 1, Nres = 3, Tprop = 25 µs (5 km)

Late

ncy

[µs]

)LJXUH� ���� :RUVW�FDVH� ODWHQF\� ZKHQ� WKH� RQO\� VORWV� D� QRGH� JHWV� DUH� WKRVHRUGLQDU\�VORWV�IRU�ZKLFK�LW�LV�WKH�VORW�LQLWLDWRU��&DVH�%�LV�DVVXPHG�

161

}1|

max{___

1L7

71717

VNHZ

SURSLSDVV6,VORWLJDSODW

≤≤

++=(23)

where 1JDSBL is the number of slots between the start of slot VL and the startof slot VL+1 (V0 in next cycle if L = 1). The number of slots for which the sourcenode is handing over the role of being slot initiator to the next node duringthis time is denoted as 16,BSDVVBL.

The maximum aggregated throughput in the network is:

SURSVORWUHVRUG

VORWUHVRUG

70711

037116

+++

=)(

)(max

(24)

where 3 is the average number of transmissions possible per slot as aconsequence of spatial slot reuse and Case A is assumed. The maximumaggregated throughput is plotted in Figure 64 for 3 = 1. When 3 > 1, thethroughput plotted in the figure is simply multiplied by 3. At low values of7SURS, throughputs near 3 can be achieved even for low values of 0. Thecorresponding throughput for Case B is:

0 10 20 30 40 500

50

100

150

200

250

300

Number of nodes

o A, Nord=3, Nres=1, Tprop=50 µs+ B, Nord=3, Nres=1, Tprop=50 µs* A, Nord=1, Nres=6, Tprop=20 µsx B, Nord=1, Nres=6, Tprop=20 µs

A, Nord=1, Nres=3, Tprop=2.5 µsB, Nord=1, Nres=3, Tprop=2.5 µs

Late

ncy

[µs]

)LJXUH�����&RPSDULVRQ�RI�ZRUVW�FDVH�ODWHQF\�IRU�&DVHV�$�DQG�%�

162

SURSVORW

VORW

707

0376

+=

max(25)

and is plotted in Figure 65. As can be seen, the maximum throughput forCase B does not vary with respect to 1RUG and 1UHV. When 1RUG + 1UHV = 1,Case A and Case B give the same aggregated throughput. If 1RUG + 1UHV > 1,however, Case A gives better aggregated throughput.

The same performance difference as in the case of aggregated throughput isfound when looking at the throughput corresponding to one slot per cycle.The single-slot throughput is relevant information for all kinds of traffic,but especially when calculating guaranteed bandwidth for an RTVC. It alsodetermines the minimum throughput a node gets if it only can send in asingle ordinary slot per cycle, i.e., the only ordinary slot for which the nodeis slot initiator if 1RUG = 1. The single slot throughputs are:

SURSVORWUHVRUG

VORW

70711

76

++=

)(max(26)

0 10 20 30 40 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of nodes

Thr

ough

put

* Nord + Nres = 4, Tprop = 2.5 µs (500 m)o Nord + Nres = 1, Tprop = 2.5 µs (500 m)x Nord + Nres = 4, Tprop = 25 µs (5 km)+ Nord + Nres = 1, Tprop = 25 µs (5 km)

)LJXUH�����0D[LPXP�DJJUHJDWHG�WKURXJKSXW��&DVH�$�LV�DVVXPHG�

163

and

))((maxSURSVORWUHVRUG

VORW

70711

76

++= (27)

for Cases A and B, respectively.

���� 6XPPDU\

We have presented a fiber-ribbon based ring network with services forparallel and distributed real-time systems. A key component of the networkarchitecture is the flexible control channel that can be configured to be usedfor different types of network control. Examples of this are the low levelsupport for barrier synchronization, global reduction, and reliabletransmission. A great advantage is that the low level support can beimplemented with little or no modifications to existing hardware. Highthroughputs can be achieved in the network, especially in systems withsome kind of pipelined dataflow between the nodes. The network offers real-

0 10 20 30 40 500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of nodes

Thr

ough

put

x Tprop = 0.025 µs (5 m)* Tprop = 0.25 µs (50 m)+ Tprop = 2.5 µs (500 m)o Tprop = 25 µs (5 km)

)LJXUH�����0D[LPXP�DJJUHJDWHG�WKURXJKSXW��&DVH�%�LV�DVVXPHG�

164

time services for logical connections with guaranteed performance, RTVCs,and for separate messages, best effort and guarantee seeking messages. Ananalysis of worst-case latency and deterministic throughput has beenprovided for two variants of time slot organization. One offers higherthroughput while the other offers lower latency in some situations. Alsoworth mentioning is that the network can be built today using fiber-opticoff-the-shelf components and that this is ongoing work.

165

����3DFNHW�DQG�&LUFXLW�6ZLWFKHG�)LEHU�5LEERQ3LSHOLQH�5LQJ�1HWZRUN

The physical ring of the second ring network proposed in this thesis is sub-divided into two networks carrying different kinds of traffic [Jonsson et al.1997B]. Nine of the fibers are used for time multiplexed circuit switchedtraffic; eight fibers are for data and one is for clocking. The tenth fiber isdedicated to packet switched traffic using, for example, a token ringprotocol. This fiber also carries control messages to reconfigure the TDMAschedule (i.e., circuit establishment) for the other nine fibers. This networkis a good choice when the main data flow in the network does not changerapidly. Compared to the CC-FPR network, the network for both packet andcircuit switched traffic is slightly simpler at the expense of somewhatreduced support of dynamic traffic patterns. However, in many systems,only a fraction of the traffic is, e.g., "bursty".

Circuit switched and packet switched traffic are discussed in Sections 10.1and 10.2, respectively. Section 10.3 describes circuit establishment, while acase study is provided in Section 10.4. Mode changes are treated in Section10.5, and a summary is given in Section 10.6.

�����&LUFXLW�6ZLWFKHG�7UDIILF

For circuit switched traffic, the first nine fibers in each link form a highspeed channel. All of the high speed channels together form a high speedring network for circuit switched traffic. The access is divided into slots asin an ordinary TDMA network. However, in each slot, the network can bedivided into segments as in the CC-FPR network. Also, for each slot, there isalways a slot initiator node. The same kind of asynchronous slotsynchronization method is also used.

The access is cyclic, and each cycle consists of . slots. In a typical case, . isa multiple of 0, where 0 is the number of nodes, and each node is the slotinitiator in ./0 slots. An example of an agreed schedule for a network with. = 0 = 5 slots per cycle is shown in Figure 66. Each column represents onetime slot and contains information on how the ring is segmented in thatslot. Each number in a column is the node index of the owner of thecorresponding link. The bold type numbers indicate the current slotinitiator. In each segment and slot, one, and only one, node can be the ownerof the links and hence has the right to use the segment links fortransmission. In the first slot in the example, node P1 (slot initiator) ownsthe link between itself and node P2. Hence, it can transmit to node P2 butnot to any other node. In the same slot, node P2 can transmit to any of

166

nodes P3, P4, P5, or P1. The choice is made by the process that owns thecircuit (logical connection) to which the slot segment is associated. Amulticast to two or more of these nodes is also possible.

In the third slot, the link between node P1 and node P2 is free. Although thelink is free, node P1 must not disturb the asynchronous slot synchronizationtechnique and therefore transmits an empty packet to node P2. In the fifthslot, node P5 has the capability of transmitting a broadcast packet (a packetto all other nodes in the ring).

The same reasoning about bandwidth utilization according to theasynchronous slot synchronization as in the CC-FPR network holds for thisnetwork as well. The maximum aggregated throughput of the network madepossible by the asynchronous slot synchronization method, 6PD[, is:

SURSVORW

VORW

7.7

.376 +

=max

(28)

where . is the number of slots per cycle, 3 is the average number of packettransmissions in each slot, 7VORW is the duration of one slot, and 7SURS is thetotal propagation delay around the ring.

The latency grows linearly with distance, measured in number of hops(repeating latency in each node). Adding to the latency is also thepropagation delay between source and destination node, as well as the delayuntil the first available slot for transmission. By distributing tasks in such away as to minimize the number of hops, both latency and remainingbandwidth will be improved.

/LQN2ZQHUV

'DWD�VORWV

/LQNV � � � � �

1 − 2 � 5 - 5 5

2 − 3 2 � 2 5 5

3 − 4 2 2 � 3 5

4 − 5 2 2 3 � -

5 − 1 2 5 3 5 �

)LJXUH�����([DPSOH�RI�DQ�DOORFDWLRQ�VFKHPH�IRU�WKH�OLQNVLQ� D� ILYH�QRGH� V\VWHP�� 7KH� VORW� LQLWLDWRUV� DUH� LQ� EROGW\SH��DQG�GLIIHUHQW� VHJPHQWV�KDYH�GLIIHUHQW�EDFNJURXQGVKDGLQJ�

167

����� 3DFNHW�6ZLWFKHG�7UDIILF

The tenth fibers from all of the links are combined to form a ring networktotally dedicated to packet switched traffic. An ordinary ring protocol can beused. However, there are two requirements: (L) it must be possible to haltthe protocol when special packets for circuit establishment are to betransmitted (see Section 10.3), and (LL) the latency must be upper boundedto assure transmission of the packets for circuit establishment. When using,e.g., a token ring protocol on the packet network, this network will supportlow latency communication for sporadic packets at moderate traffic rates. Atthe same time, it is assured that the circuit switched traffic (often real-timetraffic) is not disturbed by packet switched traffic.

�����&LUFXLW�(VWDEOLVKPHQW

When a node is to establish a new circuit, it searches for slots where therequired links are free so that the allocation of a new segment can be made.First, the node’s own slots (i.e., where the node itself is the slot initiator) aresearched. When too few slots (actually only a segment in each slot) for thecircuit can be allocated, the search is continued in other slots. In that case, aspecial UHTXHVW� SDFNHW� is transmitted on the packet network to ask othernodes to allocate the desired segment in their slots. This packet isimmediately followed by a FROOHFW�SDFNHW�to collect information on the successof the slot segment allocations.

The request packet, which is broadcast to all other nodes, containsinformation about the links required and the number of slots needed. Eachnode then checks whether any of its own slots have the required free links.If so, it prepares to modify the collect packet when it arrives (beforeforwarding it), to notify the requesting node of which slots have beenallocated. However, if any of the previous nodes have already allocated slotsegments and modified the collect packet, the number of slots needed wouldbe decreased by the corresponding number of allocated slots. The number ofslots still needed is indicated by a dedicated field in the collect packet. Inthis way, allocation of more than the necessary number of slots is avoided.However, several nodes can each allocate some of the slots needed, andinformation about all of these allocations is added to the same collectpacket.

When the requesting node receives the collect packet after one round, itdecides whether the number of allocated slots is sufficient. If not, it sends arelease packet. Otherwise, it can start using the established circuitimmediately.

168

�����&DVH�6WXG\

With the same assumptions as are made in the case study described inSection 9.6 (Page 155), we will now show that the second pipeline ringnetwork is also feasible for the chosen radar signal processing application.Since we still assume links with ten fibers and 800 Mbit/s per fiber, abandwidth of 6.4 Gbit/s is dedicated to circuit switched traffic on eight of thefibers, while 800 Mbit/s is dedicated to packet switched traffic on one fiber.

We also assume there are still 45 slots per cycle, which gives a bandwidth of133 Mbit/s per cycle and slot. In the case of the second network, this meansthat each cycle is divided into . = 45 slots per cycle for circuit switchedtraffic. The allocation scheme in Figure 59 on Page 157 then also holds forthis network, leaving Slots 1 through 15 free. Another possibility is to have. = 12 slots per cycle. In that case, one slot corresponds to 500 Mbit/s, andall bandwidths in Figure 58 on Page 156 are divisible by the slot bandwidth.

�����0RGH�&KDQJHV

It is possible to use a number of different working modes in a radar system.The task of one mode can, for example, be to scan the whole working range,while the task of another mode may be to track a certain object. Normally,the algorithm mapping and communication patterns are different for twodifferent modes. The change of the circuits at mode changes can beperformed in two different ways: (L) switching between the various slotallocation schemes for a known set of modes, schemes that are staticallystored in each node, and (LL) dynamically changing the slot allocationscheme in each node at a mode change by establishing new circuits, asdescribed in Section 10.3.

In the first case, a mode change request is broadcast by the subsystemresponsible for mode changes (which is the master node). Immediately aftertransmission of the request packet, an acknowledge packet is transmitted.The acknowledge packet is halted in each node until the node is preparedfor the requested mode change. In this way, the master node knows that allinvolved nodes are prepared for the mode change when it receives theacknowledge packet after one turn around the ring. Whether other packetsare allowed during a mode change depends on the tolerable latency of themode change.

In the second case, the mode change is initiated by the master node in thesame manner as a broadcast packet. However, each of the nodes involved isthen responsible for requesting its required bandwidth. Each node alsosends its own acknowledge (or negative acknowledge if it failed to establish

169

the required circuits) packet to the master node, indicating that it isprepared for the mode change.

When all the nodes have been prepared for the mode change, the systemwill change to the new mode in the next batch. The packets coming from theantenna in the new batch will be tagged to indicate the new mode. In thatway, the nodes will be triggered to change to the new mode. Nodes that areplaced later in the signal processing chain are triggered by the packetsgenerated by succeeding nodes. Even if a node has a totally different job todo (and a different communication pattern) in the new mode, it can betriggered in this way. This is possible as long as two different batches aredata independent.

����� 6XPPDU\

We have presented a ring network in which very high throughputs can beachieved, especially in systems having some kind of pipelined dataflowbetween the nodes. The network supports packet switched traffic at thesame time as guaranteed bandwidth is supported through circuit switching.In a typical system, circuits can be set up for time critical dataflows,guaranteeing that they are not disturbed by, e.g., control information. Thesenetwork features are highly appreciated in, e.g., radar signal processingsystems. It is also worth mentioning that the network can be built todayusing fiber-optic off-the-shelf components.

171

����&RQFOXVLRQV�DQG�)XWXUH�:RUN

�����&RQFOXVLRQV

Two different kinds of networks have been in focus, WDM star networks andfiber-ribbon ring networks. Protocols and real-time services were proposedand evaluated for both networks (see Chapters 7 and 9, respectively).Because the WDM star network implements a distributed crossbar it hasbetter support for general communication patterns, compared to the ring.From the perspective of fault-tolerance (which, however, not has beenaddressed in this thesis) the passive optical star seems to be more reliablethan a uni-directional ring, due to its passive nature. On the other hand, thering network allows for simpler transceiver design as a consequence of therelaxed clock synchronization (because of the asynchronous slot-synchronization method), lack of tuning latencies, and the fact that nopropagation delay measurements has to be done. In practice, this can leadto shorter time-slots which, in turn, reduces the latencies. Two otherimportant factors are what kind of traffic (communication pattern etc.) thatshould be carried by the network and how the different technologies willmature.

A star-of-stars WDM network has also been proposed, offering a scalablenetwork architecture (see Chapter 8). Moreover, a simplified variant of thefiber-ribbon ring network with less support for dynamic traffic was proposed(Chapter 10). Again, the choice of network depends much upon things suchas the communication patterns in the application. The WDM star networkas well as the two ring networks have been shown to be feasible in a high-performance radar system, and the star-of-stars network is offered as amore scalable alternative. More general analyses of the networks, with afocus on support for real-time communication have also been presented.

�����2QJRLQJ�DQG�)XWXUH�ZRUN

A network demonstrator of the control channel based ring network iscurrently being built, and some initial tests with two nodes have beenperformed (see Figure 67). Some building blocks included in a node are:

• Optoelectronic conversion and fiber-ribbon attachement (MotorolaOPTOBUS™).

• Protocol processor (a commercial DSP board together with anadditional memory and I/O board).

172

• Dual-ported memories and various control logic (the big cards on whichother cards are mounted − see Figure 67). There are three memorybanks for transmission, reception, and protocol interactions betweenthe two processors, respectively.

• Control channel logic implemented in an FPGA (not shown in thefigure.

• Application processor (a commercial DSP board together with anadditional memory and I/O board).

A layered protocol hierarchy is also being developed for use in thedemonstrator.

Other possible directions for future work are:

• Further development of methods for heterogeneous real-time servicesin high performance networks, i.e., services with different QoS.Funding for a project with this focus has recently been granted.

• Investigation of how similar protocols and network architectures canbe used in free-space systems. Such activities have already been

)LJXUH� ����1HWZRUN�GHPRQVWUDWRU�ZLWK� WZR� QRGHV�� 7KH� FDEOH� EXQGOH� LQ� WKHXSSHU� SDUW� RI� WKH� ILJXUH� FRQWDLQV� WZR� ILEHU�ULEERQV� HQGLQJ� LQ� HDFK� QRGH�V2372%86�PRGXOH�

173

initiated and include collaboration with research groups on photonicsand micro-electronics at Chalmers University of Technology.

• More work with a focus on group communication and similar services.

• Further work on real-time services in hierarchical networks.

• Development of methods for, e.g., system startup, nodeinsertion/deletion, and fault tolerance. Much research efforts have beenpaid to these problems and standards exist for specific LANs.

Finally, it should be noted that the work has been valuable for theindustrial partner, Ericsson Microwave Systems, and may have impact onfuture computer system designs for, e.g., radar signal processing.

175

5HIHUHQFHV

[Acampora and Karol 1989] A. S. Acampora and M. J. Karol, "An overviewof lightwave packet networks," ,(((�1HWZRUN, pp. 29-41, Jan. 1989.

[Adams 1994] J. L. Adams, “Orwell,” &RPSXWHU� 1HWZRUNV� DQG� ,6'16\VWHPV, vol. 26, pp. 771-784, Mar. 1994.

[Agarwal 1991] A. Agarwal, "Limits on interconnection networkperformance," ,(((�7UDQVDFWLRQV�RQ�3DUDOOHO�DQG�'LVWULEXWHG�6\VWHPV, vol.2, no. 4, pp. 398-412, Oct. 1991.

[Agarwal et al. 1995] A. Agarwal, R. Bianchini, D. Chaiken, K. L. Johnson,D. Kranz, J. Kubiatowicz, B.-H. Lim, K. Mackenzie, and D. Yeung, "TheMIT-Alewife machine: architecture and performance," 3URF�� ��QG,QWHUQDWLRQDO� 6\PSRVLD� RQ� &RPSXWHU� $UFKLWHFWXUH� �,6&$���, SantaMargherita Ligure, Italy, June 22-24, 1995.

[Agerwala et al. 1995] T. Agerwala, J. L. Martin, J. H. Mirza, D. C. Sadler,D. M. Dias, and M. Snir, "SP2 system architecture," ,%0�6\VWHPV�-RXUQDO,vol. 34, no. 2, pp. 152-184, 1995.

[Agrawal 1992] G. P. Agrawal, )LEHU�2SWLF�&RPPXQLFDWLRQ�6\VWHPV. JohnWiley & Sons, Inc., 1992, ISBN 0-471-54286-5.

[Ahlander 1996] A. Åhlander, "Using multiple SIMD architectures for multi-channel radar signal processing," /LFHQWLDWH�WKHVLV��'HSDUWPHQW�RI�&RPSXWHU(QJLQHHULQJ�� &KDOPHUV� 8QLYHUVLW\� RI� 7HFKQRORJ\�� *|WHERUJ�� 6ZHGHQ, Nov.1996, ISBN 91-7197-394-x.

[Alexander et al. 1993] S. B. Alexander et al., “A precompetitive consortiumon wide-band all-optical networks,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol.11, no. 5/6, pp. 714-732, May/June 1993.

[Almasi and Gottlieb 1994] G. S. Almasi and A. Gottlieb, Eds., +LJKO\3DUDOOHO� &RPSXWLQJ. The Benjamin/Cummings Publishing Company, Inc.,Redwood City, CA, USA, 1994, ISBN 0-8053-0443-6.

[Anderson and Cornelius 1992] T. M. Anderson and R. S. Cornelius, "High-performance switching with Fibre Channel," 3URF�� ��WK� $QQXDO� ,(((,QWHUQDWLRQDO� &RPSXWHU� &RQIHUHQFH� �&203&21� 6SULQJ� ª���� 'LJHVW, SanFrancisco, CA, USA, Feb. 24-28, 1992, pp. 261-264.

176

[Anderson et al. 1995] T. E. Anderson, D. E. Culler, and D. A. Patterson, "Acase for NOW (Networks of Workstations)," ,(((�0LFUR, vol. 15, no. 1, pp.54-64, Feb. 1995.

[Arnould et al. 1989] E. A. Arnould, F. J. Bitz, E. C. Cooper, H. T. Kung, R.D. Sansom, and P. A. Steenkiste, "The design of Nectar: a networkbackplane for heterogeneous multicomputers," 3URF�� $63/26�,,,, Boston,MA, USA, Apr. 3-6, 1989, pp. 205-216.

[Aronson et al. 1998] L. B. Aronson, B. E. Lemoff, L. A. Buckman, and D. W.Dolfi, “ Low-cost multimode WDM for local area networks up to 10 Gb/s,”,(((� 3KRWRQLFV� 7HFKQRORJ\� /HWWHUV," vol. 10, no. 10, pp. 1489-1491, Oct.1998.

[Arthurs et al. 1988] E. Arthurs, J. M. Cooper, M. S. Goodman, H.Kobrinski, M. Tur, M. P. Vecchi, "Multiwavelength optical crossconnect forparallel-processing computers," (OHFWURQLFV� /HWWHUV, vol. 24, no. 2, pp. 119-120, Jan. 21, 1988.

[Arvind et al. 1991] K. Arvind, K. Ramamritham, and J. A. Stankovic, “Alocal area network architecture for communication in distributed real-timesystems,” -RXUQDO� RI� 5HDO�7LPH� 6\VWHPV, vol. 3, no. 2, pp. 115-147, May1991.

[Bagley et al. 1990] M. Bagley, R. Wyatt, D. J. Elton, H. J. Wickes, P. C.Spurdens, C. P. Seltzer, D. M. Cooper, and W. J. Devlin, "242 nm continuoustuning from a GRIN-SC-MQW-BH InGaAsP laser in an extended cavity,"(OHFWURQLFV�/HWWHUV, vol. 26, no. 4, pp. 267-269, Feb. 15, 1990.

[Bantz and Bauchot 1994] D. F. Bantz and F. J. Bauchot, "Wireless LANdesign alternatives," ,(((�1HWZRUN, pp. 43-53, Mar./Apr. 1994.

[Banu and Dunlop 1992] M. Banu and A. E. Dunlop, "Clock RecoveryCircuits with Instantaneous Locking," (OHFWURQLFV�/HWWHUV, vol. 28, no. 23, pp.2127-2130, 5 Nov., 1992.

[Barry et al. 1996] Barry et al., "All-optical network consortium - ultrafastTDM networks," ,(((�-RXUQDO� RQ�6HOHFWHG�$UHDV� LQ�&RPPXQLFDWLRQV, vol.14, no. 5, pp. 999-1013, June 1996.

[Batcher 1980] K. E. Batcher, “Design of a massively parallel processor,”,(((�7UDQVDFWLRQV�RQ�&RPSXWHUV, vol. 29, no. 9, pp. 836-840, Sept. 1980.

[Batcher 1980B] K. E. Batcher, "Architecture of a massively parallelprocessor," 3URF�� �WK� ,QWHUQDWLRQDO� 6\PSRVLD� RQ� &RPSXWHU� $UFKLWHFWXUH�,6&$���, May 1980.

177

[Beecroft et al. 1994] J. Beecroft, "Meiko CS-2 interconnect ELAN-ELITEdesign," 3DUDOOHO�&RPSXWLQJ, vol. 20, pp. 1627-1638, no. 10, 1994.

[Bengtsson et al. 1993] L. Bengtsson, A. Linde, B. Svensson, and A.Åhlander, “The REMAP massively parallel computer platform for neuralcomputations," 3URF��7KLUG�,QWHUQDWLRQDO�&RQIHUHQFH�RQ�0LFURHOHFWURQLFV�IRU1HXUDO�1HWZRUNV��0,&521(852������, Edinburgh, Scotland, UK, Apr. 6-8, 1993.

[Bengtsson et al. 1994] L. Bengtsson, K. Nilsson, and B. Svensson, “A high-performance embedded massively parallel processing system,” 3URF�� ,(((DQG�(XURPLFUR�&RQIHUHQFH��03&6ª���, Ischia, Italy, May 1994, pp. 201-206.

[Bergenhem and Olsson 1999] C. Bergenhem and J. Olsson, “Protocol suiteand demonstrator for a high performance real-time network,” 0DVWHU�WKHVLV�&HQWUH�IRU�&RPSXWHU�$UFKLWHFWXUH��&&$���+DOPVWDG�8QLYHUVLW\��6ZHGHQ, Jan.1999.

[Bergman et al. 1998] L. Bergman, J. Morookian, and C. Yeh, “An all-opticallong-distance multi-Gbytes/s bit-parallel WDM single-fiber link,” -RXUQDO�RI/LJKWZDYH�7HFKQRORJ\, vol. 16, no. 9, pp. 1577-1582, Sept. 1998.

[Bergman et al. 1998B] L. A. Bergman, C. Yeh, and J. Morookian, ”Towardsthe realization of multi-km × Gbyte/sec bit-parallel WDM single fibercomputer links,” 3URF�� �WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���� Las Vegas, NV, USA,June 15-17, 1998, pp. 218-223.

[Betzos and Mitkas 1998] G. A. Betzos and P. A. Mitkas, ”Performanceevaluation of massively parallel processing architectures with three-dimensional optical interconnections," $SSOLHG�2SWLFV, vol. 37, no. 2, pp. 315-325, Jan. 10, 1998.

[Bhoedjang et al. 1998] R. Bhoedjang, T. Rühl, and H. Bal, "Efficientmulticast on Myrinet using link-level flow control," 3URF��RI�WKH�,QWHUQDWLRQDO&RQIHUHQFH� RQ�3DUDOOHO� 3URFHVVLQJ� �,&33ª���, Minneapolis, MN, USA, Aug.10-14, 1998.

[Bhoedjang et al. 1998B] R. A. F. Bhoedjang, T. Rühl, and H. E. Bal, “User-level network interface protocols,” &RPSXWHU, vol. 31, no. 11, pp. 53-60, Nov.1998.

[Bhuyan et al. 1989] L. N. Bhuyan, Q. Yang, and D. P. Agrawal,“Performance of multiprocessor interconnection networks,” &RPSXWHU, vol.22, no. 2, pp. 25-37, Feb. 1989.

178

[Bhuyan et al. 1989B] L. N. Bhuyan, D. Ghosal, and Q. Yang, “Approximateanalysis of single and multiple ring networks,” ,(((� 7UDQVDFWLRQV� RQ&RPSXWHUV, vol. 38, no. 7, pp. 1027-1040, July 1989.

[Blank 1990] T. Blank, "The MasPar MP-1 architecture," 3URF����WK�$QQXDO,(((� ,QWHUQDWLRQDO� &RPSXWHU� &RQIHUHQFH� �&203&21� 6SULQJ� ª���� 'LJHVW,San Francisco, CA, USA, Feb. 26 - Mar. 2, 1990, pp. 20-24.

[Blumenthal et al. 1994] D. J. Blumenthal, P. R. Prucnal, and J. R. Sauer,”Photonic packet switches: architectures and experimentalimplementations,” 3URFHHGLQJV�RI� WKH� ,(((, vol. 82, no. 11, pp. 1650-1667,Nov. 1994.

[Boden et al. 1995] N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik,C. L. Seitz, J. N. Seizovic, and W.-K. Su, "Myrinet: a gigabit-per-second localarea network," ,(((�0LFUR, vol. 15, no. 1, pp. 29-36, Feb. 1995.

[Boggess and Shirley 1997] T. Boggess and F. Shirley, "High-performancescalable computing for real-time applications," 3URF�� RI� WKH� 6L[WK,QWHUQDWLRQDO� &RQIHUHQFH� RQ� &RPSXWHU� &RPPXQLFDWLRQV� DQG� 1HWZRUNV�,&�1ª���, Las Vegas, NV, USA, Sept. 22-25, 1997, pp. 332-335.

[Bogineni and Dowd 1992] K. Bogineni and P.W. Dowd, "A CollisionlessMultiple Access Protocol for a Wavelength Division Multiplexed Star-Coupled Configuration: Architecture and Performance Analysis," -RXUQDO�RI/LJKWZDYH�7HFKQRORJ\, vol. 10, no. 11, pp. 1688-1699, Nov. 1992.

[Bogineni et al. 1993] K. Bogineni, K. M. Sivalingam, and P. W. Dowd,"Low-complexity multiple access protocols for wavelength-divisionmultiplexed photonic networks," ,(((� -RXUQDO� RQ� 6HOHFWHG� $UHDV� LQ&RPPXQLFDWLRQV, vol. 11, no. 4, pp. 590-604, May 1993.

[Bohr 1998] M. Bohr, “Silicon trends and limits for advancedmicroprocessors,” &RPPXQLFDWLRQV� RI� WKH� $&0, vol. 41, no. 3, pp. 80-87,Mar. 1998.

[Boisseau et al. 1994] M. Boisseau, M. Demange, and J.-M. Munier, +LJK6SHHG�1HWZRUNV. John Wiley & Sons, Ltd., West Sussex, England, UK, 1994,ISBN 0-471-95109-9.

[Borella et al. 1997] M. S. Borella, J. P. Jue, D. Banerjee, B. Ramamurthy,and B. Mukherjee, ”Optical components for WDM lightwave networks,”3URFHHGLQJV�RI�WKH�,(((, vol. 85, no. 8, pp. 1274-1307, Aug. 1997.

[Borella et al. 1998] A. Borella, G. Cancellieri, and F. Chiaraluce,:DYHOHQJWK�'LYLVLRQ�0XOWLSOH�$FFHVV�2SWLFDO�1HWZRUNV. Artech House, Inc.,Norwood, MA, USA, 1998, ISBN 0-89006-657-4.

179

[Borella et al. 1998B] A. Borella, G. Cancellieri, and F. Chiaraluce,:DYHOHQJWK�'LYLVLRQ�0XOWLSOH�$FFHVV�2SWLFDO�1HWZRUNV. Artech House, Inc.,Norwood, MA, USA, pp. 278-282, 1998, ISBN 0-89006-657-4.

[Brackett 1990] C. A. Brackett, "Dense wavelength division multiplexingnetworks: principles and applications," ,(((�-RXUQDO� RQ�6HOHFWHG�$UHDV� LQ&RPPXQLFDWLRQV, vol. 8, no. 6, pp. 948-964, Aug. 1990.

[Brackett 1991] C. A. Brackett, “On the capacity of multiwavelength optical-star packet switches,” ,(((�/76, pp. 33-37, May 1991.

[Brackett 1996] C. A. Brackett, “Foreword: Is there an emerging consensuson WDM networking?,” -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 14, no. 6, pp.936-941, June 1996.

[Buckman et al. 1998] L. Buckman, A. Yuen, K. Giboney, P. Rosenberg, J.Straznicky, K. Wu, and D. Dolfi, "Parallel optical interconnects," 3URF��+RW,QWHUFRQQHFWV�9,, Stanford, CA, USA, Aug. 13-15, 1998, pp. 137-143.

[Bursky 1994] D. Bursky, "Parallel optical links move data at 3 Gbits/s,"(OHFWURQLF�'HVLJQ, vol. 42, no. 24, pp. 79-82, Nov. 21, 1994.

[Carlile 1993] B. R. Carlile, "Algorithms and design: the CRAY APP shared-memory system," 3URF�� ��WK� $QQXDO� ,(((� ,QWHUQDWLRQDO� &RPSXWHU&RQIHUHQFH� �&203&21� 6SULQJ� ª���� 'LJHVW, San Francisco, CA, USA, pp.312-320, 1993.

[Carroll et al. 1993] J. E. Carroll, L. M. Zhang, and M. E. Bray, "A Braggabout lasers," (OHFWURQLFV��&RPPXQLFDWLRQ�(QJLQHHULQJ�-RXUQDO, pp. 325-337, 1993.

[Casavant et al. 1996] T. L. Casavant, P. Tvrdík, and F. Plášil, Eds.,3DUDOOHO�&RPSXWHUV��7KHRU\�DQG�3UDFWLFH. IEEE Computer Society Press, LosAlamitos, CA, USA, 1996, ISBN 0-8186-5162-8.

[Caulfield 1998] H. J. Caulfield, “Perspectives in optical computing,”&RPSXWHU, vol. 31, no. 2, pp. 22-25, Feb. 1998.

[Cerisola et al. 1995] M. Cerisola, T. K. Fong, R. T. Hofmeister, L. G.Kazovsky, C. L. Lu, P. Poggiolini, and D. J. M. Sabido IX, "Subcarriermultiplexing of packet headers in a WDM optical network and a novelultrafast header clock-recovery technique," &RQIHUHQFH� RQ� 2SWLFDO� )LEHU&RPPXQLFDWLRQ��2)&ª���7HFKQLFDO�'LJHVW, pp. 273-274, 1995.

180

[Chamberlain et al. 1998] R. D. Chamberlain, M. A. Franklin, R. B.Krchnavek, and B. H. Baysal, ”Design of an optically-interconnectedmultiprocessor,” 3URF�� �WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���� Las Vegas, NV, USA,June 15-17, 1998, pp. 114-122.

[Chang et al. 1995] Chang et al., "Subcarrier multiplexing and ATM/SONETclear-channel transmission in a reconfigurable multiwavelength all-opticalnetwork test bed," &RQIHUHQFH� RQ� 2SWLFDO� )LEHU� &RPPXQLFDWLRQ�� 2)&ª��7HFKQLFDO�'LJHVW, pp. 269-270, 1995.

[Chang-Hasnain et al. 1991] C. J. Chang-Hasnain, M. W. Maeda, J. P.Harbison, L. T. Florez, and C. Lin, “Monolithic multiple wavelength surfaceemitting laser arrays,” -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 9, no. 12, Dec.1991.

[Chang-Hasnain et al. 1991B] C. J. Chang-Hasnain, J. R. Wullert, J. P.Harbison, L. T. Florez, N. G. Stoffel, and M. W. Maeda, “Rastered,uniformly separated wavelengths emitted from a two-dimensional vertical-cavity surface-emitting laser arrays," $SSOLHG�3K\VLFV�/HWWHUV, vol. 58, no. 1,pp. 31-33, Jan. 7, 1991.

[Chapuran 1991] T. E. Chapuran, S. S. Wagner, R. C. Menendez, H. E.Tohme, and L. A. Wang, "Broadband multichannel WDM transmission withsuperluminescent diodes and LEDS," 3URF�� ,(((� *OREDO7HOHFRPPXQLFDWLRQV�&RQIHUHQFH��*/2%(&20ª���, pp. 612-618, 1991.

[Charlesworth 1998] A. Charlesworth, "Starfire: extending the SMPenvelope," ,(((�0LFUR, vol. 18, no. 1, pp. 39-49, Jan./Feb. 1998.

[Chatterjee and Pawlowski 1999] S. Chatterjee and S. Pawlowski, “All-optical networks,” &RPPXQLFDWLRQV� RI� WKH� $&0, vol. 42, no. 6, pp. 74-83,June 1999.

[Chawki et al. 1995] M. J. Chawki, V. Tholey, E. Delevaque, S. Boj, and E.Gay, "Wavelength reuse in a WDM unidirectional ring network by using afiber-grating add/drop multiplexer," &RQIHUHQFH� RQ� 2SWLFDO� )LEHU&RPPXQLFDWLRQ��2)&ª���7HFKQLFDO�'LJHVW, pp. 272-273, 1995.

[Chen and Georganas 1993] M. Chen and N. D. Georganas, "Multimediadata delivery in high-speed multichannel networks," 3URF�� &DQDGLDQ&RQIHUHQFH� RQ� (OHFWULFDO� DQG� &RPSXWHU� (QJLQHHULQJ� �&&(&(ª���, pp. 636-639, 1993.

181

[Chen et al. 1990] M. Chen, N. R. Dono, and R. Ramaswami, "A media-access protocol for packet-switched wavelength division multiaccessmetropolitan area networks," ,(((� -RXUQDO� RQ� 6HOHFWHG� $UHDV� LQ&RPPXQLFDWLRQV, vol. 8, no. 6, pp. 1048-1057, Aug. 1990.

[Chen et al. 1991] J. Chen, H. Ahmadi, and Y. Ofek, "Performance study ofthe MetaRing with Gb/s links," 3URF�� RI� WKH� ��WK� &RQIHUHQFH� RQ� /RFDO&RPSXWHU�1HWZRUNV��/&1ª���, Minneapolis, MN, USA, Oct. 14-17, 1991, pp.137-143.

[Chen et al. 1998] C.-H. Chen, B. Hoanca, C. B. Kuznia, A. A. Sawchuk, andJ.-M. Wu, ”Architecture and optical system design for TRANslucent SmartPixel ARray (TRANSPAR) chips," 3URF�� 2SWLFV� LQ� &RPSXWLQJ� �2&���,Brugge, Belgium, June 17-20, 1998, pp. 316-319.

[Chen et al. 1998B] C.-H. Chen, B. Hoanca, C. B. Kuznia, A. A. Sawchuk,and J.-M. Wu, ”TRANslucent Smart Pixel ARray (TRANSPAR) chips forhigh throughput networks and SIMD signal processing,” 3URF�� �WK,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���� Las Vegas, NV, USA, June 15-17, 1998, pp.42-49.

[Cheriton et al. 1991] D. R. Cheriton, H. A. Goosen, and P. D. Boyle,“Paradigm: a highly scalable shared-memory multicomputer architecture,”&RPSXWHU, vol. 24, no. 2, pp. 33-46, Feb. 1991.

[Cheung 1990] K.-W. Cheung, "Acoustooptic tunable filters in narrowbandWDM networks: system issues and network applications," ,(((�-RXUQDO�RQ6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 8, no. 6, pp. 1015-1025, Aug. 1990.

[Cheung 1992] K.-W. Cheung, "Design and implementation considerationsfor wavelength-division multiplexed (WDM) photonic dual bus," 3URF��,(((,QWHUQDWLRQDO� &RQIHUHQFH� RQ� &RPPXQLFDWLRQV� �,&&ª���, Chicago, IL, USA,June 1992, pp. 848-854.

[Chiarulli et al. 1994] D. M. Chiarulli, S. P. Levitan, R. P. Melhem, J. P.Teza, and G. Gravenstreter, ”Multiprocessor interconnection networks usingpartitioned optical passive star (POPS) topologies and distributed control,”3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV�0332,ª���� Cancun, Mexico, Apr. 26-27, 1994, pp. 70-80.

[Chieng and Minasian 1994] Y. T. Chieng and R. A. Minasian, “Tunableerbium-doped fiber laser with a reflection Mach-Zehnder interferometer,”,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 6, no. 2, pp. 153-156, Feb. 1994.

182

[Chiang et al. 1996] T.-K. Chiang, S. K. Agrawal, D. T. Mayweather, D.Sadot, C. F. Barry, M. Hickey, and L. G. Kazovsky, "Implementation ofSTARNET: a WDM computer communication network," ,(((� -RXUQDO� RQ6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 14, no. 5, pp. 824-839, June 1996.

[Chipalkatti et al. 1992] R. Chipalkatti, Z. Zhang and A. S. Acampora, “Highspeed communication protocols for optical star coupler using WDM,” 3URF�,1)2&20ª��, Florence, Italy, May 1992, pp. 2124-2133.

[Chlamtac and Ganz 1988] I. Chlamtac and A. Ganz, "A multibus traincommunication (AMTRAC) architecture for high-speed fiber optic networks,",(((�-RXUQDO�RQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 6, no. 6, pp. 903-912, July 1988.

[Cho et al. 1995] W. Cho, C. Shim, M.-L. Song, J.-Y. Lee, and S. B. Lee,"MCDQDB (multi-channel DQDB) using WDM (wavelength divisionmultiplexing)," 3URF�� ,(((� *OREDO� 7HOHFRPPXQLFDWLRQV� &RQIHUHQFH�*/2%(&20ª���, vol. 3, pp. 2205-2209, Nov. 1997.

[Chollet et al. 1996] F. Chollet, J.-P. Goedgebuer, H. Porte, and A. Hamel,“Electrooptic narrow linewidth wavelength tuning and intensity modulationof an erbium fiber ring laser,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 8,no. 8, pp. 1009-1011, Aug. 1996.

[Chraplyvy et al. 1996] Chraplyvy et al., “1-Tb/s transmission experiment,”,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 8, no. 9, pp. 1264-1266, Sept. 1996.

[Christensen and Haney 1997] M. P. Christensen and M. W. Haney, ”Two-bounce free-space arbitrary interconnection architecture,” 3URF�� 0DVVLYHO\3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Montreal,Canada, June 22-24, 1997, pp. 61-67.

[Coldren et al. 1998] L. A. Coldren, E. R. Hegblom, Y. A. Akulova, J. Ko, E.M. Strzelecka, and S. Y. Hu, ”Vertical-cavity lasers for parallel opticalinterconnects,” 3URF�� �WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���� Las Vegas, NV, USA,June 15-17, 1998, pp. 2-10.

[Comer 1995] D. E. Comer, ,QWHUQHWZRUNLQJ�ZLWK�7&3�,3��9RO�,��3ULQFLSOHV�3URWRFROV��DQG�$UFKLWHFWXUH. third edition, Prentice-Hall, Inc., Upper SaddleRiver, NJ, USA, 1995, ISBN 0-13-227836-7.

[Coquin and Cheung 1988] G. A. Coquin and K. W. Cheung, "Electronicallytunable external-cavity semiconductor laser," (OHFWURQLFV�/HWWHUV, vol. 24, no.10, pp. 599-600, May 12, 1988.

183

[Cremer et al. 1992] C. Cremer, N. Emeis, M. Schier, G. Heise, G.Ebbinghaus, and L. Stoll, “Grating spectrograph integrated with photodiode array in InGaAsP/InGaAs/InP,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV,"vol. 4, no. 1, pp. 108-110, Jan. 1992.

[Dally 1992] W. J. Dally, "Virtual-channel flow control," ,(((�7UDQVDFWLRQVRQ�3DUDOOHO�DQG�'LVWULEXWHG�6\VWHPV, vol. 3, no. 2, pp. 194-205, Mar. 1992.

[Dally and Seitz 1987] W. J. Dally and C. L. Seitz, “Deadlock-free messagerouting in multiprocessor interconnection networks,” ,(((�7UDQVDFWLRQV�RQ&RPSXWHUV, vol. 36, no. 5, pp. 547-553, May 1987.

[Dally et al. 1993] W. J. Dally, J. S.Keen, and M. D. Noakes, "The J-Machine architecture and evaluation," 3URF�� ��WK� $QQXDO� ,(((,QWHUQDWLRQDO� &RPSXWHU� &RQIHUHQFH� �&203&21� 6SULQJ� ª���� 'LJHVW, SanFrancisco, CA, USA, pp. 183-188, 1993.

[Dally et al. 1998] W. J. Dally, M.-J. E. Lee, F.-T. An, J. Poulton, and S.Tell, ”High-performance electrical signaling,” 3URF�� �WK� ,QWHUQDWLRQDO&RQIHUHQFH�RQ�0DVVLYHO\�3DUDOOHO�3URFHVVLQJ�XVLQJ�2SWLFDO� ,QWHUFRQQHFWLRQV�0332,ª���� Las Vegas, NV, USA, June 15-17, 1998, pp. 11-16.

[Darcie 1987] T. E. Darcie, “Subcarrier multiplexing for multiple-accesslightwave networks,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol. 5, no. 8, pp.1103-1110, Aug. 1987.

[Davies and Ghani 1983] P. Davies and F. A. Ghani, “Access protocols for anoptical-fibre ring network,” &RPSXWHU� &RPPXQLFDWLRQV, vol. 6, no. 4, pp.185-191, Aug. 1983.

[Davis et al. 1992] E. W. Davis, T. Nordström, and B. Svensson, "Issues andapplications driving research in non-conforming massively parallelprocessors," 3URF�� RI� WKH�1HZ�)URQWLHUV��D�:RUNVKRS�RQ�)XWXUH�'LUHFWLRQ�RI0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ. Scherson Ed., McLean, Virginia, pp. 68-78,1992.

[Decegama 1989] A. L. Decegama, 7KH� 7HFKQRORJ\� RI� 3DUDOOHO� 3URFHVVLQJ�3DUDOOHO� 3URFHVVLQJ� $UFKLWHFWXUHV� DQG� 9/6,�+DUGZDUH� 9ROXPH� ,. Prentice-Hall, Inc., Englewood Cliffs, NJ, USA, 1989, ISBN 0-13-898438-7.

[DeCusatis 1998] C. DeCusatis, “Optical data communications:fundamentals and future directions,” 2SWLFDO� (QJLQHHULQJ, vol. 37, no. 12,pp. 3082-3099, Dec. 1998.

184

[Delorme et al. 1996] F. Delorme, S. Grosmaire, G. Alibert, S. Slempkes,and A. Ougazzaden, “Simple multiwavelength device fabrication techniqueusing a simple-grating holographic exposure,” ,(((� 3KRWRQLFV� 7HFKQRORJ\/HWWHUV," vol. 8, no. 7, pp. 867-869, July 1996.

[Dong et al. 1998] L. Dong, R. Melhem, and D. Mossé, "Time slot allocationfor real-time messages with negotiable distance constraints," 3URF�� 5HDO�7LPH�7HFKQRORJ\�DQG�$SSOLFDWLRQV�6\PSRVLXP, pp. 131-136, 1998.

[Dono et al. 1990] N. R. Dono, P. E. Green, K. Liu, R. Ramaswami, and F. F.Tong, "A wavelength division multiple access network for computercommunication," ,(((�-RXUQDO�RQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 8,no. 6, pp. 983-994, Aug. 1990.

[Dowd 1991] P. W. Dowd, "Random access protocols for high-speedinterprocessor communication based on an optical passive star topology,"-RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 9, no. 6, pp. 799-808, June 1991.

[Dowd and Chu 1994] P. W. Dowd and J. Chu, ”Photonic architectures fordistributed shared memory multiprocessors,” 3URF�� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���� Cancun, Mexico, Apr.26-27, 1994, pp. 151-161.

[Dowd et al. 1993] P. W. Dowd, K. K. Bogineni, A. Aly, and J. A. Perreault,“Hierarchical scalable photonic architectures for high-performance processorinterconnection,” ,(((�7UDQVDFWLRQV�RQ�&RPSXWHUV, vol. 42, no. 9, pp. 1105-1120, Sept. 1993.

[Dowd et al. 1995] P. W. Dowd, J. A. Perreault, J. C. Chu, D. C. Hoffmeister,and D. Crouse, "LIGHTNING: A scalable dynamically reconfigurablehierarchical WDM network for high-performance clustering�"� 3URF�� RI� WKHIRXUWK� ,(((� ,QWHUQDWLRQDO� 6\PSRVLXP� RQ� +LJK� 3HUIRUPDQFH� 'LVWULEXWHG&RPSXWLQJ��+3'&���, Pentagon City, Virginia, Aug. 1995.

[Dowd et al. 1996] P. Dowd et al., “LIGHTNING network and systemsarchitecture,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol. 14, no. 6, pp. 1371-1387, June 1996.

[Duan and Wilmsen 1998] C. Duan and C. W. Wilmsen, "OptoelectronicATM switch using VCSEL and smart detector arrays," 3URF�� 2SWLFV� LQ&RPSXWLQJ��2&���, Brugge, Belgium, June 17-20, 1998, pp. 103-106.

[Duato et al. 1997] J. Duato, S. Yalamanchili, and L. Ni, ,QWHUFRQQHFWLRQ1HWZRUNV�� DQ� (QJLQHHULQJ� $SSURDFK. IEEE Computer Society Press, LosAlamitos, CA, USA, 1997, ISBN 0-8186-7800-3.

185

[Eicken et al. 1995] T. von Eicken, A. Basu, and V. Buch, "Low-latencycommunication over ATM networks using active messages," ,(((� 0LFUR,vol. 15, no. 1, pp. 46-53, Feb. 1995.

[Einstein 1996] T. Einstein, "RACEway Interlink - a real-timemulticomputing interconnect fabric for high-performance VMEbus systems,"90(EXV�6\VWHPV, Spring 1996.

[Elrefaie 1993] A. F. Elrefaie, "Multiwavelength survivable ring networkarchitectures," 3URF�� ,(((� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� &RPPXQLFDWLRQV�,&&ª���, Geneva, Switzerland, May 23-26, 1993, pp. 1245-1251.

[Emerson 1995] S. Emerson, "Evaluation of a data communication model forswitched fibre channel," ,(((�1HWZRUN, pp. 38-44, Nov./Dec. 1995.

[Eng 1988] K. Y. Eng, "A photonic knockout switch for high-speed packetnetworks," ,(((�-RXUQDO�RQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 6, no. 7,pp. 1107-1116, Aug. 1988.

[Engebretsen et al. 1996] D. R. Engebretsen, D. M. Kuchta, R. C. Booth, J.D. Crow, and W. G. Nation, "Parallel fiber-optic SCI links," ,(((�0LFUR, vol.16, no. 1, pp. 20-26, Feb. 1996.

[Eriksen et al. 1995] P. Eriksen, K. Gustafsson, M. Niburg, G. Palmskog, M.Robertsson, and K. Åkermark, "The Apollo demonstrator − new low-costtechnologies for optical interconnects," (ULFVVRQ�5HYLHZ, vol. 72, no. 2, 1995.

[Ettenberg 1998] M. Ettenberg, "Fundamentals of semiconductor lasers,"3KRWRQLFV�6SHFWUD, vol. 32, no. 2, pp. 148-153, 1998.

[Felderman et al. 1994] R. Felderman, A. DeSchon, D. Cohen, and G. Finn,“ATOMIC: a high speed local communciation architecture,” -RXUQDO�RI�+LJK6SHHG�1HWZRUNV, vol. 3, no. 1, pp. 1-30, 1994.

[Feldman et al. 1987] M. R. Feldman, S. C. Esener, C. C. Guest, and S. H.Lee, ”Comparison between optical and electrical interconnects based onpower and speed considerations," $SSOLHG�2SWLFV, vol. 27, no. 9, pp. 1742-1751, May 1, 1988.

[Fibre Systems 1998] "Parallel optics can feed the clamour for speed," )LEUH6\VWHPV� vol. 2, no. 4, pp. 27-28, May 1998.

[Finn and Mason 1996] N. Finn and T. Mason, “ATM LAN emulation,”,(((�&RPPXQLFDWLRQV�0DJD]LQH, no. 6, pp. 96-100, June 1996.

[Flipse 1998] R. Flipse, "Optical switches ease bandwidth crunch,"(XUR3KRWRQLFV, vol. 3, no. 5, pp. 44-45, Aug./Sept. 1998.

186

[Flynn 1966] M. J. Flynn, ”Very high-speed computing systems,”3URFHHGLQJV�RI�WKH�,(((, vol. 54, no. 12, pp. 1901-1909, Dec. 1966.

[Flynn 1972] M. J. Flynn, “Some computer architectures and theireffectiveness,” ,(((�7UDQVDFWLRQV�RQ�&RPSXWHUV, vol. 21, no. 9, pp. 948-960,Sept. 1972.

[Freeman 1998] R. L. Freeman, “Bits, symbols, bauds, and bandwidth,”,(((�&RPPXQLFDWLRQV�0DJD]LQH� vol. 36, no. 4, pp. 96-99, Apr. 1998.

[Frenkel 1992] A. Frenkel, “Pulse frequency division multiplexing is a newway to increase the capacity of a local fiber-optic communications network,”-RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 10, no. 11, Nov. 1992.

[Frenkel and Lin 1988] A. Frenkel and C. Lin, "Inline tunable etalon filterfor optical channel selection in high density wavelength divisionmultiplexed fibre systems," (OHFWURQLFV�/HWWHUV, vol. 24, no. 3, pp. 159-161,Feb. 4, 1988.

[Fujimoto et al. 1998] N. Fujimoto, A. Ishizuka, H. Rokugawa, and K. Mori,“Skew-free parallel optical transmission systems,” -RXUQDO� RI� /LJKWZDYH7HFKQRORJ\, vol. 16, no. 10, pp. 1822-1831, Oct. 1998.

[Fujiwara et al. 1988] M. Fujiwara, N. Shimosaka, M. Nishio, S. Suzuki, S.Yamazaki, S. Murata, and K. Kaede, "A coherent photonic wavelength-division switching system for broadband networks," 3URF�� RI� WKH� ��WK(XURSHDQ� &RQIHUHQFH� RQ� 2SWLFDO� &RPPXQLFDWLRQ� �(&2&ª���, pp. 139-142,1988.

[Gagliardi and Karp 1995] R. M. Gagliardi and S. Karp, 2SWLFDO&RPPXQLFDWLRQV. John Wiley & Sons, Inc., 1995, ISBN 0-471-54287-3.

[Galles 1997] M. Galles, "Spider: a high-speed network interconnect," ,(((0LFUR, vol. 17, no. 1, pp. 34-39, Jan./Feb. 1997.

[Ganz and Gao 1992] A. Ganz and Y. Gao, “A time-wavelength assignmentalgorithm for a WDM star network,” 3URF�� ,1)2&20ª��, Florence, Italy,May 1992, pp. 2144-2150.

[Ganz and Gao 1992B] A. Ganz and Y. Gao, "Traffic scheduling in multipleWDM star systems," 3URF�� ,(((� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ&RPPXQLFDWLRQV��,&&ª���, Chicago, IL, USA, June 1992, pp. 1468-1472.

[Ganz and Koren 1991] A. Ganz and Z. Koren, “WDM passive star -protocols and performance analysis,” 3URF��,1)2&20ª��, Bal Harbour, FL,USA, Apr. 1991, pp. 991-1000.

187

[Garrett and Li 1991] M. W. Garrett and S.-Q. Li, "A study of slot reuse indual bus multiple access networks," ,(((� -RXUQDO� RQ� 6HOHFWHG� $UHDV� LQ&RPPXQLFDWLRQV, vol. 9, no. 2, pp. 248-256, Feb. 1991.

[Geist et al. 1994] A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek,and V. Sunderam, 390�� 3DUDOOHO� 9LUWXDO� 0DFKLQH� −� $� 8VHUV� *XLGH� DQG7XWRULDO� IRU� 1HWZRUNHG� 3DUDOOHO� &RPSXWLQJ. The MIT Press, Cambridge,MA, USA, 1994.

[Georges and Lau 1993] J. B. Georges and K. Y. Lau, ”Self-pulsating laserdiodes as fast--tunable transmitters in subcarrier multiple-accessnetworks,” ,(((�3KRWRQLFV� 7HFKQRORJ\� /HWWHUV," vol. 5, no. 2, pp. 242-245,Feb. 1993.

[Gerstel 1996] O. Gerstel, "On the future of wavelength routing networks,",(((�1HWZRUN, pp. 14-20, Nov./Dec. 1996.

[Giboney et al. 1998] K. S. Giboney, L. B. Aronson, and B. E. Lemoff, "Theideal light source for datanets," ,(((�6SHFWUXP, vol. 35, no. 2, pp. 43-53,Feb. 1998.

[Gidron 1992] R. Gidron, “Teranet: a multi-gigabits per second ATMnetwork,” &RPSXWHU�&RPPXQLFDWLRQV, vol. 15, no. 3, pp. 143-152, Apr. 1992.

[Glance 1996] B. Glance, ”Wavelength-tunable add/drop optical filter,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 8, no. 2, pp. 245-247, Feb. 1996.

[Goh et al. 1998] T. Goh, M. Yasu, K. Hattori, A. Himeno, M. Okuno, and Y.Ohmori, “Low-loss and high-extinction-ratio silica-based strictlynonblocking 16 × 16 thermooptic matrix switch,” ,(((� 3KRWRQLFV7HFKQRORJ\�/HWWHUV," vol. 10, no. 6, pp. 810-812, June 1998.

[Goke and Lipovski 1973] L. R. Goke and G. J. Lipovski, "Banyan networksfor partitioning multiprocessor systems," 3URF�� �VW� ,QWHUQDWLRQDO� 6\PSRVLDRQ�&RPSXWHU�$UFKLWHFWXUH��,6&$���, 1973.

[Goldberg 1994] L. Goldberg, "ATM switching: a brief introduction,"(OHFWURQLF�'HVLJQ, pp. 87-103, Dec. 16, 1994.

[Goldberg 1997] L. Goldberg, "Moving toward the light: new opticsdevelopments reshaping electronics," (OHFWURQLF�'HVLJQ, vol. 45, no. 28, pp.65-74, Dec. 15, 1997.

[Goodman et al. 1984] J. W. Goodman, F. I. Leonberger, S.-Y. Kung, and R.A. Athale, ”Optical interconnections for VLSI systems,” 3URFHHGLQJV� RI� WKH,(((, vol. 72, no. 7, pp. 850-866, July 1984.

188

[Goodman et al. 1988] M. S. Goodman, E. Arthurs, J. M. Cooper, H.Kobrinski, and M. P. Vecchi, "Demonstration of fast wavelength tuning for ahigh performance packet switch," 3URF��RI�WKH���WK�(XURSHDQ�&RQIHUHQFH�RQ2SWLFDO�&RPPXQLFDWLRQ��(&2&ª���, pp. 255-258, 1988.

[Goodman et al. 1990] M. S. Goodman, H. Kobrinski, M. P. Vecchi, R. M.Bulley, and J. L. Gimlett, "The LAMBDANET Multiwavelength Network:Architecture, Applications, and Demonstrations," ,(((�-RXUQDO�RQ�6HOHFWHG$UHDV�LQ�&RPPXQLFDWLRQV, vol. 8, no. 6, Aug. 1990.

[Gottlieb et al. 1982] A. Gottlieb, R. Grishman, C. P. Kruskal, K. P.McAuliffe, L. Rudolph, and M. Snir, "The NYU Ultracomputer − designing aMIMD, shared-memory parallel machine," 3URF���WK�,QWHUQDWLRQDO�6\PSRVLDRQ�&RPSXWHU�$UFKLWHFWXUH��,6&$���, 1982.

[Gottlieb et al. 1983] A. Gottlieb, R. Grishman, C. P. Kruskal, K. P.McAuliffe, L. Rudolph, and M. Snir, “The NYU Ultracomputer − designingan MIMD shared memory parallel computer,” ,(((� 7UDQVDFWLRQV� RQ&RPSXWHUV, vol. 32, no. 2, pp. 175-189, Feb. 1983.

[Gourlay et al. 1998] J. Gourlay, T.-Y. Yang, J. A. B. Dines, J. F. Snowdon,and A. C. Walker, “Development of free-space digital optics in computing,”&RPSXWHU, vol. 31, no. 2, pp. 38-44, Feb. 1998.

[Greaves and Zielinski 1993] D. J. Greaves and K. Zielinski, “TheCambridge backbone network − an overview and preliminary performance,”&RPSXWHU�1HWZRUNV�DQG�,6'1�6\VWHPV, vol. 25, pp. 1127-1133, 1993.

[Green 1991] P. E. Green, Jr., “The future of fiber-optic computer networks,”&RPSXWHU, pp. 78-87, Sept. 1991.

[Green 1993] P. E. Green, )LEHU�2SWLF�1HWZRUNV. Prentice-Hall, Inc., 1993,ISBN 0-13-319492-2.

[Green 1996] P. E. Green, Jr., "Optical networking update," ,(((�-RXUQDORQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 14, no. 5, pp. 764-779, June 1996.

[Green and Ramaswami 1990] P. E. Green and R. Ramaswami, “Directdetection lightwave systems: why pay more?,” ,(((�/&6, pp. 36-49, Nov.1990.

[Grenfeldt 1998] M. Grenfeldt, "ERION − Ericsson optical networking usingWDM technology," (ULFVVRQ�5HYLHZ, vol. 75, no. 3, pp. 132-137, 1998.

189

[Guilfoyle et al. 1998] P. S. Guilfoyle, J. M. Hessenbruch, and R. V. Stone,“Free-space interconnects for high-performance optoelectronic switching,”&RPSXWHU, vol. 31, no. 2, pp. 69-75, Feb. 1998.

[Gupta 1997] N. Gupta, “Acousto-optic tunable filters,” 2SWLFV��3KRWRQLFV1HZV, vol. 8, no. 11, pp. 23-27, Nov. 1997.

[Gustavson and Li 1996] D. B. Gustavson and Q. Li, “The scalable coherentinterface (SCI),” ,(((� &RPPXQLFDWLRQV� 0DJD]LQH� no. 8, pp. 52-63, Aug.1996.

[Ha and Pinkston 1995] J.-H. Ha and T. M. Pinkston, ”The SPEED cachecoherence protocol for an optical multi-access interconnect architecture,”3URF�� �QG� ,QWHUQDWLRQDO�&RQIHUHQFH� RQ�0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� San Antonio, TX, USA, Oct 23-24,1995, pp. 98-107.

[Ha and Pinkston 1997] J.-H. Ha and T. M. Pinkston, “A token-basedchannel access protocol for wavelength division multiplexed opticallyinterconnected multiprocessors,” 3URF�� �QG� :RUNVKRS� RQ� 2SWLFV� DQG&RPSXWHU�6FLHQFH��:2&6�� Geneva, Switzerland, Apr. 1, 1997.

[Habbab et al. 1987] I. M. I. Habbab, M. Kavehrad, and C. W. Sundberg,"Protocols for very high-speed optical fiber local area networks using apassive star topology," -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 5, no. 12, pp.1782-1794, Dec. 1987.

[Hahn 1995] K. H. Hahn, "POLO – Parallel optical links for gigabyte/s datacommunications," 3URF��/(26ª��, San Francisco, CA, USA, Oct. 30 – Nov. 2,1995, vol. 1, pp. 228-229.

[Hahn 1995B] K. H. Hahn, "POLO − parallel optical links for Gigabyte datacommunications," 3URF��RI�WKH���WK�(OHFWURQLFV�&RPSRQHQWV�DQG�7HFKQRORJ\&RQIHUHQFH��(&7&ª���, pp. 368-375, 1995.

[Hahn et al. 1995] K. H. Hahn et al., "POLO: parallel optical links forworkstation clusters and switching systems," &RQIHUHQFH� RQ� 2SWLFDO� )LEHU&RPPXQLFDWLRQ��2)&ª���7HFKQLFDO�'LJHVW, pp. 112-112, 1995.

[Hahn et al. 1996] K. H. Hahn, K. S. Giboney, R. E. Wilson, J. Straznicky,E. G. Wong, M. R. Tan, R. T. Kaneshiro, D. W. Dolfi, E. H. Mueller, A. E.Plotts, D. D. Murray, J. E. Marchegiano, B. L. Booth, B. J. Sano, B.Madhavan, B. Raghavan, A. F. J. Levi, "Gigabyte/s data communicationswith the polo parallel optical link," 3URF��RI�WKH���WK�(OHFWURQLFV�&RPSRQHQWVDQG�7HFKQRORJ\�&RQIHUHQFH��(&7&ª���, May 1996.

190

[Halsall 1995] F. Halsall, 'DWD� &RPPXQLFDWLRQV�� &RPSXWHU�1HWZRUNV� DQG2SHQ� 6\VWHPV� fourth edition, Addison-Wesley Longman Ltd., Essex, UK,1995, ISBN 0-201-42293-X.

[Hamanaka 1991] K. Hamanaka, “Otical bus interconnection system usingselfloc lenses,” 2SWLFV�/HWWHUV, vol. 16, no. 6, pp. 1222-1224, Aug. 15, 1991.

[Hamel et al. 1995] A. Hamel, D. Laville, A. Tromeur, P. Mouazan, T. BenMeriem, R Richier, E. Pelletier, E. Tanguy, and S. Menard, "Multilayer add-drop multiplexers in a self-healing WDM ring network," &RQIHUHQFH� RQ2SWLFDO�)LEHU�&RPPXQLFDWLRQ��2)&ª���7HFKQLFDO�'LJHVW, pp. 84-85, 1995.

[Hariri and Lu 1996] S. Hariri and B. Lu, "ATM-based parallel anddistributed computing," in 3DUDOOHO�DQG�GLVWULEXWHG�FRPSXWLQJ�KDQGERRN. A.Y. Zomaya Ed., McGraw-Hill, Inc., pp. 811-838, 1996.

[Hartman et al. 1990] D. H. Hartman, L. A. Reith, S. F. Habiby, G. R. Lalk,B. L. Booth, J. E. Marchegiano, and J. L. Hohman, "Power economy usingpoint-to-point optical interconnects," in 0LFURHOHFWURQLF� ,QWHUFRQQHFWV� DQG3DFNDJHV�� 6\VWHP� DQG� 3URFHVV� ,QWHJUDWLRQ�� 3URF�� 63,(� YRO�� ����, S. K.Tewksbury and J. R. Carruthers, Eds., pp. 368-376, 1990.

[Hecht 1987] E. Hecht; with contributions by A. Zajac, 2SWLFV. Addison-Wesley, Inc., 1987, ISBN 0-201-11611-1.

[Hennessy and Patterson 1996] J. L. Hennessy and D. A. Patterson,&RPSXWHU� $UFKLWHFWXUH�� $� 4XDQWLWDWLYH� $SSURDFK. Morgan KaufmannPublishers, Inc., San Francisco, CA, USA, 1996, ISBN 0-55860-329-8.

[Hill 1989] G. R. Hill, “Wavelength domain optical network techniques,”3URFHHGLQJV�RI�WKH�,(((, vol. 77, no. 1, pp. 121-132, Jan. 1989.

[Hillis and Tucker 1993] W. D. Hillis and L. W. Tucker, “The CM-5Connection Machine: a scalable supercomputer,” &RPPXQLFDWLRQV� RI� WKH$&0, vol. 36, no. 11, pp. 31-40, Nov. 1993.

[Hinton and Szymanski 1995] H. S. Hinton and T. H. Szymanski,”Intelligent optical backplanes,” 3URF�� �QG� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª����San Antonio, TX, USA, Oct 23-24, 1995, pp. 133-143.

[Hirabayashi et al. 1998] K. Hirabayashi, T. Yamamoto, and S. Hino,“Optical backplane with free-space optical interconnections using tunablebeam deflectors and a mirror for bookshelf-assembled terabit per secondclass asynchronous transfer mode switch,” 2SWLFDO�(QJLQHHULQJ, vol. 37, no.4, pp. 1332-1342, Apr. 1998.

191

[Hockney and Jesshope 1988] R. W. Hockney and C. R. Jesshope, 3DUDOOHO&RPSXWHUV� ��� $UFKLWHFWXUH�� 3URJUDPPLQJ� DQG� $OJRULWKPV. Adam Hilger,Bristol, UK, 1988, ISBN 0-85274-812-4.

[Hopper and Needham 1988] A. Hopper and R. M. Needham, “TheCambridge fast ring networking system,” ,(((�7UDQVDFWLRQV�RQ�&RPSXWHUV,vol. 37, no. 10, pp. 1214-1223, Oct. 1988.

[Hord 1990] R. M. Hord, 3DUDOOHO� 6XSHUFRPSXWLQJ� LQ� 6,0'�$UFKLWHFWXUHV.CRC Press, Inc., Boca Raton, FL, USA, 1990, ISBN 0-8493-4271-6.

[Hord 1993] R. M. Hord, 3DUDOOHO�6XSHUFRPSXWLQJ� LQ�0,0'�$UFKLWHFWXUHV.CRC Press, Inc., Boca Raton, FL, USA, 1993, ISBN 0-8493-4417-4.

[Horowitz et al. 1998] M. Horowitz, C.-K. K. Yang, and S. Sidiropoulos,"High-speed electrical signaling: overview and limitations," ,(((�0LFUR, vol.18, no. 1, pp. 12-24, Jan./Feb. 1998.

[Horst 1995] R. W. Horst, "TNet: a reliable system area network," ,(((0LFUR, vol. 15, no. 1, pp. 37-45, Feb. 1995.

[Huang and Sheu 1996] N.-F. Huang and S.-T. Sheu, “A wavelengthreusing/sharing access protocol for multichannel photonic dual busnetworks,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol. 14, no. 5, pp. 678-692,May 1996.

[Huang and Sheu 1997] N.-F. Huang and S.-T. Sheu, “An efficientwavelength reusing/migrating/sharing protocol for dual bus lightwavenetworks,” -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 15, no. 1, pp. 62-75, Jan.1997.

[Huffaker and Deppe 1996] D. L. Huffaker and D. G. Deppe,”Multiwavelength, densely-packed 2 × 2 vertical-cavity surface-emittinglaser array fabricated using selective oxidation,” ,(((�3KRWRQLFV�7HFKQRORJ\/HWWHUV," vol. 8, no. 7, pp. 858-860, July 1996.

[Humblet et al. 1992] P. A. Humblet, R. Ramaswami, and K. N. Sivarajan,"An efficient communication protocol for high-speed packet-switchedmultichannel networks," 3URF�� 6,*&200ª��, Baltimore, MD, USA, Aug.17-20, 1992, pp. 2-13.

[Humblet et al. 1993] P. A. Humblet, R. Ramaswami, and K. N. Sivarajan,"An efficient communication protocol for high-speed packet-switchedmultichannel networks," ,(((� -RXUQDO� RQ� 6HOHFWHG� $UHDV� LQ&RPPXQLFDWLRQV, vol. 11, no. 4, pp. 568-578, May 1993.

192

[Hwang 1993] K. Hwang, $GYDQFHG� &RPSXWHU� $UFKLWHFWXUH. McGraw-Hill,Inc., 1993, ISBN 0-07-031622-8.

[Hwang and Briggs 1985] K. Hwang and F. A. Briggs, &RPSXWHU$UFKLWHFWXUH� DQG�3DUDOOHO� 3URFHVVLQJ. McGraw-Hill, Inc., 1985, ISBN 0-07-031556-6.

[IEEE 1993] ,(((�6WDQGDUG� IRU� 6FDODEOH� &RKHUHQW� ,QWHUIDFH� �6&,�. IEEE,New York, NY, USA, 1993, ISBN 1-55937-222-2.

[Imai et al. 1994] K. Imai, T. Ito, H. Kasahara, and N. Morita, “ATMR:asynchronous transfer mode ring protocol,” &RPSXWHU�1HWZRUNV�DQG� ,6'16\VWHPV, vol. 26, pp. 785-798, Mar. 1994.

[Irakliotis and Mitkas 1998] L. J. Irakliotis and P. A. Mitkas, “Optics: amaturing technology for better computing,” &RPSXWHU, vol. 31, no. 2, pp. 36-37, Feb. 1998.

[Irshid and Kavehrad 1992] M. I. Irshid and M. Kavehrad, “A fullytransparent fiber-optic ring architecture for WDM networks,” -RXUQDO� RI/LJKWZDYH�7HFKQRORJ\, vol. 10, no. 1, pp. 101-108, Jan. 1992.

[Isenstein 1994] B. Isenstein, "Scaling I/O bandwidth with multiprocessors,"(OHFWURQLF�'HVLJQ, June 13, 1994.

[Ishihata et al. 1997] H. Ishihata, M. Takahashi, and H. Sato, " Hardware ofAP3000 scalar parallel server," )XMLWVX� 6FLHQWLILF� DQG� 7HFKQLFDO� -RXUQDO�vol. 33, no. 1, pp. 24-30, June 1997.

[Ishikawa and McArdle 1998] M. Ishikawa and N. McArdle, “Opticallyinterconnected parallel computing systems,” &RPSXWHU, vol. 31, no. 2, pp. 61-68, Feb. 1998.

[Jafari et al. 1980] H. Jafari, T. G. Lewis, and J. D. Spragins, “Simulation ofa class of ring-structured networks,” ,(((�7UDQVDFWLRQV�RQ�&RPSXWHUV, vol.29, no. 5, pp. 385-392, May 1980.

[Jahns 1994] J. Jahns, ”Planar packaging of free-space opticalinterconnects,” 3URFHHGLQJV�RI�WKH�,(((, vol. 82, no. 11, pp. 1623-1631, Nov.1994.

[Jahns 1998] J. Jahns, ”Integrated free-space optical interconnects for chip-to-chip communications,” 3URF�� �WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\3DUDOOHO� 3URFHVVLQJ�XVLQJ�2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Las Vegas,NV, USA, June 15-17, 1998, pp. 20-23.

193

[Jain 1993] R. Jain, “FDDI: current issues and future plans,” ,(((&RPPXQLFDWLRQV�0DJD]LQH, no. 9, pp. 98-105, Sept. 1993.

[Jajszczyk and Mouftah 1993] A. Jajszczyk and H. T. Mouftah, “Photonicfast packet switching,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH, vol. 31, no. 2, pp.58-65, Feb. 1993.

[Janniello et al. 1992] F. J. Janiello, R. Ramaswami, and D. G. Steinberg, "Aprototype circuit-switched multi-wavelength optical metropolitan-areanetwork," 3URF��,(((�,QWHUQDWLRQDO�&RQIHUHQFH�RQ�&RPPXQLFDWLRQV��,&&ª���,Chicago, IL, USA, June 1992, pp. 818-823.

[Janniello et al. 1995] F. J. Janniello, R. A. Neuner, R. Ramaswami, and P.E. Green, Jr., "Multiplex-protocol optical-fiber multiplexer for remotecomputer interconnection," &RQIHUHQFH� RQ� 2SWLFDO� )LEHU� &RPPXQLFDWLRQ�2)&ª���7HFKQLFDO�'LJHVW, pp. 163-164, 1995.

[Jeong and Goodman 1996] G. Jeong and J. W. Goodman, “Long-distanceparallel data link using WDM transmission with bit-skew compensation,”-RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 14, no. 5, pp. 655-660, May 1996.

[Jia and Mukherjee 1996] F. Jia and B. Mukherjee, “MultiS-Net: a high-capacity, packet-switched, multichannel, single-hop architecture andprotocol for a local lightwave network,” -RXUQDO� RI� +LJK� 6SHHG� 1HWZRUNV,vol. 5, no. 3, pp. 221-241, 1996.

[Jia et al. 1995] F. Jia, B. Mukherjee, and J. Iness, "Scheduling variable-length messages in a single-hop multichannel local lightwave network,",(((�$&0�7UDQVDFWLRQV�RQ�1HWZRUNLQJ, vol. 3, no. 4, Aug. 1995.

[Jiang et al. 1995] C.-L. Jiang, D. J. Brown, E. H. Mueller, A. E. Plotts, E.Cornejo, S. O'Neill, A. J. Heiney, and B. H. Reysen, "LED based paralleloptical links," 3URF�� /(26ª��, San Francisco, CA, USA, Oct. 30 – Nov. 2,1995, vol. 1, pp. 224-225.

[Jiang et al. 1998] W. Jiang, L. J. Norton, P. Kiely, D. B. Schwartz, B.Gable, M. Lebby, and G. Raskin, “Vertical cavity surface emitting laser-based parallel optical data link,” 2SWLFDO� (QJLQHHULQJ, vol. 37, no. 12, pp.3113-3118, Dec. 1998.

[Jonsson 1998] M. Jonsson, “Control-channel based fiber-ribbon pipelinering network," 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���, Las Vegas, NV, USA, June 15-17, 1998, pp.158-165.

194

[Jonsson 1998B] M. Jonsson, “Two fiber-ribbon ring networks for paralleland distributed computing systems,” 2SWLFDO�(QJLQHHULQJ, vol. 37, no. 12, pp.3196-3204, Dec. 1998.

[Jonsson and Moen 1994] M. Jonsson, U. Moen, “A high performance fiber-optic time-deterministic communication network,” 0DVWHU� WKHVLV��&HQWUH� IRU&RPSXWHU�6\VWHPV�$UFKLWHFWXUH��&&$���+DOPVWDG�8QLYHUVLW\��6ZHGHQ, 1994.

[Jonsson and Svensson 1997] M. Jonsson and B. Svensson, “On inter-clustercommunication in a time-deterministic WDM star network,” WR� DSSHDU� LQ3URF�� �QG� :RUNVKRS� RQ� 2SWLFV� DQG� &RPSXWHU� 6FLHQFH� �:2&6�, Geneva,Switzerland, Apr. 1, 1997.

[Jonsson et al. 1995] M. Jonsson, K. Nilsson, and B. Svensson, “A fiber-opticinterconnection concept for scaleable massively parallel computing,” 3URF�0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���,San Antonio, TX, USA, Oct. 23-24, 1995, pp. 313-320.

[Jonsson et al. 1996] M. Jonsson, A. Åhlander, M. Taveniku, and B.Svensson, “Time-deterministic WDM star network for massively parallelcomputing in radar systems,” 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���, Maui, HI, USA, Oct. 27-29, 1996, pp.85-93.

[Jonsson et al. 1997] M. Jonsson, K. Börjesson, and M. Legardt, “Dynamictime-deterministic traffic in a fiber-optic WDM star network,” 3URF�� �WK(XURPLFUR� :RUNVKRS� RQ� 5HDO� 7LPH� 6\VWHPV, Toledo, Spain, June 11-13,1997, pp. 25-33.

[Jonsson et al. 1997B] M. Jonsson, B. Svensson, M. Taveniku, and A.Åhlander, “Fiber-ribbon pipeline ring network for high-performancedistributed computing systems,” 3URF��,QWHUQDWLRQDO�6\PSRVLXP�RQ�3DUDOOHO$UFKLWHFWXUHV��$OJRULWKPV� DQG�1HWZRUNV� �,�63$1ª���� Taipei, Taiwan, Dec.18-20, 1997, pp. 138-143.

[Jonsson et al. 1999] M. Jonsson, C. Bergenhem, and J. Olsson, “Fiber-ribbon ring network with services for parallel processing and distributedreal-time systems,” 3URF�� ,6&$� ��WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 3DUDOOHODQG�'LVWULEXWHG�&RPSXWLQJ�6\VWHPV��3'&6����, Fort Lauderdale, FL, USA,Aug. 18-20, 1999, pp. 94-101.

[Jonsson et al. 1999B] M. Jonsson, C. Bergenhem, and J. Olsson, “Fiber-ribbon ring network with services for parallel processing and distributedreal-time systems,” 3URF�� 61$57��� 5HDO�7LPH� 6\VWHPV� &RQIHUHQFH,Linköping, Sweden, Aug. 24-25, 1999, pp. 46-54.

195

[Juma 1996] S. Juma, “Bragg gratings boost data transmission rates,” /DVHU)RFXV�:RUOG, vol. 32, no. 11, pp. S5-S9, Nov. 1996.

[Kam et al. 1998] A. C. Kam, K.-Y. Siu, R. A. Barry, and E. A. Swanson,"Toward best-effort services over WDM networks with fair access andminimum bandwidth guarantee," ,(((� -RXUQDO� RQ� 6HOHFWHG� $UHDV� LQ&RPPXQLFDWLRQV, vol. 16, no. 7, pp. 1024-1039, Sept. 1998.

[Kam et al. 1998B] A. C. Kam, K.-Y. Siu, R. A. Barry, and E. A. Swanson, “Acell switching WDM broadcast LAN with bandwidth guarantee and fairaccess,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol. 16, no. 12, pp. 2146-2168,Dec. 1998.

[Kameda et al. 1993] T. Kameda, H. Mori, S. Onuki, T. Kikugawa, Y.Takahashi, F. Tsuchiya, and H. Nagai, “A DBR laser employing passive-section heaters, with 10.8 nm tuning range and 1.6 MHz linewidth,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 5, no. 6, pp. 608-610, June 1993.

[Kaminow 1989] I. P. Kaminow, "Non-coherent photonic frequency-multiplexed access networks," ,(((�1HWZRUN, pp. 4-12, Mar. 1989.

[Kaminow et al. 1988] I. P. Kaminow, P. P. Iannone, J. Stone, and L. W.Stulz, “FDMA-FSK star network with a tunable optical filterdemultiplexer,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol. 6, no. 9, pp. 1406-1414, Sept. 1988.

[Kaminow et al. 1989] I. P. Kaminow, P. P. Iannone, J. Stone, and L. W.Stulz, "A tunable vernier fiber Fabry-Perot filter for FDM demultiplexingand detection,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 1, no. 1, pp. 24-26,Jan. 1989.

[Kanjamala and Levi 1995] A. P. Kanjamala and A. F. J. Levi, "Sub-picosecond skew in multimode fiber ribbon for synchronous datatransmission," (OHFWURQLFV�/HWWHUV, vol. 31, pp. 1376-1377, 1995.

[Karstensen 1995] H. Karstensen, "Parallel optical links – PAROLI, a lowcost 12-channel optical interconnection," 3URF��/(26ª��, San Francisco, CA,USA, Oct. 30 – Nov. 2, 1995, vol. 1, pp. 226-227.

[Karstensen et al. 1995] H. Karstensen, C. Hanke, M. Honsberg, J.-R.Kropp, J. Wieland, M. Blaser, P. Weger, and J. Popp, “Parallel opticalinterconnection for uncoded data transmission with 1 Gb/s-per-channelcapacity, high dynamic range, and low power consumption,” -RXUQDO� RI/LJKWZDYH�7HFKQRORJ\, vol. 13, no. 6, pp. 1017-1030, June 1995.

196

[Karstensen et al. 1998] H. Karstensen, J. Wieland, R. Dal’Ara, and M.Blaser, “Parallel optical link for multichannel interconnections at gigabitrate,” 2SWLFDO�(QJLQHHULQJ, vol. 37, no. 12, pp. 3119-3123, Dec. 1998.

[Kasahara 1998] K. Kasahara, "Optical interconnects speed up networks,"3KRWRQLFV�6SHFWUD, vol. 32, no. 2, pp. 127-128, 1998.

[Kato et al. 1993] K. Kato, K. Okamoto, H. Okazaki, Y. Ohmori, and I.Nishi, ”Packaging of large-scale integrated-optic NxN star couplers,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV, vol. 4, no. 3, pp. 348-351, Mar. 1993.

[Kato et al. 1998] T. Kato, J. Sasaki, T. Shimoda, H. Hatakeyama, T.Tamanuki, M. Yamaguchi, M. Kitamura, and M. Itoh, "10 Gb/s photonic cellswitching with hybrid 4×4 optical matrix switch module on silica basedplanar waveguide platform," 2SWLFDO� )LEHU� &RPPXQLFDWLRQ� &RQIHUHQFH�2)&ª���7HFKQLFDO�'LJHVW, San Jose, CA, USA, Feb. 22-27, 1998, pp. 437-440.

[Kawai et al. 1995] S. Kawai, H. Kurita, and K. Kubota, ”Design of electro-photonic computer-networks with non-blocking and self-routing functions,"2SWLFDO� &RPSXWLQJ�� YRO�� ���� ����� 26$� 7HFKQLFDO� 'LJHVW� 6HULHV, Salt LakeCity, Utah, Mar. 13-16, 1995, pp. 263-265.

[Kazovsky et al. 1994] L. G. Kazovsky, T. Fong, and T. Hofmeister, “Opticallocal area network technologies,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH, vol. 32,no. 12, pp. 50-54, Dec. 1993.

[Keiser 1991] G. Keiser, 2SWLFDO�)LEHU�&RPPXQLFDWLRQV. McGraw-Hill, Inc.,1991, ISBN 0-07-100785-7.

[Kermani and Kleinrock 1979] P. Kermani and L. Kleinrock, “Virtual cut-through: a new computer communication switching technique,” &RPSXWHU1HWZRUNV, vol. 3, pp. 267-286, 1979.

[Kessler and Schwarzmeier 1993] R. E. Kessler and J. L. Schwarzmeier,"CRAY T3D: a new dimension for Cray Research," 3URF����WK�$QQXDO�,(((,QWHUQDWLRQDO� &RPSXWHU� &RQIHUHQFH� �&203&21� 6SULQJ� ª���� 'LJHVW, SanFrancisco, CA, USA, pp. 176-182, 1993.

[Kiefer and Swanson 1995] D. R. Kiefer and V. W. Swanson,”Implementation of optical clock distribution in a supercomputer," �2SWLFDO&RPSXWLQJ��YRO�����������26$�7HFKQLFDO�'LJHVW�6HULHV, Salt Lake City, Utah,Mar. 13-16, 1995, pp. 260-262.

197

[Kirkby 1990] P. A. Kirkby, "Multichannel wavelength-switchedtransmitters and recievers-new component concept for broad-band networksand distributed switching systems," -RXUQDO� RI�/LJKWZDYH�7HFKQRORJ\, vol.8, no. 2, Feb. 1990.

[Kobrinski and Cheung 1989] H. Kobrinski and K.-W. Cheung,“Wavelength-tunable optical filters: applications and technologies,” ,(((&RPPXQLFDWLRQV�0DJD]LQH, no. 10, pp. 53-63, Oct. 1989.

[Kobrinski et al. 1987] H. Kobrinski, R. M. Bulley, M. S. Goodman, M. P.Vecchi, C. A. Bracket, L. Curtis, and J. L. Gimlett, "Demonstration of highcapacity in the Lambdanet architecture: a multiwavelength opticalnetwork," (OHFWURQLFV�/HWWHUV, vol. 23, no. 16, pp. 824-826, July 30, 1987.

[Kobrinski et al. 1988] H. Kobrinski, M. P. Vecchi, E. L. Goldstein, and R.M. Bulley, "Wavelength selection with nanosecond switching times usingdistributed-feedback laser amplifiers," (OHFWURQLFV�/HWWHUV, vol. 24, no. 15, pp.969-971, July 21, 1988.

[Kobrinski et al. 1988B] H. Kobrinski, E. Arthurs, R. M. Bulley, J. M.Cooper, E. L. Goldstein, M. S. Goodman, and M. P. Vecchi, "Anoptoelectronic packet switch utilizing fast wavelength tuning," 3URF�� ,(((*OREDO�7HOHFRPPXQLFDWLRQV�&RQIHUHQFH��*/2%(&20ª���, pp. 948-953, 1988.

[Kobrinski et al. 1990] H. Kobrinski, M. P. Vecchi, M. S. Goodman, E. L.Goodman, E. L. Goldstein, T. E. Chapuran, J. M. Cooper, M. Tur, C. E. Zah,and S. G. Menocal, Jr., "Fast wavelength-switching of laser transmittersand amplifiers," ,(((�-RXUQDO�RQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 8,no. 6, pp. 1190-1202, Aug. 1990.

[Koeninger et al. 1994] R. K. Koeninger, M. Furtney, and M. Walker, "Ashared memory MPP from Cray Research," 'LJLWDO�7HFKQLFDO�-RXUQDO, vol. 6,no. 2, pp. 8-21, 1994.

[Kopetz 1997] H. Kopetz, 5HDO�7LPH� 6\VWHPV� −� 'HVLJQ� 3ULQFLSOHV� IRU'LVWULEXWHG�(PEHGGHG�$SSOLFDWLRQV. Kluwer Academic Publishers, Norwell,MA, USA, 1997, ISBN 0-7293-9894-7.

[Kopetz and Grünsteidl 1994] H. Kopetz and G. Grünsteidl, “TTP − aprotocol for fault-tolerant real-time systems,” &RPSXWHU, vol. 27, no. 1, pp.14-23, Jan 1994.

[Kopetz et al. 1989] H. Kopetz, A. Damm, C. Koza, M. Mulazzani, W.Schwabl, C. Senft, and R. Zainlinger, "Distributed fault-tolerant real-timesystems: the Mars approach," ,(((� 0LFUR, vol. 9, no. 1, pp. 25-40, Feb.1989.

198

[Krisnamoorthy et al. 1996] A. V. Krisnamoorthy, J. E. Ford, K. W. Goosen,J. A. Walker, B. Tseng, S. P. Hui, J. E. Cunningham, W. Y. Jan, T. K.Woodward, M. C. Nuss, R. G. Rozier, F. E. Kiamilev, and D. A. B. Miller,”The AMOEBA chip: an optoelectronic switch for multiprocessor networkingusing dense-WDM,” 3URF�� �UG� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Maui, HI,USA, Oct. 27-29, 1996, pp. 94-100.

[Kuchta et al. 1998] D. M. Kuchta, J. Crow, P. Pepeljugoski, K. Stawiasz, J.Trewhella, D. Booth, W. Nation, C. DeCusatis, and A. Muszynski, ”Low cost10 gigabit/s optical interconnects for parallel processing,” 3URF�� �WK,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���� Las Vegas, NV, USA, June 15-17, 1998, pp.210-215.

[Kuck et al. 1993] D. Kuck, E. Davidson, D. Lawrie, A. Sameh, C.-Q. Zhu, A.Vaidenbaum, J. Konicek, P. Yew, K. Gallivan, W. Jallby, H. WijshoffR.Bramley, U. M. Yang, P. Emrath, D. Padua, R. Eigenmann, J. Hoeflinger,G. Jaxon, Z. Li, T. Murphy, J. Andrews, and S. Turner, "The Cedar systemand an initial performance study," 3URF�� ��WK� ,QWHUQDWLRQDO� 6\PSRVLD� RQ&RPSXWHU�$UFKLWHFWXUH��,6&$���, San Diego, CA, USA, May 16-19, 1993.

[Kurokawa and Ikegami 1996] T. Kurokawa and T. Ikegami, ”Opticalinterconnection technologies based on vertical-cavity surface-emitting lasersand smart pixels,” 3URF���UG�,QWHUQDWLRQDO�&RQIHUHQFH�RQ�0DVVLYHO\�3DUDOOHO3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���� Maui, HI, USA, Oct.27-29, 1996, pp. 300-305.

[Kurokawa et al. 1998] T. Kurokawa, S. Matso, T. Nakahara, K. Tateno, Y.Ohiso, A. Wakatsuki, and H. Tsuda, ”Design approaches for VCSEL’s andVCSEL-based smart pixels toward parallel optoelectronic processingsystems," $SSOLHG�2SWLFV, vol. 37, no. 2, pp. 194-204, Jan. 10, 1998.

[Kuszmaul 1995] B. C. Kuszmaul, "The RACE network architecture," 3URF��WK�,QWHUQDWLRQDO�3DUDOOHO�3URFHVVLQJ�6\PSRVLXP��,336���, Santa Barbara,CA, USA, Apr. 25-28, 1995, pp. 508-513.

[Lai et al. 1998] Q. Lai, W. Hunziker, and H. Melchior, “Low-power compact2 × 2 thermooptic silica-on-silicon waveguide switch with fast response,”,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 10, no. 5, pp. 681-683, May 1998.

[Lane et al. 1989] T. A. Lane, J. A. Quam, B. O. Khale, and E. C. Parish,"Gigabit optical interconnections for the Connection Machine," in 2SWLFDO,QWHUFRQQHFWV�LQ�WKH�&RPSXWHU�(QYLURQPHQW��3URF��63,(�YRO������, J. Pazarisand G. R. Willenbring, Eds., pp. 24-35, 1989.

199

[Larson and Harris 1995] M. C. Larson and J. S. Harris, Jr., “Broadly-tunable resonant-cavity light-emitting diode,” ,(((� 3KRWRQLFV� 7HFKQRORJ\/HWWHUV," vol. 7, no. 11, pp. 1267-1269, Nov. 1995.

[Larson and Harris 1996] M. C. Larson and J. S. Harris, Jr., “Wide andcontinuous wavelength tuning in a vertical-cavity surface-emitting laserusing a micromachined deformable-membrane mirror," $SSOLHG� 3K\VLFV/HWWHUV, vol. 68, no. 7, pp. 891-893, Feb. 12, 1996.

[Laudon and Lenoski 1997] J. Laudon and D. Lenoski, "The SGI Origin: accNUMA highly scalable server," 3URF�� ��WK� ,QWHUQDWLRQDO� 6\PSRVLD� RQ&RPSXWHU�$UFKLWHFWXUH��,6&$���, Denver, CO, USA, June 2-4, 1997.

[Lawson et. al. 1992] H. W. Lawson; with contributions by B. Svensson andL. Wanhammar, 3DUDOOHO� 3URFHVVLQJ� LQ� ,QGXVWULDO� 5HDO�7LPH� $SSOLFDWLRQV.Prentice-Hall, Inc., 1992.

[Lee and Zah 1989] T. P. Lee and C. E. Zah, “Wavelength-tunable andsingle-frequency semiconductor lasers for photonic communicationsnetworks,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH, no. 10, pp. 42-52, Oct. 1989.

[Lee et al. 1996] T. P. Lee et al., “Multiwavelength DFB laser arraytransmitter for ONTC reconfigurable optical network testbed,” -RXUQDO� RI/LJKWZDYH�7HFKQRORJ\, vol. 14, no. 6, pp. 967-976, June 1996.

[Leiserson 1985] C. E. Leiserson, “Fat-trees: universal networks forhardware-efficient supercomputing,” ,(((�7UDQVDFWLRQV�RQ�&RPSXWHUV, vol.34, no. 10, pp. 892-901, Oct. 1985.

[Leiserson et al. 1992] C. E. Leiserson, Z. S. Abuhamdeh, D. C. Douglas, C.R. Feynman, M. N. Ganmukhi, J. V. Hill, W. D. Hillis, B. C. Kuszmaul, M.A. St. Pierre, D. S. Wells, M. C. Wong, S.-W. Yang, and R. Zak, "Thenetwork architecture of the Connection Machine CM-5," 3URF��RI�WKH�)RXUWK$QQXDO� $&0� 6\PSRVLXP� RQ� 3DUDOOHO� $OJRULWKPV� DQG� $UFKLWHFWXUHV, June1992.

[Lenoski et al. 1992] D. Lenoski, J. Laudon, T. Joe, D. Nakahira, L. Stevens,A. Gupta, and J. Hennessy, "The DASH prototype: implementation andperformance," 3URF����WK�,QWHUQDWLRQDO�6\PSRVLD�RQ�&RPSXWHU�$UFKLWHFWXUH�,6&$���, Gold Coast, Australia, May 19-21, 1992.

[Lenoski et al. 1992B] D. Lenoski, J. Laudon, K. Gharachorloo, W.-D.Weber, A. Gupta, J. Hennessy, M. Horowitz, and M. S. Lam, “The StanfordDASH multiprocessor,” &RPSXWHU, vol. 25, no. 3, pp. 63-79, Mar. 1992.

200

[Lenoski et al. 1993] D. Lenoski, J. Laudon, T. Joe, D. Nakahira, L. Stevens,A. Gupta, and J. Hennessy, "The DASH prototype: logic overhead andperformance," ,(((�7UDQVDFWLRQV�RQ�3DUDOOHO�DQG�'LVWULEXWHG�6\VWHPV, vol.4, no. 1, pp. 41-61, Jan. 1993.

[Levine and Akyildiz 1995] D. A. Levine and I. F. Akyildiz, "PROTON: amedia access control protocol for optical networks with star topology,",(((�$&0�7UDQVDFWLRQV�RQ�1HWZRUNLQJ, vol. 3, no. 2, Apr. 1995.

[Li et al. 1995] Y. Li, H. Kosaka, T. Wang, S. Kawai, and K. Kasahara,”Applications of fiber image guides to bit-parallel optical interconnections,"2SWLFDO� &RPSXWLQJ�� YRO�� ���� ����� 26$� 7HFKQLFDO� 'LJHVW� 6HULHV, Salt LakeCity, Utah, Mar. 13-16, 1995, pp. 286-288.

[Li et al. 1998] Y. Li, J. Popelek, J.-K. Rhee, L. J. Wang, T. Wang, and K.Shum, ”Demonstration of fiber-based board-level optical clock distributions,”3URF�� �WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ�0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Las Vegas, NV, USA, June 15-17,1998, pp. 224-228.

[Li et al. 1998B] Y. Li, T. Wang, and S. Kawai, ”Distributed crossbarinterconnects with vertical-cavity surface-emitting laser-angle multiplexingand fiber image guides," $SSOLHG�2SWLFV, vol. 37, no. 2, pp. 254-263, Jan. 10,1998.

[Li et al. 1998C] Y. Li, J. Ai, and T. Wang, "100×100 opto-electronic cross-connector using OPTOBUS," 3URF��2SWLFV� LQ�&RPSXWLQJ� �2&���, Brugge,Belgium, June 17-20, 1998, pp. 282-284.

[Lin et al. 1988] C. Lin, H. Kobrinski, A. Frenkel, C. A. Brackett,"Wavelength-tunable 16 optical channel transmission experiment at 2Gbit/s and 600 Mbit/s for broadband subscriber distribution," (OHFWURQLFV/HWWHUV, vol. 24, no. 19, pp. 1215-1217, Sept. 15, 1988.

[Liu and Prasanna 1998] W. Liu and V. K. Prasanna, "Utilizing the power ofhigh-performance computing," ,(((� 6LJQDO� 3URFHVVLQJ� 0DJD]LQH, vol. 15,no. 5, pp. 85-100, Sept. 1998.

[Loeb and Stilwell 1988] M. L. Loeb and G. R. Stilwell, "High-speed datatransmission on an optical fiber using a byte-wide WDM system," -RXUQDO�RI/LJKWZDYH�7HFKQRORJ\, vol. 6, no. 8, pp. 1306-1311, Aug. 1988.

[Loeb and Stilwell 1990] M. L. Loeb and G. R. Stilwell, "An algorithm forbit-skew correction in byte-wide WDM optical fiber systems," -RXUQDO� RI/LJKWZDYH�7HFKQRORJ\, vol. 8, no. 2, pp. 239-242, Aug. 1988.

201

[Lohmann et al. 1986] A. W. Lohmann, W. Stork, and G. Stucke, ”Opticalperfect shuffle," $SSOLHG�2SWLFV, vol. 25, no. 10, pp. 1530-1531, May 15, 1986.

[Louri and Gupta 1996] A. Louri and R. Gupta, ”Hierarchical optical ringinterconnection (HORN): a scalable interconnection-network formultiprocessors and massively parallel systems,” 3URF�� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���, Maui, HI, USA, Oct.27-29, 1996, pp. 247-254.

[Louri and Gupta 1997] A. Louri and R. Gupta, ”Hierarchical optical ringinterconnection (HORN): scalable interconnection network formultiprocessors and multicomputers," $SSOLHG�2SWLFV, vol. 36, no. 2, pp. 430-442, Jan. 10, 1997.

[Louri et al. 1996] A. Louri, S. Furlonge, and C. Neocleous, ”Experimentaldemonstration of the optical multi-mesh hypercube: scaleableinterconnection network for multiprocessors and multicomputers," $SSOLHG2SWLFV, vol. 35, no. 35, pp. 6909-6919, Dec. 10, 1996.

[Lovett and Clapp 1996] T. Lovett and R. Clapp, "STiNG: a CC-NUMAcomputer system for the commercial marketplace," 3URF����UG�,QWHUQDWLRQDO6\PSRVLD�RQ�&RPSXWHU�$UFKLWHFWXUH��,6&$���, May 1996.

[Lukowicz et al. 1998] P. Lukowicz, S. Sinzinger, K. Dunkel, and H.-D.Bauer, "Design of an opto-electronic VLSI/parallel fiber bus," 3URF��2SWLFV�LQ&RPSXWLQJ��2&���, Brugge, Belgium, June 17-20, 1998, pp. 289-292.

[Lund 1997] C. Lund, ”Optics inside future computers,” 3URF�� 0DVVLYHO\3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Montreal,Canada, June 22-24, 1997, pp. 156-159.

[Maeda et al. 1991] M. W. Maeda, C. Chang-Hasnain, A. Von Lehmen, H.Izadpanah, C. Lin, M. Z. Iqbal, L. Florez, and J. Harbison, “Multigigabit/soperation of 16-wavelength vertical-cavity surface-emitting laser array,”,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 3, no. 10, pp. 863-865, Oct. 1991.

[Maeno et al. 1997] Y. Maeno, A. Tajima, Y. Suemura, and N. Henmi, ”8.5Gbit/s/port synchronous optical packet-switch,” 3URF�� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Montreal, Canada,June 22-24, 1997, pp. 114-119.

[Malcolm and Zhao 1995] N. Malcolm and W. Zhao, "Hard real-timecommunication in multiple-access networks," 5HDO�7LPH�6\VWHPV, vol. 8, no.1, pp. 35-77, 1995.

202

[Maode et al. 1998] M. Maode, B. Hamidzadeh, and M. Hamdi, "Efficientscheduling algorithms for real-time service on WDM optical networks," 3URF�RI� WKH� �WK� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� &RPSXWHU� &RPPXQLFDWLRQV� DQG1HWZRUNV��,&�1ª���, Lafayette, LA, USA, Oct. 12-15, 1998.

[Marhic 1993] M. E. Marhic, “Coherent optical CDMA networks,” -RXUQDO�RI/LJKWZDYH�7HFKQRORJ\, vol. 11, no. 5/6, pp. 854-863, May/June 1993.

[Marsan et al. 1997] M. A. Marsan, A. Bianco, E. Leonardi, A. Morabito, andF. Neri, “SR3: a bandwidth-reservation MAC protocol for multimediaapplication over all-optical WDM multi-rings,” 3URF�� ,1)2&20ª��, Kobe,Japan, Apr. 7-11, 1997, pp. 761-768.

[MasPar 1992] “The design of the MasPar MP-2: a cost effective massivelyparallel computer,” :KLWH�SDSHU��0DV3DU�&RPSXWHU�&RUSRUDWLRQ��6XQQ\YDOH�&$��86$, 1992.

[McArdle et al. 1996] N. McArdle, M. Naruse, T. Komuro, H. Sakaida, M.Ishikawa, Y.Kobayashi, and H. Toyoda, ”A smart-pixel paralleloptoelectronic computing system with free-space dynamic interconnections,”3URF�� �UG� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ�0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ2SWLFDO�,QWHUFRQQHFWLRQV��0332,ª���� Maui, HI, USA, Oct. 27-29, 1996, pp.146-157.

[McArdle et al. 1997] N. McArdle, M. Naruse, T. Komuro, and M. Ishikawa,”Realisation of a smart-pixel parallel optoelectronic computing system,”3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV�0332,ª���� Montreal, Canada, June 22-24, 1997, pp. 190-195.

[McGreer 1998] K. A. McGreer, “Arrayed waveguide gratings for wavelengthrouting,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH� vol. 36, no. 12, pp. 62-68, Dec.1998.

[Mehravari 1991] N. Mehravari, "Performance and protocol improvementsfor very high speed optical fiber local area networks using a passive startopology," -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 8, no. 4, pp. 520-530, Apr.1990.

[Mellis et al. 1988] J. Mellis, S. A. Al-Chalabi, K. H. Cameron, R. Wyatt, J.C. Regnault, W. J. Devlin, M. C. Brain, "Miniature packaged external-cavitysemiconductor laser with 50 GHz continuous electrical tuning range,"(OHFWURQLFV�/HWWHUV, vol. 24, no. 16, pp. 988-989, Aug. 4, 1988.

[Mercury 1998] 5$&(�� 6HULHV� 5$&(ZD\� ,QWHUOLQN� 0RGXOHV� 'DWD� 6KHHW.Mercury Computer Systems, Inc., 1998.

203

[Mestdagh 1995] D. J. G. Mestdagh, )XQGDPHQWDOV� RI�0XOWLDFFHVV�2SWLFDO)LEHU�1HWZRUNV. Artech House, Inc., 1995, ISBN 0-89006-666-3.

[Miller and Janniello 1990] C. M. Miller and F. J. Janniello, " Passivelytemperature-compensated fibre Fabry-Perot filter and its application inwavelength division multiple access computer network," (OHFWURQLFV�/HWWHUV,vol. 26, no. 25, pp. 2122-2123, Dec. 6, 1990.

[Miller and Miller 1992] C. M. Miller and J. W. Miller, " Wavelength-locked,two-stage fibre Fabry-Perot filter for dense wavelength divisionmultiplexing in erbium-doped fibre amplifier spectrum," (OHFWURQLFV�/HWWHUV,vol. 28, no. 3, pp. 216-217, Jan. 30, 1992.

[Miura et al. 1993] K. Miura, M. Takamura, Y. Sakamoto, and S. Okada,"Overview of the Fujitsu VPP500 supercomputer," 3URF����WK�$QQXDO�,(((,QWHUQDWLRQDO� &RPSXWHU� &RQIHUHQFH� �&203&21� 6SULQJ� ª���� 'LJHVW, SanFrancisco, CA, USA, pp. 128-130, 1993.

[Modiano 1999] E. Modiano, “Topics in lightwave WDM-based packetnetworks,” ,(((� &RPPXQLFDWLRQV� 0DJD]LQH� vol. 37, no. 3, pp. 130-135,Mar. 1999.

[Moosburger and Petermann 1998] R. Moosburger and K. Petermann, “4 × 4digital optical matrix switch using polymeric oversized rib waveguides,”,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 10, no. 5, pp. 684-686, May 1998.

[Mudge et al. 1987] T. N. Mudge, J. P. Haynes, and D. C. Winsor, “Multiplebus architectures,” &RPSXWHU, vol. 20, no. 6, pp. 42-48, June 1987.

[Mukherjee 1992] B. Mukherjee, "WDM-based local lightwave networks partI: single-hop systems," ,(((�1HWZRUN, pp. 12-27, May 1992.

[Mukherjee 1992B] B. Mukherjee, "WDM-based local lightwave networkspart II: multihop systems," ,(((�1HWZRUN, pp. 20-32, July 1992.

[Nagarajan et al. 1998] R. Nagarajan, W. Sha, B. Li, and R. Craig,“Gigabyte/s parallel fiber-optic links based on edge emitting laser diodearrays,” -RXUQDO� RI�/LJKWZDYH�7HFKQRORJ\, vol. 16, no. 5, pp. 778-787, Maj1998.

[Nassehi et al. 1985] M. M. Nassehi, F. A. Tobagi, and M. E. Marhic, "Fiberoptic configurations for local area networks," ,(((� -RXUQDO� RQ� 6HOHFWHG$UHDV�LQ�&RPPXQLFDWLRQV, vol. 3, no. 6, pp. 941-949, Nov. 1985.

204

[Neff 1994] J. A. Neff, ”Optical interconects based on two-dimensionalVCSEL arrays,” 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���� Cancun, Mexico, Apr. 26-27, 1994, pp. 202-212.

[Neff et al. 1996] J. A. Neff, C. Chen, T. McLaren, C.-C. Mao, A. Fedor, W.Berseth, Y. C. Lee, and V. Morozov, ”VCSEL/CMOS smart pixel arrays forfree-space optical interconnects,” 3URF�� �UG� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª����Maui, HI, USA, Oct. 27-29, 1996, pp. 282-289.

[Neilson and Schenfeld 1998] D. T. Neilson and E. Schenfeld, ”Plasticmodules for free-space optical interconnects," $SSOLHG�2SWLFV, vol. 37, no. 14,pp. 2944-2952, May 10, 1998.

[Nesset et al. 1998] D. Nesset, T. Kelly, and D. Marcenac, “All-opticalwavelength conversion using SOA nonlinearities,” ,(((� &RPPXQLFDWLRQV0DJD]LQH� vol. 36, no. 12, pp. 56-61, Dec. 1998.

[Ni and McKinley 1993] L. M. Ni and P. K. McKinley, “A survey ofwormhole routing techniques in direct networks,” &RPSXWHU, vol. 26, no. 2,pp. 62-76, Feb. 1993.

[Nickolls 1992] J. R. Nickolls, "Interconnection architecture and packagingin massively parallel computers," 3URF�� 3DFNDJLQJ�� ,QWHUFRQQHFWV�� DQG2SWRHOHFWURQLFV�IRU�WKH�'HVLJQ�RI�3DUDOOHO�&RPSXWHUV�:RUNVKRS, Mar. 18-19,1992.

[Nilsson 1994] K. Nilsson, "Time-deterministic communication in acomputer system for embedded real-time applications," /LFHQWLDWH� WKHVLV�'HSDUWPHQW� RI� &RPSXWHU� (QJLQHHULQJ�� &KDOPHUV� 8QLYHUVLW\� RI� 7HFKQRORJ\�*|WHERUJ��6ZHGHQ, June 1994, ISBN 91-7197-004-5.

[Nilsson et al. 1993] K. Nilsson, B. Svensson, and P.-A. Wiberg, “A modular,massively parallel computer architecture for trainable real-time controlsystems,” &RQWURO�(QJLQHHULQJ�3UDFWLFH, Vol. 1, No. 4, pp. 655-661, 1993.

[Nilsson et al. 1993B] K. Nilsson, B. Svensson, and L. Bengtsson,“Massively parallel architectures for future action-oriented systems:possibilities, solutions, and implementation aspects,” 0&3$ª� ���,QWHUQDWLRQDO�:RUNVKRS�RQ�0HFKDWURQLFDO�&RPSXWHU�6\VWHPV� IRU�3HUFHSWLRQDQG�$FWLRQ, Halmstad, Sweden, June 1-3, 1993, pp. 229-240.

205

[Nishikido et al. 1995] J. Nishikido, S. Fujita, Y. Arai, Y. Akahori, S. Hino,and K. Yamasaki, “Multigigabit multichannel optical interconnectionmodule for broadband switching system,” -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\,vol. 13, no. 6, pp. 1104-1110, June 1995.

[Nishimura et al. 1997] S. Nishimura, H. Inoue, S. Hanatani, H. Matsuoka,and T. Yokota, “Optical interconnections for the massively parallelcomputer,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 9, no. 7, pp. 1029-1031,July 1997.

[Nishimura et al. 1998] S. Nishimura, H. Inoue, S. Hanatani, H. Matsuoka,and T. Yokota, “Error-free optical inter-node connection for the massivelyparallel computer,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 10, no. 1, pp.147-149, Jan. 1998.

[Nishio et al. 1993] M. Nishio, S. Suzuki, K. Takagi, Ichiro Oguro, T.Numai, K. Kasahari, and K. Kaede, “A new architecture of photonic ATMswitches,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH, vol. 31, no.4, pp. 62-68, Apr.1993.

[Numai et al. 1989] T. Numai, S. Murata, and I. Mito, “1.5 µm tunablewavelength filter using a phase-shift-controlled distributed feedback laserdiode with a wide tuning range and a high constant gain," $SSOLHG�3K\VLFV/HWWHUV, vol. 54, no. 19, pp. 1859-1860, May 8, 1989.

[Oda et al. 1989] K. Oda, N. Takato, T. Kominato, and H. Toba, "Channelselection and stabilization technique for a waveguide-type 16-channelfrequency selection switch for optical FDM distribution systems,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 1, no. 6, pp. 137-139, June 1989.

[Ofek 1994] Y. Ofek, “Overview of the MetaRing architecture,” &RPSXWHU1HWZRUNV�DQG�,6'1�6\VWHPV, vol. 26, pp. 817-829, Mar. 1994.

[Ohshima et al. 1985] S. Ohshima, T. Ito, K.-I. Donuma, H. Sugiyama, andY. Fujii, “Small loss-deviation tapered fiber star coupler for LAN,” -RXUQDORI�/LJKWZDYH�7HFKQRORJ\, vol. 3, no. 3, pp. 556-560, June 1985.

[Okamoto and Inoue 1995] K. Okamoto and Y. Inoue, "Silica-based planarlightwave circuits for WDM systems," &RQIHUHQFH� RQ� 2SWLFDO� )LEHU&RPPXQLFDWLRQ��2)&ª���7HFKQLFDO�'LJHVW, pp. 224-225, 1995.

[Okamoto et al. 1992] K. Okamoto, H. Takahashi, M. Yasu, and Y. Hibino,”Fabrication of wavelength-insensitive 8x8 star coupler,” ,(((� 3KRWRQLFV7HFKQRORJ\�/HWWHUV, vol. 4, no. 1, pp. 61-63, Jan. 1992.

206

[Okamoto et al. 1992B] K. Okamoto, H. Okazaki, Y. Ohmori, and K. Kato,”Fabrication of large scale integrated-optic NxN star couplers,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV, vol. 4, no. 9, pp. 1032-1035, Sep. 1992.

[O'Keefe and Dietz 1990] M. T. O'Keefe and H. G. Dietz, "Hardware barriersynchronization: static barrier MIMD (SBM)," 3URF�� RI� WKH� ,QWHUQDWLRQDO&RQIHUHQFH�RQ�3DUDOOHO�3URFHVVLQJ��,&33ª���, pp. 35-42, Aug. 1990.

[O'Keefe and Dietz 1990B] M. T. O'Keefe and H. G. Dietz, "Hardwarebarrier synchronization: dynamic barrier MIMD (DBM)," 3URF�� RI� WKH,QWHUQDWLRQDO�&RQIHUHQFH� RQ�3DUDOOHO� 3URFHVVLQJ� �,&33ª���, pp. 43-46, Aug.1990.

[Ozaktas 1997] H. M. Ozaktas, ”Fundamentals of optical interconnections −a review,” 3URF��0DVVLYHO\�3DUDOOHO�3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV�0332,ª���� Montreal, Canada, June 22-24, 1997, pp. 184-189.

[Ozaktas 1997B] H. M. Ozaktas, ”Toward an optimal foundationarchitecture for optoelectronic computing. Part I. Regularly interconnecteddevice planes," $SSOLHG�2SWLFV, vol. 36, no. 23, pp. 5682-5696, Aug. 10, 1997.

[Ozaktas 1997C] H. M. Ozaktas, ”Toward an optimal foundationarchitecture for optoelectronic computing. Part II. Physical construction andapplication platforms," $SSOLHG�2SWLFV, vol. 36, no. 23, pp. 5697-5705, Aug.10, 1997.

[Parker 1991] J. W. Parker, “Optical interconnection for advanced processorsystems: A review of the ESPRIT II OLIVES program,” -RXUQDO� RI/LJKWZDYH�7HFKQRORJ\, vol. 9, no. 12, pp. 1764-1773, Dec. 1991.

[Parker and Mears 1996] M. C. Parker and R. J. Mears, “Digitally tunablewavelength filter and laser,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 8, no.8, pp. 1007-1008, Aug. 1996.

[Parker et al. 1992] J. W. Parker, P. J. Ayliffe, T. V. Clapp, M. C. Geear, P.M. Harrison, and R. G. Peall, "Multifibre bus for rack-to-rack interconnectsbased on opto-hybrid transmitter/receiver array pair," (OHFWURQLFV� /HWWHUV,vol. 28, no. 8, pp. 801-803, April 9, 1992.

[Patel et al. 1990] J. S. Patel, M. A. Saifi, D. W. Berreman, C. Lin, N.Andreadakis, and S. D. Lee, “Electrically tunable optical filter for infraredwavelength using liquid crystals in a Fabry-Perot etalon," $SSOLHG�3K\VLFV/HWWHUV, vol. 57, no. 17, pp. 1718-1720, Oct. 22, 1990.

207

[Peall 1995] R. G. Peall, "Development in multi-channel opticalinterconnects under ESPRIT III SPIBOC," 3URF�� /(26ª��, San Francisco,CA, USA, Oct. 30 – Nov. 2, 1995, vol. 1, pp. 222-223.

[Pedersen et al. 1998] R. J. S. Pedersen, B. Mikkelsen, B. F. Jørgensen, M.Nissov, K. E. Stubkjaer, K. Wünstel, K. Daub, E. Lach, G. Laube, W. Idler,M. Schilling, P. Doussiere, and F. Pommerau, "WDM cross-connect cascadebased on all-optical wavelength converters for routing and wavelength slotinterchanging using a reduced number of internal wavelengths," 2SWLFDO)LEHU�&RPPXQLFDWLRQ�&RQIHUHQFH��2)&ª���7HFKQLFDO�'LJHVW, San Jose, CA,USA, Feb. 22-27, 1998, pp. 58-59.

[Peterson and Davie 1996] L. L. Peterson and B. S. Davie, &RPSXWHU1HWZRUNV�� D� 6\VWHPV� $SSURDFK. Morgan Kaufmann Publishers, Inc., SanFrancisco, CA, USA, 1996, ISBN 0-55860-368-9.

[Pieris and Sasaki 1994] G. R. Pieris and G. H. Sasaki, "Schedulingtransmissions in WDM broadcast-and-select networks," ,(((�$&07UDQVDFWLRQV�RQ�1HWZRUNLQJ, vol. 2, no. 2, Apr. 1994.

[Pinkston and Kuznia 1997] T. M. Pinkston and C. Kuznia, ”Smart-pixel-based network interface chip," $SSOLHG�2SWLFV, vol. 36, no. 20, pp. 4871-4880,July 10, 1997.

[Pinkston and Warnakulasuriya 1997] T. M. Pinkston and S.Warnakulasuriya, "On deadlocks in interconnection networks," 3URF�� ��WK,QWHUQDWLRQDO� 6\PSRVLD� RQ� &RPSXWHU� $UFKLWHFWXUH� �,6&$���, Denver, CO,USA, June 2-4, 1997.

[Pinkston et al. 1998] T. M. Pinkston, M. Raksapatcharawong, and Y. Choi,"WARP II: an optoelectronic fully adaptive network router chip," 3URF�2SWLFV�LQ�&RPSXWLQJ��2&���, Brugge, Belgium, June 17-20, 1998, pp. 311-315.

[Prucnal et al. 1986] P. R. Prucnal, M. A. Santoro, and S. K. Sehgal,"Ultrafast all-optical synchronous multiple access fiber networks," ,(((-RXUQDO�RQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 4, no. 9, pp. 1484-1493,Dec. 1996.

[Prucnal et al. 1986B] P. R. Prucnal, M. A. Santoro, T. R. Fan, “Spreadspectrum fiber-optic local area network using optical processing,” -RXUQDO�RI/LJKWZDYH�7HFKQRORJ\, vol. 4, no. 5, pp. 547-554, May 1986.

208

[Prucnal et al. 1994] P. R. Prucnal, I. Glesk, and J. P. Sokoloff,”Demonstration of all-optical self-clocked demultiplexing of TDM data at250 Gb/s,” 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���, Cancun, Mexico, Apr. 26-27, 1994, pp. 106-117.

[Prylli and Tourancheau 1998] L. Prylli and B. Tourancheau, " BIP: a newprotocol designed for high performance networking on Myrinet," 3URF��RI�WKH�VW�:RUNVKRS� RQ� 3HUVRQDO� &RPSXWHU� EDVHG�1HWZRUNV� RI�:RUNVWDWLRQV� �3&�12:ª���, Orlando, FL, USA, Apr. 3, 1998.

[Raghavan et al. 1999] B. Raghavan, Y.-G. Kim, T.-Y. Chuang, B.Madhavan, and A. F. J. Levi, "A Gbyte/s parallel fiber-optic networkinterface for multimedia applications," ,(((�1HWZRUN, vol. 13, no. 1, pp. 20-28, Jan./Feb. 1999.

[Ramamurthy et al. 1995] S. Ramamurthy, B. Mukherjee, and B. Moslehi,“Optimized partitioning of a wavelength distributed data interface ringnetwork,” -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 13, no. 10, pp. 1971-1984,Oct. 1995.

[Ramaswami 1993] R. Ramaswami, “Multiwavelength lightwave networksfor computer communication,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH, vol. 31, no.2, pp. 78-88, Feb. 1993.

[Ray and Jiang 1995] S. Ray and H. Jiang, ”A reconfigurable optical busstructure for shared memory multiprocessors with improved performance,”3URF�� �QG� ,QWHUQDWLRQDO�&RQIHUHQFH� RQ�0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� San Antonio, TX, USA, Oct 23-24,1995, pp. 108-115.

[Ray and Mukherjee 1995] S. Ray and S. Mukherjee, “On optimal placementof erasure nodes on a dual bus network,” 3URF��,1)2&20ª��, Boston, MA,USA, Apr. 2-6, 1995, pp. 883-890.

[Reed and Grunwald 1987] D. A. Reed and D. C. Grunwald, “Theperformance of multicomputer interconnection networks,” &RPSXWHU, vol. 20,no. 6, pp. 63-73, June 1987.

[Reif and Yoshida 1994] J. H. Reif and A. Yoshida, ”Free space opticalmessage routing for high performance parallel computers,” 3URF��0DVVLYHO\3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Cancun,Mexico, Apr. 26-27, 1994, pp. 37-44.

209

[Reilly 1994] P. Reilly, “PDH, broadband ISDN, ATM, and all that: a guideto modern WAN networking, and how it evolved,” 6LOLFRQ� *UDSKLFV�� ,QF.,1994.

[Robertsson et al. 1995] M. Robertsson, K. Engberg, P. Eriksen, H.Hesselbom, M. Niburg, and G. Palmskog, "Optical interconnects inpackaging for telecom applications," 3URF�� RI� WKH� ��WK� (XURSHDQ0LFURHOHFWURQLFV�&RQIHUHQFH, pp. 580-591, 1995.

[Rom and Sidi 1990] R. Rom and M. Sidi, 0XOWLSOH� $FFHVV� 3URWRFROV�3HUIRUPDQFH�DQG�$QDO\VLV. Springer-Verlag, Inc., 1990, ISBN 0-387-97253-6.

[Ross 1989] F. E. Ross, "An overview of FDDI: the fiber distributed datainterface," ,(((�-RXUQDO�RQ�6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 7, no. 7,pp. 1043-1051, Sept. 1989.

[Rouskas and Ammar 1995] G. N. Rouskas and M. H. Ammar, "Analysis andoptimization of transmission schedules for single-hop WDM networks,",(((�$&0�7UDQVDFWLRQV�RQ�1HWZRUNLQJ, vol. 3, no. 2, Apr. 1995.

[Rudolph 1998] L. Rudolph, ”Do parallel computers really need opticalinterconnection networks?,” 3URF���WK�,QWHUQDWLRQDO�&RQIHUHQFH�RQ�0DVVLYHO\3DUDOOHO� 3URFHVVLQJ�XVLQJ�2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Las Vegas,NV, USA, June 15-17, 1998, pp. 176-183.

[Sachs and Varma 1996] M. W. Sachs and A. Varma, “Fibre channel andrelated standards,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH� no. 8, pp. 40-50, Aug.1996.

[Sadot and Boimovich 1998] D. Sadot and E. Boimovich, “Tunable opticalfilters for dense WDM networks,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH� vol. 36,no. 12, pp. 50-55, Dec. 1998.

[Salehi 1989] J. A. Salehi, "Code division multiple-access techniques inoptical fiber networks - Part I: fundamental principles," ,(((�7UDQVDFWLRQVRQ�&RPPXQLFDWLRQV, vol 37, no. 8, pp. 824-833, Aug. 1989.

[Salehi and Brackett 1989] J. A. Salehi and C. A. Brackett, "Code divisionmultiple-access techniques in optical fiber networks - Part II: Systemsperformance analysis," ,(((�7UDQVDFWLRQV�RQ�FRPPXQLFDWLRQV, vol 37, no. 8,pp. 834-842, Aug. 1989.

[Sano and Levi 1998] B. J. Sano and A. F. J. Levi, "Networks for theprofessional campus environment," in 0XOWLPHGLD� 7HFKQRORJ\� IRU$SSOLFDWLRQV� B. Sheu and M. Ismail, Eds., McGraw-Hill, Inc., pp. 413-427,1998, ISBN 0-7803-1174-4.

210

[Sano et al. 1996] B. Sano, B. Madhavan, and A. F. J. Levi, "8 Gbps CMOSinterface for parallel fiber-optic interconnects," (OHFWURQLFV�/HWWHUV, vol. 32,pp. 2262-2263, 1996.

[Saunders 1996] S. Saunders, 7KH� 0F*UDZ�+LOO� +LJK�6SHHG� /$1V+DQGERRN. McGraw-Hill, New York, NY, USA, 1996, ISBN 0-07-057199-6.

[Sawchuk et al. 1987] A. A. Sawchuk, B. K. Jenkins, C. S. Raghavendra, andA. Varma, “Optical crossbar networks,” &RPSXWHU, vol. 20, no. 6, pp. 50-60,June 1987.

[Schenfeld 1995] E. Schenfeld, ”Massively parallel processing with opticalinterconnections: what can be done, should be and must not be done byoptics," �2SWLFDO�&RPSXWLQJ��YRO�����������26$�7HFKQLFDO�'LJHVW�6HULHV, SaltLake City, Utah, Mar. 13-16, 1995, pp. 16-18.

[Schenfeld 1996] E. Schenfeld, "Massively parallel processing with opticalinterconnections," in 3DUDOOHO� DQG�'LVWULEXWHG�&RPSXWLQJ�+DQGERRN� A. Y.Zomaya Ed., McGraw-Hill, Inc., pp. 780-810, 1996.

[Scherson and Youssef 1994] I. D. Scherson and A. S. Youssef, Eds.,,QWHUFRQQHFWLRQ� 1HWZRUNV� IRU� +LJK�3HUIRUPDQFH� 3DUDOOHO� &RPSXWHUV. IEEEComputer Society Press, Los Alamitos, CA, USA, 1994, ISBN 0-8186-6197-6.

[Schwartz et al. 1996] D. B. Schwartz, K. Y. Chun, N. Choi, D. Diaz, S.Planer, G. Raskin, and S. G. Shook, ”OPTOBUS I: performance of a 4 Gb/soptical interconnect,” 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV��0332,ª���, Maui, HI, USA, Oct. 27-29, 1996, pp. 256-263.

[Scott 1996] S. Scott, "The GigaRing channel," ,(((�0LFUR, vol. 16, no. 1,pp. 27-34, Feb. 1996.

[Seitz 1985] C. L. Seitz, “The Cosmic Cube,” &RPPXQLFDWLRQV�RI� WKH�$&0,vol. 28, no. 1, pp. 22-33, Jan. 1985.

[Seitz and Su 1993] C. L. Seitz and W.-K. Su, "A family of routing andcommunication chips based on the Mosaic," 5HVHDUFK�RQ�,QWHJUDWHG�6\VWHPV�3URFHHGLQJV�RI�WKH������6\PSRVLXP, pp. 320-337, 1993.

[Selvakennedy and Ramani 1996] S. Selvakennedy and A. K. Ramani,"Analysis of piggybacked token-passing MAC protocol with variable buffersize for WDM star coupled photonic network," 3URF�� �WK� ,QWHUQDWLRQDO&RQIHUHQFH�RQ�+LJK�3HUIRUPDQFH�&RPSXWLQJ��+L3&����, Trivandrum, India,Dec. 19-22, 1997, pp. 307-312.

211

[Semaan and Humblet 1993] G. Semaan and P. Humblet, ”Timing anddispersion in WDM optical star networks,” 3URF�� ,1)2&20ª��, SanFrancisco, CA, USA, 1993, pp. 573-577.

[Senior et al. 1998] J. M. Senior, M. R. Handley, and M. S. Leeson,“Developments in wavelength division multiple access networking,” ,(((&RPPXQLFDWLRQV�0DJD]LQH� vol. 36, no. 12, pp. 28-36, Dec. 1998.

[Seo et al. 1996] S.-W. Seo, K. Bergman, and P. R. Prucnal, "Transparentoptical networks with time-division multiplexing," ,(((� -RXUQDO� RQ6HOHFWHG�$UHDV�LQ�&RPPXQLFDWLRQV, vol. 14, no. 5, pp. 1039-1051, June 1996.

[Sethu et al. 1998] H. Sethu, C. Stunkel, and R. Stucke, "IBM RS/6000 SPinterconnection network topologies for large systems," 3URF�� RI� WKH,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 3DUDOOHO� 3URFHVVLQJ� �,&33ª���, Minneapolis,MN, USA, Aug. 10-14, 1998.

[Shahid and Holland 1996] M. A. Shahid and W. R. Holland, ”Flexibleoptical backplane interconnections,” 3URF�� �UG� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª����Maui, HI, USA, Oct. 27-29, 1996, pp. 178-185.

[Shankaranarayanan et al. 1991] N. K. Shankaranarayanan, Stuart D.Elby, and Kam Y. Lau, “WDMA/subcarrier-FDMA lightwave networks:limitations due to optical beat interference,” -RXUQDO� RI� /LJKWZDYH7HFKQRORJ\, vol. 9, no. 7, Jul. 1991.

[Siala et al. 1994] S. Siala, A. P. Kanjamala, R. N. Nottenburg, and A. F. J.Levi, "Low skew multimode ribbon fibres for parallel opticalcommunication," (OHFWURQLFV�/HWWHUV, vol. 30, no. 21, pp. 1784-1786, Oct. 13,1994.

[Siegel 1990] H. J. Siegel, ,QWHUFRQQHFWLRQ�1HWZRUNV�IRU�/DUJH�6FDOH�3DUDOOHO3URFHVVLQJ. McGraw-Hill, Inc., 1990, ISBN 0-669-03594-7.

[Siegel et al. 1981] H. J. Siegel, L. J. Siegel, F. C. Kemmerer, P. T. Mueller,Jr., H. E. Smalley, Jr., and S. D. Smith, “PASM: a partionable SIMD/MIMDsystem for image processing and pattern recognition,” ,(((�7UDQVDFWLRQV�RQ&RPSXWHUV, vol. 30, no. 12, pp. 934-947, Dec. 1981.

[Silicon 1994] 3RZHU� &KDOOHQJH� 7HFKQLFDO� 5HSRUW� Silicon Graphics, Inc.,1994.

[Sinzinger 1998] S. Sinzinger, "Planar optics as the technological platformfor optical interconnects," 3URF�� 2SWLFV� LQ� &RPSXWLQJ� �2&���, Brugge,Belgium, June 17-20, 1998, pp. 40-43.

212

[Sivalingam 1994] K. M. Sivalingam, "High-speed communication protocolsfor all-optical wavelength division multiplexed computer networks,"'RFWRUDO� WKHVLV�� 'HSDUWPHQW� RI� &RPSXWHU� 6FLHQFH�� 6WDWH� 8QLYHUVLW\� RI� 1HZ<RUN�DW�%XIIDOR��%XIIDOR��1<��86$, June 1994.

[Sivalingam and Dowd 1995] K. M. Sivalingam and P. W. Dowd, “Amultilevel WDM access protocol for an optically interconnectedmultiprocessor system,” -RXUQDO� RI� /LJKWZDYH� 7HFKQRORJ\, vol. 13, no. 11,pp. 2152-2167, Nov. 1995.

[Sivalingam et al. 1992] K. M. Sivalingam, K. Bogineni, and P. W. Dowd,"Pre-allocation media access control protocols for multiple access WDMphotonic networks," 3URF�� $&0�6,*&200ª��, Baltimore, MD, USA, Aug.17-20, 1992, pp. 14-26.

[Smeyne and Nickolls 1995] A. L. Smeyne and J. R. Nickolls, "A ruggedscalable parallel system," 3URF�� �WK� ,QWHUQDWLRQDO� 3DUDOOHO� 3URFHVVLQJ6\PSRVLXP��,336���, Santa Barbara, CA, USA, Apr. 25-28, 1995, pp. 502-507.

[Soole et al. 1992] J. B. D. Soole, K. Poguntke, A. Scherer, H. P. LeBlanc, C.Chang-Hasnain, J. R. Hayes, C. Caneau, R. Bhat, and M. A. Koza, "Multistripe array grating integrated cavity (MAGIC) laser: a newsemiconductor laser for WDM applications," (OHFWURQLFV�/HWWHUV, vol. 28, no.19, pp. 1805-1807, Sept. 10, 1992.

[Sorel et al. 1996] Y. Sorel, A. O’Hare, J.-F. Kerdiles, and P.-L. François,”160 Gb/s in 28-nm WDM transmission over 238 km of DSF with standardfiber compensation,”� ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 8, no. 5, pp.727-729, May 1996.

[Spirit et al. 1994] D. M. Spirit, A. D. Ellis, and P. E. Barnsley, “Opticaltime division multiplexing: systems and networks,” ,(((�&RPPXQLFDWLRQV0DJD]LQH, no. 12, pp. 56-62, Dec. 1994.

[Stallings 1997] W. Stallings, 'DWD� DQG� &RPSXWHU� &RPPXQLFDWLRQV. fifthedition, Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1997, ISBN 0-13-571274-2.

[Stankovic 1988] J. A. Stankovic, “Misconceptions about real-timecomputing,” &RPSXWHU, vol. 21, no. 10, pp. 10-19, Oct. 1988.

[Steenkiste 1996] P. Steenkiste, "Network-based multicomputers: apractical supercomputer architecture," ,(((�7UDQVDFWLRQV� RQ�3DUDOOHO� DQG'LVWULEXWHG�6\VWHPV, vol. 7, no. 8, pp. 861-875, Aug. 1996.

213

[Stenström 1990] P. Stenström, “A survey of cache coherence schemes formultiprocessors,” &RPSXWHU, vol. 23, no. 6, pp. 12-24, June 1990.

[Stojmenovic 1996] I. Stojmenovic, "Direct interconnection networks," in3DUDOOHO�DQG�'LVWULEXWHG�&RPSXWLQJ�+DQGERRN� A. Y. Zomaya Ed., McGraw-Hill, Inc., pp. 537-567, 1996.

[Straus and Kawasaki 1987] J. Straus and B. Kawasaki, "Passive opticalcomponents," in 2SWLFDO�)LEHU� 7UDQVPLVVLRQ. E. E. Basch Ed., SAMS, pp.241-264, 1987.

[Stunkel 1997] C. B. Stunkel, ”Commercial MPP networks: time for optics?,”3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV�0332,ª���� Montreal, Canada, June 22-24, 1997, pp. 90-95.

[Stunkel et al. 1994] C. B. Stunkel, D. G. Shea, B. Abali, M. Atkins, C. A.Bender, D. G. Grice, P. H. Hochschild, D. J. Joseph, B. J. Nathanson, R. A.Swetz, R. F. Stucke, M. Tsao, and P. R. Varker, “The SP2 communicationsubsystem,” :KLWH� SDSHU�� ,%0� 7KRPDV� -�� :DWVRQ� 5HVHDUFK� &HQWHU�<RUNWRZQ�+HLJKWV��1<��86$, Aug. 1994.

[Stunkel et al. 1995] C. B. Stunkel, D. G. Shea, B. Abali, M. G. Atkins, C. A.Bender, D. G. Grice, P. Hochschild, D. J. Joseph, B. J. Nathanson, R. A.Swetz, R. F. Stucke, M. Tsao, P. R. Varker, "The SP2 high-performanceswitch," ,%0�6\VWHPV�-RXUQDO, vol. 34, no. 2, pp. 185-204, 1995.

[Su and Olshansky 1993] S. F. Su and R. Olshansky, “Performance ofWDMA networks with baseband data packets and subcarrier multiplexedcontrol channels,” ,(((� 3KRWRQLFV� 7HFKQRORJ\� /HWWHUV, vol. 5, no. 2, Feb.1993.

[Sudhakar et al. 1991] G. N. M. Sudhakar, N. D. Georganas, and M.Kavehrad, "Slotted aloha and reservation aloha protocols for very high-speed optical fiber local area networks using passive star topology," -RXUQDORI�/LJKWZDYH�7HFKQRORJ\, vol. 9, no. 10, pp. 1411-1422, Oct. 1991.

[Sun et al. 1998] Z. J. Sun, K. A. McGreer, and J. N. Broughton,“Demultiplexer with 120 channels and 0.29-nm channel spacing,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 10, no. 1, pp. 90-92, Jan. 1998.

[Supmonchai and Szymanski 1998] B. Supmonchai and T. Szymanski,"High speed VLSI concentrators for terabit intelligent optical backplanes,"3URF��2SWLFV�LQ�&RPSXWLQJ��2&���, Brugge, Belgium, June 17-20, 1998, pp.306-310.

[Suzuki 1994] T. Suzuki, “ATM adaptation layer protocol,” ,(((&RPPXQLFDWLRQV�0DJD]LQH, no. 4, pp. 80-83, Apr. 1994.

214

[Svensson and Wiberg 1993] B. Svensson and P.-A. Wiberg, “Autonomoussystems demand new computer system architectures and new developmentstrategies,” 3URF�� RI� WKH� �WK� DQQXDO� &RQIHUHQFH� RI� WKH� ,(((� ,QGXVWULDO(OHFWURQLFV�6RFLHW\��,(&21�ª���, Maui, Hawaii, USA, Nov. 15-19, 1993, pp.27-31.

[Szymanski 1995] T. H. Szymanski, ”Intelligent optical backplanes,"2SWLFDO� &RPSXWLQJ�� YRO�� ���� ����� 26$� 7HFKQLFDO� 'LJHVW� 6HULHV, Salt LakeCity, Utah, Mar. 13-16, 1995, pp. 11-13.

[Szymanski and Hinton 1995] T. Szymanski and H. S. Hinton, ”Design of aterabit free-space photonic backplane for parallel computing,” 3URF�� �QG,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV� �0332,ª���� San Antonio, TX, USA, Oct 23-24, 1995, pp.16-27.

[Szymanski and Supmonchai 1996] T. H. Szymanski and B. Supmonchai,”Reconfigurable computing with optical backplanes − an economic argumentfor optical interconnects,” 3URF�� �UG� ,QWHUQDWLRQDO�&RQIHUHQFH� RQ�0DVVLYHO\3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Maui, HI,USA, Oct. 27-29, 1996, pp. 321-328.

[Szymanski et al. 1998] T. H. Szymanski, A. Au, M. Lafrenière-Roula, V.Tyan, B. Supmonchai, J. Wong, B. Zerrouk, and S. T. Obenaus, ”Terabitoptical local area networks for multiprocessing systems," $SSOLHG� 2SWLFV,vol. 37, no. 2, pp. 264-275, Jan. 10, 1998.

[Tabuchi and Ishikawa 1990] H. Tabuchi and H. Ishikawa, "Externalgrating tunable MQW laser with wide tuning range of 240 nm," (OHFWURQLFV/HWWHUV, vol. 26, no. 11, pp. 742-743, May 24, 1990.

[Tajima et al. 1998] A. Tajima, N. Kitamura, S. Takahashi, S. Kitamura, Y.Maeno, Y. Suemura, and N. Henmi, “10-Gb/s/port gated divider passivecombiner optical switch with single-mode-to-multimode combiner,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 10, no. 1, pp. 162-164, Jan. 1998.

[Taveniku 1997] M. Taveniku, "SIMD architectures for radar signalprocessing and artificial neural networks," 'RFWRUDO� WKHVLV�� 'HSDUWPHQW� RI&RPSXWHU� (QJLQHHULQJ�� &KDOPHUV� 8QLYHUVLW\� RI� 7HFKQRORJ\�� *|WHERUJ�6ZHGHQ, Nov. 1997, ISBN 91-7197-564-0.

[Taveniku et al. 1996] M. Taveniku, A. Åhlander, M. Jonsson, and B.Svensson, “A multiple SIMD mesh architecture for multi-channel radarprocessing,” 3URF�� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 6LJQDO� 3URFHVVLQJ$SSOLFDWLRQV��7HFKQRORJ\��,&63$7���, Boston, MA, USA, Oct. 7-10, 1996,pp. 1421-1427.

215

[Taveniku et al. 1998] M. Taveniku, A. Ahlander, M. Jonsson, and B.Svensson, "The VEGA moderately parallel MIMD, moderately parallelSIMD, architecture for high performance array signal processing," 3URF���WK� ,QWHUQDWLRQDO� 3DUDOOHO� 3URFHVVLQJ� 6\PSRVLXP� � �WK� 6\PSRVLXP� RQ3DUDOOHO� DQG� 'LVWULEXWHG� 3URFHVVLQJ� �,336�63'3ª���, Orlando, FL, USA,Mar. 30 - Apr. 3, 1998, pp. 226-232.

[Tanenbaum 1996] A. S. Tanenbaum, &RPSXWHU� 1HWZRUNV. Prentice-Hall,Inc., Upper Saddle River, NJ, USA, 1996, ISBN 0-13-349945-6.

[Teitelbaum 1998] K. Teitelbaum, ”Crossbar tree networks for embeddedsignal processing applications,” 3URF�� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� Las Vegas, NV, USA, June 15-17,1998, pp. 200-207.

[Teza et al. 1995] J. P. Teza, D. M. Chiarulli, S. P. Levitan, and R. G.Melhem, ”Multiprocessor architectures using partioned optical passive starinterconnection networks," �2SWLFDO�&RPSXWLQJ��YRO�����������26$�7HFKQLFDO'LJHVW�6HULHV, Salt Lake City, Utah, Mar. 13-16, 1995, pp. 23-25.

[Thinking Machines 1991] 7KH� &RQQHFWLRQ� 0DFKLQH� &0����� 6HULHV7HFKQLFDO�6XPPDU\� Thinking Machines, Corp., June 1991.

[Toba et al. 1989] H. Toba, K. Oda, K. Nosu, and N. Takato, "16-channeloptical FDM distribution/transmission experiment utilising multichannelfrequency stabiliser and waveguide frequency selection switch," (OHFWURQLFV/HWWHUV, vol. 25, no. 9, pp. 574-576, Apr. 27, 1989.

[Toba et al. 1990] H. Toba, K. Oda, K. Nakanishi, N. Shibata, K. Nosu, N.Takato, and K. Sato, "100-channel optical FDM transmission /distributionat 622 Mbit/s over 50 km utilising a waveguide frequency selection switch,"(OHFWURQLFV�/HWWHUV, vol. 26, no. 6, pp. 376-377, Mar. 15, 1990.

[Toba et al. 1993] H. Toba, K. Nakanishi, K. Oda, K. Inoue, and T.Kominato, ”A 100-channel optical FDM six-stage in-line amplifier systememploying tunable gain equalizers,”� ,(((� 3KRWRQLFV� 7HFKQRORJ\� /HWWHUV,"vol. 5, no. 2, pp. 248-251, Feb. 1993.

[Tolmie and Renwick 1993] D. Tolmie and J. Renwick, "HIPPI: simplicityyields success," ,(((�1HWZRUN, pp. 28-32, Jan. 1993.

[Tomaševiü and Milutinoviü 1994] M. Tomaševiü and V. Milutinoviü,"Hardware approaches to cache coherence in shared-memorymultiprocessors, Part 1," ,(((�0LFUR, vol. 14, no. 5, pp. 52-59, Oct. 1994.

216

[Tomaševiü and Milutinoviü 1994B] M. Tomaševiü and V. Milutinoviü,"Hardware approaches to cache coherence in shared-memorymultiprocessors, Part 2," ,(((�0LFUR, vol. 14, no. 5, pp. 61-66, Oct. 1994.

[Tong 1998] F. Tong, “Multiwavelength receivers for WDM systems,” ,(((&RPPXQLFDWLRQV�0DJD]LQH� vol. 36, no. 12, pp. 42-49, Dec. 1998.

[Tooley 1996] F. A. P. Tooley, ”Optically interconnected electronics –challanges and choices,” 3URF��0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ�2SWLFDO,QWHUFRQQHFWLRQV��0332,ª���� Maui, HI, USA, Oct. 27-29, 1996, pp. 138-145.

[TriQuint 1992] 'DWD�&RPPXQLFDWLRQV�3URGXFWV�'DWD�%RRN� TriQuint, 1992.

[Truong et al. 1995] H. L. Truong, W. W. Ellington Jr., J. L. Boudec, A. X.Meier, and J. W. Pace, “LAN emulation on an ATM network,” ,(((&RPPXQLFDWLRQV�0DJD]LQH, no. 5, pp. 70-85, May 1995.

[Tseng and Chen 1982] C.-W. Tseng and B.-U. Chen, "D-Net. A new schemefor high data rate optical local area networks," 3URF�� ,(((� *OREDO7HOHFRPPXQLFDWLRQV�&RQIHUHQFH��*/2%(&20ª���, pp. 949-955, 1982.

[Tyan et al. 1996] H.-Y. Tyan, C.-J. Hou, B. Wang, C.-C. Han, "Onsupporting time-constrained communications in WDMA-based star-coupledoptical networks," 3URF����WK�,(((�5HDO�7LPH�6\VWHPV�6\PSRVLXP, pp. 175-184, 1996.

[Uchida 1997] N. Uchida, "Hardware of VX/VPP300/VPP700 series ofvector-parallel supercomputer systems," )XMLWVX� 6FLHQWLILF� DQG� 7HFKQLFDO-RXUQDO� vol. 33, no. 1, pp. 6-14, June 1997.

[USC 1997] 32/2�7HFKQLFDO�6XPPDU\� University of Southern California,Oct. 1997.

[Varma and Raghavendra 1994] A. Varma and C. S. Raghavendra, Eds.,,QWHUFRQQHFWLRQ� 1HWZRUNV� IRU� 0XOWLSURFHVVRUV� DQG� 0XOWLFRPSXWHUV�� 7KHRU\DQG�3UDFWLFH. IEEE Computer Society Press, Los Alamitos, CA, USA, 1994,ISBN 0-8186-4972-2.

[Wagner and Chapuran 1992] S. S. Wagner and T. E. Chapuran,"Multiwavelength ring networks for switch consolidation andinterconnection," 3URF�� ,(((� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� &RPPXQLFDWLRQV�,&&ª���, Chicago, IL, USA, June 1992, pp. 1173-1179.

[Wailes and Meyer 1991] T. S. Wailes and D. G. Meyer, “Multiple channelarchitecture: a new optical interconnection strategy for massively parallelcomputers,” -RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 9, no. 12, pp. 1702-1716,Dec. 1991.

217

[Wang et al. 1997] B. Wang, C.-J. Hou, and C.-C. Han, “On dynamicallyestablishing and terminating isochronous message streams in WDMA-basedlocal area lightwave networks,” 3URF��,1)2&20ª��, Kobe, Japan, Apr. 7-11,1997, pp. 1261-1269.

[Weber et al. 1997] W.-D. Weber, S. Gold, P. Helland, T. Shimizu, T. Wicki,and W. Wilcke, "The Mercury interconnect architecture: a cost-effectiveinfrastructure for high-performance servers," 3URF�� ��WK� ,QWHUQDWLRQDO6\PSRVLD�RQ�&RPSXWHU�$UFKLWHFWXUH��,6&$���, Denver, CO, USA, June 2-4,1997.

[Weiner et al. 1988] A. M. Weiner, J. P. Heritage, and J. A. Salehi,“Encoding and decoding of femtosecond pulses,” 2SWLFV�/HWWHUV, vol. 13, no. 4,pp. 300-302, Apr. 1988.

[Westmore 1991] R. J. Westmore, "SYMFONET: interconnect technology formultinode computing," (OHFWURQLFV�/HWWHUV, vol. 27, no. 9, pp. 697-698, Apr.25, 1991.

[Wiberg 1993] P.-A. Wiberg, “Distributed systems for time-deterministicexecution and incremental development,” 0&3$ª� ���� ,QWHUQDWLRQDO:RUNVKRS� RQ�0HFKDWURQLFDO� &RPSXWHU� 6\VWHPV� IRU� 3HUFHSWLRQ� DQG� $FWLRQ,Halmstad, Sweden, June 1-3, 1993, pp. 309-318.

[Wickman et al. 1999] R. W. Wickman, B. R. Pecor, J. P. Greene, D. A.Barneson, and G. Raskin, "Clustering supercomputer nodes using high-density parallel-optic interconnects," in 63,(�&RQIHUHQFH� RQ�2SWRHOHFWURQLF,QWHUFRQQHFWV�9,��63,(�YRO������, San Jose, CA, USA, Jan. 27-29, 1999, pp.104-114.

[Wong and Yum 1994] P. C. Wong and T.-S. P. Yum, "Design and analysis ofa pipeline ring protocol," ,(((�7UDQVDFWLRQV�RQ�FRPPXQLFDWLRQV, vol. 42, no.2/3/4, pp. 1153-1161, Feb./Mar./Apr. 1994.

[Wong et al. 1995] Y.-M. Wong, D. J. Muehlner, C. C. Faudskar, D. B.Buchholz, M. Fishteyn, J. L. Brandner, W. J. Parzygnat, R. A. Morgan, T.Mullally, R. E. Leibenguth, G. D. Guth, M. W. Focht, K. G. Glogovsky, J. L.Zilko, J. V. Gates, P. J. Anthony, B. H. Tyrone, Jr., T. J. Ireland, D. H.Lewis, Jr., D. F. Smith, S. F. Nati, D. K. Lewis, D. L. Rogers, H. A. Aispain,S. M. Gowda, S. G. Walker, Y. H. Kwark, R. J. S. Bates, D. M. Kuchta, J. D.Crow, “Technology development of a high-density 32-channel 16-Gb/s opticaldata link for optical interconnection applications for the optoelectronictechnology consortium (OETC),” -RXUQDO� RI�/LJKWZDYH�7HFKQRORJ\, vol. 13,no. 6, pp. 995-1016, June 1995.

218

[Wooten et al. 1996] E. L. Wooten, R. L. Stone, E. W. Miles, and E. M.Bradley, “Rapidly tunable narrowband wavelength filter using LiNbO3unbalanced Mach-Zehnder interferometers,” -RXUQDO� RI� /LJKWZDYH7HFKQRORJ\, vol. 14, no. 11, pp. 2530-2536, Nov. 1996.

[Xu and Herzog 1988] M. Xu and J. H. Herzog, “Concurrent token ringprotocol,” 3URF��,1)2&20ª��, pp. 145-154, 1988.

[Yan et al. 1996] A. Yan, A. Ganz, and C. M. Krishna, “A distributedadaptive protocol providing real-time services on WDM-based LANs,” 3URF�,1)2&20ª��, 1996, San Francisco, CA, USA, Mar. 24-28, 1996, pp. 962-969.

[Yan et al. 1996B] A. Yan, A. Ganz, and C. M. Krishna, “A distributedadaptive protocol providing real-time services on WDM-based LAN’s,”-RXUQDO�RI�/LJKWZDYH�7HFKQRORJ\, vol. 14, no. 6, pp. 1245-1254, June 1996.

[Yatagai et al. 1996] T. Yatagai, S. Kawai, and H. Huang, ”Opticalcomputing and interconnects,” 3URFHHGLQJV� RI� WKH� ,(((, vol. 84, no. 6, pp.828-852, June 1996.

[Yayla et al. 1998] G. I. Yayla, P. J. Marchand, and S. C. Esener, ”Speed andenergy analysis of digital interconnections: comparison of on-chip, off-chip,and free-space technologies," $SSOLHG�2SWLFV, vol. 37, no. 2, pp. 205-227, Jan.10, 1998.

[Yoshikawa et al. 1997] T. Yoshikawa, S. Araki, K. Miyoshi, Y. Suemura, N.Henmi, T. Nagahori, H. Matsuoka, and T. Yokota, “Skewless optical data-link subsystem for massively parallel processors using 8 Gb/s × 1.1 Gb/sMMF array optical module,” ,(((�3KRWRQLFV�7HFKQRORJ\�/HWWHUV," vol. 9, no.12, pp. 1625-1627, Dec. 1997.

[Yoshikawa et al. 1997B] T. Yoshikawa, H. Matsuoka, T. Yokota, and J.Shimada, ”Parallel optical interconnection for massively parallel processorRWC-1,” 3URF��0DVVLYHO\�3DUDOOHO�3URFHVVLQJ�XVLQJ�2SWLFDO�,QWHUFRQQHFWLRQV�0332,ª���� Montreal, Canada, June 22-24, 1997, pp. 4-9.

[Yu and Bhattacharya 1997] C.-C. Yu and S. Bhattacharya, "Dynamicscheduling of real-time messages over an optical network," 3URF��RI�WKH�6L[WK,QWHUQDWLRQDO� &RQIHUHQFH� RQ� &RPSXWHU� &RPPXQLFDWLRQV� DQG� 1HWZRUNV�,&�1ª���, Las Vegas, NV, USA, Sept. 22-25, 1997, pp. 336-339.

[Yun and Kavehrad 1992] G. Yun and M. Kavehrad, ”1 × 1 passive fiberstar coupler using grating degeneration and sandwich structures,” ,(((3KRWRQLFV�7HFKQRORJ\�/HWWHUV, vol. 4, no. 9, pp. 1035-1037, Sep. 1992.

219

[Zaccarin and Kavehrad 1993] D. Zaccarin and M. Kavehrad, ”An opticalCDMA system based on spectral encoding of LED,” ,(((� 3KRWRQLFV7HFKQRORJ\�/HWWHUV," vol. 4, no. 4, pp. 479-482, Apr. 1993.

[Zah et al. 1988] C. E. Zah et al., "Monolithic integration of multiwavelengthcompressive-strained multiquantum-well distributed-feedback laser arraywith star coupler and optical amplifiers," (OHFWURQLFV�/HWWHUV, vol. 28, no. 25,pp. 2361-2362, 3 Dec., 1992.

[Zhang 1995] H. Zhang, ”Service disciplines for guaranteed performance inpacket-switching networks,” 3URFHHGLQJV� RI� WKH� ,(((, vol. 83, no. 10, pp.1374-1396, Oct. 1995.

[Zhao et al. 1995] C. Zhao, T.-H. Oh, and R. T. Chen, ”General purposebidirectional optical backplane: high-performance bus for multiprocessorsystems,” 3URF�� �QG� ,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO3URFHVVLQJ� XVLQJ� 2SWLFDO� ,QWHUFRQQHFWLRQV� �0332,ª���� San Antonio, TX,USA, Oct 23-24, 1995, pp. 188-195.

[Zhao et al.1996] C. Zhao, J. Liu, and R. T. Chen, ”Hybrid optoelectronicbackplane bus for multiprocessor-based computing systems,” 3URF�� �UG,QWHUQDWLRQDO� &RQIHUHQFH� RQ� 0DVVLYHO\� 3DUDOOHO� 3URFHVVLQJ� XVLQJ� 2SWLFDO,QWHUFRQQHFWLRQV��0332,ª���� Maui, HI, USA, Oct. 27-29, 1996, pp. 313-320.

[Zirngibl et al. 1996] M. Zirngibl, C. R. Doerr, and L. W. Stulz, “Study ofspectral slicing for local access applications,” ,(((� 3KRWRQLFV� 7HFKQRORJ\/HWWHUV," vol. 8, no. 5, pp. 721-723, May 1996.

[Zirngibl 1998] M. Zirngibl, “Multifrequency lasers and applications inWDM networks,” ,(((�&RPPXQLFDWLRQV�0DJD]LQH� vol. 36, no. 12, pp. 39-41, Dec. 1998.

221

$EEUHYLDWLRQV

ADM Add-Drop Multiplexer

APD Avalanche Photo Diode

ATM Asynchronous Transfer Mode

AWG Arrayed Waveguide Grating

BS Barrier Synchronization

CAN Control Area Network

CBR Constant Bit Rate

CC-FPR Control Channel based Fiber-ribbon Pipeline Ring

CDMA Code Division Multiple Access

CFAR Constant False Alarm Ratio

CM Computation Module

COW Cluster of Workstations

CPI Coherent Processing Interval

CSMA Carrier-Sense Multiple-Access

CSMA/CD CSMA with Collision Detection

DBR Distributed Bragg Region

DFB Distributed FeedBack

DS-SS Direct Sequence Spread Spectrum

DT-WDMA Dynamic Time-Wavelength Division Multi Access

EDF Earliest Deadline First

FDDI Fibre Distributed Data Interface

FDM Frequency Division Multiplexing

FIFO First In First Out

FIG Fiber Imaging Guide

FWHM Full Width Half Maximum

GR Global Reduction

ILP Instruction Level Parallelism

I-TDMA Interleaved TDMA

LED Light Emitting Diode

222

MAC Medium Access Control

MAGIC Multistripe Array Grating Integrated Cavity

MIMD Multiple Instruction streams Multiple Data streams

MSIMD Multiple SIMD arrays

NOW Network of Workstations

PCB Printed Circuit Board

PE Processing Element

PFDM Pulse Frequency Division Multiplexing

PIN P-insulator-N

PON Passive Optical Networks

PVM Parallel Virtual Machine

QoS Quality of Service

RTVC Real-Time Virtual Channels

SAN System Area Network

SCMA SubCarrier Division Multiple Access

SDM Space Division Multiplexing

SI Slot Initiator

SIMD Single Instruction stream Multiple Data streams

SLM Spatial Light Modulator

SPMD Same Program Multiple Data

TDMA Time Division Multiple Access

TTP Time-Triggered Protocol

UBR Unspecified Bit Rate

VBR Variable Bit Rate

VCSEL Vertical Cavity Surface Emitting Laser

WDDI Wavelength Distributed Data Interface

WDM Wavelength Division Multiplexing

WDMA Wavelength Division Multiple Access