Backplane Technology Overview for AdvancedTCA
-
Upload
huichenphd -
Category
Education
-
view
1.294 -
download
2
Transcript of Backplane Technology Overview for AdvancedTCA
![Page 1: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/1.jpg)
B k l T h l O i Backplane Technology Overview for AdvancedTCA
Theresienstr. 9080290 MunichGermany
Hui ChenSupervisor: Rainer Ohlendorf
www.lis.ei.tum.deJuly, 13th 2007
© Institute forIntegrated Systems
![Page 2: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/2.jpg)
Outline
Introduction to AdvancedTCAI t t P t lInterconnect Protocols① Ethernet② InfiniBand③ StarFabric④ PCI Express⑤ RapidIO
Comparison of Interconnect Protocols① Feature② Protocol Efficiency② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration
C l i
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 2
Conclusion
![Page 3: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/3.jpg)
What is AdvancedTCA ?
• AdvancedTCA (Advanced Telecom Computing Architecture)Standard chassis/ backplane solution for high-availability communications equipmentAdopted in 2003
• AdvancedTCA Chassis
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 3
![Page 4: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/4.jpg)
Base Spec of AdvancedTCA
• Board dimension1.2 inches in width 280 mm deep, ~322 mm high
• Power supply & Cooling-48 V power feeds200 Watts per board
Board of AdvancedTCA
• Backplane architectureZ 1 P M tZone 1: Power, ManagementZone 2: Data Transport InterfaceZone 3: User-defined I/O
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 4
Backplane of AdvancedTCA
![Page 5: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/5.jpg)
Data Transport Interface
• Base Interface: Dual StarSupports only Ethernet
• Fabric Interface: Dual Star/ Full MeshSupports various protocolsMax. 4 channelsMax. 4 channels
• Dual StarRedundant switch: eliminates single point of
(a) Dual Star configuration
Redundant switch: eliminates single point of failureEffective for large redundant systems
F ll M h• Full Mesh+) Higher data throughput-) Higher pins count & costEff ti f ll t
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 5
Effective for small systems(b) Full Mesh configuration
![Page 6: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/6.jpg)
Outline
Introduction to AdvancedTCA
Interconnect ProtocolsInterconnect Protocols① Ethernet② InfiniBand③ StarFabric④ PCI Express⑤ RapidIO⑤ RapidIO
Comparison of Interconnect Protocols① Feature② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 6
Conclusion
![Page 7: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/7.jpg)
Protocol ①― Ethernet
• Application:Initial in Local Area NetworkSuccessful in backplanebut defective in congestion management
• Congestion managementCongestion managementPAUSE function: all data flows are paused when only one flow is congested→ congestion spreading, when complex data pathsg p g, p p
• TopologyP2P connections: Ethernet Switch ↔ Endpoints Ethernet Topology
• Frame
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 7
Header: large
![Page 8: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/8.jpg)
Protocol ②― InfiniBand ⑴
• ApplicationInitial in System Area Network
l i b k l h i t h i
Processor Node
now also in backplane, chassis-to-chassisSupercomputer, Linux cluster
• Topology• TopologyInfiniBand Switches + Channel AdaptersHCA: Host Channel Adapter in processor node
TCATCA: Target Channel Adapter in I/O nodeHCA talks TCAs to enable remotestorage and connectivity InfiniBand Topology
I/O Node
• Packet format
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 8
Larger payload size than Ethernet
![Page 9: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/9.jpg)
Protocol ②― InfiniBand ⑵
• Three link widths x1
x4Links x1 x4 x12 x4
x12
Links x1 x4 x12
Data Rate (Gb/s) 2 8 24
• Supports Quality of Service (QoS)QoS: a method to prioritize network traffic
Ensures most important data gets thru the network as quickly as possibleEnsures most important data gets thru the network as quickly as possibleDefines traffic to many streams, streams to differentiated classes;reserves bandwidth to streams and classes
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 9
![Page 10: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/10.jpg)
Protocol ② ― InfiniBand ⑶
• Supports Quality of ServiceThru 16 Virtual Lanes (VLs)
Multiplex independent data streams → the same physical link
Represent a set of send & receive pbuffers in a port
Each port: one management VL (VL15) & up to 15 data VLs
Priorities
VL15 (highest)
Virtual Lanes
VL14
VL0
…© Institute for
Integrated SystemsHauptseminar am Lehrstuhl für Integrierte Systeme
Hui Chen, July 13th 2007 10
![Page 11: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/11.jpg)
Protocol ③ ― StarFabric ⑴
• ApplicationBackplane, chassis-to-chassisEmbedded distributed processing(e.g. multiprocessor systems)
• TopologyStarFabric Switch: six-port
Connects other switches/ bridges
StarFabric TopologyCascadable
StarFabric BridgeTranslates other protocols (e.g. PCI) intoSt F b i t ffiStarFabric traffic
• Not scalableO t l t 2 Gb/
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 11
Operates only at 2 Gb/s
![Page 12: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/12.jpg)
Protocol ③ ― StarFabric ⑵
• Packet format
• Supports Quality of Service7 classes of traffic7 classes of traffic
asynchronous traffic (data)multicast…isochronous traffic (voice & video)
• Backwards compatible to PCIConverting a serial fabric to parallel PCI
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 12
![Page 13: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/13.jpg)
Protocol ④ ― PCI Express ⑴
• Successor to PCIBackplane, chip-to-chip interconnectPC (e.g. Graphics cards)
• TopologySwitchesMultiple Endpoints: I/O devicesHost Bridge
To create an I/O hierarchy
• Six link widths:PCI Express Topology
Links x1 x2 x4 x8 x16 x32
Data Rate (Gb/s) 2 4 8 16 32 64
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 13
![Page 14: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/14.jpg)
Protocol ④ ― PCI Express ⑵
• Packet format
• Supports Quality of ServiceThr 8 Traffic Classes (TCs) & 8 Virt al Channels (VCs)Thru 8 Traffic Classes (TCs) & 8 Virtual Channels (VCs)
Each port : up to 8 VCs. VC0 (lowest priority), VC7 (highest)
TCs: differentiated servicesE h k t TC t d t i hi h VC b ff t d k t i tEach packet a TC: to determine which VC buffer to drop packet into
TC/VC mapping: ☺ TC[0:6]→VC0 TC7→VC[0:1]
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 14
An Example of TC/VC mapping
![Page 15: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/15.jpg)
Protocol ⑤― RapidIO ⑴
• ApplicationBackplane, chip-to-chip interconnectEmbedded systems (e.g. DSPs)Networking (e.g. Carrier Cards)
T l• TopologyA Switch, multiple EndpointsLarge systems: one / more host processors
For system exploration initializationFor system exploration, initializationMonitors system-level activity
• Two link widths (x1 x4)RapidIO Topology
Two link widths (x1, x4)
Links x1 x4
Data Rate (Gb/s) 1 2 2.5 4 8 10
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 15
![Page 16: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/16.jpg)
Protocol ⑤― RapidIO ⑵
• Packet format
Header: smaller than Ethernet:1-byte source address ~ 28 Endpoints → sufficient for backplane applicationsPayload size: relatively small
• Supports Quality of ServiceSix logical flows: as prioritized classes
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 16
![Page 17: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/17.jpg)
Outline
Introduction to AdvancedTCA
Interconnect ProtocolsInterconnect Protocols① Ethernet② InfiniBand③ StarFabric③ StarFabric④ PCI Express⑤ RapidIO
Comparison of Interconnect Protocols① Feature② Protocol Efficiency② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 17
Conclusion
![Page 18: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/18.jpg)
Comparison – Feature
Feature GigE InfiniBand StarFabric PCI Expr RapidIOFeature GigE InfiniBand StarFabric PCI Expr. RapidIO
Transport Media
PCB, Copper & Fiber
PCB, Copper & Fiber
PCB &CAT5 cable
PCB, Copper & Optical PCB only
Scalable Link Widths None Yes None Yes Yes
PCI Migration None Encapsulated, requires SW
Yes, transparent
Yes, transparent
Yes,requires SWq p p requires SW
Max Signal Channel Rate 1 Gbps 2 Gbps 2 Gbps 2 Gbps 2.5 Gbps
Max Payload 1500 B tes 4096 B t 128 B t 4096 B t 256 B tesySize 1500 Bytes 4096 Bytes 128 Bytes 4096 Bytes 256 Bytes
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 18
![Page 19: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/19.jpg)
Comparison – Protocol Efficiency
• Protocol Efficiency = Payload / Packet size
• Payload < ~500B : RapidIO, highest efficiency smallest overheadPayload > ~500B : PCI Express highestPayload > ~500B : PCI Express, highest
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 19
![Page 20: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/20.jpg)
Comparison – Effective Bandwidth
• Effective Bandwidth = Protocol efficiency × Bandwidth
• InfiniBand x4; PCIe x4; RapidIO 4x 2Gbps
O ll Gi E l t• Overall: GigE, lowest
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 20
![Page 21: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/21.jpg)
Comparison – System Level
• Quality of Service (QoS)
Gi E I fi iB d St F b i PCI E R idIOGigE InfiniBand StarFabric PCI Expr. RapidIO
Prioritized Traffic Classes - 16 7 8 6
P f• Performance
GigE InfiniBand StarFabric PCI Expr. RapidIO
M B d idth fMax Bandwidth for AdvancedTCA, x4 1 Gbps 8 Gbps 2 Gbps 8 Gbps 10 Gbps
Latency High (~100 us)
Medium (1.29-2.6 us)
Medium (< 3 us)
Low (0.3-0.7 us)
Low (< 100 ns)
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 21
![Page 22: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/22.jpg)
Outline
Introduction to AdvancedTCA
Interconnect ProtocolsInterconnect Protocols① Ethernet② InfiniBand③ StarFabric③ StarFabric④ PCI Express⑤ RapidIO
Comparison of Interconnect Protocols① Feature② Protocol Efficiency③ Effective Bandwidth④ System Level Consideration
Conclusion
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 22
Conclusion
![Page 23: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/23.jpg)
Conclusion
• Low bandwidth backplane applicationsEthernet +) well understood, low risk
) hi h h d hi h l-) high overhead, high latency-) not scalable
• High bandwidth (tradeoffs of flexibility vs. latency, overhead)g ( y y, )InfiniBand and StarFabric
Complex header: +) for complex routing-) reduces effective bandwidth
RapidIOSmaller packets: +) to lower latency
-) overhead: a higher percentage of the packet) d ff ti b d idth-) reduces effective bandwidth
PCI ExpressMiddle: +) design goal: software transparency
-) constrains the protocol raises latency
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 23
-) constrains the protocol, raises latency
![Page 24: Backplane Technology Overview for AdvancedTCA](https://reader033.fdocuments.in/reader033/viewer/2022060201/5599a9b21a28abe4498b4631/html5/thumbnails/24.jpg)
Thank you for your attention !
Any questions ?
Special thanks to:
Rainer Ohlendorf
© Institute forIntegrated Systems
Hauptseminar am Lehrstuhl für Integrierte SystemeHui Chen, July 13th 2007 24