DATA CENTER OPTICS - ECOC 2018 | :: The Leading ... Center Optics ¢ Fiber versus Copper in the Data...
62
DATA CENTER OPTICS ECOC MARKET FOCUS, SEPTEMBER 24, 2013 Lisa Huff, CDCP Principal Analyst, Discerning Analytics Chief Technology Analyst, DataCenterStocks.com [email protected]
Transcript of DATA CENTER OPTICS - ECOC 2018 | :: The Leading ... Center Optics ¢ Fiber versus Copper in the Data...
DATA CENTER OPTICS ECOC MARKET FOCUS, SEPTEMBER 24, 2013
Lisa Huff, CDCP Principal Analyst, Discerning Analytics Chief Technology Analyst, DataCenterStocks.com [email protected]
AGENDA
¢ Data Center Networking Trends ¢ Data Center Optics ¢ Fiber versus Copper in the Data Center ¢ Vertical Market Summary
September 24, 2013
2
DATA CENTER NETWORKING TRENDS
¢ Higher speed networks ¢ Server trends ¢ Data center power and cooling
September 24, 2013
3
DATA CENTERS DEFINED
Facility Purpose Typical Data Rates
Typical # of Racks
Typical Business
TR Cross-Connections & Switches
100 Mbps – 1Gbps
< 5 All
Small DC (Reduced)
Servers, CPU, UPS, Switches
1 Gbps 5 - 20
Small Financial Hospitals
Medium DC (Basic)
Servers, CPU, UPS, POE, Local SAN,
Switches
1 - 10 Gbps
20 - 100 Banks Drug Co.
Universities
Large DC (Distributed)
Servers, CPU, UPS, POE,
Local & Remote SANs, Switches, Asset
Management
1 - 40Gbps > 100 Large Financial Retail
Insurance Large Research
Labs Co-location Facilities
September 24, 2013
4
LAN AND DATA CENTER GROWTH DRIVERS ¢ Government Initiatives
ó Data Policy Directive (DPD) – European Union (E.U.)
ó Basel Capital Accord (2006) ó HIPAA (U.S.), PIPEDA (Canada) ó Sarbanes-Oxley Act (US) ó HITECH (US) ó Broadband Initiative (US)
¢ Redundancy to protect against catastrophic loss
¢ Web 2.0/Internet Data Centers ¢ Cloud Computing
ó Both corporate and consumer
September 24, 2013
0.0 5.0
10.0 15.0 20.0 25.0 30.0 35.0 40.0
2006 2008 2010 2012 2014
Enterprise Storage Growth
Million Terabytes
Source: DataCenterStocks.com
5
DATA CENTER NETWORK ARCHITECTURE Septem
ber 24, 2013
100M/1G
10G
Multiple 1G 10G
1G
40G
40G
10G
100G
6
TOP OF RACK (TOR) SWITCHING Septem
ber 24, 2013
Source: Extreme Networks
7
DATA CENTER NETWORK TRENDS END OF ROW (EOR) SWITCHING
September 24, 2013
Source: Brad Hedlund, Cisco Systems
8
DATA CENTER NETWORK TRENDS ETHERNET NETWORK ARCHITECTURE
September 24, 2013
Top of Rack (ToR)
End of Row (EoR) 9
DATA CENTER NETWORK TRENDS LAN DATA RATE PROGRESSION
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000
8,000
2013 2014 2015 2016 2017
Mill
ions
of D
olla
rs
Year
Ethernet Optical Modules and AOCs Revenue
10G 40G 100G
September 24, 2013
10 Source: Discerning Analytics, LLC
DATA CENTER CABLING TRENDS
¢ Higher speed networks ¢ Server trends ¢ Data center power and cooling
September 24, 2013
11
X86 SERVER FORECAST BY DATA RATE Septem
ber 24, 2013
Source: DataCenterStocks.com
0.00
5.00
10.00
15.00
20.00
25.00
30.00
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
Mili
ons
of P
orts
Year
100M 1G 10G 40G 100G
12
SERVER TRENDS - MARKET PENETRATION BY DATA RATE
Time to Reach (years) Ethernet
Speed (Mbps) Million Port Sale
(after Ratification) 60% Market Penetration
10 1 100 1 5
1,000 2 9 10,000 6 11 40,000 7 13+
100,000 9 15+
September 24, 2013
13
SERVER TRENDS – FORM FACTORS BLADE SERVERS
• Servers on a card, containing processors, memory, integrated network controllers, an optional Fibre Channel host bus adapter (HBA) and other input/output (IO) ports.
September 24, 2013
Rack Server Blade Server 14
SERVER TRENDS Septem
ber 24, 2013
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Perc
enta
ge o
f Se
rver
s
Year
Pedestal Rack Blade
15
SERVERS VIRTUALIZATION – IMPACT ON THE NETWORK
¢ Virtualization is the creation of a virtual (rather than actual) version of something, such as an operating system, a server, a storage device or network resources
¢ Drastically reduces the number of servers needed in the data center
¢ Impacts the amount of cabling needed
¢ Impacts the network architecture – need faster server-to-server connections
September 24, 2013
16 Source: CA Associates, 2011
SERVER TRENDS BANDWIDTH DENSITY (Gbps/sqft or /rack)
¢ Server virtualization ó Pushing network connection data rates up ó Changing network traffic patterns from hierarchal to
horizontal ¢ Server Form Factor ó Blades
September 24, 2013
Source: HP
17
DATA CENTER CABLING TRENDS
¢ Higher speed networks ¢ Server trends ¢ Data center power and cooling
September 24, 2013
18
DATA CENTER TRENDS - POWER CONSUMPTION Septem
ber 24, 2013
Source: Oracle/Sun Microsystems 19
ASHRAE, Datacom Equipment Power Trends and Cooling Applications, 2005. © American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., www.ashrae.org.
kW/ft2
kW/rack
DATA CENTER HEAT LOAD TRENDS
September 24, 2013
20
MODULAR DESIGNS BASED ON COOLING REQUIREMENTS
September 24, 2013
Hot/Cold aisle containment POD with Liebert XDV overhead spot cooling
Source: Oracle/Sun Microsystems
Hot/Cold aisle containment POD with APC in-row cooling
21
DATA CENTER TRENDS COLD AISLE CONTAINMENT
September 24, 2013
22
COOLING TRENDS MOVING CABLING TO ABOVE CABINETS
September 24, 2013
23
COOLING TRENDS STRUCTURED CABLING
September 24, 2013
24
AGENDA
¢ Data Center Networking Trends ¢ Data Center Optics ¢ Fiber versus Copper in the Data Center ¢ Vertical Market Summary
September 24, 2013
25
GIGABIT ETHERNET OPTICAL VARIANTS
September 24, 2013
26
Ethernet Variant
Min. Reach (m)
Form Factors Media Wavelength
Standard IEEE 802.3
1000-BASE- SX 300 GBIC, SFP LOMF 850nm z
LX 5000 GBIC, SFP SMF 1310nm
ZX 70,000 GBIC, SFP SMF 1550nm Defacto
Source: Avago Technologies
GBIC SFP
10-GIGABIT ETHERNET OPTICAL VARIANTS
September 24, 2013
27
Ethernet Variant
Data Rate (Gb/s)
Min. Reach (m)
Form Factors Media Wavelength
Standard IEEE 802.3
10GBASE-
SR 10 300 XENPAK, XFP, X2, SFP+
LOMF 850nm
ae-2002/-2005
LR 10,000 300-pin, XENPAK, XFP, X2, SFP+
SFM 1310nm
LX4 4x2.5 300 XENPAK, X2 FDDI-grade MMF 850nm
ER 10 40,000 XENPAK, XFP, X2 SMF 1550nm
ZR 80,000 XFP, SFP+ SMF 1550nm
LRM 220 XFP, SFP+ FDDI-grade MMF 1310nm
aq-2006/-2008
Source: Finisar
300-pin XENPAK X2 XFP SFP+
40-GIGABIT ETHERNET OPTICAL VARIANTS
September 24, 2013
28
Ethernet Variant
Data rate (Gb/s)
Min. Reach (m)
Form Factors
Media Wavelength Standard IEEE 802.3
40GBASE-
SR4 4x10 100/125 QSFP+ LOMF 850nm ba
LR4 4x10 10,000 QSFP+, CFP SMF 1310nm
FR 40 2,000 CFP SMF 1310/1550nm bg
Source: JDSU
Source: Reflex Photonics
QSFP+
CFP
100 GIGABIT ETHERNET OPTICAL VARIANTS Ethernet Variant
Data rate (Gb/s)
Min. Reach (m)
Form Factors Media Wavelength
Standard IEEE 802.3
100GBASE-
SR10 10x10 100/150 CXP, CFP LOMF 850nm ba
LR4 4x25 10,000 CFP, CFP2, CFP4, QSFP28
SMF 1310nm
ER4 4x25 40,000 CFP SMF 1310nm
LR10 10x10 2,000 CFP SMF 1550nm Not supported
September 24, 2013
29
CXP CFP2
Source: Avago Technologies Source: Avago Technologies
Source: Oclaro
CFP Source: Ethernet Alliance
Source: Cisco Systems
CPAK
DATA CENTER ETHERNET OPTICAL DEVELOPMENT STANDARDS
¢ IEEE P802.3bm 40 Gb/s and 100Gb/s Fiber Optic Task Force ó 4x10G 40km SMF development ó 4x25G 500m SMF development ó 4x25G 20m and 100m LOMF development
¢ Study Groups ó 400 Gb/s Ethernet
¢ established in May 2013 ¢ Working on project request
September 24, 2013
30
DATA CENTER ETHERNET OPTICAL ROADMAP
September 24, 2013
31 Source: Ethernet Alliance, OFC 2013
FIBRE CHANNEL IN THE DATA CENTER Septem
ber 24, 2013
32
INFINIBAND AND HIGH PERFORMANCE COMPUTING
¢ Today’s Supercomputer
September 24, 2013
Source: Sun/Oracle
33
INFINIBAND ROADMAP
¢ Switched fabric communications link primarily used in high-performance computing.
September 24, 2013
2002 2005 2008 2011 2013 SDR DDR QDR FDR EDR
1X 2.5 (1x2.5) 5 10 14 26 4X 10 (4x2.5) 20 40 56 104 8X 20 (8x2.5) 40 80 112 208 12X 30 (12x2.5) 60 120 168 312
Aggregate Data Rates (Gbps)
Supercomputer Juropa (Jülich) #10 worldwide
Source: FCI
34
PRE-TERMINATED FIBER ASSEMBLIES EXAMPLE
September 24, 2013
35
Source: Corning
Source: Brocade
AGENDA
¢ Data Center Networking Trends ¢ Data Center Optics ¢ Fiber versus Copper in the Data Center ¢ Vertical Market Summary
September 24, 2013
36
2012 DATA CENTER CABLING MIX Septem
ber 24, 2013
63%
37%
Copper Fiber
Data Center Media Mix
6%
94%
Copper Fiber
SAN Media Mix
Source: DataCenterStocks.com 2012 Research 37
2017 DATA CENTER CABLING MIX Septem
ber 24, 2013
40%
60%
Copper Fiber
Data Center Media Mix
2%
98%
Copper Fiber
SAN Media Mix
Source: DataCenterStocks.com 2012 Research 38
DATA CENTER LAYOUT Septem
ber 24, 2013
Source: 42u.com 39
AIRFLOW: HOT AISLE/COLD AISLE
September 24, 2013
3’ - 4’
Cabinet or Rack
PERFED TILES
CABLE TRAYS POWER
CABLES
Front
Rear Rear
PERFED TILES
CABLE TRAYS POWER
CABLES
Cabinet or Rack
Cabinet or Rack
CABLE TRAYS or LADDER
RACK
CABLE TRAYS or LADDER
RACK
CABLE TRAYS or LADDER
RACK
RAISED FLOOR
CEILING
40
DATA CENTER COPPER CABLING UNDER RAISED FLOOR
September 24, 2013
41
DATA CENTER COPPER CABLING ABOVE CABINETS
September 24, 2013
42
AIRFLOW: HOT AISLE/COLD AISLE
September 24, 2013 Wire Basket
0.23”
0.25”
8.0”
4.0”
Air flow
240 channel Fiber solution 50/125 LOMF MDP
10 cables, Fill Ratio < 1 % 242 channel Cat 6 UTP solution
242 cables at 50% Fill Ratio *channels include 2 fibers
Source: Berk-Tek, A Nexans Company 43
DATA CENTER CABLING FIBER VERSUS COPPER COST COMPARISON SUMMARY
September 24, 2013
Model details: •Distribution prices based on 16-port 10-Gig switch blades for Cisco Catalyst 6509 • WS-X6716-10T-3CXL • WS-X6716-10G-3CXL w/X2 10GBASE-SR modules • Patch cords are 2m CAT6A, OM3 LOMF or SMF • Labor rate of $60/hour • Port cost computed by total cost of switch blade and transceivers/number of ports
Connection Type
Port Cost Patch Cords
Horizontal Cable and
Connectivity
Installation Labor per Channel
Total per Channel Cost
Fiber SM FT $7,370.00 54.00$ 91.53$ 43.67$ 7,559.20$ Fiber LOMF FT 3,350.00$ 45.10$ 111.60$ 34.67$ 3,541.36$ Fiber LOMF PT 3,350.00$ 45.10$ 172.54$ 6.25$ 3,573.89$ Copper UTP 1,762.50$ 17.88$ 223.53$ 81.34$ 2,085.25$ Copper FTP 1,762.50$ 27.72$ 225.00$ 132.78$ 2,148.00$
44
FIBER VERSUS COPPER COST COMPARISON WITH POWER CONSUMPTION
48-port GigE Blade
Blade Watts
# Blades/ Chassis
16-port 10GigE Blade
Blade Watts
# Blades/ Chassis
Total Watts
W/port
WS-X6748-SFP (Fiber) 280 7
WS-X6716-10G-3CXL 572 1 2532 7
WS-X6748-GE-TX (Copper 1) 377 7
WS-X6716-10T-3CXL 597 1 3236 9
WS-X6548-GE-TX (Copper 2) 189 7
WS-X6716-10T-3CXL 597 1 1920 5
September 24, 2013
Optical solution saves 1.4 kW/rack 1W power savings yields 2.84W total
Total possible savings = 4.kW/rack 45
FIBER VERSUS COPPER THE ‘CASCADE’ EFFECT
September 24, 2013
Source: DVL
46
FIBER VERSUS COPPER ENERGY SAVINGS ECONOMICS – SWITCH PORTS
¢ Save about $3000.00/rack/year with optical solution (at $0.12/kWh)
¢ ROI would take about 32 years ¢ To get a 10 year payback ó Electricity would have to cost $0.77/kWh ó Installed optical port needs to be < $1000.00 more
than an installed copper port ¢ Why not analyze a server rack? ¢ For further study ó Quantify airflow inefficiency with copper cabling
September 24, 2013
47
OTHER COMPELLING REASONS TO INSTALL FIBER
¢ Latency – nanoseconds versus microseconds ¢ Lower power consumption at higher data rates ¢ Smaller – will allow better airflow ¢ Higher density ¢ Longer reach ¢ Less weight in cable trays ¢ Not susceptible to EMI
September 24, 2013
48
DATA CENTER FIBER CABLING TYPES PROGRESSION
September 24, 2013
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2010 2011 2012 2013 2014 2015 2016 2017
OM2 OM3 OM4 OS2 49
AGENDA
¢ Data Center Networking Trends ¢ Data Center Optics ¢ Fiber versus Copper in the Data Center ¢ Vertical Market Summary
September 24, 2013
50
VERTICAL MARKET SUMMARY
Typical Business Size (sqft) # of Racks
Typical Data Rates (Gbps)
Networking Technologies
Small Financial Manufacturing K12 Schools
<1,000
5-20
1 4
Ethernet, iSCSI
Banks Drug Co.
Small Hospitals Higher Education
1,000-10,000
20-100
1 4 8
10
Ethernet, iSCSI, FC
Large Financial Retail
Insurance Large Research Labs Medium Healthcare
Web 2.0/IDCs ISPs
Co-location
>10,000
>100
1 4 8
10 16 40
100
Ethernet, iSCSI, FC, InfiniBand
September 24, 2013
51
VERTICAL MARKET BREAKDOWN DATA CENTER OPTICS
September 24, 2013
52
25%
13% 62%
Co-location Internet Enterprise
Source: Discerning Analytics, LLC 2012 Research
SUMMARY
¢ Gigabit Ethernet still the workhorse of the typical data center.
¢ 10G Ethernet is the largest revenue generator application in the data center for the next five years ó SFP+ 10GBASE-SR ó OM3 and OM4 Fiber ó SFP+ DAC 10GBASE-CR ó RJ45 10GBASE-T ó CAT6A/7 Twisted Pair
¢ Ethernet networks in the data center will continue to evolve into flatter topologies ó Higher data-rates will be pushed closer to the server
September 24, 2013
53
APPENDIX Case Study Contributors and References
September 24, 2013
DATA CENTER CASE STUDY WELLSPAN HEALTH
¢ Facilities ó Data centers support 2 hospitals, 10 health centers,
14 imaging centers, 16 laboratory locations, 55 physician offices
ó State-of-the-art 6,000 sq ft. data center in York, PA built in 2001 ¢ 480 MW, 1000 A ¢ Cooled by CRACs ¢ Continuously updated ¢ 5,100 sq. ft. additional space
ó State-of-the-art 1,400 sq ft. mirror data center in York, PA built in 2007 ¢ 480 MW, 800 A ¢ Cooled by CRACs ¢ Continuously updated
September 24, 2013
55
WELLSPAN HEALTH
¢ Network ó Three-tiered Ethernet network: Access,
Distribution/Aggregation, Core ¢ Access – Gigabit ¢ Distribution – Multiple Gigabit ¢ Core – Multiple 10G, looking at 40G
ó Topology ¢ Main Gigabit-rated cross-connect ¢ Transitioning to Top-of-Rack switching using
Cisco Nexus 2000 ¢ Cisco Nexus 5000 in Distribution/Aggregation
and Nexus 7000 in Core ó Multi-wavelength CWDM/DWDM between
buildings ó Separate FC SAN
¢ Looking at FCoE – still too expensive
September 24, 2013
56
WELLSPAN HEALTH
¢ Cabling ó Using TIA-942 as much as possible ó Copper in Access
¢ Category 6 ó Fiber in Distribution/Aggregation
¢ Pre-terminated LOMF OM2 and OM3 MPO assemblies with cassettes
ó Fiber in Core ¢ SMF between data centers and
buildings ó LOMF and SMF in SAN
September 24, 2013
57
ACKNOWLEDGEMENTS September 24, 2013
58
RESOURCES STANDARDS
¢ ANSI/BICSI 002-2011, Data Center Design and Implementation Best Practices
¢ FC-PI, Fibre Channel Physical Layer Standards ¢ IBTA InfiniBand Architecture Specification ¢ IEEE 802.3, Standard for Information Technology ¢ TIA-942, Telecommunications Infrastructure Standard for Data
Centers ¢ TIA-568, Commercial Building Cabling for Telecommunications ¢ TIA-1179, Healthcare Facility Telecommunications Infrastructure
Standard
September 24, 2013
59
RESOURCES ARTICLES, WHITE PAPERS, BLOGS, MARKET RESEARCH
¢ Huff, L. (2013), Ethernet Networks Progress to 400G, Connector Supplier
¢ Debiec, T. & Huff, L.(2013)Structured Cabling Technology and Market, Connector Supplier
¢ Huff, L. (2012) Multi-Gigabit Datacom Network Connectivity, Bishop & Associates, Inc.
¢ Huff, L. (2011) The Battle for the Converged Data Center Network, Data Center Knowledge.com
¢ Gross, D., & Huff, L. (2011). DataCenterStocks.com Blog. ¢ Huff, L. (2011). Optical Components Blog. ¢ Huff, L. (2011) Data Center Cabling – Copper versus Fiber Webinar ¢ Ponemon Institute (2011), Calculating the Cost of Data Center
Outages, Ponemon Institute and Emerson Network Power ¢ Hewlett Packard (2011) Building Virtualization-Optimized Data
Center Networks, www.hp.com. ¢ Gosai, Birendra (2011) Building the Next Generation Data Center,
CA Technologies Web site
September 24, 2013
60
RESOURCES ARTICLES, WHITE PAPERS, BLOGS, MARKET RESEARCH
¢ Huff, L. (2010). Data Center Cabling Cost Analysis. DataCenterStocks.com.
¢ Huff, L. (2010). DAC Report. DataCenterStocks.com. ¢ Huff, L. (2010). 40/100 GigE Markets: 2010 and Beyond.
Communications Industry Researchers. ¢ Huff, L. (2010). New Revenue Opportunities for Optical
Interconnects: A Market and Technology Forecast. Communications Industry Researchers.
¢ Cisco (2009), Data Center Top-of-Rack Architecture Design, Cisco ¢ Cisco (2007), Cisco Data Center Infrastructure 2.5 Design Guide,
Cisco ¢ APC (2005), Determining Total Cost of Ownership for Data
Center and Network Room Infrastructure, TechRepublic.com
September 24, 2013
61
MULTI-SOURCE AGREEMENTS (MSAS) AND SFF ¢ 10G ó SFP+: SFF-8431
¢ 40G ó QSFP+: SFF-8436
¢ 100G ó CXP: InfiniBand Supplement ó CFP/CFP2/CFP4: CFP MSA ó 10x10: 10x10 MSA ó QSFP28: SFF-8661
¢ 400G ó CDFP
September 24, 2013
62
