Network Architecture

Tewfiq El Maliki, HES-SO

Data Center Physical Layout

2

DATA CENTERS

• Small: < 1 MW, 4 racks per unit

• Medium: 1-4 MW, 10 racks per unit

• Large: > 4 MW, 20 racks per unit

• Built-in cooling, high PUE (power usage effectiveness) 1.02 PUE =

Power In/Power Used

• Rapid deployment

3

Câblage non-structuré et structuré

4

DC Equipment Cabinets

5

ANSI/TIA-942-2005 Standard 2005

• Main Distribution Area (MDA)

• Horizontal Distribution Area (HDA)

• Equipment Distribution Area (EDA)

• Zone Distribution Area (ZDA)

6

Link technologies : Copper or Fiber

• Fiber

• Cheaper per unit distance

• Low weight (10g/m)

• Max distance for 10G: 300m (with cheap optics)

• Typically multi-mode fiber (MMF)

• Copper

• Cheaper for short runs (< 8m)

• Heavier (100g/m), larger size cable

• Max distance for 10G : ~8m (real) 15m (spec) 30m (future?)

• Typically Twin-axial cable

7

Network Topology

8

Power

9

Energy Problem

• Pretty good data centers have efficiency of 1.7

• 0.7 Watts lost for each 1W delivered to the servers

• Breakdown:

• Servers: 59%

• Distribution losses: 8%

• Cooling: 33%

10

Data Centers Cost and Topology

11

DATA Centers

12

Performance Isolation

• VLANs typically provide only reachability isolation

• Problem of one service sending/receiving much traffic

13

Poor server to server connection

• Data centers run 2 kinds of apps

• Internal computation ~80% paquets stay

• Outward facing

14

ToR or EoR

15

Architecture

16

Clos Network

17

Mansoon Clos network Example

18

Server-to-server Forwarding

19

DCN Requirements

Needs to be Scalable, Secure, Shared, Standardized, and Simplified (5 S's)

Converged Infrastructure: Servers, storage, and network have to work together

Workload Mobility: Large L2 domains required for VM mobilityEast-West Traffic: Significant server-to-server traffic as compared

to server to user. One Facebook request required 88 cache looks, 35 database lookups, 392 backend RPC calls. Intranet traffic 935X the http request/response [Farrington]

Storage traffic on Ethernet: Congestion management on Ethernet (Errors)

Ref: A. Kindlness, "The Forester Wave: Data Center Networking Hardware," Jan 23, 2013, http://ca.westcon.com/documents/46488/forrester_wave_data_cetner_networking_hw_q1_2013.pdfRef: N. Farringon and A. Andreyev, “Face book’s Data Center Network Architecture,”2013 IEEE Optical Interconnect Conference, http://nathanfarrington.com/papers/facebook-oic13.pdf

Facebook Network –Source NF & AA

21