ANSI/BICSI 002, Data Center Design and Implementation Best Practices

Rick Ciordia, PE, RCDD, DCDC, RTPM, CT

ETTG, PSC.

CALA District Chair - BICSI

• Develop standards within all facets of Information & Communications Technology (ICT) infrastructure design and installation

• Details:

– Over 450 member worldwide

– Accredited by ANSI

– Develops international open to use/“royalty free” standards and best practices

BICSI International Standards Program

BICSI Standards Within ICT

IEEE ISO/IEC & TIA

BICSI

Defines the message and transmission characteristics

Defines the transmission media and system specifications

Defines how to design solutions using transmission media and systems

Standards Development Cycle

Industry Needs

Technology Trends

Designers

Suppliers

Clients

Use

Idea to

Consensus

Body

Draft

Consensus

Ballot

Review of

Results

• Building/Facilities

– K12 / Primary Education Facilities (001)

– Data Centers (002)**

– Healthcare Facilities (004)

• Systems

– ESS (005)*

– DAS (006)**

• Installation

– Cable Installation (NECA/BICSI 568)

– ICT Bonding and Grounding (NECA/BICSI 607)*

• Other

– BIM for ICT (003)

BICSI Published Standards

* Available in Spanish ** Translation into Spanish planned or in progress

Current Activities

New Standards

• DAS Design – Moving to publication

• ICT Systems for Intelligent Buildings

• OSP Installation and Construction – General Principles

– Aerial Pathway

Revision Activities

• Revision to BICSI 005, ESS – IP Fire Alarm

• Revision to BICSI 001, Educational Facilities – Expanded scope to include

secondary, graduate, post-graduate facilities

About ANSI (American National Standards Institute)

• Promotes standards use within United States

• Accreditation body

– Standards Development

– Credentialing Bodies (ISO 17024)

– Testing Laboratories (ISO 17025)

• Ensures open and unbiased standards development processes

• Does not create standards

Reach of BICSI Standards

ANSI/BICSI 002-2014

Data Center Design and Implementation Best Practices

Data Center Standards

ANSI/TIA-942-A (2012) Telecommunications Infrastructure Standard

for Data Centers

118 pages

ISO/IEC 24764 (2010) Information technology — Generic cabling

systems for data centres

44 pages

ASHRAE TC9.9 (2012) Thermal Guidelines for Data Processing

Environments, 3rd edition

150 pages

CENELEC EN 50173-5:2007 / A2:2012

Information technology — Generic cabling systems - Part 5: Data centres

48 pages

CENELEC EN 50600 (2012-) Information technology — Data centre

facilities and infrastructures

Multiple Documents

ANSI/BICSI 002-2014 Data Center Design and Implementation

Best Practices

534 pages

Comparing the Data Center Standards

• Data Center Design – BICSI 002, CENELEC 50600

• Computer Room and Cabling Infrastructure – TIA 942-A

• Cabling Infrastructure – ISO/IEC 24764, CENELEC EN 50173-5

• Environmental Conditions – ASHRAE TC 9.9

• Others – Local / Country

– Certifications

Design Standard - CENELEC 50600

EN 50600-1 (2012) General Concepts 32 Pages

EN 50600-2-6 Management and

Operational Information

EN 50600-2-5 Security Systems

EN 50600-2-4 Telecommunications

Cabling Infrastructure

EN 50600-2-1 (2014) Building Construction

36 Pages

EN 50600-2-2 (2014) Power Distribution

40 Pages

EN 50600-2-3 (2014) Environmental Control

32 Pages

BICSI 002, Data Center Design and Implementation Best Practices

• More than just cabling

• Expands requirements from other standards

• Covers additional topics not within other standards

BICSI 002-2014 Breakdown

Site / Building

18%

Electrical 26%

HVAC / Mechanical

13%

Cabling / Networks

22%

Security 14%

Other 7%

Represents 408 pages of normative content across 13 sections

Content Revision and Expansions

• Availability Classes

• Modular Data Centers*

• Hot/Cold Aisles

• Mechanical Systems*

• DCIM*

• Circuit Maps and DC Power

• Cabinet Airflow and Cabling Capacity

• “Green” / Efficiency*

• Building Structure

• Site Hazards

• Data Center Services Outsourcing Model*

• Bonding & Grounding

• Commissioning

• Network Security*

• Telecommunications Cabling

• (And More …)

* Indicates all new content to this edition

BICSI Design Classes and Selection Methodology

Finding the Right Design

• Methodologies include:

– Loss prevention vs. initial build cost

– Total Cost of Ownership (TCO)

– Desired availability level

• All provide a starting point for design requirements

• Based on three questions

1. How much downtime per year will be allowed for maintenance?

2. During scheduled operation, what is the maximum allowed downtime?

3. What is downtime’s impact to operations?

• Answers will indicate design class for starting point of requirements

BICSI Design Class Determination

Interaction of Answers

Define

Operational

Requirements

Define Impact

of Downtime

Identify Required

Availability Class

Define

Operational

Availability

Requirements

• Class 0: Single path, and fails to meet one or more criteria of Class 1

• Class 1: Single path

• Class 2: Single path with redundant components

• Class 3: Concurrently maintainable & operable

• Class 4: Fault tolerant

BICSI DC Design Classes

Availability Class Prefixes

• Class Fx: Facility (Electrical & Mechanical)

• Class Cx: Cable Plant

• Class Nx: Network Infrastructure

• Class Sx: Data Processing and Storage Systems

• Class Ax: Applications

Class F1 Electrical Example

Electrical Distribution

Mechanical

Switchgear

Non-Critical

Loads

Mechanical

Loads

Critical

Loads

UPS

PDU

Utility

Static

Bypass

Maintenance

Bypass

Transfer

Switchgear

Alternate

Power

Source

Class F2 Electrical Example

Electrical Distribution

Mechanical

Switchgear

Non-Critical

Loads

Mechanical

Loads

Critical

Loads

UPS

PDU

Utility

Static

Bypass

Maintenance Bypass

Transfer

Switchgear

Alternate

Power Source N

Alternate

Power Source +1

N

UPS

+1

Electrical Class F3

Electrical Distribution

Mechanical

Switchgear

Non-Critical

Loads

Mechanical

Loads

UPS

PDU

Critical

Loads

Static

Bypass

Maintenance

Bypass

N

UPS

+1

Electrical Distribution

Mechanical

Switchgear

Transfer

Switchgear

Alternate

Power Source N

Alternate

Power Source +1

Non-Critical

Loads

Output Distribution Switchboard Alternate Switchboard

PDU

Critical Mechanical

Switchgear

Critical Mechanical

Switchgear

Critical

Fans/Pumps

Utility

Mechanical

Loads

Electrical Class F4

Electrical Distribution

Mechanical

Switchgear

Non-Critical

Loads

Mechanical

Loads

UPS

PDU

Critical

Loads

Static

Bypass Maintenance

Bypass

N

UPS

+1

Electrical Distribution

Mechanical

Switchgear UPS

Static

Bypass Maintenance

Bypass

UPS

Non-Critical

Loads

N +1

Output Distribution Switchboard Output Distribution Switchboard

PDU

Critical Mechanical

Switchgear

Critical Mechanical

Switchgear

Critical

Fans/Pumps

Utility Transfer

Switchgear

Alternate

Power Source N

Alternate

Power Source +1

Transfer

Switchgear

Alternate

Power Source N

Alternate

Power Source +1

Critical

Loads

Non-Critical

Loads

Non-Critical

Loads

Normal

UPS

N

Stand-

by

UPS

+1

Overlap Switch

(Manual)

PDU-A

Sta

tic B

ypass Input

Main Stand-by

Normal

UPS

N

Stand-

by

UPS

+1

Sta

tic B

ypass Input

To Mechanical Systems

See Section 10 for configuration

of mechanical systems

Normal

UPS

N

Stand-

by

UPS

+1

Overlap Switch

(Manual)

PDU-B

Sta

tic B

ypass Input

Main Stand-by

Sourc

e 2

Sourc

e 1

Sourc

e 2

Sourc

e 1

Sourc

e 2

Sourc

e 1

Sourc

e 2

Sourc

e 1

Source

2

Source

1 Source

2

Source

1

Medium Voltage Electrical Distribution 2 Medium Voltage Electrical Distribution 1

Medium Voltage

Distribution 1

High Voltage

Receiving 1

High to Medium Voltage

Transformer 1

Medium Voltage

Distribution 2

Alternate

Power Source

Alternate

Power Source N

+

1

High Voltage

Receiving 3

Utility Utility

Switchgear

High Voltage

Receiving 2

High to Medium Voltage

Transformer 2

Class F4 Electrical “Catcher”

Telecommunication Classes

ITE ITE ITE ITE

Equip

Dist Area

Equip

Dist Area

Equip

Dist Area

Equip

Dist Area

ITE ITE ITE ITE

Equip

Dist Area

Zone Dist

Area

Equip

Dist Area

Equip

Dist Area

Equip

Dist Area

Only required for large computer rooms due to port

counts or distance limitations

Horiz Dist

Area

Horiz Dist

Area

Horiz Dist

Area

Horiz Dist

Area

Main Dist Area

Service

Provider

Service

Provider

Entrance Room

Telecomunicaciones

Room

Computer

Room

Entrance Room

Main Dist Area

Service

Provider

Service

Provider

Maintenance

Hole

Maintenance

Hole

Zone Dist

Area

Zone Dist

Area

Class C1 Class C3 Class C2 Class C4

Offices,

Operations Center,

Support Rooms

Chiller

Air-Cooled

Condenser

(Dry-Cooler)

CRAH

• No redundancy present

• All power feeds from common upstream distribution

• No ability to be maintained under load

Mechanical Class F1

Air-Cooled

Condensers

(Dry-Coolers)

Chiller Chiller

CRAH

Mechanical Class F2

• Redundant critical components

• All power feeds from common upstream distribution

• Only redundant components able to be maintained under load

Mechanical Class F3

• Redundancy in all areas

• Equipment and controls with redundant systems fed from different electrical distribution

• Maintainable when actions do not reduce cooling capacity below "N+1"

Chiller

Air-Cooled

Condensers

(Dry-Coolers)

Chiller

CRAH

Air-Cooled Condensers (Dry-Coolers)

Chiller Chiller Chiller

CRAH

Mechanical Class F4

• Redundant equipment and piping for maintenance

• Power feed so that cooling capacity does not drop below "N" when maintaining mechanical or upstream electrical distribution

• Maintainable when actions do not decrease cooling capacity below "N"

Class N0/N1 and N4 Network

Class S4 System and A4 Application

Availability and Multi-Site Data Centers

• Prior to virtualization, subclasses aligned through data center

• Today, a single data center may not have alignment

• Availability class methodology can in discussions about using multiple data centers to achieve availability target

Example: Class 3 Availability Using Three Class 2 Data Centers

Modular Data Center Design

• Modular may describe:

– How a data center is segmented

– Method of construction/implementation

• Both can assist in allowing for different availability needs for one site

Modular/Containerized Data Center Space Adjacencies

F2

F3 F4

Modular Design Planning Aides

BICSI-002, 2014 Supplement

Are BICSI & Uptime Similar?

• ANSI/BICSI 002-2014

This standard provides a reference of common terminology and design practice … a framework for

the process to determine facility criticality and to develop optimum design & implementation solutions

• Uptime Tiers “Only data center benchmarking system developed by and for data center owners

Performance-based on fundamental concepts

– Not a checklist, design menu, or cookbook”

Source: Uptime Institute: Tier Classification System & Operational Sustainability presented by Dana Smith, Director of Development, Uptime Institute at BICSI Andino 2012

Standards

BICSI

Colombia

o NTC-6064 - Tecnología de la Información. Cableado Genérico para instalaciones de clientes.

Especifica el cableado genérico para uso dentro de instalaciones que pueden comprender una o varias edificaciones en un predio (campus).

Comprende el cableado balanceado y el cableado de fibra óptica.

Referencia: ISO/IEC 11801

Colombia (Cont.)

o RITEL - Reglamento Técnico para Redes Internas de Telecomunicaciones

Establece las medidas relacionadas con el diseño/

construcción y puesta en servicio de las redes internas de

telecomunicaciones.

Referencia: ISO/IEC 11801

Costa Rica

o Colegio Federado de Ingenieros y Arquitectos Todos los planos de instalaciones electricas y de

telecomunicaciones, de voz y datos deberan cumplir donde corresponda con: El NEC Codigo Electrico Nacional (NFPA 70) en su ultima version en

espanol.

NFPA 70 E “Norma para la seguridad electrica de los empleados en los lugares de trabajo”, en su ultima version en espanol.

Las ultimas revisiones y adenda aprobados de las normas ANSI/EIA/TIA 568, 569, 570, 606, 607.

Mexico

o NMX-I-108-NYCE-2006, Telecomunicaciones – Cableado – Cableado estructurado – Puesta a tierra en sistemas de telecomunicaciones

o NMX-I-132-NYCE-2006, Telecomunicaciones – Cableado – Cableado estructurado – Especificaciones de las pruebas de cableado balanceado – Parte 01: Cableado instalado

o NMX-I-154-NYCE-2008, Telecomunicaciones – Cableado – Cableado estructurado – Cableado genérico residencial

o NMX-I-248-NYCE-2008, Telecomunicaciones – Cableado – Cableado estructurado – Cableado de Telecomunicaciones para edificios comerciales – Especificaciones y métodos de prueba

o NMX-I-279-NYCE-2009, Telecomunicaciones - Cableado - Cableado estructurado - Canalizaciones y espacios para cableado de telecomunicaciones en edificios comerciales

Mexico (Cont.)

o NMX-I-14763-1-NYCE-2010, Telecomunicaciones – Cableado – Cableado estructurado – Implementación y operación de cableado en edificios comerciales – Parte 1: Administración

o NMX-I-24764-NYCE-2013, Tecnología de la información – Sistema de cableado genérico para centros de datos

o NMX-J-364/I-ANCE-2011, Instalaciones eléctricas – Parte 1: Principios fundamentales, planeación de características generales, definiciones

o NMX-J-C-I-489-ANCE-ONNCCE-NYCE-2014, Centros de datos de alto desempeño sustentable y energético – Requisitos y métodos de comprobación

o NOM-001-SEDE-2012, Instalaciones Eléctricas (Utilización)

ANCE - Asociación Nacional de Normalización y Certificación del Sector Eléctrico NMX - Norma Mexicana NOM - Norma Oficial Mexicana NYCE - Normalización y Certificación Electrónica ONNCCE - Organismo Nacional de Normalización y Certificación de la Construcción y Edificación

Standards In Use

Country NEC TIA ISO BICSI Uptime ICREA National Other

Chile X X Colombia RL X R X X X RN Costa Rica RLN R X El Salvador X X X Guatemala X X X Honduras X X X Mexico RN X X X X X RN Panama X X X Peru X X Puerto Rico R X X Republica Dominicana

X

Uruguay X X X X Venezuela RL R R R

Notes: X = Being Used Not Required RN = National Version

R = Required by law, regulation, etc. RL = National Version based on NEC

RLN = NEC version adopted as national

DC Standards

Organization BICSI TIA ISO CENELEC ASHRAE ICREA

Standard BICSI 002

(2014) TIA 942A

(2012) ISO 24764

(2010) EN 50173-5 / EN 50600

ASHRAE TC9.9

ICREA-Std-131-2013

Recognition Worldwide US /Canada

(Int’l) Worldwide

EU (Int’l)

US/Canada (Int’l)

Int’l (21

Countries)

Are

as

Co

vere

d

Electrical Yes Yes Yes

Mechanical Yes Yes

Telecom Yes Yes Yes Draft Yes

Environment Yes Yes Yes Yes Yes

Site Yes

Security Yes Draft Yes

Dat

a C

en

ter

Security Standards

Building System Standards

Environmental Conditions

Fire Code

Electrical Standards

Cabling Standards

Cabling Spaces Standards

Building Code

Electrical Code

Government Regulations

Planned Interoperability

(BIC

SI 0

02

-20

14

)

Security Standards

Building System Standards

Environmental Conditions

Fire Code

Electrical Standards

Cabling Standards

Cabling Spaces Standards

Building Code

Electrical Code

Government Regulations

Interoperability Demonstrated

NFPA, ASIS, Government

TIA 862, AHSRAE 13

ASHRAE

NFPA 1, NFPA 72

IEEE, NFPA, NECA

TIA 568

TIA 942

International Building Code (IBC)

NFPA 70 (NEC)

FCC, OSHA

(BIC

SI 0

02

-20

14

) U

nit

ed

Sta

tes

RCMP, Government, ISO

National Fire Code of Canada (NFC)

CSA, IEEE

National Building Code of Canada (NBC)

CSA C22.1 (CEC)

Industry Canada, Labour Program

Can

ada

Co

sta

Ric

a

Government

Government, NFPA

IEEE, NFPA

Government

NFPA 70 (NEC)

Government

Co

lom

bia

Government, ISO

ISO

RITEL, ISO

ISO

Why BICSI 002?

“If you fail to plan,

you are planning to fail!” “¡Si fallas en planificar,

estas planificando fallar!”

- Benjamin Franklin

Thank You! Rick Ciordia, PE, RCDD, DCDC, RTPM, CT

ETTG, PSC.

rciordia@ettgpr.com