No Interruptions: Commissioning Measures in Preventing ...
Transcript of No Interruptions: Commissioning Measures in Preventing ...
By Ryan S. Lean and Phil Karnicki
No Interruptions: Commissioning Measures in Preventing Unplanned Downtime
AIA Quality Assurance
The Building Commissioning Association is a Registered Provider with The American Institute of Architects Continuing Education Systems (AIA/CES). Credit(s) earned on completion of this program will be reported to AIA/CES for AIA members. Certificates of the Completion for both AIA members and non-AIA members are available upon request.
This program is registered with AIA/CES for continuing professional education. As such, it does not include content that may be deemed or construed to be an approval or endorsement by the AIA of any material of construction or any method or manner of handling, using, distributing, or dealing in any material or product.
Questions related to specific materials, methods, and services will be addressed at the conclusion of this presentation.
BCxA Annual Conference – Chicagoland, IL – September 2019 2
Learning Objectives
BCxA Annual Conference – Chicagoland, IL – September 2019 3
1. Understand the importance of systematic planning at each stage to minimize unforeseen risks.
2. Identify and plan around risks associated with critical testing procedures.3. Understand the challenges associated with each trade and the impacts
they have on each other. 4. Leave with a greater appreciation of how difficult it is to bring new
systems online in a critical environment.
BCxA Annual Conference – Chicagoland, IL – September 2019 4
Chapter 1: OverviewWhy Does It Matter?
New Construction Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 5
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
• Typical apartment/hotel room units (PTAC, FCU, HP)
• Standard chiller plant or condenser water system
• Steam or hot water boiler plants• Standard fire alarm and fire pump
(wet sprinkler)• Service switchgear and distribution
panels• Life safety emergency generator
Residential/Hospitality Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 6
CR
ITIC
AL M
ETER
HIGH
LOW
• Typical terminal units for open office (VAV, FPB)
• Air handling system solution• Chiller plant or condenser water
system (if core and shell)• Packaged air conditioning units
(electric closets, etc.)• Steam or hot water boiler plants• Fire pump and fire tanks• Service switchgear and distribution
panels• Emergency generator serving LS and
emergency loads
Commercial Office Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 7
CR
ITIC
AL M
ETER
HIGH
LOW
Cultural Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 8
• Custom solution for open galleries (VAV, FPB)
• Air handling system solution with humidity control
• Chiller plant • Steam or hot water boiler plants• Fire pump and dry/gas fire
protection systems• Service switchgear and
distribution panels• Emergency generator serving
LS and emergency loads
CR
ITIC
AL M
ETER
HIGH
LOW
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 9
• Redundancy across all systems• Air handling units for ventilation for trading• Packaged A/C units for data centers
(CRAC, CRAH)• Chiller plant (air and water cooled)• Electrical heating (for data centers with low
office reqs.)• Fire pump and dry/gas fire protection
systems• Service switchgear and distribution panels• Emergency generator serving LS and
emergency loads• UPS for data center
CR
ITIC
AL M
ETER
HIGH
LOW
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 10
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 11
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
MOP’s:Call everyone!
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 12
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
MOP’s:Call everyone!
PFPC, STR, BMS and FPT all
on same day
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 13
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
Night and
weekend testing
MOP’s:Call everyone!
PFPC, STR, BMS and FPT all
on same day
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 14
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
Night and
weekend testing
MOP’s:Call everyone!
IST in phases as equipment comes online
PFPC, STR, BMS and FPT all
on same day
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 15
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
Night and
weekend testing
MOP’s:Call everyone!
IST in phases as equipment comes online
Turnover immediate
(no deficiencies)
PFPC, STR, BMS and FPT all
on same day
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 16
STRManufacturer’s Start-Up
BMSBMS Programming
TABHVAC Testing, Adjusting and Balancing
FPTFunctional Performance Test
ISTIntegrated Systems Test
FATFactory Acceptance Testing
PFPCPre-Functional Performance Checklist
Cx Design ReviewCx Submittal Review
Systems Manual Training VerificationWarranty Review
Night and
weekend testing
MOP’s:Call everyone!
IST in phases as equipment comes online
Turnover immediate
(no deficiencies)
PFPC, STR, BMS and FPT all
on same dayTesting in isolation prior to swingover
Critical Facility Commissioning Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 17
• Commissioning is controlled chaos Critical nature of Data Center testing
requires attention Clear processes drive:
◦ Reduced stress ◦ Reduced confusion◦ Increased awareness
Practice, practice, practice (rehearsal)
Critical Facility Overview
BCxA Annual Conference – Chicagoland, IL – September 2019 18
• What is a data center?• Needs Power Cooling Security
• Requirements SLA’s (service level agreements) Contracts MOPs, SOPs, EOPs, etc.
Critical Facility Cx Flow Process
BCxA Annual Conference – Chicagoland, IL – September 2019 19
• Factory Acceptance Testing (FAT) Imperative to have both operating engineers and owner at FAT.
• Start-Up (STR) critically important.• Integrated Systems Test (IST) Scripts to be specific and updated (based on current technologies) Imperative for Client to review prior to testing
Critical Facility Cx Flow Process
BCxA Annual Conference – Chicagoland, IL – September 2019 20
• Method of Procedure (MOP)
Critical Facility Cx Flow Process
BCxA Annual Conference – Chicagoland, IL – September 2019 21
• Method of Procedure (MOP) Determine very early which entity owns MOP development.
Something to be determined in the Cx contract. Most existing data centers have developed their own MOPs for
existing equipment maintenance. CxP to provide and offer input on Cx process with a focus on that
specific facilities requirements. Help to manage RISK:
◦ Procedures change risk level (lose redundancy, add single points of failure).
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 22
• HVAC Equipment Air handling units for ventilation for trading Packaged A/C units for data centers (CRAC, CRAH) Chiller plant (air and water cooled) Hot water plants (only for office/trading component) Electrical heating (for data centers with low office reqs.)
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 23
• Electrical Equipment Service/distribution switchgear Emergency generators Uninterruptible power supply (UPS) Static transfer switches (STS) Power distribution units (PDU) Automatic transfer switches (ATS) Lighting and lighting controls
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 24
• Plumbing Equipment Hot water heaters
• Fire Protection Equipment Fire pump Preaction systems Dry pipe systems Nitrogen systems VESDA
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 25
• Low Voltage Systems Building management system (BMS) Security SCADA
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 26
• Simulate your IST conditions
Critical Facility Commissioning Projects
BCxA Annual Conference – Chicagoland, IL – September 2019 27
• Integrated System Testing (IST) Must simulate the conditions (load test). Account for proper logistics (e.g.; where to put load banks). Test the electrical with the HVAC (i.e.; Room Verification Test). Use rack mounted load banks if available.
BCxA Annual Conference – Chicagoland, IL – September 2019 28
Chapter 2: Specific Challenges
Working with Existing Systems
29
Critical Facility Commissioning in Existing BuildingsHVAC
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 30
CRACs/CRAHs and chiller plants to be tied into existing systems.
Data center clients must be protected from any type of outage (e.g.; CHW plant goes
down due to inadvertent alarm).
Challenges
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 31
CRACs/CRAHs and chiller plants to be tied into existing systems.
Data center clients must be protected from any type of outage (e.g.; CHW plant goes
down due to inadvertent alarm).
Have MovinCool ready in case remaining CRACs/CRAHs go down.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 32
CRACs/CRAHs and chiller plants to be tied into existing systems.
Data center clients must be protected from any type of outage (e.g.; CHW plant goes
down due to inadvertent alarm).
Have MovinCool ready in case remaining CRACs/CRAHs go down.
If redundant CHW plant, prepare valve arrangement for quick changeover. If no
redundant CHW plant, potential arrangement for roll-up ACC.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 33
SLAs will have temperature requirements for server racks and notices to clients if the
temperatures are out of range.
Testing including ISTs and Room Verification Testing, needs to be done within specific time
frames and windows.
Challenges
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 34
SLAs will have temperature requirements for server racks and notices to clients if the
temperatures are out of range.
Testing including ISTs and Room Verification Testing, needs to be done within specific time
frames and windows.
Establish baseline trends prior to swing over. Confirm all sensor calibration prior to
swing over.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 35
SLAs will have temperature requirements for server racks and notices to clients if the
temperatures are out of range.
Testing including ISTs and Room Verification Testing, needs to be done within specific time
frames and windows.
Establish baseline trends prior to swing over. Confirm all sensor calibration prior to
swing over.
Coordinate load bank requirements and logistics of placement, including all cabling
and time for setup and breakdown.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 36
Control points to be integrated into existing BMS software.
• DP, OA, flow sensors.• Programming that selects lowest value or
an average.• Chiller plant line-ups, pump rotation, etc.
Challenges
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 37
Control points to be integrated into existing BMS software.
• DP, OA, flow sensors.• Programming that selects lowest value or
an average.• Chiller plant line-ups, pump rotation, etc.
Functionally test each unique piece of equipment prior to integration.
Confirm building standard prior to matching approved BMS submittal.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 38
“False” alarms during swing over (cause shutdowns, maybe cascading).
Migration to corporate system (before you get locked out).
Challenges
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 39
“False” alarms during swing over (cause shutdowns, maybe cascading).
Migration to corporate system (before you get locked out).
Functionally test each unique piece of equipment prior to integration.
Confirm no existing alarms on network.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsHVAC
BCxA Annual Conference – Chicagoland, IL – September 2019 40
“False” alarms during swing over (cause shutdowns, maybe cascading).
Migration to corporate system (before you get locked out).
Functionally test each unique piece of equipment prior to integration.
Confirm no existing alarms on network.
Coordinate with corporate for date (moving target) for migration.
Finish commissioning prior to migration.
Challenges Potential Solutions
BCxA Annual Conference – Chicagoland, IL – September 2019 41
Critical Facility Commissioning in Existing BuildingsElectrical
Critical Facility Commissioning in Existing BuildingsElectrical
BCxA Annual Conference – Chicagoland, IL – September 2019 42
Generators, UPS, STS, PDU, switchgear and controls tied into existing systems.
Single-ended vs. double-ended power. Power cannot be interrupted to the clients.
Challenges
Critical Facility Commissioning in Existing BuildingsElectrical
BCxA Annual Conference – Chicagoland, IL – September 2019 43
Generators, UPS, STS, PDU, switchgear and controls tied into existing systems.
Single-ended vs. double-ended power. Power cannot be interrupted to the clients.
Must engage client on potential testing pathways. Develop a mock control system.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsElectrical
BCxA Annual Conference – Chicagoland, IL – September 2019 44
Generators, UPS, STS, PDU, switchgear and controls tied into existing systems.
Single-ended vs. double-ended power. Power cannot be interrupted to the clients.
Must engage client on potential testing pathways. Develop a mock control system.
Dry run testing procedures. Utilize existing transfer systems to protect downstream loads
(with client permission).
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsElectrical
BCxA Annual Conference – Chicagoland, IL – September 2019 45
Cascading issues that could cause an upstream circuit breaker to trip open.
Upstream breakers may be supporting client loads.
Lighting controls for office spaces.
Challenges
Critical Facility Commissioning in Existing BuildingsElectrical
BCxA Annual Conference – Chicagoland, IL – September 2019 46
Cascading issues that could cause an upstream circuit breaker to trip open.
Upstream breakers may be supporting client loads.
Lighting controls for office spaces.
Mitigate risk by performing dry runs and developing concrete back-out plans.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsElectrical
BCxA Annual Conference – Chicagoland, IL – September 2019 47
Cascading issues that could cause an upstream circuit breaker to trip open.
Upstream breakers may be supporting client loads.
Lighting controls for office spaces.
Mitigate risk by performing dry runs and developing concrete back-out plans.
Coordination to access client spaces.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsPlumbing & Fire Protection
BCxA Annual Conference – Chicagoland, IL – September 2019 48
Tying into existing piping.
Challenges
Critical Facility Commissioning in Existing BuildingsPlumbing & Fire Protection
BCxA Annual Conference – Chicagoland, IL – September 2019 49
Tying into existing piping.Drain down lines in order to perform work if available. Confirm flushing procedures
followed prior to tie-in.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsPlumbing & Fire Protection
BCxA Annual Conference – Chicagoland, IL – September 2019 50
Pre-action and dry pipe sprinkler systems over operating equipment.
Accidental discharge of gaseous systems.
False alarms during testing (evacuate an operating building).
Challenges
Critical Facility Commissioning in Existing BuildingsPlumbing & Fire Protection
BCxA Annual Conference – Chicagoland, IL – September 2019 51
Pre-action and dry pipe sprinkler systems over operating equipment.
Accidental discharge of gaseous systems.
False alarms during testing (evacuate an operating building).
Provide protection under piping near operating equipment.
Take equipment offline if allowed.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsPlumbing & Fire Protection
BCxA Annual Conference – Chicagoland, IL – September 2019 52
Pre-action and dry pipe sprinkler systems over operating equipment.
Accidental discharge of gaseous systems.
False alarms during testing (evacuate an operating building).
Provide protection under piping near operating equipment.
Take equipment offline if allowed.
Have safety provisions including egress paths open and masks ready.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsPlumbing & Fire Protection
BCxA Annual Conference – Chicagoland, IL – September 2019 53
Pre-action and dry pipe sprinkler systems over operating equipment.
Accidental discharge of gaseous systems.
False alarms during testing (evacuate an operating building).
Provide protection under piping near operating equipment.
Take equipment offline if allowed.
Have safety provisions including egress paths open and masks ready.
Take the building offline prior to testing.
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsSecurity
BCxA Annual Conference – Chicagoland, IL – September 2019 54
New spaces need security to be coordinated with existing spaces.
Access to these spaces may be difficult with Client requirements.
Challenges
Critical Facility Commissioning in Existing BuildingsSecurity
BCxA Annual Conference – Chicagoland, IL – September 2019 55
New spaces need security to be coordinated with existing spaces.
Access to these spaces may be difficult with Client requirements.
Engage security vendor/designer for existing spaces. Submittal coordination is critical
(existing vs. new install).
Challenges Potential Solutions
Critical Facility Commissioning in Existing BuildingsSecurity
BCxA Annual Conference – Chicagoland, IL – September 2019 56
Challenges
New spaces need security to be coordinated with existing spaces.
Access to these spaces may be difficult with Client requirements.
Potential Solutions
Engage security vendor/designer for existing spaces. Submittal coordination is critical
(existing vs. new install).
Always inform clients of ongoing work and utilize processes in place in order to gain
access to spaces.
Critical Facility Commissioning in Existing BuildingsInfrastructure
BCxA Annual Conference – Chicagoland, IL – September 2019 57
Required partial or localized PTP on the newly installed equipment.
Separate systems fed by the entire infrastructure.
Challenges: Pull-The-Plug (Blackout) Testing
Critical Facility Commissioning in Existing BuildingsInfrastructure
BCxA Annual Conference – Chicagoland, IL – September 2019 58
Required partial or localized PTP on the newly installed equipment.
Separate systems fed by the entire infrastructure.
Confirm which transfer switches feed new systems and lock out others.
Challenges: Pull-The-Plug (Blackout) Testing Potential Solutions
Critical Facility Commissioning in Existing BuildingsInfrastructure
BCxA Annual Conference – Chicagoland, IL – September 2019 59
Challenges: Pull-The-Plug (Blackout) Testing
Required partial or localized PTP on the newly installed equipment.
Separate systems fed by the entire infrastructure.
Potential Solutions
Confirm which transfer switches feed new systems and lock out others.
Confirm which transfer switches feed new systems and lock out others.
BCxA Annual Conference – Chicagoland, IL – September 2019 60
Chapter 3: FinSummary and Takeaways
BCxA Annual Conference – Chicagoland, IL – September 2019 61
• Lessons Learned:
Summary
BCxA Annual Conference – Chicagoland, IL – September 2019 62
• Key take-aways: Labeling
◦ UPS Boards◦ Breaker Panels◦ EPO◦ Pull Stations◦ Etc.
Summary
BCxA Annual Conference – Chicagoland, IL – September 2019 63
• Lessons Learned:
Summary
BCxA Annual Conference – Chicagoland, IL – September 2019 64
• Key take-aways: It takes a village.
◦ Don’t go at this yourself◦ Lots of overnight testing◦ Lots of writing and rewriting the test procedures◦ Avoid fatigue – leads to errors
Summary
BCxA Annual Conference – Chicagoland, IL – September 2019 65
• Key take-aways: Owner required challenges. Service level agreement challenges. Existing building data centers more challenging than new data centers
due to potential for outage (power or cooling).
Summary
BCxA Annual Conference – Chicagoland, IL – September 2019 66
• Key take-aways: As data center launch nears, pressure increases. Coordination and communication among all parties involved in the
commissioning process is crucial. Existing building processes/standards must be followed at all points of
commissioning.
Summary
Ryan S. LeanAssociate PartnerJaros, Baum & Bolles212.530.9447 | [email protected]
67
Phil KarnickiFacilities ManagerCBRE DCS732.740.6661 | [email protected]